Prerequisites for Discovery on AWS¶

Privacera Discovery module leverages AWS services such as S3, DynamoDB, and SQS for scanning data. To enable this, you need to create the necessary AWS resources and IAM roles for the Discovery and Portal pods to access these resources. Privacera Manager can create these resources for you or you can create them manually and provide their ARNs during the installation of the Discovery module.

Here are the prerequisites for setting up Privacera Discovery on AWS:

AWS S3 bucket and path¶

An AWS S3 bucket and path is required to store the configuration for Privacera Discovery. It is recommended to create a dedicated bucket for Privacera Discovery. You can use a sub-folder in the bucket to store the configurations and temporary files for Discovery. You need to make sure that the IAM roles for the Discovery and Portal pods have read/write access to this bucket and/or the sub-folder.

You will need to provide the bucket name and path to Privacera Manager during the installation configuration. This bucket can be created manually or let Privacera Manager create it for you.

Examples are (replace acme or acme-prod with your bucket name):

1. Separate bucket for each environment:

   s3://acme-prod/privacera-discovery-config/privacera-prod

2. Common bucket, but different folders for each environment:

   s3://acme/privacera-discovery-config/privacera-prod

AWS DynamoDB tables¶

AWS DynamoDB tables are required to store the metadata for Privacera Discovery. The recommended naming convention for these tables are privacera_*_DEPLOYMENT_ENV_NAME. You can create these tables manually or let Privacera Manager create them for you.

Assuming your DEPLOYMENT_ENV_NAME is privacera-prod, the table names could be suffixed with privacera-prod as shown above. If you are manually creating the tables, you can use the following naming convention and schema, but replace the DEPLOYMENT_ENV_NAME with your actual deployment environment name.

The table names and their corresponding hash key, and range key are as follows:

AWS SQS¶

An AWS SQS queue is used for real-time scanning. The change events for the S3 objects are retrieved from the SQS queue and processed by the Discovery pods. Privacera Manager can create these for you or you can create them manually. The recommended naming convention for the SQS queue is privacera_bucket_sqs_DEPLOYMENT_ENV_NAME.

Examples are (replace privacera-prod with your DEPLOYMENT_ENV_NAME):

privacera_bucket_sqs_privacera-prod

IAM Policies For Discovery¶

There are 2 sets of IAM policies required for Privacera Discovery.

1. For Privacera Manager: Permissions to create the AWS resources required for Privacera Discovery by Privacera Manager. This is optional. You can have Privacera Manager create it during installation or you can create these resources manually and provide their ARNs during installation of the Discovery module. These IAM policies should be attached to the EC2 instance where Privacera Manager is running.
2. For Discovery Services: IAM roles for the Discovery and Portal pods to access the AWS resources. This is mandatory for scanning AWS services such as S3 and DynamoDB. These roles need to be created manually and configured in the Privacera Manager during installation. You can limit the access to only the required resources that will be scanned by Discovery.

Step 1: IAM Role for Privacera Manager¶

The following additional IAM policy must be attached to the Privacera Manager EC2 instance to enable the creation of AWS resources for Discovery, such as DynamoDB tables, S3 buckets, and SQS queues.

The below IAM policies provide the create and update policies for the following AWS Resources:

1
2
3

Resource":"arn:aws:dynamodb:<AWS_REGION>:<ACCOUNT_ID>:table/privacera*
Resource":"arn:aws:s3:::<DISCOVERY_BUCKET>"
Resource":"arn:aws:sqs:<AWS_REGION>:<ACCOUNT_ID>:privacera*"

After you created the IAM policies, you can attach them to the role used by your Privacera Manager EC2 instance. (e.g. privacera-manager-role-privacera-prod)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

     {
    "Version":"2012-10-17",
    "Statement":[
        {
            "Sid":"CreateDynamodb",
            "Effect":"Allow",
            "Action":[
                "dynamodb:CreateTable",
                "dynamodb:DescribeTable",
                "dynamodb:ListTables",
                "dynamodb:TagResource",
                "dynamodb:UntagResource",
                "dynamodb:UpdateTable",
                "dynamodb:UpdateTableReplicaAutoScaling",
                "dynamodb:UpdateTimeToLive",
                "dynamodb:DescribeTimeToLive",
                "dynamodb:ListTagsOfResource",
                "dynamodb:DescribeContinuousBackups"
            ],
            "Resource":"arn:aws:dynamodb:<AWS_REGION>:<ACCOUNT_ID>:table/privacera*"
        },
        {
            "Sid":"CreateS3Bucket",
            "Effect":"Allow",
            "Action":[
                "s3:CreateBucket",
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation"

            ],
            "Resource":[
                "arn:aws:s3:::<DISCOVERY_BUCKET>"
            ]
        },
        {
            "Sid":"CreateSQSMessages",
            "Effect":"Allow",
            "Action":[
                "sqs:CreateQueue",
                "sqs:ListQueues"
            ],
            "Resource":[
                "arn:aws:sqs:<AWS_REGION>:<ACCOUNT_ID>:privacera*"
            ]
        }
    ]
    }

Step 2: IAM Role for Discovery Services¶

Privacera Discovery runs on Apache Spark, and its pods require access to AWS resources to scan data. The IAM roles for the Discovery and Portal pods must be created manually and configured in Privacera Manager during installation.

The Discovery and Portal pods require the following IAM policies to access the AWS resources. The recommendation is to create these policies and attach them to the IAM roles for the Discovery and Portal pods.

Here are the recommended IAM Role names and the policies to be attached to them:

1. Role Name: privacera-discovery-role-DEPLOYMENT_ENV_NAME (e.g. privcera-discovery-role-privacera-prod)
2. Discovery Service Policies: privacera-discovery-service-policies-DEPLOYMENT_ENV_NAME (e.g. privacera-discovery-service-policies-privacera-prod)
3. Discovery Scan Policies: privacera-discovery-scan-policies-DEPLOYMENT_ENV_NAME (e.g. privacera-discovery-scan-policies-privacera-prod)

The above role will be attached to the following pods (DEPLOYMENT_ENV_NAME will be replaced with your actual deployment environment name):

1. Privacera Portal (In Self-Managed Deployment) or Privacera Discovery Admin Console (In PrivaceraCloud Data Plane Deployment)
2. Discovery Driver and Executor pods
3. Discover pKafka pod (If real-time scanning is enabled)

graph TD

    subgraph RolesAndPolicies
        A[privacera-discovery-role]
        B[privacera-discovery-service-policies]
        C[privacera-discovery-scan-policies]
    end

    subgraph Pods
        D[Portal or\nDiscovery Admin Console]
        E[Discovery Spark]
        F[pKafka]
    end


    B --> A
    C --> A

    A --> D
    A --> E
    A --> F

a. Discovery Service Policies¶

1
2
3

Resource":"arn:aws:dynamodb:<AWS_REGION>:<ACCOUNT_ID>:table/privacera*
Resource":"arn:aws:s3:::<DISCOVERY_BUCKET>"
Resource":"arn:aws:sqs:<AWS_REGION>:<ACCOUNT_ID>:privacera*"

After you created the IAM policies, you can attach them to the role used by your Discovery and Portal pods. (e.g. privacera-discovery-role-privacera-prod)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Dynamodb",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGet*",
                "dynamodb:DescribeStream",
                "dynamodb:DescribeTable",
                "dynamodb:Get*",
                "dynamodb:Query",
                "dynamodb:Scan",
                "dynamodb:BatchWrite*",
                "dynamodb:Update*",
                "dynamodb:Put*"
            ],
            "Resource": "arn:aws:dynamodb:<AWS_REGION>:<ACCOUNT_ID>:table/privacera_*_DEPLOYMENT_ENV_NAME"
        },
        {
            "Sid": "S3ObjectAllpermissions",
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Put*",
                "s3:Get*",
                "s3:Delete*"
            ],
            "Resource": [
                "arn:aws:s3:::<DISCOVERY_BUCKET>/PATH/*",
                "arn:aws:s3:::<DISCOVERY_BUCKET>"
            ]
        },
        {
            "Sid": "SQSObjectAllpermissions",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues",
                "sqs:SendMessage",
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Resource": [
                "arn:aws:sqs:<AWS_REGION>:<ACCOUNT_ID>:privacera_*_DEPLOYMENT_ENV_NAME"
            ]
        }
    ]
}

b. Discovery Scan Policies¶

It is highly recommended to set this up during the installation phase, but if you won't be scanning data in S3, then you can skip this step.

1
2
3
4

Resource":"arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME1>/*"
Resource":"arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME1>"
Resource":"arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME2>/*"
Resource":"arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME2>"

After you created the IAM policies, you can attach them to the role used by your Discovery and Portal pods. (e.g. privacera-discovery-role-privacera-prod)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3ObjectScanOnlypermissions",
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": [
                "arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME1>/*",
                "arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME1>",
                "arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME2>/*",
                "arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME2>"
            ]
        }
    ]
}

c. Assign IAM roles to EKS Service Accounts¶

The IAM role (privacera-discovery-role-privacera-prod) created above should be assigned to the EKS service accounts for the Discovery and Portal pods.

1
2
3
4

discovery-consumer-privacera-sa
discovery-privacera-sa
portal-privacera-sa
pkafka-privacera-sa

You can follow the instructions here for creating the IAM role for service accounts.

Final Checklist¶

• Create IAM policies and roles for Privacera Manager to create AWS resources required for Privacera Discovery (optional).
• Create an S3 bucket and path for storing the configurations and temporary files for Privacera Discovery or let Privacera Manager create it for you.
• Create DynamoDB tables to store metadata and tags or let Privacera Manager create them for you.
• Create an SQS queue for real-time scanning or let Privacera Manager create it for you.
• Create IAM policies and roles for the Discovery and Portal pods to access the AWS resources.
• Create IAM policies for the Discovery and Portal pods to scan the S3 bucket (optional).
• Assign the IAM roles to the EKS Service Accounts for the Discovery and Portal pods.

Comments