Prerequisites for Discovery on AWS¶

Privacera Discovery module leverages AWS services such as S3, DynamoDB, and SQS for scanning data. To enable this, you need to create the necessary AWS resources and IAM roles for the Discovery and Portal pods to access these resources. Privacera Manager can create these resources for you or you can create them manually and provide their ARNs during the installation of the Discovery module.

Here are the prerequisites for setting up Privacera Discovery on AWS:

AWS S3 bucket and path¶

An AWS S3 bucket and path is required to store the configuration for Privacera Discovery. It is recommended to create a dedicated bucket for Privacera Discovery. You can use a sub-folder in the bucket to store the configurations and temporary files for Discovery. You need to make sure that the IAM roles for the Discovery and Portal pods have read/write access to this bucket and/or the sub-folder.

You will need to provide the bucket name and path to Privacera Manager during the installation configuration. This bucket can be created manually or let Privacera Manager create it for you.

Examples are (replace acme or acme-prod with your bucket name):

1. Separate bucket for each environment:

DISCOVERY_BUCKET_NAME: s3://acme-prod/privacera-discovery-config/privacera-prod

1. Common bucket, but different folders for each environment:

DISCOVERY_BUCKET_NAME: s3://acme/privacera-discovery-config/privacera-prod

AWS DynamoDB tables¶

AWS DynamoDB tables are required to store the metadata for Privacera Discovery. During installation, Privacera Discovery will create the DynamoDB tables. If you are creating the tables manually, you can use the following naming convention and schema, but replace <DEPLOYMENT_ENV_NAME> with your actual deployment environment name.

The table names and their corresponding hash key, and range key are as follows:

AWS SQS¶

An AWS SQS queue is used for real-time scanning. The change events for the S3 objects are retrieved from the SQS queue and processed by the Discovery pods. Privacera Manager can create these for you or you can create them manually. The recommended naming convention for the SQS queue is privacera_bucket_sqs_DEPLOYMENT_ENV_NAME.

Examples are (replace privacera-prod with your DEPLOYMENT_ENV_NAME):

privacera_bucket_sqs_privacera-prod

IAM Policies For Discovery¶

There are 2 sets of IAM policies required for Privacera Discovery.

1. For Privacera Manager: Permissions to create the AWS resources required for Privacera Discovery by Privacera Manager. This is optional. You can have Privacera Manager create it during installation or you can create these resources manually and provide their ARNs during installation of the Discovery module. These IAM policies should be attached to the EC2 instance where Privacera Manager is running.
2. For Discovery Services: IAM roles for the Discovery and Portal pods to access the AWS resources. This is mandatory for scanning AWS services such as S3 and DynamoDB. These roles need to be created manually and configured in the Privacera Manager during installation. You can limit the access to only the required resources that will be scanned by Discovery.

Step 1: IAM Role for Privacera Manager¶

The following additional IAM policy must be attached to the Privacera Manager EC2 instance to enable the creation of AWS resources for Discovery, such as DynamoDB tables, S3 buckets, and SQS queues.

The below IAM policies provide the create and update policies for the following AWS Resources:

1
2
3

Resource":"arn:aws:dynamodb:<AWS_REGION>:<ACCOUNT_ID>:table/privacera*
Resource":"arn:aws:s3:::<DISCOVERY_BUCKET_NAME>"
Resource":"arn:aws:sqs:<AWS_REGION>:<ACCOUNT_ID>:privacera*"

After you created the IAM policies, you can attach them to the role used by your Privacera Manager EC2 instance. (e.g. privacera-manager-role-privacera-prod)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

     {
    "Version":"2012-10-17",
    "Statement":[
        {
            "Sid":"CreateDynamodb",
            "Effect":"Allow",
            "Action":[
                "dynamodb:CreateTable",
                "dynamodb:DescribeTable",
                "dynamodb:ListTables",
                "dynamodb:TagResource",
                "dynamodb:UntagResource",
                "dynamodb:UpdateTable",
                "dynamodb:UpdateTableReplicaAutoScaling",
                "dynamodb:UpdateTimeToLive",
                "dynamodb:DescribeTimeToLive",
                "dynamodb:ListTagsOfResource",
                "dynamodb:DescribeContinuousBackups"
            ],
            "Resource":"arn:aws:dynamodb:<AWS_REGION>:<ACCOUNT_ID>:table/privacera*"
        },
        {
            "Sid":"CreateS3Bucket",
            "Effect":"Allow",
            "Action":[
                "s3:CreateBucket",
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation"

            ],
            "Resource":[
                "arn:aws:s3:::<DISCOVERY_BUCKET_NAME>"
            ]
        },
        {
            "Sid":"CreateSQSMessages",
            "Effect":"Allow",
            "Action":[
                "sqs:CreateQueue",
                "sqs:ListQueues"
            ],
            "Resource":[
                "arn:aws:sqs:<AWS_REGION>:<ACCOUNT_ID>:privacera*"
            ]
        }
    ]
    }

Step 2: IAM Role for Discovery Services¶

Privacera Discovery runs on Apache Spark, and its pods require access to AWS resources to scan data. The IAM roles for the Discovery and Portal pods must be created manually and configured in Privacera Manager during installation.

The Discovery and Portal pods require the following IAM policies to access the AWS resources. The recommendation is to create these policies and attach them to the IAM roles for the Discovery and Portal pods.

Here are the recommended IAM Role names and the policies to be attached to them:

1. Role Name: privacera-discovery-role-DEPLOYMENT_ENV_NAME (e.g. privcera-discovery-role-privacera-prod)
2. Discovery Service Policies: privacera-discovery-service-policies-DEPLOYMENT_ENV_NAME (e.g. privacera-discovery-service-policies-privacera-prod)
3. Discovery Scan Policies: privacera-discovery-scan-policies-DEPLOYMENT_ENV_NAME (e.g. privacera-discovery-scan-policies-privacera-prod)

The above role will be attached to the following pods (DEPLOYMENT_ENV_NAME will be replaced with your actual deployment environment name):

1. Privacera Portal (In Self-Managed Deployment) or Privacera Discovery Admin Console (In PrivaceraCloud Data Plane Deployment)
2. Discovery Driver and Executor pods
3. Discover pKafka pod (If real-time scanning is enabled)

Privacera Manager assigns the IAM role (privacera-discovery-role-privacera-prod) to the EKS service accounts for the relevant pods during installation.

Following kubernetes service accounts will be created by Privacera Manager during the installation process.

1
2
3
4

discovery-consumer-privacera-sa
discovery-privacera-sa
portal-privacera-sa
pkafka-privacera-sa

graph TD

    subgraph RolesAndPolicies
        A[privacera-discovery-role]
        B[privacera-discovery-service-policies]
        C[privacera-discovery-scan-policies]
    end

    subgraph Pods
        D[Portal or\nDiscovery Admin Console]
        E[Discovery Spark]
        F[pKafka]
    end


    B --> A
    C --> A

    A --> D
    A --> E
    A --> F

a. Discovery Service Policies¶

1
2
3

Resource":"arn:aws:dynamodb:<AWS_REGION>:<ACCOUNT_ID>:table/privacera*
Resource":"arn:aws:s3:::<DISCOVERY_BUCKET_NAME>"
Resource":"arn:aws:sqs:<AWS_REGION>:<ACCOUNT_ID>:privacera*"

After you created the IAM policies, you can attach them to the role used by your Discovery and Portal pods. (e.g. privacera-discovery-role-privacera-prod)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Dynamodb",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGet*",
                "dynamodb:DescribeStream",
                "dynamodb:DescribeTable",
                "dynamodb:Get*",
                "dynamodb:Query",
                "dynamodb:Scan",
                "dynamodb:BatchWrite*",
                "dynamodb:Update*",
                "dynamodb:Put*"
            ],
            "Resource": "arn:aws:dynamodb:<AWS_REGION>:<ACCOUNT_ID>:table/privacera_*_<DEPLOYMENT_ENV_NAME>"
        },
        {
            "Sid": "S3ObjectAllpermissions",
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Put*",
                "s3:Get*",
                "s3:Delete*"
            ],
            "Resource": [
                "arn:aws:s3:::<DISCOVERY_BUCKET_NAME>/PATH/*",
                "arn:aws:s3:::<DISCOVERY_BUCKET_NAME>"
            ]
        },
        {
            "Sid": "S3ObjectListAllpermissions",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        },
        {
            "Sid": "SQSObjectAllpermissions",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues",
                "sqs:SendMessage",
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Resource": [
                "arn:aws:sqs:<AWS_REGION>:<ACCOUNT_ID>:privacera_*_<DEPLOYMENT_ENV_NAME>"
            ]
        }
    ]
}

b. Discovery Scan Policies¶

It is highly recommended to set this up during the installation phase, but if you won't be scanning data in S3, then you can skip this step.

1
2
3
4

Resource":"arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME1>/*"
Resource":"arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME1>"
Resource":"arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME2>/*"
Resource":"arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME2>"

After you created the IAM policies, you can attach them to the role used by your Discovery and Portal pods. (e.g. privacera-discovery-role-privacera-prod)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3ObjectScanOnlypermissions",
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": [
                "arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME1>/*",
                "arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME1>",
                "arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME2>/*",
                "arn:aws:s3:::<DISCOVERY_SCAN_BUCKET_NAME2>"
            ]
        }
    ]
}

c. Add trust relationship to the IAM role¶

The IAM role (privacera-discovery-role-privacera-prod) must be updated with a trust relationship condition to allow access.

To get Cluster’s OpenID to update the trust relationship, you can follow the below steps:

1. Go to AWS Console > Amazon Elastic Kubernetes Service.
2. Click on your cluster name.
3. Copy the OpenID Connect Provider URL (e.g. https://oidc.eks.us-east-1.amazonaws.com/id/D3B53XXXXXXXXXXXXXXXXXXXXD4357F).
4. Replace the id in federated section with the OpenID connect number.(e.g. arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>)

For more information on OIDC, refer to the OpenID Connect documentation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
           "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>:aud": "sts.amazonaws.com",
            "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>:sub": "system:serviceaccount:<DEPLOYMENT_ENV_NAME>:discovery-privacera-sa"
        }
      }
    },
    {
        "Effect": "Allow",
        "Principal": {
            "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>:aud": "sts.amazonaws.com",
            "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>:sub": "system:serviceaccount:<DEPLOYMENT_ENV_NAME>:discovery-consumer-privacera-sa"
        }
      }
    },
    {
        "Effect": "Allow",
        "Principal": {
            "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>:aud": "sts.amazonaws.com",
            "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>:sub": "system:serviceaccount:<DEPLOYMENT_ENV_NAME>:portal-privacera-sa"
        }
      }
    },
    {
        "Effect": "Allow",
        "Principal": {
            "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>:aud": "sts.amazonaws.com",
            "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OPEN_ID_CONNECT_NUMBER>:sub": "system:serviceaccount:<DEPLOYMENT_ENV_NAME>:pkafka-privacera-sa"
        }
      }
    },
    {
        "Sid": "AssumeRole",
        "Effect": "Allow",
        "Principal": {
        "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/<DISCOVERY_ROLE_NAME>"
    },
    "Action": "sts:AssumeRole"
   }
 ]
}

Final Checklist¶

• Create IAM policies and roles for Privacera Manager to create AWS resources required for Privacera Discovery (optional).
• Create an S3 bucket and path for storing the configurations and temporary files for Privacera Discovery or let Privacera Manager create it for you.
• Create DynamoDB tables to store metadata and tags or let Privacera Manager create them for you.
• Create an SQS queue for real-time scanning or let Privacera Manager create it for you.
• Create IAM policies and roles for the Discovery and Portal pods to access the AWS resources.
• Create IAM policies for the Discovery and Portal pods to scan the S3 bucket (optional).
• Add trust relationship to the IAM role

And you have the values for the following placeholders:

• DISCOVERY_BUCKET_NAME: Discovery configuration bucket name and path.
• DynamoDB table names for storing metadata and tags (only if you have created the tables manually).
  • SCAN_REQUEST_TABLE: DynamoDB table name for storing scan requests
  • RESOURCE_TABLE: DynamoDB table name for storing resource metadata
  • ALERT_TABLE: DynamoDB table name for storing alerts
  • AUDIT_SUMMARY_TABLE: DynamoDB table name for storing audit summary
  • ACTIVE_SCANS_TABLE: DynamoDB table name for storing active scans
  • STATE_TABLE: DynamoDB table name for storing state
• DISCOVERY_BUCKET_SQS_NAME: Amazon SQS Queue name (only if you have created the queue manually).
• IAM Role For Discovery and Portal Pods: ARN of the IAM role created for Discovery driver, executor and Portal pods. You can set the same value as privacera-discovery-role-privacera-prod from prerequisites.

Comments