Skip to main content

Privacera Platform

AWS IAM
:

When running in AWS, the Privacera Manager host virtual machine requires privileges in order to complete the deployment of Privacera Platform components. Additionally, once installed, Privacera Platform components will also require privileges to provide access to targeted data repositories and in order to execute. The specific access required will depend on the functions requested and the scope of data coverage requested.

AWS uses a 'policy/role/object' paradigm known as "AWS Identity and Access Management" (IAM) in order to assign and manage access and functionality rights. Roles and policies are both IAM objects, but are created and managed somewhat independently of each other. Access and rights are defined in one or more policies. Policies are then attached to Roles. A Role, may be attached to a user account or an instance. When attached to an instance the role is known as an instance profile.

Policies may be created using the AWS console, or aws command line. They can be represented, stored, imported, or exported in JSON format. This document contains a library of Policies. In this guide, to create a recommended policy, you will select a policy from the Privacera Manager policy library, import/copy it to the console, modify it to meet your specific enterprise requirements, and save it as a named policy.

In a subsequent step, you will attach one or more of these policies to a Role, and then to the Privacera Manager host.

AWS IAM role and attach policy(s)

  1. In AWS Console. Open IAM Services.

  2. Click Roles on the left side navigation and then click Create role

  3. Create Role: Choose a use case. Select 'EC2' use case. (This "Allows EC2 instances to call AWS services on 'your' behalf").

    Click Next: Permissions to transition to the next wizard page.

  4. Create Role: Attach permissions policies. Using Filter policies, search for the previously created policy (e.g. 'privacera_s3_all'). Select it, (click in the checkbox).

    Click Next: Tags.

  5. Create Role: Add tags. Optionally add a tag based on your enterprise resource tag standards. Click Next: Review.

  6. Create Role: Review. Enter a Role name such as 'privacera_s3_role'. Click Create role.

  7. Confirm the Role has been created by searching for it in the Role list.

AWS IAM create and attach policy

  1. In AWS Console. Open IAM Services.

  2. Click Policies on the left side navigation and then click Create Policy.

  3. Click on the JSON tab.

  4. Select a Policy from the list below, copy and paste it into the JSON edit box in the AWS Create policy dialog. Click Review policy (at the bottom of the page).

    Full S3 Access - All Buckets

                                  {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DataServerS3FullAccess",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

    Limited S3 Access - Limited Buckets

    Note

    (That in this example policy accessible buckets are represented as "<PLEASE_ASSIGN_BUCKET_NAME_x>". Assign or adjust this sample policy for your enterprise and selected controlled S3 buckets.)

                                  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DataServerS3Limited",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObjectAcl",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:DeleteBucket",
        "s3:ListBucketMultipartUploads",
        "s3:GetBucketAcl",
        "s3:GetBucketPolicy",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:PutObjectAcl"
        ],
      "Resource": [
        "arn:aws:s3:::<PLEASE_ASSIGN_BUCKET_NAME_1>/*",
        "arn:aws:s3:::<PLEASE_ASSIGN_BUCKET_NAME_2>",
        "arn:aws:s3:::<PLEASE_ASSIGN_BUCKET_NAME_3>/*",
        "arn:aws:s3:::<PLEASE_ASSIGN_BUCKET_NAME_4>"
        ]
    },
    {
      "Sid": "DataServerS3ListAndCreateBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:HeadBucket",
        "s3:CreateBucket"
        ],
      "Resource": "*"
    }
  ]
}

    DynamoDB Access

                                  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DataserverDynamoDBAccess",
      "Effect": "Allow",
      "Action": [
          "dynamodb:Query",
          "dynamodb:PutItem",
          "dynamodb:DeleteItem",
          "dynamodb:Scan",
          "dynamodb:UpdateItem",
          "dynamodb:CreateTable",
          "dynamodb:DescribeTable",
          "dynamodb:DeleteTable",
          "dynamodb:UpdateTable",
          "dynamodb:GetItem",
          "dynamodb:CreateBackup",
          "dynamodb:BatchGetItem",
          "dynamodb:BatchWriteItem",
          "dynamodb:TagResource",
          "dynamodb:UntagResource"
      ],
      "Resource": [
          "*"
      ]
    },
    {
      "Sid": "DataserverDynamoDBAccessListing",
      "Effect": "Allow",
      "Action": [
          "dynamodb:ListTables",
          "dynamodb:ListBackups"
      ],
      "Resource": "*"
    }
  ]
}

    Kinesis Access

                                  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManageStreams",
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:DeleteStream",
        "kinesis:DescribeStreamSummary",
        "kinesis:CreateStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:DescribeStream",
        "kinesis:PutRecords",
        "kinesis:AddTagsToStream",
        "kinesis:DecreaseStreamRetentionPeriod",
        "kinesis:IncreaseStreamRetentionPeriod",
        "kinesis:ListTagsForStream",
        "kinesis:RemoveTagsFromStream",
        "kinesis:RegisterStreamConsumer",
        "kinesis:DeregisterStreamConsumer",
        "kinesis:DescribeStreamConsumer",
        "kinesis:ListStreamConsumers",
        "kinesis:DisableEnhancedMonitoring",
        "kinesis:EnableEnhancedMonitoring",
        "kinesis:UpdateShardCount",
        "kinesis:MergeShards",
        "kinesis:SplitShard",
        "kinesis:StartStreamEncryption",
        "kinesis:StopStreamEncryption",
        "kinesis:ListShards"
      ],
      "Resource": "*"
    },
    {
      "Sid": "KinesisListing",
      "Effect": "Allow",
      "Action": [
        "kinesis:ListStreams",
        "kinesis:DescribeLimits"
      ],
      "Resource": "*"
    }
  ]
}

    Firehose Access (Requires IAM Policy and Trust Relationship

                                  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DataserverKinesisFirehoseAccess",
      "Effect": "Allow",
      "Action": [
        "firehose:DescribeDeliveryStream",
        "firehose:DeleteDeliveryStream",
        "firehose:PutRecord",
        "firehose:CreateDeliveryStream",
        "firehose:UpdateDestination"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DataserverKinesisFirehoseListingAccess",
      "Effect": "Allow",
      "Action": "firehose:ListDeliveryStreams",
      "Resource": "*"
    },
    {
      "Sid": "DataserverKinesisFirehosePassRoleAccess",
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:PassRole"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DataserverKinesisFirehoseKinesisAccess",
      "Effect": "Allow",
      "Action": [
        "kinesis:GetShardIterator",
        "kinesis:DescribeStream",
        "kinesis:GetRecords"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DataserverKinesisS3Access",
      "Effect": "Allow",
      "Action": [
          "s3:PutObject",
          "s3:CreateBucket",
          "s3:ListBucket"
      ],
      "Resource": "*"
    }]
}

    Add a Trust Relationship for Role "privacera-access-role':

                                  {
"Sid": "DataserverKinesisAssumeRole",
"Effect":"Allow",
"Principal":{
    "Service":"firehose.amazonaws.com"
},
"Action":"sts:AssumeRole"
}

    Lamda Access

                                  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DataserverLambdaManagementAccess",
      "Effect": "Allow",
      "Action": [
          "lambda:CreateFunction",
          "lambda:InvokeFunction",
          "lambda:GetEventSourceMapping",
          "lambda:GetFunction",
          "lambda:DeleteFunction",
          "lambda:DeleteEventSourceMapping"
      ],
      "Resource": [
          "*"
      ]
    },
    {
      "Sid": "DataserverLambdaManagementListing",
      "Effect": "Allow",
      "Action": [
          "lambda:ListFunctions",
          "lambda:ListEventSourceMappings",
          "lambda:CreateEventSourceMapping"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DataserverLambdaKinesisStreamRead",
        "Effect": "Allow",
        "Action": [
            "kinesis:SubscribeToShard",
            "kinesis:DescribeStreamSummary",
            "kinesis:GetShardIterator",
            "kinesis:GetRecords",
            "kinesis:DescribeStream"
        ],
      "Resource": "*"
    },
    {
      "Sid": "DataserverLambdaKinesisListing",
      "Effect": "Allow",
      "Action": [
          "kinesis:ListStreams",
          "kinesis:ListShards"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DataserverLambdaS3BucketsListing",
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    }
  ]
}

    Add a Trust Relationship for AWS IAM role 'privacera-access-role'

                                  {
"Sid": "DataserverLambdaAssumeRole",
"Effect":"Allow",
"Principal":{
    "Service":"lambda.amazonaws.com"
},
"Action":"sts:AssumeRole"
}

    Athena Access

                                  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DataServerAthenaAccess",
      "Effect": "Allow",
      "Action": [
        "athena:TagResource",
        "athena:UntagResource",
        "athena:StartQueryExecution",
        "athena:GetQueryResultsStream",
        "athena:DeleteWorkGroup",
        "athena:GetQueryResults",
        "athena:DeleteNamedQuery",
        "athena:UpdateWorkGroup",
        "athena:GetNamedQuery",
        "athena:CreateWorkGroup",
        "athena:ListTagsForResource",
        "athena:ListQueryExecutions",
        "athena:ListNamedQueries",
        "athena:GetWorkGroup",
        "athena:CreateNamedQuery",
        "athena:GetQueryExecution",
        "athena:StopQueryExecution",
        "athena:BatchGetNamedQuery",
        "athena:BatchGetQueryExecution"
      ],
      "Resource": [
        "arn:aws:athena:*:*:workgroup/primary"
      ]
    },
    {
      "Sid": "DataServerAthenaGlue",
      "Effect": "Allow",
      "Action": [
          "glue:CreateDatabase",
          "glue:DeleteDatabase",
          "glue:GetDatabase",
          "glue:GetDatabases",
          "glue:UpdateDatabase",
          "glue:CreateTable",
          "glue:DeleteTable",
          "glue:BatchDeleteTable",
          "glue:UpdateTable",
          "glue:GetTable",
          "glue:GetTables",
          "glue:BatchCreatePartition",
          "glue:CreatePartition",
          "glue:DeletePartition",
          "glue:BatchDeletePartition",
          "glue:UpdatePartition",
          "glue:GetPartition",
          "glue:GetPartitions",
          "glue:BatchGetPartition",
          "glue:GetCatalogImportStatus"
      ],
      "Resource": [
          "*"
      ]
    }
  ]
}

    Glue Access

                                  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:CreateDatabase",
        "glue:DeleteDatabase",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:UpdateDatabase",
        "glue:CreateTable",
        "glue:DeleteTable",
        "glue:BatchDeleteTable",
        "glue:UpdateTable",
        "glue:GetTable",
        "glue:GetTables",
        "glue:BatchCreatePartition",
        "glue:CreatePartition",
        "glue:DeletePartition",
        "glue:BatchDeletePartition",
        "glue:UpdatePartition",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:BatchGetPartition",
        "glue:GetCatalogImportStatus"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

  5. In Create policy: Review Policy, give each policy a descriptive name, such as 'privacera_s3_all_policy', or 'privacera_s3_limited_policy'. Suggested practice is to use 'privacera_' as a prefix for each policy created for Privacera Manager or Privacera Platform.

  6. Click Create policy at the bottom of the dialog.

Attach Policy to Privacera Host IAM Role

In your initial creation of the Privacera Host VM, you created a role, with the suggested name: "Privacera_PM_Role". Note that as this role is already attached to the Privacera Host virtual machine, it will convey any attached policy rights to the Privacera Host.

  1. If you are not already in the AWS console at IAM: Policies, open it now:

    1. In AWS Console. Open IAM Services.

    2. Click Policies on the left side navigation and then click Create Policy.

  2. Locate the Policy(s) to be attached by searching for each by name in the Create Policy dialog. (Use a substring such as "privacera" to find all with this name prefix.)

  3. Select a Policy to attach by clicking on the 'radio' button to the left of the policy name.

  4. Click the Policy actions menu at the top of this dialog. Select Attach. This will open the Attach policy dialog.

  5. Select the Privacera_PM_Role, and Attach Policy (at the bottom of the dialog). This will attach the policy to the Privacera_PM_Role, and those rights will be conveyed to the Privacera Manager Host virtual machine.

AWS IAM role and policy for Databricks
Add S3 IAM role to Databricks

  1. Login to Databricks and click on top-right menu.

  2. Click the Admin Console.

  3. Click the IAM Roles tab.

  4. Click the +Add IAM Role.

  5. Enter the Instance Profile ARN which you have created in step 1 Create IAM Role and Policy to Access S3 Bucket

    [image64.jpg]

    Databricks validates that this Instance Profile ARN is both syntactically and semantically correct. To validate semantic correctness, Databricks does a dry run by launching a cluster with this IAM role. Any failure in this dry run produces a validation error in the UI.

  6. Click Add.

  7. You can specify the users who can launch clusters with the IAM role. (Optional)

    [image65.jpg]

Launch Cluster with S3 IAM Role

  1. Login to Databricks and click the Clusters from left menu.

  2. Select or create a cluster.

  3. Expand the Advanced Options section, under Instances tab, select the IAM role from the IAM Role drop-down list. This drop-down includes all of the IAM roles that are available for the cluster.

    [image66.jpg]

PostgreSQL PolicySync
Lambda Setup for PostgreSQL Audits

This AWS Lambda function will send the audits from AWS CloudWatch to SQS queue.

Create an Audit policy

Create a policy to be attached while creating an AWS Lambda function (discussed below) to send audit information to the SQS Queue.

  1. Login to AWS Console and go to the Policies section from IAM Service.

  2. Click on Create Policy and go to the JSON tab.

  3. Copy the policy below and enter it in the JSON textbox.

                                     {
  "Version":"2012-10-17",
  "Statement":[
      {
        "Effect":"Allow",
        "Action":"logs:CreateLogGroup",
        "Resource":"arn:aws:logs:${REGION}:${ACCOUNT_ID}:*"
      },
      {
        "Effect":"Allow",
        "Action":[
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource":[
            "arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/lambda/${LAMBDA_FUNCTION_NAME}:*"
        ]
      },
      {
        "Effect":"Allow",
        "Action":"sqs:SendMessage",
        "Resource":"arn:aws:sqs:${REGION}:${ACCOUNT_ID}:${SQS_QUEUE_NAME}"
      }
  ]
}

  4. Click Review Policy.

  5. Enter a name for the policy. For example, privacera-postgres-audits-lambda-execution-policy.

  6. Click Create Policy.

Create an IAM Role

  1. In the AWS Console, go to Roles.

  2. Click Create Role.

  3. Select Lamda as the use case, and click Next Permissions.

  4. In Attach permi ssion policies, search for the policy created above, and select it.

  5. Click Next: Tags.

  6. Click Next: Review.

  7. Give a name for the role. For example, privacera-postgres-audits-lambda-execution-role.

  8. Click Create Role.

Create Lambda Function

  1. In the AWS Console, go to the Lambda service.

  2. Click Create Function.

  3. Configure the following in Basic Information:

    • Name: privacera-postgres-${RDS_CLUSTER_NAME}-audits

    • Runtime: Node.js 12.x

  4. In Choose or create an execution role, select Use existing role.

  5. Search and select for the IAM role created above.

  6. Click Create Function.

  7. In the Designer view, click Add Trigger.

    • Select CloudWatch Logs.

    • In the Log group, enter ${YOUR_RDS_LOG_GROUP}.

    • In the Filter name, add auditTrigger.

  8. Click Add.

  9. Access the Lambda Code Editor, and add the following code.

    // CloudWatch logs encoding
var encoding = process.env.ENCODING || 'utf-8';  // default is utf-8
var awsRegion = process.env.REGION || 'us-east-1';
var sqsQueueURL = process.env.SQS_QUEUE_URL;
var ignoreDatabase = process.env.IGNORE_DATABASE;
var ignoreUsers = process.env.IGNORE_USERS;

var ignoreDatabaseArray = ignoreDatabase.split(',');
var ignoreUsersArray = ignoreUsers.split(',');

// Import the AWS SDK
const AWS = require('aws-sdk');

// Configure the region
AWS.config.update({region: awsRegion});

exports.handler = function (event, context, callback) {

    var zippedInput = Buffer.from(event.awslogs.data, 'base64');

        zlib.gunzip(zippedInput, function (e, buffer) {
        if (e) {
            callback(e);
        }

        var awslogsData = JSON.parse(buffer.toString(encoding));

        // Create an SQS service object
        const sqs = new AWS.SQS({apiVersion: '2012-11-05'});

        console.log(awslogsData);
        if (awslogsData.messageType === 'DATA_MESSAGE') {

            // Chunk log events before posting
            awslogsData.logEvents.forEach(function (log) {

                //// Remove any trailing \n
                console.log(log.message)

                // Checking if message falls under ignore users/database
                var sendToSQS = true;

                if(sendToSQS) {

                    for(var i = 0; i < ignoreDatabaseArray.length; i++) {
                      if(log.message.toLowerCase().indexOf("@" + ignoreDatabaseArray[i]) !== -1) {
                            sendToSQS = false;
                            break;
                      }
                    }
                }

                if(sendToSQS) {

                    for(var i = 0; i < ignoreUsersArray.length; i++) {
                      if(log.message.toLowerCase().indexOf(ignoreUsersArray[i] + "@") !== -1) {
                            sendToSQS = false;
                            break;
                      }
                    }
                }

                if(sendToSQS) {

                    let sqsOrderData = {
                        MessageBody: JSON.stringify(log),
                        MessageDeduplicationId: log.id,
                        MessageGroupId: "Audits",
                        QueueUrl: sqsQueueURL
                    };

                    // Send the order data to the SQS queue
                    let sendSqsMessage = sqs.sendMessage(sqsOrderData).promise();

                    sendSqsMessage.then((data) => {
                        console.log("Sent to SQS");
                    }).catch((err) => {
                        console.log("Error in Sending to SQS = " + err);
                    });

                }
            });
        }
    });
};

  10. In the Code Editor of the Lambda code, go to Environment Variables > Manage Environment Variables > Add environment variables and set the following variables.

    • REGION: ${REGION}

    • SQS_QUEUE_URL: ${SQS_QUEUE_URL}

    • IGNORE_DATABASE: ${POSTGRESQL_DB}

    • IGNORE_USERS: ${POSTGRES_ADMIN_USER}

  11. Click Save. It saves the environment variables.

  12. In Designer view, click Save.

IAM Role for EC2

Create the following IAM Policy and associate it with the IAM role attached to the EC2 instance where PolicySync is installed.

                           {
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "sqs:DeleteMessage",
            "sqs:GetQueueUrl",
            "sqs:ListDeadLetterSourceQueues",
            "sqs:ReceiveMessage",
            "sqs:GetQueueAttributes"
         ],
         "Resource":"${SQS_QUEUE_ARN}"
      },
      {
         "Effect":"Allow",
         "Action":"sqs:ListQueues",
         "Resource":"*"
      }
   ]
}