Skip to main content

Privacera Documentation

Table of ContentsTable of Contents

Multiple AWS accounts support in DataServer

Dataserver supports the use of multiple AWS account profiles at the same time. You can use Databricks to run Spark queries on data stored in buckets across multiple AWS accounts.

The following are the steps to configure multiple account for Dataserver:

  1. SSH to the instance where Privacera manager is installed.

  2. Copy vars.dataserver.aws.yml file from sample-vars to custom vars.

    cd ~/privacera/privacera-manager
cp config/sample-vars/vars.dataserver.aws.yml config/custom-vars/

  3. Open file in text editor.

    vi config/custom-vars/vars.dataserver.aws.yml

  4. Update the configuration properties in the vars.dataserver.aws.yml file.

    DATASERVER_AWS_MULTI_ACCOUNT_PROFILE_ENABLE: "true"
DATASERVER_AWS_PROFILE_NAMES: ”<PLEASE_CHANGE>”
DATASERVER_AWS_PROFILE_NAME_DEFAULT: ”<PLEASE_CHANGE>”

    Property

    Description

    DATASERVER_AWS_MULTI_ACCOUNT_PROFILE_ENABLE

    Set the value of this property to true to Enable multi-account access in Privacera Dataserver.

    DATASERVER_AWS_PROFILE_NAMES

    Add the comma-separated profile names that need to be created. For example: dev,qa

    DATASERVER_AWS_PROFILE_NAME_DEFAULT

    Set the default profile name. For example: dev

  5. Set the following DATASERVER_AWS_PROFILE_PROPERTIES as needed for each profile:

    Example:

    DATASERVER_AWS_PROFILE_PROPERTIES:
  - PROFILE_NAME: "main"
  PROFILE_PROPERTIES:
    - AUTHN_METHOD: "CUSTOM_IAM"
    - IAM_ARN: "arn:aws:iam::xxxxxxxxxxxx"
    - REGION: "us-east-1"
    - S3_ENCRYPTION_ENABLE:"bucketA,bucketB*|SSE-C|jkabcxiuahsc"
  - PROFILE_NAME: "external"
  PROFILE_PROPERTIES:
    - AUTHN_METHOD: "CUSTOM_IAM"
    - IAM_ARN: "arn:aws:iam::xxxxxxxxxxxxxxxxxxxxxxx"
    - IAM_ROLE_EXTERNAL_ID: "test1234"
    - REGION: "us-east-1"

    Property

    Description

    AUTHN_METHOD

    This profile uses one of the following authentication methods:

    • ACCESS_KEY

    • CUSTOM_IAM

    • INSTANCE_IAM

    Depending on the AUTHN_METHOD, a few parameters are required and should have non-null values.

    If it is ACCESS_KEY, then the variables ACCESS KEY and SECRET KEY are required.

    If it is CUSTOM_IAM, then IAM ARN is a required property.

    If it is INSTANCE_IAM, nothing is required.

    IAM_ARN

    Add the custom IAM Role ARN.

    ACCESS_KEY

    If it is ACCESS_KEY, then it is a mandatory property.

    SECRET_KEY

    If it is ACCESS_KEY, then SECRET_KEY is mandatory property.

    REGION

    Set the AWS region.

    USE_REGIONAL_ENDPOINT

    By default the property is set to false.

    If the value is set to true, then it forcefully uses the value configured REGION property for all s3 requests.

    DEFAULT_DNS

    Set the default DNS server for AWS. The default value is amazonaws.com.

    IAM_ROLE_EXTERNAL_ID

    Set the External ID configured in the IAM role.

    STS_ROLE

    Set the AWS Security Token Service (STS) role.

    STS_DURATION_SECONDS

    Set the AWS STS duration in seconds. The default values is 3600.

    S3_SIGNED_URL_EXPIRE_MS

    Set the expire time for AWS S3 signed URL. The default value is 10 minutes.

    S3_EXECUTION_TIMEOUT_MS

    Set the AWS S3 client execution timeout in milliseconds. The default value is 60 * 1000.

    CUSTOM_S3_ENDPOINT_ENABLE

    Set the property to connect to a custom S3 endpoint such as, MINIO instead of AWS S3. The default value is false.

    CUSTOM_S3_ENDPOINT_HOST

    If the CUSTOM_S3_ENDPOINT_ENABLE property is set to true, then set the S3 endpoint host to connect to instead of AWS S3.

    CUSTOM_S3_ENDPOINT_PORT

    If the CUSTOM_S3_ENDPOINT_ENABLE property is set to true, then set the S3 endpoint port to connect to instead of AWS S3.

    CUSTOM_S3_ENDPOINT_SSL

    Set the value to true to enable S3 endpoint SSL.

    S3_ENCRYPTION_ENABLE

    Set the value to true to enable encryption for AWS S3 bucket. The default value is false.

    S3_ENCRYPTION_BUCKET_DATA

    If you need to provide data for multiple buckets and S3_ENCRYPTION_ENABLE is set to true, the data should be in the format:

    S3_ENCRYPTION_BUCKET_NAMES|S3_ENCRYPTION_SSETYPE|S3_ENCRYPTION_SSEKEY

    Example:

    "bucketA,bucketB*|SSE-C|jkabcxiuahsc" 
"bucketC|SSE-KMS|jkabcxiuahsc"

    Note

    SSE KEY is not required for SSE TYPE SSE-S3, but it is required for SSE TYPE SSE-C and optional for SSE TYPE SSE-KMS.

    ATHENA_S3_USE_S3POLICY

    Set this property to true for AWS Athena to use S3 policy.

    ATHENA_RESULTS_STORAGE_URL

    Set the bucket location for Athena results. For example: s3://bucketName/

    ATHENA_RESULTS_STORAGE_ENCRYPT_TYPE

    Set the Encryption options for bucket location for Athena results. For example: SSE_S3, SSE_KMS or CSE_KMS.

    ATHENA_RESULTS_STORAGE_ENCRYPT_KMS_KEY

    The KMS key ARN or ID is used if the value of the ATHENA_RESULTS_STORAGE_ENCRYPT_TYPE property is SSE_KMS or CSE_KMS.

    GLUE_USE_S3POLICY

    Set this property to true for AWS Glue to use S3 policy.

    GLUE_USE_KINESIS_POLICY

    Set this property to true for AWS Glue to use Kinesis policy.

    KINESIS_BYPASS_ACTIONS_LIST

    Set the list of Kinesis actions to bypass authorization. For example: GetShardIterator, GetRecords.

    The default value is none.

    DATABRICKS_DELTA_IAM_ROLE_ARN

    Set the IAM role ARN for Databricks delta use-cases.

    REQUEST_INCLUDE_USERINFO

    Set this property to true if you want to include Privacera User Information in AWS outgoing CloudWatch requests.. The default value is false.

  6. Update Privacera Manager.

    cd ~/privacera/privacera-manager
./privacera-manager.sh update

Use case with Platform Portal after adding multiple AWS accounts

Platform Portal behavior with the addition of multiple AWS accounts:

  • If you are upgrading Privacera Manager and the dataserver has multiple AWS profiles and the portal already has an AWS S3 application, it will not add or modify AWS S3 information with profile support.

    To make the portal support multiple profiles after the PM update, you first need to backup the application properties.

    Do the following steps to backup the AWS S3 application properties:

    1. Go to the Platform portal.

    2. Under Settings > select Data Source Registration.

    3. Click the edit icon next to the AWS S3 application.

    4. Click Application Properties.

    5. Click Import/Export properties, and then select Export Properties.

    Note

    If you are using discovery, avoid performing application deletion because the data sources will be lost.

    After exporting, the customer can delete the application and restart the Portal.

  • It will work as expected for a fresh Privacera Manager installation.