Skip to content

Setup for BigQuery Connector - Access Management

This section outlines the procedure for setting up GCP BigQuery. Please ensure that you have completed the prerequisites before beginning the setup.

Configuration of Self-Managed Instance for BigQuery with Portal UI

Prerequisites

  1. Obtain a Service Account key JSON file by following the instructions in Attach IAM Role to Service Account.

  2. Migration (Optional: Follow only if you have existing connector setup running)

    • Important Notes:

      1. Replace with the type of connector you're using (e.g. bigquery, mysql, etc.).
      2. Replace with the environment name of your connector (e.g. prod, dev, etc.).

    • Steps :

      1. Backup Existing Applications Before proceeding with the migration, create a backup of the existing connector configurations. Note: Please change with connector type.

        Bash
        1
2
        mkdir -p ~/privacera/backup/connector-config
cp ~/privacera/privacera-manager/config/custom-vars/connectors/<CONNECTOR_TYPE>/*

        Example

        Bash
        1
2
        mkdir -p ~/privacera/backup/connector-config
cp -r ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/* ~/privacera/backup/connector-config

      2. Steps to find CONNECTOR_ENV name Considering you have created bigquery connector at location as ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/prod/vars.connector.bigquery.yml. In the above example prod will be your CONNECTOR_ENV name.

      3. Create an Application on a portal with the same name as name.

1. Creating & Downloading Connector Configuration from Portal

  1. In the Portal for Self-Managed, navigate to Settings → Applications.

  2. On the Applications screen, select BigQuery.

  3. Enter the application Name and Description, then click Save

    • The name can be any meaningful identifier, such as bigquery-connector-1.
    • The description can provide additional context, e.g. BigQuery Connector for dev account xyz.

  4. Open the newly created BigQuery application and enable the Access Management option by toggling the button.

  5. Under the Basic tab, provide values for the following fields:

    Field Description
    BigQuery Project Location Add the project location, e.g. us.
    BigQuery Project ID Use the BigQuery Project ID from GCP, e.g. projectXXX.
    Service Account Email Provide the email address of the service account with the required GCP permissions.
    Service Account JSON Key Content Paste the content of the Service Account JSON key.
    Projects to Set Access Control Policies Enter the Project IDs to be managed.
    Native Public Group Identity Name Specify the identity for access grants in policies referencing public groups: ALL_AUTHENTICATED_USERS or ALL_USERS.
    Enable Audit Indicate whether to fetch access audit data from the data source. Allowed values: true or false.

  6. Under the Advanced tab, provide values for these fields:

    Field Description
    Flag for GCP use case Set to true if the connector runs on a GCP instance.
    GCP custom IAM roles scope Define the scope for custom IAM roles. Allowed values: project or org.
    Users to exclude when fetching access audits List users (comma-separated) whose access audits you want to exclude.
    Datasets to set access control policies Enter datasets managed by PolicySync (comma-separated). Format: <PROJECT_ID>.<DATASET_NAME>.
    Projects to Set Access Control Policies Specify the Project IDs to be managed.

  7. Click the Save button.

  8. The configured BigQuery connector will now appear under Applications.

  9. Click on the BigQuery icon to see the configured connector. Next, click the Download icon in the action column of the connector. This will download a zipped file which will be required in further steps.

2. Modifying the Downloaded YML File

  1. Add the name of the Service Account key JSON file (without the path) to the downloaded YML file. Goto file location

    Bash
    1
    cd <file_path>/
    Open the file in an editor:
    Bash
    1
    vi vars.connector.bigquery.yml
    Add the following line:
    YAML
    1
    CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME: "policysync-gbq-service-account.json"
    Ensure the JSON file name matches correctly.

  2. Alternatively, use the following shell command to append the line to the YML file:

    Bash
    1
    echo 'CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME: "policysync-gbq-service-account.json"' >> vars.connector.bigquery.yml

3. Uploading Files to the Instance

  1. Upload the Service Account key JSON file to the appropriate directory on the instance:

    Bash
    1
    scp <local_path>/policysync-gbq-service-account.json <host>:/home/<user>/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/

  2. Copy the configuration zip file to the correct location in the Privacera Manager environment:

  3. For a single application:

    Bash
    1
    scp <local_path>/connectors.zip <host>:/home/<user>/privacera/privacera-manager/config/custom-vars/connectors

  4. For multiple applications:

    Bash
    1
    scp <local_path>/connectors.zip <host>:/home/<user>/privacera/privacera-manager/config/custom-vars

  5. SSH to the instance and navigate to the location where zip is copied and run command to unzip folder. Note: If replace warning comes up then replace.

    Bash
    1
2
    cd ~/privacera/privacera-manager/config/custom-vars/connectors
unzip connectors.zip

Finishing the Setup

SSH into the instance and run the following commands to update your Privacera Manager platform instance:

Bash
1
2
cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade

Please perform the following steps to configure the GCP BigQuery connector:

  1. SSH into the instance where Privacera is installed.

  2. Run the following command to navigate to the /config directory.

    Bash
    1
    cd ~/privacera/privacera-manager/config

  3. Run the following command to create a new directory:

    Bash
    1
    mkdir -p custom-vars/connectors/bigquery/instance1

  4. Run the following command to copy the sample vars:

    Bash
    1
    cp sample-vars/vars.connector.bigquery.yml custom-vars/connectors/bigquery/instance1/

  5. Run the following command to open the .yml file to be edited.

    Bash
    1
    vi custom-vars/connectors/bigquery/instance1/vars.connector.bigquery.yml

  6. Transfer the Service Account key JSON file, obtained during Attach IAM Role to Service Account, to the /custom-vars/connectors/bigquery/instance1/ directory. Subsequently, modify the vars.connector.bigquery.yml file by updating the variable CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME with the name of the newly placed JSON file.

  7. Modify the following properties:

    • CONNECTOR_BIGQUERY_PROJECT_ID - Enter the GCP Bigquery Project ID you want to use.

    • CONNECTOR_BIGQUERY_PROJECT_LOCATION - Set GCP Bigquery Project Location.

    • CONNECTOR_BIGQUERY_USE_VM_CREDENTIALS - Set this to true, if you are running connector on GCP instance.

    • CONNECTOR_BIGQUERY_OAUTH_SERVICE_ACCOUNT_EMAIL - Set the service account email which have access to GCP resources required by PolicySync.

    • CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME - Set Private key file name if
      CONNECTOR_BIGQUERY_USE_VM_CREDENTIALS is set to False.

    • CONNECTOR_BIGQUERY_CUSTOM_IAM_ROLES_SCOPE - Set the scope of the custom IAM roles to be created. Allowed values: "project" or "org".

    • CONNECTOR_BIGQUERY_ORGANIZATION_ID - Enter your GCP organization ID if you want to create IAM roles at the organization level.

    • CONNECTOR_BIGQUERY_AUDIT_ENABLE - Set to "true" to enable auditing for BigQuery.

    • CONNECTOR_BIGQUERY_AUDIT_EXCLUDED_USERS - Enter a comma-separated list of users whose access audits you want to exclude. If not set, it excludes the service account set in CONNECTOR_BIGQUERY_OAUTH_SERVICE_ACCOUNT_EMAIL by default.

    • CONNECTOR_BIGQUERY_MANAGE_PROJECT_LIST - Enter a comma-separated list of project IDs that PolicySync will manage.

    • CONNECTOR_BIGQUERY_MANAGE_DATASET_LIST - Enter a comma-separated list of datasets that PolicySync will manage. Format: <PROJECT_ID>.<DATASET_NAME>.

    • CONNECTOR_BIGQUERY_GRANT_UPDATES - Set to "true" to enable PolicySync to perform grant and revoke updates on access control.

    • CONNECTOR_BIGQUERY_NATIVE_PUBLIC_GROUP_IDENTITY_NAME - Enter the public group identity name for access grants. Allowed values: "ALL_AUTHENTICATED_USERS", "ALL_USERS", or any valid Google group name.

    • CONNECTOR_BIGQUERY_ENABLE - Set to "true" to enable PolicySync for BigQuery.

    • CONNECTOR_BIGQUERY_MANAGE_ENTITY_PREFIX - Enter a comma-separated list of prefixes for users, groups, and roles that PolicySync will manage. For example: dev_*, sa_*. Leave blank to manage all entities.

  8. Once the properties have been configured, execute the following commands to update your Privacera Manager platform instance:

    Bash
    1
2
3
    cd ~/privacera/privacera-manager

./privacera-manager.sh update

  1. In PrivaceraCloud, navigate to Settings -> Applications.

  2. On the Applications screen, select BigQuery.

  3. Enter the application Name and Description, then click Save. The name can be any designation of your choice, such as BigQuery Connector for account 123456.

  4. Open the BigQuery application.

  5. Enable the Access Management option using the toggle button.

  6. Under the BASIC tab, enter the values for:

    • BigQuery Project Location : us

    • BigQuery Project Id : projectXXX

    • Service Account Email : Use the service account email which have access to GCP resources required by PolicySync.

    • Service Account Json Key Content : Use Service Account JSON

    • Projects to Set Access Control Policies : Project IDs you want to manage

    • Native Public Group Identity Name : Specifies the public group identity to be used for access grants in policies referring to public groups. Allowed values: ALL_AUTHENTICATED_USERS (All GCP project authenticated users) or ALL_USERS (All Google authenticated users).

    • Enable Audit : Specifies whether Privacera fetches access audit data from the data source. Allowed values: true or false.

  7. Click SAVE.

  8. The configured BigQuery connector appears under Applications.

  9. Once saved and enabled, the BigQuery connector will start. Then you can hover on the VIEW LOGS button to check the status, either Running or Stopped.

  10. Perform following steps to restart the BigQuery connector application:

    1. Go to Settings → Applications → select the****BigQuery** connector application** .

    2. Edit the application → Disable it → and Save it.

    3. Open the same application again and then: Enable it → and Save it.

Comments