Skip to content

Setup for BigQuery Connector - Access Management

This section outlines the procedure for setting up GCP BigQuery. Please ensure that you have completed the prerequisites before beginning the setup.

Configuration of Self-Managed Instance for BigQuery with Portal UI

Prerequisites

  1. Obtain Authentication Credentials

    • Choose one of the following methods for authentication:
      • Workload Identity
        If you're deploying the connector on a GCP instance and prefer to use Workload Identity for authentication, follow the instructions in Enable Workload Identity to set up Workload Identity authentication.
      • Service Account Key JSON File
        If you are using a service account key, follow the instructions in Attach IAM Role to Service Account to obtain the Service Account key in JSON format.

  2. Migration (Optional: Follow only if you have existing connector setup running)

    • Important Notes:

      1. Replace <CONNECTOR_TYPE> with the type of connector you're using (e.g. bigquery, mysql, etc.).
      2. Replace <CONNECTOR_ENV> with the environment name of your connector (e.g. prod, dev, etc.).

    • Steps :

      1. Backup Existing Applications Before proceeding with the migration, create a backup of the existing connector configurations. Note: Please change with connector type.

        Bash
        1
2
        mkdir -p ~/privacera/backup/connector-config
cp ~/privacera/privacera-manager/config/custom-vars/connectors/<CONNECTOR_TYPE>/*

        Example

        Bash
        1
2
        mkdir -p ~/privacera/backup/connector-config
cp -r ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/* ~/privacera/backup/connector-config

      2. Steps to find CONNECTOR_ENV name Considering you have created bigquery connector at location as ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/prod/vars.connector.bigquery.yml. In the above example prod will be your CONNECTOR_ENV name.

      3. Create an Application on a portal with the same name as name.

Creating & Downloading Connector Configuration from Portal

  1. In the Portal for Self-Managed, navigate to Settings → Applications.

  2. On the Applications screen, select BigQuery.

  3. Enter the application Name and Description, then click Save

    • The name can be any meaningful identifier, such as bigquery-connector-1.
    • The description can provide additional context, e.g. BigQuery Connector for dev account xyz.

  4. On a successful Save action, you should see the Access Management card below the BigQuery card on the left side of the popup. Enable it by clicking on the toggle.

  5. Under the Basic tab, provide values for the following fields:

    Field Required Description
    BigQuery Project Location Yes Add the project location, e.g. us.
    BigQuery Project ID Yes Use the BigQuery Project ID from GCP, e.g. projectXXX.
    Service Account Email Yes Provide the email address of the service account with the required GCP permissions.
    Service Account JSON Key Content Paste the content of the Service Account JSON key if you have chosen Service Account JSON key authentication; this step is not required if deploying the connector on a GCP instance and using Workload Identity.
    Projects to Set Access Control Policies Yes Enter the Project IDs to be managed.
    Native Public Group Identity Name Yes Specify the identity for access grants in policies that reference public groups: ALL_AUTHENTICATED_USERS or ALL_USERS.
    Enable Access Audit No Indicate whether to fetch access audit data from the data source. Allowed values: true or false.

  6. Under the Advanced tab, provide values for these fields:

    Field Required Description
    Flag for GCP use case Yes Set to true if the connector runs on a GCP instance and you have followed the steps for setting up Workload Identity authentication.
    GCP custom IAM roles scope No Define the scope for custom IAM roles. Allowed values: project or org.
    Datasets to set access control policies No Enter datasets managed by PolicySync (comma-separated). Format: <PROJECT_ID>.<DATASET_NAME>.
    Projects to Set Access Control Policies No Specify the Project IDs to be managed.
    Users to exclude when fetching access audits No List the users whose access audits you want to exclude, separated by commas.

  7. Click the Save button.

  8. The configured BigQuery connector will now appear under Connected Applications.

  9. Click on the BigQuery icon to see the configured connector. Next, click the Download icon in the action column of the created connector. This will download a zipped file which will be required in further steps.

Modifying the Downloaded YML File (Avoid if using Workload Identity authentication)

Add the name of the Service Account key JSON file (without the path) to the downloaded YML file. Goto file location

Bash
1
cd <file_path>/
Open the file in an editor:
Bash
1
vi vars.connector.bigquery.yml
Add the following line:
YAML
1
CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME: "policysync-gbq-service-account.json"
Save the file and exit the editor. Ensure the JSON file name matches correctly.

Uploading Files to the Instance

  1. Upload the Service Account key JSON file to the appropriate directory on the instance:

    Bash
    1
    scp <local_path>/policysync-gbq-service-account.json <host>:/home/<user>/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/

  2. Copy the configuration zip file to the correct location in the Privacera Manager environment:

    • For a single application:
      Bash
      1
      scp <local_path>/connectors.zip <host>:/home/<user>/privacera/privacera-manager/config/custom-vars/connectors
      • For multiple applications:
        Bash
        1
        scp <local_path>/connectors.zip <host>:/home/<user>/privacera/privacera-manager/config/custom-vars

  3. SSH to the instance and navigate to the location where zip is copied and run command to unzip folder. Note: If replace warning comes up then replace.

    Bash
    1
2
    cd ~/privacera/privacera-manager/config/custom-vars/connectors
unzip connectors.zip

Finishing the Setup

SSH into the instance and run the following commands to update your Privacera Manager platform instance:

Bash
1
2
3
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade

Conclusion

After completing the above steps, your Privacera Manager platform instance will be updated with the new configurations.

Please note that only the following fields will be updated automatically in the instance. For all other fields, you may need to update them manually on the instance.

Fields those will be updated automatically

  1. Projects to set access control policies
  2. Datasets to set access control policies
  3. Tables to set access control policies

To edit these fields, navigate to the respective connectors and make the necessary changes in Basic & Advanced tabs. Click on Save button and the changes will be stored.

To update rest of the fields, please update the YML file in your instance.

Please perform the following steps to configure the GCP BigQuery connector:

  1. SSH into the instance where Privacera is installed.

  2. Run the following command to navigate to the /config directory.

    Bash
    1
    cd ~/privacera/privacera-manager/config

  3. Run the following command to create a new directory:

    Bash
    1
    mkdir -p custom-vars/connectors/bigquery/instance1

  4. Run the following command to copy the sample vars:

    Bash
    1
    cp sample-vars/vars.connector.bigquery.yml custom-vars/connectors/bigquery/instance1/

  5. Run the following command to open the .yml file to be edited.

    Bash
    1
    vi custom-vars/connectors/bigquery/instance1/vars.connector.bigquery.yml

  6. Transfer the Service Account key JSON file, obtained during Attach IAM Role to Service Account, to the /custom-vars/connectors/bigquery/instance1/ directory. If using Workload Identity, ensure that the CONNECTOR_BIGQUERY_USE_VM_CREDENTIALS variable is set to true, and you do not need the JSON file. If not using Workload Identity, update the vars.connector.bigquery.yml file by setting the CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME variable to the name of the newly transferred JSON file.

  7. Modify the following properties:

    • CONNECTOR_BIGQUERY_PROJECT_ID - Enter the GCP Bigquery Project ID you want to use.

    • CONNECTOR_BIGQUERY_PROJECT_LOCATION - Set GCP Bigquery Project Location.

    • CONNECTOR_BIGQUERY_USE_VM_CREDENTIALS - Set this true if you are running the connector on a GCP instance and have performed the steps mentioned here.

    • CONNECTOR_BIGQUERY_OAUTH_SERVICE_ACCOUNT_EMAIL - Set the service account email with access to the GCP resources required by PolicySync. If CONNECTOR_BIGQUERY_USE_VM_CREDENTIALS is set to true, the service account from the Workload Identity setup will be used.

    • CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME - Specify the name of the service account's private key file (in JSON format) if CONNECTOR_BIGQUERY_USE_VM_CREDENTIALS is set to false. If CONNECTOR_BIGQUERY_USE_VM_CREDENTIALS is true, the private key file is not required, as Workload Identity will be used.

    • CONNECTOR_BIGQUERY_CUSTOM_IAM_ROLES_SCOPE - Set the scope of the custom IAM roles to be created. Allowed values: "project" or "org".

    • CONNECTOR_BIGQUERY_ORGANIZATION_ID - Enter your GCP organization ID if you want to create IAM roles at the organization level.

    • CONNECTOR_BIGQUERY_AUDIT_ENABLE - Set to "true" to enable auditing for BigQuery.

    • CONNECTOR_BIGQUERY_AUDIT_EXCLUDED_USERS - Enter a comma-separated list of users whose access audits you want to exclude. If not set, it excludes the service account set in CONNECTOR_BIGQUERY_OAUTH_SERVICE_ACCOUNT_EMAIL by default.

    • CONNECTOR_BIGQUERY_MANAGE_PROJECT_LIST - Enter a comma-separated list of project IDs that PolicySync will manage.

    • CONNECTOR_BIGQUERY_MANAGE_DATASET_LIST - Enter a comma-separated list of datasets that PolicySync will manage. Format: <PROJECT_ID>.<DATASET_NAME>.

    • CONNECTOR_BIGQUERY_GRANT_UPDATES - Set to "true" to enable PolicySync to perform grant and revoke updates on access control.

    • CONNECTOR_BIGQUERY_NATIVE_PUBLIC_GROUP_IDENTITY_NAME - Enter the public group identity name for access grants. Allowed values: "ALL_AUTHENTICATED_USERS", "ALL_USERS", or any valid Google group name.

    • CONNECTOR_BIGQUERY_ENABLE - Set to "true" to enable PolicySync for BigQuery.

    • CONNECTOR_BIGQUERY_MANAGE_ENTITY_PREFIX - Enter a comma-separated list of prefixes for users, groups, and roles that PolicySync will manage. For example: dev_*, sa_*. Leave blank to manage all entities.

  8. Once the properties have been configured, execute the following commands to update your Privacera Manager platform instance:

    Bash
    1
2
3
    cd ~/privacera/privacera-manager

./privacera-manager.sh update

  1. In PrivaceraCloud, navigate to Settings -> Applications.

  2. On the Applications screen, select BigQuery.

  3. Enter the application Name and Description, then click Save. The name can be any designation of your choice, such as BigQuery Connector for account 123456.

  4. Open the BigQuery application.

  5. Enable the Access Management option using the toggle button.

  6. Under the BASIC tab, enter the values for:

    • BigQuery Project Location : us

    • BigQuery Project Id : projectXXX

    • Service Account Email : Use the service account email which have access to GCP resources required by PolicySync.

    • Service Account Json Key Content : Use Service Account JSON

    • Projects to Set Access Control Policies : Project IDs you want to manage

    • Native Public Group Identity Name : Specifies the public group identity to be used for access grants in policies referring to public groups. Allowed values: ALL_AUTHENTICATED_USERS (All GCP project authenticated users) or ALL_USERS (All Google authenticated users).

    • Enable Audit : Specifies whether Privacera fetches access audit data from the data source. Allowed values: true or false.

  7. Click SAVE.

  8. The configured BigQuery connector appears under Applications.

  9. Once saved and enabled, the BigQuery connector will start. Then you can hover on the VIEW LOGS button to check the status, either Running or Stopped.

  10. Perform following steps to restart the BigQuery connector application:

    1. Go to Settings → Applications → select the****BigQuery** connector application** .

    2. Edit the application → Disable it → and Save it.

    3. Open the same application again and then: Enable it → and Save it.

Comments