Dynamic resource onboarding through Portal

Dynamic resource onboarding through Portal¶

In BigQuery connector, we have simplified the onboarding process for resources such as projects, datasets, tables in Privacera by replacing manual backend edits with an intuitive, user-friendly UI. This interface will allow administrators to easily add, update, and remove resources.

Prerequisites¶

1. Configure Ops-server
   Ensure that the Ops-server is up and running by following the instructions here: Ops-Server Setup

2. Obtain Authentication Credentials
   Choose one of the following methods for authentication:

   • Workload Identity
     If you're deploying the connector on a GCP instance and prefer to use Workload Identity for authentication, follow the instructions in Enable Workload Identity to set up Workload Identity authentication.
   • Service Account Key JSON File
     If you are using a service account json, follow the instructions in Attach IAM Role to Service Account to obtain the Service Account Key in JSON format.

3. Migration (Optional)
   Follow these steps only if you have an existing connector instance configured and wish to migrate its configurations to the portal.

   Note

   1. Replace <CONNECTOR_TYPE> with the type of connector you're using (e.g. bigquery, mysql, etc.).
   2. Replace <CONNECTOR_ENV> with the environment name of your connector (e.g. prod, dev, etc.).
   • SSH into the instance where Privacera Manager is installed.

   • Backup Existing Applications
     Before proceeding with the migration, create a backup of the existing connector configurations.

     Bash
     1 2	mkdir -p ~/privacera/backup/connector-config cp ~/privacera/privacera-manager/config/custom-vars/connectors/<CONNECTOR_TYPE>/*

     Example

     Bash
     1 2	mkdir -p ~/privacera/backup/connector-config cp -r ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/* ~/privacera/backup/connector-config

   • Find CONNECTOR_ENV name
     Considering you have created bigquery connector instance at location as:
     ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/prod/vars.connector.bigquery.yml.
     In this case prod will be your CONNECTOR_ENV name.

   • Create an Application on a portal with the same name as <CONNECTOR_ENV> name.

Creating & Downloading Connector Configuration from Portal¶

1. In the Portal for Self-Managed, navigate to Settings → Applications.

2. On the Applications screen, select BigQuery.

3. Enter the application Name and Description, then click Save

   • The name can be any meaningful identifier, such as bigquery-connector-1.
   • The description can provide additional context, e.g. BigQuery Connector for dev account xyz.

4. Click on the application name or icon, and a popup will open. You should see the Access Management card below the BigQuery card on the left side of the popup. Enable it by clicking on the toggle.

5. Under the Basic tab, provide values for the following fields:

   Note

   • To view the property details, click the icon next to the property name.
   • Fields with asterisks (*) are mandatory.

   Field	Description
   BigQuery Project Location	Add the project location, e.g. us.
   BigQuery Project ID	Use the BigQuery Project ID from GCP, e.g. projectXXX.
   Service Account Email	Provide the email address of the service account with the required GCP permissions.
   Projects to Set Access Control Policies	Enter the Project IDs to be managed.
   Native Public Group Identity Name	Specify the identity for access grants in policies that reference public groups: ALL_AUTHENTICATED_USERS or ALL_USERS.
   Enable Access Audit	Indicate whether to fetch access audit data from the data source. Allowed values: true or false.
   If Enable Access Audit is set to true, then configure the following properties
   Project id having the bigquery audits datasets configured	The unique ID of the Google Cloud project linked to BigQuery audit logs.
   Dataset name to fetch the access audits	The name of the dataset for retrieving access audit logs in BigQuery.

6. Under the Advanced tab, enter value for the following mandatory fields:

   Field	Description
   Flag for GCP use case	Set to true if the connector runs on a GCP instance and you have followed the steps for setting up Workload Identity authentication.

7. For dynamic resource onboarding, provide values for the following fields as required.

   Field	Description
   Datasets to set access control policies	Enter datasets managed by PolicySync (comma-separated). Format: <PROJECT_ID>.<DATASET_NAME>.
   Tables/Views to Set Access Control Policies	Enter tables managed by PolicySync (comma-separated). Format: <PROJECT_ID>.<DATASET_NAME>.<TABLE_NAME>.

8. To use Native row filter and masking, adjust the following toggle settings:

   • Enable: Enforce BigQuery native row filter
   • Disable: Enforce masking policies using secure views
   • Disable: Enforce row filter policies using secure views
   • Disable: Create secure view for all tables/views
   • Disable: Enable dataadmin

   Note

   • To use Native Tag-based Masking, add following property in your connector YAML file.

     YAML
     1	ENABLE_TAG_MASKING: "true"

9. Click the Save button.

10. The configured BigQuery connector will now be visible in the Connected Applications section.

11. Click on the BigQuery icon to view the configured connector instance.

12. Next, download the created connector instance.

    • Download a single instance: Click on the icon in the action column of the corresponding application row.
    • Download multiple instances: Click on the DOWNLOAD ALL CONFIGURATION option in the top-right corner.

Uploading Files to the Privacera Manager host¶

1. Copy the configuration zip file to the following path at Privacera Manager host:

   • In case of single or multiple applications of same service type e.g. bigquery copy to:

     Bash
     1	~/privacera/privacera-manager/config/custom-vars/connectors/

   • In case of multiple applications of different service type e.g. bigquery, snowflake copy to:

     Bash
     1	~/privacera/privacera-manager/config/custom-vars/

2. SSH to the instance and navigate to the location where zip is copied and run command to unzip folder.

   Note

   • If a replacement warning is displayed, ensure that existing configurations are backed up (if needed), and then proceed with the replacement.
   • The replacement will only replace files with the same name; other files within the folder remain unchanged.
   • Ensure there are no duplicate variables across multiple YAML files.
   • Make sure the unzip command is installed on the system.

   Bash
   1	unzip connectors.zip

Configure Service Account JSON File (Avoid if you're using Workload Identity Authentication)¶

1. Create a Service Account JSON file as mentioned in point 2 of the Prerequisites.

   Bash
   1 2	cd ~/privacera/privacera-manager vi config/custom-vars/connectors/bigquery/<CONNECTOR_ENV>/policysync-gbq-service-account.json

2. Add the Service Account JSON content in the policysync-gbq-service-account.json file. Save the file and exit the editor.
3. Update the connector YML file
   • Open the connector YML file in an editor

     Bash
     1	vi config/custom-vars/connectors/bigquery/<CONNECTOR_ENV>/vars.connector.bigquery.yml

   • Add the following variable and save the file:

     YAML
     1	CONNECTOR_BIGQUERY_OAUTH_PRIVATE_KEY_FILE_NAME: "policysync-gbq-service-account.json"

   • Save the file and exit the editor. Ensure the JSON file name matches correctly.

Finishing the Setup¶

After all the changes are done you can start the connector by running the following instructions:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

1
2

cd ~/privacera/privacera-manager
./privacera-manager.sh setup

1
2

cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade

1
2

cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Comments