Skip to content

Resource Loading Configuration

You can configure the BigQuery connector to optimize how it loads resources such as datasets, tables, and columns from BigQuery. These settings control the loading methodology, thread allocation, and batch processing to ensure optimal performance for your environment.

The connector supports both multi-threaded and single-threaded resource loading approaches. Multi-threaded loading can significantly improve performance in environments with large numbers of resources, while single-threaded loading provides more predictable resource usage patterns.

Resource Loading Properties:

Property Name Description Default Value Supported Values
LOAD RESOURCES KEY Controls which method to be used to load resources from BigQuery load_multi_thread load_from_dataset_columns, load_multi_thread
LOAD RESOURCES LOAD TABLE THREADS Maximum number of threads to load the tables inside the configured databases 2 Any numeric value
LOAD RESOURCES LOAD TABLE MIN THREADS Minimum number of threads to load the tables inside the configured databases 2 Any numeric value
LOAD RESOURCES LOAD COLUMN THREADS Maximum number of threads to load the columns inside the configured tables 3 Any numeric value
LOAD RESOURCES LOAD COLUMN MIN THREADS Minimum number of threads to load the columns inside the configured tables 3 Any numeric value
LOAD TABLE SIZE Load table batch size in a single thread task 500 Any numeric value

Configuration Steps

Warning

  • Higher thread counts can improve performance but may increase memory usage and BigQuery API quotas.
  • Replace the example values with your actual configuration values.

Restart Required

Any changes to these properties require restarting the BigQuery connector application for the updates to take effect.

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. Select BigQuery from the list of Connected Applications.

  3. Click on the application name or the icon, then click on Access Management → ADVANCED tab.

  4. Add the following properties under the Add New Custom Properties section:

    Bash
    1
2
3
4
5
6
    ranger.policysync.connector.0.load.resources=load_multi_thread
ranger.policysync.connector.0.load.resources.load.table.thread.count=4
ranger.policysync.connector.0.load.resources.load.table.thread.min.count=2
ranger.policysync.connector.0.load.resources.load.column.thread.count=6
ranger.policysync.connector.0.load.resources.load.column.thread.min.count=3
ranger.policysync.connector.0.load.table.size=1000

  5. Click SAVE to apply the changes.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/vars.connector.bigquery.yml

  3. Add or modify the following properties:

    YAML
    1
2
3
4
5
6
    CONNECTOR_BIGQUERY_LOAD_RESOURCES_KEY: "load_multi_thread"
CONNECTOR_BIGQUERY_LOAD_RESOURCES_LOAD_TABLE_THREADS: "4"
CONNECTOR_BIGQUERY_LOAD_RESOURCES_LOAD_TABLE_MIN_THREADS: "2"
CONNECTOR_BIGQUERY_LOAD_RESOURCES_LOAD_COLUMN_THREADS: "6"
CONNECTOR_BIGQUERY_LOAD_RESOURCES_LOAD_COLUMN_MIN_THREADS: "3"
CONNECTOR_BIGQUERY_LOAD_TABLE_SIZE: "1000"

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud, go to Settings → Applications.

  2. Select BigQuery from the list of Connected Applications.

  3. Click on the application name or the icon, then click on Access Management → ADVANCED tab.

  4. Add the following properties under the Add New Custom Properties section:

    Bash
    1
2
3
4
5
6
    ranger.policysync.connector.0.load.resources=load_multi_thread
ranger.policysync.connector.0.load.resources.load.table.thread.count=4
ranger.policysync.connector.0.load.resources.load.table.thread.min.count=2
ranger.policysync.connector.0.load.resources.load.column.thread.count=6
ranger.policysync.connector.0.load.resources.load.column.thread.min.count=3
ranger.policysync.connector.0.load.table.size=1000

  5. Click SAVE.

  6. Once saved and enabled, the BigQuery connector will start. You can hover on the VIEW LOGS button to check the status, either Running or Stopped.

Note

Perform the following steps only if the connector does not reflect the updated configuration and requires a restart.

Restart the BigQuery Connector:

  1. Go to Settings → Applications → select the BigQuery connector application.

  2. Edit the application → Disable it → and Save it.

  3. Open the same application again and then: Enable it and Save it.

Comments