Skip to content

Tag Reconcile Loader Configuration

Goal

This guide explains how to configure the BigQuery connector to automatically reconcile tags between the BigQuery connector and Apache Ranger.

  1. Service Tag Reconciliation: Reconciles tags between the BigQuery connector and Apache Ranger
  2. Ranger Tag Reconciliation: Reconciles tags between Ranger/Portal and BigQuery, ensuring tags and their column mappings remain synchronized

Prerequisites

Before you begin, ensure the following:

  • Privacera Manager is installed and the base installation is operational.
  • The BigQuery connector is configured and running.
  • Apache Ranger is properly configured and accessible.

Tag Masking Requirement

These tag reconciliation loader properties will only work if tag masking is enabled in the connector.

Service Tag Reconciliation

Overview

Service Tag Reconciliation maintains tag consistency between your BigQuery connector and Apache Ranger by:

  • Monitoring tag differences: Compares tags loaded by the connector with those present in Ranger
  • Automatic reconciliation: Creates missing tags in Ranger or removes orphaned tags
  • Configurable intervals: Runs at specified intervals to ensure ongoing consistency

Service Tag Reconcile Loader Properties:

Property Name Description Default Value Supported Values
CONNECTOR_BIGQUERY_TAG_RECONCILE_LOADER_ENABLED Enable or disable the service tag reconcile loader functionality true true, false
CONNECTOR_BIGQUERY_SERVICE_TAG_RECONCILE_SYNC_INTERVAL Set the interval for service tag reconcile sync process in seconds 540 Any numeric value in seconds

Configuration Steps

Warning

  • Enabling this feature will automatically create and delete tags in Ranger based on connector data.
  • Lower interval values result in more frequent reconciliation but may increase system load.
  • Be sure to replace the example values with your actual configuration values.

Restart Required

Any changes to these properties require a restart of the BigQuery connector application for the updates to take effect.

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. Select BigQuery from the list of Connected Applications.

  3. Click on the application name or the icon, then click on Access Management → ADVANCED tab.

  4. To update the tag reconcile sync interval, add the following property under the Add New Custom Properties section:

    Bash
    1
    ranger.policysync.connector.0.sync.servicetag.reconcile.interval.sec=540

  5. To disable the tag reconcile loader, add the following property under the Add New Custom Properties section:

    Bash
    1
    ranger.policysync.connector.0.tag.reconcile.loading.enabled=false

  6. Click SAVE to apply the changes.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/vars.connector.bigquery.yml

  3. To update the tag reconcile sync interval, add or modify the following property:

    YAML
    1
2
    # Set tag reconcile sync interval (in seconds)
CONNECTOR_BIGQUERY_SERVICE_TAG_RECONCILE_SYNC_INTERVAL: "540"

  4. To disable the tag reconcile loader, add or modify the following property:

    YAML
    1
2
    # Disable tag reconcile loader
CONNECTOR_BIGQUERY_TAG_RECONCILE_LOADER_ENABLED: "false"

  5. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud, navigate to Settings → Applications.

  2. Select BigQuery from the list of Connected Applications.

  3. Click on the application name or the icon, then click on Access Management → ADVANCED tab.

  4. To update the tag reconcile sync interval, add the following property under the Add New Custom Properties section:

    Bash
    1
    ranger.policysync.connector.0.sync.servicetag.reconcile.interval.sec=540

  5. To disable the tag reconcile loader, add the following property under the Add New Custom Properties section:

    Bash
    1
    ranger.policysync.connector.0.tag.reconcile.loading.enabled=false

  6. Click SAVE.

  7. Once saved and enabled, the BigQuery connector will start. You can hover on the VIEW LOGS button to check the status, either Running or Stopped.

Note

Perform the following steps only if the connector does not reflect the updated configuration and requires a restart.

Restart the BigQuery Connector:

  1. Go to Settings → Applications → select the BigQuery connector application.

  2. Edit the application → Disable it → and Save it.

  3. Open the same application again and then: Enable it and Save it.

Ranger Tag Reconciliation

Overview

Ranger Tag Reconciliation ensures tags and their resource mappings (column-level policy tags) remain synchronized between Ranger/Portal and BigQuery. This advanced reconciliation feature:

  • Detects missing tags: Identifies tags that exist in Ranger but were manually deleted from BigQuery
  • Recreates tags automatically: Restores deleted policy tags in BigQuery Data Catalog
  • Restores tag mappings: Automatically reapplies column-level tag mappings for recreated tags
  • Monitors existing mappings: Checks if tag-resource mappings were manually removed from BigQuery columns and reapplies them
  • Prevents drift: Ensures Ranger remains the authoritative source for tag definitions and mappings

Ranger Tag Reconciliation Properties:

Property Name Description Default Value Supported Values
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_ENABLE Enable or disable Ranger tag reconciliation functionality false true, false
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_INTERVAL_SEC Set the interval for Ranger tag reconciliation process in seconds 420 Any numeric value in seconds (default: 7 minutes)
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_AT_RESTART_ENABLE Run reconciliation immediately at connector startup false true, false

Configuration Steps

Important Considerations

  • This feature focuses on tag creation and restoration only - it does not delete tags from BigQuery
  • Ranger is the authoritative source of truth for all tag definitions and mappings
  • Lower interval values result in faster recovery but may increase system load and API calls to BigQuery
  • Both normal (flat) and hierarchical policy tag structures are supported
  • Be sure to replace the example values with your actual configuration values

Restart Required

Any changes to these properties require a restart of the BigQuery connector application for the updates to take effect.

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. Select BigQuery from the list of Connected Applications.

  3. Click on the application name or the icon, then click on Access Management → ADVANCED tab.

  4. To enable Ranger tag reconciliation, add the property under Add New Custom Properties, update the index based on the connector you are configuring :

    Bash
    1
    ranger.policysync.connector.0.sync.rangertag.reconcile.enable=true

  5. To update the reconciliation interval (optional), add the following property:

    Bash
    1
    ranger.policysync.connector.0.sync.rangertag.reconcile.interval.sec=420

  6. To enable reconciliation at connector restart (optional), add the following property:

    Bash
    1
    ranger.policysync.connector.0.sync.rangertag.reconcile.at.restart.enable=true

  7. Click SAVE to apply the changes.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/vars.connector.bigquery.yml

  3. To enable Ranger tag reconciliation, add or modify the following property:

    YAML
    1
2
    # Enable Ranger tag reconciliation
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_ENABLE: "true"

  4. To update the reconciliation interval (optional), add or modify the following property:

    YAML
    1
2
    # Set Ranger tag reconciliation interval (in seconds, default: 7 minutes)
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_INTERVAL_SEC: "420"

  5. To enable reconciliation at connector restart (optional), add or modify the following property:

    YAML
    1
2
    # Run reconciliation at connector startup
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_AT_RESTART_ENABLE: "true"

  6. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud, navigate to Settings → Applications.

  2. Select BigQuery from the list of Connected Applications.

  3. Click on the application name or the icon, then click on Access Management → ADVANCED tab.

  4. To enable Ranger tag reconciliation, add the property under Add New Custom Properties, update the index based on the connector you are configuring :

    Bash
    1
    ranger.policysync.connector.0.sync.rangertag.reconcile.enable=true

  5. To update the reconciliation interval (optional), add the following property:

    Bash
    1
    ranger.policysync.connector.0.sync.rangertag.reconcile.interval.sec=420

  6. To enable reconciliation at connector restart (optional), add the following property:

    Bash
    1
    ranger.policysync.connector.0.sync.rangertag.reconcile.at.restart.enable=true

  7. Click SAVE.

  8. Once saved and enabled, the BigQuery connector will start. You can hover on the VIEW LOGS button to check the status, either Running or Stopped.

Note

Perform the following steps only if the connector does not reflect the updated configuration and requires a restart.

Restart the BigQuery Connector:

  1. Go to Settings → Applications → select the BigQuery connector application.

  2. Edit the application → Disable it → and Save it.

  3. Open the same application again and then: Enable it and Save it.

Use Cases

Scenario 1: Tag Accidentally Deleted from BigQuery

  • A policy tag is manually deleted from BigQuery Data Catalog
  • Ranger tag reconciliation detects the missing tag on next cycle
  • The tag is automatically recreated in BigQuery
  • All column mappings that previously had this tag are automatically restored

Scenario 2: Column Tag Manually Removed

  • A user removes a policy tag from specific columns in BigQuery (tag still exists)
  • Ranger tag reconciliation detects the missing column mapping
  • The tag is automatically reapplied to those columns

Monitoring

You can monitor the reconciliation process through connector logs. Look for these log messages:

Text Only
1
2
3
4
5
6
INFO  RangerTagReconcileLoader - Found X service tag defs in RocksDB for reconciliation
INFO  RangerTagReconcileLoader - Recreating missing tag in target system: <tag_name>
INFO  RangerTagReconcileLoader - Successfully reconciled and added tag: <tag_name>
INFO  RangerTagReconcileLoader - Reconciling resource mappings for X recreated tags
INFO  RangerTagReconcileLoader - Successfully reapplied tag-resource mapping for resource: <resource_key>
INFO  RangerTagReconcileLoader - STATS: RangerTagReconcileLoader(connector=BigQuery): {timeTaken:XXXms,changeLogId:XX,isSuccess:true,tagsAdded:X,tagsFailed:X,resourceMappingsAdded:X,resourceMappingsFailed:X}

Comparison: Service Tag vs Ranger Tag Reconciliation

Aspect Service Tag Reconciliation Ranger Tag Reconciliation
Direction BigQuery → Ranger Ranger → BigQuery
Purpose Sync connector-discovered tags to Ranger Sync Ranger tags to BigQuery
Source of Truth BigQuery Connector Ranger/Portal
Tag Creation Creates tags in Ranger Creates policy tags in BigQuery
Tag Deletion Can delete orphaned tags from Ranger Does not delete tags
Resource Mappings N/A Handles column-level tag mappings
Typical Use Case Initial tag discovery and sync Disaster recovery, manual deletion protection
Default State Enabled Disabled
Interval 540 seconds (9 minutes) 420 seconds (7 minutes)

Best Practice

Enable both reconciliation types for comprehensive tag management:

  • Service Tag Reconciliation: Keeps Ranger updated with tags discovered from BigQuery
  • Ranger Tag Reconciliation: Ensures BigQuery reflects all tags defined in Ranger and protects against manual deletions