Tag Reconcile Loader Configuration¶

Goal¶

This guide explains how to configure the BigQuery connector to automatically reconcile tags and tag permissions between the BigQuery connector and Apache Ranger.

The BigQuery connector supports the following reconciliation types:

Service Tag Reconciliation¶

Reconciles tags between the BigQuery connector and Apache Ranger. Use this when useServiceTag=true (Native Tag-Based Masking).

Ranger Tag Reconciliation¶

When using Ranger tags (useServiceTag=false), this includes two types of reconciliation:

Tag and Resource Mapping Reconciliation: Reconciles tags and their column mappings between Ranger/Portal and BigQuery
Tag Permission Reconciliation: Reconciles tag-based access policies and tag-based masking policies between Ranger/Portal and BigQuery

Prerequisites¶

Before you begin, ensure the following:

Privacera Manager is installed and the base installation is operational.
The BigQuery connector is configured and running.
Apache Ranger is properly configured and accessible.

Tag Masking Requirement

These tag reconciliation loader properties will only work if tag masking is enabled in the connector.

Choosing the Right Reconciliation Type¶

The type of reconciliation you can use depends on the useServiceTag property configuration:

`useServiceTag` Value	Tag Type	Reconciliation Type to Use
`true` (default)	Service Tags (Native Tag-Based Masking)	Service Tag Reconciliation
`false`	Ranger Tags (Ranger Tag-Based Masking)	Ranger Tag Reconciliation (Tag & Mapping or Permissions)

Property Configuration

For Service Tags (useServiceTag=true): Use Service Tag Reconciliation to sync tags discovered from BigQuery back to Ranger. For detailed implementation, refer to the Native Tag Masking Guide.
For Ranger Tags (useServiceTag=false): Use Ranger Tag Reconciliation to sync tags and permissions from Ranger/Portal to BigQuery. For detailed implementation, refer to the Ranger Tag-Based Masking Guide.

The useServiceTag property is configured in the Native Row Filter and Tag Masking configuration page under the ADVANCED tab.

Service Tag Reconciliation¶

Overview¶

Service Tag Reconciliation maintains tag consistency between your BigQuery connector and Apache Ranger by:

Monitoring tag differences: Compares tags loaded by the connector with those present in Ranger
Automatic reconciliation: Creates missing tags in Ranger or removes orphaned tags
Configurable intervals: Runs at specified intervals to ensure ongoing consistency

When to Use Service Tag Reconciliation

Use this reconciliation type when useServiceTag=true (default), which means you are using Native Tag-Based Masking with Service Tags. Service tags are created in BigQuery and synchronized into Privacera as read-only tags. For detailed implementation steps, refer to the Native Tag Masking Guide.

Service Tag Reconcile Loader Properties:

Property Name	Description	Default Value	Supported Values
CONNECTOR_BIGQUERY_TAG_RECONCILE_LOADER_ENABLED	Enable or disable the service tag reconcile loader functionality	`true`	`true`, `false`
CONNECTOR_BIGQUERY_SERVICE_TAG_RECONCILE_SYNC_INTERVAL	Set the interval for service tag reconcile sync process in seconds	`540`	Any numeric value in seconds

Configuration Steps¶

Warning

Enabling this feature will automatically create and delete tags in Ranger based on connector data.
Lower interval values result in more frequent reconciliation but may increase system load.
Be sure to replace the example values with your actual configuration values.

Restart Required

Any changes to these properties require a restart of the BigQuery connector application for the updates to take effect.

Self Managed (Portal Configuration)Self Managed (YAML Configuration) and Data PlanePrivacera Cloud

Navigate to Settings → Applications in the Self-Managed Portal.
Select BigQuery from the list of Connected Applications.
Click on the application name or the icon, then click on Access Management → ADVANCED tab.
To update the tag reconcile sync interval, add the following property under the Add New Custom Properties section:
Bash
1
ranger.policysync.connector.0.sync.servicetag.reconcile.interval.sec=540
To disable the tag reconcile loader, add the following property under the Add New Custom Properties section:
Bash
1
ranger.policysync.connector.0.tag.reconcile.loading.enabled=false
Click SAVE to apply the changes.

SSH to the instance where Privacera Manager is installed.
Run the following command to open the .yml file to be edited.

If you have multiple connectors, then replace instance1 with the appropriate connector instance name.
Bash
1
vi ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/vars.connector.bigquery.yml

To update the tag reconcile sync interval, add or modify the following property:

YAML
# Set tag reconcile sync interval (in seconds)
CONNECTOR_BIGQUERY_SERVICE_TAG_RECONCILE_SYNC_INTERVAL: "540"

To disable the tag reconcile loader, add or modify the following property:

YAML
# Disable tag reconcile loader
CONNECTOR_BIGQUERY_TAG_RECONCILE_LOADER_ENABLED: "false"

Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh setup
Step 2 - Apply the Privacera Manager helm charts.
Bash
1 2
cd ~/privacera/privacera-manager ./pm_with_helm.sh upgrade
Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh post-install

In PrivaceraCloud, navigate to Settings → Applications.
Select BigQuery from the list of Connected Applications.
Click on the application name or the icon, then click on Access Management → ADVANCED tab.
To update the tag reconcile sync interval, add the following property under the Add New Custom Properties section:
Bash
1
ranger.policysync.connector.0.sync.servicetag.reconcile.interval.sec=540
To disable the tag reconcile loader, add the following property under the Add New Custom Properties section:
Bash
1
ranger.policysync.connector.0.tag.reconcile.loading.enabled=false
Click SAVE.
Once saved and enabled, the BigQuery connector will start. You can hover on the VIEW LOGS button to check the status, either Running or Stopped.

Note

Perform the following steps only if the connector does not reflect the updated configuration and requires a restart.

Restart the BigQuery Connector:

Go to Settings → Applications → select the BigQuery connector application.
Edit the application → Disable it → and Save it.
Open the same application again and then: Enable it and Save it.

Ranger Tag Reconciliation¶

When using Ranger tags, you can configure two types of reconciliation to ensure tags and their permissions remain synchronized between Ranger/Portal and BigQuery:

Tag and Resource Mapping Reconciliation: Ensures tags and their column mappings remain synchronized
Tag Permission Reconciliation: Ensures tag-based access and masking policies remain synchronized

When to Use Ranger Tag Reconciliation

Use this reconciliation type when useServiceTag=false, which means you are using Ranger Tag-Based Masking. Ranger tags are defined and managed in Privacera/Ranger and automatically created in BigQuery. For detailed implementation steps, refer to the Ranger Tag-Based Masking Guide.

Tag and Resource Mapping Reconciliation¶

Overview¶

Tag and Resource Mapping Reconciliation ensures tags and their resource mappings (column-level policy tags) remain synchronized between Ranger/Portal and BigQuery. This advanced reconciliation feature:

Detects missing tags: Identifies tags that exist in Ranger but got accidently/manually deleted from BigQuery
Recreates tags automatically: Restores deleted policy tags in BigQuery Data Catalog
Restores tag mappings: Automatically reapplies column-level tag mappings for recreated tags
Monitors existing mappings: Checks if tag-resource mappings were manually removed from BigQuery columns and reapplies them
Prevents drift: Ensures Ranger remains the authoritative source for tag definitions and mappings

Required Configuration

To use Ranger Tag Reconciliation, the following property must be set before configuring the reconciliation settings:

Properties
ranger.policysync.connector.0.use.service.tags=false

Where to configure:

Access Management → Advanced settings of the BigQuery connector
vars.connector.bigquery.yml when performing setup using YAML for Self-Managed or Data Plane deployments

Purpose:

This property selects Ranger Tag Reconciliation as the active tag synchronization mechanism.

Tag and Resource Mapping Reconciliation Properties:

Property Name	Description	Default Value	Supported Values
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_ENABLE	Enable or disable Ranger tag reconciliation functionality	`false`	`true`, `false`
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_INTERVAL_SEC	Set the interval for Ranger tag reconciliation process in seconds	`420`	Any numeric value in seconds (default: 7 minutes)
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_AT_RESTART_ENABLE	Run reconciliation immediately at connector startup	`false`	`true`, `false`

Configuration Steps¶

Important Considerations

This feature focuses on tag creation and restoration only - it does not delete tags from BigQuery
Ranger is the authoritative source of truth for all tag definitions and mappings
Lower interval values result in faster recovery but may increase system load and API calls to BigQuery
Both normal (flat) and hierarchical policy tag structures are supported
Be sure to replace the example values with your actual configuration values

Restart Required

Any changes to these properties require a restart of the BigQuery connector application for the updates to take effect.

Self Managed (Portal Configuration)Self Managed (YAML Configuration) and Data PlanePrivacera Cloud

Navigate to Settings → Applications in the Self-Managed Portal.
Select BigQuery from the list of Connected Applications.
Click on the application name or the icon, then click on Access Management → ADVANCED tab.
To enable Ranger tag reconciliation, add the property under Add New Custom Properties, update the index based on the connector you are configuring :
Bash
1
ranger.policysync.connector.0.sync.rangertag.reconcile.enable=true
To update the reconciliation interval (optional), add the following property:
Bash
1
ranger.policysync.connector.0.sync.rangertag.reconcile.interval.sec=420
To enable reconciliation at connector restart (optional), add the following property:
Bash
1
ranger.policysync.connector.0.sync.rangertag.reconcile.at.restart.enable=true
Click SAVE to apply the changes.

SSH to the instance where Privacera Manager is installed.
Run the following command to open the .yml file to be edited.

If you have multiple connectors, then replace instance1 with the appropriate connector instance name.
Bash
1
vi ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/vars.connector.bigquery.yml

To enable Ranger tag reconciliation, add or modify the following property:

YAML
# Enable Ranger tag reconciliation
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_ENABLE: "true"

To update the reconciliation interval (optional), add or modify the following property:

YAML
# Set Ranger tag reconciliation interval (in seconds, default: 7 minutes)
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_INTERVAL_SEC: "420"

To enable reconciliation at connector restart (optional), add or modify the following property:

YAML
# Run reconciliation at connector startup
CONNECTOR_BIGQUERY_RANGER_TAG_RECONCILE_AT_RESTART_ENABLE: "true"

Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh setup
Step 2 - Apply the Privacera Manager helm charts.
Bash
1 2
cd ~/privacera/privacera-manager ./pm_with_helm.sh upgrade
Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh post-install

In PrivaceraCloud, navigate to Settings → Applications.
Select BigQuery from the list of Connected Applications.
Click on the application name or the icon, then click on Access Management → ADVANCED tab.
To enable Ranger tag reconciliation, add the property under Add New Custom Properties, update the index based on the connector you are configuring :
Bash
1
ranger.policysync.connector.0.sync.rangertag.reconcile.enable=true
To update the reconciliation interval (optional), add the following property:
Bash
1
ranger.policysync.connector.0.sync.rangertag.reconcile.interval.sec=420
To enable reconciliation at connector restart (optional), add the following property:
Bash
1
ranger.policysync.connector.0.sync.rangertag.reconcile.at.restart.enable=true
Click SAVE.
Once saved and enabled, the BigQuery connector will start. You can hover on the VIEW LOGS button to check the status, either Running or Stopped.

Note

Perform the following steps only if the connector does not reflect the updated configuration and requires a restart.

Restart the BigQuery Connector:

Go to Settings → Applications → select the BigQuery connector application.
Edit the application → Disable it → and Save it.
Open the same application again and then: Enable it and Save it.

Use Cases¶

Scenario 1: Tag Accidentally Deleted from BigQuery

A policy tag is manually deleted from BigQuery Data Catalog
Tag reconciliation detects the missing tag on next cycle
The tag is automatically recreated in BigQuery
All column mappings that previously had this tag are automatically restored

Scenario 2: Column Tag Manually Removed

A user removes a policy tag from specific columns in BigQuery (tag still exists)
Tag reconciliation detects the missing column mapping
The tag is automatically reapplied to those columns

Monitoring¶

You can monitor the reconciliation process through connector logs. Look for these log messages:

Text Only

INFO  RangerTagReconcileLoader - Found X service tag defs in RocksDB for reconciliation
INFO  RangerTagReconcileLoader - Recreating missing tag in target system: <tag_name>
INFO  RangerTagReconcileLoader - Successfully reconciled and added tag: <tag_name>
INFO  RangerTagReconcileLoader - Reconciling resource mappings for X recreated tags
INFO  RangerTagReconcileLoader - Successfully reapplied tag-resource mapping for resource: <resource_key>
INFO  RangerTagReconcileLoader - STATS: RangerTagReconcileLoader(connector=BigQuery): {timeTaken:XXXms,changeLogId:XX,isSuccess:true,tagsAdded:X,tagsFailed:X,resourceMappingsAdded:X,resourceMappingsFailed:X}

Tag Permission Reconciliation¶

Overview¶

Ranger Tag Permission Reconciliation ensures tag-based access policies and tag-based masking policies remain synchronized between Ranger/Portal and BigQuery. This reconciliation feature focuses on permissions rather than tag definitions:

Detects missing permissions: Identifies tag-based access policies (fine-grained reader roles) that exist in Ranger but were manually removed from BigQuery policy tags
Detects missing masking policies: Identifies tag-based data masking policies that exist in Ranger but were manually removed from BigQuery data policies
Reapplies permissions automatically: Restores fine-grained reader role bindings on BigQuery policy tags
Reapplies masking policies: Restores data masking policies for tags that had them configured
Prevents permission drift: Ensures Ranger remains the authoritative source for tag-based access and masking policies
Handles multiple taxonomies: Supports tags mapped to multiple policy tags across different taxonomies within same project location

Tag Permission Reconciliation Properties:

Property Name	Description	Default Value	Supported Values
CONNECTOR_BIGQUERY_RANGER_TAG_PERMISSIONS_ENABLE	Enable or disable Ranger tag permission reconciliation functionality	`false`	`true`, `false`
CONNECTOR_BIGQUERY_RANGER_TAG_PERMISSIONS_INTERVAL_SEC	Set the interval for Ranger tag permission reconciliation process in seconds	`420`	Any numeric value in seconds (default: 7 minutes)
CONNECTOR_BIGQUERY_RANGER_TAG_PERMISSIONS_SYNC_AT_RESTART_ENABLE	Run permission reconciliation immediately at connector startup	`false`	`true`, `false`

Configuration Steps¶

Important Considerations

This feature focuses on permission reconciliation only - it does not modify tag definitions or tag-resource mappings
Ranger is the authoritative source of truth for all tag-based access and masking policies
Handles tags mapped to single or multiple taxonomies

Restart Required

Any changes to these properties require a restart of the BigQuery connector application for the updates to take effect.

Self Managed (Portal Configuration)Self Managed (YAML Configuration) and Data PlanePrivacera Cloud

Navigate to Settings → Applications in the Self-Managed Portal.
Select BigQuery from the list of Connected Applications.
Click on the application name or the icon, then click on Access Management → ADVANCED tab.
To enable Ranger tag permission reconciliation, add the property under Add New Custom Properties, update the index based on the connector you are configuring :
Bash
1
ranger.policysync.connector.0.sync.rangertag.permissions.enable=true
To update the reconciliation interval (optional), add the following property:
Bash
1
ranger.policysync.connector.0.sync.rangertag.permissions.interval.sec=420
To enable reconciliation at connector restart (optional), add the following property:
Bash
1
ranger.policysync.connector.0.sync.rangertag.permissions.sync.at.restart.enable=true
Click SAVE to apply the changes.

SSH to the instance where Privacera Manager is installed.
Run the following command to open the .yml file to be edited.

If you have multiple connectors, then replace instance1 with the appropriate connector instance name.
Bash
1
vi ~/privacera/privacera-manager/config/custom-vars/connectors/bigquery/instance1/vars.connector.bigquery.yml

To enable Ranger tag permission reconciliation, add or modify the following property:

YAML
# Enable Ranger tag permission reconciliation
CONNECTOR_BIGQUERY_RANGER_TAG_PERMISSIONS_ENABLE: "true"

To update the reconciliation interval (optional), add or modify the following property:

YAML
# Set Ranger tag permission reconciliation interval (in seconds, default: 7 minutes)
CONNECTOR_BIGQUERY_RANGER_TAG_PERMISSIONS_INTERVAL_SEC: "420"

To enable reconciliation at connector restart (optional), add or modify the following property:

YAML
# Run permission reconciliation at connector startup
CONNECTOR_BIGQUERY_RANGER_TAG_PERMISSIONS_SYNC_AT_RESTART_ENABLE: "true"

Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh setup
Step 2 - Apply the Privacera Manager helm charts.
Bash
1 2
cd ~/privacera/privacera-manager ./pm_with_helm.sh upgrade
Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh post-install

In PrivaceraCloud, navigate to Settings → Applications.
Select BigQuery from the list of Connected Applications.
Click on the application name or the icon, then click on Access Management → ADVANCED tab.
To enable Ranger tag permission reconciliation, add the property under Add New Custom Properties, update the index based on the connector you are configuring :
Bash
1
ranger.policysync.connector.0.sync.rangertag.permissions.enable=true
To update the reconciliation interval (optional), add the following property:
Bash
1
ranger.policysync.connector.0.sync.rangertag.permissions.interval.sec=420
To enable reconciliation at connector restart (optional), add the following property:
Bash
1
ranger.policysync.connector.0.sync.rangertag.permissions.sync.at.restart.enable=true
Click SAVE.
Once saved and enabled, the BigQuery connector will start. You can hover on the VIEW LOGS button to check the status, either Running or Stopped.

Note

Perform the following steps only if the connector does not reflect the updated configuration and requires a restart.

Restart the BigQuery Connector:

Go to Settings → Applications → select the BigQuery connector application.
Edit the application → Disable it → and Save it.
Open the same application again and then: Enable it and Save it.

Use Cases¶

Scenario 1: Fine-Grained Reader Role Manually Removed

A user manually removed from fine-grained reader role (roles/datacatalog.categoryFineGrainedReader) from a policy tag in BigQuery
Tag permission reconciliation detects the missing permission on next cycle
The fine-grained reader role is automatically reapplied to the policy tag with the correct principals (users, groups, roles)

Scenario 2: Data Masking Policy Manually Removed

A data masking policy is manually deleted from a BigQuery data policy associated with a tag
A user manually removed from data masking policy will also be restored
Tag permission reconciliation detects the missing masking policy
The data masking policy is automatically recreated and applied with the correct mask type and principals

Scenario 3: Multiple Taxonomy Permissions Inconsistency

A tag is mapped to multiple columns across different taxonomies
Permissions are manually modified on one policy tag but not others, causing inconsistency
Tag permission reconciliation detects the inconsistency and reapplies permissions uniformly across all policy tags

Monitoring¶

You can monitor the permission reconciliation process through connector logs. Look for these log messages:

Text Only

1
2

INFO  RangerTagPermissionLoader - Found X tag permissions for reconciliation
INFO  RangerTagPermissionLoader - STATS: RangerTagPermissionLoader(connector=BigQuery): {timeTaken:XXXms,changeLogId:XX,isSuccess:true,permissionsReapplied:X,permissionsFailed:X}

Comparison: Service Tag vs Ranger Tag Reconciliation¶

Aspect	Service Tag Reconciliation	Ranger Tag Reconciliation
Direction	BigQuery → Ranger	Ranger → BigQuery
Purpose	Sync connector-discovered tags to Ranger	Sync Ranger tags, column mappings, and tag-based permissions to BigQuery
Source of Truth	BigQuery Connector	Ranger/Portal
Tag Creation	Creates tags in Ranger	Creates policy tags in BigQuery (Tag & Mapping aspect)
Tag Deletion	Can delete orphaned tags from Ranger	Does not delete tags
Resource Mappings	N/A	Handles column-level tag mappings (Tag & Mapping aspect)
Permissions	N/A	Handles fine-grained reader roles and data masking policies (Permissions aspect)
Sub-Types	N/A	Tag & Mapping: Reconciles tags and column mappings Permissions: Reconciles tag-based access and masking policies
Typical Use Case	Initial tag discovery and sync	Disaster recovery, manual deletion protection, permission drift recovery
Default State	Enabled	Disabled
Interval	540 seconds (9 minutes)	420 seconds (7 minutes)

Best Practice

Enable both reconciliation types for comprehensive tag and permission management:

Service Tag Reconciliation: Keeps Ranger updated with tags discovered from BigQuery
Ranger Tag Reconciliation: Ensures BigQuery reflects all tags, mappings, and permissions defined in Ranger, protecting against manual deletions and permission removals

Prev topic: Advanced Configuration