Dedicated Changelog Queue for Policy Changes¶

Configure the Databricks Unity Catalog connector to process policy changes on their own queue, so a policy change is applied without waiting behind a long resource sync or reconcile.

Overview¶

By default, the connector processes every offline changelog (policy changes, principal changes, resource syncs and reconciles) one after another on a single queue. A policy change that arrives while a large resource sync is running waits until that sync finishes.

When the dedicated changelog queue is enabled:

Policy changes (RANGER_POLICIES) and, optionally, principal changes (RANGER_PRINCIPALS) go to a separate fast queue with its own processing thread.
Resource syncs, reconciles, role, GDS and user-store changes stay on the offline queue and run in parallel with the fast queue.
Role, GDS and user-store changes are applied by the offline queue once the latest policy engine is published, so they are not delayed by policy changes but can take longer than policy changes to reach Unity Catalog.
When two changes touch the same resource at the same time, the newer one wins. A change based on older data never overwrites a newer result.
If a policy change cannot get to a resource within 30 seconds because another change is still applying to it, the resource is set aside and re-applied shortly after, instead of holding up the rest of the policy change. A resource set aside this way is re-applied after a connector restart too.

Use this when policy changes must reach Unity Catalog quickly in an environment with long-running resource syncs or reconciles.

Configuration properties¶

Property	Description	Default	Allowed values
`CONNECTOR_DATABRICKS_UNITY_CATALOG_DEDICATED_QUEUE_ENABLED`	Processes the changelog types below on a dedicated queue and thread. When `false`, all changelogs share the offline queue.	`false`	`true`, `false`
`CONNECTOR_DATABRICKS_UNITY_CATALOG_DEDICATED_QUEUE_CHANGELOG_TYPES`	Comma-separated changelog types routed to the dedicated queue when it is enabled. `RANGER_POLICIES` is always included, and `RANGER_PRINCIPALS` is the only other supported value; any other value is logged and ignored.	`RANGER_POLICIES`	`RANGER_POLICIES`, `RANGER_PRINCIPALS`

Matching custom properties (Self-Managed Portal):

Custom property	Default
`ranger.policysync.connector.0.dedicated.queue.enabled`	`false`
`ranger.policysync.connector.0.dedicated.queue.changelog.types`	`RANGER_POLICIES`

Not supported with tag-based masking or tag sync

The dedicated changelog queue does not support tag-based masking or tag sync in this version. If either is enabled for the connector, the connector ignores CONNECTOR_DATABRICKS_UNITY_CATALOG_DEDICATED_QUEUE_ENABLED and keeps using the single offline queue:

Feature	Property	PM variable	Default
Tag-based masking	`ranger.policysync.connector.0.tag.masking.enabled`, or the global `tag.masking.enabled` when the connector property is not set	`ENABLE_TAG_MASKING`	`false`
Tag sync	`ranger.policysync.connector.0.purview.tag.sync.enabled`	`CONNECTOR_PURVIEW_TAG_SYNC_ENABLED`	`false`

Both are false by default, so you only need to change them if you turned either feature on. The connector checks them every time it starts, so the result is the same on every start and a change takes effect after one restart.

Ranger tag changelogs

RANGER_TAGS is not supported on the dedicated queue. It cannot be added to CONNECTOR_DATABRICKS_UNITY_CATALOG_DEDICATED_QUEUE_CHANGELOG_TYPES, and Ranger tag changelogs always stay on the offline queue. A Ranger tag service linked to the Unity Catalog service does not turn the dedicated queue off.

JDBC connection sizing

With the dedicated queue, up to three changelog processors run at the same time: the fast queue, the offline queue and, when the on-demand dual queue is enabled, the on-demand queue. Each processor uses up to 10 worker threads by default (custom property ranger.policysync.connector.0.resource.process.thread.max), and each worker can hold a JDBC connection. If the processors' workers together (2 × workers, or 3 × workers with the on-demand dual queue) exceed CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_MAX_POOL_SIZE (custom property ranger.policysync.connector.0.jdbc.maximum.pool.size, default 30), the connector ignores the flag and keeps using the single offline queue.

If the optional changelog thread pool V2 (CONNECTOR_CHANGELOG_USE_THREAD_POOL_V2, false by default) is turned on, each processor uses exactly 10 worker threads, whatever resource.process.thread.max is set to.

Setup¶

Configure the properties and restart the connector to apply the changes.

Self-Managed (Portal Configuration)Self-Managed (YAML Configuration) and Data Plane

Navigate to Settings → Applications in the Self-Managed Portal.
From the list of Connected Applications, select Databricks Unity Catalog.
Select the application name or the icon to edit. Then, go to the Access Management tab.

Under the ADVANCED tab, under Add New Custom Properties, add:

Properties
ranger.policysync.connector.0.dedicated.queue.enabled=true
ranger.policysync.connector.0.dedicated.queue.changelog.types=RANGER_POLICIES

Select SAVE to apply the changes.

SSH to the instance where Privacera Manager is installed.
Run the following command to open the .yml file to be edited.

If you have multiple connectors, then replace instance1 with the appropriate connector instance name.
Bash
```
vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml
```

Add or set the following properties:

YAML
CONNECTOR_DATABRICKS_UNITY_CATALOG_DEDICATED_QUEUE_ENABLED: "true"
CONNECTOR_DATABRICKS_UNITY_CATALOG_DEDICATED_QUEUE_CHANGELOG_TYPES: "RANGER_POLICIES"

Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh setup
Step 2 - Apply the Privacera Manager helm charts.
Bash
1 2
cd ~/privacera/privacera-manager ./pm_with_helm.sh upgrade
Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh post-install

To turn the feature off, set CONNECTOR_DATABRICKS_UNITY_CATALOG_DEDICATED_QUEUE_ENABLED (or the custom property ranger.policysync.connector.0.dedicated.queue.enabled) to false, then update the Privacera Manager instance or, for a custom property, restart the connector. The connector returns to the single offline queue and removes the per-resource tracking data it kept for the dedicated queue.

Verify the configuration¶

On startup with the flag on, the connector log shows dedicated changelog queue enabled, followed by the changelog types that use the fast queue.
If the connector ignored the flag, the log shows an ERROR containing falling back to the single offline queue, and the connector audits a DEDICATED QUEUE FALLBACK activity with the reason (tags or pool_sizing).
Connector metrics with the prefix ps_core_fast_changelog_task_queue_ show the fast queue's activity, for example the age of the oldest queued changelog per queue, the number of resources waiting for a re-apply, and how often a policy change had to set a resource aside because another change was still applying to it.

Prev topic: Advanced Configuration