Parallel Grant Statements for Databricks Unity Catalog¶

Configure the Databricks Unity Catalog connector to run the GRANT and REVOKE statements of one resource in parallel, so a policy change that produces many statements is applied faster.

Note

This feature applies only when the connector manages permissions over JDBC (the default). It has no effect when the connector uses the Databricks API to manage permissions.

Overview¶

By default, the connector applies the GRANT and REVOKE statements of a resource one at a time. When parallel grant statements are enabled, it runs several of them at the same time, so policy changes are applied faster.

Revokes still run before grants, so the final permissions always match your policies.
A failed statement is retried as before and does not stop the others.

Use this when policy changes often produce many grant statements, for example a table shared with many users, groups or roles.

Configuration properties¶

Property	Description	Default
`CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_PARALLEL_GRANT_UPDATES`	Run the `GRANT` / `REVOKE` statements of one resource in parallel, on both changelog queues. `false` runs them inline and in order.	`false`
`CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_THREAD_COUNT`	Helper threads per changelog queue (on-demand and offline each get this many). Size the JDBC connection pool for this value (see below). Used only when the feature is enabled.	`16`
`CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_MAX_INFLIGHT_PER_RESOURCE`	Maximum statements of one resource running at the same time, including the thread that processes the resource (at most the helper thread count + 1). Used only when the feature is enabled.	`16`

Matching custom properties (Portal and PrivaceraCloud):

Custom property	Default
`ranger.policysync.connector.0.enable.parallel.grant.updates`	`false`
`ranger.policysync.connector.0.parallel.grant.updates.thread.count`	`16`
`ranger.policysync.connector.0.parallel.grant.updates.max.inflight.per.resource`	`16`

Increase the JDBC connection pool size

Each grant statement that runs in parallel uses its own JDBC connection, so this feature needs a larger connection pool than the default (30). The connector does not check the pool size: with a pool that is too small, statements wait for a free connection at busy times, which slows policy changes down.

Set CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_MAX_POOL_SIZE in Privacera Manager (custom property ranger.policysync.connector.0.jdbc.maximum.pool.size) to at least:

loader threads + 2 × (resource.process threads + helper threads)

With the default thread counts (21 loader threads, 10 resource.process threads) and 16 helper threads, this is 73. Recompute it if you changed the resource or permission loader thread counts or ranger.policysync.connector.0.resource.process.thread.max.

Also set the minimum idle connections (custom property ranger.policysync.connector.0.jdbc.min.idle.connection, default 3) to at least the helper thread count, so the first statements after an idle period do not wait for new connections. Privacera Manager has no variable for this property; add it as a custom property.

Setup¶

Configure the properties and restart the connector to apply the changes.

Self-Managed (Portal Configuration)Self-Managed (YAML Configuration) and Data PlanePrivaceraCloud

Navigate to Settings → Applications in the Self-Managed Portal.
From the list of Connected Applications, select Databricks Unity Catalog.
Select the application name or the icon to edit. Then, go to the Access Management tab.

Under the ADVANCED tab, under Add New Custom Properties, add:

Properties
ranger.policysync.connector.0.enable.parallel.grant.updates=true
ranger.policysync.connector.0.parallel.grant.updates.thread.count=16
ranger.policysync.connector.0.parallel.grant.updates.max.inflight.per.resource=16
ranger.policysync.connector.0.jdbc.maximum.pool.size=73
ranger.policysync.connector.0.jdbc.min.idle.connection=16

Select SAVE to apply the changes.

SSH to the instance where Privacera Manager is installed.
Run the following command to open the .yml file to be edited.

If you have multiple connectors, then replace instance1 with the appropriate connector instance name.
Bash
```
vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml
```

Add or set the following properties:

YAML
CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_PARALLEL_GRANT_UPDATES: "true"
CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_THREAD_COUNT: "16"
CONNECTOR_DATABRICKS_UNITY_CATALOG_PARALLEL_GRANT_UPDATES_MAX_INFLIGHT_PER_RESOURCE: "16"
CONNECTOR_DATABRICKS_UNITY_CATALOG_JDBC_MAX_POOL_SIZE: "73"

Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh setup
Step 2 - Apply the Privacera Manager helm charts.
Bash
1 2
cd ~/privacera/privacera-manager ./pm_with_helm.sh upgrade
Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.
Bash
1 2
cd ~/privacera/privacera-manager ./privacera-manager.sh post-install

In PrivaceraCloud portal, navigate to Settings → Applications.
On the Connected Applications screen, select Databricks Unity Catalog.
Select the pen icon or the Account Name to modify the settings.
On the Edit Application screen, go to Access Management → ADVANCED tab.

Under Add New Custom Properties, add:

Properties
ranger.policysync.connector.0.enable.parallel.grant.updates=true
ranger.policysync.connector.0.parallel.grant.updates.thread.count=16
ranger.policysync.connector.0.parallel.grant.updates.max.inflight.per.resource=16
ranger.policysync.connector.0.jdbc.maximum.pool.size=73
ranger.policysync.connector.0.jdbc.min.idle.connection=16

Select SAVE to apply the changes.

Monitoring¶

When the feature is enabled, the connector:

Writes a grant_apply stats line to the connector log every minute in which grant statements ran, with the statement count and timings and the JDBC connection pool usage. A waiting count above 0 in the pool part means statements are waiting for a connection: increase the pool size.
Logs a warning when a grant statement runs longer than the GRANT statement timeout, or longer than the JDBC statement timeout (30 minutes by default) when no GRANT statement timeout is set (see Fine Tuning JDBC Connection).
Includes the time a statement waited before it started in the ACCESS audit time, shown as queueTime in the audit extra info.

To turn the feature off, set CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_PARALLEL_GRANT_UPDATES (or the custom property ranger.policysync.connector.0.enable.parallel.grant.updates) to false and restart the connector.

Prev topic: Advanced Configuration