Skip to content

Databricks Unity Catalog Resource Loader Configuration

This section outlines the configuration settings for resource loading behavior and threading when using the Databricks Unity Catalog Connector. These settings help improve performance by enabling parallel loading of Unity Catalog entities such as schemas, tables, columns, functions, and volumes.

Connector Type

These configurations apply only when the Databricks Unity Catalog connector is configured via JDBC.

Configuration Parameters

You can configure the following:

  • Optional toggle to enable column loading
  • Optional toggle to enable model resource loading
  • Thread counts for each supported resource type

If these parameters are not specified, default values may apply, depending on the system and connector version.

Setup

Warning

  • All configuration values must be entered as strings for Self Managed (YAML Configuration) and Data Plane.
  • Thread-related parameters should be tuned based on the volume of metadata and available system resources.

Usage Guide

  • Load Columns: Enables loading of column metadata. Defaults to true if not set.
  • Thread properties: Controls the number of threads used to load each resource type. Increase values for faster loading at the cost of higher CPU/memory usage.
  • These properties only take effect when the Databricks Unity Catalog connector is configured via JDBC.

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. From the list of Connected Applications, select Databricks Unity Catalog.

  3. Click the icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access Management → ADVANCED tab.

  5. Under ADVANCED tab, add the following properties under Add New Custom Properties:

    Bash
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
    # enables loading of column metadata
ranger.policysync.connector.0.load.columns=true

# Thread Configuration – Schema
ranger.policysync.connector.0.load.resources.load.schema.thread.count=2
ranger.policysync.connector.0.load.resources.load.schema.thread.min.count=2

# Thread Configuration – Table
ranger.policysync.connector.0.load.resources.load.table.thread.count=2
ranger.policysync.connector.0.load.resources.load.table.thread.min.count=2

# Thread Configuration – Column
ranger.policysync.connector.0.load.resources.load.column.thread.count=3
ranger.policysync.connector.0.load.resources.load.column.thread.min.count=3

# Thread Configuration – Function
ranger.policysync.connector.0.load.resources.load.function.thread.min.count=2
ranger.policysync.connector.0.load.resources.load.function.thread.count=2

# Thread Configuration – Volume
ranger.policysync.connector.0.load.resources.load.volume.thread.min.count=2
ranger.policysync.connector.0.load.resources.load.volume.thread.count=2

  6. Click SAVE to apply the changes.

  1. SSH into the instance where Privacera Manager is installed.

  2. Open the Unity Catalog connector configuration file:

    Note

    Replace instance1 with your actual connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks_unity/instance1/vars.connector.databricks_unity.yml

  3. Configure the resource loader parameters as needed:

    YAML
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
    # Optional Toggle – enables loading of column metadata
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_COLUMNS: "true"

# Optional Toggle – enables loading of Model resource
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_MODEL_ENABLED: "true"

# Thread Configuration – Schema
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_SCHEMA_THREADS: "2"
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_SCHEMA_MIN_THREADS: "2"

# Thread Configuration – Table
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_TABLE_THREADS: "2"
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_TABLE_MIN_THREADS: "2"

# Thread Configuration – Column
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_COLUMN_THREADS: "3"
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_COLUMN_MIN_THREADS: "3"

# Thread Configuration – Function
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_FUNCTION_THREADS: "2"
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_FUNCTION_MIN_THREADS: "2"

# Thread Configuration – Model
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_MODEL_THREADS: "2"
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_MODEL_MIN_THREADS: "2"

# Thread Configuration – Volume
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_VOLUME_THREADS: "2"
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_LOAD_VOLUME_MIN_THREADS: "2"

  4. After updating the configuration, apply the changes by running:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud portal, navigate to Settings → Applications.

  2. On the Connected Applications screen, select Databricks Unity Catalog.

  3. Click the icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access Management → ADVANCED tab.

  5. Enable Load Column metadata to load column-level metadata.

  6. Under ADVANCED tab, add the following properties under Add New Custom Properties:

    Bash
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
    # Thread Configuration – Schema
ranger.policysync.connector.0.load.resources.load.schema.thread.count=2
ranger.policysync.connector.0.load.resources.load.schema.thread.min.count=2

# Thread Configuration – Table
ranger.policysync.connector.0.load.resources.load.table.thread.count=2
ranger.policysync.connector.0.load.resources.load.table.thread.min.count=2

# Thread Configuration – Column
ranger.policysync.connector.0.load.resources.load.column.thread.count=3
ranger.policysync.connector.0.load.resources.load.column.thread.min.count=3

# Thread Configuration – Function
ranger.policysync.connector.0.load.resources.load.function.thread.min.count=2
ranger.policysync.connector.0.load.resources.load.function.thread.count=2

# Thread Configuration – Volume
ranger.policysync.connector.0.load.resources.load.volume.thread.min.count=2
ranger.policysync.connector.0.load.resources.load.volume.thread.count=2

  7. Click SAVE to apply the changes.