Skip to content

Skip Child Resource Loading for On-Demand Sync

Configure the Databricks Unity Catalog connector so on-demand sync loads only the resources that were requested, instead of descending into the full subtree.

Overview

By default, an on-demand request for a parent resource (for example, a catalog or schema) causes the connector to load that parent and all of its children. When skip-child loading is enabled, the connector:

  • Loads only the resources named in the on-demand request
  • Skips JDBC loads for child resources that were not requested
  • Retains already-stored children so reconciliation does not delete them
  • Honors the * wildcard for tables to load all tables under a schema (for example, catalog.schema.*)

Use this when on-demand requests, including Event Hub events and API-triggered requests, often target parent-only resources and a full subtree scan is unnecessary overhead.

Configuration property

  • Property name: CONNECTOR_DATABRICKS_UNITY_CATALOG_ONDEMAND_SKIP_CHILD_LOAD_ENABLED

    Value Behavior
    false (default) On-demand requests descend and load the full subtree under the requested parent
    true On-demand requests load only the resources named in the request: skip unrequested children, retain already-stored children, and honor * for tables to load all tables under a schema (for example, catalog.schema.*)

Warning

When CONNECTOR_DATABRICKS_UNITY_CATALOG_ONDEMAND_SKIP_CHILD_LOAD_ENABLED is set to true and an on-demand request targets only a parent (for example, a schema with no table key), children that already exist in the connector store are retained, not reloaded. Send an explicit table request or use * (for example, catalog.schema.*) when you need those tables refreshed from Unity Catalog.

Setup

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. From the list of Connected Applications, select Databricks Unity Catalog.

  3. Select the application name or the icon to edit. Then, go to the Access Management tab.

  4. Under the ADVANCED tab, under Add New Custom Properties, add:

    Properties
    1
    ranger.policysync.connector.0.ondemand.skip.child.load.enabled=true

  5. Select SAVE to apply the changes.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Add or set the following property:

    YAML
    1
    CONNECTOR_DATABRICKS_UNITY_CATALOG_ONDEMAND_SKIP_CHILD_LOAD_ENABLED: "true"

  4. Once the property is configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install