Skip to content

Foreign Table Recreate Detection in Scheduled Syncs

Configure whether a scheduled resource sync of the Databricks Unity Catalog connector treats a changed created time on a foreign-catalog table as a drop and recreate.

Overview

During a resource sync, the connector compares each table's created time with the one it stored. A newer created time means the table was dropped and recreated in the source, so the connector replaces the table and re-applies all its permissions.

For tables in foreign catalogs, a scheduled sync reads the created time from SHOW TABLE EXTENDED. For some foreign tables this is not a stable creation time: it can change between syncs while the table is unchanged. Those tables are then recreated and re-granted on every scheduled sync, which sends many redundant GRANT statements.

When CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_FOREIGN_TABLE_CREATED_TIME_RECREATE_ENABLED is false:

  • A scheduled sync does not treat a changed created time on a foreign-catalog table as a recreate, so it does not re-apply the table's permissions. Other changes to the table, such as a changed column, are still applied as an update.
  • On-demand syncs, tables in regular (non-foreign) catalogs and other resource types are unaffected.

What false gives up

A scheduled sync cannot tell an unstable created time from a real drop and recreate in the source, so with false it misses real recreates too:

  • An on-demand event for the table still recreates it and re-applies its grants.

Keep the default true unless scheduled syncs recreate foreign tables that were not dropped in the source. To check, turn on resource-delta.log for one scheduled sync and, for a few tables it lists with decision=RECREATED, confirm in the source that the table was not dropped and recreated (for AWS Glue, the CloudTrail events DeleteTable and CreateTable).

Configuration properties

Property Description Default
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_FOREIGN_TABLE_CREATED_TIME_RECREATE_ENABLED When true, a scheduled sync treats a changed created time on a foreign-catalog table as a drop and recreate and re-applies all its permissions. When false, it does not (see Overview). true

Setup

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. From the list of Connected Applications, select Databricks Unity Catalog.

  3. Select the application name or the icon to edit. Then, go to the Access Management tab.

  4. Under the ADVANCED tab, under Add New Custom Properties, add:

    Properties
    1
    ranger.policysync.connector.0.load.resources.foreign.table.created.time.recreate.enabled=false

  5. Select SAVE to apply the changes.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Add or set the following property:

    YAML
    1
    CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_FOREIGN_TABLE_CREATED_TIME_RECREATE_ENABLED: "false"

    Set this property in the .yml file

    Do not add the matching custom property to connector-custom.properties. The generated connector configuration already contains the key, and the first occurrence wins, so a value in connector-custom.properties is ignored.

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Verify the configuration

After a resource sync that recreated a table because of its created time, did not treat a created-time change as a recreate, skipped a stale copy or could not read an owner, policy-sync.log shows one INFO line:

Text Only
1
ServiceResourceLoader(connector=<connector>, changelogId=<id>): 0 resources recreated because their created time changed; 1250 created-time changes not treated as a recreate (0 of them moved back); 0 skipped because a newer copy was stored during the load; 0 owners not read (stored owner kept)

With the property false, created-time changes not treated as a recreate counts the foreign tables that the property kept from being recreated. A table that was really dropped and recreated in the source is counted again on every scheduled sync until an on-demand event for it recreates it.

For one line per table with the decision and its reason, turn on resource-delta.log.