Skip to content

Resource Sync Change Log

Configure the Databricks Unity Catalog connector to write resource-delta.log: one line for each resource that a resource sync changes, with what the connector decided, why, and what differed.

Overview

During a resource sync, the connector compares each resource it loads with the copy it stored, then adds, recreates, updates or deletes it. Use resource-delta.log when a sync changes more resources than expected, for example tables that are recreated and re-granted on every scheduled sync, to see which resources changed and why.

When CONNECTOR_DATABRICKS_UNITY_CATALOG_LOG_RESOURCE_DELTA_DETAILS_ENABLED is true, the connector writes one line for each resource that a scheduled or on-demand resource sync adds, recreates, updates, deletes or skips. Unchanged resources get no line, except a table whose created time changed while nothing else did.

resource-delta.log is a diagnostic: turn it on to investigate, and off again once you have the information you need.

Configuration properties

Property Description Default
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOG_RESOURCE_DELTA_DETAILS_ENABLED Writes one line to resource-delta.log for each resource that a resource sync changes. false
CONNECTOR_RESOURCE_DELTA_MAX_LOG_FILES Number of rolled resource-delta.log files to keep. 20
CONNECTOR_RESOURCE_DELTA_MAX_LOG_FILE_SIZE Size at which resource-delta.log rolls to a new file. 25MB

Matching custom property (Self-Managed Portal):

Custom property Default
ranger.policysync.connector.0.log.resource.delta.details.enabled false

The size of resource-delta.log can be set only with the Privacera Manager properties.

Setup

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. From the list of Connected Applications, select Databricks Unity Catalog.

  3. Select the application name or the icon to edit. Then, go to the Access Management tab.

  4. Under the ADVANCED tab, under Add New Custom Properties, add:

    Properties
    1
    ranger.policysync.connector.0.log.resource.delta.details.enabled=true

  5. Select SAVE to apply the changes.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Add or set the following properties. The two size properties are optional:

    YAML
    1
2
3
    CONNECTOR_DATABRICKS_UNITY_CATALOG_LOG_RESOURCE_DELTA_DETAILS_ENABLED: "true"
CONNECTOR_RESOURCE_DELTA_MAX_LOG_FILES: "20"
CONNECTOR_RESOURCE_DELTA_MAX_LOG_FILE_SIZE: "25MB"

    Set this property in the .yml file

    Do not add the matching custom property to connector-custom.properties. The generated connector configuration already contains the key, and the first occurrence wins, so a value in connector-custom.properties is ignored.

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Read resource-delta.log

The connector writes resource-delta.log in its log folder (/workdir/policysync/logs). Each line explains one resource:

Text Only
1
decision=UPDATED | reason=CHANGED | loader=OFFLINE | changeLogId=5165639 | resource=table:uc_e2e^s^t | createdTime(new=1, saved=2, check=OLDER_THAN_SAVED) | source(new=SHOW_TABLE_EXTENDED, saved=DESCRIBE_TABLE_EXTENDED) | changed=[createdTime, owner(saved=a, new=b)]
Field Meaning
decision What the sync did with the resource (see the next table).
reason Why, for example NEW_RESOURCE, CREATED_TIME_NEWER, CREATED_TIME_NOT_A_RECREATE, CHANGED or NOT_IN_LOAD.
loader OFFLINE for a scheduled sync, ON_DEMAND for an on-demand event.
changeLogId The changelog of the resource sync that made the decision.
resource The resource type and name.
createdTime The loaded (new) and stored (saved) created time, and how they compare (check): SAME, NEWER, NOT_TRUSTED (newer, but not treated as a recreate; see Foreign Table Recreate Detection in Scheduled Syncs), OLDER_THAN_SAVED, UNKNOWN (not read) or SAVED_MISSING.
source Which read produced each created time for a foreign-catalog table: typically SHOW_TABLE_EXTENDED in a scheduled sync and DESCRIBE_TABLE_EXTENDED in an on-demand sync.
changed What differs between the stored and the loaded resource, with both values.
Decision Meaning
ADDED New resource.
RECREATED The created time is newer and trusted: the resource was replaced and all its permissions re-applied.
UPDATED The stored resource was updated. Its stored created time is kept.
CREATEDTIME_IGNORED_NO_CHANGE The created time changed but was not treated as a recreate, and nothing else changed. Nothing was written.
MIGRATED The stored resource had no created time. It was filled in, with no permission changes.
SKIPPED_STALE A newer copy of the resource was stored while the sync ran. Nothing was changed.
DELETED The resource is no longer in the source and was removed.