Configure API-Driven On-Demand Sync for Databricks Unity Catalog Connector¶

Overview¶

API-driven On-Demand Sync enables real-time, targeted policy synchronization in Databricks Unity Catalog by allowing you to invoke resource synchronization through REST API calls. Unlike scheduled sync operations that process all resources periodically, API-driven On-Demand Sync provides programmatic control to trigger immediate synchronization for specific resources when changes occur.

This path uses the same On-Demand Sync V2 task pipeline as event-driven (Azure Event Hub) On-Demand Sync; you can enable the API server alone, Event Hub alone, or both, depending on how you want to trigger sync tasks.

Key Benefits¶

How It Works¶

API-driven On-Demand Sync follows this workflow:

1. API Invocation: Your application sends a REST request to create a sync task.
2. Request Processing: The connector API server validates the request (including JWT authentication).
3. Task Creation: A sync task is created and processed by the On-Demand Sync V2 pipeline.
4. Resource Sync: The connector loads the resources specified in the request from Unity Catalog and applies policy updates.
5. Response: The API returns task identifiers and status; you can poll task status with the GET endpoint.

Configuration Properties¶

Privacera Manager must enable both the API server and On-Demand Sync V2 so that API server settings are written to the connector configuration.

Required Properties¶

Optional Properties¶

Setup¶

Step 1: Edit Connector Configuration¶

SSH to the instance where Privacera is installed and edit your connector configuration file:

1
2

cd ~/privacera/privacera-manager/config
vi custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

Step 2: Add API-Driven On-Demand Sync Configuration¶

Add the following configuration to your connector YAML file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Enable API Server for On-Demand Sync
CONNECTOR_DATABRICKS_UNITY_CATALOG_API_SERVER_ENABLED: "true"

# Enable On-Demand Sync V2 (required with API server)
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_V2_ENABLED: "true"

# Enable External Access
CONNECTOR_DATABRICKS_UNITY_CATALOG_EXTERNAL_ACCESS_ENABLED: "true"

# Optional: Kubernetes NGINX Ingress (default: false)
CONNECTOR_DATABRICKS_UNITY_CATALOG_K8S_NGINX_INGRESS_ENABLE: "false"

Step 3: Deploy Configuration¶

Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

1
2

cd ~/privacera/privacera-manager
./privacera-manager.sh setup

1
2

cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade

1
2

cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

API Usage¶

Retrieve Host¶

The host is generated during post-install. After running ./privacera-manager.sh post-install, a service-urls.txt file is generated in the output directory. To retrieve the host:

1. SSH to the instance where Privacera is installed and navigate to the Privacera Manager directory:

   Bash
   1	cd ~/privacera/privacera-manager

2. Read the host URL from the service-urls.txt file:

   Bash
   1	cat output/service-urls.txt

3. For the required connector instance, copy the external URL from the output to use in the API endpoint URL.

Endpoint¶

The REST API endpoint for triggering on-demand sync:

1

POST https://<host>/policysync/api/v1/task

Replace <host> with your connector API server hostname (from the Retrieve Host section).

Authentication¶

The API requires JWT authentication. After deployment, Privacera Manager generates a token under the connector API server keys path. Replace <instance-name> with your connector instance name (for example, instance1):

1

cat config/api-server-keys/connector/databricks-unity-catalog/<instance-name>/jwt/connector-databricks-unity-catalog-<instance-name>-jwt-token.txt

Use the token value in the Authorization header when calling the API.

Request Headers¶

Request Payload Structure¶

Send a JSON body with task type RESOURCE_SYNC and a requestInfo.resources array. Resource identifiers use catalog, schema, and table (Unity Catalog naming), not database/schema as in Snowflake.

Sample Request Payload¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
    "type": "RESOURCE_SYNC",
    "requestInfo": {
        "resources": [
            {
                "type": "table",
                "values": {
                    "catalog": "main",
                    "schema": "sales",
                    "table": "customers"
                }
            }
        ]
    }
}

Payload Field Descriptions¶

Resource Types and Values¶

Catalog:

1
2
3
4
5
6

{
  "type": "catalog",
  "values": {
    "catalog": "main"
  }
}

Schema:

1
2
3
4
5
6
7

{
  "type": "schema",
  "values": {
    "catalog": "main",
    "schema": "sales"
  }
}

Table:

1
2
3
4
5
6
7
8

{
  "type": "table",
  "values": {
    "catalog": "main",
    "schema": "sales",
    "table": "customers"
  }
}

View:

1
2
3
4
5
6
7
8

{
  "type": "view",
  "values": {
    "catalog": "main",
    "schema": "sales",
    "table": "revenue_by_region"
  }
}

Multiple Resources Example¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

{
    "type": "RESOURCE_SYNC",
    "requestInfo": {
        "resources": [
            {
                "type": "catalog",
                "values": {
                    "catalog": "hr_catalog"
                }
            },
            {
                "type": "schema",
                "values": {
                    "catalog": "hr_catalog",
                    "schema": "employee_data"
                }
            },
            {
                "type": "table",
                "values": {
                    "catalog": "hr_catalog",
                    "schema": "employee_data",
                    "table": "employees"
                }
            }
        ]
    }
}

Optional Event-Style Fields¶

For parity with Event Hub payloads, you may include optional fields such as id, appType, appSubType, source, and createTime. See Configure Event-Driven On-Demand Sync for examples.

Retrieve Task Status¶

To check the status of a sync task, use the GET endpoint with the task ID returned from the POST call:

1

GET https://<host>/policysync/api/v1/tasks/<taskId>

Example response:

1
2
3
4
5

{
    "taskId": "574057046565109672",
    "status": "SUCCESS",
    "message": "Sync completed successfully"
}

Task Status¶

After processing, each sync task will have one of the following statuses: