Configure Event-Driven On-Demand Sync for Snowflake Connector¶

Overview¶

Event-driven On-Demand Sync enables real-time, targeted policy synchronization in Snowflake based on specific resource changes. Unlike scheduled sync operations that process all resources periodically, On-Demand Sync allows you to trigger immediate synchronization for specific resources when changes occur.

Key Benefits¶

How It Works¶

Event-driven On-Demand Sync follows this workflow:

1. Event Trigger: External systems publish sync request events to Azure Event Hub
2. Connector Receives: The Snowflake Connector listens to the Event Hub for incoming sync requests
3. Batch Processing: Multiple events are batched together for efficient processing
4. Resource Sync: The connector loads resources(specified in the incoming sync requests) from Snowflake and applies policy changes.
5. Completion: Audit records track the sync operation with SUCCESS, FAILED, or SKIPPED status

Prerequisites¶

Before enabling Event-Driven On-Demand Sync, ensure you have:

1. Azure Event Hub namespace created in your Azure subscription
2. Connection string with appropriate permissions (Send/Listen)

Configuration Properties¶

Required Properties¶

Optional Properties¶

Azure Failure Hub Configuration (for failure tracking)¶

Setup¶

Step 1: Edit Connector Configuration¶

SSH to the instance where Privacera is installed and edit your connector configuration file:

1
2

cd ~/privacera/privacera-manager/config
vi custom-vars/connectors/snowflake/instance1/vars.connector.snowflake.yml

Step 2: Add On-Demand Sync Configuration¶

Add the following configuration to your connector YAML file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Enable Event-Driven On-Demand Sync
CONNECTOR_SNOWFLAKE_ON_DEMAND_V2_ENABLED: "true"
CONNECTOR_SNOWFLAKE_LOAD_RESOURCES_KEY: "load_multi_thread"

# Azure Event Hub Configuration
CONNECTOR_ON_DEMAND_V2_AZURE_EVENT_HUB_ENABLED: "true"
CONNECTOR_ON_DEMAND_V2_AZURE_CONNECTION_STRING: "Endpoint=sb://<update_namespace>.servicebus.windows.net/;SharedAccessKeyName=<update_shared_access_key_name>;SharedAccessKey=<update_shared_key>"
CONNECTOR_ON_DEMAND_V2_AZURE_EVENT_HUB_NAME: "<update_event_hub_name>"
CONNECTOR_ON_DEMAND_V2_AZURE_CONSUMER_GROUP: "$Default"

# Optional: Batch Processing Configuration
CONNECTOR_ON_DEMAND_V2_BATCH_SIZE: "50"
CONNECTOR_ON_DEMAND_V2_BATCH_TIMEOUT_MS: "5000"
CONNECTOR_ON_DEMAND_V2_QUEUE_THRESHOLD: "100"

Azure Failure Hub Configuration (for failure tracking)¶

Add the following to your connector YAML when you want to track failures in a dedicated Event Hub:

1
2
3
4

# Azure Failure Hub (optional - for failure tracking)
CONNECTOR_ON_DEMAND_V2_AZURE_FAILURE_HUB_ENABLED: "true"
CONNECTOR_ON_DEMAND_V2_AZURE_FAILURE_HUB_CONNECTION_STRING: "Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=<key_name>;SharedAccessKey=<key>"
CONNECTOR_ON_DEMAND_V2_AZURE_FAILURE_HUB_NAME: "failure-tracker"

Step 3: Deploy Configuration¶

Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

1
2

cd ~/privacera/privacera-manager
./privacera-manager.sh setup

1
2

cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade

1
2

cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Event Payload Structure¶

To trigger an on-demand sync, publish a JSON event to the Azure Event Hub with the following structure.

Sending events to Event Hub¶

Follow Microsoft's documentation to send events to your Event Hub. For example:

• REST API — Send event (REST API) for sending events via HTTP.
• Python — Send and receive events using Python for a quickstart with the Azure Event Hubs Python SDK.

Similar quickstarts and SDK guides for other languages and platforms are available in the Azure Event Hubs documentation.

Sample Event Payload¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

{
  "id": "005",
  "type": "RESOURCE_SYNC",
  "appType": "PS_CONNECTOR",
  "appSubType": "SNOWFLAKE",
  "requestInfo": {
    "resources": [
      {
        "type": "table",
        "values": {
          "database": "AP_PS_OMNI_PROD_DB",
          "schema": "TEST_SCHEMA1",
          "table": "TEST_DATA3"
        }
      }
    ]
  },
  "source": "KAFKA",
  "createTime": "2026-01-14T10:00:00Z"
}

Payload Field Descriptions¶

Resource Types and Values¶

Database:

1
2
3
4
5
6

{
  "type": "database",
  "values": {
    "database": "SALES_DB"
  }
}

Schema:

1
2
3
4
5
6
7

{
  "type": "schema",
  "values": {
    "database": "SALES_DB",
    "schema": "PUBLIC"
  }
}

Table:

1
2
3
4
5
6
7
8

{
  "type": "table",
  "values": {
    "database": "SALES_DB",
    "schema": "PUBLIC",
    "table": "CUSTOMERS"
  }
}

Multiple Resources Example¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

{
  "id": "006",
  "type": "RESOURCE_SYNC",
  "appType": "PS_CONNECTOR",
  "appSubType": "SNOWFLAKE",
  "requestInfo": {
    "resources": [
      {
        "type": "database",
        "values": {
          "database": "HR_DB"
        }
      },
      {
        "type": "schema",
        "values": {
          "database": "HR_DB",
          "schema": "EMPLOYEE_DATA"
        }
      },
      {
        "type": "table",
        "values": {
          "database": "HR_DB",
          "schema": "EMPLOYEE_DATA",
          "table": "EMPLOYEES"
        }
      }
    ]
  },
  "source": "ACCESS_REQUEST",
  "createTime": "2026-01-21T10:30:00Z"
}

Task Status¶

After processing, each sync task will have one of the following statuses: