On-Demand Sync for PolicySync Connectors¶

By default, PolicySync connectors synchronizes changes to metadata in the data sources at regular intervals. However, if the synchronization interval is not sufficient for your use case, or if you want to avoid incurring compute costs on your data sources, you can configure the connectors to synchronize changes on-demand.

The following events are supported for On-Demand Sync:

1. Resources
2. Service principals
3. Permissions

Reference Clients¶

When Privacera Ops Server with On-Demand Sync is enabled, clients can be set up to send change events to the Ops Server. Below are the reference clients that can be used to send events to the Ops Server:

1. MSK Ops Client
2. EventBridge Ops Client

Supported Connectors¶

Connector Name	Supported
Lakeformation (Pull Mode)	Yes
Lakeformation (Push Mode)	Yes
Databricks SQL Analytics	Yes
Redshift	Yes

Prerequisites¶

Before configuring On-Demand Sync for PolicySync connectors, ensure the following:

1. Privacera Basic Installation is completed.
2. Ops Server is set up. Refer to the setup guide for detailed steps.

Steps to Configure On-Demand Sync¶

1. SSH into the Privacera instance.

2. Navigate to the privacera-manager directory:

   Bash
   1	cd ~/privacera/privacera-manager

3. Copy the sample variable files:

   Bash
   1	cp config/sample-vars/vars.ops-bridge.yaml config/custom-vars/

4. Open the vars.ops-bridge.yaml file for editing:

   Bash
   1	vi config/custom-vars/vars.ops-bridge.yaml

5. Configure the Ops Bridge settings as explained below.

Configuration Properties¶

OPS_BRIDGE_ENABLE¶

• Description: Enables the Ops Bridge configuration. Set this to "true" to activate Ops Bridge.
• Default Value: "false"
• Example:

  YAML
  1	OPS_BRIDGE_ENABLE: "true"

OPS_BRIDGE_CONNECTOR_PROPERTIES¶

• Description: This property holds a list of sources and their associated connectors. Each source can have multiple connectors, and each connector can have multiple properties. These properties define how each connector behaves and integrates with the system.
• Example:

  YAML
  1 2 3 4 5	OPS_BRIDGE_CONNECTOR_PROPERTIES: - OPS_BRIDGE_SOURCE: "REST_API" CONNECTORS: - CONNECTOR_NAME: "policysync_databricks-sql-analytics_prod" APP_SUB_TYPE: "databricks_sql_analytics"

OPS_BRIDGE_SOURCE¶

• Description: Represents the name of the source (e.g., REST API, MSK, etc.). Replace <PLEASE_CHANGE> with the actual source name.

• Example:

  YAML
  1	OPS_BRIDGE_SOURCE: "REST_API"

CONNECTORS¶

• Description: A list of connectors associated with the source. Each connector is unique and follows a naming pattern for easy identification. Multiple connectors can be defined under the CONNECTORS section.

CONNECTOR_NAME:¶

• A unique name, formed using the pattern: policysync_<CONNECTOR_NAME>_<CONNECTOR_ENV>
  • <CONNECTOR_NAME>: The name of the connector. Supported values includes :
    • databricks-sql-analytics, lakeformation, redshift.
  • <CONNECTOR_ENV>: The environment (e.g., dev, prod).
  • Example
    • If you have configured the databricks_sql_analytics connector in the prod environment, the directory structure will be as follows: Where databricks_sql_analytics represents the connector name (<CONNECTOR_NAME>) and prod represents the environment (<CONNECTOR_ENV>).

      Bash
      1	~/privacera/privacera-manager/config/custom-vars/connectors/databricks-sql-analytics/prod/vars.connector.databricks.sql.analytics.yml

    • Based on the above configuration, the connector name will be:

      Bash
      1	CONNECTOR_NAME: "policysync_databricks-sql-analytics_prod"

APP_SUB_TYPE¶

• Description: The type of the application for the connector.
• Supported values include:
  • databricks_sql_analytics, lakeformation, redshift.
• Example:

  YAML
  1	APP_SUB_TYPE: "databricks_sql_analytics"

PARENT_RESOURCES - (Optional)¶

• Description: A set of key-value pairs defining the parent resources associated with the connector, such as catalog IDs, regions, etc. Replace <PLEASE_CHANGE> with actual resource details.

• Example:

  YAML
  1 2 3	PARENT_RESOURCES: - catalog: "12345678" - region: "us-east-1"

Adding Multiple Connectors¶

You can define multiple connectors under the CONNECTORS section. Here’s an example of how to configure multiple connectors for different environments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

OPS_BRIDGE_CONNECTOR_PROPERTIES:
  - OPS_BRIDGE_SOURCE: "REST_API"
    CONNECTORS:
      - CONNECTOR_NAME: "policysync_databricks-sql-analytics_prod"
        APP_SUB_TYPE: "databricks_sql_analytics"
      - CONNECTOR_NAME: "policysync_lakeformation_dev"
        APP_SUB_TYPE: "lakeformation"
        PARENT_RESOURCES:
          - catalog: "12345678"
          - region: "us-east-1"

Enable On-demand Processing in Connector¶

1. Edit the configuration file for the policysync connector:

   • Example: For Databricks SQL Analytics connector, edit the below file. Where databricks-sql-analytics is the connector name (<CONNECTOR_NAME>) and prod is the environment (<CONNECTOR_ENV>).

     Bash
     1	vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-sql-analytics/prod/vars.connector.databricks.sql.analytics.yml

2. Add the following property to enable on-demand processing:

   YAML
   1	CONNECTOR_ON_DEMAND_PROCESSING_ENABLE: "true"

Setup Privacera Manager and Run Helm Upgrade¶

1. Once the properties have been configured, execute the following commands to update your Privacera Manager platform instance:

   a. Generate the helm charts.

   Bash
   1 2	cd ~/privacera/privacera-manager ./privacera-manager.sh setup

   b. Apply the helm charts.

   Bash
   1	./pm_with_helm.sh upgrade

On-Demand Sync Task of specific connector¶

1. Login in Privacera Portal.
2. Navigate to Resource Policies.
3. Locate the configured connector repository with the icon.
4. Click to view the task list.

Comments