Skip to content

Resource Federation synchronization between Unity Catalog and AWS Lakeformation

Unity Catalog resources can be federated in AWS Lakeformation. These resources need additional implicit read permissions on IRC Service Principal only then these resources gets accessible and synchronized on the AWS Lakeformation side.

Feature overview

To enable the feature, set CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_GRANT_IRC_DSP_PRINCIPAL_ON_NEW_TABLE as "true". Once enabled, the UC connector will grant implicit read permissions on UC native event based synced tables to IRC Service Principal.

Whether a UC table is treated as a native table is determined by CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_NATIVE_TABLE_EVENT_SOURCE. For UC tables that are synced by on‑demand event, the event source must use the same label as this property.

Once implicit read permission is granted to the IRC Service Principal, an on‑demand event is sent targeting the Lakeformation connector and the resource is then synchronized in the Lakeformation connector.

Configuration Properties

Property Description Default Value
CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_GRANT_IRC_DSP_PRINCIPAL_ON_NEW_TABLE Enable implicit IRC DSP permissions on newly added native UC tables (SELECT, USE_CATALOG, USE_SCHEMA). false
CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_PRINCIPAL_NAME IRC DSP principal identifier (app ID) for implicit grants. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_NATIVE_TABLE_EVENT_SOURCE Event source label for native‑table grants. "UC_APP"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_APP_INSTANCE_ID Target Lakeformation app instance ID for on‑demand federation Resource Sync events. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_CATALOG Target Lakeformation catalog used in on‑demand federation events. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_REGION Target AWS region for Lakeformation on‑demand federation events. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATED_UC_APP_INSTANCE_ID Optional property. UC app instance ID to include as source in Lakeformation on‑demand federation events. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_OPS_SERVER_CALL_BATCH_SIZE Batch size for ops‑server calls when sending on‑demand federation events targeting Lakeformation. 100

Setup

Warning

Replace the example values shown below with values appropriate for your environment.

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. From the list of Connected Applications, select Databricks Unity Catalog.

  3. Click on the application name or the icon to edit. Then, go to the Access Management -> ADVANCED tab.

  4. Under Add New Custom Properties, add the following properties:

    Bash
    1
2
3
4
5
6
7
8
    ranger.policysync.connector.0.enable.grant.irc.dsp.principal.on.new.table=true
ranger.policysync.connector.0.irc.dsp.principal.name=8976ab98-1276-487c-3dg4-d6734f987s98
ranger.policysync.connector.0.irc.dsp.native.table.event.source=UC_APP
ranger.policysync.connector.0.on.demand.federation.aws.lakeformation.app.instance.id=prod
ranger.policysync.connector.0.on.demand.federation.aws.lakeformation.catalog=federated_catalog1_on_aws_lf
ranger.policysync.connector.0.on.demand.federation.aws.lakeformation.region=us-east-1
ranger.policysync.connector.0.on.demand.federated.uc.app.instance.id=dev
ranger.policysync.connector.0.on.demand.federation.ops.server.call.batch.size=50

  5. Click SAVE to apply the changes.

To enable federated resource synchronization between Unity Catalog and AWS Lakeformation, update the following properties in Privacera Manager:

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Set the following properties to enable federated resource synchronization between Unity Catalog and AWS Lakeformation:

    YAML
    1
2
3
4
5
6
7
8
    CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_GRANT_IRC_DSP_PRINCIPAL_ON_NEW_TABLE: "true"
CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_PRINCIPAL_NAME: "8976ab98-1276-487c-3dg4-d6734f987s98"
CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_NATIVE_TABLE_EVENT_SOURCE: "UC_APP"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_APP_INSTANCE_ID: "prod"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_CATALOG: "federated_catalog1_on_aws_lf"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_REGION: "us-east-1"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATED_UC_APP_INSTANCE_ID: "dev"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_OPS_SERVER_CALL_BATCH_SIZE: "50"

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install