Skip to content

Resource Federation synchronization between Unity Catalog and AWS Lakeformation

Unity Catalog resources can be federated in AWS Lakeformation. These resources need additional implicit read permissions on IRC Service Principal only then these resources gets accessible and synchronized on the AWS Lakeformation side.

Feature overview

To enable the feature, set CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_GRANT_IRC_DSP_PRINCIPAL_ON_NEW_TABLE as "true". Once enabled, the UC connector will grant implicit read permissions on UC native event based synced tables to IRC Service Principal.

Whether a UC table is treated as a native table is determined by CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_NATIVE_TABLE_EVENT_SOURCE. For UC tables that are synced by on‑demand event, the event source must use the same label as this property.

Once implicit read permission is granted to the IRC Service Principal, an on‑demand event is sent targeting the Lakeformation connector and the resource is then synchronized in the Lakeformation connector.

Additionally, to synchronize federated tables during a scheduled sync or full sync, implicit permissions must be granted on those tables.

To enable this behavior, set the CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_LOAD_FOREIGN_CATALOG_TABLE_TYPES configuration property to True. When this property is enabled, the connector loads table types from the information schema during scheduled sync. This allows implicit permissions to be granted on federated tables that are discovered during the scheduled sync.

Configuration Properties

Add IRC DSP principle in ignored user list property to avoid permission collision with implicit permissions for this IRC service principle.

Property Description Default Value
CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_GRANT_IRC_DSP_PRINCIPAL_ON_NEW_TABLE Enable implicit IRC DSP permissions on newly added native UC tables (SELECT, USE_CATALOG, USE_SCHEMA). false
CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_PRINCIPAL_NAME IRC DSP principal identifier (app ID) for implicit grants. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_NATIVE_TABLE_EVENT_SOURCE Event source label for native‑table grants. "UC_APP"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_APP_INSTANCE_ID Target Lakeformation app instance ID for on‑demand federation Resource Sync events. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_CATALOG Target Lakeformation catalog used in on‑demand federation events. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_REGION Target AWS region for Lakeformation on‑demand federation events. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATED_UC_APP_INSTANCE_ID Optional property. UC app instance ID to include as source in Lakeformation on‑demand federation events. ""
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_OPS_SERVER_CALL_BATCH_SIZE Batch size for ops‑server calls when sending on‑demand federation events targeting Lakeformation. 100
CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_LOAD_FOREIGN_CATALOG_TABLE_TYPES Enables federation synchronization for tables loaded via scheduled sync false

Setup

Warning

Replace the example values shown below with values appropriate for your environment.

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. From the list of Connected Applications, select Databricks Unity Catalog.

  3. Click on the application name or the icon to edit. Then, go to the Access Management -> ADVANCED tab.

  4. Under Add New Custom Properties, add the following properties:

    Bash
    1
2
3
4
5
6
7
8
9
    ranger.policysync.connector.0.enable.grant.irc.dsp.principal.on.new.table=true
ranger.policysync.connector.0.irc.dsp.principal.name=8976ab98-1276-487c-3dg4-d6734f987s98
ranger.policysync.connector.0.irc.dsp.native.table.event.source=UC_APP
ranger.policysync.connector.0.on.demand.federation.aws.lakeformation.app.instance.id=prod
ranger.policysync.connector.0.on.demand.federation.aws.lakeformation.catalog=federated_catalog1_on_aws_lf
ranger.policysync.connector.0.on.demand.federation.aws.lakeformation.region=us-east-1
ranger.policysync.connector.0.on.demand.federated.uc.app.instance.id=dev
ranger.policysync.connector.0.on.demand.federation.ops.server.call.batch.size=50
ranger.policysync.connector.0.enable.load.foreign.catalog.table.types=true

  5. Click SAVE to apply the changes.

To enable federated resource synchronization between Unity Catalog and AWS Lakeformation, update the following properties in Privacera Manager:

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Set the following properties to enable federated resource synchronization between Unity Catalog and AWS Lakeformation:

    YAML
    1
2
3
4
5
6
7
8
9
    CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_GRANT_IRC_DSP_PRINCIPAL_ON_NEW_TABLE: "true"
CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_PRINCIPAL_NAME: "8976ab98-1276-487c-3dg4-d6734f987s98"
CONNECTOR_DATABRICKS_UNITY_CATALOG_IRC_DSP_NATIVE_TABLE_EVENT_SOURCE: "UC_APP"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_APP_INSTANCE_ID: "prod"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_CATALOG: "federated_catalog1_on_aws_lf"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_AWS_LAKEFORMATION_REGION: "us-east-1"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATED_UC_APP_INSTANCE_ID: "dev"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ON_DEMAND_FEDERATION_OPS_SERVER_CALL_BATCH_SIZE: "50"
CONNECTOR_DATABRICKS_UNITY_CATALOG_ENABLE_LOAD_FOREIGN_CATALOG_TABLE_TYPES: "true"

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install