Skip to content

Support Multiple Unity Catalog Connectors for MetaStores

A MetaStore in Unity Catalog serves as a centralized metadata management system that contains technical metadata about your data assets, including catalogs, schemas, and tables. It acts as a single source of truth for your data structure and permissions across multiple Databricks workspaces. When a MetaStore is associated with multiple workspaces, the permissions defined within it are consistently applied across all associated workspaces, ensuring uniform access control.

Key characteristics of MetaStores in Databricks Unity Catalog include:

  • Contains technical metadata about catalogs, schemas, and tables
  • Manages permissions centrally for all associated workspaces
  • Ensures consistent access control across all connected workspaces
  • Can be shared across multiple Databricks workspaces
  • Supports multiple independent MetaStores within a Databricks account

Recommendations for Multiple MetaStores

When working with multiple MetaStores in Unity Catalog, it is recommended to configure a dedicated Privacera Unity Catalog connector for each MetaStore. This approach provides several benefits:

  1. Clear Separation of Concerns: Each connector is responsible for managing permissions within a single MetaStore, preventing any cross-MetaStore permission conflicts.

  2. Independent Management: Changes to permissions in one MetaStore won't affect other MetaStores, allowing for more granular control.

  3. Simplified Troubleshooting: When issues arise, having dedicated connectors makes it easier to identify and resolve problems specific to each MetaStore.

  4. Scalability: As your organization grows and adds new MetaStores, you can easily add new connectors without impacting existing configurations.

Best Practices for Multiple MetaStore Configuration

  1. Unique Connector Names: Use descriptive names for each connector instance that clearly identify the MetaStore it manages.

  2. Separate Configuration Files: Maintain separate configuration files for each connector to avoid confusion and potential misconfigurations.

⚠ Limitations

  1. It is not recommended to use Privacera's Connector to create and manager User/Groups in Databricks Account. Read here for more information.
  2. System Catalog and any shared resources within the same MetaStore should be managed by only one connector. All other connectors should be explicitly configured to ignore it.

Managing Resources with Multiple Unity Catalog Connectors

When multiple Unity Catalog connectors are configured to manage the same MetaStore, it's crucial to avoid having multiple connectors manage the same resources. This is because when multiple connectors manage the same resource (such as catalogs, schemas, or tables), they can overwrite each other's policies, leading to inconsistent access control and unexpected behavior.

Best Practices for Resource Management

To ensure consistent and reliable access control when using multiple connectors:

  1. Avoid Resource Overlap: Configure each connector to manage a distinct set of resources using the resource management configuration. This prevents policy conflicts and overwrites.

  2. System Catalog Management: The system catalogs should be managed by only one connector. All other connectors should be explicitly configured to ignore it.

  3. Audit Configuration: Configure only one connector to retrieve audit logs to prevent duplicate audit entries and ensure consistent audit trails.

Example: Managing System Catalog Permissions

When multiple Databricks workspaces share a System Catalog, it's critical to have only one Unity Catalog (UC) connector managing its permissions. This prevents policy conflicts that could result in:

  • Inconsistent permissions across workspaces
  • Intermittent access revokes
  • Unexpected permission changes

The steps below show how to properly configure UC connectors for System Catalog management.

These configuration steps can be adapted for other resources like user catalogs, schemas, or tables. Adjust the property names and values based on the specific resource type.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    Update <instance_name> with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/<instance_name>/vars.connector.databricks.unity.catalog.yml

  3. On the UC connector designated to manage the system catalog, set the following property:

    YAML
    1
    CONNECTOR_DATABRICKS_UNITY_CATALOG_MANAGE_CATALOG_LIST: "system, catalog_prod1, catalog_prod2"

  4. On all other UC connectors, configure the include and ignore properties:

    YAML
    1
2
    CONNECTOR_DATABRICKS_UNITY_CATALOG_IGNORE_CATALOG_LIST: "system"
CONNECTOR_DATABRICKS_UNITY_CATALOG_MANAGE_CATALOG_LIST: "catalog_dev1, catalog_dev2"

  5. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud portal, navigate to Settings -> Applications.

  2. On the Connected Applications screen, select Databricks Unity Catalog.

  3. For the UC connector managing the system resources (e.g., system catalog):

    1. Click the pen icon or the Account Name to modify the settings.

    2. On the Edit Application screen, go to Access Management -> BASIC tab.

    3. Update Catalogs to set access control policies: system, catalog_prod1, catalog_prod2.

    4. Click SAVE to apply the changes.

  4. For all other UC connectors, configure them to ignore the system resources and manage others:

    1. Click the pen icon or the Account Name to modify the settings.

    2. On the Edit Application screen, go to Access Management -> ADVANCED tab.

    3. Update Catalogs to ignore while setting access control policies: system.

    4. Update Catalogs to set access control policies: catalog_dev1, catalog_dev2.

    5. Click SAVE to apply the changes.

  5. Restart the Databricks Unity Catalog connector application:

    1. Disable the Access Management toggle and save.
    2. Re-enable the Access Management toggle, then save it.

Creating and Managing Principals in Unity Catalog

Privacera's connector for Databricks Unity Catalog allows you to create and manage principals (users and groups) in the Unity Catalog. However, if you are using multiple connectors for the same Databricks Account, then you need to ensure that only one connector is responsible for creating principals in the Unity Catalog. This prevents conflicts and ensures that the principal creation process is consistent and reliable. Since this can be a complex process, we recommend not to enable this feature unless you are familiar with the implications of having multiple connectors managing the same Databricks Account.

Read Creating and Managing Principals in Databricks Unity Catalog for more information on how to create and manage principals in Unity Catalog.

This is applicable while using multiple connectors for the same Databricks Account, regardless if you are using the same MetaStore in UnityCatalog or different MetaStores

Comments