Skip to content

Manage Foreign GDC Catalog

Prerequisite

This guide walks you through the steps to set up HMS Federation with AWS Glue in Unity Catalog. By completing this setup, you can query metadata from an external AWS Glue Data Catalog as if it were natively registered within Unity Catalog.

DBX Doc reference: Databricks Glue HMS Federation Documentation

⚠ Limitations

  • The connector supports only the JDBC connection method for managing permissions in Unity Catalog.
  • The connector supports only native masking and row-level filters (RLF) for applying column masking and RLF policies.
  • Databricks Unity Catalog provides read-only access to the federated Glue Data Catalog (GDC).
  • The name of a foreign catalog must not exceed 236 characters.

Setup

Note

  • The connector automatically detects whether a catalog is foreign or regular when loading resources. No additional configuration is required to specify the catalog type.

Warning

  • Values are case-sensitive.
  • Use fully qualified names for schemas, tables/views, and functions (e.g., catalog1.schema1.*).
  • Replace all example values with your actual resource names.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Set the following properties to enable the connector to manage the permissions for foreign catalogs, schemas and tables in the Databricks Unity Catalog:

    YAML
    1
2
3
    CONNECTOR_DATABRICKS_UNITY_CATALOG_MANAGE_CATALOG_LIST: "test_catalog"
CONNECTOR_DATABRICKS_UNITY_CATALOG_MANAGE_SCHEMA_LIST: "test_catalog1.schema1"
CONNECTOR_DATABRICKS_UNITY_CATALOG_MANAGE_TABLE_LIST: "test_catalog1.schema1.table1"

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Note

  • GRANT statements for write operations (e.g., CREATE TABLE, CREATE SCHEMA) can be applied to foreign catalog resources; however, they will have no effect, as Databricks does not permit write operations on foreign catalogs.

Native Masking and RLF

The connector supports only native masking and row-level filtering (RLF) for enforcing column masking and RLF policies on foreign catalog tables.

  • To enforce these policies:
    1. Privacera creates functions and attaches them to the relevant columns (for masking) and tables (for row filters).
    2. These functions are stored in a new schema within a catalog that is local to the Databricks workspace.
    3. Since GDC foreign catalogs do not support the creation of new schemas, the connector automatically creates a local catalog in the Databricks workspace.
    4. A corresponding schema, matching the name of the original schema in the foreign catalog, will be created in the local catalog to store these functions.

Local Catalog Naming Convention

  • The local catalog name is automatically generated using a predefined prefix followed by the name of the foreign catalog.
  • By default, this prefix is set to privacera_security_ to ensure the local catalog is easily identifiable.
  • Example: If the foreign catalog name is test_glue_catalog, the corresponding local catalog will be named privacera_security_test_glue_catalog

Comments