Skip to content

Setup Access Management for Databricks SQL

Runtime tab

The Runtime tab describes the Omni Connector on a Runtime Plane, which is a separate connector from PolicySync. Sections without a Runtime tab apply to PolicySync only.

This section outlines the steps to configure the Databricks SQL connector in Privacera. Ensure that all prerequisites are met before proceeding.

Best Practice for Enabling the Connector

This connector manages access control policies for Databricks SQL. If there are existing policies in Databricks SQL, they will be overwritten by policies from Privacera. It is strongly recommended to start by managing access for a limited set of resources before enabling the connector for all resources. For example, you can create a test database in Databricks SQL and use Privacera to manage its access control policies. Once you're confident with the setup, you can extend the connector to manage all databases.

Create Instance of Databricks SQL

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to navigate to the /config directory.

    Bash
    1
    cd ~/privacera/privacera-manager/config

  3. Create a new directory for the Databricks SQL connector configuration.

    Note

    In the example below, instance1 is the name of the connector instance. You can change this name to uniquely identify your installed connector configuration. The connector instance name should consist of only hyphens and alphanumeric characters.

    Bash
    1
    mkdir -p custom-vars/connectors/databricks-sql-analytics/instance1

  4. Copy the sample connector configuration file to your custom directory:

    Bash
    1
    cp -n sample-vars/vars.connector.databricks.sql.analytics.yml custom-vars/connectors/databricks-sql-analytics/instance1/

  5. Run the following command to open the .yml file to be edited:

    Bash
    1
    vi custom-vars/connectors/databricks-sql-analytics/instance1/vars.connector.databricks.sql.analytics.yml

  1. In PrivaceraCloud, navigate to Settings -> Applications.

  2. On the Applications screen, select Databricks SQL application under Available connections.

  3. Enter the application Name and Description, then click Save. Name could be any name of your choice. E.g. Databricks SQL Connector for account 123456.

  4. Open the Databricks SQL application.

  5. Enable the Access Management option with toggle button.

The Omni Connector is configured from the Trust3 AI portal on a Runtime Plane (Self-Managed or Partially Managed (D2P)). For an overview and the data platform prerequisites, see Omni Connector.

  1. In the portal, navigate to Settings → Runtime Plane and open your Runtime Plane.

  2. Open the Connectors tab and select Add New.

  3. Select Databricks SQL (Hive Metastore) from the available connectors.

  4. Enter the Name and an optional Description.

    • The name can contain lowercase letters, numbers and hyphens, with a single space allowed only between words. It must start and end with a letter or number, and be unique within the Runtime Plane. For example, dbx-sql-1.

  5. Select the Deployment Mode, then select Continue.

    • Shared: the connector runs inside the Runtime Plane's shared omni-connector application.
    • Dedicated: the connector runs in its own application.
    • The deployment mode cannot be changed after the connector is created. See Deployment mode.

Connection Details

  1. Specify the Databricks workspace URL. For example, https://dev-environment.cloud.databricks.com.
  2. Provide the Databricks personal access token used to authenticate with the Databricks API.
  3. Enter the JDBC URL, default database, and username for the Databricks connection.

Replace <workspace-url>, <jdbc-url>, <db_name>, and <databricks-access-token> with your actual values.

Default Repository Behavior

By default, the Databricks SQL connector uses the SQL repository. If you prefer to use the Hive repository instead, refer to the corresponding tab below for configuration steps.

YAML
1
2
3
4
5
CONNECTOR_DATABRICKS_SQL_ANALYTICS_HOST_URL: "https://<workspace-url>.cloud.databricks.com"
CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_URL: "<jdbc-url>"
CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_DB: "<db_name>"
CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_USERNAME: "token"
CONNECTOR_DATABRICKS_SQL_ANALYTICS_JDBC_PASSWORD: "<databricks-access-token>"
Hive Repository Support

To use the Hive repository, add the following property:

YAML
CONNECTOR_DATABRICKS_SQL_ANALYTICS_USE_HIVE_ACCESS_POLICIES: "true"

  • Enter these fields under the BASIC tab:

    • Databricks SQL workspace URL: https://<workspace-url>.cloud.databricks.com
    • Databricks SQL JDBC url: <jdbc-url>
    • Databricks SQL default database: <db_name>
    • Databricks SQL JDBC username: token
    • Databricks SQL JDBC password: <databricks-access-token>

  • Update the Advanced tab with the following property:

    • System config: privacera-databricks_sql_analytics-system-config.json
Hive Repository Support

To enable the Hive repository, configure the following system property:

  • System config: privacera-databricks_sql_analytics-hive-system-config.json

Under ACCESS CONTROL CONFIGS, in the Databricks SQL (Hive Metastore) section, enter:

  • SQL warehouse JDBC URL: jdbc:databricks://<host>:443/default;httpPath=/sql/1.0/warehouses/<id>. Do not include UID, PWD or AuthMech.
  • Auth type: OAuth M2M (service principal) (default) or Personal access token (PAT).
  • OAuth client ID (M2M) and OAuth client secret (M2M): shown for OAuth M2M.
  • Access token (PAT): shown for PAT.
  • Session catalog: default hive_metastore.

Session catalog decides where a grant lands

Governed names are two-part (database.table), so the catalog the session resolves to decides which objects a grant applies to. On a Unity Catalog workspace, the warehouse's own default may not be hive_metastore. Enter - to skip the pin for a warehouse that rejects USE CATALOG.

Note

The Omni Connector for Databricks SQL governs the legacy hive metastore only. To govern Unity Catalog, create a Databricks Unity Catalog connector.

See Prerequisites on the data platform.

Managed Databases

This property is used to specify a comma-separated list of database names to which access control policies will be applied. Wildcards are supported.

Example: test_db1,test_db2,sales_*.

Warning

  • Values are case-sensitive.
  • Replace the below example value with your actual value.

Recommended Best Practice

It is advisable to test the connector with a test database before enabling it for all databases.

YAML
1
CONNECTOR_DATABRICKS_SQL_ANALYTICS_MANAGE_DATABASE_LIST: "test_database"

Under the BASIC tab, enter the value for:

  • Databases to set access control policies: test_database

Under ACCESS CONTROL CONFIGS, in the Managed Resources section, enter one database name or wildcard pattern per line under Include.

Example: test_db1

A new connector governs nothing until you add an Include entry

Include is empty by default, which means deny-all. Enter * alone to govern everything.

For tables, exclusions and resource types, see Manage Resources List.

Enforcing Grants/Revoke

This property enables the enforcement of GRANT and REVOKE operations for managing users, groups, and roles.

YAML
1
CONNECTOR_DATABRICKS_SQL_ANALYTICS_GRANT_UPDATES: "true"

Under the BASIC tab, enable the toggle:

  • Enable policy enforcements and user/group/role management

There is no separate setting. The connector applies grants and revokes on the resources it manages, using the authority of its credential. Make sure the credential can grant and revoke on those resources.

Apply the Configuration

After all the changes are done you can start the connector by running the following instructions:

Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

Bash
1
2
cd ~/privacera/privacera-manager
./privacera-manager.sh setup
Step 2 - Apply the Privacera Manager helm charts.
Bash
1
2
cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

Bash
1
2
cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. Once all the required fields are filled, click Save.

  2. The configured Databricks SQL connector appears under Connected Applications.

  3. Once saved and enabled, the Databricks SQL connector will start. Then you can hover on the VIEW LOGS button to check the status, either Running or Stopped.

  4. Perform the following steps to restart the Databricks SQL connector application:

    1. Go to Settings → Applications → select the Databricks SQL connector application.

    2. Edit the application by disabling the Access Management option with toggle button and then Save it.

    3. Open the same application again and then enable the Access Management option with toggle button and then Save it.

  1. Optionally, set the Scan Schedule under ACCESS CONTROL CONFIGS. Both schedules are off by default, so a new connector scans only when triggered manually. For example, set Metadata scan schedule to 0 0 */4 * * * and Permission sweep schedule to 0 0 */8 * * *. See Scan schedule.

  2. Select Save.

  3. The connector starts after it is saved.

    • Shared: saving restarts the shared omni-connector pod, so every Shared connector on the Runtime Plane is briefly interrupted. To check the status, select View status on the connector row, which opens the shared application under System Applications.
    • Dedicated: the connector's own application starts. Enable, restart and resource controls are on the connector itself.