Skip to content

Manage Access Audits

The Databricks Unity Catalog connector provides access auditing capabilities to help organizations track user activity, ensure compliance, and enhance security monitoring. This feature allows you to dynamically filter audit logs based on specific conditions. If no conditions are specified, the connector applies a default query based on the selected audit mode.

Audit Configuration

Audit Enable: Enables audit logs for the Databricks Unity Catalog. Set this property to true to enable audit logs.

Audit Table Path: Specifies the table location for external audits. Use the format catalog_name.schema_name.table_name.

To set up audits, follow these instructions for creating an external table for audits.

  1. Audit Unity Catalog Events: Sends activity logs from each workspace to cloud-supported file storage.
  2. Manage external locations and storage credentials: Create an external location in each cloud provider.

  3. Create an External Table: Run the following command to create the external table at the specified location.

    Replace <catalog_name>, <schema_name>, <table_name>, and <location> with your actual values.

    SQL
    1
2
3
4
5
6
    CREATE TABLE <catalog_name>.<schema_name>.<table_name> (
    accountId STRING, actionName STRING, requestId STRING, response STRING, timestamp BIGINT, workspaceId STRING, date DATE
) 
USING json 
PARTITIONED BY (workspaceId, date) 
LOCATION '<location>';

Audit Mode: Defines the type of audit logs to fetch. Refer to the Enable verbose audit logs to enable verbose logging on the workspace. The connector supports the following audit modes:

Audit Mode Description Requires Verbose Logging? Use Cases
simple Fetches audit logs with basic information. No Lightweight auditing of key access events (e.g., table reads, writes).
verbose Fetches query history from warehouses and notebooks. Yes In-depth analysis of user activity, including tracking of SQL queries and command execution.
external-simple Fetches audit logs with basic information from an external location. No When audit logs are stored outside of Databricks and need to be processed from an external storage source, ensure that the proper integration is in place for efficient data handling.
external-verbose Fetches detailed audit logs from an external location. Yes Used when detailed query history and execution logs are required, but the logs are stored externally for scalability or compliance reasons.
workspace_api(deprecated) Loads query history using the workspace query history API (for legacy support). No Only used for backward compatibility with older implementations and not recommended for new deployments.

Audit Excluded Users: Comma-separated list of users to exclude from the audits (e.g.,user1@gmail.com).

Audit Workspace IDs: Comma-separated list of workspace IDs to fetch audits from (e.g., 1023707303840399). If not provided, audits are fetched from all workspaces.

Audit Query Condition: Custom condition for filtering audit logs based on action_name (for simple/verbose) or actionname (for external modes). If provided, the connector uses a dynamic query to load audit logs. If not, the connector applies the default query based on the selected audit mode. Example:

Bash
1
`action_name IN ("commandFinish"\\,"commandSubmit"\\,"runCommand")`

Connector Behavior

  1. Audit Mode: simple/verbose

    • Executes the simple/verbose query to load audits from the system table.
    • If an audit query condition is provided, the connector executes a dynamic query.
    • Use action_name as the column name in the condition. For example:
      SQL
      1
      action_name IN ("commandFinish"\\,"commandSubmit"\\,"runCommand")

  2. Audit Mode: external-simple/external-verbose

    • Executes the external-simple/external-verbose query to load audits from an external table.
    • If an audit query condition is provided, the connector executes an external-dynamic query.
    • Use actionname as the column name in the condition. For example:
      SQL
      1
      actionname IN ("commandFinish"\\,"commandSubmit"\\,"runCommand")

Note

  • Use action_name for simple/verbose mode and actionname for external-simple/external-verbose mode.
  • Escape commas in query conditions using \\.

Setup

Warning

  • Provide a fully qualified name for the audit table path. E.g. catalog1.schema1.audits_table.
  • Replace the example values with your actual values.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Update or add the following properties

    YAML
    1
2
3
4
5
6
    CONNECTOR_DATABRICKS_UNITY_CATALOG_AUDIT_ENABLE: "true"
CONNECTOR_DATABRICKS_UNITY_CATALOG_AUDIT_TABLE_PATH: "catalog1.schema1.audits_table"
CONNECTOR_DATABRICKS_UNITY_CATALOG_AUDIT_MODE: "simple"
CONNECTOR_DATABRICKS_UNITY_CATALOG_AUDIT_EXCLUDED_USERS: "user1@gmail.com"
CONNECTOR_DATABRICKS_UNITY_CATALOG_AUDIT_WORKSPACE_IDS: "1023707303840399"
CONNECTOR_DATABRICKS_UNITY_CATALOG_AUDIT_QUERY_CONDITION: "action_name IN ("commandFinish"\\,"commandSubmit"\\,"runCommand")"

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud portal, navigate to Settings -> Applications.

  2. On the Connected Applications screen, select Databricks Unity Catalog.

  3. Click the pen icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access Management.

  5. Under BASIC tab, enable Enable access audits with the toggle button.

  6. Under ADVANCED tab, enter the following values:

    • Audit mode: simple
    • Users to exclude when fetching access audits: user1@gmail.com
    • Workspaces to get audits from: 1023707303840399

  7. Enter the following properties under Add New Custom Properties:

    Bash
    1
2
    ranger.policysync.connector.0.audit.table.path=catalog1.schema1.audits_table
ranger.policysync.connector.0.audit.query.condition=action_name IN ("commandFinish"\\,"commandSubmit"\\,"runCommand")

  8. Click SAVE to apply the changes.

Comments