Skip to content

Preventing Thread Pool Stalls in Databricks Unity Catalog

This document provides guidance on preventing thread pool stalls during resource and permission loading in Databricks Unity Catalog. The strategy is to introduce configurable timeouts that monitor thread activity and safely shut down non-responsive thread pools, and to optionally enable the ThreadPool V2 implementation.

Thread pool stalls can occur when threads in a multithreaded execution pool become inactive and fail to make progress. To address this, configurable timeouts are introduced to monitor activity. If a pool remains idle beyond the defined threshold, it is considered stalled and the associated executors are safely shut down. This helps the connector recover on the next scheduled run without manual intervention.

Configuration Properties

Property Feature Description Default Value Possible Values
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_THREAD_POOL_WAIT_TIMEOUT_MINUTES Resource Thread Stall Timeout Maximum duration (in minutes) a resource-loading thread pool can wait without making progress before it is shut down. 1200 minutes Any positive integer (in minutes)
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_PERMISSIONS_THREAD_POOL_WAIT_TIMEOUT_MINUTES Permission Thread Stall Timeout Maximum duration (in minutes) a permission-loading thread pool can wait without making progress before it is shut down. 1200 minutes Any positive integer (in minutes)
CONNECTOR_DATABRICKS_UNITY_CATALOG_USE_THREADPOOL_V2 ThreadPool V2 Enables the V2 thread pool executor for improved shutdown coordination and metrics. true true, false

Setup

Warning

Replace the <PLEASE_CHANGE> placeholder with appropriate values for your environment.

To configure the stall timeouts and optionally enable ThreadPool V2, add the following properties in Privacera Manager for the Unity Catalog connector instance:

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited:

    Note

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Set the following properties:

    YAML
    1
2
3
4
5
6
7
8
    # Maximum wait time (in minutes) before resource-loading pools are considered stalled (default: 1200 minutes)
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_RESOURCES_THREAD_POOL_WAIT_TIMEOUT_MINUTES: "<PLEASE_CHANGE>"

# Maximum wait time (in minutes) before permission-loading pools are considered stalled (default: 1200 minutes)
CONNECTOR_DATABRICKS_UNITY_CATALOG_LOAD_PERMISSIONS_THREAD_POOL_WAIT_TIMEOUT_MINUTES: "<PLEASE_CHANGE>"

# Enable ThreadPool V2 (default: true)
CONNECTOR_DATABRICKS_UNITY_CATALOG_USE_THREADPOOL_V2: "<PLEASE_CHANGE>"

  1. In PrivaceraCloud portal, navigate to Settings -> Applications.

  2. On the Connected Applications screen, select Databricks Unity Catalog.

  3. Click the pen icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access Management -> ADVANCED tab.

  5. Under Add New Custom Properties, add the following properties:

    Bash
    1
2
3
    ranger.policysync.connector.0.use.thread.pool.executor.v2=
ranger.policysync.connector.0.load.resources.thread.pool.wait.timeout.minutes=<PLEASE_CHANGE>
ranger.policysync.connector.0.load.permissions.thread.pool.wait.timeout.minutes=<PLEASE_CHANGE>

Comments