Skip to content

Preventing Thread Pool Stalls in Databricks SQL

This document provides guidance on mitigating thread pool stalls during resource and permission loading in Databricks SQL (DBX). The primary strategy involves introducing a configurable timeout to monitor thread activity and safely terminate non-responsive threads. Thread pool stalls can occur when threads in a multithreaded execution pool become inactive and fail to make progress. To address this, the system uses a configurable timeout to monitor thread behavior. If a thread remains idle beyond the defined threshold, it is flagged as stalled and safely terminated. Before shutting down a parent thread (e.g., a database-level thread), the system ensures that all dependent child threads (e.g., table loaders) have completed their execution. This prevents premature termination and ensures that all resources are properly and fully loaded.

Configuration Properties

Property Feature Description Default Value Possible Values
CONNECTOR_DATABRICKS_SQL_ANALYTICS_LOAD_RESOURCES_THREAD_POOL_WAIT_TIMEOUT_MINUTES JDBC Resource Thread Stall Timeout Maximum duration (in minutes) a resource-loading thread can remain idle without making progress before being forcefully shut down. 1200 minutes Any positive integer (in minutes)
CONNECTOR_DATABRICKS_SQL_ANALYTICS_LOAD_PERMISSION_THREAD_POOL_WAIT_TIMEOUT_MINUTES JDBC Permission Thread Stall Timeout Maximum duration (in minutes) a permission-loading thread can remain idle without making progress before being forcefully shut down. 1200 minutes Any positive integer (in minutes))

Setup

Warning

Replace the <PLEASE_CHANGE> placeholder with appropriate values for your environment.

To configure stall timeouts for multi-threaded execution (both resource and permission loading), add the following properties in Privacera Manager:

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited:

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
2
3
4
5
    # The maximum time to wait for each stalled thread during resource loading. (default: 1200 minutes)
vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-sql-analytics/instance1/vars.connector.databricks.sql.analytics.yml

# The maximum time to wait for each stalled thread during permission loading. (default: 1200 minutes)
CONNECTOR_DATABRICKS_SQL_ANALYTICS_LOAD_PERMISSION_THREAD_POOL_WAIT_TIMEOUT_MINUTES: "<PLEASE_CHANGE>"

  3. Set the following property to configure thread stall timeout:

    YAML
    1
2
    # The maximum time to wait for thread for each thread when stalled. (default: 1200 minutes)
CONNECTOR_DATABRICKS_SQL_ANALYTICS_LOAD_RESOURCES_THREAD_POOL_WAIT_TIMEOUT_MINUTES: "<PLEASE_CHANGE>"

  1. In PrivaceraCloud portal, navigate to Settings -> Applications.

  2. On the Connected Applications screen, select Databricks SQL.

  3. Click the pen icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access Management -> ADVANCED tab.

  5. Under Add New Custom Properties, add the following property:

    Bash
    1
2
    ranger.policysync.connector.0.load.resources.thread.pool.wait.timeout.minutes=<PLEASE_CHANGE>
ranger.policysync.connector.0.load.permissions.thread.pool.wait.timeout.minutes=<PLEASE_CHANGE>

  6. Click SAVE to apply the changes.

Comments