Skip to content

Preventing DBX SCIM API throttling

The Databricks Unity Catalog connector uses the Databricks SCIM API to manage users, groups, and roles within the Databricks Unity Catalog. The Databricks SCIM API has rate limits, which can throttle the connector if exceeded. To prevent throttling, you can configure the connector to handle retries of DBX SCIM API requests.

Setup

To prevent throttling, update the following properties in Privacera Manager:

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Set the following properties to enable the connector to handle retrying of DBX SCIM API requests:

    YAML
    1
2
3
4
5
    # The maximum number of retry attempts to make when the DBX SCIM API limit is exceeded. By default, the value is set to 2.
CONNECTOR_DATABRICKS_UNITY_CATALOG_DBX_API_LIMIT_EXCEEDED_MAX_RETRY_ATTEMPTS: "<PLEASE_CHANGE>"

# The time interval in seconds to wait before retrying the DBX SCIM API request when the API limit is exceeded. By default the value is set to 20 seconds.
CONNECTOR_DATABRICKS_UNITY_CATALOG_DEFAULT_TIME_INTERVAL_API_LIMIT_EXCEEDED_IN_SECONDS: "<PLEASE_CHANGE>"

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Comments