Skip to content

Preventing DBX SCIM API throttling

The Databricks Unity Catalog connector uses the Databricks SCIM API to manage users, groups, and roles within the Databricks Unity Catalog. The Databricks SCIM API has rate limits, which can throttle the connector if exceeded. To prevent throttling, you can configure the connector to handle retries of DBX SCIM API requests.

Configuration Properties

There are two key properties that control the retry behavior:

  1. Maximum Retry Attempts: The number of times the connector will retry a failed API request due to rate limiting.

    • Default value: 2

  2. Retry Interval: The time to wait between retry attempts in seconds.

    • Default value: 20 seconds

Setup

Warning

  • Replace the <PLEASE_CHANGE> placeholder with appropriate values for your environment.

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. From the list of Connected Applications, select Databricks Unity Catalog.

  3. Click on the application name or the icon to edit. Then, go to the Access Management -> ADVANCED tab.

  4. Under Add New Custom Properties, add following properties:

    Bash
    1
2
    ranger.policysync.connector.0.dbx.api.limit.exceeded.max.retry.attempts=<PLEASE_CHANGE>
ranger.policysync.connector.0.default.time.interval.api.limit.exceeded.in.seconds=<PLEASE_CHANGE>

  5. Click SAVE to apply the changes.

To prevent throttling, update the following properties in Privacera Manager:

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Set the following properties to enable the connector to handle retrying of DBX SCIM API requests:

    YAML
    1
2
3
4
5
    # The maximum number of retry attempts to make when the DBX SCIM API limit is exceeded. By default, the value is set to 2.
CONNECTOR_DATABRICKS_UNITY_CATALOG_DBX_API_LIMIT_EXCEEDED_MAX_RETRY_ATTEMPTS: "<PLEASE_CHANGE>"

# The time interval in seconds to wait before retrying the DBX SCIM API request when the API limit is exceeded. By default the value is set to 20 seconds.
CONNECTOR_DATABRICKS_UNITY_CATALOG_DEFAULT_TIME_INTERVAL_API_LIMIT_EXCEEDED_IN_SECONDS: "<PLEASE_CHANGE>"

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud portal, navigate to Settings -> Applications.

  2. On the Connected Applications screen, select Databricks Unity Catalog.

  3. Click the pen icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access Management -> ADVANCED tab.

  5. Under Add New Custom Properties, add following properties:

    Bash
    1
2
    ranger.policysync.connector.0.dbx.api.limit.exceeded.max.retry.attempts=<PLEASE_CHANGE>
ranger.policysync.connector.0.default.time.interval.api.limit.exceeded.in.seconds=<PLEASE_CHANGE>

  6. Click SAVE to apply the changes.

Comments