Skip to content

Preventing DBX SCIM API throttling

The Databricks Unity Catalog connector uses the Databricks SCIM API to manage users, groups, and roles within the Databricks Unity Catalog. The Databricks SCIM API has rate limits, which can throttle the connector if exceeded. To prevent throttling, you can configure the connector to handle retries of DBX SCIM API requests.

Configuration Properties

The following properties configure SCIM API retry behavior and related throttling settings:

Property Description Default Value
Maximum Retry Attempts Number of retry attempts for a failed API request due to rate limiting. 31
Retry Interval Wait time between retry attempts (in seconds). 20
Group Update Delay Delay interval between consecutive patch API calls to update group users (in milliseconds). 6000
Group ID Cache Timeout Cache timeout for group IDs in minutes. 15
Min Retry Delay Minimum retry delay in seconds for API limit exceeded scenarios. 1

Setup

Warning

Replace the example values shown below with values appropriate for your environment.

  1. Navigate to Settings → Applications in the Self-Managed Portal.

  2. From the list of Connected Applications, select Databricks Unity Catalog.

  3. Click on the application name or the icon to edit. Then, go to the Access Management -> ADVANCED tab.

  4. Under Add New Custom Properties, add following properties:

    Bash
    1
2
3
4
5
    ranger.policysync.connector.0.dbx.api.limit.exceeded.max.retry.attempts=35
ranger.policysync.connector.0.default.time.interval.api.limit.exceeded.in.seconds=30
ranger.policysync.connector.0.dbx.retry.delay.min.seconds=5
ranger.policysync.connector.0.dbx.group.id.cache.timeout.minutes=20
ranger.policysync.connector.0.dbx.api.consecutive.group.user.update.delay.interval.ms=6000

  5. Click SAVE to apply the changes.

To prevent throttling, update the following properties in Privacera Manager:

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-unity-catalog/instance1/vars.connector.databricks.unity.catalog.yml

  3. Set the following properties to enable the connector to handle retrying of DBX SCIM API requests:

    YAML
    1
2
3
4
5
    CONNECTOR_DATABRICKS_UNITY_CATALOG_DBX_API_LIMIT_EXCEEDED_MAX_RETRY_ATTEMPTS: "35"
CONNECTOR_DATABRICKS_UNITY_CATALOG_DEFAULT_TIME_INTERVAL_API_LIMIT_EXCEEDED_IN_SECONDS: "30"
CONNECTOR_DATABRICKS_UNITY_CATALOG_API_RETRY_DELAY_MIN_SECONDS: "5"
CONNECTOR_DATABRICKS_UNITY_CATALOG_GROUP_ID_CACHE_TIMEOUT_MINUTES: "20"
CONNECTOR_DATABRICKS_UNITY_CATALOG_API_CONSECUTIVE_GROUP_USER_UPDATE_DELAY_INTERVAL_MS: "6000"

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud portal, navigate to Settings -> Applications.

  2. On the Connected Applications screen, select Databricks Unity Catalog.

  3. Click the pen icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access Management -> ADVANCED tab.

  5. Under Add New Custom Properties, add following properties:

    Bash
    1
2
3
4
5
    ranger.policysync.connector.0.dbx.api.limit.exceeded.max.retry.attempts=35
ranger.policysync.connector.0.default.time.interval.api.limit.exceeded.in.seconds=30
ranger.policysync.connector.0.dbx.retry.delay.min.seconds=5
ranger.policysync.connector.0.dbx.group.id.cache.timeout.minutes=20
ranger.policysync.connector.0.dbx.api.consecutive.group.user.update.delay.interval.ms=6000

  6. Click SAVE to apply the changes.

Comments