Skip to content

Preventing DBX SCIM API throttling

The Databricks SQL connector uses the Databricks SCIM API to manage users, groups, and roles within Databricks SQL. The Databricks SCIM API enforces rate limits that, if exceeded, can cause the connector to be throttled. To prevent throttling, you can configure the connector to automatically retry DBX SCIM API requests.

Configuration Properties

The following properties configure SCIM API retry behavior and related throttling settings:

Property Description Default Value
Maximum Retry Attempts Number of retry attempts for a failed API request due to rate limiting. 31
Retry Interval Wait time between retry attempts (in seconds). 20
Group Update Delay Delay interval between consecutive patch API calls to update group users (in milliseconds). 6000
Group ID Cache Timeout Cache timeout for group IDs in minutes. 15
Min Retry Delay Minimum retry delay in seconds for API limit exceeded scenarios. 1
Max Retry Delay Maximum retry delay in seconds for API limit exceeded scenarios. 10

Setup

Warning

Replace the example values shown below with values appropriate for your environment.

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited:

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/databricks-sql-analytics/instance1/vars.connector.databricks.sql.analytics.yml
    
  3. Add or update the following properties to enable the connector to handle retrying of DBX SCIM API requests:

    YAML
    1
    2
    3
    4
    5
    6
    CONNECTOR_DATABRICKS_SQL_ANALYTICS_API_LIMIT_EXCEEDED_MAX_RETRY_ATTEMPTS: "35"
    CONNECTOR_DATABRICKS_SQL_ANALYTICS_TIME_INTERVAL_BETWEEN_APPLY_UPDATE_RETRY_ATTEMPT_IN_SEC: "30"
    CONNECTOR_DATABRICKS_SQL_ANALYTICS_API_RETRY_DELAY_MIN_SECONDS: "5"
    CONNECTOR_DATABRICKS_SQL_ANALYTICS_API_RETRY_DELAY_MAX_SECONDS: "15"
    CONNECTOR_DATABRICKS_SQL_ANALYTICS_GROUP_ID_CACHE_TIMEOUT_MINUTES: "20"
    CONNECTOR_DATABRICKS_SQL_ANALYTICS_API_CONSECUTIVE_GROUP_USER_UPDATE_DELAY_INTERVAL_MS: "6000"
    

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    cd ~/privacera/privacera-manager
    ./privacera-manager.sh setup
    
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    cd ~/privacera/privacera-manager
    ./pm_with_helm.sh upgrade
    
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    cd ~/privacera/privacera-manager
    ./privacera-manager.sh post-install
    
  1. In PrivaceraCloud portal, navigate to SettingsApplications.

  2. On the Connected Applications screen, select Databricks SQL.

  3. Click on the icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access ManagementADVANCED tab.

  5. Under Add New Custom Properties, add the following properties:

    Bash
    1
    2
    3
    4
    5
    6
    ranger.policysync.connector.0.dbx.api.limit.exceeded.max.retry.attempts=35
    ranger.policysync.connector.0.time.interval.between.apply.update.retry.attempt.in.seconds=30
    ranger.policysync.connector.0.dbx.retry.delay.min.seconds=5
    ranger.policysync.connector.0.dbx.retry.delay.max.seconds=15
    ranger.policysync.connector.0.dbx.group.id.cache.timeout.minutes=20
    ranger.policysync.connector.0.dbx.api.consecutive.group.user.update.delay.interval.ms=6000
    

  6. Click SAVE to apply the changes.

Comments