Skip to content

Parallel Resource Loading

Enable parallel resource loading to reduce resource sync time when the MSSQL connector manages a large number of databases.

Note

This feature is disabled by default and only takes effect when more than one database is configured in the managed database list. See Manage Resources List for details on configuring the database list.

Setup

  1. SSH to the instance where Privacera Manager is installed.

  2. Run the following command to open the .yml file to be edited.

    If you have multiple connectors, then replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/mssql/instance1/vars.connector.mssql.yml

  3. Set the following properties to enable parallel resource loading:

    YAML
    1
2
3
4
    CONNECTOR_MSSQL_LOAD_RESOURCES_MULTITHREADED_ENABLED: "true"
CONNECTOR_MSSQL_LOAD_RESOURCES_DATABASE_THREAD_COUNT: "<please_change>"
CONNECTOR_MSSQL_LOAD_RESOURCES_THREAD_POOL_WAIT_TIMEOUT_MINUTES: "<please_change>"
CONNECTOR_MSSQL_JDBC_MAX_POOL_SIZE: "<please_change>"
    • CONNECTOR_MSSQL_LOAD_RESOURCES_MULTITHREADED_ENABLED — Enables parallel per-database resource discovery. Default is false.
    • CONNECTOR_MSSQL_LOAD_RESOURCES_DATABASE_THREAD_COUNT — Maximum number of parallel worker threads. Default is 3.
    • CONNECTOR_MSSQL_LOAD_RESOURCES_THREAD_POOL_WAIT_TIMEOUT_MINUTES — Maximum minutes to wait for all worker threads to complete. Default is 60. Increase to 120 for large deployments.
    • CONNECTOR_MSSQL_JDBC_MAX_POOL_SIZE — JDBC connection pool size. Must be greater than or equal to the thread count when parallel loading is enabled. Default is 15.

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

  1. In PrivaceraCloud portal, navigate to Settings → Applications.

  2. On the Connected Applications screen, select MSSQL.

  3. Select the pen icon or the Account Name to modify the settings.

  4. On the Edit Application screen, go to Access Management → ADVANCED tab.

  5. Under Add New Custom Properties, add the following properties:

    Properties
    1
2
3
4
    ranger.policysync.connector.0.load.resources.multithreaded.enabled=true
ranger.policysync.connector.0.load.resources.load.database.thread.count=<please_change>
ranger.policysync.connector.0.load.resources.thread.pool.wait.timeout.minutes=<please_change>
ranger.policysync.connector.0.jdbc.maximum.pool.size=<please_change>
    • load.resources.multithreaded.enabled — Enables parallel per-database resource discovery. Default is false.
    • load.resources.load.database.thread.count — Maximum number of parallel worker threads. Default is 3.
    • load.resources.thread.pool.wait.timeout.minutes — Maximum minutes to wait for all worker threads to complete. Default is 60. Increase to 120 for large deployments.
    • jdbc.maximum.pool.size — JDBC connection pool size. Must be greater than or equal to the thread count when parallel loading is enabled. Default is 15.

  6. Select SAVE to apply the changes.