Skip to content

AWS Permission Loader Configuration – Lake Formation Push Mode

This section describes the configuration settings for permission loading behavior and threading when operating in Lake Formation Push Mode. These settings help optimize performance when working with large sets of AWS Lake Formation permissions.

Configuration Parameters

You can configure the following:

  • Permission loading mode
  • Thread counts and batching for permission loading in multi-threaded mode

If these parameters are not specified, default values will be applied automatically, as described below.

Defaults

  • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_KEY defaults to load_multi_thread (multi-threaded).
  • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_BATCH_SIZE defaults to 1000 (multi-threaded).
  • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_THREAD_POOL_WAIT_TIMEOUT_MINUTES defaults to "1200" (20 hours).
  • CONNECTOR_LAKEFORMATION_USE_THREAD_POOL_EXECUTOR_V2 defaults to "true".

Setup

Warning

  • All configuration values must be entered as strings.
  • Thread-related parameters should be carefully tuned based on the volume of permissions data and the available system resources.

  1. SSH into the instance where Privacera Manager is installed.

  2. Open the Lake Formation connector configuration file:

    Note

    Replace instance1 with your actual connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/lakeformation/instance1/vars.connector.lakeformation.push.yml

  3. Configure the permission loader parameters as needed:

    YAML
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
    # Permission Loading Mode (optional – defaults to load_multi_thread)
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_KEY: "load_multi_thread"

# Thread Configuration – Permission (only used when using load_multi_thread)
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_THREADS: "3"
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_MIN_THREADS: "1"

# Batch Size for Permission Processing
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_BATCH_SIZE: "1000"

# Thread Pool Executor Configuration
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_THREAD_POOL_WAIT_TIMEOUT_MINUTES: "1200"
CONNECTOR_LAKEFORMATION_USE_THREAD_POOL_EXECUTOR_V2: "true"

    Usage Guide

    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_KEY: Sets permission loading mode. Use "load" for single-threaded or "load_multi_thread" for multi-threaded mode (recommended for large datasets).
    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_THREADS: Number of threads to use for permission loading (multi-threaded mode only).
    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_MIN_THREADS: Minimum number of threads that will remain active during processing.
    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_BATCH_SIZE: Number of resource permissions processed in each batch.
    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_THREAD_POOL_WAIT_TIMEOUT_MINUTES: Maximum wait time (in minutes) for permission loader thread pool tasks to complete. Defaults to "1200" (20 hours). Increase this value for large-scale permission loading operations.
    • CONNECTOR_LAKEFORMATION_USE_THREAD_POOL_EXECUTOR_V2: Enables Thread Pool Executor v2 for improved permission loading performance and better thread management. Set to "true" to use the enhanced thread pool executor. Defaults to "true".

  4. After updating the configuration, apply the changes by running:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - (Optional) Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on. This step is not required if you are updating only connector properties.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install