Skip to content

AWS Permission Loader Configuration – Lake Formation Push Mode

This section describes the configuration settings for permission loading behavior and threading when operating in Lake Formation Push Mode. These settings help optimize performance when working with large sets of AWS Lake Formation permissions.

Configuration Parameters

You can configure the following:

  • Permission loading mode
  • Thread counts and batching for permission loading in multi-threaded mode

If these parameters are not specified, default values will be applied automatically, as described below.

Defaults

  • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_KEY defaults to load_multi_thread (multi-threaded).
  • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_BATCH_SIZE defaults to 1000 (multi-threaded).

Setup

Warning

  • All configuration values must be entered as strings.
  • Thread-related parameters should be carefully tuned based on the volume of permissions data and the available system resources.

  1. SSH into the instance where Privacera Manager is installed.

  2. Open the Lake Formation connector configuration file:

    Note

    Replace instance1 with your actual connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/lakeformation/instance1/vars.connector.lakeformation.push.yml

  3. Configure the permission loader parameters as needed:

    YAML
    1
2
3
4
5
6
7
8
9
    # Permission Loading Mode (optional – defaults to load_multi_thread)
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_KEY: "load_multi_thread"

# Thread Configuration – Permission (only used when using load_multi_thread)
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_THREADS: "3"
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_MIN_THREADS: "1"

# Batch Size for Permission Processing
CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_BATCH_SIZE: "1000"

    Usage Guide

    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_KEY: Sets permission loading mode. Use "load" for single-threaded or "load_multi_thread" for multi-threaded mode (recommended for large datasets).
    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_THREADS: Number of threads to use for permission loading (multi-threaded mode only).
    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_MIN_THREADS: Minimum number of threads that will remain active during processing.
    • CONNECTOR_LAKEFORMATION_LOAD_PERMISSIONS_BATCH_SIZE: Number of resource permissions processed in each batch.

  4. After updating the configuration, apply the changes by running:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Comments