Skip to content

AWS API Batch Sizes – Lake Formation Push Mode

This section outlines the configurable batch size settings for various AWS API calls used when operating in Lake Formation Push Mode. These settings help optimize connector performance and manage AWS API rate limits effectively.

Configuration Parameters

You can configure batch sizes for the following AWS services:

  • AWS Glue
  • AWS Lake Formation
  • AWS IAM

If not specified, all parameters default to a batch size of 100.

Setup

Warning

  • All batch size values must be integers.
  • Tune these values based on your workload and AWS API rate limits to avoid throttling.
  • These settings are applicable only in Lake Formation Push Mode.

  1. SSH into the instance where Privacera Manager is installed.

  2. Open the Lake Formation connector configuration file for editing:

    Note

    If you have multiple connectors, replace instance1 with the appropriate connector instance name.

    Bash
    1
    vi ~/privacera/privacera-manager/config/custom-vars/connectors/lakeformation/instance1/vars.connector.lakeformation.push.yml

  3. Set the following properties to configure API batch sizes:

    YAML
    1
2
3
    CONNECTOR_LAKEFORMATION_AWS_GLUE_API_BATCH_SIZE: "100"
CONNECTOR_LAKEFORMATION_AWS_LAKEFORMATION_API_BATCH_SIZE: "100"
CONNECTOR_LAKEFORMATION_AWS_IAM_API_BATCH_SIZE: "100"

    Usage Guide

    • CONNECTOR_LAKEFORMATION_AWS_GLUE_API_BATCH_SIZE: Used for listing Glue databases and tables.
    • CONNECTOR_LAKEFORMATION_AWS_LAKEFORMATION_API_BATCH_SIZE: Used for listing Lake Formation tags, resources, and permissions.
    • CONNECTOR_LAKEFORMATION_AWS_IAM_API_BATCH_SIZE: Used for listing IAM users, roles, and groups.

  4. Once the properties are configured, run the following commands to update your Privacera Manager platform instance:

    Step 1 - Setup which generates the helm charts. This step usually takes few minutes.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
    Step 2 - Apply the Privacera Manager helm charts.
    Bash
    1
2
    cd ~/privacera/privacera-manager
./pm_with_helm.sh upgrade
    Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.

    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

Comments