AWS Resource Loader Configuration – Lake Formation¶
This section describes the configuration settings for resource loading behavior and threading when operating in Lake Formation Push/Pull Mode. These settings are designed to optimize performance when working with large datasets in AWS Lake Formation.
Configuration Parameters¶
You can configure the following:
- Resource loading mode
- Optional toggles for tag and data location loading
- Thread counts for each resource type when using multi-threaded loading
If these parameters are not specified, default values will be applied automatically, as described below.
Defaults¶
CONNECTOR_LAKEFORMATION_LOAD_RESOURCES_KEY
defaults toload_multi_thread
.CONNECTOR_LAKEFORMATION_LOAD_DATA_LOCATION_ENABLED
defaults to"true"
.CONNECTOR_LAKEFORMATION_LOAD_TAG_ENABLED
defaults to"true"
.
Setup¶
Warning
- All configuration values must be entered as strings.
- Thread-related parameters should be carefully tuned based on the volume of permissions data and the available system resources.
-
SSH into the instance where Privacera Manager is installed.
-
Open the Lake Formation connector configuration file:
For Push Mode¶
Note
Replace
instance1
with your actual connector instance name.Bash For Pull Mode¶
Note
Replace
instance1
with your actual connector instance name.Bash -
Configure the resource loader parameters as needed:
Usage Guide
- CONNECTOR_LAKEFORMATION_LOAD_RESOURCES_KEY: Sets resource loading mode.
load_multi_thread
is the default and recommended for large datasets. - CONNECTOR_LAKEFORMATION_LOAD_DATA_LOCATION_ENABLED: Enables loading of data location resource. Defaults to
"true"
if not set. Only supported in multi-threaded mode. - CONNECTOR_LAKEFORMATION_LOAD_TAG_ENABLED: Enables loading of tag resource. Defaults to
"true"
if not set. Only supported in multi-threaded mode. - Thread properties: Specify the number of threads for loading each resource type. Tune according to system capabilities.
- CONNECTOR_LAKEFORMATION_LOAD_RESOURCES_KEY: Sets resource loading mode.
-
After updating the configuration, apply the changes by running:
Step 1 - Setup which generates the helm charts. This step usually takes few minutes.
Step 2 - Apply the Privacera Manager helm charts. Step 3 - Post-installation step which generates Plugin tar ball, updates Route 53 DNS and so on.