Setup for Access Management for EMR Serverless¶

Perform following steps to configure EMR Serverless connector:

1. SSH into the instance where Privacera Manager is installed.

2. Navigate to the /config directory by running the following command:

   Bash
   1	cd ~/privacera/privacera-manager/config

3. Copy the sample variables by running the following command:

   Bash
   1	cp sample-vars/vars.emr-serverless.yml custom-vars/

4. Open the .yml file for editing by running the following command:

   Bash
   1	vi custom-vars/vars.emr-serverless.yml

5. Modify the following properties. You can get the supported versions from the AWS EMR Serverless from AWS EMR Serverless Versions

   Bash
   1 2 3 4 5 6 7 8	EMR_SERVERLESS_ENABLE: "true" # EMR serverless version e.g. emr-7.2.0:latest EMR_SERVERLESS_VERSION: "<PLEASE_CHANGE>" # The unique name of the EMR Serverless application within your AWS account # to avoid conflicts with other applications. Example: privacera1-emr-serverless EMR_SERVERLESS_APP_NAME: "<PLEASE_CHANGE>"

   Note

   EMR_SERVERLESS_VERSION is the EMR Serverless Spark Docker image tag, which you can get from this link.

6. Once the properties are configured, update your Privacera Manager platform instance by following the commands

   Bash
   1 2	cd ~/privacera/privacera-manager ./privacera-manager.sh post-install

7. Once the post-install process is complete, you will see emr-serverless folder in the ~/privacera/privacera-manager/output directory, with the following folder structure:

   Bash
   1 2 3 4 5 6 7	output/ ├── emr-serverless/ │ ├── olac/ │ │ ├── Dockerfile_Privacera_Spark_OLAC │ │ ├── setup_emrserverless_spark_olac.sh │ │ ├── spark_custom_conf │ │ ├── spark_custom_conf.zip

1. To build and push the Docker image, you need to copy the following Docker files and other configuration files to the EC2 instance where you can build the Docker image or you can build in the same EC2 instance where Privacera Manager is installed.

   Bash
   1 2 3 4	## Files to copy Dockerfile_Privacera_Spark_OLAC setup_emrserverless_spark_olac.sh spark_custom_conf.zip

2. Once the required files are on the EC2 instance where you can build the Docker image, run the following command. You can set the following environments variables before running the command or replace the values in the command itself.

   Here are some global variables that you need to set before running the command:

   Variable Name	Description	Sample Value
   aws_account_id	Your AWS account ID.	"123456789012"
   region	The AWS region where your ECR repository is located.	"us-east-1"
   ecr_repo_name	The name of your ECR repository where the Docker image will be pushed.	"privacera/emr-serverless-spark-olac"
   tag	The tag for the Docker image.	v1.0

   You can set the following environments variables before running the command or replace the values in the command itself.

   Bash
   1 2 3 4	aws_account_id=<PLEASE_CHANGE> region=<PLEASE_CHANGE> ecr_repo_name=<PLEASE_CHANGE> tag=<PLEASE_CHANGE>

3. Ensure that you have the necessary IAM permissions to manage customized Docker image in your Amazon Elastic Container Registry (ECR). You can use the IAM policies on this AWS documentation link to grant the necessary permissions.

4. Create ECR repository by running the following command. This is a one-time setup.

   Bash
   1 2 3	aws ecr create-repository \ --repository-name ${ecr_repo_name} \ --region ${region}

5. Run the following command to log into AWS Elastic Container Registry (ECR) repository that you just created. Make sure to set the environment variables or replace the variables before running the command.

   Bash
   1 2 3 4	# Login to ECR repo aws ecr get-login-password --region ${region} | \ docker login --username AWS \ --password-stdin ${aws_account_id}.dkr.ecr.${region}.amazonaws.com

6. Make sure you have buildx support in your docker cli by running -

   Bash
   1	docker buildx --help

   If the above command fails, then you need to enable buildx by following the instructions at this to install Docker buildx.

7. Follow these steps to build a multi-arch Docker image that can be used to run on AWS Intel/AMD EC2 instances and AWS Graviton EC2 instances for EMR Serverless.

   First build a builder instance with the following command. This allows you to build the multi-architecture Docker image for the CPU architecture of your build host.

   Bash
   1 2 3 4	# --use option can set as the builder as a default builder docker buildx create \ --name multi-arch-builder \ --driver=docker-container

   Then follow this command build to build the EMR Serverless Docker image for both ARM and x86 platforms.

   Since it is a multi-architecture image, it cannot be loaded in your docker engine but has to be pushed into ECR directly. This step also pushes into the ECR repository. Later you can do a docker pull from ECR which will load the correct platform image based on the architecture of the host.

   Bash
   1 2 3 4 5 6 7	docker buildx build \ --file ./Dockerfile_Privacera_Spark_OLAC \ --tag ${aws_account_id}.dkr.ecr.${region}.amazonaws.com/${ecr_repo_name}:${tag} \ --platform linux/arm64/v8,linux/amd64 \ --builder multi-arch-builder \ --push \ .

8. To verify that the Docker image was created successfully, run the following commands. Make sure to set the environment variables or replace the variables before running the command.

   Bash
   1 2 3 4	# this command will show the manifest which should have both the ARM # and x86 platforms docker manifest inspect \ ${aws_account_id}.dkr.ecr.${region}.amazonaws.com/${ecr_repo_name}:${tag}

   Bash
   1 2 3 4	# this will pull the correct platform image based on the architecture of # the host docker pull \ ${aws_account_id}.dkr.ecr.${region}.amazonaws.com/${ecr_repo_name}:${tag}

   Bash
   1 2 3	# This command will open the bash shell in the docker container. docker run -it --rm --entrypoint /bin/bash \ ${aws_account_id}.dkr.ecr.${region}.amazonaws.com/${ecr_repo_name}:${tag}

   Once inside the container, you can inspect the environment to ensure it’s set up correctly. There should be /opt/privacera folder inside the image. Run exit to exit. This should also delete the container since the --rm flag was used.