Connect Elastic MapReduce from Amazon application to PrivaceraCloud

This topic describes how to connect an Elastic MapReduce from Amazon application to PrivaceraCloud for access control.

Connect EMR application

Go to Settings > Applications.
In the Applications screen, select EMR.
Enter the application Name and Description, and then click Save.
Click the toggle button to enable Access Management for your application.
Click Save.

EMR Spark access control types

EMR Spark supports two types of access control: Fine-Grained Access Control (FGAC) and Object Level Access Control (OLAC). EMR FGAC and OLAC are mutually exclusive.

Kerberos required for EMR FGAC or OLAC

Note

To support Privacera FGAC or OLAC, the EMR application must be configured with Kerberos.

EMR Spark OLAC

The advantages of EMR Spark OLAC:

It allows you to access existing AWS S3 resource location that you are trying to access with Spark.
It uses privacera_s3 service for resource-based access control and privacera_tag service for tag-based access control.
It uses the signed-authorization implementation from Privacera.

EMR Spark FGAC

When FGAC is installed and enabled, each data user query is parsed by Spark and authorized by the PrivaceraCloud Spark Plug-In. All resources referred to by the query must be accessible to the requesting user.

PrivaceraCloud configuration

Obtain shared key

Note

This is only needed for OLAC setup.

Obtain or determine a character string to serve as a "shared key" between PrivaceraCloud and the AWS EMR cluster. We'll refer to this as {SHARED_KEY} in the configuration steps below.

Connect S3 application to PrivaceraCloud. For more information about how to connect S3 application, see Connect S3 to PrivaceraCloud.
Select the connected S3 application, and then click the edit (pen) icon.
In the the ADVANCED tab, add the following property:
Substitute actual value for {SHARED KEY}.
```
dataserver.shared.secret={SHARED_KEY}
```
Click Save.

Obtain EMR script download URL

Obtain your account unique call-in <emr-script-download-url> to allow the EMR cluster to obtain additional scripts and setup from PrivaceraCloud:

In PrivaceraCloud portal, go to Settings > API Keys .
Use an existing Active Api Key or create a new one. Set Expiry = Never Expires.
Click the API Key Info (i) button.
On the API Key Info page, click the COPY URL button in front of the AWS EMR Setup Script to store the emr-script-download-url.

AWS IAM roles using CloudFormation setup

The following two IAM roles need to be created before launching the cluster. These can be created easily with minimal permission using the IAM roles creation template.
- Node role: EmrPrivaceraNodeRole
- App data access role: EmrPrivaceraDataAcessRole
If required, you can modify the template based on your requirements.
To create role, use the following AWS CLI CloudFormation command:
Note
This must be executed from an EC2 that has permission to create a CloudFormation stack.
```
aws --region <AWS-REGION> cloudformation create-stack --stack-name privacera-emr-role-creation --template-body file://emr-roles-creation-template.json --capabilities CAPABILITY_NAMED_IAM
```
For more information about how to create a stack using a CloudFormation template, see Create CloudFormation stack.

IAM roles creation template

 {
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description":"Create roles and policies for use by Privacera-Protected EMR Clusters",
  "Resources":{
    "EmrRestrictedRole":{
      "Type":"AWS::IAM::Role",
      "Properties":{
        "RoleName":{
          "Fn::Join":[
            "",
            [
              "EmrPrivaceraNodeRole"
            ]
          ]
        },
        "AssumeRolePolicyDocument":{
          "Version":"2012-10-17",
          "Statement":[
            {
              "Effect":"Allow",
              "Principal":{
                "Service":[
                  "ec2.amazonaws.com"
                ]
              },
              "Action":[
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path":"/"
      }
    },
    "EmrRestrictedPolicy":{
      "Type":"AWS::IAM::ManagedPolicy",
      "Properties":{
        "ManagedPolicyName":{
          "Fn::Join":[
            "",
            [
              "EMRPrivaceraNodePolicy"
            ]
          ]
        },
        "PolicyDocument":{
          "Version":"2012-10-17",
          "Statement":[
            {
              "Sid":"EmrServiceLimited",
              "Effect":"Allow",
              "Action":[
                "glue:CreateDatabase",
                "glue:UpdateDatabase",
                "glue:DeleteDatabase",
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:CreateTable",
                "glue:UpdateTable",
                "glue:DeleteTable",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetTableVersions",
                "glue:CreatePartition",
                "glue:BatchCreatePartition",
                "glue:UpdatePartition",
                "glue:DeletePartition",
                "glue:BatchDeletePartition",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:BatchGetPartition",
                "glue:CreateUserDefinedFunction",
                "glue:UpdateUserDefinedFunction",
                "glue:DeleteUserDefinedFunction",
                "glue:GetUserDefinedFunction",
                "glue:GetUserDefinedFunctions",
                "ec2:Describe*",
                "elasticmapreduce:Describe*",
                "elasticmapreduce:ListBootstrapActions",
                "elasticmapreduce:ListClusters",
                "elasticmapreduce:ListInstanceGroups",
                "elasticmapreduce:ListInstances",
                "elasticmapreduce:ListSteps"
              ],
              "Resource":"*"
            },
            {
              "Sid":"EmrS3Limited",
              "Effect":"Allow",
              "Action":"s3:*",
              "Resource":[
                "arn:aws:s3:::*.elasticmapreduce/*",
                "arn:aws:s3:::elasticmapreduce/*",
                "arn:aws:s3:::elasticmapreduce",
                "arn:aws:s3:::<PLEASE_UPDATE EMR_CLUSTER_LOG_S3_PATH>",
                "arn:aws:s3:::<PLEASE_UPDATE EMR_CLUSTER_HIVE_WAREHOUSE_S3_PATH>"
              ]
            },
            {
              "Sid":"EmrAssumeIAM",
              "Effect":"Allow",
              "Action":"sts:AssumeRole",
              "Resource":[
                "arn:aws:iam::<PLEASE_UPDATE AWS_ACCCOUNT_ID>:role/EmrPrivaceraDataAccessRole"
              ]
            }
          ]
        },
        "Roles":[
          {
            "Ref":"EmrRestrictedRole"
          }
        ]
      }
    },
    "EmrRoleForApps":{
      "Type":"AWS::IAM::Role",
      "Properties":{
        "RoleName":{
          "Fn::Join":[
            "",
            [
              "EmrPrivaceraDataAccessRole"
            ]
          ]
        },
        "AssumeRolePolicyDocument":{
          "Version":"2012-10-17",
          "Statement":[
            {
              "Effect":"Allow",
              "Principal":{
                "Service":[
                  "ec2.amazonaws.com"
                ]
              },
              "Action":[
                "sts:AssumeRole"
              ]
            },
            {
              "Effect":"Allow",
              "Principal":{
                "AWS":{
                  "Fn::GetAtt":[
                    "EmrRestrictedRole",
                    "Arn"
                  ]
                }
              },
              "Action":"sts:AssumeRole"
            }
          ]
        },
        "Path":"/"
      }
    },
    "DataAccessPolicy":{
      "Type":"AWS::IAM::ManagedPolicy",
      "Properties":{
        "ManagedPolicyName":{
          "Fn::Join":[
            "",
            [
              "EmrPrivaceraDataAcessPolicy"
            ]
          ]
        },
        "PolicyDocument":{
          "Version":"2012-10-17",
          "Statement":[
            {
              "Sid":"S3DataAccess",
              "Effect":"Allow",
              "Action":[
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:DeleteBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketAcl",
                "s3:GetBucketPolicy",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:PutObjectAcl"
              ],
              "Resource":[
                "arn:aws:s3:::*.elasticmapreduce/*",
                "arn:aws:s3:::elasticmapreduce/*",
                "arn:aws:s3:::elasticmapreduce",
                "arn:aws:s3:::<PLEASE_UPDATE EMR_CLUSTER_LOG_S3_PATH>",
                "arn:aws:s3:::<PLEASE_UPDATE EMR_CLUSTER_HIVE_WAREHOUSE_S3_PATH>"
              ]
            }
          ]
        },
        "Roles":[
          {
            "Ref":"EmrRoleForApps"
          }
        ]
      }
    },
    "EmrRestrictedRoleProfile":{
      "Type":"AWS::IAM::InstanceProfile",
      "Properties":{
        "InstanceProfileName":{
          "Ref":"EmrRestrictedRole"
        },
        "Roles":[
          {
            "Ref":"EmrRestrictedRole"
          }
        ]
      }
    },
    "EmrRoleForAppsProfile":{
      "Type":"AWS::IAM::InstanceProfile",
      "Properties":{
        "InstanceProfileName":{
          "Ref":"EmrRoleForApps"
        },
        "Roles":[
          {
            "Ref":"EmrRoleForApps"
          }
        ]
      }
    }
  },
  "Outputs":{
    "EMRRestrictedRole":{
      "Value":{
        "Ref":"EmrRestrictedRole"
      }
    },
    "EmrRoleForApps":{
      "Value":{
        "Ref":"EmrRoleForApps"
      }
    }
  }
}

Create a security configuration

You can create a security configuration using:

CloudFormation setup (Recommended)
AWS EMR console (Manually)

Create a security configuration using CloudFormation setup (Recommended)

A new security configuration must be created with the Kerberos server that will be connected to the EMR cluster. Using the EMR security configuration template, you can easily create this with minimal permission.
If required, you can modify the template emr-security-config-template.json based on your requirements.
To create security configuration, use the following AWS CLI CloudFormation command:
Note
This must be executed from an EC2 that has permission to create a CloudFormation stack.
```
aws --region <AWS-REGION> cloudformation create-stack --stack-name 
privacera-emr-security-config-creation  --template-body 
file://emr-security-config-template.json
```
For more information about how to create a stack using a CloudFormation template, see Create CloudFormation stack.

EMR security configuration template

Cluster Dedicated KDC Configuration Template

Note

This template assumes you have a cluster-specific KDC with Cross Realm Trust enabled.

{
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description":"Create Security Configuration for use by Privacera-Protected EMR Clusters",
  "Resources":{
    "SecurityConfiguration":{
      "Type":"AWS::EMR::SecurityConfiguration",
      "Properties":{
        "Name":"EmrPrivaceraSecurityGroup",
        "SecurityConfiguration":{
          "AuthorizationConfiguration":{
            "EmrFsConfiguration":{
              "RoleMappings":[
                {
                  "Role":"arn:aws:iam::<PLEASE_UPDATE AWS_ACCCOUNT_ID>:role/EmrPrivaceraDataAccessRole",
                  "IdentifierType":"User",
                  "Identifiers":[
                    "hive;presto;trino"
                  ]
                }
              ]
            }
          },
          "AuthenticationConfiguration":{
            "KerberosConfiguration":{
              "Provider":"ClusterDedicatedKdc",
              "ClusterDedicatedKdcConfiguration":{
                "TicketLifetimeInHours":24,
                "CrossRealmTrustConfiguration":{
                  "AdminServer":"<PLEASE_UPDATE>",
                  "Domain":"<PLEASE_UPDATE>",
                  "KdcServer":"<PLEASE_UPDATE>",
                  "Realm":"<PLEASE_UPDATE>"
                }
              }
            }
          }
        }
      }
    }
  }
}

External KDC Configuration Template

Note

This is required if you enable Multiple Master Nodes.

{
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description":"Create Security Configuration for use by Privacera-Protected EMR Clusters",
  "Resources":{
    "SecurityConfiguration":{
      "Type":"AWS::EMR::SecurityConfiguration",
      "Properties":{
        "Name":"EmrPrivaceraSecurityGroup",
        "SecurityConfiguration":{
          "AuthorizationConfiguration":{
            "EmrFsConfiguration":{
              "RoleMappings":[
                {
                  "Role":"arn:aws:iam::<PLEASE_UPDATE AWS_ACCCOUNT_ID>:role/EmrPrivaceraDataAccessRole",
                  "IdentifierType":"User",
                  "Identifiers":[
                    "hive;presto;trino"
                  ]
                }
              ]
            }
          },
          "AuthenticationConfiguration":{
            "KerberosConfiguration":{
              "Provider":"ExternalKdc",
              "ExternalKdcConfiguration":{
                "TicketLifetimeInHours":24,
                "KdcServerType":"Single",
                "AdminServer":"<PLEASE_UPDATE>",
                "KdcServer":"<PLEASE_UPDATE>"
              }
            }
          }
        }
      }
    }
  }
}

Manually create a security configuration using AWS EMR console

To create a security configuration using the AWS EMR console:

Login to the AWS EMR console.
In the left navigation, select Security Configuration > Create New Security Configuration.
Enter a Name for Security Configuration. For example, emr_sec_config.
Navigate to the Authentication section, check Enable Kerberos authentication, and enter the Kerberos environment details as follows:
- Provider: Cluster dedicated KDC
- TicketLifetime: 24 hours
- Cross-realm trust
  - Realm: EXAMPLE.COM
    You must use the actual URL instead of the domain EXAMPLE.COM.
  - Domain: example.com
  - Admin server: sever.admin.com
  - KDC server: server.example.com
Select Use IAM roles for EMRFS requests to Amazon S3.
- IAM Role: select the App data access role created in AWS IAM roles using CloudFormation setup.
- Under Basis for access select an identifier type ( User ) from the list and enter corresponding identifiers (hadoop;hive;presto;trino).

Create EMR cluster

Prerequisites

To enable access control for Trino or PrestoSQL in EMR, the specific application need to be enabled prior to configuring EMR.

Go to Settings > Applications.
In the Applications screen, select Trino or Presto as per your EMR version.
Note
If the EMR version is 6.4.0 or above, then select Trino.
If the EMR version is 6.x to 6.3.1, then select Presto.
Enter the application Name and Description, and then click Save.
Click the toggle button to enable Access Management for your application.

Kerberos required for EMR FGAC or OLAC

Note

To support Privacera FGAC or OLAC, EMR application must be running on Kerberos.

You can create EMR cluster using:

CloudFormation setup (Recommended)
- Using CloudFormation EMR templates
- Using Cloud formation AWS CLI
- Using Cloud formation AWS Console
AWS EMR console (Manually)

Create EMR cluster using CloudFormation setup (Recommended)

Create EMR cluster using CloudFormation EMR templates

To create an EMR cluster, use the CloudFormation templates listed below. You can modify the templates to meet your needs. It is recommended to maintain the same common variables from the previous setup steps.

EMR template: Spark_OLAC, Hive, PrestoDB (for EMR versions 5.33.1)

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-EMR-Spark_OLAC-Hive-Presto"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<KDC-USER-PASSWORD>"
    },
    "CrossRealmTrustPrincipalPassword":{
      "Description":"KDC Cross Realm Trust Principal password",
      "Type":"String",
      "Default":"<KDC-PRINCIPAL-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"EC2.INTERNAL",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          },
          "CrossRealmTrustPrincipalPassword":{
            "Ref":"CrossRealmTrustPrincipalPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":1,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-olac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark OLAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-olac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Presto"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"presto-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR template: Spark_FGAC, Hive, PrestoDB (for EMR versions 5.33.1)

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-EMR-Spark_FGAC-Hive-Presto"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<KDC-USER-PASSWORD>"
    },
    "CrossRealmTrustPrincipalPassword":{
      "Description":"KDC Cross Realm Trust Principal password",
      "Type":"String",
      "Default":"<KDC-PRINCIPAL-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"EC2.INTERNAL",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          },
          "CrossRealmTrustPrincipalPassword":{
            "Ref":"CrossRealmTrustPrincipalPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":1,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Presto"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"presto-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR Template for Multiple Master Node: Spark_OLAC, Hive, PrestoDB (for EMR versions 5.33.1 )

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-Multiple-Master-EMR-Spark_OLAC-Hive-Presto"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<PLEASE_UPDATE EXTERNAL-KDC-USER-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"<PLEASE_UPDATE EXTERNAL-KDC-REALM>",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":3,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-olac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark OLAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-olac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Presto"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          },
          {
            "Name":"Oozie"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"presto-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.security.credstore.java-keystore-provider.password-file":"",
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          },
          {
            "Classification":"oozie-site",
            "ConfigurationProperties":{
              "oozie.service.JPAService.jdbc.driver":"org.mariadb.jdbc.Driver",
              "oozie.service.JPAService.jdbc.url":"jdbc:mysql://<PLEASE_UPDATE EXTERNAL_DB_HOST>:<PLEASE_UPDATE EXTERNAL_DB_PORT>/<PLEASE_UPDATE EXTERNAL_DB_NAME>?createDatabaseIfNotExist=true",
              "oozie.service.JPAService.jdbc.username":"<PLEASE_UPDATE EXTERNAL_JDBC_USER>",
              "oozie.service.JPAService.jdbc.password":"<PLEASE_UPDATE EXTERNAL_JDBC_PASSWORD>"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"hue-ini",
            "ConfigurationProperties":{
              
            },
            "Configurations":[
              {
                "Classification":"desktop",
                "ConfigurationProperties":{
                  
                },
                "Configurations":[
                  {
                    "Classification":"database",
                    "ConfigurationProperties":{
                      "engine":"mysql",
                      "host":"<PLEASE_UPDATE EXTERNAL_DB_HOST>",
                      "port":"<PLEASE_UPDATE EXTERNAL_DB_HOST>",
                      "user":"<PLEASE_UPDATE EXTERNAL_DB_USER>",
                      "password":"<PLEASE_UPDATE EXTERNAL_DB_PASSWORD>",
                      "name":"<PLEASE_UPDATE EXTERNAL_DB_NAME>"
                    },
                    "Configurations":[
                      
                    ]
                  }
                ]
              }
            ]
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR Template for Multiple Master Node: Spark_FGAC, Hive, PrestoDB (for EMR versions 5.33.1 )

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-Multiple-Master-EMR-Spark_FGAC-Hive-Presto"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<PLEASE_UPDATE EXTERNAL-KDC-USER-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"<PLEASE_UPDATE EXTERNAL-KDC-REALM>",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":3,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Presto"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          },
          {
            "Name":"Oozie"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"presto-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.security.credstore.java-keystore-provider.password-file":"",
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          },
          {
            "Classification":"oozie-site",
            "ConfigurationProperties":{
              "oozie.service.JPAService.jdbc.driver":"org.mariadb.jdbc.Driver",
              "oozie.service.JPAService.jdbc.url":"jdbc:mysql://<PLEASE_UPDATE EXTERNAL_DB_HOST>:<PLEASE_UPDATE EXTERNAL_DB_PORT>/<PLEASE_UPDATE EXTERNAL_DB_NAME>?createDatabaseIfNotExist=true",
              "oozie.service.JPAService.jdbc.username":"<PLEASE_UPDATE EXTERNAL_JDBC_USER>",
              "oozie.service.JPAService.jdbc.password":"<PLEASE_UPDATE EXTERNAL_JDBC_PASSWORD>"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"hue-ini",
            "ConfigurationProperties":{
              
            },
            "Configurations":[
              {
                "Classification":"desktop",
                "ConfigurationProperties":{
                  
                },
                "Configurations":[
                  {
                    "Classification":"database",
                    "ConfigurationProperties":{
                      "engine":"mysql",
                      "host":"<PLEASE_UPDATE EXTERNAL_DB_HOST>",
                      "port":"<PLEASE_UPDATE EXTERNAL_DB_HOST>",
                      "user":"<PLEASE_UPDATE EXTERNAL_DB_USER>",
                      "password":"<PLEASE_UPDATE EXTERNAL_DB_PASSWORD>",
                      "name":"<PLEASE_UPDATE EXTERNAL_DB_NAME>"
                    },
                    "Configurations":[
                      
                    ]
                  }
                ]
              }
            ]
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR Template: Spark_OLAC, Hive, Trino (for EMR versions 6.11.0 )

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-EMR-Spark_OLAC-Hive-Trino"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<KDC-USER-PASSWORD>"
    },
    "CrossRealmTrustPrincipalPassword":{
      "Description":"KDC Cross Realm Trust Principal password",
      "Type":"String",
      "Default":"<KDC-PRINCIPAL-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"EC2.INTERNAL",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          },
          "CrossRealmTrustPrincipalPassword":{
            "Ref":"CrossRealmTrustPrincipalPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":1,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-olac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark OLAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-olac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Trino"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"trino-config",
            "ConfigurationProperties":{
              "internal-communication.shared-secret":"<PLEASE_UPDATE>"
            }
          },
          {
            "Classification":"trino-connector-hive",
            "ConfigurationProperties":{
              "connector.name":"hive",
              "hive.metastore":"glue",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR Template: Spark_FGAC, Hive, Trino (for EMR versions 6.11.0 )

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-EMR-Spark_FGAC-Hive-Trino"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<KDC-USER-PASSWORD>"
    },
    "CrossRealmTrustPrincipalPassword":{
      "Description":"KDC Cross Realm Trust Principal password",
      "Type":"String",
      "Default":"<KDC-PRINCIPAL-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"EC2.INTERNAL",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          },
          "CrossRealmTrustPrincipalPassword":{
            "Ref":"CrossRealmTrustPrincipalPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":1,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark FGAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Trino"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"trino-config",
            "ConfigurationProperties":{
              "internal-communication.shared-secret":"<PLEASE_UPDATE>"
            }
          },
          {
            "Classification":"trino-connector-hive",
            "ConfigurationProperties":{
              "connector.name":"hive",
              "hive.metastore":"glue",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR template: Spark_OLAC, Hive, Trino (for EMR versions 6.4.0 and above)

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-EMR-Spark_OLAC-Hive-Trino"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<KDC-USER-PASSWORD>"
    },
    "CrossRealmTrustPrincipalPassword":{
      "Description":"KDC Cross Realm Trust Principal password",
      "Type":"String",
      "Default":"<KDC-PRINCIPAL-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"EC2.INTERNAL",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          },
          "CrossRealmTrustPrincipalPassword":{
            "Ref":"CrossRealmTrustPrincipalPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":1,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fbac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark OLAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fbac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Trino"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.privacera.emr.glue.PrivaceraGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"trino-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR Template for Multiple Master Node: Spark_OLAC, Hive, Trino (for EMR version 6.4.0 and above)

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-Multiple-Master-EMR-Spark_OLAC-Hive-Trino"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<PLEASE_UPDATE EXTERNAL-KDC-USER-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"<PLEASE_UPDATE EXTERNAL-KDC-REALM>",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":3,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fbac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark OLAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fbac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Trino"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          },
          {
            "Name":"Oozie"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"trino-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.security.credstore.java-keystore-provider.password-file":"",
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          },
          {
            "Classification":"oozie-site",
            "ConfigurationProperties":{
              "oozie.service.JPAService.jdbc.driver":"org.mariadb.jdbc.Driver",
              "oozie.service.JPAService.jdbc.url":"jdbc:mysql://<PLEASE_UPDATE EXTERNAL_DB_HOST>:<PLEASE_UPDATE EXTERNAL_DB_PORT>/<PLEASE_UPDATE EXTERNAL_DB_NAME>?createDatabaseIfNotExist=true",
              "oozie.service.JPAService.jdbc.username":"<PLEASE_UPDATE EXTERNAL_JDBC_USER>",
              "oozie.service.JPAService.jdbc.password":"<PLEASE_UPDATE EXTERNAL_JDBC_PASSWORD>"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"hue-ini",
            "ConfigurationProperties":{
              
            },
            "Configurations":[
              {
                "Classification":"desktop",
                "ConfigurationProperties":{
                  
                },
                "Configurations":[
                  {
                    "Classification":"database",
                    "ConfigurationProperties":{
                      "engine":"mysql",
                      "host":"<PLEASE_UPDATE EXTERNAL_DB_HOST>",
                      "port":"<PLEASE_UPDATE EXTERNAL_DB_HOST>",
                      "user":"<PLEASE_UPDATE EXTERNAL_DB_USER>",
                      "password":"<PLEASE_UPDATE EXTERNAL_DB_PASSWORD>",
                      "name":"<PLEASE_UPDATE EXTERNAL_DB_NAME>"
                    },
                    "Configurations":[
                      
                    ]
                  }
                ]
              }
            ]
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR template: Spark_OLAC, Hive, PrestoSQL (for EMR versions 6.x to 6.3.1)

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-EMR-Spark_OLAC-Hive-PrestoSQL"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<KDC-USER-PASSWORD>"
    },
    "CrossRealmTrustPrincipalPassword":{
      "Description":"KDC Cross Realm Trust Principal password",
      "Type":"String",
      "Default":"<KDC-PRINCIPAL-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"EC2.INTERNAL",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          },
          "CrossRealmTrustPrincipalPassword":{
            "Ref":"CrossRealmTrustPrincipalPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":1,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fbac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark OLAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fbac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"PrestoSQL"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"prestosql-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR template: Spark_FGAC, Hive, Trino (for EMR versions 6.4.0 and above)

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-EMR-Spark_FGAC-Hive-Trino"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<KDC-USER-PASSWORD>"
    },
    "CrossRealmTrustPrincipalPassword":{
      "Description":"KDC Cross Realm Trust Principal password",
      "Type":"String",
      "Default":"<KDC-PRINCIPAL-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"EC2.INTERNAL",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          },
          "CrossRealmTrustPrincipalPassword":{
            "Ref":"CrossRealmTrustPrincipalPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":1,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark FGAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Trino"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"trino-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR Template for Multiple Master Node: Spark_FGAC, Hive, Trino (for EMR version 6.4.0 and above)

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-Multiple-Master-EMR-Spark_FGAC-Hive-Trino"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<PLEASE_UPDATE EXTERNAL-KDC-USER-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"<PLEASE_UPDATE EXTERNAL-KDC-REALM>",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":3,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark FGAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"Trino"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          },
          {
            "Name":"Oozie"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"trino-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.security.credstore.java-keystore-provider.password-file":"",
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          },
          {
            "Classification":"oozie-site",
            "ConfigurationProperties":{
              "oozie.service.JPAService.jdbc.driver":"org.mariadb.jdbc.Driver",
              "oozie.service.JPAService.jdbc.url":"jdbc:mysql://<PLEASE_UPDATE EXTERNAL_DB_HOST>:<PLEASE_UPDATE EXTERNAL_DB_PORT>/<PLEASE_UPDATE EXTERNAL_DB_NAME>?createDatabaseIfNotExist=true",
              "oozie.service.JPAService.jdbc.username":"<PLEASE_UPDATE EXTERNAL_JDBC_USER>",
              "oozie.service.JPAService.jdbc.password":"<PLEASE_UPDATE EXTERNAL_JDBC_PASSWORD>"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"hue-ini",
            "ConfigurationProperties":{
              
            },
            "Configurations":[
              {
                "Classification":"desktop",
                "ConfigurationProperties":{
                  
                },
                "Configurations":[
                  {
                    "Classification":"database",
                    "ConfigurationProperties":{
                      "engine":"mysql",
                      "host":"<PLEASE_UPDATE EXTERNAL_DB_HOST>",
                      "port":"<PLEASE_UPDATE EXTERNAL_DB_HOST>",
                      "user":"<PLEASE_UPDATE EXTERNAL_DB_USER>",
                      "password":"<PLEASE_UPDATE EXTERNAL_DB_PASSWORD>",
                      "name":"<PLEASE_UPDATE EXTERNAL_DB_NAME>"
                    },
                    "Configurations":[
                      
                    ]
                  }
                ]
              }
            ]
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

EMR template: Spark_FGAC, Hive, PrestoSQL (for EMR versions 6.x to 6.3.1)

{
  "Parameters":{
    "CLUSTERNAME":{
      "Description":"Name of the emr cluster",
      "Type":"String",
      "Default":"PCloud-EMR-Spark_FGAC-Hive-PrestoSQL"
    },
    "EMRRegion":{
      "Description":"aws region name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "EMRVersion":{
      "Description":"Emr version",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "MasterSecurityGroup":{
      "Description":"Emr master/edge node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "SlaveSecurityGroup":{
      "Description":"Emr worker/slave node security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "ServiceAccessSecurityGroup":{
      "Description":"Emr service access security group",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "Ec2SubnetId":{
      "Description":"Ec2 subnet id",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "HiveMetaStoreS3Path":{
      "Description":"Hive metastore s3 path",
      "Type":"String",
      "Default":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/hive_warehouse"
    },
    "Ec2KeyName":{
      "Description":"Ec2 keypair name",
      "Type":"String",
      "Default":"<EC2-SSH-KEY_PAIR-NAME>"
    },
    "Market":{
      "Description":"Ec2 Instance market type",
      "Type":"String",
      "Default":"ON_DEMAND"
    },
    "KDCName":{
      "Description":"KDC Name",
      "Type":"String",
      "Default":"<PLEASE_UPDATE>"
    },
    "KdcAdminPassword":{
      "Description":"KDC admin user password",
      "Type":"String",
      "Default":"<KDC-USER-PASSWORD>"
    },
    "CrossRealmTrustPrincipalPassword":{
      "Description":"KDC Cross Realm Trust Principal password",
      "Type":"String",
      "Default":"<KDC-PRINCIPAL-PASSWORD>"
    },
    "PrivaceraDownloadUrl":{
      "Description":"Privacera Base Download Url",
      "Type":"String",
      "Default":"https://privaceracloud.com/api/public/get/emr_script/<PLEASE UPDATED API-KEY>"
    }
  },
  "Resources":{
    "EMRCLUSTER":{
      "Type":"AWS::EMR::Cluster",
      "Properties":{
        "Name":{
          "Ref":"CLUSTERNAME"
        },
        "KerberosAttributes":{
          "Realm":"EC2.INTERNAL",
          "KdcAdminPassword":{
            "Ref":"KdcAdminPassword"
          },
          "CrossRealmTrustPrincipalPassword":{
            "Ref":"CrossRealmTrustPrincipalPassword"
          }
        },
        "SecurityConfiguration":{
          "Ref":"KDCName"
        },
        "VisibleToAllUsers":true,
        "EbsRootVolumeSize":15,
        "Instances":{
          "MasterInstanceGroup":{
            "InstanceCount":1,
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Master Instance Group"
          },
          "CoreInstanceGroup":{
            "InstanceCount":"<PLEASE_UPDATE>",
            "InstanceType":"m5.xlarge",
            "Market":{
              "Fn::Sub":"${Market}"
            },
            "Name":"Core Instance Group"
          },
          "Ec2KeyName":{
            "Ref":"Ec2KeyName"
          },
          "EmrManagedSlaveSecurityGroup":{
            "Fn::Sub":"${SlaveSecurityGroup}"
          },
          "EmrManagedMasterSecurityGroup":{
            "Fn::Sub":"${MasterSecurityGroup}"
          },
          "ServiceAccessSecurityGroup":{
            "Fn::Sub":"${ServiceAccessSecurityGroup}"
          },
          "Ec2SubnetId":{
            "Fn::Sub":"${Ec2SubnetId}"
          },
          "TerminationProtected":true
        },
        "BootstrapActions":[
          {
            "Name":"Install Privacera Plugins on Master Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=true"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          },
          {
            "Name":"Install Spark FGAC in Core Node",
            "ScriptBootstrapAction":{
              "Path":"s3://elasticmapreduce/bootstrap-actions/run-if",
              "Args":[
                {
                  "Fn::Sub":"instance.isMaster=false"
                },
                {
                  "Fn::Sub":"wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fgac"
                }
              ]
            }
          }
        ],
        "Applications":[
          {
            "Name":"Hive"
          },
          {
            "Name":"Spark"
          },
          {
            "Name":"PrestoSQL"
          },
          {
            "Name":"Zeppelin"
          },
          {
            "Name":"Livy"
          },
          {
            "Name":"Hue"
          }
        ],
        "Configurations":[
          {
            "Classification":"spark",
            "ConfigurationProperties":{
              "maximizeResourceAllocation":"true"
            },
            "Configurations":[
              
            ]
          },
          {
            "Classification":"spark-hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"hive-site",
            "ConfigurationProperties":{
              "hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
              "hive.metastore.schema.verification":"false",
              "hive.server2.enable.doAs":"false",
              "parquet.column.index.access":"true",
              "fs.s3a.impl":"com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
              "hive.metastore.warehouse.dir":{
                "Ref":"HiveMetaStoreS3Path"
              }
            }
          },
          {
            "Classification":"prestosql-connector-hive",
            "ConfigurationProperties":{
              "hive.metastore":"glue",
              "hive.allow-drop-table":"true",
              "hive.allow-add-column":"true",
              "hive.allow-rename-column":"true",
              "connector.name":"hive-hadoop2",
              "hive.config.resources":"/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
              "hive.s3-file-system-type":"EMRFS",
              "hive.hdfs.impersonation.enabled":"false",
              "hive.allow-drop-column":"true",
              "hive.allow-rename-table":"true"
            }
          },
          {
            "Classification":"livy-conf",
            "ConfigurationProperties":{
              "livy.impersonation.enabled":"true"
            }
          },
          {
            "Classification":"core-site",
            "ConfigurationProperties":{
              "hadoop.proxyuser.livy.groups":"*",
              "hadoop.proxyuser.livy.hosts":"*"
            }
          }
        ],
        "LogUri":"s3://<PLEASE_UPDATE BUCKET_NAME>/emr/emr_logs/",
        "JobFlowRole":"<PLEASE_UPDATE NODE_ROLE_NAME>",
        "ServiceRole":"EMR_DefaultRole",
        "ReleaseLabel":{
          "Fn::Sub":"${EMRVersion}"
        }
      }
    }
  }
}

Create EMR cluster using CloudFormation AWS CLI

Run the following command to create AWS EMR:

aws --region <AWS-REGION> cloudformation create-stack --stack-name 
privacera-emr-creation  --template-body file://<EMR-TEMPLATE-JSON-FILE-PATH>

Create CloudFormation stack

Follow these steps to create a stack on the AWS CloudFormation console:

Login in to the AWS Management Console and navigate to Cloudformation Console.
Click the Create Stack on the Stacks page.
On the Specify template page, select Template is ready option, and then Upload a template file.
Click the Choose File button, and select your modified emr template json file.
Click the Next button.
On the Specify stack details page, enter a stack name in the Stack name box.
Click the Next button.
Update Configure stack options as per your requirements.
Click the Next button.
Click the Create Stack button.
You will see the progress of your stack in the CloudFormation > Stacks section.

Manually create EMR cluster using AWS EMR console

Follow these steps to manually create AWS EMR cluster using AWS EMR console:

Login to AWS Management Console and navigate to EMR Console.
Click the Create cluster button.
click Go to advanced, and select Release. For example, emr-6.4.0

Configure applications for AWS EMR cluster

Follow these steps to configure individual application for AWS EMR cluster:

Select additional applications as per your environment, such as Spark, Hadoop, Hive, Trino or Presto.
In Edit software settings, select Enter configuration and add the following individual application's configuration array.

Spark configuration array

{
  "Classification": "spark",
  "ConfigurationProperties": {
    "maximizeResourceAllocation": "true"
  },
  "Configurations": []
},
{
  "Classification": "spark-hive-site",
  "ConfigurationProperties": {
    "hive.metastore.warehouse.dir": "s3://<bucket-name>/<path>"
  }
}

Hive configuration array

{
  "Classification": "hive-site",
  "ConfigurationProperties": {
    "javax.jdo.option.ConnectionURL": "",
    "javax.jdo.option.ConnectionDriverName": "org.mariadb.jdbc.Driver",
    "javax.jdo.option.ConnectionUserName": "root",
    "javax.jdo.option.ConnectionPassword": "welcome1",
    "hive.server2.enable.doAs": "false",
    "parquet.column.index.access": "true",
    "fs.s3a.impl": "com.amazon.ws.emr.hadoop.fs.EmrFileSystem",
    "hive.metastore.warehouse.dir": "s3://<bucket-name>/<path>"
  }
}

Trino/PrestoSQL configuration array

Note

If the EMR version is 6.4.0 or above, use trino in place of {application} in the array. Use prestosql for older versions.

Replace {application} based on the component you select.

For example:

For Trino, use trino-connector-hive
For PrestoSQL, use prestosql-connecor-hive

Presto and Trino/Presto-sql are incompatible. Only one at a time should be used.

Glue configuration:

{
  "Classification": "<application>-connector-hive",
  "ConfigurationProperties": {
    "hive.metastore": "glue",
    "hive.allow-drop-table": "true",
    "hive.allow-add-column": "true",
    "hive.allow-rename-column": "true",
    "connector.name": "hive-hadoop2",
    "hive.config.resources": "/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
    "hive.s3-file-system-type": "EMRFS",
    "hive.hdfs.impersonation.enabled": "false",
    "hive.allow-drop-column": "true",
    "hive.allow-rename-table": "true"
  }
}

EHM configuration:

{
  "Classification": "<application>-connector-hive",
  "ConfigurationProperties": {
    "hive.allow-drop-table": "true",
    "hive.allow-add-column": "true",
    "hive.allow-rename-column": "true",
    "connector.name": "hive-hadoop2",
    "hive.config.resources": "/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml",
    "hive.s3-file-system-type": "EMRFS",
    "hive.hdfs.impersonation.enabled": "false",
    "hive.allow-drop-column": "true",
    "hive.allow-rename-table": "true"
  }
}

Bootstrap actions

In Hardware settings, select Networking, Node, and Instance values as appropriate for your environment.
Go to General cluster settings, configure the cluster name, logging, debugging, and termination protection as needed for your environment.
Configure the General cluster settings by including two scripts that install the Privacera Signing Agent on both the master and worker nodes.
In Additional Options, expand Bootstrap Actions, select bootstrap action Run if, and then click Configure and add.
In the Bootstrap actions dialog, set the name of the master and core node to Privacera Signing Agent.
Copy the following script into Optional Arguments using your own emr-script-download-url script URL. See Obtain EMR script download URL.
Click Add when finished.

EMR OLAC (Object Level Access Control)

Optional Arguments for Privacera installation script:

Master node

instance.isMaster=true "wget <emr-script-download-url>; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fbac"

Core node

instance.isMaster=false "wget <emr-script-download-url>; chmod +x ./privacera_emr.sh ; sudo ./privacera_emr.sh spark-fbac"

Optional Arguments for Privacera installation script with delta.

Export the following additional variables in the Bootstrap actions to enable Delta Lake:

SPARK_DELTA_LAKE_ENABLE
SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL

Additionally, export the following variable for EMR version 6.10.0 or above.

Note

The following variable is not applicable for the EMR versions lower than 6.10.0.

SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL

Master node

The following master node is applicable for EMR version lower than 6.10.0.

instance.isMaster=true "export SPARK_DELTA_LAKE_ENABLE=enable-spark-deltalake; export SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL=<DELTA_LAKE_CORE_JAR_DOWNLOAD_URL>; wget <emr-script-download-url>; chmod +x ./privacera_emr.sh ; sudo -E ./privacera_emr.sh spark-fbac"

The following master node is applicable for EMR version 6.10.0 or above.

instance.isMaster=true "export SPARK_DELTA_LAKE_ENABLE=enable-spark-deltalake; export SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL=<DELTA_LAKE_CORE_JAR_DOWNLOAD_URL>; export SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL=<SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL> ; wget <emr-script-download-url> wget <emr-script-download-url>; chmod +x ./privacera_emr.sh ; sudo -E ./privacera_emr.sh spark-fbac"

Core node

The following core node is applicable for EMR version lower than 6.10.0.

instance.isMaster=false "export SPARK_DELTA_LAKE_ENABLE=enable-spark-deltalake; export SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL=<DELTA_LAKE_CORE_JAR_DOWNLOAD_URL>; wget <emr-script-download-url>; chmod +x ./privacera_emr.sh ; sudo -E ./privacera_emr.sh spark-fbac"

The following core node is applicable for EMR version 6.10.0 or above.

instance.isMaster=false "export SPARK_DELTA_LAKE_ENABLE=enable-spark-deltalake; export SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL=<DELTA_LAKE_CORE_JAR_DOWNLOAD_URL>; export SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL=<SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL> ; wget <emr-script-download-url>; chmod +x ./privacera_emr.sh ; sudo -E ./privacera_emr.sh spark-fbac"

Note

Ensure the following:

The Delta Lake core jar and Delta Lake Storage (Applicable only for EMR version 6.10.0 or above) are dependent on the Spark version. You must choose the correct version for your EMR.
Obtain and update the appropriate download URL for the Delta Lake core jar and Delta Lake Storage (Applicable only for EMR version 6.10.0 or above).
Delta Lake and Spark version compatibility. For more information about Delta Lake releases, see Compatibility with Apache Spark.

Optional Arguments for Privacera installation script with STS endpoints:

Export the following two additional variables in the Bootstrap actions to configure with the list of users who can access STS endpoints. You must include hive, trino, or presto users in the list for the services to work.

EMR_STS_STATE
EMR_STS_USERS_LIST

Master node

"EMR_STS_STATE=unblock ; export EMR_STS_USERS_LIST=hive,trino,<COMMA_SEPARATED_LIST_OF_USERS> ; 
wget <emr-script-download-url>; 
chmod +x ./privacera_emr.sh ; 
sudo -E ./privacera_emr.sh spark-olac"

Core node

instance.isMaster=false 
"EMR_STS_STATE=unblock ; export EMR_STS_USERS_LIST=hive,trino,<COMMA_SEPARATED_LIST_OF_USERS> ; 
wget <emr-script-download-url>; 
chmod +x ./privacera_emr.sh ; 
sudo -E ./privacera_emr.sh spark-olac"

EMR FGAC (Fine Grained Access Control)

Optional Arguments for Privacera installation script:

Master node

instance.isMaster=true 
"wget <emr-script-download-url>; 
chmod +x ./privacera_emr.sh ; 
sudo ./privacera_emr.sh spark-fgac"

Core node

instance.isMaster=false 
"wget <emr-script-download-url>; 
chmod +x ./privacera_emr.sh ; 
sudo ./privacera_emr.sh spark-fgac"

Optional Arguments for Privacera installation script with delta.

Export the following two additional variables in the Bootstrap actions to enable Delta Lake:

SPARK_DELTA_LAKE_ENABLE
SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL

Additionally, export the following variable for EMR version 6.10.0 or above.

Note

The following variable is not applicable for the EMR versions lower than 6.10.0.

SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL

Master Node

The following master node is applicable for EMR version lower than 6.10.0.

instance.isMaster=true 
"export SPARK_DELTA_LAKE_ENABLE=enable-spark-deltalake; 
export SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL=<DELTA_LAKE_CORE_JAR_DOWNLOAD_URL>; 
wget <emr-script-download-url>; 
chmod +x ./privacera_emr.sh ; 
sudo -E ./privacera_emr.sh spark-fgac"

The following master node is applicable for EMR version 6.10.0 or above.

instance.isMaster=true "export SPARK_DELTA_LAKE_ENABLE=enable-spark-deltalake; export SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL=<DELTA_LAKE_CORE_JAR_DOWNLOAD_URL>; export SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL=<SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL> ; wget <emr-script-download-url>; chmod +x ./privacera_emr.sh ; sudo -E ./privacera_emr.sh spark-fgac"

Core Node

The following core node is applicable for EMR version lower than 6.10.0.

instance.isMaster=false 
"export SPARK_DELTA_LAKE_ENABLE=enable-spark-deltalake; 
export SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL=<DELTA_LAKE_CORE_JAR_DOWNLOAD_URL>; 
wget <emr-script-download-url>; 
chmod +x ./privacera_emr.sh ; 
sudo -E ./privacera_emr.sh spark-fgac"

The following core node is applicable for EMR version 6.10.0 or above.

instance.isMaster=false "export SPARK_DELTA_LAKE_ENABLE=enable-spark-deltalake; export SPARK_DELTA_LAKE_CORE_JAR_DOWNLOAD_URL=<DELTA_LAKE_CORE_JAR_DOWNLOAD_URL>; export SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL=<SPARK_DELTA_LAKE_STORAGE_JAR_DOWNLOAD_URL> ;wget <emr-script-download-url>; chmod +x ./privacera_emr.sh ; sudo -E ./privacera_emr.sh spark-fgac"

Note

Ensure the following:

The Delta Lake core jar and Delta Lake Storage (Applicable only for EMR version 6.10.0 or above) are dependent on the Spark version. You must choose the correct version for your EMR.
Obtain and update the appropriate download URL for the Delta Lake core jar and Delta Lake Storage (Applicable only for EMR version 6.10.0 or above).
Delta Lake and Spark version compatibility. For more information about Delta Lake releases, see Compatibility with Apache Spark.

Optional Arguments for Privacera installation script with STS endpoints:

EMR_STS_STATE
EMR_STS_USERS_LIST

Master node

instance.isMaster=true 
"EMR_STS_STATE=unblock ; export EMR_STS_USERS_LIST=hive,trino,<COMMA_SEPARATED_LIST_OF_USERS> ; 
wget <emr-script-download-url>; 
chmod +x ./privacera_emr.sh ; 
sudo -E ./privacera_emr.sh spark-fgac"

Core node

instance.isMaster=false 
"EMR_STS_STATE=unblock ; export EMR_STS_USERS_LIST=hive,trino,<COMMA_SEPARATED_LIST_OF_USERS> ; 
wget <emr-script-download-url>; 
chmod +x ./privacera_emr.sh ; 
sudo -E ./privacera_emr.sh spark-fgac"

Use a custom policy repository with EMR

Prerequisite

Perform following steps to create a custom policy repository to use an EMR with a custom policy repository:

Login to PrivaceraCloud Portal.
On the Privacera Portal home page, expand Access Management, and then click the Resource Policies.
Click the three dots menu on the service for which you want to create a custom policy repository.
Click Add Service.
Add the values in the given fields as follows:
- Service Name: name of your service. For example, teamA_access_policy
- Select Tag Service: select privacera_tag to apply tag based policies for this custom repository.
- Username: It should be service user that is, hive
- Password: dummy
- jdbc.driverClassName: dummy or org.apache.hive.jdbc.HiveDriver
- jdbc.url: dummy
Click Save.

Optional Arguments

Following are the optional arguments for Privacera installation script with custom Hive repository.

Master node

instance.isMaster=true 
"export EMR_HIVE_SERVICE_NAME=<hive_repo_name>; 
export EMR_TRINO_HIVE_SERVICE_NAME=<trino_hive_repo_name>; 
export EMR_SPARK_HIVE_SERVICE_NAME=<spark_hive_repo_name>; 
wget <emr-script-download-url> ; 
chmod +x ./privacera_emr.sh ; 
sudo -E ./privacera_emr.sh spark-fgac"

Core node

instance.isMaster=false 
"export EMR_HIVE_SERVICE_NAME=<hive_repo_name>; 
export EMR_TRINO_HIVE_SERVICE_NAME=<trino_hive_repo_name>; 
export EMR_SPARK_HIVE_SERVICE_NAME=<spark_hive_repo_name>; 
wget <emr-script-download-url> ; chmod +x ./privacera_emr.sh ; 
sudo -E ./privacera_emr.sh spark-fgac"

Configure security options in EMR cluster

This section provides instruction to configure security options in EMR cluster.

Perform following steps to configure security options in EMR cluster:

In Security Options, select security options as per your environment.
Open Security Configuration, and select the configuration you created earlier. See Create a security configuration.
For example, "emr_sec_config".
In the following fields, enter values:
- Realm
- KDC admin password
Click Create cluster to complete.

Privacera Documentation

Table of ContentsTable of Contents

Connect Elastic MapReduce from Amazon application to PrivaceraCloud

Connect EMR application

EMR Spark access control types

Kerberos required for EMR FGAC or OLAC

Note

EMR Spark OLAC

EMR Spark FGAC

PrivaceraCloud configuration

Obtain shared key

Note

Obtain EMR script download URL

AWS IAM roles using CloudFormation setup

Note

IAM roles creation template

Create a security configuration

Create a security configuration using CloudFormation setup (Recommended)

Note

EMR security configuration template

Cluster Dedicated KDC Configuration Template

Note

External KDC Configuration Template

Note

Manually create a security configuration using AWS EMR console

Create EMR cluster

Note

Kerberos required for EMR FGAC or OLAC

Note

Create EMR cluster using CloudFormation setup (Recommended)

Create EMR cluster using CloudFormation EMR templates

EMR template: Spark_OLAC, Hive, PrestoDB (for EMR versions 5.33.1)

EMR template: Spark_FGAC, Hive, PrestoDB (for EMR versions 5.33.1)

EMR Template for Multiple Master Node: Spark_OLAC, Hive, PrestoDB (for EMR versions 5.33.1 )

EMR Template for Multiple Master Node: Spark_FGAC, Hive, PrestoDB (for EMR versions 5.33.1 )

EMR Template: Spark_OLAC, Hive, Trino (for EMR versions 6.11.0 )

EMR Template: Spark_FGAC, Hive, Trino (for EMR versions 6.11.0 )

EMR template: Spark_OLAC, Hive, Trino (for EMR versions 6.4.0 and above)

EMR Template for Multiple Master Node: Spark_OLAC, Hive, Trino (for EMR version 6.4.0 and above)

EMR template: Spark_OLAC, Hive, PrestoSQL (for EMR versions 6.x to 6.3.1)

EMR template: Spark_FGAC, Hive, Trino (for EMR versions 6.4.0 and above)

EMR Template for Multiple Master Node: Spark_FGAC, Hive, Trino (for EMR version 6.4.0 and above)

EMR template: Spark_FGAC, Hive, PrestoSQL (for EMR versions 6.x to 6.3.1)

Create EMR cluster using CloudFormation AWS CLI

Create CloudFormation stack

Manually create EMR cluster using AWS EMR console

Configure applications for AWS EMR cluster

Spark configuration array

Hive configuration array

Trino/PrestoSQL configuration array

Note

Bootstrap actions

EMR OLAC (Object Level Access Control)

Note

Note

EMR FGAC (Fine Grained Access Control)

Note

Note

Use a custom policy repository with EMR

Configure security options in EMR cluster