Skip to content

Deny Unsupported SQL Primitives by Default on EMR

Overview

Use this configuration to enable the deny-by-default mechanism for unsupported SQL Primitives in EMR Spark queries.

Configure

  1. SSH to the instance where Privacera is installed.
  2. Run the following command to navigate to the /custom-vars directory.
    Bash
    1
    cd ~/privacera/privacera-manager/config/custom-vars
  3. Edit the vars.emr.yml file
    Bash
    1
    vi vars.emr.yml
  4. Uncomment the below-mentioned flag to set its value to true.
    Bash
    1
    EMR_SPARK_DENY_UNSUPPORTED_SQL_PRIMITIVES_ENABLED: "true"
  5. Once the properties are updated, run below command to generate the plugin configurations:
    Bash
    1
2
    cd ~/privacera/privacera-manager
./privacera-manager.sh post-install
  6. Follow the steps given here to deploy the EMR Cluster with changes.

Add EMR_SPARK_DENY_UNSUPPORTED_SQL_PRIMITIVES_ENABLED in EMR bootstrap action script to enable the Spark Plugin to Deny Unsupported SQL Primitives by Default.

privacera-emr-bootstrap-action-spark_fgac
JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
"BootstrapActions": [
{
  "Name": "Install Privacera Plugins on Master Node",
  "ScriptBootstrapAction": {
    "Path": "s3://elasticmapreduce/bootstrap-actions/run-if",
    "Args": [
      {
        "Fn::Sub": "instance.isMaster=true"
      },
      {
        "Fn::Sub": "export EMR_SPARK_DENY_UNSUPPORTED_SQL_PRIMITIVES_ENABLED=true; wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo -E ./privacera_emr.sh spark-fgac"
      }
    ]
  }
},
{
  "Name": "Install Spark FGAC in Core Node",
  "ScriptBootstrapAction": {
    "Path": "s3://elasticmapreduce/bootstrap-actions/run-if",
    "Args": [
      {
        "Fn::Sub": "instance.isMaster=false"
      },
      {
        "Fn::Sub": "export EMR_SPARK_DENY_UNSUPPORTED_SQL_PRIMITIVES_ENABLED=true; wget ${PrivaceraDownloadUrl}/privacera_emr.sh ; chmod +x ./privacera_emr.sh ; sudo -E ./privacera_emr.sh spark-fgac"
      }
    ]
  }
}
]