Skip to content

Using S3 Bucket for Init Scripts in Databricks Clusters with FGAC

This section describes how to upload the privacera_custom_conf.zip and initialization script files, specifically ranger_enable.sh and ranger_enable_scala.sh, to the S3 location in the Databricks environment.

This is applicable only Self Managed and Data Plane deployment of Privacera

Pre-requisites

  1. The EC2 instance where Privacera Manager is installed should have the permission to copy init scripts to the S3 bucket.

Setup

  1. SSH into the instance where Privacera Manager is installed.
  2. Go to the config directory.
    Bash
    1
    cd ~/privacera/privacera-manager/config
  3. Run the following command to open the vars.databricks.plugin.yml file to be edited.
    Bash
    1
    vi custom-vars/vars.databricks.plugin.yml
  4. To upload ranger_enable.sh, ranger_enable_scala.sh and privacera_custom_conf.zip files to the S3 location, update the following properties in the vars.databricks.plugin.yml file:
    Bash
    1
2
3
4
5
6
7
8
9
    # To upload files to Databricks Workspace. Default: `true`.
DATABRICKS_INIT_SCRIPT_WORKSPACE_FLAG_ENABLE: "false"

# To upload the `init scripts` and `privacera_cust_conf.zip` to the S3 location.
DATABRICKS_CUST_CONF_PATH: "s3://<bucket-name>/<path>"

# To upload the plugin installation logs to the S3 location. By default, the logs will be uploaded to the path 
configured in `DATABRICKS_CUST_CONF_PATH`.
PRIVACERA_CLUSTER_LOGS_DIR: "s3://<bucket-name>/<path>"
  5. Once the properties are configured, run the Privacera Manager to update the configurations and upload to S3
    Bash
    1
2
3
    cd ~/privacera/privacera-manager
./privacera-manager.sh setup
./pm_with_helm.sh upgrade

Run the following command to run the post install steps:

Bash
1
2
cd ~/privacera/privacera-manager
./privacera-manager.sh post-install

The init script is not uploaded automatically. Download the connector bundle, upload the script it contains to the S3 path, then set the cluster's init script Source to S3 - see Upload the init script.

Note

The cluster's instance profile needs s3:GetObject permission on that path.