Databricks Spark Fine-Grained Access Control Plugin [FGAC] [Python, SQL]
Configuration
-
Run the following commands.
cd ~/privacera/privacera-manager cp config/sample-vars/vars.databricks.plugin.yml config/custom-vars/ vi config/custom-vars/vars.databricks.plugin.yml
-
Edit the following properties to allow Privacera Platform to connect to your Databricks host. For property details and description, refer to the Configuration Properties below.
DATABRICKS_HOST_URL: "<PLEASE_UPDATE>" DATABRICKS_TOKEN: "<PLEASE_UPDATE>" DATABRICKS_WORKSPACES_LIST: - alias: DEFAULT databricks_host_url: "{{DATABRICKS_HOST_URL}}" token: "{{DATABRICKS_TOKEN}}" DATABRICKS_MANAGE_INIT_SCRIPT: "true" DATABRICKS_ENABLE: "true"
Note
You can also add custom properties that are not included by default. See Databricks.
-
Run the following commands.
cd ~/privacera/privacera-manager ./privacera-manager.sh update
-
(Optional) By default, policies under the default service name, privacera_hive, are enforced. You can customize a different service name and enforce policies defined in the new name. See Configure Service Name for Databricks Spark Plugin.
Configuration Properties
Property Name | Description | Example Values |
---|---|---|
DATABRICKS_HOST_URL | Enter the URL where the Databricks environment is hosted. | For AZURE Databricks, DATABRICKS_HOST_URL: "https://xdx-66506xxxxxxxx.2.azuredatabricks.net/?o=665066931xxxxxxx" For AWS Databricks DATABRICKS_HOST_URL: "https://xxx-7xxxfaxx-xxxx.cloud.databricks.com" |
DATABRICKS_TOKEN |
Enter the token. To generate the token,1. Login to your Databricks account. 2. Click the user profile icon in the upper right corner of your Databricks workspace. 3. Click User Settings. 4. Click the Generate New Token button. 5. Optionally enter a description (comment) and expiration period. 6. Click the Generate button. 7. Copy the generated token. |
DATABRICKS_TOKEN: "xapid40xxxf65xxxxxxe1470eayyyyycdc06" |
DATABRICKS_WORKSPACES_LIST |
Add multiple Databricks workspaces to connect to Ranger.
Note: |
|
DATABRICKS_ENABLE | If set to 'true' Privacera Manager will create the Databricks cluster Init script "ranger_enable.sh" to: '~/privacera/privacera-manager/output/databricks/ranger_enable.sh. |
"true" "false" |
DATABRICKS_MANAGE_INIT_SCRIPT |
If set to 'true' Privacera Manager will upload Init script ('ranger_enable.sh') to the identified Databricks Host. If set to 'false' upload the following two files to the DBFS location. The files can be located at *~/privacera/privacera-manager/output/databricks*.
| "true" "false" |
DATABRICKS_SPARK_PLUGIN_AGENT_JAR | Use the Java agent to assign a string of extra JVM options to pass to the Spark driver. | -javaagent:/databricks/jars/privacera-agent.jar |
DATABRICKS_SPARK_PRIVACERA_CUSTOM_CURRENT_USER_UDF_NAME |
Property to map logged-in user to Ranger user for row-filter policy. It is mapped with the Databricks cluster-level property |
current_user() |
DATABRICKS_SPARK_PRIVACERA_VIEW_LEVEL_MASKING_ROWFILTER_EXTENSION_ENABLE | Property to enable masking, row-filter and data_admin access on view.
Property to enable masking, row-filter and data_admin access on view. This property is a Privacera Manager (PM) property It is mapped with the Databricks cluster-level property |
false |
DATABRICKS_SQL_CLUSTER_POLICY_SPARK_CONF |
Configure Databricks Cluster policy. Add the following JSON in the text area:
|
|
DATABRICKS_POST_PLUGIN_COMMAND_LIST | Note: This property is not part of the default YAML file, but can be added, if required. Use this property, if you want to run a specific set of commands in the Databricks init script. |
The following example will be added to the cluster init script to allow Athena JDBC via data access server. DATABRICKS_POST_PLUGIN_COMMAND_LIST: - sudo iptables -I OUTPUT 1 -p tcp -m tcp --dport 8181 -j ACCEPT - sudo curl -k -u user:password {{PORTAL_URL}}/api/dataserver/cert?type=dataserver_jks -o /etc/ssl/certs/dataserver.jks - sudo chmod 755 /etc/ssl/certs/dataserver.jks |
DATABRICKS_SPARK_PYSPARK_ENABLE_PY4J_SECURITY |
This property allows you to backlist APIs to enable security. This property is a Privacera Manager (PM) property It is mapped with the Databricks cluster-level property |
The following example will be added to the cluster init script to allow Athena JDBC via data access server. DATABRICKS_POST_PLUGIN_COMMAND_LIST: - sudo iptables -I OUTPUT 1 -p tcp -m tcp --dport 8181 -j ACCEPT - sudo curl -k -u user:password {{PORTAL_URL}}/api/dataserver/cert?type=dataserver_jks -o /etc/ssl/certs/dataserver.jks - sudo chmod 755 /etc/ssl/certs/dataserver.jks |
Managing init Script
If DATABRICKS_ENABLE is 'true' and DATABRICKS_MANAGE_INIT_SCRIPT is "true", the Init script will be uploaded automatically to your Databricks host. The Init Script will be uploaded to dbfs:/privacera/<DEPLOYMENT_ENV_NAME>/ranger_enable.sh
, where <DEPLOYMENT_ENV_NAME>
is the value of DEPLOYMENT_ENV_NAME
mentioned in vars.privacera.yml
.
If DATABRICKS_ENABLE is 'true' and DATABRICKS_MANAGE_INIT_SCRIPT is "false" the Init script must be uploaded to your Databricks host.
Note
To avoid the manual steps below, you should set DATABRICKS_MANAGE_INIT_SCRIPT=true and follow the instructions on the Automatic Upload tab above.
-
Open a terminal and connect to Databricks account using your Databricks login credentials/token.
-
Connect using login credentials:
-
If you're using login credentials, then run the following command.
databricks configure --profile privacera
-
Enter the Databricks URL.
Databricks Host (should begin with https://): https://dbc-xxxxxxxx-xxxx.cloud.databricks.com/
-
Enter the username and password.
Username: email-id@yourdomain.com Password:
-
-
Connect using Databricks token:
-
If you don't have a Databricks token, you can generate one. For more information, refer Generate a personal access token.
-
If you're using token, then run the following command.
databricks configure --token --profile privacera
-
Enter the Databricks URL.
Databricks Host (should begin with https://): https://dbc-xxxxxxxx-xxxx.cloud.databricks.com/
-
Enter the token.
Token:
-
-
-
To check if the connection to your Databricks account is established, run the following command.
dbfs ls dbfs:/ --profile privacera
You should see the list of files in the output, if you are connected to your account.
-
Upload files manually to Databricks.
-
Copy the following files to DBFS, which are available in the PM host at the location,
~/privacera/privacera-manager/output/databricks
:- ranger_enable.sh
- privacera_spark_plugin.conf
- privacera_spark_plugin_job.conf
- privacera_custom_conf.zip
-
Run the following command. For the value of
<DEPLOYMENT_ENV_NAME>
, you can get it from the file,~/privacera/privacera-manager/config/vars.privacera.yml
.export DEPLOYMENT_ENV_NAME=<DEPLOYMENT_ENV_NAME> dbfs mkdirs dbfs:/privacera/${DEPLOYMENT_ENV_NAME} --profile privacera dbfs cp ranger_enable.sh dbfs:/privacera/${DEPLOYMENT_ENV_NAME}/ --profile privacera dbfs cp privacera_spark_plugin.conf dbfs:/privacera/${DEPLOYMENT_ENV_NAME}/ --profile privacera dbfs cp privacera_spark_plugin_job.conf dbfs:/privacera/${DEPLOYMENT_ENV_NAME}/ --profile privacera dbfs cp privacera_custom_conf.zip dbfs:/privacera/${DEPLOYMENT_ENV_NAME}/ --profile privacera
-
Verify the files have been uploaded.
dbfs ls dbfs:/privacera/${DEPLOYMENT_ENV_NAME}/ --profile privacera
The Init Script will be uploaded to
dbfs:/privacera/<DEPLOYMENT_ENV_NAME>/ranger_enable.sh
, where<DEPLOYMENT_ENV_NAME>
is the value ofDEPLOYMENT_ENV_NAME
mentioned invars.privacera.yml
.
-
Configure Databricks Cluster
-
Once the update completes successfully, log on to the Databricks console with your account and open the target cluster, or create a new target cluster.
-
Open the Cluster dialog. enter Edit mode.
-
In the Configuration tab, in Edit mode, Open Advanced Options (at the bottom of the dialog) and then the Spark tab.
-
Add the following content to the Spark Config edit box. For more information on the Spark config properties, click here.
spark.databricks.cluster.profile serverless spark.databricks.isv.product privacera spark.driver.extraJavaOptions -javaagent:/databricks/jars/privacera-agent.jar spark.databricks.repl.allowedLanguages sql,python,r
spark.databricks.cluster.profile serverless spark.databricks.repl.allowedLanguages sql,python,r spark.driver.extraJavaOptions -javaagent:/databricks/jars/ranger-spark-plugin-faccess-2.0.0-SNAPSHOT.jar spark.databricks.isv.product privacera spark.databricks.pyspark.enableProcessIsolation true
Note
-
From Privacera 5.0.6.1 Release onwards, it is recommended to replace the Old Properties with the New Properties. However, the Old Properties will also continue to work.
-
For Databricks versions < 7.3, Old Properties should only be used since the versions are in extended support.
-
-
In the Configuration tab, in Edit mode, Open Advanced Options (at the bottom of the dialog) and then set init script path. For the
<DEPLOYMENT_ENV_NAME>
variable, enter the deployment name as defined for theDEPLOYMENT_ENV_NAME
variable in thevars.privacera.yml
.dbfs:/privacera/<DEPLOYMENT_ENV_NAME>/ranger_enable.sh
-
Save (Confirm) this configuration.
-
Start (or Restart) the selected Databricks Cluster.
Related Information
For further reading, see:
-
To enable view-level access control (via Data-Admin), and view-level row-level filtering and column masking, add the property
DATABRICKS_SPARK_PRIVACERA_VIEW_LEVEL_MASKING_ROWFILTER_EXTENSION_ENABLE: "true"
in custom-vars. Search for this property in Spark Plugin Properties for more information. To learn how to use the property, see Apply View-level Access Control. -
By default, certain python packages are blocked on the Databricks cluster for security compliance. If you still wish to use these packages, see Whitelisting Py4j Packages.
-
If you want to enable JWT-based user authentication for your Databricks clusters, see JWT for Databricks.
-
If you want PM to add cluster policies in Databricks, see Configure Databricks Cluster Policy.
-
If you want to add additional Spark properties for your Databricks cluster, see Spark Properties for Databricks Cluster.
Validation
In order to help evaluate the use of Privacera with Databricks, Privacera provides a set of Privacera Manager 'demo' notebooks. These can be downloaded from Privacera S3 repository using either your favorite browser, or a command line 'wget'. Use the notebook/sql sequence that matches your cluster.
-
Download using your browser (just click on the correct file for your cluster, below:
https://privacera.s3.amazonaws.com/public/pm-demo-data/databricks/PrivaceraSparkPlugin.sql
If AWS S3 is configured from your Databricks cluster: https://privacera.s3.amazonaws.com/public/pm-demo-data/databricks/PrivaceraSparkPluginS3.sql
If ADLS Gen2 is configured from your Databricks cluster: https://privacera.s3.amazonaws.com/public/pm-demo-data/databricks/PrivaceraSparkPluginADLS.sql
or, if you are working from a Linux command line, use the 'wget' command to download.
wget https://privacera.s3.amazonaws.com/public/pm-demo-data/databricks/PrivaceraSparkPlugin.sql -O PrivaceraSparkPlugin.sql
wget https://privacera.s3.amazonaws.com/public/pm-demo-data/databricks/PrivaceraSparkPluginS3.sql -O PrivaceraSparkPluginS3.sql
wget https://privacera.s3.amazonaws.com/public/pm-demo-data/databricks/PrivaceraSparkPluginADLS.sql -O PrivaceraSparkPluginADLS.sql
-
Import the Databricks notebook:
Login to Databricks Console ->
Select Workspace -> Users -> Your User ->
Click on drop down ->
Click on Import and Choose the file downloaded -
Follow the suggested steps in the text of the notebook to exercise and validate Privacera with Databricks.