Skip to main content

Privacera Documentation

Table of ContentsTable of Contents

Add custom Spark configuration for Databricks on Privacera Platform

To add custom properties in the Databricks cluster init script, you need to create a custom configuration file and add your custom spark properties to it.

To do so, follow these steps:

  1. Run the following commands to create a custom configuration file.

    cd ~/privacera/privacera-manager
vi config/custom-properties/databricks-spark.conf

  2. Add the properties one below the other, and then save the file. For example, you can add the following property and save the file:

    "spark.databricks.delta.formatCheck.enabled"="false"

    Note

    Avoid putting comments, extra words, or blank lines in the config file.

  3. Run the following command.

    cd ~/privacera/privacera-manager
./privacera-manager.sh update

Configure Databricks cluster policy

  1. Add the following two properties to vars.databricks.plugin.yml:

    • DATABRICKS_SQL_CLUSTER_POLICY_SPARK_CONF

    • DATABRICKS_SCALA_CLUSTER_POLICY_SPARK_CONF

    For example:

    DATABRICKS_SQL_CLUSTER_POLICY_SPARK_CONF:
- Note: first spark conf
    key : "spark.hadoop.first.spark.test"
    value: "test1"
- Note: second spark first spark conf
    key: "spark.hadoop.first.spark.test2"
    value: "test2"
DATABRICKS_SCALA_CLUSTER_POLICY_SPARK_CONF:
- Note: first spark conf
    key : "spark.hadoop.first.spark.test"
    value: "test1"
- Note: second spark first spark conf
    key: "spark.hadoop.first.spark.test2"
    value: "test2"

  2. To add custom properties with Java agent, add the following property to vars.databricks.plugin.yml:

    • DATABRICKS_SPARK_PLUGIN_AGENT_JAR

    For example:

    DATABRICKS_SPARK_PLUGIN_AGENT_JAR: " -Dmy.custom.propery=test -javaagent:/databricks/jars/privacera-agent.jar"