Skip to main content

Privacera Documentation

Add custom Spark configuration for Databricks on Privacera Platform

To add custom properties in the Databricks cluster init script, you need to create a custom configuration file and add your custom spark properties to it.

To do so, follow these steps:

  1. Run the following commands to create a custom configuration file.

    cd ~/privacera/privacera-manager
    vi config/custom-properties/databricks-spark.conf
    
  2. Add the properties one below the other, and then save the file. For example, you can add the following property and save the file:

    "spark.databricks.delta.formatCheck.enabled"="false"
    

    Note

    Avoid putting comments, extra words, or blank lines in the config file.

  3. Run the following command.

    cd ~/privacera/privacera-manager
    ./privacera-manager.sh update
    

Configure Databricks cluster policy

  1. Add the following two properties to vars.databricks.plugin.yml:

    • DATABRICKS_SQL_CLUSTER_POLICY_SPARK_CONF

    • DATABRICKS_SCALA_CLUSTER_POLICY_SPARK_CONF

    For example:

    DATABRICKS_SQL_CLUSTER_POLICY_SPARK_CONF:
    - Note: first spark conf
        key : "spark.hadoop.first.spark.test"
        value: "test1"
    - Note: second spark first spark conf
        key: "spark.hadoop.first.spark.test2"
        value: "test2"
    DATABRICKS_SCALA_CLUSTER_POLICY_SPARK_CONF:
    - Note: first spark conf
        key : "spark.hadoop.first.spark.test"
        value: "test1"
    - Note: second spark first spark conf
        key: "spark.hadoop.first.spark.test2"
        value: "test2"
    
  2. To add custom properties with Java agent, add the following property to vars.databricks.plugin.yml:

    • DATABRICKS_SPARK_PLUGIN_AGENT_JAR

    For example:

    DATABRICKS_SPARK_PLUGIN_AGENT_JAR: " -Dmy.custom.propery=test -javaagent:/databricks/jars/privacera-agent.jar"