Skip to content

Troubleshooting for Access Management for Apache Spark OLAC

Steps to Retrieve Logs for Troubleshooting

To enable debug for Apache Spark OLAC, before executing the build_spark_plugin.sh script, update the log4j2.properties file to enable debug logging.

  1. Navigate to the oss-plugin/config plugin directory:

    Bash
    1
    cd ~privacera-oss-plugin/config

  2. Open the log4j2.properties file and update the following property:

    Bash
    1
2
3
    vi log4j2.properties

logger.privacera.level = debug

  3. Once the pod is up, navigate to the ~/tmp/<user> folder.

    Bash
    1
    cd ~/tmp/<user>

  4. Logs will be saved in file privacera.log

  5. Command to get logs on machine:

    Bash
    1
    kubectl cp <SPARK_NAME_SPACE>/<SPARK_PLUGIN_POD_IP>:/tmp/<user>/privacera.log <destination_path>

Configuration Issues Causing Access Control Failures in Apache Spark Jobs and Spark Operator Deployments

This guide helps you troubleshoot configuration issues that prevent Privacera from enforcing access control in Apache Spark jobs and deployments using Spark Operator.

Overwriting Java Options in Spark Job Code

Problem

Setting spark.executor.extraJavaOptions or spark.driver.extraJavaOptions directly in your Spark job code (for example, in Python or Scala) overrides the default Privacera agent configuration. As a result, the Privacera agent may fail to load correctly.

Default Configuration

By default, the following properties are defined in /opt/spark/conf/spark-defaults.conf:

spark-defaults.conf
spark.executor.extraJavaOptions -javaagent:/opt/spark/jars/privacera-agent.jar -Dlog4j.configurationFile=file:///privacera-conf/log4j2.properties

spark.driver.extraJavaOptions -javaagent:/opt/spark/jars/privacera-agent.jar -Dlog4j.configurationFile=file:///privacera-conf/log4j2.properties

Incorrect Configuration Example

Instead of overwriting these properties, append your custom Java options to the existing Privacera configuration. This ensures that the Privacera agent settings are preserved. For example, in Python:

Python
1
2
3
4
spark = SparkSession.builder \
    .config("spark.executor.extraJavaOptions", "<PROPERTY>") \
    .config("spark.driver.extraJavaOptions", "<PROPERTY>") \
    .getOrCreate()

Impact

  • The Privacera agent and required JARs are not loaded on the Spark driver or executors.
  • The Spark job may still succeed; however, Privacera will not be able to enforce access control.

Solution

Instead of overwriting these properties, append your options to the existing Privacera configuration. For example, in Python:

Python
1
2
3
4
spark = SparkSession.builder \
    .config("spark.executor.extraJavaOptions", "-javaagent:/opt/spark/jars/privacera-agent.jar -Dlog4j.configurationFile=file:///privacera-conf/log4j2.properties <your-additional-options>") \
    .config("spark.driver.extraJavaOptions", "-javaagent:/opt/spark/jars/privacera-agent.jar -Dlog4j.configurationFile=file:///privacera-conf/log4j2.properties <your-additional-options>") \
    .getOrCreate()