Run Spark Session

  1. Navigate to ${SPARK_HOME}/bin folder and export the JWT token

    Bash
    1
2
    cd <SPARK_HOME>/bin
export JWT_TOKEN="<JWT_TOKEN>"

  2. Start spark-session (choose one of spark-shell, pyspark, or spark-sql)

    • To pass the JWT token directly as a command-line argument, use the following configuration when connecting to the cluster

      Bash
      1
2
      ./<spark-shell | pyspark | spark-sql> \
--conf "spark.hadoop.privacera.jwt.token.str=${JWT_TOKEN}"

    • To use the file path containing the JWT token, use the following configuration:

      Bash
      1
2
      ./<spark-shell | pyspark | spark-sql> \
--conf "spark.hadoop.privacera.jwt.token=<path-to-jwt-token-file>"