Run Spark Session

1. Navigate to ${SPARK_HOME}/bin folder and export the JWT token

   Bash
   1 2	cd <SPARK_HOME>/bin export JWT_TOKEN="<JWT_TOKEN>"

2. Start spark-session (choose one of spark-shell, pyspark, or spark-sql)

   • To pass the JWT token directly as a command-line argument, use the following configuration when connecting to the cluster

     Bash
     1 2	./<spark-shell | pyspark | spark-sql> \ --conf "spark.hadoop.privacera.jwt.token.str=${JWT_TOKEN}"

   • To use the file path containing the JWT token, use the following configuration:

     Bash
     1 2	./<spark-shell | pyspark | spark-sql> \ --conf "spark.hadoop.privacera.jwt.token=<path-to-jwt-token-file>"

Comments