Run Spark Session with Executor

  1. SSH into driver pod of the Namespace

  2. Export the following variables

    Bash
    1
2
3
4
5
6
7
    SPARK_NAME_SPACE=<SPARK_NAME_SPACE>
SPARK_IMAGE=<SPARK_IMAGE>

SVC_ACCOUNT=privacera-sa-spark-plugin
K8S_MASTER=k8s://https://kubernetes.default.svc

export JWT_TOKEN="<JWT_TOKEN>"

  3. Run the following command to start spark session with executors

    Bash
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    /opt/spark/bin/spark-shell \
  --master ${K8S_MASTER} \
  --deploy-mode client \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=${SVC_ACCOUNT} \
  --conf spark.kubernetes.namespace=${SPARK_NAME_SPACE} \
  --conf spark.kubernetes.driver.request.cores=0.001 \
  --conf spark.driver.memory=1g \
  --conf spark.kubernetes.executor.request.cores=0.001 \
  --conf spark.executor.memory=1g \
  --conf spark.kubernetes.container.image=${SPARK_IMAGE} \
  --conf spark.kubernetes.container.image.pullPolicy=Always \
  --conf spark.driver.host=${SPARK_PLUGIN_POD_IP} \
  --conf spark.driver.port=7077 \
  --conf spark.blockManager.port=7078 \
  --conf spark.kubernetes.executor.secrets.privacera-spark-secret=/privacera-secret \
  --conf "spark.hadoop.privacera.jwt.token.str=${JWT_TOKEN}"

Comments