Skip to content

Spark shell 4.x — JLine Warnings on Console

On Apache Spark 4.x, spark-shell showing repeated console warnings such as:

Text Only
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
WARN org.jline :115 Failed to save history
java.nio.file.AccessDeniedException: /nonexistent
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
    at java.base/java.nio.file.Files.createDirectory(Files.java:700)
    at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
    at java.base/java.nio.file.Files.createDirectories(Files.java:793)
    at org.jline.reader.impl.history.DefaultHistory.internalWrite(DefaultHistory.java:223)
    ...
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

The Scala REPL writes history to $HOME/.scala_history. Spark 4.x ships JLine 3, whose history path uses Files.createDirectories() before writing; with HOME=/nonexistent, that step raises AccessDeniedException. Spark 3.x paths that relied on JLine 2 often failed more quietly for the same HOME value, so the issue tends to surface after upgrading to 4.x even when the image policy is unchanged. This warning does not affect query execution or access management.

Solution

Two approaches are available:

  • Update log4j2.properties
  • Create required directory in the image

Option 1 - Update log4j2.properties

Add the following block to log4j2.properties (Spark 4.x) to reduce repeated JLine console noise. This affects logging only; it does not fix history persistence.

Properties
1
2
3
4
5
6
# Spark 4.x JLine
logger.jline.name = org.jline
logger.jline.level = ERROR
logger.jline.additivity = false
logger.jline.appenderRefs = rolling
logger.jline.appenderRef.rolling.ref = fileLogger

Use the log4j2.properties file that is packaged into the custom image (see Setup).

Option 2 - Create required directory in the image

Create nonexistent directory, as JLine 3 expecting this to write history, Add the following to the Dockerfile

Docker
1
2
3
# Fix for JLine warnings (Spark 4.x shell)
RUN mkdir -p /nonexistent
RUN chown -R ${USER_NAME}:${GROUP_NAME} /nonexistent

Rebuild, push, and redeploy

Both log4j2.properties and Dockerfile changes are applied by building a new image and rolling it out. After saving the edits used in the image build: