Skip to content

AWS EMR Serverless - Access - Spark OLAC with Delta

To use Delta with EMR Serverless, update the existing Application configuration by adding the following properties under spark-defaults section.

This is just for your reference. You can modify the properties as per your requirement.

Application configuration for OLAC:

Add the following to the spark-defaults classification section in the Application configuration.

JSON
1
2
3
"spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog",
"spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
"spark.jars": "/usr/share/aws/delta/lib/delta-spark.jar,/usr/share/aws/delta/lib/delta-storage.jar"

Application configuration for OLAC_FGAC:

Add the following to the spark-defaults classification section in the Application configuration.

JSON
1
2
3
"spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog",
"spark.sql.extensions": "com.privacera.spark.agent.SparkSQLExtension,io.delta.sql.DeltaSparkSessionExtension",
"spark.jars": "/usr/share/aws/delta/lib/delta-spark.jar,/usr/share/aws/delta/lib/delta-storage.jar"

Comments