8615f354a4
### What changes were proposed in this pull request? This is an improvement, we mount all the user specific configuration files(except the templates and spark properties files) from `SPARK_CONF_DIR` at the point of spark-submit, to both executor and driver pods. Currently, only `spark.properties` is mounted, only on driver. ### Why are the changes needed? `SPARK_CONF_DIR` hosts several configuration files, for example, 1) `spark-defaults.conf` - containing all the spark properties. 2) `log4j.properties` - Logger configuration. 3) `core-site.xml` - Hadoop related configuration. 4) `fairscheduler.xml` - Spark's fair scheduling policy at the job level. 5) `metrics.properties` - Spark metrics. 6) Any user specific - library or framework specific configuration file. At the moment, we can cannot propagate these files to the driver and executor configuration directory. There is a design doc, with more details, and this patch is currently providing a reference implementation. Please take a look at the doc and comment, how we can improve. [google docs link to the doc](https://bit.ly/spark-30985) ### Further scope Support user defined configMaps. ### Does this PR introduce any user-facing change? Yes, previously the user configuration files(e.g. hdfs-site.xml, log4j.properties etc...) were not propagated by default, now after this patch it is propagated to driver and executor pods' `SPARK_CONF_DIR`. ### How was this patch tested? Added tests. Also manually tested, by deploying it to a minikube cluster and observing the additional configuration files were present, and taking effect. For example, changes to log4j.properties was properly applied to executors. Closes #27735 from ScrapCodes/SPARK-30985/spark-conf-k8s-propagate. Authored-by: Prashant Sharma <prashsh1@in.ibm.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
---|---|---|
.. | ||
kubernetes | ||
mesos | ||
yarn |