ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/conf
History
guoxiaolong bdcd6e4c68 [SPARK-20995][CORE] Spark-env.sh.template' should add 'YARN_CONF_DIR' configuration instructions. 
## What changes were proposed in this pull request?

Ensure that `HADOOP_CONF_DIR` or `YARN_CONF_DIR` points to the directory which contains the (client side) configuration files for the Hadoop cluster.
These configs are used to write to HDFS and connect to the YARN ResourceManager. The
configuration contained in this directory will be distributed to the YARN cluster so that all
containers used by the application use the same configuration.

Sometimes, `HADOOP_CONF_DIR` is set to the hdfs configuration file path. So, YARN_CONF_DIR should be set to the yarn configuration file path.

My project configuration item of 'spark-env.sh ' is as follows:
![1](https://cloud.githubusercontent.com/assets/26266482/26819987/d4acb814-4ad3-11e7-8458-a21aea57a53d.png)

'HADOOP_CONF_DIR' configuration file path. List the relevant documents below:
![3](https://cloud.githubusercontent.com/assets/26266482/26820116/47b6b9fe-4ad4-11e7-8131-fe07c8d8bc21.png)

'YARN_CONF_DIR' configuration file path. List the relevant documents below:
![2](https://cloud.githubusercontent.com/assets/26266482/26820078/274ad79a-4ad4-11e7-83d4-ff359dbb397c.png)

So, 'Spark-env.sh.template' should add 'YARN_CONF_DIR' configuration instructions.

## How was this patch tested?

manual tests

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: guoxiaolong <guo.xiaolong1@zte.com.cn>
Author: 郭小龙 10207633 <guo.xiaolong1@zte.com.cn>
Author: guoxiaolongzte <guo.xiaolong1@zte.com.cn>

Closes #18212 from guoxiaolongzte/SPARK-20995.
2017-06-09 09:26:30 +01:00
..
docker.properties.template [SPARK-20781] the location of Dockerfile in docker.properties.templat is wrong 2017-05-19 20:47:30 +01:00
fairscheduler.xml.template [SPARK-10718] [BUILD] Update License on conf files and corresponding excludes file update 2015-09-22 11:03:21 +01:00
log4j.properties.template [SPARK-14134][CORE] Change the package name used for shading classes. 2016-04-06 19:33:51 -07:00
metrics.properties.template [SPARK-13238][CORE] Add ganglia dmax parameter 2016-08-05 13:07:52 -07:00
slaves.template [SPARK-10718] [BUILD] Update License on conf files and corresponding excludes file update 2015-09-22 11:03:21 +01:00
spark-defaults.conf.template [SPARK-10718] [BUILD] Update License on conf files and corresponding excludes file update 2015-09-22 11:03:21 +01:00
spark-env.sh.template [SPARK-20995][CORE] Spark-env.sh.template' should add 'YARN_CONF_DIR' configuration instructions. 2017-06-09 09:26:30 +01:00