ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36377][DOCS] Re-document "Options read in YARN client/cluster mode" section in spark-env.sh.template

Browse source 
### What changes were proposed in this pull request?

Edit spark-env.sh.template for better documentation.

### Why are the changes needed?

Some options  (e.g. SPARK_CONF_DIR, SPARK_EXECUTOR_CORES, etc.) are read by any mode but are in the "Options read in YARN client/cluster mode" that might confuse users, so we should separate YARN only options from others.

### Does this PR introduce _any_ user-facing change?

Yes, docs changed.

### How was this patch tested?

Just a manual checking.
<img width="336" alt="options" src="https://user-images.githubusercontent.com/87687356/127804872-02a27337-f304-47df-affa-a8c4e9e98fe5.png">

Closes #33604 from yutoacts/SPARK-36377.

Authored-by: Yuto Akutsu <yuto.akutsu@nttdata.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
Yuto Akutsu 2021-08-10 11:05:39 +09:00 committed by Hyukjin Kwon
parent ab897109a3
commit c040593fb4
1 changed files with 7 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
conf/spark-env.sh.template
View file

@ -32,14 +32,18 @@
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
# - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos
# Options read in YARN client/cluster mode
# Options read in any mode
# - SPARK_CONF_DIR, Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN
# - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Executor (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_DRIVER_MEMORY, Memory for Driver (e.g. 1000M, 2G) (Default: 1G)
# Options read in any cluster manager using HDFS
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# Options read in YARN client/cluster mode
# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN
# Options for the daemons used in the standalone deploy mode
# - SPARK_MASTER_HOST, to bind the master to a different IP address or hostname
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master