From c040593fb432f75f7e22e3d32a6979142a125375 Mon Sep 17 00:00:00 2001 From: Yuto Akutsu Date: Tue, 10 Aug 2021 11:05:39 +0900 Subject: [PATCH] [SPARK-36377][DOCS] Re-document "Options read in YARN client/cluster mode" section in spark-env.sh.template ### What changes were proposed in this pull request? Edit spark-env.sh.template for better documentation. ### Why are the changes needed? Some options (e.g. SPARK_CONF_DIR, SPARK_EXECUTOR_CORES, etc.) are read by any mode but are in the "Options read in YARN client/cluster mode" that might confuse users, so we should separate YARN only options from others. ### Does this PR introduce _any_ user-facing change? Yes, docs changed. ### How was this patch tested? Just a manual checking. options Closes #33604 from yutoacts/SPARK-36377. Authored-by: Yuto Akutsu Signed-off-by: Hyukjin Kwon --- conf/spark-env.sh.template | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/conf/spark-env.sh.template b/conf/spark-env.sh.template index 3c003f45ed..a2f1380692 100755 --- a/conf/spark-env.sh.template +++ b/conf/spark-env.sh.template @@ -32,14 +32,18 @@ # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data # - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos -# Options read in YARN client/cluster mode +# Options read in any mode # - SPARK_CONF_DIR, Alternate conf dir. (Default: ${SPARK_HOME}/conf) -# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files -# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN # - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1). # - SPARK_EXECUTOR_MEMORY, Memory per Executor (e.g. 1000M, 2G) (Default: 1G) # - SPARK_DRIVER_MEMORY, Memory for Driver (e.g. 1000M, 2G) (Default: 1G) +# Options read in any cluster manager using HDFS +# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files + +# Options read in YARN client/cluster mode +# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN + # Options for the daemons used in the standalone deploy mode # - SPARK_MASTER_HOST, to bind the master to a different IP address or hostname # - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master