3697232b7d
I've updated default values in comments, documentation, and in the command line builder to be 1g based on comments in the JIRA. I've also updated most usages to point at a single variable defined in the Utils.scala and JavaUtils.java files. This wasn't possible in all cases (R, shell scripts etc.) but usage in most code is now pointing at the same place. Please let me know if I've missed anything. Will the spark-shell use the value within the command line builder during instantiation? Author: Ilya Ganelin <ilya.ganelin@capitalone.com> Closes #7132 from ilganeli/SPARK-3071 and squashes the following commits: 4074164 [Ilya Ganelin] String fix 271610b [Ilya Ganelin] Merge branch 'SPARK-3071' of github.com:ilganeli/spark into SPARK-3071 273b6e9 [Ilya Ganelin] Test fix fd67721 [Ilya Ganelin] Update JavaUtils.java 26cc177 [Ilya Ganelin] test fix e5db35d [Ilya Ganelin] Fixed test failure 39732a1 [Ilya Ganelin] merge fix a6f7deb [Ilya Ganelin] Created default value for DRIVER MEM in Utils that's now used in almost all locations instead of setting manually in each 09ad698 [Ilya Ganelin] Update SubmitRestProtocolSuite.scala 19b6f25 [Ilya Ganelin] Missed one doc update 2698a3d [Ilya Ganelin] Updated default value for driver memory
52 lines
3.2 KiB
Bash
Executable file
52 lines
3.2 KiB
Bash
Executable file
#!/usr/bin/env bash
|
||
|
||
# This file is sourced when running various Spark programs.
|
||
# Copy it as spark-env.sh and edit that to configure Spark for your site.
|
||
|
||
# Options read when launching programs locally with
|
||
# ./bin/run-example or ./bin/spark-submit
|
||
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
|
||
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
|
||
# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program
|
||
# - SPARK_CLASSPATH, default classpath entries to append
|
||
|
||
# Options read by executors and drivers running inside the cluster
|
||
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
|
||
# - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program
|
||
# - SPARK_CLASSPATH, default classpath entries to append
|
||
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
|
||
# - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos
|
||
|
||
# Options read in YARN client mode
|
||
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
|
||
# - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2)
|
||
# - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1).
|
||
# - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G)
|
||
# - SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 1G)
|
||
# - SPARK_YARN_APP_NAME, The name of your application (Default: Spark)
|
||
# - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests (Default: ‘default’)
|
||
# - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed with the job.
|
||
# - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job.
|
||
|
||
# Options for the daemons used in the standalone deploy mode
|
||
# - SPARK_MASTER_IP, to bind the master to a different IP address or hostname
|
||
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master
|
||
# - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y")
|
||
# - SPARK_WORKER_CORES, to set the number of cores to use on this machine
|
||
# - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g)
|
||
# - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker
|
||
# - SPARK_WORKER_INSTANCES, to set the number of worker processes per node
|
||
# - SPARK_WORKER_DIR, to set the working directory of worker processes
|
||
# - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")
|
||
# - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y")
|
||
# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y")
|
||
# - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y")
|
||
# - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers
|
||
|
||
# Generic options for the daemons used in the standalone deploy mode
|
||
# - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf)
|
||
# - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs)
|
||
# - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp)
|
||
# - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER)
|
||
# - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)
|