[SPARK-1896] Respect spark.master (and --master) before MASTER in spark-shell
The hierarchy for configuring the Spark master in the shell is as follows: ``` MASTER > --master > spark.master (spark-defaults.conf) ``` This is inconsistent with the way we run normal applications, which is: ``` --master > spark.master (spark-defaults.conf) > MASTER ``` I was trying to run a shell locally on a standalone cluster launched through the ec2 scripts, which automatically set `MASTER` in spark-env.sh. It was surprising to me that `--master` didn't take effect, considering that this is the way we tell users to set their masters [here](http://people.apache.org/~pwendell/spark-1.0.0-rc7-docs/scala-programming-guide.html#initializing-spark). Author: Andrew Or <andrewor14@gmail.com> Closes #846 from andrewor14/shell-master and squashes the following commits: 2cb81c9 [Andrew Or] Respect spark.master before MASTER in REPL
This commit is contained in:
parent
8edbee7d1b
commit
cce77457e0
|
@ -962,11 +962,10 @@ class SparkILoop(in0: Option[BufferedReader], protected val out: JPrintWriter,
|
|||
private def getMaster(): String = {
|
||||
val master = this.master match {
|
||||
case Some(m) => m
|
||||
case None => {
|
||||
case None =>
|
||||
val envMaster = sys.env.get("MASTER")
|
||||
val propMaster = sys.props.get("spark.master")
|
||||
envMaster.orElse(propMaster).getOrElse("local[*]")
|
||||
}
|
||||
propMaster.orElse(envMaster).getOrElse("local[*]")
|
||||
}
|
||||
master
|
||||
}
|
||||
|
|
Loading…
Reference in a new issue