[SPARK-8687] [YARN] Fix bug: Executor can't fetch the new set configuration in yarn-client

Spark initi the properties CoarseGrainedSchedulerBackend.start
```scala
    // TODO (prashant) send conf instead of properties
    driverEndpoint = rpcEnv.setupEndpoint(
      CoarseGrainedSchedulerBackend.ENDPOINT_NAME, new DriverEndpoint(rpcEnv, properties))
```
Then the yarn logic will set some configuration but not update in this `properties`.
So `Executor` won't gain the `properties`.

[Jira](https://issues.apache.org/jira/browse/SPARK-8687)

Author: huangzhaowei <carlmartinmax@gmail.com>

Closes #7066 from SaintBacchus/SPARK-8687 and squashes the following commits:

1de4f48 [huangzhaowei] Ensure all necessary properties have already been set before startup ExecutorLaucher

(cherry picked from commit 1b0c8e6104)
Signed-off-by: Andrew Or <andrew@databricks.com>
This commit is contained in:
huangzhaowei 2015-07-01 23:14:13 -07:00 committed by Andrew Or
parent e33c0f0a49
commit 7cbfef23aa

View file

@ -41,7 +41,6 @@ private[spark] class YarnClientSchedulerBackend(
* This waits until the application is running.
*/
override def start() {
super.start()
val driverHost = conf.get("spark.driver.host")
val driverPort = conf.get("spark.driver.port")
val hostport = driverHost + ":" + driverPort
@ -56,6 +55,12 @@ private[spark] class YarnClientSchedulerBackend(
totalExpectedExecutors = args.numExecutors
client = new Client(args, conf)
appId = client.submitApplication()
// SPARK-8687: Ensure all necessary properties have already been set before
// we initialize our driver scheduler backend, which serves these properties
// to the executors
super.start()
waitForApplication()
monitorThread = asyncMonitorApplication()
monitorThread.start()