[SPARK-8687] [YARN] Fix bug: Executor can't fetch the new set configuration in yarn-client
Spark initi the properties CoarseGrainedSchedulerBackend.start
```scala
// TODO (prashant) send conf instead of properties
driverEndpoint = rpcEnv.setupEndpoint(
CoarseGrainedSchedulerBackend.ENDPOINT_NAME, new DriverEndpoint(rpcEnv, properties))
```
Then the yarn logic will set some configuration but not update in this `properties`.
So `Executor` won't gain the `properties`.
[Jira](https://issues.apache.org/jira/browse/SPARK-8687)
Author: huangzhaowei <carlmartinmax@gmail.com>
Closes #7066 from SaintBacchus/SPARK-8687 and squashes the following commits:
1de4f48 [huangzhaowei] Ensure all necessary properties have already been set before startup ExecutorLaucher
(cherry picked from commit 1b0c8e6104
)
Signed-off-by: Andrew Or <andrew@databricks.com>
This commit is contained in:
parent
e33c0f0a49
commit
7cbfef23aa
|
@ -41,7 +41,6 @@ private[spark] class YarnClientSchedulerBackend(
|
|||
* This waits until the application is running.
|
||||
*/
|
||||
override def start() {
|
||||
super.start()
|
||||
val driverHost = conf.get("spark.driver.host")
|
||||
val driverPort = conf.get("spark.driver.port")
|
||||
val hostport = driverHost + ":" + driverPort
|
||||
|
@ -56,6 +55,12 @@ private[spark] class YarnClientSchedulerBackend(
|
|||
totalExpectedExecutors = args.numExecutors
|
||||
client = new Client(args, conf)
|
||||
appId = client.submitApplication()
|
||||
|
||||
// SPARK-8687: Ensure all necessary properties have already been set before
|
||||
// we initialize our driver scheduler backend, which serves these properties
|
||||
// to the executors
|
||||
super.start()
|
||||
|
||||
waitForApplication()
|
||||
monitorThread = asyncMonitorApplication()
|
||||
monitorThread.start()
|
||||
|
|
Loading…
Reference in a new issue