ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-18027][YARN] .sparkStaging not clean on RM ApplicationNotFoundException

Browse source 
## What changes were proposed in this pull request?

Cleanup YARN staging dir on all `KILLED`/`FAILED` paths in `monitorApplication`

## How was this patch tested?

Existing tests

Author: Sean Owen <sowen@cloudera.com>

Closes #15598 from srowen/SPARK-18027.
This commit is contained in:
Sean Owen 2016-10-26 14:23:11 +02:00
parent 6c7d094ec4
commit 2978136475
No known key found for this signature in database
GPG key ID: BEB3956D6717BDDC
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
View file

@ -1059,9 +1059,11 @@ private[spark] class Client(
} catch {
case e: ApplicationNotFoundException =>
logError(s"Application $appId not found.")
cleanupStagingDir(appId)
return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED)
case NonFatal(e) =>
logError(s"Failed to contact YARN for application $appId.", e)
// Don't necessarily clean up staging dir because status is unknown
return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED)
}
val state = report.getYarnApplicationState