ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting killed on yarn restart.

Browse source 
[SPARK-5613] Added a  catch block to catch the ApplicationNotFoundException. Without this catch block the thread gets killed on occurrence of this exception. This Exception occurs when yarn restarts and tries to find an application id for a spark job which got interrupted due to yarn getting stopped.
See the stacktrace in the bug for more details.

Author: Kashish Jain <kashish.jain@guavus.com>

Closes #4392 from kasjain/branch-1.2 and squashes the following commits:

4831000 [Kashish Jain] SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting killed on yarn restart.
This commit is contained in:
Kashish Jain 2015-02-06 13:47:23 -08:00 committed by Andrew Or
parent b3872e00d1
commit ca66159a4f
1 changed files with 9 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
View file

@ -20,6 +20,7 @@ package org.apache.spark.scheduler.cluster
import scala.collection.mutable.ArrayBuffer
import org.apache.hadoop.yarn.api.records.{ApplicationId, YarnApplicationState}
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException
import org.apache.spark.{SparkException, Logging, SparkContext}
import org.apache.spark.deploy.yarn.{Client, ClientArguments}
@ -133,8 +134,14 @@ private[spark] class YarnClientSchedulerBackend(
val t = new Thread {
override def run() {
while (!stopping) {
var state: YarnApplicationState = null
try {
val report = client.getApplicationReport(appId)
val state = report.getYarnApplicationState()
state = report.getYarnApplicationState()
} catch {
case e: ApplicationNotFoundException =>
state = YarnApplicationState.KILLED
}
if (state == YarnApplicationState.FINISHED ||
state == YarnApplicationState.KILLED ||
state == YarnApplicationState.FAILED) {