ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-9522] [SQL] SparkSubmit process can not exit if kill application when HiveThriftServer was starting

Browse source 
When we start HiveThriftServer, we will start SparkContext first, then start HiveServer2, if we kill application while HiveServer2 is starting then SparkContext will stop successfully, but SparkSubmit process can not exit.

Author: linweizhong <linweizhong@huawei.com>

Closes #7853 from Sephiroth-Lin/SPARK-9522.
This commit is contained in:
linweizhong 2015-09-17 22:25:24 -07:00 committed by Yin Huai
parent d009da2f5c
commit 93c7650ab6
2 changed files with 7 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
core/src/main/scala/org/apache/spark/SparkContext.scala
View file

@ -97,7 +97,7 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
val startTime = System.currentTimeMillis()
private val stopped: AtomicBoolean = new AtomicBoolean(false)
private[spark] val stopped: AtomicBoolean = new AtomicBoolean(false)
private def assertNotStopped(): Unit = {
if (stopped.get()) {

6
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala
View file

@ -93,6 +93,12 @@ object HiveThriftServer2 extends Logging {
} else {
None
}
// If application was killed before HiveThriftServer2 start successfully then SparkSubmit
// process can not exit, so check whether if SparkContext was stopped.
if (SparkSQLEnv.sparkContext.stopped.get()) {
logError("SparkContext has stopped even if HiveServer2 has started, so exit")
System.exit(-1)
}
} catch {
case e: Exception =>
logError("Error starting HiveThriftServer2", e)