ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-19688][STREAMING] Not to read spark.yarn.credentials.file from checkpoint.

Browse source 
## What changes were proposed in this pull request?

Reload the `spark.yarn.credentials.file` property when restarting a streaming application from checkpoint.

## How was this patch tested?

Manual tested with 1.6.3 and 2.1.1.
I didn't test this with master because of some compile problems, but I think it will be the same result.

## Notice

This should be merged into maintenance branches too.

jira: [SPARK-21008](https://issues.apache.org/jira/browse/SPARK-21008)

Author: saturday_s <shi.indetail@gmail.com>

Closes #18230 from saturday-shi/SPARK-21008.
This commit is contained in:
saturday_s 2017-06-19 10:24:29 -07:00 committed by Marcelo Vanzin
parent 9a145fd796
commit e92ffe6f17
1 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
View file

@ -55,6 +55,9 @@ class Checkpoint(ssc: StreamingContext, val checkpointTime: Time)
"spark.master", "spark.master",
"spark.yarn.keytab", "spark.yarn.keytab",
"spark.yarn.principal", "spark.yarn.principal",
"spark.yarn.credentials.file",
"spark.yarn.credentials.renewalTime",
"spark.yarn.credentials.updateTime",
"spark.ui.filters") "spark.ui.filters")
val newSparkConf = new SparkConf(loadDefaults = false).setAll(sparkConfPairs) val newSparkConf = new SparkConf(loadDefaults = false).setAll(sparkConfPairs)