ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-27486][CORE][TEST] Enable History server storage information test in the HistoryServerSuite

Browse source 
## What changes were proposed in this pull request?

We have disabled a test related to storage in the History server suite after SPARK-13845. But, after SPARK-22050, we can store the information about block updated events to eventLog, if we enable "spark.eventLog.logBlockUpdates.enabled=true".

   So, we can enable the test, by adding an eventlog corresponding to the application, which has enabled the configuration, "spark.eventLog.logBlockUpdates.enabled=true"

## How was this patch tested?
Existing UTs

Closes #24390 from shahidki31/enableRddStorageTest.

Authored-by: Shahid <shahidki31@gmail.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
This commit is contained in:
Shahid 2019-04-19 08:12:20 -07:00 committed by Sean Owen
parent 31488e1ca5
commit 16bbe0f798
2 changed files with 15 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json Normal file
View file

@ -0,0 +1,10 @@
{
"id" : 0,
"name" : "0",
"numPartitions" : 8,
"numCachedPartitions" : 0,
"storageLevel" : "Memory Deserialized 1x Replicated",
"memoryUsed" : 0,
"diskUsed" : 0,
"partitions" : [ ]
}

8
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
View file

@ -174,9 +174,11 @@ class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with Matchers
"executor node blacklisting unblacklisting" -> "applications/app-20161115172038-0000/executors",
"executor memory usage" -> "applications/app-20161116163331-0000/executors",
"app environment" -> "applications/app-20161116163331-0000/environment"
// Todo: enable this test when logging the even of onBlockUpdated. See: SPARK-13845
// "one rdd storage json" -> "applications/local-1422981780767/storage/rdd/0"
"app environment" -> "applications/app-20161116163331-0000/environment",
// Enable "spark.eventLog.logBlockUpdates.enabled", to get the storage information
// in the history server.
"one rdd storage json" -> "applications/local-1422981780767/storage/rdd/0"
)
// run a bunch of characterization tests -- just verify the behavior is the same as what is saved