ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-11877] Prevent agg. fallback conf. from leaking across test suites

Browse source 
This patch fixes an issue where the `spark.sql.TungstenAggregate.testFallbackStartsAt` SQLConf setting was not properly reset / cleared at the end of `TungstenAggregationQueryWithControlledFallbackSuite`. This ended up causing test failures in HiveCompatibilitySuite in Maven builds by causing spilling to occur way too frequently.

This configuration leak was inadvertently introduced during test cleanup in #9618.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #9857 from JoshRosen/clear-fallback-prop-in-test-teardown.
This commit is contained in:
Josh Rosen 2015-11-20 00:46:29 -08:00
parent 3e1d120ced
commit a66142dece
1 changed files with 19 additions and 21 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

40
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala
View file

@ -868,29 +868,27 @@ class TungstenAggregationQueryWithControlledFallbackSuite extends AggregationQue
override protected def checkAnswer(actual: => DataFrame, expectedAnswer: Seq[Row]): Unit = {
(0 to 2).foreach { fallbackStartsAt =>
sqlContext.setConf(
"spark.sql.TungstenAggregate.testFallbackStartsAt",
fallbackStartsAt.toString)
withSQLConf("spark.sql.TungstenAggregate.testFallbackStartsAt" -> fallbackStartsAt.toString) {
// Create a new df to make sure its physical operator picks up
// spark.sql.TungstenAggregate.testFallbackStartsAt.
// todo: remove it?
val newActual = DataFrame(sqlContext, actual.logicalPlan)
// Create a new df to make sure its physical operator picks up
// spark.sql.TungstenAggregate.testFallbackStartsAt.
// todo: remove it?
val newActual = DataFrame(sqlContext, actual.logicalPlan)
QueryTest.checkAnswer(newActual, expectedAnswer) match {
case Some(errorMessage) =>
val newErrorMessage =
s"""
|The following aggregation query failed when using TungstenAggregate with
|controlled fallback (it falls back to sort-based aggregation once it has processed
|$fallbackStartsAt input rows). The query is
|${actual.queryExecution}
|
|$errorMessage
""".stripMargin
QueryTest.checkAnswer(newActual, expectedAnswer) match {
case Some(errorMessage) =>
val newErrorMessage =
s"""
|The following aggregation query failed when using TungstenAggregate with
|controlled fallback (it falls back to sort-based aggregation once it has processed
|$fallbackStartsAt input rows). The query is
|${actual.queryExecution}
|
|$errorMessage
""".stripMargin
fail(newErrorMessage)
case None =>
fail(newErrorMessage)
case None =>
}
}
}
}