[SPARK-11877] Prevent agg. fallback conf. from leaking across test suites
This patch fixes an issue where the `spark.sql.TungstenAggregate.testFallbackStartsAt` SQLConf setting was not properly reset / cleared at the end of `TungstenAggregationQueryWithControlledFallbackSuite`. This ended up causing test failures in HiveCompatibilitySuite in Maven builds by causing spilling to occur way too frequently. This configuration leak was inadvertently introduced during test cleanup in #9618. Author: Josh Rosen <joshrosen@databricks.com> Closes #9857 from JoshRosen/clear-fallback-prop-in-test-teardown.
This commit is contained in:
parent
3e1d120ced
commit
a66142dece
|
@ -868,29 +868,27 @@ class TungstenAggregationQueryWithControlledFallbackSuite extends AggregationQue
|
||||||
|
|
||||||
override protected def checkAnswer(actual: => DataFrame, expectedAnswer: Seq[Row]): Unit = {
|
override protected def checkAnswer(actual: => DataFrame, expectedAnswer: Seq[Row]): Unit = {
|
||||||
(0 to 2).foreach { fallbackStartsAt =>
|
(0 to 2).foreach { fallbackStartsAt =>
|
||||||
sqlContext.setConf(
|
withSQLConf("spark.sql.TungstenAggregate.testFallbackStartsAt" -> fallbackStartsAt.toString) {
|
||||||
"spark.sql.TungstenAggregate.testFallbackStartsAt",
|
// Create a new df to make sure its physical operator picks up
|
||||||
fallbackStartsAt.toString)
|
// spark.sql.TungstenAggregate.testFallbackStartsAt.
|
||||||
|
// todo: remove it?
|
||||||
|
val newActual = DataFrame(sqlContext, actual.logicalPlan)
|
||||||
|
|
||||||
// Create a new df to make sure its physical operator picks up
|
QueryTest.checkAnswer(newActual, expectedAnswer) match {
|
||||||
// spark.sql.TungstenAggregate.testFallbackStartsAt.
|
case Some(errorMessage) =>
|
||||||
// todo: remove it?
|
val newErrorMessage =
|
||||||
val newActual = DataFrame(sqlContext, actual.logicalPlan)
|
s"""
|
||||||
|
|The following aggregation query failed when using TungstenAggregate with
|
||||||
|
|controlled fallback (it falls back to sort-based aggregation once it has processed
|
||||||
|
|$fallbackStartsAt input rows). The query is
|
||||||
|
|${actual.queryExecution}
|
||||||
|
|
|
||||||
|
|$errorMessage
|
||||||
|
""".stripMargin
|
||||||
|
|
||||||
QueryTest.checkAnswer(newActual, expectedAnswer) match {
|
fail(newErrorMessage)
|
||||||
case Some(errorMessage) =>
|
case None =>
|
||||||
val newErrorMessage =
|
}
|
||||||
s"""
|
|
||||||
|The following aggregation query failed when using TungstenAggregate with
|
|
||||||
|controlled fallback (it falls back to sort-based aggregation once it has processed
|
|
||||||
|$fallbackStartsAt input rows). The query is
|
|
||||||
|${actual.queryExecution}
|
|
||||||
|
|
|
||||||
|$errorMessage
|
|
||||||
""".stripMargin
|
|
||||||
|
|
||||||
fail(newErrorMessage)
|
|
||||||
case None =>
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
Loading…
Reference in a new issue