ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-21652][SQL] Fix rule confliction between InferFiltersFromConstraints and ConstantPropagation

Browse source 
## What changes were proposed in this pull request?

For the given example below, the predicate added by `InferFiltersFromConstraints` is folded by `ConstantPropagation` later, this leads to unconverged optimize iteration:
```
Seq((1, 1)).toDF("col1", "col2").createOrReplaceTempView("t1")
Seq(1, 2).toDF("col").createOrReplaceTempView("t2")
sql("SELECT * FROM t1, t2 WHERE t1.col1 = 1 AND 1 = t1.col2 AND t1.col1 = t2.col AND t1.col2 = t2.col")
```

We can fix this by adjusting the indent of the optimize rules.

## How was this patch tested?

Add test case that would have failed in `SQLQuerySuite`.

Author: Xingbo Jiang <xingbo.jiang@databricks.com>

Closes #19099 from jiangxb1987/unconverge-optimization.
This commit is contained in:
Xingbo Jiang 2017-09-05 13:12:39 -07:00 committed by gatorsmile
parent 8c954d2cd1
commit fd60d4fa6c
2 changed files with 16 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
View file

@ -79,11 +79,12 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
PushProjectionThroughUnion, PushProjectionThroughUnion,
ReorderJoin, ReorderJoin,
EliminateOuterJoin, EliminateOuterJoin,
InferFiltersFromConstraints,
BooleanSimplification,
PushPredicateThroughJoin, PushPredicateThroughJoin,
PushDownPredicate, PushDownPredicate,
LimitPushDown, LimitPushDown,
ColumnPruning, ColumnPruning,
InferFiltersFromConstraints,
// Operator combine // Operator combine
CollapseRepartition, CollapseRepartition,
CollapseProject, CollapseProject,

14
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file

@ -2663,4 +2663,18 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
// In unit test, Spark will fail the query if memory leak detected. // In unit test, Spark will fail the query if memory leak detected.
spark.range(100).groupBy("id").count().limit(1).collect() spark.range(100).groupBy("id").count().limit(1).collect()
} }
test("SPARK-21652: rule confliction of InferFiltersFromConstraints and ConstantPropagation") {
withTempView("t1", "t2") {
Seq((1, 1)).toDF("col1", "col2").createOrReplaceTempView("t1")
Seq(1, 2).toDF("col").createOrReplaceTempView("t2")
val df = sql(
"""
|SELECT *
|FROM t1, t2
|WHERE t1.col1 = 1 AND 1 = t1.col2 AND t1.col1 = t2.col AND t1.col2 = t2.col
""".stripMargin)
checkAnswer(df, Row(1, 1, 1))
}
}
} }