ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-23021][SQL] AnalysisBarrier should override innerChildren to print correct explain output

Browse source 
## What changes were proposed in this pull request?
`AnalysisBarrier` in the current master cuts off explain results for parsed logical plans;
```
scala> Seq((1, 1)).toDF("a", "b").groupBy("a").count().sample(0.1).explain(true)
== Parsed Logical Plan ==
Sample 0.0, 0.1, false, -7661439431999668039
+- AnalysisBarrier Aggregate [a#5], [a#5, count(1) AS count#14L]
```
To fix this, `AnalysisBarrier` needs to override `innerChildren` and this pr changed the output to;
```
== Parsed Logical Plan ==
Sample 0.0, 0.1, false, -5086223488015741426
+- AnalysisBarrier
      +- Aggregate [a#5], [a#5, count(1) AS count#14L]
         +- Project [_1#2 AS a#5, _2#3 AS b#6]
            +- LocalRelation [_1#2, _2#3]
```

## How was this patch tested?
Added tests in `DataFrameSuite`.

Author: Takeshi Yamamuro <yamamuro@apache.org>

Closes #20247 from maropu/SPARK-23021-2.
This commit is contained in:
Takeshi Yamamuro 2018-01-14 22:26:21 +08:00 committed by gatorsmile
parent 66738d29c5
commit 990f05c803
2 changed files with 18 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
View file

@ -903,6 +903,7 @@ case class Deduplicate(
* This analysis barrier will be removed at the end of analysis stage.
*/
case class AnalysisBarrier(child: LogicalPlan) extends LeafNode {
override protected def innerChildren: Seq[LogicalPlan] = Seq(child)
override def output: Seq[Attribute] = child.output
override def isStreaming: Boolean = child.isStreaming
override def doCanonicalize(): LogicalPlan = child.canonicalized

17
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveExplainSuite.scala
View file

@ -171,4 +171,21 @@ class HiveExplainSuite extends QueryTest with SQLTestUtils with TestHiveSingleto
sql("EXPLAIN EXTENDED CODEGEN SELECT 1")
}
}
test("SPARK-23021 AnalysisBarrier should not cut off explain output for parsed logical plans") {
val df = Seq((1, 1)).toDF("a", "b").groupBy("a").count().limit(1)
val outputStream = new java.io.ByteArrayOutputStream()
Console.withOut(outputStream) {
df.explain(true)
}
assert(outputStream.toString.replaceAll("""#\d+""", "#0").contains(
s"""== Parsed Logical Plan ==
|GlobalLimit 1
|+- LocalLimit 1
| +- AnalysisBarrier
| +- Aggregate [a#0], [a#0, count(1) AS count#0L]
| +- Project [_1#0 AS a#0, _2#0 AS b#0]
| +- LocalRelation [_1#0, _2#0]
|""".stripMargin))
}
}