ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36020][SQL][FOLLOWUP] RemoveRedundantProjects should retain the LOGICAL_PLAN_TAG tag

Browse source 
### What changes were proposed in this pull request?

This is a followup of https://github.com/apache/spark/pull/33222 .

https://github.com/apache/spark/pull/33222 made a mistake that, `RemoveRedundantProjects` may lose the `LOGICAL_PLAN_TAG` tag, even though the logical plan link is retained. This was actually caught by the test `LogicalPlanTagInSparkPlanSuite`, but was not being taken care of.

There is no problem so far, but losing information can always lead to potential bugs.

### Why are the changes needed?

fix a mistake

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

existing test

Closes #33442 from cloud-fan/minor.

Authored-by: Wenchen Fan <wenchen@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This commit is contained in:
Wenchen Fan 2021-07-21 14:03:06 +08:00
parent efcce23b91
commit 94aece4325
2 changed files with 6 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantProjects.scala
View file

@ -49,7 +49,11 @@ object RemoveRedundantProjects extends Rule[SparkPlan] {
plan match {
case p @ ProjectExec(_, child) =>
if (isRedundant(p, child, requireOrdering) && canRemove(p, child)) {
removeProject(child, requireOrdering)
val newPlan = removeProject(child, requireOrdering)
// The `newPlan` should retain the logical plan link already. We call `setLogicalLink`
// here to make sure the `newPlan` sets the `LOGICAL_PLAN_TAG` tag.
newPlan.setLogicalLink(child.logicalLink.get)
newPlan
} else {
p.mapChildren(removeProject(_, false))
}

2
sql/core/src/test/scala/org/apache/spark/sql/execution/LogicalPlanTagInSparkPlanSuite.scala
View file

@ -131,7 +131,7 @@ class LogicalPlanTagInSparkPlanSuite extends TPCDSQuerySuite with DisableAdaptiv
}
private def getLogicalPlan(node: SparkPlan): LogicalPlan = {
node.logicalLink.getOrElse {
node.getTagValue(SparkPlan.LOGICAL_PLAN_TAG).getOrElse {
fail(node.getClass.getSimpleName + " does not have a logical plan link")
}
}