ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-31975][SQL] Show AnalysisException when WindowFunction is used without WindowExpression

Browse source 
### What changes were proposed in this pull request?

Add WindowFunction check at `CheckAnalysis`.

### Why are the changes needed?
Provide friendly error msg.

**BEFORE**
```scala
scala> sql("select rank() from values(1)").show
java.lang.UnsupportedOperationException: Cannot generate code for expression: rank()
```

**AFTER**
```scala
scala> sql("select rank() from values(1)").show
org.apache.spark.sql.AnalysisException: Window function rank() requires an OVER clause.;;
Project [rank() AS RANK()#3]
+- LocalRelation [col1#2]
```

### Does this PR introduce _any_ user-facing change?

Yes, user wiill be given a better error msg.

### How was this patch tested?

Pass the newly added UT.

Closes #28808 from ulysses-you/SPARK-31975.

Authored-by: ulysses <youxiduo@weidian.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This commit is contained in:
ulysses 2020-07-07 13:39:04 +00:00 committed by Wenchen Fan
parent 5d296ed39e
commit 2e23da2bda
2 changed files with 16 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
View file

@ -158,6 +158,11 @@ trait CheckAnalysis extends PredicateHelper {
case g: GroupingID => case g: GroupingID =>
failAnalysis("grouping_id() can only be used with GroupingSets/Cube/Rollup") failAnalysis("grouping_id() can only be used with GroupingSets/Cube/Rollup")
case e: Expression if e.children.exists(_.isInstanceOf[WindowFunction]) &&
!e.isInstanceOf[WindowExpression] =>
val w = e.children.find(_.isInstanceOf[WindowFunction]).get
failAnalysis(s"Window function $w requires an OVER clause.")
case w @ WindowExpression(AggregateExpression(_, _, true, _, _), _) => case w @ WindowExpression(AggregateExpression(_, _, true, _, _), _) =>
failAnalysis(s"Distinct window functions are not supported: $w") failAnalysis(s"Distinct window functions are not supported: $w")

11
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
View file

@ -884,4 +884,15 @@ class AnalysisSuite extends AnalysisTest with Matchers {
Seq("Intersect can only be performed on tables with the compatible column types. " + Seq("Intersect can only be performed on tables with the compatible column types. " +
"timestamp <> double at the second column of the second table")) "timestamp <> double at the second column of the second table"))
} }
test("SPARK-31975: Throw user facing error when use WindowFunction directly") {
assertAnalysisError(testRelation2.select(RowNumber()),
Seq("Window function row_number() requires an OVER clause."))
assertAnalysisError(testRelation2.select(Sum(RowNumber())),
Seq("Window function row_number() requires an OVER clause."))
assertAnalysisError(testRelation2.select(RowNumber() + 1),
Seq("Window function row_number() requires an OVER clause."))
}
} }