ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-34457][SQL] DataSource V2: Add default null ordering to SortDirection

Browse source 
### What changes were proposed in this pull request?

This PR adds a default null ordering to public `SortDirection` to match the Catalyst behavior.

### Why are the changes needed?

The SQL standard does not define the default null ordering for a sort direction. That's why it is up to a query engine to assign one. We need to standardize this in our public connector expressions to avoid ambiguity. That's why I propose to match the behavior in our Catalyst expressions.

### Does this PR introduce _any_ user-facing change?

Yes, it affects unreleased connector expression API.

### How was this patch tested?

Existing tests.

Closes #31580 from aokolnychyi/spark-34457.

Authored-by: Anton Okolnychyi <aokolnychyi@apple.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This commit is contained in:
Anton Okolnychyi 2021-03-11 05:47:31 +00:00 committed by Wenchen Fan
parent 2a6e68e1f7
commit 72263797bc
2 changed files with 33 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Expressions.java
View file

@ -177,4 +177,17 @@ public class Expressions {
public static SortOrder sort(Expression expr, SortDirection direction, NullOrdering nullOrder) {
return LogicalExpressions.sort(expr, direction, nullOrder);
}
/**
* Create a sort expression.
*
* @param expr an expression to produce values to sort
* @param direction direction of the sort
* @return a SortOrder
*
* @since 3.2.0
*/
public static SortOrder sort(Expression expr, SortDirection direction) {
return LogicalExpressions.sort(expr, direction, direction.defaultNullOrdering());
}
}

21
sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/SortDirection.java
View file

@ -19,14 +19,33 @@ package org.apache.spark.sql.connector.expressions;
import org.apache.spark.annotation.Experimental;
import static org.apache.spark.sql.connector.expressions.NullOrdering.NULLS_FIRST;
import static org.apache.spark.sql.connector.expressions.NullOrdering.NULLS_LAST;
/**
* A sort direction used in sorting expressions.
* <p>
* Each direction has a default null ordering that is implied if no null ordering is specified
* explicitly.
*
* @since 3.2.0
*/
@Experimental
public enum SortDirection {
ASCENDING, DESCENDING;
ASCENDING(NULLS_FIRST), DESCENDING(NULLS_LAST);
private final NullOrdering defaultNullOrdering;
SortDirection(NullOrdering defaultNullOrdering) {
this.defaultNullOrdering = defaultNullOrdering;
}
/**
* Returns the default null ordering to use if no null ordering is specified explicitly.
*/
public NullOrdering defaultNullOrdering() {
return defaultNullOrdering;
}
@Override
public String toString() {