[SPARK-24640][SQL] Return NULL
from size(NULL)
by default
### What changes were proposed in this pull request? Set the default value of the `spark.sql.legacy.sizeOfNull` config to `false`. That changes behavior of the `size()` function for `NULL`. The function will return `NULL` for `NULL` instead of `-1`. ### Why are the changes needed? There is the agreement in the PR https://github.com/apache/spark/pull/21598#issuecomment-399695523 to change behavior in Spark 3.0. ### Does this PR introduce any user-facing change? Yes. Before: ```sql spark-sql> select size(NULL); -1 ``` After: ```sql spark-sql> select size(NULL); NULL ``` ### How was this patch tested? By the `check outputs of expression examples` test in `SQLQuerySuite` which runs expression examples. Closes #26051 from MaxGekk/sizeof-null-returns-null. Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
This commit is contained in:
parent
ef1e8495ba
commit
4e6d31f570
|
@ -79,7 +79,7 @@ trait BinaryArrayExpressionWithImplicitCast extends BinaryExpression
|
|||
_FUNC_(expr) - Returns the size of an array or a map.
|
||||
The function returns -1 if its input is null and spark.sql.legacy.sizeOfNull is set to true.
|
||||
If spark.sql.legacy.sizeOfNull is set to false, the function returns null for null input.
|
||||
By default, the spark.sql.legacy.sizeOfNull parameter is set to true.
|
||||
By default, the spark.sql.legacy.sizeOfNull parameter is set to false.
|
||||
""",
|
||||
examples = """
|
||||
Examples:
|
||||
|
@ -88,7 +88,7 @@ trait BinaryArrayExpressionWithImplicitCast extends BinaryExpression
|
|||
> SELECT _FUNC_(map('a', 1, 'b', 2));
|
||||
2
|
||||
> SELECT _FUNC_(NULL);
|
||||
-1
|
||||
NULL
|
||||
""")
|
||||
case class Size(child: Expression) extends UnaryExpression with ExpectsInputTypes {
|
||||
|
||||
|
|
|
@ -1865,7 +1865,7 @@ object SQLConf {
|
|||
.doc("If it is set to true, size of null returns -1. This behavior was inherited from Hive. " +
|
||||
"The size function returns null for null input if the flag is disabled.")
|
||||
.booleanConf
|
||||
.createWithDefault(true)
|
||||
.createWithDefault(false)
|
||||
|
||||
val LEGACY_REPLACE_DATABRICKS_SPARK_AVRO_ENABLED =
|
||||
buildConf("spark.sql.legacy.replaceDatabricksSparkAvro.enabled")
|
||||
|
|
Loading…
Reference in a new issue