[SPARK-27619][SQL][FOLLOWUP] Rename 'spark.sql.legacy.useHashOnMapType' to 'spark.sql.legacy.allowHashOnMapType'
### What changes were proposed in this pull request? Renamed configuration from `spark.sql.legacy.useHashOnMapType` to `spark.sql.legacy.allowHashOnMapType`. ### Why are the changes needed? Better readability of configuration. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing UTs. Closes #27719 from iRakson/SPARK-27619_FOLLOWUP. Authored-by: iRakson <raksonrakesh@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This commit is contained in:
parent
eba2076ca3
commit
a40a2f8338
|
@ -224,7 +224,7 @@ license: |
|
||||||
|
|
||||||
- Since Spark 3.0, when casting string value to integral types(tinyint, smallint, int and bigint), datetime types(date, timestamp and interval) and boolean type, the leading and trailing whitespaces (<= ASCII 32) will be trimmed before converted to these type values, e.g. `cast(' 1\t' as int)` results `1`, `cast(' 1\t' as boolean)` results `true`, `cast('2019-10-10\t as date)` results the date value `2019-10-10`. In Spark version 2.4 and earlier, while casting string to integrals and booleans, it will not trim the whitespaces from both ends, the foregoing results will be `null`, while to datetimes, only the trailing spaces (= ASCII 32) will be removed.
|
- Since Spark 3.0, when casting string value to integral types(tinyint, smallint, int and bigint), datetime types(date, timestamp and interval) and boolean type, the leading and trailing whitespaces (<= ASCII 32) will be trimmed before converted to these type values, e.g. `cast(' 1\t' as int)` results `1`, `cast(' 1\t' as boolean)` results `true`, `cast('2019-10-10\t as date)` results the date value `2019-10-10`. In Spark version 2.4 and earlier, while casting string to integrals and booleans, it will not trim the whitespaces from both ends, the foregoing results will be `null`, while to datetimes, only the trailing spaces (= ASCII 32) will be removed.
|
||||||
|
|
||||||
- Since Spark 3.0, An analysis exception will be thrown when hash expressions are applied on elements of MapType. To restore the behavior before Spark 3.0, set `spark.sql.legacy.useHashOnMapType` to true.
|
- Since Spark 3.0, an analysis exception will be thrown when hash expressions are applied on elements of MapType. To restore the behavior before Spark 3.0, set `spark.sql.legacy.allowHashOnMapType` to `true`.
|
||||||
|
|
||||||
- Since Spark 3.0, numbers written in scientific notation(e.g. `1E2`) would be parsed as Double. In Spark version 2.4 and earlier, they're parsed as Decimal. To restore the behavior before Spark 3.0, you can set `spark.sql.legacy.exponentLiteralAsDecimal.enabled` to `true`.
|
- Since Spark 3.0, numbers written in scientific notation(e.g. `1E2`) would be parsed as Double. In Spark version 2.4 and earlier, they're parsed as Decimal. To restore the behavior before Spark 3.0, you can set `spark.sql.legacy.exponentLiteralAsDecimal.enabled` to `true`.
|
||||||
|
|
||||||
|
|
|
@ -256,11 +256,11 @@ abstract class HashExpression[E] extends Expression {
|
||||||
TypeCheckResult.TypeCheckFailure(
|
TypeCheckResult.TypeCheckFailure(
|
||||||
s"input to function $prettyName requires at least one argument")
|
s"input to function $prettyName requires at least one argument")
|
||||||
} else if (children.exists(child => hasMapType(child.dataType)) &&
|
} else if (children.exists(child => hasMapType(child.dataType)) &&
|
||||||
!SQLConf.get.getConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE)) {
|
!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_HASH_ON_MAPTYPE)) {
|
||||||
TypeCheckResult.TypeCheckFailure(
|
TypeCheckResult.TypeCheckFailure(
|
||||||
s"input to function $prettyName cannot contain elements of MapType. In Spark, same maps " +
|
s"input to function $prettyName cannot contain elements of MapType. In Spark, same maps " +
|
||||||
"may have different hashcode, thus hash expressions are prohibited on MapType " +
|
"may have different hashcode, thus hash expressions are prohibited on MapType elements." +
|
||||||
s"elements. To restore previous behavior set ${SQLConf.LEGACY_USE_HASH_ON_MAPTYPE.key} " +
|
s" To restore previous behavior set ${SQLConf.LEGACY_ALLOW_HASH_ON_MAPTYPE.key} " +
|
||||||
"to true.")
|
"to true.")
|
||||||
} else {
|
} else {
|
||||||
TypeCheckResult.TypeCheckSuccess
|
TypeCheckResult.TypeCheckSuccess
|
||||||
|
|
|
@ -2258,7 +2258,7 @@ object SQLConf {
|
||||||
.booleanConf
|
.booleanConf
|
||||||
.createWithDefault(false)
|
.createWithDefault(false)
|
||||||
|
|
||||||
val LEGACY_USE_HASH_ON_MAPTYPE = buildConf("spark.sql.legacy.useHashOnMapType")
|
val LEGACY_ALLOW_HASH_ON_MAPTYPE = buildConf("spark.sql.legacy.allowHashOnMapType")
|
||||||
.doc("When set to true, hash expressions can be applied on elements of MapType. Otherwise, " +
|
.doc("When set to true, hash expressions can be applied on elements of MapType. Otherwise, " +
|
||||||
"an analysis exception will be thrown.")
|
"an analysis exception will be thrown.")
|
||||||
.booleanConf
|
.booleanConf
|
||||||
|
|
|
@ -2130,10 +2130,11 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
test("SPARK-27619: when spark.sql.legacy.useHashOnMapType is true, hash can be used on Maptype") {
|
test(s"SPARK-27619: When ${SQLConf.LEGACY_ALLOW_HASH_ON_MAPTYPE.key} is true, hash can be " +
|
||||||
|
"used on Maptype") {
|
||||||
Seq("hash", "xxhash64").foreach {
|
Seq("hash", "xxhash64").foreach {
|
||||||
case hashExpression =>
|
case hashExpression =>
|
||||||
withSQLConf(SQLConf.LEGACY_USE_HASH_ON_MAPTYPE.key -> "true") {
|
withSQLConf(SQLConf.LEGACY_ALLOW_HASH_ON_MAPTYPE.key -> "true") {
|
||||||
val df = spark.createDataset(Map() :: Nil)
|
val df = spark.createDataset(Map() :: Nil)
|
||||||
checkAnswer(df.selectExpr(s"$hashExpression(*)"), sql(s"SELECT $hashExpression(map())"))
|
checkAnswer(df.selectExpr(s"$hashExpression(*)"), sql(s"SELECT $hashExpression(map())"))
|
||||||
}
|
}
|
||||||
|
|
Loading…
Reference in a new issue