[SPARK-34935][SQL] CREATE TABLE LIKE should respect the reserved table properties
### What changes were proposed in this pull request? CREATE TABLE LIKE should respect the reserved properties of tables and fail if specified, using `spark.sql.legacy.notReserveProperties` to restore. ### Why are the changes needed? Make DDLs consistently treat reserved properties ### Does this PR introduce _any_ user-facing change? YES, this is a breaking change as using `create table like` w/ reserved properties will fail. ### How was this patch tested? new test Closes #32025 from yaooqinn/SPARK-34935. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
This commit is contained in:
parent
39d5677ee3
commit
7cffacef18
|
@ -73,6 +73,8 @@ license: |
|
|||
|
||||
- In Spark 3.2, the timestamps subtraction expression such as `timestamp '2021-03-31 23:48:00' - timestamp '2021-01-01 00:00:00'` returns values of `DayTimeIntervalType`. In Spark 3.1 and earlier, the type of the same expression is `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
|
||||
|
||||
- In Spark 3.2, `CREATE TABLE .. LIKE ..` command can not use reserved properties. You need their specific clauses to specify them, for example, `CREATE TABLE test1 LIKE test LOCATION 'some path'`. You can set `spark.sql.legacy.notReserveProperties` to `true` to ignore the `ParseException`, in this case, these properties will be silently removed, for example: `TBLPROPERTIES('owner'='yao')` will have no effect. In Spark version 3.1 and below, the reserved properties can be used in `CREATE TABLE .. LIKE ..` command but have no side effects, for example, `TBLPROPERTIES('location'='/tmp')` does not change the location of the table but only create a headless property just like `'a'='b'`.
|
||||
|
||||
## Upgrading from Spark SQL 3.0 to 3.1
|
||||
|
||||
- In Spark 3.1, statistical aggregation function includes `std`, `stddev`, `stddev_samp`, `variance`, `var_samp`, `skewness`, `kurtosis`, `covar_samp`, `corr` will return `NULL` instead of `Double.NaN` when `DivideByZero` occurs during expression evaluation, for example, when `stddev_samp` applied on a single element set. In Spark version 3.0 and earlier, it will return `Double.NaN` in such case. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.statisticalAggregate` to `true`.
|
||||
|
|
|
@ -467,8 +467,9 @@ class SparkSqlAstBuilder extends AstBuilder {
|
|||
|
||||
val storage = toStorageFormat(location, serdeInfo, ctx)
|
||||
val properties = Option(ctx.tableProps).map(visitPropertyKeyValues).getOrElse(Map.empty)
|
||||
val cleanedProperties = cleanTableProperties(ctx, properties)
|
||||
CreateTableLikeCommand(
|
||||
targetTable, sourceTable, storage, provider, properties, ctx.EXISTS != null)
|
||||
targetTable, sourceTable, storage, provider, cleanedProperties, ctx.EXISTS != null)
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
@ -24,6 +24,7 @@ import org.apache.spark.sql.catalyst.TableIdentifier
|
|||
import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedAlias, UnresolvedAttribute, UnresolvedRelation, UnresolvedStar}
|
||||
import org.apache.spark.sql.catalyst.expressions.{Ascending, AttributeReference, Concat, SortOrder}
|
||||
import org.apache.spark.sql.catalyst.plans.logical._
|
||||
import org.apache.spark.sql.connector.catalog.TableCatalog
|
||||
import org.apache.spark.sql.execution.command._
|
||||
import org.apache.spark.sql.execution.datasources.{CreateTempViewUsing, RefreshResource}
|
||||
import org.apache.spark.sql.internal.StaticSQLConf
|
||||
|
@ -342,4 +343,11 @@ class SparkSqlParserSuite extends AnalysisTest {
|
|||
test("CLEAR CACHE") {
|
||||
assertEqual("CLEAR CACHE", ClearCacheCommand)
|
||||
}
|
||||
|
||||
test("CREATE TABLE LIKE COMMAND should reject reserved properties") {
|
||||
Seq(TableCatalog.PROP_OWNER, TableCatalog.PROP_PROVIDER).foreach { reserved =>
|
||||
intercept(s"CREATE TABLE target LIKE source TBLPROPERTIES ($reserved='howdy')",
|
||||
"reserved")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
Loading…
Reference in a new issue