ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-27253][SQL][FOLLOW-UP] Add a legacy flag to restore old session init behavior

Browse source 
## What changes were proposed in this pull request?

Add a legacy flag to restore the old session init behavior, where SparkConf defaults take precedence over configs in a parent session.

Closes #24540 from jose-torres/oss.

Authored-by: Jose Torres <torres.joseph.f+github@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
Jose Torres 2019-05-07 20:04:09 -07:00 committed by Dongjoon Hyun
parent 303ee3fce0
commit 83f628b57d
3 changed files with 15 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/sql-migration-guide-upgrade.md
View file

@ -122,7 +122,7 @@ license: |
- Since Spark 3.0, `TIMESTAMP` literals are converted to strings using the SQL config `spark.sql.session.timeZone`. In Spark version 2.4 and earlier, the conversion uses the default time zone of the Java virtual machine.
- In Spark version 2.4, when a spark session is created via `cloneSession()`, the newly created spark session inherits its configuration from its parent `SparkContext` even though the same configuration may exist with a different value in its parent spark session. Since Spark 3.0, the configurations of a parent `SparkSession` have a higher precedence over the parent `SparkContext`.
- In Spark version 2.4, when a spark session is created via `cloneSession()`, the newly created spark session inherits its configuration from its parent `SparkContext` even though the same configuration may exist with a different value in its parent spark session. Since Spark 3.0, the configurations of a parent `SparkSession` have a higher precedence over the parent `SparkContext`. The old behavior can be restored by setting `spark.sql.legacy.sessionInitWithConfigDefaults` to `true`.
- Since Spark 3.0, parquet logical type `TIMESTAMP_MICROS` is used by default while saving `TIMESTAMP` columns. In Spark version 2.4 and earlier, `TIMESTAMP` columns are saved as `INT96` in parquet files. To set `INT96` to `spark.sql.parquet.outputTimestampType` restores the previous behavior.

7
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
View file

@ -138,4 +138,11 @@ object StaticSQLConf {
.intConf
.checkValue(_ >= 0, "Must be set greater or equal to zero")
.createWithDefault(Int.MaxValue)
val SQL_LEGACY_SESSION_INIT_WITH_DEFAULTS =
buildStaticConf("spark.sql.legacy.sessionInitWithConfigDefaults")
.doc("Flag to revert to legacy behavior where a cloned SparkSession receives SparkConf " +
"defaults, dropping any overrides in its parent SparkSession.")
.booleanConf
.createWithDefault(false)
}

8
sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala
View file

@ -85,7 +85,13 @@ abstract class BaseSessionStateBuilder(
* with its [[SparkConf]] only when there is no parent session.
*/
protected lazy val conf: SQLConf = {
parentState.map(_.conf.clone()).getOrElse {
parentState.map { s =>
val cloned = s.conf.clone()
if (session.sparkContext.conf.get(StaticSQLConf.SQL_LEGACY_SESSION_INIT_WITH_DEFAULTS)) {
mergeSparkConf(cloned, session.sparkContext.conf)
}
cloned
}.getOrElse {
val conf = new SQLConf
mergeSparkConf(conf, session.sparkContext.conf)
conf