ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-35770][SQL] Parse YearMonthIntervalType from JSON

Browse source 
### What changes were proposed in this pull request?
Parse YearMonthIntervalType from JSON.

### Why are the changes needed?
This will allow to store year-month intervals as table columns into Hive external catalog.

### Does this PR introduce _any_ user-facing change?
People can store year-month interval types as json string.

### How was this patch tested?
Added UT.

Closes #32929 from AngersZhuuuu/SPARK-35770.

Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
This commit is contained in:
Angerszhuuuu 2021-06-17 09:51:47 +03:00 committed by Max Gekk
parent 8aeed08d04
commit 0e554d44df
2 changed files with 8 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala
View file

@ -37,6 +37,7 @@ import org.apache.spark.sql.catalyst.util.StringUtils.StringConcat
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.StoreAssignmentPolicy
import org.apache.spark.sql.internal.SQLConf.StoreAssignmentPolicy.{ANSI, STRICT}
import org.apache.spark.sql.types.YearMonthIntervalType._
import org.apache.spark.util.Utils
/**
@ -173,8 +174,10 @@ object DataType {
DoubleType, FloatType, ShortType, ByteType, StringType, CalendarIntervalType,
// TODO(SPARK-35732): Parse DayTimeIntervalType from JSON
DayTimeIntervalType(),
// TODO(SPARK-35770): Parse YearMonthIntervalType from JSON
YearMonthIntervalType(), TimestampWithoutTZType)
YearMonthIntervalType(YEAR, YEAR),
YearMonthIntervalType(MONTH, MONTH),
YearMonthIntervalType(YEAR, MONTH),
TimestampWithoutTZType)
.map(t => t.typeName -> t).toMap
}

3
sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
View file

@ -256,6 +256,9 @@ class DataTypeSuite extends SparkFunSuite {
checkDataTypeFromJson(VarcharType(10))
checkDataTypeFromDDL(VarcharType(11))
yearMonthIntervalTypes.foreach(checkDataTypeFromJson)
yearMonthIntervalTypes.foreach(checkDataTypeFromDDL)
dayTimeIntervalTypes.foreach(checkDataTypeFromDDL)