ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-35860][SQL] Support UpCast between different field of YearMonthIntervalType/DayTimeIntervalType

Browse source 
### What changes were proposed in this pull request?
Support UpCast between different field of YearMonthIntervalType/DayTimeIntervalType

### Why are the changes needed?
Since in our encoder we handle Period/Duration as default  YearMonthIntervalType/DayTimeIntervalType, when we use udf to handle this type, it will upcast all type of YearMonthIntervalType/DayTimeIntervalType to default YearMonthIntervalType/DayTimeIntervalType, so we need to support this.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Added Ut

Closes #33035 from AngersZhuuuu/SPARK-35860.

Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
This commit is contained in:
Angerszhuuuu 2021-06-23 11:32:13 +03:00 committed by Max Gekk
parent 7c1a9dd3f5
commit 758b423a31
2 changed files with 15 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
View file

@ -174,6 +174,9 @@ object Cast {
resolvableNullability(f1.nullable, f2.nullable) && canUpCast(f1.dataType, f2.dataType)
}
case (_: DayTimeIntervalType, _: DayTimeIntervalType) => true
case (_: YearMonthIntervalType, _: YearMonthIntervalType) => true
case (from: UserDefinedType[_], to: UserDefinedType[_]) if to.acceptsType(from) => true
case _ => false

12
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
View file

@ -672,6 +672,18 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
atomicTypes.foreach { atomicType =>
assert(Cast.canUpCast(NullType, atomicType))
}
dayTimeIntervalTypes.foreach { from =>
dayTimeIntervalTypes.foreach { to =>
assert(Cast.canUpCast(from, to))
}
}
yearMonthIntervalTypes.foreach { from =>
yearMonthIntervalTypes.foreach { to =>
assert(Cast.canUpCast(from, to))
}
}
}
test("SPARK-27671: cast from nested null type in struct") {