ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-11379][SQL] ExpressionEncoder can't handle top level primitive type correctly

Browse source 
For inner primitive type(e.g. inside `Product`), we use `schemaFor` to get the catalyst type for it, https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L403.

However, for top level primitive type, we use `dataTypeFor`, which is wrong.

Author: Wenchen Fan <wenchen@databricks.com>

Closes #9337 from cloud-fan/encoder.
This commit is contained in:
Wenchen Fan 2015-10-29 11:17:03 +01:00 committed by Michael Armbrust
parent 3dfa4ea526
commit 87f28fc240
2 changed files with 2 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
View file

@ -170,7 +170,7 @@ trait ScalaReflection {
.getOrElse(BoundReference(ordinal, dataType, false))
/** Returns the current path or throws an error. */
def getPath = path.getOrElse(BoundReference(0, dataTypeFor(tpe), true))
def getPath = path.getOrElse(BoundReference(0, schemaFor(tpe).dataType, true))
tpe match {
case t if !dataTypeFor(t).isInstanceOf[ObjectType] =>

1
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
View file

@ -57,6 +57,7 @@ class ExpressionEncoderSuite extends SparkFunSuite {
encodeDecodeTest(false)
encodeDecodeTest(1.toShort)
encodeDecodeTest(1.toByte)
encodeDecodeTest("hello")
encodeDecodeTest(PrimitiveData(1, 1, 1, 1, 1, 1, true))