ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-12834] Change ser/de of JavaArray and JavaList

Browse source 
https://issues.apache.org/jira/browse/SPARK-12834

We use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780

Author: Xusen Yin <yinxusen@gmail.com>

Closes #10772 from yinxusen/SPARK-12834.
This commit is contained in:
Xusen Yin 2016-01-25 22:41:52 -08:00 committed by Joseph K. Bradley
parent b66afdeb52
commit ae47ba718a
1 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
View file

@ -1490,7 +1490,11 @@ private[spark] object SerDe extends Serializable {
initialize()
def dumps(obj: AnyRef): Array[Byte] = {
new Pickler().dumps(obj)
obj match {
// Pickler in Python side cannot deserialize Scala Array normally. See SPARK-12834.
case array: Array[_] => new Pickler().dumps(array.toSeq.asJava)
case _ => new Pickler().dumps(obj)
}
}
def loads(bytes: Array[Byte]): AnyRef = {