[SPARK-12834] Change ser/de of JavaArray and JavaList
https://issues.apache.org/jira/browse/SPARK-12834 We use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780 Author: Xusen Yin <yinxusen@gmail.com> Closes #10772 from yinxusen/SPARK-12834.
This commit is contained in:
parent
b66afdeb52
commit
ae47ba718a
|
@ -1490,7 +1490,11 @@ private[spark] object SerDe extends Serializable {
|
|||
initialize()
|
||||
|
||||
def dumps(obj: AnyRef): Array[Byte] = {
|
||||
new Pickler().dumps(obj)
|
||||
obj match {
|
||||
// Pickler in Python side cannot deserialize Scala Array normally. See SPARK-12834.
|
||||
case array: Array[_] => new Pickler().dumps(array.toSeq.asJava)
|
||||
case _ => new Pickler().dumps(obj)
|
||||
}
|
||||
}
|
||||
|
||||
def loads(bytes: Array[Byte]): AnyRef = {
|
||||
|
|
Loading…
Reference in a new issue