spark-instrumented-optimizer/python
Davies Liu 0d8cdf0ede [SPARK-3681] [SQL] [PySpark] fix serialization of List and Map in SchemaRDD
Currently, the schema of object in ArrayType or MapType is attached lazily, it will have better performance but introduce issues while serialization or accessing nested objects.

This patch will apply schema to the objects of ArrayType or MapType immediately when accessing them, will be a little bit slower, but much robust.

Author: Davies Liu <davies.liu@gmail.com>

Closes #2526 from davies/nested and squashes the following commits:

2399ae5 [Davies Liu] fix serialization of List and Map in SchemaRDD
2014-09-27 12:21:37 -07:00
..
docs [SPARK-3430] [PySpark] [Doc] generate PySpark API docs using Sphinx 2014-09-16 12:51:58 -07:00
lib [SPARK-2305] [PySpark] Update Py4J to version 0.8.2.1 2014-07-29 19:02:06 -07:00
pyspark [SPARK-3681] [SQL] [PySpark] fix serialization of List and Map in SchemaRDD 2014-09-27 12:21:37 -07:00
test_support [SPARK-3634] [PySpark] User's module should take precedence over system modules 2014-09-24 12:10:09 -07:00
.gitignore SPARK-1004. PySpark on YARN 2014-04-29 23:24:34 -07:00
epydoc.conf [SPARK-3491] [MLlib] [PySpark] use pickle to serialize data in MLlib 2014-09-19 15:01:11 -07:00
run-tests [SPARK-3491] [MLlib] [PySpark] use pickle to serialize data in MLlib 2014-09-19 15:01:11 -07:00