ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/core/src/main
History
Josh Rosen efa80a531e [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer 
This PR fixes an issue where PySpark broadcast variables caused NullPointerExceptions if KryoSerializer was used.  The fix is to register PythonBroadcast with Kryo so that it's deserialized with a KryoJavaSerializer.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #3831 from JoshRosen/SPARK-4882 and squashes the following commits:

0466c7a [Josh Rosen] Register PythonBroadcast with Kryo.
d5b409f [Josh Rosen] Enable registrationRequired, which would have caught this bug.
069d8a7 [Josh Rosen] Add failing test for SPARK-4882
2014-12-30 09:29:52 -08:00
..
java/org/apache/spark SPARK-4567. Make SparkJobInfo and SparkStageInfo serializable 2014-12-09 16:26:07 -08:00
resources/org/apache/spark [SPARK-4920][UI]:current spark version in UI is not striking. 2014-12-22 14:14:39 -08:00
scala/org/apache/spark [SPARK-4882] Register PythonBroadcast with Kryo so that PySpark works with KryoSerializer 2014-12-30 09:29:52 -08:00