spark-instrumented-optimizer/sql/catalyst/src/main
Reynold Xin 84ea287178 [SPARK-10914] UnsafeRow serialization breaks when two machines have different Oops size.
UnsafeRow contains 3 pieces of information when pointing to some data in memory (an object, a base offset, and length). When the row is serialized with Java/Kryo serialization, the object layout in memory can change if two machines have different pointer width (Oops in JVM).

To reproduce, launch Spark using

MASTER=local-cluster[2,1,1024] bin/spark-shell --conf "spark.executor.extraJavaOptions=-XX:-UseCompressedOops"

And then run the following

scala> sql("select 1 xx").collect()

Author: Reynold Xin <rxin@databricks.com>

Closes #9030 from rxin/SPARK-10914.
2015-10-08 17:25:14 -07:00
..
java/org/apache/spark/sql [SPARK-10914] UnsafeRow serialization breaks when two machines have different Oops size. 2015-10-08 17:25:14 -07:00
scala/org/apache/spark/sql [SPARK-8848] [SQL] Refactors Parquet write path to follow parquet-format 2015-10-08 16:18:35 -07:00