ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-7314] [SPARK-3524] [PYSPARK] upgrade Pyrolite to 4.4

Browse source 
This PR upgrades Pyrolite to 4.4, which contains the bug fix for SPARK-3524 and some other performance improvements (e.g., SPARK-6288). The artifact is still under `org.spark-project` on Maven Central since there is no official release published there.

Author: Xiangrui Meng <meng@databricks.com>

Closes #5850 from mengxr/SPARK-7314 and squashes the following commits:

2ed4a95 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-7314
da3c2dd [Xiangrui Meng] remove my repo
fe7e29b [Xiangrui Meng] switch to maven central
6ddac0e [Xiangrui Meng] reverse the machine code for float/double
d2d5b5b [Xiangrui Meng] change back to 4.4
7824a9c [Xiangrui Meng] use Pyrolite 3.1
cc3903a [Xiangrui Meng] upgrade Pyrolite to 4.4-0 for testing

(cherry picked from commit e9b16e67c6)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
This commit is contained in:
Xiangrui Meng 2015-05-04 23:52:42 -07:00
parent 48655d10ed
commit 21ed108fa7
2 changed files with 3 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
core/pom.xml
View file

@ -362,7 +362,7 @@
<dependency> <dependency>
<groupId>org.spark-project</groupId> <groupId>org.spark-project</groupId>
<artifactId>pyrolite</artifactId> <artifactId>pyrolite</artifactId>
<version>2.0.1</version> <version>4.4</version>
</dependency> </dependency>
<dependency> <dependency>
<groupId>net.sf.py4j</groupId> <groupId>net.sf.py4j</groupId>

7
core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
View file

@ -56,16 +56,13 @@ private[spark] object SerDeUtil extends Logging {
// {'\0', 0, 0, 0} /* Sentinel */ // {'\0', 0, 0, 0} /* Sentinel */
// }; // };
// TODO: support Py_UNICODE with 2 bytes // TODO: support Py_UNICODE with 2 bytes
// FIXME: unpickle array of float is wrong in Pyrolite, so we reverse the
// machine code for float/double here to workaround it.
// we should fix this after Pyrolite fix them
val machineCodes: Map[Char, Int] = if (ByteOrder.nativeOrder().equals(ByteOrder.BIG_ENDIAN)) { val machineCodes: Map[Char, Int] = if (ByteOrder.nativeOrder().equals(ByteOrder.BIG_ENDIAN)) {
Map('c' -> 1, 'B' -> 0, 'b' -> 1, 'H' -> 3, 'h' -> 5, 'I' -> 7, 'i' -> 9, Map('c' -> 1, 'B' -> 0, 'b' -> 1, 'H' -> 3, 'h' -> 5, 'I' -> 7, 'i' -> 9,
'L' -> 11, 'l' -> 13, 'f' -> 14, 'd' -> 16, 'u' -> 21 'L' -> 11, 'l' -> 13, 'f' -> 15, 'd' -> 17, 'u' -> 21
) )
} else { } else {
Map('c' -> 1, 'B' -> 0, 'b' -> 1, 'H' -> 2, 'h' -> 4, 'I' -> 6, 'i' -> 8, Map('c' -> 1, 'B' -> 0, 'b' -> 1, 'H' -> 2, 'h' -> 4, 'I' -> 6, 'i' -> 8,
'L' -> 10, 'l' -> 12, 'f' -> 15, 'd' -> 17, 'u' -> 20 'L' -> 10, 'l' -> 12, 'f' -> 14, 'd' -> 16, 'u' -> 20
) )
} }
override def construct(args: Array[Object]): Object = { override def construct(args: Array[Object]): Object = {