71af030b46
After this patch, we can run PySpark in PyPy (testing with PyPy 2.3.1 in Mac 10.9), for example:
```
PYSPARK_PYTHON=pypy ./bin/spark-submit wordcount.py
```
The performance speed up will depend on work load (from 20% to 3000%). Here are some benchmarks:
Job | CPython 2.7 | PyPy 2.3.1 | Speed up
------- | ------------ | ------------- | -------
Word Count | 41s | 15s | 2.7x
Sort | 46s | 44s | 1.05x
Stats | 174s | 3.6s | 48x
Here is the code used for benchmark:
```python
rdd = sc.textFile("text")
def wordcount():
rdd.flatMap(lambda x:x.split('/'))\
.map(lambda x:(x,1)).reduceByKey(lambda x,y:x+y).collectAsMap()
def sort():
rdd.sortBy(lambda x:x, 1).count()
def stats():
sc.parallelize(range(1024), 20).flatMap(lambda x: xrange(5024)).stats()
```
Author: Davies Liu <davies.liu@gmail.com>
Closes #2144 from davies/pypy and squashes the following commits:
9aed6c5 [Davies Liu] use protocol 2 in CloudPickle
4bc1f04 [Davies Liu] refactor
b20ab3a [Davies Liu] pickle sys.stdout and stderr in portable way
3ca2351 [Davies Liu] Merge branch 'master' into pypy
fae8b19 [Davies Liu] improve attrgetter, add tests
591f830 [Davies Liu] try to run tests with PyPy in run-tests
c8d62ba [Davies Liu] cleanup
f651fd0 [Davies Liu] fix tests using array with PyPy
1b98fb3 [Davies Liu] serialize itemgetter/attrgetter in portable ways
3c1dbfe [Davies Liu] Merge branch 'master' into pypy
42fb5fa [Davies Liu] Merge branch 'master' into pypy
cb2d724 [Davies Liu] fix tests
9986692 [Davies Liu] Merge branch 'master' into pypy
25b4ca7 [Davies Liu] support PyPy
117 lines
3.6 KiB
Bash
Executable file
117 lines
3.6 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
|
|
# Figure out where the Spark framework is installed
|
|
FWDIR="$(cd "`dirname "$0"`"; cd ../; pwd)"
|
|
|
|
# CD into the python directory to find things on the right path
|
|
cd "$FWDIR/python"
|
|
|
|
FAILED=0
|
|
|
|
rm -f unit-tests.log
|
|
|
|
# Remove the metastore and warehouse directory created by the HiveContext tests in Spark SQL
|
|
rm -rf metastore warehouse
|
|
|
|
function run_test() {
|
|
echo "Running test: $1"
|
|
|
|
SPARK_TESTING=1 "$FWDIR"/bin/pyspark $1 2>&1 | tee -a unit-tests.log
|
|
|
|
FAILED=$((PIPESTATUS[0]||$FAILED))
|
|
|
|
# Fail and exit on the first test failure.
|
|
if [[ $FAILED != 0 ]]; then
|
|
cat unit-tests.log | grep -v "^[0-9][0-9]*" # filter all lines starting with a number.
|
|
echo -en "\033[31m" # Red
|
|
echo "Had test failures; see logs."
|
|
echo -en "\033[0m" # No color
|
|
exit -1
|
|
fi
|
|
}
|
|
|
|
echo "Running PySpark tests. Output is in python/unit-tests.log."
|
|
|
|
export PYSPARK_PYTHON="python"
|
|
|
|
# Try to test with Python 2.6, since that's the minimum version that we support:
|
|
if [ $(which python2.6) ]; then
|
|
export PYSPARK_PYTHON="python2.6"
|
|
fi
|
|
|
|
echo "Testing with Python version:"
|
|
$PYSPARK_PYTHON --version
|
|
|
|
run_test "pyspark/rdd.py"
|
|
run_test "pyspark/context.py"
|
|
run_test "pyspark/conf.py"
|
|
run_test "pyspark/sql.py"
|
|
# These tests are included in the module-level docs, and so must
|
|
# be handled on a higher level rather than within the python file.
|
|
export PYSPARK_DOC_TEST=1
|
|
run_test "pyspark/broadcast.py"
|
|
run_test "pyspark/accumulators.py"
|
|
run_test "pyspark/serializers.py"
|
|
unset PYSPARK_DOC_TEST
|
|
run_test "pyspark/shuffle.py"
|
|
run_test "pyspark/tests.py"
|
|
run_test "pyspark/mllib/_common.py"
|
|
run_test "pyspark/mllib/classification.py"
|
|
run_test "pyspark/mllib/clustering.py"
|
|
run_test "pyspark/mllib/linalg.py"
|
|
run_test "pyspark/mllib/random.py"
|
|
run_test "pyspark/mllib/recommendation.py"
|
|
run_test "pyspark/mllib/regression.py"
|
|
run_test "pyspark/mllib/stat.py"
|
|
run_test "pyspark/mllib/tests.py"
|
|
run_test "pyspark/mllib/tree.py"
|
|
run_test "pyspark/mllib/util.py"
|
|
|
|
# Try to test with PyPy
|
|
if [ $(which pypy) ]; then
|
|
export PYSPARK_PYTHON="pypy"
|
|
echo "Testing with PyPy version:"
|
|
$PYSPARK_PYTHON --version
|
|
|
|
run_test "pyspark/rdd.py"
|
|
run_test "pyspark/context.py"
|
|
run_test "pyspark/conf.py"
|
|
run_test "pyspark/sql.py"
|
|
# These tests are included in the module-level docs, and so must
|
|
# be handled on a higher level rather than within the python file.
|
|
export PYSPARK_DOC_TEST=1
|
|
run_test "pyspark/broadcast.py"
|
|
run_test "pyspark/accumulators.py"
|
|
run_test "pyspark/serializers.py"
|
|
unset PYSPARK_DOC_TEST
|
|
run_test "pyspark/shuffle.py"
|
|
run_test "pyspark/tests.py"
|
|
fi
|
|
|
|
if [[ $FAILED == 0 ]]; then
|
|
echo -en "\033[32m" # Green
|
|
echo "Tests passed."
|
|
echo -en "\033[0m" # No color
|
|
fi
|
|
|
|
# TODO: in the long-run, it would be nice to use a test runner like `nose`.
|
|
# The doctest fixtures are the current barrier to doing this.
|