ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/dev
History
Josh Rosen 7bbbe380c5 [SPARK-5161] Parallelize Python test execution 
This commit parallelizes the Python unit test execution, significantly reducing Jenkins build times.  Parallelism is now configurable by passing the `-p` or `--parallelism` flags to either `dev/run-tests` or `python/run-tests` (the default parallelism is 4, but I've successfully tested with higher parallelism).

To avoid flakiness, I've disabled the Spark Web UI for the Python tests, similar to what we've done for the JVM tests.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #7031 from JoshRosen/parallelize-python-tests and squashes the following commits:

feb3763 [Josh Rosen] Re-enable other tests
f87ea81 [Josh Rosen] Only log output from failed tests
d4ded73 [Josh Rosen] Logging improvements
a2717e1 [Josh Rosen] Make parallelism configurable via dev/run-tests
1bacf1b [Josh Rosen] Merge remote-tracking branch 'origin/master' into parallelize-python-tests
110cd9d [Josh Rosen] Fix universal_newlines for Python 3
cd13db8 [Josh Rosen] Also log python_implementation
9e31127 [Josh Rosen] Log Python --version output for each executable.
a2b9094 [Josh Rosen] Bump up parallelism.
5552380 [Josh Rosen] Python 3 fix
866b5b9 [Josh Rosen] Fix lazy logging warnings in Prospector checks
87cb988 [Josh Rosen] Skip MLLib tests for PyPy
8309bfe [Josh Rosen] Temporarily disable parallelism to debug a failure
9129027 [Josh Rosen] Disable Spark UI in Python tests
037b686 [Josh Rosen] Temporarily disable JVM tests so we can test Python speedup in Jenkins.
af4cef4 [Josh Rosen] Initial attempt at parallelizing Python test execution
2015-06-29 21:32:40 -07:00
..
audit-release [SQL] [Minor] Fix for SqlApp.scala 2015-04-13 18:23:35 -07:00
create-release [HOTFIX] Fixing errors in name mappings 2015-06-10 22:56:36 -07:00
sparktestsupport [SPARK-5161] Parallelize Python test execution 2015-06-29 21:32:40 -07:00
tests [WIP][HOTFIX][SPARK-4123]: Fix bug in PR dependency (all deps. removed issue) 2015-04-13 22:31:44 -07:00
.gitignore [SPARK-6219] Reuse pep8.py 2015-04-18 16:46:28 -07:00
change-version-to-2.10.sh [SPARK-7087] [BUILD] Fix path issue change version script 2015-04-23 17:23:15 -04:00
change-version-to-2.11.sh [SPARK-7087] [BUILD] Fix path issue change version script 2015-04-23 17:23:15 -04:00
check-license [SPARK-6773][Tests]Fix RAT checks still passed issue when download rat jar failed 2015-04-10 20:02:35 +01:00
github_jira_sync.py Fix install jira-python 2015-05-23 09:14:07 -07:00
lint-python [SPARK-8583] [SPARK-5482] [BUILD] Refactor python/run-tests to integrate with dev/run-tests module system 2015-06-27 20:24:34 -07:00
lint-r [SPARK-8495] [SPARKR] Add a .lintr file to validate the SparkR files and the lint-r script 2015-06-20 16:10:14 -07:00
lint-r.R [SPARK-8495] [SPARKR] Add a .lintr file to validate the SparkR files and the lint-r script 2015-06-20 16:10:14 -07:00
lint-scala [SPARK-2627] [PySpark] have the build enforce PEP 8 automatically 2014-08-06 12:58:24 -07:00
merge_spark_pr.py [SPARK-8652] [PYSPARK] Check return value for all uses of doctest.testmod() 2015-06-26 08:12:22 -07:00
mima [minor] [build] Set java options when generating mima ignores. 2015-04-21 16:35:37 -07:00
README.md Merge pull request #565 from pwendell/dev-scripts. Closes #565. 2014-02-08 23:13:34 -08:00
run-tests [SPARK-5161] Parallelize Python test execution 2015-06-29 21:32:40 -07:00
run-tests-codes.sh [SPARK-7017] [BUILD] [PROJECT INFRA] Refactor dev/run-tests into Python 2015-06-17 12:00:34 -07:00
run-tests-jenkins [SPARK-7017] [BUILD] [PROJECT INFRA] Refactor dev/run-tests into Python 2015-06-17 12:00:34 -07:00
run-tests.py [SPARK-5161] Parallelize Python test execution 2015-06-29 21:32:40 -07:00
scalastyle [SPARK-7722] [STREAMING] Added Kinesis to style checker 2015-05-21 13:50:08 -07:00

README.md

Spark Developer Scripts

This directory contains scripts useful to developers when packaging, testing, or committing to Spark.

Many of these scripts require Apache credentials to work correctly.