27ef3629dd
### What changes were proposed in this pull request?
This PR mainly proposes to run only relevant tests just like Jenkins PR builder does. Currently, GitHub Actions always run full tests which wastes the resources.
In addition, this PR also fixes 3 more issues very closely related together while I am here.
1. The main idea here is: It reuses the existing logic embedded in `dev/run-tests.py` which Jenkins PR builder use in order to run only the related test cases.
2. While I am here, I fixed SPARK-32292 too to run the doc tests. It was because other references were not available when it is cloned via `checkoutv2`. With `fetch-depth: 0`, the history is available.
3. In addition, it fixes the `dev/run-tests.py` to match with `python/run-tests.py` in terms of its options. Environment variables such as `TEST_ONLY_XXX` were moved as proper options. For example,
```bash
dev/run-tests.py --modules sql,core
```
which is consistent with `python/run-tests.py`, for example,
```bash
python/run-tests.py --modules pyspark-core,pyspark-ml
```
4. Lastly, also fixed the formatting issue in module specification in the matrix:
```diff
- network_common, network_shuffle, repl, launcher
+ network-common, network-shuffle, repl, launcher,
```
which incorrectly runs build/test the modules.
### Why are the changes needed?
By running only related tests, we can hugely save the resources and avoid unrelated flaky tests, etc.
Also, now it runs the doctest of `dev/run-tests.py` properly, the usages are similar between `dev/run-tests.py` and `python/run-tests.py`, and run `network-common`, `network-shuffle`, `launcher` and `examples` modules too.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Manually tested in my own forked Spark:
https://github.com/HyukjinKwon/spark/pull/7
https://github.com/HyukjinKwon/spark/pull/8
https://github.com/HyukjinKwon/spark/pull/9
https://github.com/HyukjinKwon/spark/pull/10
https://github.com/HyukjinKwon/spark/pull/11
https://github.com/HyukjinKwon/spark/pull/12
Closes #29086 from HyukjinKwon/SPARK-32292.
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
||
|---|---|---|
| .. | ||
| create-release | ||
| deps | ||
| sparktestsupport | ||
| tests | ||
| .gitignore | ||
| .rat-excludes | ||
| .scalafmt.conf | ||
| appveyor-guide.md | ||
| appveyor-install-dependencies.ps1 | ||
| change-scala-version.sh | ||
| check-license | ||
| checkstyle-suppressions.xml | ||
| checkstyle.xml | ||
| github_jira_sync.py | ||
| lint-java | ||
| lint-python | ||
| lint-r | ||
| lint-r.R | ||
| lint-scala | ||
| make-distribution.sh | ||
| merge_spark_pr.py | ||
| mima | ||
| pip-sanity-check.py | ||
| README.md | ||
| requirements.txt | ||
| run-pip-tests | ||
| run-tests | ||
| run-tests-jenkins | ||
| run-tests-jenkins.py | ||
| run-tests.py | ||
| sbt-checkstyle | ||
| scalafmt | ||
| scalastyle | ||
| test-dependencies.sh | ||
| tox.ini | ||
Spark Developer Scripts
This directory contains scripts useful to developers when packaging, testing, or committing to Spark.
Many of these scripts require Apache credentials to work correctly.