cdd694c52b
## What changes were proposed in this pull request? ### Background For the current status, the test script that generates coverage information was merged into Spark, https://github.com/apache/spark/pull/20204 So, we can generate the coverage report and site by, for example: ``` run-tests-with-coverage --python-executables=python3 --modules=pyspark-sql ``` like `run-tests` script in `./python`. ### Proposed change The next step is to host this coverage report via `github.io` automatically by Jenkins (see https://spark-test.github.io/pyspark-coverage-site/). This uses my testing account for Spark, spark-test, which is shared to Felix and Shivaram a long time ago for testing purpose including AppVeyor. To cut this short, this PR targets to run the coverage in [spark-master-test-sbt-hadoop-2.7](https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/) In the specific job, it will clone the page, and rebase the up-to-date PySpark test coverage from the latest commit. For instance as below: ```bash # Clone PySpark coverage site. git clone https://github.com/spark-test/pyspark-coverage-site.git # Remove existing HTMLs. rm -fr pyspark-coverage-site/* # Copy generated coverage HTMLs. cp -r .../python/test_coverage/htmlcov/* pyspark-coverage-site/ # Check out to a temporary branch. git symbolic-ref HEAD refs/heads/latest_branch # Add all the files. git add -A # Commit current HTMLs. git commit -am "Coverage report at latest commit in Apache Spark" # Delete the old branch. git branch -D gh-pages # Rename the temporary branch to master. git branch -m gh-pages # Finally, force update to our repository. git push -f origin gh-pages ``` So, it is a one single up-to-date coverage can be shown in the `github-io` page. The commands above were manually tested. ### TODOs - [x] Write a draft HyukjinKwon - [x] `pip install coverage` to all python implementations (pypy, python2, python3) in Jenkins workers - shaneknapp - [x] Set hidden `SPARK_TEST_KEY` for spark-test's password in Jenkins via Jenkins's feature This should be set in both PR builder and `spark-master-test-sbt-hadoop-2.7` so that later other PRs can test and fix the bugs - shaneknapp - [x] Set an environment variable that indicates `spark-master-test-sbt-hadoop-2.7` so that that specific build can report and update the coverage site - shaneknapp - [x] Make PR builder's test passed HyukjinKwon - [x] Fix flaky test related with coverage HyukjinKwon - 6 consecutive passes out of 7 runs This PR will be co-authored with me and shaneknapp ## How was this patch tested? It will be tested via Jenkins. Closes #23117 from HyukjinKwon/SPARK-7721. Lead-authored-by: Hyukjin Kwon <gurwls223@apache.org> Co-authored-by: hyukjinkwon <gurwls223@apache.org> Co-authored-by: shane knapp <incomplete@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
110 lines
4.5 KiB
Markdown
110 lines
4.5 KiB
Markdown
# Apache Spark
|
|
|
|
[](https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7)
|
|
[](https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark)
|
|
[](https://spark-test.github.io/pyspark-coverage-site)
|
|
|
|
Spark is a fast and general cluster computing system for Big Data. It provides
|
|
high-level APIs in Scala, Java, Python, and R, and an optimized engine that
|
|
supports general computation graphs for data analysis. It also supports a
|
|
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
|
|
MLlib for machine learning, GraphX for graph processing,
|
|
and Spark Streaming for stream processing.
|
|
|
|
<http://spark.apache.org/>
|
|
|
|
|
|
## Online Documentation
|
|
|
|
You can find the latest Spark documentation, including a programming
|
|
guide, on the [project web page](http://spark.apache.org/documentation.html).
|
|
This README file only contains basic setup instructions.
|
|
|
|
## Building Spark
|
|
|
|
Spark is built using [Apache Maven](http://maven.apache.org/).
|
|
To build Spark and its example programs, run:
|
|
|
|
build/mvn -DskipTests clean package
|
|
|
|
(You do not need to do this if you downloaded a pre-built package.)
|
|
|
|
You can build Spark using more than one thread by using the -T option with Maven, see ["Parallel builds in Maven 3"](https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3).
|
|
More detailed documentation is available from the project site, at
|
|
["Building Spark"](http://spark.apache.org/docs/latest/building-spark.html).
|
|
|
|
For general development tips, including info on developing Spark using an IDE, see ["Useful Developer Tools"](http://spark.apache.org/developer-tools.html).
|
|
|
|
## Interactive Scala Shell
|
|
|
|
The easiest way to start using Spark is through the Scala shell:
|
|
|
|
./bin/spark-shell
|
|
|
|
Try the following command, which should return 1000:
|
|
|
|
scala> sc.parallelize(1 to 1000).count()
|
|
|
|
## Interactive Python Shell
|
|
|
|
Alternatively, if you prefer Python, you can use the Python shell:
|
|
|
|
./bin/pyspark
|
|
|
|
And run the following command, which should also return 1000:
|
|
|
|
>>> sc.parallelize(range(1000)).count()
|
|
|
|
## Example Programs
|
|
|
|
Spark also comes with several sample programs in the `examples` directory.
|
|
To run one of them, use `./bin/run-example <class> [params]`. For example:
|
|
|
|
./bin/run-example SparkPi
|
|
|
|
will run the Pi example locally.
|
|
|
|
You can set the MASTER environment variable when running examples to submit
|
|
examples to a cluster. This can be a mesos:// or spark:// URL,
|
|
"yarn" to run on YARN, and "local" to run
|
|
locally with one thread, or "local[N]" to run locally with N threads. You
|
|
can also use an abbreviated class name if the class is in the `examples`
|
|
package. For instance:
|
|
|
|
MASTER=spark://host:7077 ./bin/run-example SparkPi
|
|
|
|
Many of the example programs print usage help if no params are given.
|
|
|
|
## Running Tests
|
|
|
|
Testing first requires [building Spark](#building-spark). Once Spark is built, tests
|
|
can be run using:
|
|
|
|
./dev/run-tests
|
|
|
|
Please see the guidance on how to
|
|
[run tests for a module, or individual tests](http://spark.apache.org/developer-tools.html#individual-tests).
|
|
|
|
There is also a Kubernetes integration test, see resource-managers/kubernetes/integration-tests/README.md
|
|
|
|
## A Note About Hadoop Versions
|
|
|
|
Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported
|
|
storage systems. Because the protocols have changed in different versions of
|
|
Hadoop, you must build Spark against the same version that your cluster runs.
|
|
|
|
Please refer to the build documentation at
|
|
["Specifying the Hadoop Version and Enabling YARN"](http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn)
|
|
for detailed guidance on building for a particular distribution of Hadoop, including
|
|
building for particular Hive and Hive Thriftserver distributions.
|
|
|
|
## Configuration
|
|
|
|
Please refer to the [Configuration Guide](http://spark.apache.org/docs/latest/configuration.html)
|
|
in the online documentation for an overview on how to configure Spark.
|
|
|
|
## Contributing
|
|
|
|
Please review the [Contribution to Spark guide](http://spark.apache.org/contributing.html)
|
|
for information on how to get started contributing to the project.
|