## What changes were proposed in this pull request?
This is a stripped down version of the `KubernetesClusterSchedulerBackend` for Spark with the following components:
- Static Allocation of Executors
- Executor Pod Factory
- Executor Recovery Semantics
It's step 1 from the step-wise plan documented [here](https://github.com/apache-spark-on-k8s/spark/issues/441#issuecomment-330802935).
This addition is covered by the [SPIP vote](http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-td22147.html) which passed on Aug 31 .
## How was this patch tested?
- The patch contains unit tests which are passing.
- Manual testing: `./build/mvn -Pkubernetes clean package` succeeded.
- It is a **subset** of the entire changelist hosted in http://github.com/apache-spark-on-k8s/spark which is in active use in several organizations.
- There is integration testing enabled in the fork currently [hosted by PepperData](spark-k8s-jenkins.pepperdata.org:8080) which is being moved over to RiseLAB CI.
- Detailed documentation on trying out the patch in its entirety is in: https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html
cc rxin felixcheung mateiz (shepherd)
k8s-big-data SIG members & contributors: mccheah ash211 ssuchter varunkatta kimoonkim erikerlandson liyinan926 tnachen ifilonenko
Author: Yinan Li <liyinan926@gmail.com>
Author: foxish <ramanathana@google.com>
Author: mcheah <mcheah@palantir.com>
Closes#19468 from foxish/spark-kubernetes-3.
## What changes were proposed in this pull request?
Since Spark 2.1.0, Travis CI was supported by SPARK-15207 for automated PR verification (JDK7/JDK8 maven compilation and Java Linter) and contributors can see the additional result via their Travis CI dashboard (or PC).
This PR aims to make `.travis.yml` up-to-date by removing JDK7 which was removed via SPARK-19550.
## How was this patch tested?
See the result via Travis CI.
- https://travis-ci.org/dongjoon-hyun/spark/builds/207111713
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes#17143 from dongjoon-hyun/SPARK-19801.
## What changes were proposed in this pull request?
- Remove support for Hadoop 2.5 and earlier
- Remove reflection and code constructs only needed to support multiple versions at once
- Update docs to reflect newer versions
- Remove older versions' builds and profiles.
## How was this patch tested?
Existing tests
Author: Sean Owen <sowen@cloudera.com>
Closes#16810 from srowen/SPARK-19464.
## What changes were proposed in this pull request?
Move Mesos code into a mvn module
## How was this patch tested?
unit tests
manually submitting a client mode and cluster mode job
spark/mesos integration test suite
Author: Michael Gummelt <mgummelt@mesosphere.io>
Closes#14637 from mgummelt/mesos-module.
## What changes were proposed in this pull request?
Currently, Java Linter is disabled in Jenkins tests.
https://github.com/apache/spark/blob/master/dev/run-tests.py#L554
However, as of today, Spark has 721 java files with 97362 code (without blank/comments). It's about 1/3 of Scala.
```
--------------------------------------------------------------------------------
Language files blank comment code
--------------------------------------------------------------------------------
Scala 2353 62819 124060 318747
Java 721 18617 23314 97362
```
This PR aims to take advantage of Travis CI to handle the following static analysis by adding a single file, `.travis.yml` without any additional burden on the existing servers.
- Java Linter
- JDK7/JDK8 maven compile
Note that this PR does not propose to remove some of the above work items from the Jenkins. It's possible, but we need to observe the Travis CI stability for a while. The main goal of this issue is to remove committer's overhead on linter-related PRs (the original PR and the fixation PR).
## How was this patch tested?
Pass the Travis CI tests. Please see the following link.
https://travis-ci.org/dongjoon-hyun/spark/builds/128595350https://travis-ci.org/dongjoon-hyun/spark/builds/128708372
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes#12980 from dongjoon-hyun/SPARK-15207.
This PR amortizes the cost of downloading all the jars and compiling core across more test cases. In one anecdotal run this change takes the cumulative time down from ~80 minutes to ~40 minutes.
Author: Michael Armbrust <michael@databricks.com>
Closes#255 from marmbrus/travis and squashes the following commits:
506b22d [Michael Armbrust] Cut down the granularity of travis tests so we can amortize the cost of compilation.
This is not intended to replace Jenkins immediately, and Jenkins will remain the CI of reference for merging pull requests in the near term. Long term, it is possible that Travis will give us better integration with github, so we are investigating its use.
Author: Michael Armbrust <michael@databricks.com>
Closes#230 from marmbrus/travis and squashes the following commits:
93f9a32 [Michael Armbrust] Add Apache license to .travis.yml
d7c0e78 [Michael Armbrust] Initial experimentation with Travis CI configuration