e9b2070ab2
## What changes were proposed in this pull request? This is a stripped down version of the `KubernetesClusterSchedulerBackend` for Spark with the following components: - Static Allocation of Executors - Executor Pod Factory - Executor Recovery Semantics It's step 1 from the step-wise plan documented [here](https://github.com/apache-spark-on-k8s/spark/issues/441#issuecomment-330802935). This addition is covered by the [SPIP vote](http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-td22147.html) which passed on Aug 31 . ## How was this patch tested? - The patch contains unit tests which are passing. - Manual testing: `./build/mvn -Pkubernetes clean package` succeeded. - It is a **subset** of the entire changelist hosted in http://github.com/apache-spark-on-k8s/spark which is in active use in several organizations. - There is integration testing enabled in the fork currently [hosted by PepperData](spark-k8s-jenkins.pepperdata.org:8080) which is being moved over to RiseLAB CI. - Detailed documentation on trying out the patch in its entirety is in: https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html cc rxin felixcheung mateiz (shepherd) k8s-big-data SIG members & contributors: mccheah ash211 ssuchter varunkatta kimoonkim erikerlandson liyinan926 tnachen ifilonenko Author: Yinan Li <liyinan926@gmail.com> Author: foxish <ramanathana@google.com> Author: mcheah <mcheah@palantir.com> Closes #19468 from foxish/spark-kubernetes-3.
51 lines
1.7 KiB
YAML
51 lines
1.7 KiB
YAML
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
# Spark provides this Travis CI configuration file to help contributors
|
|
# check Scala/Java style conformance and JDK7/8 compilation easily
|
|
# during their preparing pull requests.
|
|
# - Scalastyle is executed during `maven install` implicitly.
|
|
# - Java Checkstyle is executed by `lint-java`.
|
|
# See the related discussion here.
|
|
# https://github.com/apache/spark/pull/12980
|
|
|
|
# 1. Choose OS (Ubuntu 14.04.3 LTS Server Edition 64bit, ~2 CORE, 7.5GB RAM)
|
|
sudo: required
|
|
dist: trusty
|
|
|
|
# 2. Choose language and target JDKs for parallel builds.
|
|
language: java
|
|
jdk:
|
|
- oraclejdk8
|
|
|
|
# 3. Setup cache directory for SBT and Maven.
|
|
cache:
|
|
directories:
|
|
- $HOME/.sbt
|
|
- $HOME/.m2
|
|
|
|
# 4. Turn off notifications.
|
|
notifications:
|
|
email: false
|
|
|
|
# 5. Run maven install before running lint-java.
|
|
install:
|
|
- export MAVEN_SKIP_RC=1
|
|
- build/mvn -T 4 -q -DskipTests -Pkubernetes -Pmesos -Pyarn -Pkinesis-asl -Phive -Phive-thriftserver install
|
|
|
|
# 6. Run lint-java.
|
|
script:
|
|
- dev/lint-java
|