8b94eff1ca
### What changes were proposed in this pull request? This PR upgrades Kubernetes and Minikube version for integration tests and removes/updates the old code for this new version. Details of this changes: - As [discussed in the mailing list](http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html): updating Minikube version from v0.34.1 to v1.7.3 and kubernetes version from v1.15.12 to v1.17.3. - making Minikube version checked and fail with an explanation when the test is started with on a version < v1.7.3. - removing minikube status checking code related to old Minikube versions - in the Minikube backend using fabric8's `Config.autoConfigure()` method to configure the kubernetes client to use the `minikube` k8s context (like it was in [one of the Minikube's example](https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/kubectl/equivalents/ConfigUseContext.java#L36)) - Introducing `persistentVolume` test tag: this would be a temporary change to skip PVC tests in the Kubernetes integration test, as currently the PCV tests are blocking the move to Docker as Minikube's driver (for details please check https://issues.apache.org/jira/browse/SPARK-34738). ### Why are the changes needed? With the current suggestion one can run into several problems without noticing the Minikube/kubernetes version is the problem. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? It was tested on Mac with [this script](https://gist.github.com/attilapiros/cd58a16bdde833c80c5803c337fffa94#file-check_minikube_versions-zsh) which installs each Minikube versions from v1.7.2 (including this version to test the negative case of the version check) and runs the integration tests. It was started with: ``` ./check_minikube_versions.zsh > test_log 2>&1 ``` And there was only one build failure the rest was successful: ``` $ grep "BUILD SUCCESS" test_log | wc -l 26 $ grep "BUILD FAILURE" test_log | wc -l 1 ``` It was for Minikube v1.7.2 and the log is: ``` KubernetesSuite: *** RUN ABORTED *** java.lang.AssertionError: assertion failed: Unsupported Minikube version is detected: minikube version: v1.7.2.For integration testing Minikube version 1.7.3 or greater is expected. at scala.Predef$.assert(Predef.scala:223) at org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:52) at org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33) at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:163) at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) at org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.org$scalatest$BeforeAndAfter$$super$run(KubernetesSuite.scala:43) at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273) at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271) ... ``` Moreover I made a test with having multiple k8s cluster contexts, too. Closes #31829 from attilapiros/SPARK-34736. Lead-authored-by: “attilapiros” <piros.attila.zsolt@gmail.com> Co-authored-by: attilapiros <piros.attila.zsolt@gmail.com> Signed-off-by: attilapiros <piros.attila.zsolt@gmail.com>
184 lines
4.3 KiB
Bash
Executable file
184 lines
4.3 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
set -exo errexit
|
|
TEST_ROOT_DIR=$(git rev-parse --show-toplevel)
|
|
|
|
DEPLOY_MODE="minikube"
|
|
IMAGE_REPO="docker.io/kubespark"
|
|
SPARK_TGZ="N/A"
|
|
IMAGE_TAG="N/A"
|
|
JAVA_IMAGE_TAG=
|
|
BASE_IMAGE_NAME=
|
|
JVM_IMAGE_NAME=
|
|
PYTHON_IMAGE_NAME=
|
|
R_IMAGE_NAME=
|
|
SPARK_MASTER=
|
|
NAMESPACE=
|
|
SERVICE_ACCOUNT=
|
|
CONTEXT=
|
|
INCLUDE_TAGS="k8s"
|
|
EXCLUDE_TAGS=
|
|
JAVA_VERSION="8"
|
|
BUILD_DEPENDENCIES_MVN_FLAG="-am"
|
|
HADOOP_PROFILE="hadoop-3.2"
|
|
MVN="$TEST_ROOT_DIR/build/mvn"
|
|
|
|
SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version 2>/dev/null\
|
|
| grep -v "INFO"\
|
|
| grep -v "WARNING"\
|
|
| tail -n 1)
|
|
|
|
export SCALA_VERSION
|
|
echo $SCALA_VERSION
|
|
|
|
# Parse arguments
|
|
while (( "$#" )); do
|
|
case $1 in
|
|
--image-repo)
|
|
IMAGE_REPO="$2"
|
|
shift
|
|
;;
|
|
--image-tag)
|
|
IMAGE_TAG="$2"
|
|
shift
|
|
;;
|
|
--java-image-tag)
|
|
JAVA_IMAGE_TAG="$2"
|
|
shift
|
|
;;
|
|
--deploy-mode)
|
|
DEPLOY_MODE="$2"
|
|
shift
|
|
;;
|
|
--spark-tgz)
|
|
SPARK_TGZ="$2"
|
|
shift
|
|
;;
|
|
--spark-master)
|
|
SPARK_MASTER="$2"
|
|
shift
|
|
;;
|
|
--namespace)
|
|
NAMESPACE="$2"
|
|
shift
|
|
;;
|
|
--service-account)
|
|
SERVICE_ACCOUNT="$2"
|
|
shift
|
|
;;
|
|
--context)
|
|
CONTEXT="$2"
|
|
shift
|
|
;;
|
|
--include-tags)
|
|
INCLUDE_TAGS="k8s,$2"
|
|
shift
|
|
;;
|
|
--exclude-tags)
|
|
EXCLUDE_TAGS="$2"
|
|
shift
|
|
;;
|
|
--base-image-name)
|
|
BASE_IMAGE_NAME="$2"
|
|
shift
|
|
;;
|
|
--jvm-image-name)
|
|
JVM_IMAGE_NAME="$2"
|
|
shift
|
|
;;
|
|
--python-image-name)
|
|
PYTHON_IMAGE_NAME="$2"
|
|
shift
|
|
;;
|
|
--r-image-name)
|
|
R_IMAGE_NAME="$2"
|
|
shift
|
|
;;
|
|
--java-version)
|
|
JAVA_VERSION="$2"
|
|
shift
|
|
;;
|
|
--hadoop-profile)
|
|
HADOOP_PROFILE="$2"
|
|
shift
|
|
;;
|
|
--skip-building-dependencies)
|
|
BUILD_DEPENDENCIES_MVN_FLAG=""
|
|
;;
|
|
*)
|
|
echo "Unexpected command line flag $2 $1."
|
|
exit 1
|
|
;;
|
|
esac
|
|
shift
|
|
done
|
|
|
|
properties=(
|
|
-Djava.version=$JAVA_VERSION \
|
|
-Dspark.kubernetes.test.sparkTgz=$SPARK_TGZ \
|
|
-Dspark.kubernetes.test.imageTag=$IMAGE_TAG \
|
|
-Dspark.kubernetes.test.imageRepo=$IMAGE_REPO \
|
|
-Dspark.kubernetes.test.deployMode=$DEPLOY_MODE \
|
|
-Dtest.include.tags=$INCLUDE_TAGS
|
|
)
|
|
|
|
if [ -n "$JAVA_IMAGE_TAG" ];
|
|
then
|
|
properties=( ${properties[@]} -Dspark.kubernetes.test.javaImageTag=$JAVA_IMAGE_TAG )
|
|
fi
|
|
|
|
if [ -n "$NAMESPACE" ];
|
|
then
|
|
properties=( ${properties[@]} -Dspark.kubernetes.test.namespace=$NAMESPACE )
|
|
fi
|
|
|
|
if [ -n "$SERVICE_ACCOUNT" ];
|
|
then
|
|
properties=( ${properties[@]} -Dspark.kubernetes.test.serviceAccountName=$SERVICE_ACCOUNT )
|
|
fi
|
|
|
|
if [ -n "$CONTEXT" ];
|
|
then
|
|
properties=( ${properties[@]} -Dspark.kubernetes.test.kubeConfigContext=$CONTEXT )
|
|
fi
|
|
|
|
if [ -n "$SPARK_MASTER" ];
|
|
then
|
|
properties=( ${properties[@]} -Dspark.kubernetes.test.master=$SPARK_MASTER )
|
|
fi
|
|
|
|
if [ -n "$EXCLUDE_TAGS" ];
|
|
then
|
|
properties=( ${properties[@]} -Dtest.exclude.tags=$EXCLUDE_TAGS )
|
|
fi
|
|
|
|
BASE_IMAGE_NAME=${BASE_IMAGE_NAME:-spark}
|
|
JVM_IMAGE_NAME=${JVM_IMAGE_NAME:-${BASE_IMAGE_NAME}}
|
|
PYTHON_IMAGE_NAME=${PYTHON_IMAGE_NAME:-${BASE_IMAGE_NAME}-py}
|
|
R_IMAGE_NAME=${R_IMAGE_NAME:-${BASE_IMAGE_NAME}-r}
|
|
|
|
properties+=(
|
|
-Dspark.kubernetes.test.jvmImage=$JVM_IMAGE_NAME
|
|
-Dspark.kubernetes.test.pythonImage=$PYTHON_IMAGE_NAME
|
|
-Dspark.kubernetes.test.rImage=$R_IMAGE_NAME
|
|
-Dlog4j.logger.org.apache.spark=DEBUG
|
|
)
|
|
|
|
$TEST_ROOT_DIR/build/mvn install -f $TEST_ROOT_DIR/pom.xml -pl resource-managers/kubernetes/integration-tests $BUILD_DEPENDENCIES_MVN_FLAG -Pscala-$SCALA_VERSION -P$HADOOP_PROFILE -Pkubernetes -Pkubernetes-integration-tests ${properties[@]}
|