From afa6aee4f5ea270db5331e48ad08e0b176cdd2a0 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Mon, 26 Oct 2020 15:29:12 -0700 Subject: [PATCH] [SPARK-33237][K8S][TESTS] Use default Hadoop-3.2 profile from K8s IT Jenkins job ### What changes were proposed in this pull request? This PR aims to use `hadoop-3.2` profile in K8s IT Jenkins jobs. - [x] Switch the default value of `HADOOP_PROFILE` from `hadoop-2.7` to `hadoop-3.2`. - [x] Remove `-Phadoop2.7` from Jenkins K8s IT job. - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/configure **BEFORE** ``` ./dev/make-distribution.sh --name ${DATE}-${REVISION} --r --pip --tgz -DzincPort=${ZINC_PORT} \ -Phadoop-2.7 -Pkubernetes -Pkinesis-asl -Phive -Phive-thriftserver ``` **AFTER** ``` ./dev/make-distribution.sh --name ${DATE}-${REVISION} --r --pip --tgz -DzincPort=${ZINC_PORT} \ -Pkubernetes -Pkinesis-asl -Phive -Phive-thriftserver ``` ### Why are the changes needed? Since Apache Spark 3.1.0, Hadoop 3 is the default. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Check the Jenkins K8s IT log and result. - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34899/ ``` + /home/jenkins/workspace/SparkPullRequestBuilder-K8s/build/mvn clean package -DskipTests -DzincPort=4021 -Pkubernetes -Pkinesis-asl -Phive -Phive-thriftserver Using `mvn` from path: /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.6.3/bin/mvn [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] ``` Closes #30153 from dongjoon-hyun/SPARK-33237. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- .../integration-tests/dev/dev-run-integration-tests.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh b/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh index 9c03a97ef1..b72a4f7491 100755 --- a/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh +++ b/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh @@ -35,7 +35,7 @@ CONTEXT= INCLUDE_TAGS="k8s" EXCLUDE_TAGS= JAVA_VERSION="8" -HADOOP_PROFILE="hadoop-2.7" +HADOOP_PROFILE="hadoop-3.2" MVN="$TEST_ROOT_DIR/build/mvn" SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version 2>/dev/null\