[SPARK-35462][BUILD][K8S] Upgrade Kubernetes-client to 5.4.0 to support K8s 1.21 models
### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` from 5.3.1 to 5.4.0 to support K8s 1.21 models officially. ### Why are the changes needed? `kubernetes-client` 5.4.0 has `Kubernetes Model v1.21.0` - https://github.com/fabric8io/kubernetes-client/releases/tag/v5.4.0 ### Does this PR introduce _any_ user-facing change? No. This is a dev-only change. ### How was this patch tested? Pass the CIs including Jenkins K8s IT. - https://github.com/apache/spark/pull/32612#issuecomment-845456039 I tested K8s IT with the following versions. - minikube version: v1.20.0 - K8s Client Version: v1.21.0 - Server Version: v1.21.0 ``` KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - Launcher client dependencies - SPARK-33615: Launcher client archives - SPARK-33748: Launcher python client respecting PYSPARK_PYTHON - SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python - Launcher python client dependencies using a zip file - Test basic decommissioning - Test basic decommissioning with shuffle cleanup - Test decommissioning with dynamic allocation & shuffle cleanups - Test decommissioning timeouts - Run SparkR on simple dataframe.R example Run completed in 17 minutes, 18 seconds. Total number of tests run: 26 Suites: completed 2, aborted 0 Tests: succeeded 26, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #32612 from dongjoon-hyun/SPARK-35462. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
Dongjoon Hyun
parent
38fbc0b4f7
commit
3757c1803d
|
|
@ -154,26 +154,27 @@ jsr305/3.0.0//jsr305-3.0.0.jar
|
|||
jta/1.1//jta-1.1.jar
|
||||
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
|
||||
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
|
||||
kubernetes-client/5.3.1//kubernetes-client-5.3.1.jar
|
||||
kubernetes-model-admissionregistration/5.3.1//kubernetes-model-admissionregistration-5.3.1.jar
|
||||
kubernetes-model-apiextensions/5.3.1//kubernetes-model-apiextensions-5.3.1.jar
|
||||
kubernetes-model-apps/5.3.1//kubernetes-model-apps-5.3.1.jar
|
||||
kubernetes-model-autoscaling/5.3.1//kubernetes-model-autoscaling-5.3.1.jar
|
||||
kubernetes-model-batch/5.3.1//kubernetes-model-batch-5.3.1.jar
|
||||
kubernetes-model-certificates/5.3.1//kubernetes-model-certificates-5.3.1.jar
|
||||
kubernetes-model-common/5.3.1//kubernetes-model-common-5.3.1.jar
|
||||
kubernetes-model-coordination/5.3.1//kubernetes-model-coordination-5.3.1.jar
|
||||
kubernetes-model-core/5.3.1//kubernetes-model-core-5.3.1.jar
|
||||
kubernetes-model-discovery/5.3.1//kubernetes-model-discovery-5.3.1.jar
|
||||
kubernetes-model-events/5.3.1//kubernetes-model-events-5.3.1.jar
|
||||
kubernetes-model-extensions/5.3.1//kubernetes-model-extensions-5.3.1.jar
|
||||
kubernetes-model-metrics/5.3.1//kubernetes-model-metrics-5.3.1.jar
|
||||
kubernetes-model-networking/5.3.1//kubernetes-model-networking-5.3.1.jar
|
||||
kubernetes-model-node/5.3.1//kubernetes-model-node-5.3.1.jar
|
||||
kubernetes-model-policy/5.3.1//kubernetes-model-policy-5.3.1.jar
|
||||
kubernetes-model-rbac/5.3.1//kubernetes-model-rbac-5.3.1.jar
|
||||
kubernetes-model-scheduling/5.3.1//kubernetes-model-scheduling-5.3.1.jar
|
||||
kubernetes-model-storageclass/5.3.1//kubernetes-model-storageclass-5.3.1.jar
|
||||
kubernetes-client/5.4.0//kubernetes-client-5.4.0.jar
|
||||
kubernetes-model-admissionregistration/5.4.0//kubernetes-model-admissionregistration-5.4.0.jar
|
||||
kubernetes-model-apiextensions/5.4.0//kubernetes-model-apiextensions-5.4.0.jar
|
||||
kubernetes-model-apps/5.4.0//kubernetes-model-apps-5.4.0.jar
|
||||
kubernetes-model-autoscaling/5.4.0//kubernetes-model-autoscaling-5.4.0.jar
|
||||
kubernetes-model-batch/5.4.0//kubernetes-model-batch-5.4.0.jar
|
||||
kubernetes-model-certificates/5.4.0//kubernetes-model-certificates-5.4.0.jar
|
||||
kubernetes-model-common/5.4.0//kubernetes-model-common-5.4.0.jar
|
||||
kubernetes-model-coordination/5.4.0//kubernetes-model-coordination-5.4.0.jar
|
||||
kubernetes-model-core/5.4.0//kubernetes-model-core-5.4.0.jar
|
||||
kubernetes-model-discovery/5.4.0//kubernetes-model-discovery-5.4.0.jar
|
||||
kubernetes-model-events/5.4.0//kubernetes-model-events-5.4.0.jar
|
||||
kubernetes-model-extensions/5.4.0//kubernetes-model-extensions-5.4.0.jar
|
||||
kubernetes-model-flowcontrol/5.4.0//kubernetes-model-flowcontrol-5.4.0.jar
|
||||
kubernetes-model-metrics/5.4.0//kubernetes-model-metrics-5.4.0.jar
|
||||
kubernetes-model-networking/5.4.0//kubernetes-model-networking-5.4.0.jar
|
||||
kubernetes-model-node/5.4.0//kubernetes-model-node-5.4.0.jar
|
||||
kubernetes-model-policy/5.4.0//kubernetes-model-policy-5.4.0.jar
|
||||
kubernetes-model-rbac/5.4.0//kubernetes-model-rbac-5.4.0.jar
|
||||
kubernetes-model-scheduling/5.4.0//kubernetes-model-scheduling-5.4.0.jar
|
||||
kubernetes-model-storageclass/5.4.0//kubernetes-model-storageclass-5.4.0.jar
|
||||
lapack/2.2.0//lapack-2.2.0.jar
|
||||
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
|
||||
libfb303/0.9.3//libfb303-0.9.3.jar
|
||||
|
|
|
|||
|
|
@ -125,26 +125,27 @@ jsr305/3.0.0//jsr305-3.0.0.jar
|
|||
jta/1.1//jta-1.1.jar
|
||||
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
|
||||
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
|
||||
kubernetes-client/5.3.1//kubernetes-client-5.3.1.jar
|
||||
kubernetes-model-admissionregistration/5.3.1//kubernetes-model-admissionregistration-5.3.1.jar
|
||||
kubernetes-model-apiextensions/5.3.1//kubernetes-model-apiextensions-5.3.1.jar
|
||||
kubernetes-model-apps/5.3.1//kubernetes-model-apps-5.3.1.jar
|
||||
kubernetes-model-autoscaling/5.3.1//kubernetes-model-autoscaling-5.3.1.jar
|
||||
kubernetes-model-batch/5.3.1//kubernetes-model-batch-5.3.1.jar
|
||||
kubernetes-model-certificates/5.3.1//kubernetes-model-certificates-5.3.1.jar
|
||||
kubernetes-model-common/5.3.1//kubernetes-model-common-5.3.1.jar
|
||||
kubernetes-model-coordination/5.3.1//kubernetes-model-coordination-5.3.1.jar
|
||||
kubernetes-model-core/5.3.1//kubernetes-model-core-5.3.1.jar
|
||||
kubernetes-model-discovery/5.3.1//kubernetes-model-discovery-5.3.1.jar
|
||||
kubernetes-model-events/5.3.1//kubernetes-model-events-5.3.1.jar
|
||||
kubernetes-model-extensions/5.3.1//kubernetes-model-extensions-5.3.1.jar
|
||||
kubernetes-model-metrics/5.3.1//kubernetes-model-metrics-5.3.1.jar
|
||||
kubernetes-model-networking/5.3.1//kubernetes-model-networking-5.3.1.jar
|
||||
kubernetes-model-node/5.3.1//kubernetes-model-node-5.3.1.jar
|
||||
kubernetes-model-policy/5.3.1//kubernetes-model-policy-5.3.1.jar
|
||||
kubernetes-model-rbac/5.3.1//kubernetes-model-rbac-5.3.1.jar
|
||||
kubernetes-model-scheduling/5.3.1//kubernetes-model-scheduling-5.3.1.jar
|
||||
kubernetes-model-storageclass/5.3.1//kubernetes-model-storageclass-5.3.1.jar
|
||||
kubernetes-client/5.4.0//kubernetes-client-5.4.0.jar
|
||||
kubernetes-model-admissionregistration/5.4.0//kubernetes-model-admissionregistration-5.4.0.jar
|
||||
kubernetes-model-apiextensions/5.4.0//kubernetes-model-apiextensions-5.4.0.jar
|
||||
kubernetes-model-apps/5.4.0//kubernetes-model-apps-5.4.0.jar
|
||||
kubernetes-model-autoscaling/5.4.0//kubernetes-model-autoscaling-5.4.0.jar
|
||||
kubernetes-model-batch/5.4.0//kubernetes-model-batch-5.4.0.jar
|
||||
kubernetes-model-certificates/5.4.0//kubernetes-model-certificates-5.4.0.jar
|
||||
kubernetes-model-common/5.4.0//kubernetes-model-common-5.4.0.jar
|
||||
kubernetes-model-coordination/5.4.0//kubernetes-model-coordination-5.4.0.jar
|
||||
kubernetes-model-core/5.4.0//kubernetes-model-core-5.4.0.jar
|
||||
kubernetes-model-discovery/5.4.0//kubernetes-model-discovery-5.4.0.jar
|
||||
kubernetes-model-events/5.4.0//kubernetes-model-events-5.4.0.jar
|
||||
kubernetes-model-extensions/5.4.0//kubernetes-model-extensions-5.4.0.jar
|
||||
kubernetes-model-flowcontrol/5.4.0//kubernetes-model-flowcontrol-5.4.0.jar
|
||||
kubernetes-model-metrics/5.4.0//kubernetes-model-metrics-5.4.0.jar
|
||||
kubernetes-model-networking/5.4.0//kubernetes-model-networking-5.4.0.jar
|
||||
kubernetes-model-node/5.4.0//kubernetes-model-node-5.4.0.jar
|
||||
kubernetes-model-policy/5.4.0//kubernetes-model-policy-5.4.0.jar
|
||||
kubernetes-model-rbac/5.4.0//kubernetes-model-rbac-5.4.0.jar
|
||||
kubernetes-model-scheduling/5.4.0//kubernetes-model-scheduling-5.4.0.jar
|
||||
kubernetes-model-storageclass/5.4.0//kubernetes-model-storageclass-5.4.0.jar
|
||||
lapack/2.2.0//lapack-2.2.0.jar
|
||||
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
|
||||
libfb303/0.9.3//libfb303-0.9.3.jar
|
||||
|
|
|
|||
2
pom.xml
2
pom.xml
|
|
@ -204,7 +204,7 @@
|
|||
<arrow.version>2.0.0</arrow.version>
|
||||
<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
|
||||
<leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
|
||||
<kubernetes-client.version>5.3.1</kubernetes-client.version>
|
||||
<kubernetes-client.version>5.4.0</kubernetes-client.version>
|
||||
|
||||
<test.java.home>${java.home}</test.java.home>
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue