a0c76a8755
### What changes were proposed in this pull request? This PR aims to upgrade K8s client to 5.3.1. ### Why are the changes needed? This will bring the latest bug fixes. - https://github.com/fabric8io/kubernetes-client/releases/tag/v5.3.1 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. K8s IT is manually tested like the following. ``` KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage - Launcher client dependencies - SPARK-33615: Launcher client archives - SPARK-33748: Launcher python client respecting PYSPARK_PYTHON - SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python - Launcher python client dependencies using a zip file - Test basic decommissioning - Test basic decommissioning with shuffle cleanup - Test decommissioning with dynamic allocation & shuffle cleanups - Test decommissioning timeouts - Run SparkR on simple dataframe.R example Run completed in 18 minutes, 33 seconds. Total number of tests run: 27 Suites: completed 2, aborted 0 Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0 All tests passed. [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 3.2.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 3.959 s] [INFO] Spark Project Tags ................................. SUCCESS [ 7.830 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 3.457 s] [INFO] Spark Project Networking ........................... SUCCESS [ 5.496 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.239 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 9.006 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 2.422 s] [INFO] Spark Project Core ................................. SUCCESS [02:17 min] [INFO] Spark Project Kubernetes Integration Tests ......... SUCCESS [21:05 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 23:59 min [INFO] Finished at: 2021-05-05T11:59:19-07:00 [INFO] ------------------------------------------------------------------------ ``` Closes #32443 from dongjoon-hyun/SPARK-35319. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> |
||
---|---|---|
.. | ||
kubernetes | ||
mesos | ||
yarn |