571a6f0574
## What changes were proposed in this pull request? Support client mode for the Kubernetes scheduler. Client mode works more or less identically to cluster mode. However, in client mode, the Spark Context needs to be manually bootstrapped with certain properties which would have otherwise been set up by spark-submit in cluster mode. Specifically: - If the user doesn't provide a driver pod name, we don't add an owner reference. This is for usage when the driver is not running in a pod in the cluster. In such a case, the driver can only provide a best effort to clean up the executors when the driver exits, but cleaning up the resources is not guaranteed. The executor JVMs should exit if the driver JVM exits, but the pods will still remain in the cluster in a COMPLETED or FAILED state. - The user must provide a host (spark.driver.host) and port (spark.driver.port) that the executors can connect to. When using spark-submit in cluster mode, spark-submit generates the headless service automatically; in client mode, the user is responsible for setting up their own connectivity. We also change the authentication configuration prefixes for client mode. ## How was this patch tested? Adding an integration test to exercise client mode support. Author: mcheah <mcheah@palantir.com> Closes #21748 from mccheah/k8s-client-mode. |
||
|---|---|---|
| .. | ||
| dev | ||
| scripts | ||
| src/test | ||
| pom.xml | ||
| README.md | ||
| layout | title |
|---|---|
| global | Spark on Kubernetes Integration Tests |
Running the Kubernetes Integration Tests
Note that the integration test framework is currently being heavily revised and is subject to change. Note that currently the integration tests only run with Java 8.
The simplest way to run the integration tests is to install and run Minikube, then run the following:
dev/dev-run-integration-tests.sh
The minimum tested version of Minikube is 0.23.0. The kube-dns addon must be enabled. Minikube should run with a minimum of 3 CPUs and 4G of memory:
minikube start --cpus 3 --memory 4096
You can download Minikube here.
Integration test customization
Configuration of the integration test runtime is done through passing different arguments to the test script. The main useful options are outlined below.
Re-using Docker Images
By default, the test framework will build new Docker images on every test execution. A unique image tag is generated,
and it is written to file at target/imageTag.txt. To reuse the images built in a previous run, or to use a Docker image tag
that you have built by other means already, pass the tag to the test script:
dev/dev-run-integration-tests.sh --image-tag <tag>
where if you still want to use images that were built before by the test framework:
dev/dev-run-integration-tests.sh --image-tag $(cat target/imageTag.txt)
Spark Distribution Under Test
The Spark code to test is handed to the integration test system via a tarball. Here is the option that is used to specify the tarball:
--spark-tgz <path-to-tgz>- set<path-to-tgz>to point to a tarball containing the Spark distribution to test.
TODO: Don't require the packaging of the built Spark artifacts into this tarball, just read them out of the current tree.
Customizing the Namespace and Service Account
--namespace <namespace>- set<namespace>to the namespace in which the tests should be run.--service-account <service account name>- set<service account name>to the name of the Kubernetes service account to use in the namespace specified by the--namespace. The service account is expected to have permissions to get, list, watch, and create pods. For clusters with RBAC turned on, it's important that the right permissions are granted to the service account in the namespace through an appropriate role and role binding. A reference RBAC configuration is provided indev/spark-rbac.yaml.