spark-instrumented-optimizer/resource-managers/kubernetes
Dmytro Melnychenko 77cd133e96 [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
### What changes were proposed in this pull request?

Don't open watch when not needed

### Why are the changes needed?
In spark-submit, we currently open a pod watch for any spark submission. If WAIT_FOR_APP_COMPLETION is false, we then immediately ignore the result of the watcher and break out of the watcher.

When submitting spark applications at scale, this is a source of operational pain, since opening the watch relies on opening a websocket, which tends to run into subtle networking issues around negotiating the websocket connection.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Standard tests

Closes #34095 from slothspot/spark-35174.

Authored-by: Dmytro Melnychenko <dmytro.i.am@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
2021-09-24 11:04:59 -07:00
..
core [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false 2021-09-24 11:04:59 -07:00
docker/src/main/dockerfiles/spark [SPARK-36806][K8S][R] Use R 4.0.4 in K8s R image 2021-09-20 10:52:45 -07:00
integration-tests [SPARK-36058][K8S] Add support for statefulset APIs in K8s 2021-08-25 17:38:57 -07:00