spark-instrumented-optimizer/resource-managers
Dmytro Melnychenko 77cd133e96 [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false
### What changes were proposed in this pull request?

Don't open watch when not needed

### Why are the changes needed?
In spark-submit, we currently open a pod watch for any spark submission. If WAIT_FOR_APP_COMPLETION is false, we then immediately ignore the result of the watcher and break out of the watcher.

When submitting spark applications at scale, this is a source of operational pain, since opening the watch relies on opening a websocket, which tends to run into subtle networking issues around negotiating the websocket connection.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Standard tests

Closes #34095 from slothspot/spark-35174.

Authored-by: Dmytro Melnychenko <dmytro.i.am@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
2021-09-24 11:04:59 -07:00
..
kubernetes [SPARK-35174][K8S] Avoid opening watch when waitAppCompletion is false 2021-09-24 11:04:59 -07:00
mesos [SPARK-36173][CORE] Support getting CPU number in TaskContext 2021-08-04 21:14:01 -05:00
yarn Revert "[SPARK-35672][CORE][YARN] Pass user classpath entries to executors using config instead of command line" 2021-09-24 10:35:39 +09:00