d273a2bb0f
This PR is based on an existing/previou PR - https://github.com/apache/spark/pull/19045 ### What changes were proposed in this pull request? This changes adds a decommissioning state that we can enter when the cloud provider/scheduler lets us know we aren't going to be removed immediately but instead will be removed soon. This concept fits nicely in K8s and also with spot-instances on AWS / preemptible instances all of which we can get a notice that our host is going away. For now we simply stop scheduling jobs, in the future we could perform some kind of migration of data during scale-down, or at least stop accepting new blocks to cache. There is a design document at https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?usp=sharing ### Why are the changes needed? With more move to preemptible multi-tenancy, serverless environments, and spot-instances better handling of node scale down is required. ### Does this PR introduce any user-facing change? There is no API change, however an additional configuration flag is added to enable/disable this behaviour. ### How was this patch tested? New integration tests in the Spark K8s integration testing. Extension of the AppClientSuite to test decommissioning seperate from the K8s. Closes #26440 from holdenk/SPARK-20628-keep-track-of-nodes-which-are-going-to-be-shutdown-r4. Lead-authored-by: Holden Karau <hkarau@apple.com> Co-authored-by: Holden Karau <holden@pigscanfly.ca> Signed-off-by: Holden Karau <hkarau@apple.com> |
||
---|---|---|
.. | ||
decommission-slave.sh | ||
slaves.sh | ||
spark-config.sh | ||
spark-daemon.sh | ||
spark-daemons.sh | ||
start-all.sh | ||
start-history-server.sh | ||
start-master.sh | ||
start-mesos-dispatcher.sh | ||
start-mesos-shuffle-service.sh | ||
start-slave.sh | ||
start-slaves.sh | ||
start-thriftserver.sh | ||
stop-all.sh | ||
stop-history-server.sh | ||
stop-master.sh | ||
stop-mesos-dispatcher.sh | ||
stop-mesos-shuffle-service.sh | ||
stop-slave.sh | ||
stop-slaves.sh | ||
stop-thriftserver.sh |