Dongjoon Hyun cf07036d9b [SPARK-35593][K8S][CORE] Support shuffle data recovery on the reused PVCs ... ### What changes were proposed in this pull request? Previously, the following two commits allow driver-owned on-demand PVC reuse. - SPARK-35182 Support driver-owned on-demand PVC - SPARK-35416 Support PersistentVolumeClaim Reuse This PR aims to recover the shuffle data on those remounted PVCs. The lifecycle of PVCs are tied to the one of Spark jobs. Since this is K8s specific feature, `ShuffleDataIO` plugin is used. ### Why are the changes needed? Although Pod is killed, we can remount PVCs and recover some data from it. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the newly added test cases. Closes #32730 from dongjoon-hyun/SPARK-RECOVER-SHUFFLE-DATA. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>		2021-06-10 16:06:58 -07:00
..
kubernetes	[SPARK-35593][K8S][CORE] Support shuffle data recovery on the reused PVCs	2021-06-10 16:06:58 -07:00
mesos	[SPARK-34520][CORE] Remove unused SecurityManager references	2021-02-24 20:38:03 -08:00
yarn	[SPARK-34472][YARN] Ship ivySettings file to driver in cluster mode	2021-04-20 13:35:57 -05:00