Venkata krishnan Sowrirajan 0a98e51b84 [SPARK-32923][FOLLOW-UP] Clean up older shuffleMergeId shuffle files when finalize request for higher shuffleMergeId is received ... ### What changes were proposed in this pull request? Clean up older shuffleMergeId shuffle files when finalize request for higher shuffleMergeId is received when no blocks pushed for the corresponding shuffleMergeId. This is identified as part of https://github.com/apache/spark/pull/33034#discussion_r680610872. ### Why are the changes needed? Without this change, older shuffleMergeId files won't be cleaned up properly. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added changes to existing unit test to address this case. Closes #33605 from venkata91/SPARK-32923-follow-on. Authored-by: Venkata krishnan Sowrirajan <vsowrirajan@linkedin.com> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com> (cherry picked from commit d8169493b6) Signed-off-by: Mridul Muralidharan <mridulatgmail.com>		2021-08-04 03:30:56 -05:00
..
kvstore	[SPARK-35824][CORE][TESTS] Convert LevelDBSuite.IntKeyType from a nested class to a normal class	2021-06-19 11:36:01 -07:00
network-common	[SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage retries for push-based shuffle	2021-08-01 23:30:02 -05:00
network-shuffle	[SPARK-32923][FOLLOW-UP] Clean up older shuffleMergeId shuffle files when finalize request for higher shuffleMergeId is received	2021-08-04 03:30:56 -05:00
network-yarn	[SPARK-35259][SHUFFLE] Update ExternalBlockHandler Timer variables to expose correct units	2021-07-24 21:27:56 +08:00
sketch	[SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters	2021-06-17 06:29:33 +00:00
tags	[SPARK-34578][SQL][TESTS][TEST-MAVEN] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library	2021-03-02 16:52:27 +09:00
unsafe	[SPARK-36081][SPARK-36066][SQL] Update the document about the behavior change of trimming characters for cast	2021-07-13 20:29:05 +08:00