Ye Zhou 1907f0ac57 [SPARK-35546][SHUFFLE] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way ... ### What changes were proposed in this pull request? This is one of the patches for SPIP SPARK-30602 which is needed for push-based shuffle. ### Summary of the change: When Executor registers with Shuffle Service, it will encode the merged shuffle dir created and also the application attemptId into the ShuffleManagerMeta into Json. Then in Shuffle Service, it will decode the Json string and get the correct merged shuffle dir and also the attemptId. If the registration comes from a newer attempt, the merged shuffle information will be updated to store the information from the newer attempt. This PR also refactored the management of the merged shuffle information to avoid concurrency issues. ### Why are the changes needed? Refer to the SPIP in SPARK-30602. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added unit tests. The reference PR with the consolidated changes covering the complete implementation is also provided in SPARK-30602. We have already verified the functionality and the improved performance as documented in the SPIP doc. Closes #33078 from zhouyejoe/SPARK-35546. Authored-by: Ye Zhou <yezhou@linkedin.com> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com> (cherry picked from commit c77acf0bbc) Signed-off-by: Mridul Muralidharan <mridulatgmail.com>		2021-07-20 00:04:16 -05:00
..
kvstore	[SPARK-35824][CORE][TESTS] Convert LevelDBSuite.IntKeyType from a nested class to a normal class	2021-06-19 11:36:01 -07:00
network-common	[SPARK-35546][SHUFFLE] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way	2021-07-20 00:04:16 -05:00
network-shuffle	[SPARK-35546][SHUFFLE] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way	2021-07-20 00:04:16 -05:00
network-yarn	[SPARK-35258][SHUFFLE][YARN] Add new metrics to ExternalShuffleService for better monitoring	2021-06-28 02:36:17 -05:00
sketch	[SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters	2021-06-17 06:29:33 +00:00
tags	[SPARK-34578][SQL][TESTS][TEST-MAVEN] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library	2021-03-02 16:52:27 +09:00
unsafe	[SPARK-36081][SPARK-36066][SQL] Update the document about the behavior change of trimming characters for cast	2021-07-13 20:29:05 +08:00