Ye Zhou cabc36b54d [SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching ... ### What changes were proposed in this pull request? Remove the appAttemptId from TransportConf, and parsing through SparkEnv. ### Why are the changes needed? Push based shuffle will fail if there are any attemptId set in the SparkConf, as the attemptId is not set correctly in Driver. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested within our Yarn cluster. Without this PR, the Driver will fail to finalize the shuffle merge on all the mergers. After the patch, Driver can successfully finalize the shuffle merge and the push based shuffle can work fine. Also with unit test to verify the attemptId is being set in the BlockStoreClient in Driver. Closes #34018 from zhouyejoe/SPARK-36772. Authored-by: Ye Zhou <yezhou@linkedin.com> Signed-off-by: Gengliang Wang <gengliang@apache.org>		2021-09-18 15:51:57 +08:00
..
kvstore	[SPARK-36603][CORE] Use WeakReference not SoftReference in LevelDB	2021-08-29 09:29:23 -07:00
network-common	[SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching	2021-09-18 15:51:57 +08:00
network-shuffle	[SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching	2021-09-18 15:51:57 +08:00
network-yarn	[SPARK-36460][SHUFFLE] Pull out NoOpMergedShuffleFileManager inner class outside	2021-08-10 10:19:24 +08:00
sketch	[SPARK-35996][BUILD] Setting version to 3.3.0-SNAPSHOT	2021-07-02 13:47:36 -07:00
tags	[SPARK-35996][BUILD] Setting version to 3.3.0-SNAPSHOT	2021-07-02 13:47:36 -07:00
unsafe	[SPARK-36704][CORE] Expand exception handling to more Java 9 cases where reflection is limited at runtime, when reflecting to manage DirectByteBuffer settings	2021-09-11 13:38:10 -05:00