Ye Zhou d4d8a6320f [SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching ... ### What changes were proposed in this pull request? Remove the appAttemptId from TransportConf, and parsing through SparkEnv. ### Why are the changes needed? Push based shuffle will fail if there are any attemptId set in the SparkConf, as the attemptId is not set correctly in Driver. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested within our Yarn cluster. Without this PR, the Driver will fail to finalize the shuffle merge on all the mergers. After the patch, Driver can successfully finalize the shuffle merge and the push based shuffle can work fine. Also with unit test to verify the attemptId is being set in the BlockStoreClient in Driver. Closes #34018 from zhouyejoe/SPARK-36772. Authored-by: Ye Zhou <yezhou@linkedin.com> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit cabc36b54d) Signed-off-by: Gengliang Wang <gengliang@apache.org>		2021-09-18 15:52:14 +08:00
..
src	[SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching	2021-09-18 15:52:14 +08:00
pom.xml	Preparing development version 3.2.1-SNAPSHOT	2021-08-31 17:04:14 +00:00