ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/common
History
Ye Zhou d4d8a6320f [SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching 
### What changes were proposed in this pull request?
Remove the appAttemptId from TransportConf, and parsing through SparkEnv.

### Why are the changes needed?
Push based shuffle will fail if there are any attemptId set in the SparkConf, as the attemptId is not set correctly in Driver.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Tested within our Yarn cluster. Without this PR, the Driver will fail to finalize the shuffle merge on all the mergers. After the patch, Driver can successfully finalize the shuffle merge and the push based shuffle can work fine.
Also with unit test to verify the attemptId is being set in the BlockStoreClient in Driver.

Closes #34018 from zhouyejoe/SPARK-36772.

Authored-by: Ye Zhou <yezhou@linkedin.com>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
(cherry picked from commit cabc36b54d)
Signed-off-by: Gengliang Wang <gengliang@apache.org>
2021-09-18 15:52:14 +08:00
..
kvstore Preparing development version 3.2.1-SNAPSHOT 2021-08-31 17:04:14 +00:00
network-common [SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching 2021-09-18 15:52:14 +08:00
network-shuffle [SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching 2021-09-18 15:52:14 +08:00
network-yarn Preparing development version 3.2.1-SNAPSHOT 2021-08-31 17:04:14 +00:00
sketch Preparing development version 3.2.1-SNAPSHOT 2021-08-31 17:04:14 +00:00
tags Preparing development version 3.2.1-SNAPSHOT 2021-08-31 17:04:14 +00:00
unsafe [SPARK-36704][CORE] Expand exception handling to more Java 9 cases where reflection is limited at runtime, when reflecting to manage DirectByteBuffer settings 2021-09-11 13:38:20 -05:00