0042ad575a
### What changes were proposed in this pull request? We added shuffle block fetch optimization in SPARK-9853. In ShuffleBlockFetcherIterator, we merge single blocks into batch blocks. During merging, we should count merged blocks for `maxBlocksInFlightPerAddress`, not original single blocks. ### Why are the changes needed? If `maxBlocksInFlightPerAddress` is specified, like set it to 1, it should mean one batch block, not one original single block. Otherwise, it will conflict with batch shuffle fetch. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Unit test. Closes #26930 from viirya/respect-max-blocks-inflight. Lead-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Liang-Chi Hsieh <liangchi@uber.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> |
||
|---|---|---|
| .. | ||
| kvstore | ||
| network-common | ||
| network-shuffle | ||
| network-yarn | ||
| sketch | ||
| tags | ||
| unsafe | ||