9c157a490b
### What changes were proposed in this pull request? Currently, we calculate the `remoteBlockBytes` based on the original block info list. It's not efficient. Usually, it costs more ~25% time to be spent here. If the original reducer size is big but the actual reducer size is small due to automatically partition coalescing of AQE, the reducer will take more time to calculate `remoteBlockBytes`. We can reduce this cost via remote requests which contain merged block info lists. ### Why are the changes needed? improve task performance ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? new unit tests and verified manually. Closes #33109 from yaooqinn/SPARK-35910. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||