aa4c10025a
Introduces the concept of a `MapOutputMetadata` opaque object that can be returned from map output writers. Note that this PR only proposes the API changes on the shuffle writer side. Following patches will be proposed for actually accepting the metadata on the driver and persisting it in the driver's shuffle metadata storage plugin. ### Why are the changes needed? For a more complete design discussion on this subject as a whole, refer to [this design document](https://docs.google.com/document/d/1Aj6IyMsbS2sdIfHxLvIbHUNjHIWHTabfknIPoxOrTjk/edit#). ### Does this PR introduce any user-facing change? Enables additional APIs for the shuffle storage plugin tree. Usage will become more apparent as the API evolves. ### How was this patch tested? No tests here, since this is only an API-side change that is not consumed by core Spark itself. Closes #28616 from mccheah/return-map-output-metadata. Authored-by: mcheah <mcheah@palantir.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||