1299c8a81d
### What changes were proposed in this pull request? Spark has a hardcode list to contain known shuffle managers, which has two values now. It does not contain user's custom shuffle manager which is set through Spark config "spark.shuffle.manager". We hit issue when set "spark.shuffle.manager" with our own shuffle manager plugin (Uber Remote Shuffle Service implementation, https://github.com/uber/RemoteShuffleService). Other users will hit same issue when they implement their own shuffle manager. It is better to remove that knownManagers hardcode list, to support user's custom shuffle manager implementation. ### Why are the changes needed? Spark has shuffle manager API to support custom shuffle manager implementation. The hardcoded known managers list does not consider that shuffle manager config value which could be set by user. Thus better to remove this hardcoded known managers list. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Current Spark unit test already covers the code path. Closes #29916 from boy-uber/knownManagers. Lead-authored-by: Bo Yang <boy@uber.com> Co-authored-by: Bo Yang <boy-uber@users.noreply.github.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> |
||
|---|---|---|
| .. | ||
| kvstore | ||
| network-common | ||
| network-shuffle | ||
| network-yarn | ||
| sketch | ||
| tags | ||
| unsafe | ||