0811666ab1
### What changes were proposed in this pull request? This PR proposes to avoid scheduling the (non-zombie) TaskSetManager which has no pending tasks. ### Why are the changes needed? Currently, Spark always tries to schedule a (non-zombie) TaskSetManager even if it has no pending tasks. This causes notable problems for the barrier TaskSetManager: 1. `calculateAvailableSlots` can be called for multiple times for a launched barrier TaskSetManager; 2. user would see "Skip current round of resource offers for barrier stage" log message for a launched barrier TaskSetManager all the time until the barrier TaskSetManager finishes, which is quite confused. Besides, scheduling a TaskSetManager always involves many function invocations even if there're no pending tasks. Therefore, I think we can skip those un-schedulable TasksetManagers to avoid the potential overhead. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass existing tests. Closes #29750 from Ngone51/filter-out-unschedulable-stage. Authored-by: yi.wu <yi.wu@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||