28dbae85aa
In SPARK-1946(PR #900), configuration <code>spark.scheduler.minRegisteredExecutorsRatio</code> was introduced. However, in standalone mode, there is a race condition where isReady() can return true because totalExpectedExecutors has not been correctly set. Because expected executors is uncertain in standalone mode, the PR try to use CPU cores(<code>--total-executor-cores</code>) as expected resources to judge whether SchedulerBackend is ready. Author: li-zhihui <zhihui.li@intel.com> Author: Li Zhihui <zhihui.li@intel.com> Closes #1525 from li-zhihui/fixre4s and squashes the following commits: e9a630b [Li Zhihui] Rename variable totalExecutors and clean codes abf4860 [Li Zhihui] Push down variable totalExpectedResources to children classes ca54bd9 [li-zhihui] Format log with String interpolation 88c7dc6 [li-zhihui] Few codes and docs refactor 41cf47e [li-zhihui] Fix race condition at SchedulerBackend.isReady in standalone mode |
||
|---|---|---|
| .. | ||
| alpha | ||
| common/src | ||
| stable | ||
| pom.xml | ||
| README.md | ||
YARN DIRECTORY LAYOUT
Hadoop Yarn related codes are organized in separate directories to minimize duplicated code.
-
common : Common codes that do not depending on specific version of Hadoop.
-
alpha / stable : Codes that involve specific version of Hadoop YARN API.
alpha represents 0.23 and 2.0.x stable represents 2.2 and later, until the API changes again.
alpha / stable will build together with common dir into a single jar