38873d5196
### What changes were proposed in this pull request? This PR adds the check to see whether the allocated executors for the waiting application is empty before recognizing it as a possible hang application. ### Why are the changes needed? It's a bugfix. The warning means there are not enough resources for the application to launch at least one executor. But we can still successfully run a job under this warning, which means it does have launched executor. ### Does this PR introduce _any_ user-facing change? Yes. Before this PR, when using local cluster mode to start spark-shell, e.g. `./bin/spark-shell --master "local-cluster[2, 1, 1024]"`, the user would always see the warning: ``` 20/06/06 22:21:02 WARN Utils: Your hostname, C02ZQ051LVDR resolves to a loopback address: 127.0.0.1; using 192.168.1.6 instead (on interface en0) 20/06/06 22:21:02 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 20/06/06 22:21:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly. NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly. Spark context Web UI available at http://192.168.1.6:4040 Spark context available as 'sc' (master = local-cluster[2, 1, 1024], app id = app-20200606222107-0000). Spark session available as 'spark'. 20/06/06 22:21:07 WARN Master: App app-20200606222107-0000 requires more resource than any of Workers could have. 20/06/06 22:21:07 WARN Master: App app-20200606222107-0000 requires more resource than any of Workers could have. ``` After this PR, the warning has gone. ### How was this patch tested? Tested manually. Closes #28742 from Ngone51/fix_warning. Authored-by: yi.wu <yi.wu@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
---|---|---|
.. | ||
benchmarks | ||
src | ||
pom.xml |