ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/yarn
History
Thomas Graves 62c5576099 [SPARK-3375] spark on yarn container allocation issues 
If yarn doesn't get the containers immediately it stops asking for them and the yarn application hangs with never getting any executors.

The issue here is that we are sending the number of containers as 0 after we send the original one of X. on the yarn side this clears out the original request.

For a ping we should just send empty asks.

Author: Thomas Graves <tgraves@apache.org>

Closes #2275 from tgravescs/SPARK-3375 and squashes the following commits:

74b6820 [Thomas Graves] send empty resource requests when we aren't asking for containers
2014-09-05 09:56:22 -05:00
..
alpha [SPARK-3375] spark on yarn container allocation issues 2014-09-05 09:56:22 -05:00
common/src [SPARK-3260] yarn - pass acls along with executor launch 2014-09-05 09:54:40 -05:00
stable [SPARK-3375] spark on yarn container allocation issues 2014-09-05 09:56:22 -05:00
pom.xml [SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix) 2014-07-28 12:07:30 -07:00
README.md fix docs for yarn 2014-01-03 14:14:35 +08:00

README.md

YARN DIRECTORY LAYOUT

Hadoop Yarn related codes are organized in separate directories to minimize duplicated code.

  • common : Common codes that do not depending on specific version of Hadoop.

  • alpha / stable : Codes that involve specific version of Hadoop YARN API.

alpha represents 0.23 and 2.0.x stable represents 2.2 and later, until the API changes again.

alpha / stable will build together with common dir into a single jar