ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/yarn
History
Patrick Wendell 995fdc96bc Assorted clean-up for Spark-on-YARN. 
In particular when the HADOOP_CONF_DIR is not not specified.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #488 from pwendell/hadoop-cleanup and squashes the following commits:

fe95f13 [Patrick Wendell] Changes based on Andrew's feeback
18d09c1 [Patrick Wendell] Review comments from Andrew
17929cc [Patrick Wendell] Assorted clean-up for Spark-on-YARN.
2014-04-22 19:22:06 -07:00
..
alpha Clean up and simplify Spark configuration 2014-04-21 10:26:33 -07:00
common/src Assorted clean-up for Spark-on-YARN. 2014-04-22 19:22:06 -07:00
stable Clean up and simplify Spark configuration 2014-04-21 10:26:33 -07:00
pom.xml SPARK-1252. On YARN, use container-log4j.properties for executors 2014-04-07 13:28:14 -05:00
README.md fix docs for yarn 2014-01-03 14:14:35 +08:00

README.md

YARN DIRECTORY LAYOUT

Hadoop Yarn related codes are organized in separate directories to minimize duplicated code.

  • common : Common codes that do not depending on specific version of Hadoop.

  • alpha / stable : Codes that involve specific version of Hadoop YARN API.

alpha represents 0.23 and 2.0.x stable represents 2.2 and later, until the API changes again.

alpha / stable will build together with common dir into a single jar