ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/yarn
History
Patrick Wendell c3f5e07533 SPARK-1121: Include avro for yarn-alpha builds 
This lets us explicitly include Avro based on a profile for 0.23.X
builds. It makes me sad how convoluted it is to express this logic
in Maven. @tgraves and @sryza curious if this works for you.

I'm also considering just reverting to how it was before. The only
real problem was that Spark advertised a dependency on Avro
even though it only really depends transitively on Avro through
other deps.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #49 from pwendell/avro-build-fix and squashes the following commits:

8d6ee92 [Patrick Wendell] SPARK-1121: Add avro to yarn-alpha profile
2014-03-02 15:18:19 -08:00
..
alpha SPARK-1121: Include avro for yarn-alpha builds 2014-03-02 15:18:19 -08:00
common/src SPARK-1051. On YARN, executors don't doAs submitting user 2014-02-28 12:43:01 -06:00
stable SPARK-1121: Include avro for yarn-alpha builds 2014-03-02 15:18:19 -08:00
pom.xml SPARK-1121: Include avro for yarn-alpha builds 2014-03-02 15:18:19 -08:00
README.md fix docs for yarn 2014-01-03 14:14:35 +08:00

README.md

YARN DIRECTORY LAYOUT

Hadoop Yarn related codes are organized in separate directories to minimize duplicated code.

  • common : Common codes that do not depending on specific version of Hadoop.

  • alpha / stable : Codes that involve specific version of Hadoop YARN API.

alpha represents 0.23 and 2.0.x stable represents 2.2 and later, until the API changes again.

alpha / stable will build together with common dir into a single jar