Henry Saputra
43dfac5132
Merge branch 'master' into removesemicolonscala
2013-11-19 16:57:57 -08:00
Henry Saputra
10be58f251
Another set of changes to remove unnecessary semicolon (;) from Scala code.
...
Passed the sbt/sbt compile and test
2013-11-19 16:56:23 -08:00
tgravescs
4093e9393a
Impove Spark on Yarn Error handling
2013-11-19 12:44:00 -06:00
Henry Saputra
9c934b640f
Remove the semicolons at the end of Scala code to make it more pure Scala code.
...
Also remove unused imports as I found them along the way.
Remove return statements when returning value in the Scala code.
Passing compile and tests.
2013-11-19 10:19:03 -08:00
Henry Saputra
c33f802044
Simple cleanup on Spark's Scala code while testing core and yarn modules:
...
-) Remove some of unused imports as I found them
-) Remove ";" in the imports statements
-) Remove () at the end of method call like size that does not have size effect.
2013-11-15 10:32:20 -08:00
tgravescs
a35472e1dd
Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs.
2013-11-04 16:16:28 -06:00
tgravescs
b6571541a6
Fix the Worker to use CoarseGrainedExecutorBackend and modify classpath to be explicit
...
about inclusion of spark.jar and app.jar
2013-10-21 14:05:15 -05:00
tgravescs
cc7df2b3cc
Fix yarn build
2013-10-16 10:09:16 -05:00
Matei Zaharia
8f11c36fe1
Merge remote-tracking branch 'tgravescs/sparkYarnDistCache'
...
Closes #11
Conflicts:
docs/running-on-yarn.md
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
2013-10-10 19:34:33 -07:00
Patrick Wendell
aa9fb84994
Merging build changes in from 0.8
2013-10-05 22:07:00 -07:00
tgravescs
0fff4ee852
Adding in the --addJars option to make SparkContext.addJar work on yarn and cleanup
...
the classpaths
2013-10-03 11:52:16 -05:00
tgravescs
c021b8c202
Add default value to usage statement
2013-10-03 08:07:19 -05:00
tgravescs
bc3b20abdc
Allow users to set the application name for Spark on Yarn
2013-10-02 12:54:17 -05:00
Patrick Wendell
6079721fa1
Update build version in master
2013-09-24 11:41:51 -07:00
Y.CORP.YAHOO.COM\tgraves
a314b30733
Fix spacing so that the java.io.tmpdir doesn't run on with SPARK_JAVA_OPTS
2013-09-23 14:48:17 -05:00
Y.CORP.YAHOO.COM\tgraves
9d4246863a
Support distributed cache files and archives on spark on yarn and attempt to cleanup the staging directory on exit
2013-09-23 09:09:59 -05:00
Patrick Wendell
e9eba8c3ce
Use different Hadoop version for YARN artifacts.
...
This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.
2013-09-13 15:34:57 -07:00
Jey Kottalam
70661246fd
Fix YARN assembly generation under Maven
2013-09-06 11:31:16 -07:00
Y.CORP.YAHOO.COM\tgraves
c8cc276110
Review comment changes and update to org.apache packaging
2013-09-03 10:50:21 -05:00
Y.CORP.YAHOO.COM\tgraves
547fc4a412
Merge remote-tracking branch 'mesos/master' into yarnUILink
...
Conflicts:
core/src/main/scala/org/apache/spark/ui/UIUtils.scala
core/src/main/scala/org/apache/spark/ui/jobs/PoolTable.scala
core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
docs/running-on-yarn.md
2013-09-03 08:36:59 -05:00
Matei Zaharia
12b2f1f9c9
Add missing license headers found with RAT
2013-09-02 12:23:03 -07:00
Matei Zaharia
5701eb92c7
Fix some URLs
2013-09-01 14:13:16 -07:00
Matei Zaharia
46eecd110a
Initial work to rename package to org.apache.spark
2013-09-01 14:13:13 -07:00
Y.CORP.YAHOO.COM\tgraves
bac46266a9
Link the Spark UI to the Yarn UI
2013-08-30 15:55:32 -05:00
Matei Zaharia
baa84e7e4c
Merge pull request #865 from tgravescs/fixtmpdir
...
Spark on Yarn should use yarn approved directories for spark.local.dir and tmp
2013-08-28 12:44:46 -07:00
Y.CORP.YAHOO.COM\tgraves
3f206bf0b5
Updated based on review comments.
2013-08-27 14:34:27 -05:00
Y.CORP.YAHOO.COM\tgraves
63dc635de6
fix typos
2013-08-26 17:06:20 -05:00
Y.CORP.YAHOO.COM\tgraves
c9464c74a1
Add ability for user to specify environment variables
2013-08-26 16:44:27 -05:00
Y.CORP.YAHOO.COM\tgraves
dfb4c697bc
Throw exception if the yarn local dirs isn't set
2013-08-26 13:57:01 -05:00
Y.CORP.YAHOO.COM\tgraves
c0b4095ee8
Change to use Yarn appropriate directories rather then /tmp or the user specified spark.local.dir
2013-08-26 12:48:37 -05:00
Matei Zaharia
aa2b89d98d
Merge remote-tracking branch 'jey/hadoop-agnostic'
...
Conflicts:
core/src/main/scala/spark/PairRDDFunctions.scala
2013-08-20 10:14:15 -07:00
Jey Kottalam
44000b10ff
Make YARN POM file valid
2013-08-18 16:23:22 -07:00
Jey Kottalam
ad580b94d5
Maven build now also works with YARN
2013-08-16 13:50:12 -07:00
Jey Kottalam
8bb0bd11ce
YARN ApplicationMaster shouldn't wait forever
2013-08-15 16:50:37 -07:00
Jey Kottalam
e2d7656ca3
re-enable YARN support
2013-08-15 16:50:37 -07:00
Jey Kottalam
bd0bab47c9
SparkEnv isn't available this early, and not needed anyway
2013-08-15 16:50:37 -07:00
Jey Kottalam
5d0785b4e5
remove hadoop-yarn's org/apache/...
2013-08-15 16:50:37 -07:00
Jey Kottalam
b877e20a33
move yarn to its own directory
2013-08-15 16:50:36 -07:00