Kay Ousterhout
150615a31e
Merge remote-tracking branch 'upstream/master' into consolidate_schedulers
...
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/ClusterScheduler.scala
2013-11-13 14:38:44 -08:00
Raymond Liu
0f2e3c6e31
Merge branch 'master' into scala-2.10
2013-11-13 16:55:11 +08:00
tgravescs
a35472e1dd
Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs.
2013-11-04 16:16:28 -06:00
Kay Ousterhout
5e91495f5c
Deduplicate Local and Cluster schedulers.
...
The code in LocalScheduler/LocalTaskSetManager was nearly identical
to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy
made making updating the schedulers unnecessarily painful and error-
prone. This commit combines the two into a single TaskScheduler/
TaskSetManager.
2013-10-30 18:48:34 -07:00
tgravescs
b6571541a6
Fix the Worker to use CoarseGrainedExecutorBackend and modify classpath to be explicit
...
about inclusion of spark.jar and app.jar
2013-10-21 14:05:15 -05:00
tgravescs
cc7df2b3cc
Fix yarn build
2013-10-16 10:09:16 -05:00
Matei Zaharia
8f11c36fe1
Merge remote-tracking branch 'tgravescs/sparkYarnDistCache'
...
Closes #11
Conflicts:
docs/running-on-yarn.md
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
2013-10-10 19:34:33 -07:00
Prashant Sharma
7be75682b9
Merge branch 'master' into wip-merge-master
...
Conflicts:
bagel/pom.xml
core/pom.xml
core/src/test/scala/org/apache/spark/ui/UISuite.scala
examples/pom.xml
mllib/pom.xml
pom.xml
project/SparkBuild.scala
repl/pom.xml
streaming/pom.xml
tools/pom.xml
In scala 2.10, a shorter representation is used for naming artifacts
so changed to shorter scala version for artifacts and made it a property in pom.
2013-10-08 11:29:40 +05:30
Patrick Wendell
aa9fb84994
Merging build changes in from 0.8
2013-10-05 22:07:00 -07:00
Prashant Sharma
c810ee0690
Merge branch 'master' into scala-2.10
...
Conflicts:
core/src/test/scala/org/apache/spark/DistributedSuite.scala
project/SparkBuild.scala
2013-10-05 15:52:57 +05:30
tgravescs
0fff4ee852
Adding in the --addJars option to make SparkContext.addJar work on yarn and cleanup
...
the classpaths
2013-10-03 11:52:16 -05:00
tgravescs
c021b8c202
Add default value to usage statement
2013-10-03 08:07:19 -05:00
tgravescs
bc3b20abdc
Allow users to set the application name for Spark on Yarn
2013-10-02 12:54:17 -05:00
Prashant Sharma
5829692885
Merge branch 'master' into scala-2.10
...
Conflicts:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala
docs/_config.yml
project/SparkBuild.scala
repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
2013-10-01 11:57:24 +05:30
Prashant Sharma
7ff4c2d399
fixed maven build for scala 2.10
2013-09-26 10:48:24 +05:30
Patrick Wendell
6079721fa1
Update build version in master
2013-09-24 11:41:51 -07:00
Y.CORP.YAHOO.COM\tgraves
a314b30733
Fix spacing so that the java.io.tmpdir doesn't run on with SPARK_JAVA_OPTS
2013-09-23 14:48:17 -05:00
Y.CORP.YAHOO.COM\tgraves
9d4246863a
Support distributed cache files and archives on spark on yarn and attempt to cleanup the staging directory on exit
2013-09-23 09:09:59 -05:00
Prashant Sharma
276c37a51c
Akka 2.2 migration
2013-09-22 08:20:12 +05:30
Patrick Wendell
e9eba8c3ce
Use different Hadoop version for YARN artifacts.
...
This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.
2013-09-13 15:34:57 -07:00
Jey Kottalam
70661246fd
Fix YARN assembly generation under Maven
2013-09-06 11:31:16 -07:00
Y.CORP.YAHOO.COM\tgraves
c8cc276110
Review comment changes and update to org.apache packaging
2013-09-03 10:50:21 -05:00
Y.CORP.YAHOO.COM\tgraves
547fc4a412
Merge remote-tracking branch 'mesos/master' into yarnUILink
...
Conflicts:
core/src/main/scala/org/apache/spark/ui/UIUtils.scala
core/src/main/scala/org/apache/spark/ui/jobs/PoolTable.scala
core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
docs/running-on-yarn.md
2013-09-03 08:36:59 -05:00
Matei Zaharia
12b2f1f9c9
Add missing license headers found with RAT
2013-09-02 12:23:03 -07:00
Matei Zaharia
5701eb92c7
Fix some URLs
2013-09-01 14:13:16 -07:00
Matei Zaharia
46eecd110a
Initial work to rename package to org.apache.spark
2013-09-01 14:13:13 -07:00
Y.CORP.YAHOO.COM\tgraves
bac46266a9
Link the Spark UI to the Yarn UI
2013-08-30 15:55:32 -05:00
Matei Zaharia
baa84e7e4c
Merge pull request #865 from tgravescs/fixtmpdir
...
Spark on Yarn should use yarn approved directories for spark.local.dir and tmp
2013-08-28 12:44:46 -07:00
Y.CORP.YAHOO.COM\tgraves
3f206bf0b5
Updated based on review comments.
2013-08-27 14:34:27 -05:00
Y.CORP.YAHOO.COM\tgraves
63dc635de6
fix typos
2013-08-26 17:06:20 -05:00
Y.CORP.YAHOO.COM\tgraves
c9464c74a1
Add ability for user to specify environment variables
2013-08-26 16:44:27 -05:00
Y.CORP.YAHOO.COM\tgraves
dfb4c697bc
Throw exception if the yarn local dirs isn't set
2013-08-26 13:57:01 -05:00
Y.CORP.YAHOO.COM\tgraves
c0b4095ee8
Change to use Yarn appropriate directories rather then /tmp or the user specified spark.local.dir
2013-08-26 12:48:37 -05:00
Matei Zaharia
aa2b89d98d
Merge remote-tracking branch 'jey/hadoop-agnostic'
...
Conflicts:
core/src/main/scala/spark/PairRDDFunctions.scala
2013-08-20 10:14:15 -07:00
Jey Kottalam
44000b10ff
Make YARN POM file valid
2013-08-18 16:23:22 -07:00
Jey Kottalam
ad580b94d5
Maven build now also works with YARN
2013-08-16 13:50:12 -07:00
Jey Kottalam
8bb0bd11ce
YARN ApplicationMaster shouldn't wait forever
2013-08-15 16:50:37 -07:00
Jey Kottalam
e2d7656ca3
re-enable YARN support
2013-08-15 16:50:37 -07:00
Jey Kottalam
bd0bab47c9
SparkEnv isn't available this early, and not needed anyway
2013-08-15 16:50:37 -07:00
Jey Kottalam
5d0785b4e5
remove hadoop-yarn's org/apache/...
2013-08-15 16:50:37 -07:00
Jey Kottalam
b877e20a33
move yarn to its own directory
2013-08-15 16:50:36 -07:00