Sandy Ryza
adf42611f1
Incorporate Tom's comments - update doc and code to reflect that core requests may not always be honored
2014-01-21 00:38:02 -08:00
Sandy Ryza
3e85b87d90
SPARK-1033. Ask for cores in Yarn container requests
2014-01-20 14:42:32 -08:00
Raymond Liu
4c22c55ad6
Address comments to fix code formats
2014-01-14 10:41:42 +08:00
Raymond Liu
161ab93989
Yarn workerRunnable refactor
2014-01-14 10:36:00 +08:00
Raymond Liu
79a5ba3497
Yarn Client refactor
2014-01-14 10:33:48 +08:00
Henry Saputra
5a8abfb70e
Address code review concerns and comments.
2014-01-12 19:15:09 -08:00
Henry Saputra
91a563608e
Merge branch 'master' into remove_simpleredundantreturn_scala
2014-01-12 10:34:13 -08:00
Henry Saputra
93a65e5fde
Remove simple redundant return statement for Scala methods/functions:
...
-) Only change simple return statements at the end of method
-) Ignore the complex if-else check
-) Ignore the ones inside synchronized
2014-01-12 10:30:04 -08:00
Thomas Graves
9bdfbc0492
Update based on review comments
2014-01-09 15:45:33 -06:00
Thomas Graves
5fccd7051d
add application type
2014-01-09 10:32:35 -06:00
Thomas Graves
c617083e47
yarn-client addJar fix and misc other
2014-01-09 10:24:35 -06:00
Raymond Liu
67af803136
Export --file for YarnClient mode to support sending extra files to worker on yarn cluster
2014-01-07 10:24:11 +08:00
Raymond Liu
da4694a0d8
Minor typo fix for yarn client
2014-01-07 10:24:10 +08:00
liguoqiang
8ddbd531a4
merge upstream/master
2014-01-03 16:06:34 +08:00
liguoqiang
b27b75f1c5
Modify spark on yarn to create SparkConf process
2014-01-03 15:34:24 +08:00
liguoqiang
010e72c079
Modify spark on yarn to create SparkConf process
2014-01-03 15:01:38 +08:00
Raymond Liu
f442afc22e
fix docs for yarn
2014-01-03 14:14:35 +08:00
Raymond Liu
18b3633e54
minor fix for loginfo
2014-01-03 12:14:38 +08:00
Raymond Liu
c59029402d
move duplicate pom config into parent pom
2014-01-03 12:14:38 +08:00
Raymond Liu
ebdfa6bb97
Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2
2014-01-03 12:14:38 +08:00
Raymond Liu
ddc5054b35
Fix yarn/README.md
2014-01-03 12:14:38 +08:00
Raymond Liu
79b6b4ddc2
Clean up unused files for yarn
2014-01-03 12:14:38 +08:00
Raymond Liu
7c96faee74
Fix pom for build yarn/2.x with yarn/common into one jar
2014-01-03 12:14:38 +08:00
Raymond Liu
c5422e02b8
merge yarn/scheduler yarn/common code into one directory
2014-01-03 12:14:37 +08:00
Raymond Liu
ad60710010
Need to send dummy hello message to actually estabilish akka connection.
2014-01-03 12:14:37 +08:00
Raymond Liu
dd6d347f4f
A few clean up for yarn 2.0 code
2014-01-03 12:14:37 +08:00
Raymond Liu
be343d2a56
Fix yarn/README.md and update docs/running-on-yarn.md
2014-01-03 12:12:38 +08:00
Raymond Liu
67cd752e74
Add README for yarn modules
2014-01-03 12:12:38 +08:00
Raymond Liu
e867e31145
some code clean up for Yarn 2.2
2014-01-03 12:12:37 +08:00
Raymond Liu
8818661721
Fix pom file for scala binary version
2014-01-03 12:12:37 +08:00
Raymond Liu
96e25e567c
Fix yarn/assemble pom file
2014-01-03 12:12:37 +08:00
Raymond Liu
aec96dd108
Change profile name new-yarn to hadoop2.2-yarn
2014-01-03 12:12:37 +08:00
Raymond Liu
d1528c7c8c
Fix pom for yarn code reorgnaize commit
2014-01-03 12:12:37 +08:00
Raymond Liu
3dc379ce5a
Reorganize yarn related codes into sub projects to remove duplicate files.
2014-01-03 12:12:37 +08:00
Thomas Graves
fced7885cb
fix yarn-client
2014-01-02 17:11:16 -06:00
Thomas Graves
c6de982be6
Fix yarn build after sparkConf changes
2014-01-02 16:50:35 -06:00
Matei Zaharia
b4ceed40d6
Merge remote-tracking branch 'origin/master' into conf2
...
Conflicts:
core/src/main/scala/org/apache/spark/SparkContext.scala
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala
core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala
core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
streaming/src/test/scala/org/apache/spark/streaming/WindowOperationsSuite.scala
2013-12-29 15:08:08 -05:00
Matei Zaharia
642029e7f4
Various fixes to configuration code
...
- Got rid of global SparkContext.globalConf
- Pass SparkConf to serializers and compression codecs
- Made SparkConf public instead of private[spark]
- Improved API of SparkContext and SparkConf
- Switched executor environment vars to be passed through SparkConf
- Fixed some places that were still using system properties
- Fixed some tests, though others are still failing
This still fails several tests in core, repl and streaming, likely due
to properties not being set or cleared correctly (some of the tests run
fine in isolation).
2013-12-28 17:13:15 -05:00
liguoqiang
2bd76f693d
Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn
2013-12-26 11:10:35 +08:00
liguoqiang
14fcef72db
Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn
2013-12-26 11:05:07 +08:00
Prashant Sharma
2573add94c
spark-544, introducing SparkConf and related configuration overhaul.
2013-12-25 00:09:36 +05:30
Kay Ousterhout
c06945cfe0
Merge remote branch 'upstream/master' into consolidate_schedulers
...
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
2013-12-20 14:39:30 -08:00
Patrick Wendell
a51f3404ad
Merge pull request #265 from markhamstra/scala.binary.version
...
DRY out the POMs with scala.binary.version
...instead of hard-coding 2.10 repeatedly.
As long as it's not a `<project>`-level `<artifactId>`, I think that we are okay parameterizing these.
2013-12-15 22:02:30 -08:00
Reynold Xin
c55e698559
Merge pull request #257 from tgravescs/sparkYarnFixName
...
Fix the --name option for Spark on Yarn
Looks like the --name option accidentally got broken in one of the merges. The Client hangs if the --name option is used right now.
2013-12-15 12:49:02 -08:00
Mark Hamstra
09ed7ddfa0
Use scala.binary.version in POMs
2013-12-15 12:39:58 -08:00
Thomas Graves
842eb55fb5
Fix the --name option for Spark on Yarn
2013-12-12 11:11:09 -06:00
Prashant Sharma
603af51bb5
Merge branch 'master' into akka-bug-fix
...
Conflicts:
core/pom.xml
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
pom.xml
project/SparkBuild.scala
streaming/pom.xml
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
2013-12-11 10:21:53 +05:30
Prashant Sharma
17db6a9041
Style fixes and addressed review comments at #221
2013-12-10 11:47:16 +05:30
Prashant Sharma
c1201f47e0
fixed yarn build
2013-12-09 13:00:50 +05:30
Aaron Davidson
40f63eb034
Merge master into 127
2013-12-08 11:16:52 -08:00
Prashant Sharma
7ad6921ae0
Incorporated Patrick's feedback comment on #211 and made maven build/dep-resolution atleast a bit faster.
2013-12-07 12:45:57 +05:30
Prashant Sharma
4e70480038
A left over akka -> akka.tcp changes
2013-12-06 12:29:53 +05:30
Prashant Sharma
17987778da
Merge branch 'master' into wip-scala-2.10
...
Conflicts:
core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala
core/src/main/scala/org/apache/spark/rdd/MapPartitionsWithContextRDD.scala
core/src/main/scala/org/apache/spark/rdd/RDD.scala
python/pyspark/rdd.py
2013-11-27 14:44:12 +05:30
Harvey Feng
afe4fe7f5e
Merge remote-tracking branch 'origin/master' into yarn-2.2
...
Conflicts:
yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
2013-11-26 15:03:03 -08:00
Harvey Feng
a67ebf4377
A few more style fixes in yarn
package.
2013-11-23 17:08:30 -08:00
Raymond Liu
ab3cefde53
Add YarnClientClusterScheduler and Backend.
...
With this scheduler, the user application is launched locally,
While the executor will be launched by YARN on remote nodes.
This enables spark-shell to run upon YARN.
2013-11-22 09:23:27 +08:00
Harvey Feng
9eae80f111
Merge branch 'master' into yarn-cleanup
...
Conflicts:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
yarn/src/main/scala/org/apache/spark/deploy/yarn/WorkerRunnable.scala
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
2013-11-21 03:41:57 -08:00
Prashant Sharma
95d8dbce91
Merge branch 'master' of github.com:apache/incubator-spark into scala-2.10-temp
...
Conflicts:
core/src/main/scala/org/apache/spark/util/collection/PrimitiveVector.scala
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
2013-11-21 12:34:46 +05:30
Henry Saputra
43dfac5132
Merge branch 'master' into removesemicolonscala
2013-11-19 16:57:57 -08:00
Henry Saputra
10be58f251
Another set of changes to remove unnecessary semicolon (;) from Scala code.
...
Passed the sbt/sbt compile and test
2013-11-19 16:56:23 -08:00
tgravescs
4093e9393a
Impove Spark on Yarn Error handling
2013-11-19 12:44:00 -06:00
Henry Saputra
9c934b640f
Remove the semicolons at the end of Scala code to make it more pure Scala code.
...
Also remove unused imports as I found them along the way.
Remove return statements when returning value in the Scala code.
Passing compile and tests.
2013-11-19 10:19:03 -08:00
Harvey Feng
a98f5a0ebb
Misc style changes in the 'yarn' package.
2013-11-17 22:35:56 -08:00
Henry Saputra
c33f802044
Simple cleanup on Spark's Scala code while testing core and yarn modules:
...
-) Remove some of unused imports as I found them
-) Remove ";" in the imports statements
-) Remove () at the end of method call like size that does not have size effect.
2013-11-15 10:32:20 -08:00
Raymond Liu
a60620b76a
Merge branch 'master' into scala-2.10
2013-11-14 12:44:19 +08:00
Kay Ousterhout
46f9c6b858
Fixed naming issues and added back ability to specify max task failures.
2013-11-13 17:12:14 -08:00
Kay Ousterhout
150615a31e
Merge remote-tracking branch 'upstream/master' into consolidate_schedulers
...
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/ClusterScheduler.scala
2013-11-13 14:38:44 -08:00
Raymond Liu
0f2e3c6e31
Merge branch 'master' into scala-2.10
2013-11-13 16:55:11 +08:00
tgravescs
a35472e1dd
Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs.
2013-11-04 16:16:28 -06:00
Kay Ousterhout
5e91495f5c
Deduplicate Local and Cluster schedulers.
...
The code in LocalScheduler/LocalTaskSetManager was nearly identical
to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy
made making updating the schedulers unnecessarily painful and error-
prone. This commit combines the two into a single TaskScheduler/
TaskSetManager.
2013-10-30 18:48:34 -07:00
tgravescs
b6571541a6
Fix the Worker to use CoarseGrainedExecutorBackend and modify classpath to be explicit
...
about inclusion of spark.jar and app.jar
2013-10-21 14:05:15 -05:00
tgravescs
cc7df2b3cc
Fix yarn build
2013-10-16 10:09:16 -05:00
Matei Zaharia
8f11c36fe1
Merge remote-tracking branch 'tgravescs/sparkYarnDistCache'
...
Closes #11
Conflicts:
docs/running-on-yarn.md
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
2013-10-10 19:34:33 -07:00
Prashant Sharma
7be75682b9
Merge branch 'master' into wip-merge-master
...
Conflicts:
bagel/pom.xml
core/pom.xml
core/src/test/scala/org/apache/spark/ui/UISuite.scala
examples/pom.xml
mllib/pom.xml
pom.xml
project/SparkBuild.scala
repl/pom.xml
streaming/pom.xml
tools/pom.xml
In scala 2.10, a shorter representation is used for naming artifacts
so changed to shorter scala version for artifacts and made it a property in pom.
2013-10-08 11:29:40 +05:30
Patrick Wendell
aa9fb84994
Merging build changes in from 0.8
2013-10-05 22:07:00 -07:00
Prashant Sharma
c810ee0690
Merge branch 'master' into scala-2.10
...
Conflicts:
core/src/test/scala/org/apache/spark/DistributedSuite.scala
project/SparkBuild.scala
2013-10-05 15:52:57 +05:30
tgravescs
0fff4ee852
Adding in the --addJars option to make SparkContext.addJar work on yarn and cleanup
...
the classpaths
2013-10-03 11:52:16 -05:00
tgravescs
c021b8c202
Add default value to usage statement
2013-10-03 08:07:19 -05:00
tgravescs
bc3b20abdc
Allow users to set the application name for Spark on Yarn
2013-10-02 12:54:17 -05:00
Prashant Sharma
5829692885
Merge branch 'master' into scala-2.10
...
Conflicts:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala
docs/_config.yml
project/SparkBuild.scala
repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
2013-10-01 11:57:24 +05:30
Prashant Sharma
7ff4c2d399
fixed maven build for scala 2.10
2013-09-26 10:48:24 +05:30
Patrick Wendell
6079721fa1
Update build version in master
2013-09-24 11:41:51 -07:00
Y.CORP.YAHOO.COM\tgraves
a314b30733
Fix spacing so that the java.io.tmpdir doesn't run on with SPARK_JAVA_OPTS
2013-09-23 14:48:17 -05:00
Y.CORP.YAHOO.COM\tgraves
9d4246863a
Support distributed cache files and archives on spark on yarn and attempt to cleanup the staging directory on exit
2013-09-23 09:09:59 -05:00
Prashant Sharma
276c37a51c
Akka 2.2 migration
2013-09-22 08:20:12 +05:30
Patrick Wendell
e9eba8c3ce
Use different Hadoop version for YARN artifacts.
...
This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.
2013-09-13 15:34:57 -07:00
Jey Kottalam
70661246fd
Fix YARN assembly generation under Maven
2013-09-06 11:31:16 -07:00
Y.CORP.YAHOO.COM\tgraves
c8cc276110
Review comment changes and update to org.apache packaging
2013-09-03 10:50:21 -05:00
Y.CORP.YAHOO.COM\tgraves
547fc4a412
Merge remote-tracking branch 'mesos/master' into yarnUILink
...
Conflicts:
core/src/main/scala/org/apache/spark/ui/UIUtils.scala
core/src/main/scala/org/apache/spark/ui/jobs/PoolTable.scala
core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
docs/running-on-yarn.md
2013-09-03 08:36:59 -05:00
Matei Zaharia
12b2f1f9c9
Add missing license headers found with RAT
2013-09-02 12:23:03 -07:00
Matei Zaharia
5701eb92c7
Fix some URLs
2013-09-01 14:13:16 -07:00
Matei Zaharia
46eecd110a
Initial work to rename package to org.apache.spark
2013-09-01 14:13:13 -07:00
Y.CORP.YAHOO.COM\tgraves
bac46266a9
Link the Spark UI to the Yarn UI
2013-08-30 15:55:32 -05:00
Matei Zaharia
baa84e7e4c
Merge pull request #865 from tgravescs/fixtmpdir
...
Spark on Yarn should use yarn approved directories for spark.local.dir and tmp
2013-08-28 12:44:46 -07:00
Y.CORP.YAHOO.COM\tgraves
3f206bf0b5
Updated based on review comments.
2013-08-27 14:34:27 -05:00
Y.CORP.YAHOO.COM\tgraves
63dc635de6
fix typos
2013-08-26 17:06:20 -05:00
Y.CORP.YAHOO.COM\tgraves
c9464c74a1
Add ability for user to specify environment variables
2013-08-26 16:44:27 -05:00
Y.CORP.YAHOO.COM\tgraves
dfb4c697bc
Throw exception if the yarn local dirs isn't set
2013-08-26 13:57:01 -05:00
Y.CORP.YAHOO.COM\tgraves
c0b4095ee8
Change to use Yarn appropriate directories rather then /tmp or the user specified spark.local.dir
2013-08-26 12:48:37 -05:00
Matei Zaharia
aa2b89d98d
Merge remote-tracking branch 'jey/hadoop-agnostic'
...
Conflicts:
core/src/main/scala/spark/PairRDDFunctions.scala
2013-08-20 10:14:15 -07:00
Jey Kottalam
44000b10ff
Make YARN POM file valid
2013-08-18 16:23:22 -07:00
Jey Kottalam
ad580b94d5
Maven build now also works with YARN
2013-08-16 13:50:12 -07:00
Jey Kottalam
8bb0bd11ce
YARN ApplicationMaster shouldn't wait forever
2013-08-15 16:50:37 -07:00
Jey Kottalam
e2d7656ca3
re-enable YARN support
2013-08-15 16:50:37 -07:00
Jey Kottalam
bd0bab47c9
SparkEnv isn't available this early, and not needed anyway
2013-08-15 16:50:37 -07:00
Jey Kottalam
5d0785b4e5
remove hadoop-yarn's org/apache/...
2013-08-15 16:50:37 -07:00
Jey Kottalam
b877e20a33
move yarn to its own directory
2013-08-15 16:50:36 -07:00