Henry Saputra
91a563608e
Merge branch 'master' into remove_simpleredundantreturn_scala
2014-01-12 10:34:13 -08:00
Henry Saputra
93a65e5fde
Remove simple redundant return statement for Scala methods/functions:
...
-) Only change simple return statements at the end of method
-) Ignore the complex if-else check
-) Ignore the ones inside synchronized
2014-01-12 10:30:04 -08:00
Thomas Graves
9bdfbc0492
Update based on review comments
2014-01-09 15:45:33 -06:00
Thomas Graves
5fccd7051d
add application type
2014-01-09 10:32:35 -06:00
Thomas Graves
c617083e47
yarn-client addJar fix and misc other
2014-01-09 10:24:35 -06:00
Raymond Liu
67af803136
Export --file for YarnClient mode to support sending extra files to worker on yarn cluster
2014-01-07 10:24:11 +08:00
Raymond Liu
da4694a0d8
Minor typo fix for yarn client
2014-01-07 10:24:10 +08:00
liguoqiang
8ddbd531a4
merge upstream/master
2014-01-03 16:06:34 +08:00
liguoqiang
b27b75f1c5
Modify spark on yarn to create SparkConf process
2014-01-03 15:34:24 +08:00
liguoqiang
010e72c079
Modify spark on yarn to create SparkConf process
2014-01-03 15:01:38 +08:00
Raymond Liu
f442afc22e
fix docs for yarn
2014-01-03 14:14:35 +08:00
Raymond Liu
18b3633e54
minor fix for loginfo
2014-01-03 12:14:38 +08:00
Raymond Liu
c59029402d
move duplicate pom config into parent pom
2014-01-03 12:14:38 +08:00
Raymond Liu
ebdfa6bb97
Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2
2014-01-03 12:14:38 +08:00
Raymond Liu
ddc5054b35
Fix yarn/README.md
2014-01-03 12:14:38 +08:00
Raymond Liu
79b6b4ddc2
Clean up unused files for yarn
2014-01-03 12:14:38 +08:00
Raymond Liu
7c96faee74
Fix pom for build yarn/2.x with yarn/common into one jar
2014-01-03 12:14:38 +08:00
Raymond Liu
c5422e02b8
merge yarn/scheduler yarn/common code into one directory
2014-01-03 12:14:37 +08:00
Raymond Liu
ad60710010
Need to send dummy hello message to actually estabilish akka connection.
2014-01-03 12:14:37 +08:00
Raymond Liu
dd6d347f4f
A few clean up for yarn 2.0 code
2014-01-03 12:14:37 +08:00
Raymond Liu
be343d2a56
Fix yarn/README.md and update docs/running-on-yarn.md
2014-01-03 12:12:38 +08:00
Raymond Liu
67cd752e74
Add README for yarn modules
2014-01-03 12:12:38 +08:00
Raymond Liu
e867e31145
some code clean up for Yarn 2.2
2014-01-03 12:12:37 +08:00
Raymond Liu
8818661721
Fix pom file for scala binary version
2014-01-03 12:12:37 +08:00
Raymond Liu
96e25e567c
Fix yarn/assemble pom file
2014-01-03 12:12:37 +08:00
Raymond Liu
aec96dd108
Change profile name new-yarn to hadoop2.2-yarn
2014-01-03 12:12:37 +08:00
Raymond Liu
d1528c7c8c
Fix pom for yarn code reorgnaize commit
2014-01-03 12:12:37 +08:00
Raymond Liu
3dc379ce5a
Reorganize yarn related codes into sub projects to remove duplicate files.
2014-01-03 12:12:37 +08:00
Thomas Graves
fced7885cb
fix yarn-client
2014-01-02 17:11:16 -06:00
Thomas Graves
c6de982be6
Fix yarn build after sparkConf changes
2014-01-02 16:50:35 -06:00
Matei Zaharia
b4ceed40d6
Merge remote-tracking branch 'origin/master' into conf2
...
Conflicts:
core/src/main/scala/org/apache/spark/SparkContext.scala
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala
core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala
core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
streaming/src/test/scala/org/apache/spark/streaming/WindowOperationsSuite.scala
2013-12-29 15:08:08 -05:00
Matei Zaharia
642029e7f4
Various fixes to configuration code
...
- Got rid of global SparkContext.globalConf
- Pass SparkConf to serializers and compression codecs
- Made SparkConf public instead of private[spark]
- Improved API of SparkContext and SparkConf
- Switched executor environment vars to be passed through SparkConf
- Fixed some places that were still using system properties
- Fixed some tests, though others are still failing
This still fails several tests in core, repl and streaming, likely due
to properties not being set or cleared correctly (some of the tests run
fine in isolation).
2013-12-28 17:13:15 -05:00
liguoqiang
2bd76f693d
Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn
2013-12-26 11:10:35 +08:00
liguoqiang
14fcef72db
Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn
2013-12-26 11:05:07 +08:00
Prashant Sharma
2573add94c
spark-544, introducing SparkConf and related configuration overhaul.
2013-12-25 00:09:36 +05:30
Kay Ousterhout
c06945cfe0
Merge remote branch 'upstream/master' into consolidate_schedulers
...
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
2013-12-20 14:39:30 -08:00
Patrick Wendell
a51f3404ad
Merge pull request #265 from markhamstra/scala.binary.version
...
DRY out the POMs with scala.binary.version
...instead of hard-coding 2.10 repeatedly.
As long as it's not a `<project>`-level `<artifactId>`, I think that we are okay parameterizing these.
2013-12-15 22:02:30 -08:00
Reynold Xin
c55e698559
Merge pull request #257 from tgravescs/sparkYarnFixName
...
Fix the --name option for Spark on Yarn
Looks like the --name option accidentally got broken in one of the merges. The Client hangs if the --name option is used right now.
2013-12-15 12:49:02 -08:00
Mark Hamstra
09ed7ddfa0
Use scala.binary.version in POMs
2013-12-15 12:39:58 -08:00
Thomas Graves
842eb55fb5
Fix the --name option for Spark on Yarn
2013-12-12 11:11:09 -06:00
Prashant Sharma
603af51bb5
Merge branch 'master' into akka-bug-fix
...
Conflicts:
core/pom.xml
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
pom.xml
project/SparkBuild.scala
streaming/pom.xml
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
2013-12-11 10:21:53 +05:30
Prashant Sharma
17db6a9041
Style fixes and addressed review comments at #221
2013-12-10 11:47:16 +05:30
Prashant Sharma
c1201f47e0
fixed yarn build
2013-12-09 13:00:50 +05:30
Aaron Davidson
40f63eb034
Merge master into 127
2013-12-08 11:16:52 -08:00
Prashant Sharma
7ad6921ae0
Incorporated Patrick's feedback comment on #211 and made maven build/dep-resolution atleast a bit faster.
2013-12-07 12:45:57 +05:30
Prashant Sharma
4e70480038
A left over akka -> akka.tcp changes
2013-12-06 12:29:53 +05:30
Prashant Sharma
17987778da
Merge branch 'master' into wip-scala-2.10
...
Conflicts:
core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala
core/src/main/scala/org/apache/spark/rdd/MapPartitionsWithContextRDD.scala
core/src/main/scala/org/apache/spark/rdd/RDD.scala
python/pyspark/rdd.py
2013-11-27 14:44:12 +05:30
Harvey Feng
afe4fe7f5e
Merge remote-tracking branch 'origin/master' into yarn-2.2
...
Conflicts:
yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
2013-11-26 15:03:03 -08:00
Harvey Feng
a67ebf4377
A few more style fixes in yarn
package.
2013-11-23 17:08:30 -08:00
Raymond Liu
ab3cefde53
Add YarnClientClusterScheduler and Backend.
...
With this scheduler, the user application is launched locally,
While the executor will be launched by YARN on remote nodes.
This enables spark-shell to run upon YARN.
2013-11-22 09:23:27 +08:00