Matei Zaharia
b4ceed40d6
Merge remote-tracking branch 'origin/master' into conf2
...
Conflicts:
core/src/main/scala/org/apache/spark/SparkContext.scala
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala
core/src/main/scala/org/apache/spark/scheduler/local/LocalScheduler.scala
core/src/main/scala/org/apache/spark/util/MetadataCleaner.scala
core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
streaming/src/test/scala/org/apache/spark/streaming/WindowOperationsSuite.scala
2013-12-29 15:08:08 -05:00
Matei Zaharia
642029e7f4
Various fixes to configuration code
...
- Got rid of global SparkContext.globalConf
- Pass SparkConf to serializers and compression codecs
- Made SparkConf public instead of private[spark]
- Improved API of SparkContext and SparkConf
- Switched executor environment vars to be passed through SparkConf
- Fixed some places that were still using system properties
- Fixed some tests, though others are still failing
This still fails several tests in core, repl and streaming, likely due
to properties not being set or cleared correctly (some of the tests run
fine in isolation).
2013-12-28 17:13:15 -05:00
liguoqiang
b662c88a24
fix this import order
2013-12-26 15:49:33 +08:00
liguoqiang
2bd76f693d
Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn
2013-12-26 11:10:35 +08:00
liguoqiang
14fcef72db
Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn
2013-12-26 11:05:07 +08:00
Prashant Sharma
2573add94c
spark-544, introducing SparkConf and related configuration overhaul.
2013-12-25 00:09:36 +05:30
azuryyu
66b7bea7f8
Make App report interval configurable during 'run on Yarn'
2013-12-24 18:16:49 +08:00
Patrick Wendell
10c0ffa1eb
One other fix
2013-12-16 22:10:55 -08:00
Patrick Wendell
c1c0f8099f
Clean-up
2013-12-16 22:01:27 -08:00
Patrick Wendell
24f8220dc8
Removing extra code in new yarn
2013-12-16 21:53:51 -08:00
Patrick Wendell
c6f95e603e
Attempt with extra repositories
2013-12-16 21:53:51 -08:00
Thomas Graves
842eb55fb5
Fix the --name option for Spark on Yarn
2013-12-12 11:11:09 -06:00
Raymond Liu
403cac9be3
Port yarn-client mode for new-yarn
2013-11-27 16:10:42 +08:00
Harvey Feng
4f1c3fa5d7
Hadoop 2.2 YARN API migration for SPARK_HOME/new-yarn
2013-11-23 17:08:30 -08:00
Harvey Feng
ab8652f2d3
Add a "new-yarn" directory in SPARK_HOME, intended to contain Hadoop-2.2 API changes.
2013-11-23 17:08:30 -08:00
