Matei Zaharia
e72afdb817
Some refactoring to make cluster scheduler pluggable.
2012-07-06 15:23:26 -07:00
Matei Zaharia
5d1a887bed
Further updates to run processes on cluster.
2012-07-01 17:13:31 -07:00
Matei Zaharia
408b5a1332
More work on deploy code (adding Worker class)
2012-06-30 16:45:57 -07:00
Matei Zaharia
2fb6e7d71e
Initial framework to get a master and web UI up.
2012-06-30 14:45:55 -07:00
Matei Zaharia
c53670b9bf
Various code style fixes, mostly from IntelliJ IDEA
2012-06-29 18:47:12 -07:00
Matei Zaharia
f58da6164e
Merge branch 'master' into dev
2012-06-15 23:47:11 -07:00
Matei Zaharia
e75b1b5cb4
Change the default broadcast implementation to a simple HTTP-based
...
broadcast. Fixes #139 .
2012-06-09 15:58:07 -07:00
Matei Zaharia
63051dd2bc
Merge in engine improvements from the Spark Streaming project, developed
...
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
2012-06-07 12:45:38 -07:00
Richard Benkovsky
8f2f736d53
Little refactoring
2012-05-22 11:04:54 +02:00
Richard Benkovsky
f162fc2beb
Formating fixed
2012-05-22 09:45:38 +02:00
Richard Benkovsky
822a4be37d
Utils.memoryBytesToString fixed
2012-05-19 15:13:20 +02:00
Reynold Xin
16461e2eda
Updated Cache's put method to use a case class for response. Previously
...
it was pretty ugly that put() should return -1 for failures.
2012-05-15 00:31:52 -07:00
Reynold Xin
019e48833f
Added the capacity to report cache usage status back to the cache
...
trackor. This is essential for building a dashboard to see the status of
caches on all slaves.
2012-05-14 18:39:04 -07:00
Matei Zaharia
3b745176e0
Bug fix to pluggable closure serialization change
2012-04-12 17:53:02 +00:00
Reynold Xin
d295ccb43c
Added a closureSerializer field in SparkEnv and use it to serialize
...
tasks.
2012-04-10 13:29:46 -07:00
Reynold Xin
968f75f6af
Added an option (spark.closure.serializer) to specify the serializer for
...
closures. This enables using Kryo as the closure serializer.
2012-04-09 21:59:56 -07:00
Matei Zaharia
7660a8b12f
Merge branch 'formatting'
...
Conflicts:
core/src/main/scala/spark/DAGScheduler.scala
core/src/main/scala/spark/SimpleShuffleFetcher.scala
core/src/main/scala/spark/SparkContext.scala
2012-02-10 10:42:14 -08:00
haoyuan
194c42ab79
Code format.
2012-02-10 08:19:53 -08:00
Matei Zaharia
8f5ed51234
Delete Spark's temporary directories when the JVM exits.
2012-02-09 22:58:24 -08:00
haoyuan
445e0bb1b5
Format the code a bit mroe.
2012-02-09 15:50:26 -08:00
Matei Zaharia
c4dd68ae21
Merge branch 'mos-bt'
...
This merge keeps only the broadcast work in mos-bt because the structure
of shuffle has changed with the new RDD design. We still need some kind
of parallel shuffle but that will be added later.
Conflicts:
core/src/main/scala/spark/BitTorrentBroadcast.scala
core/src/main/scala/spark/ChainedBroadcast.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/Utils.scala
core/src/main/scala/spark/shuffle/BasicLocalFileShuffle.scala
core/src/main/scala/spark/shuffle/DfsShuffle.scala
2011-06-26 18:22:12 -07:00
Matei Zaharia
24a1e7f838
Scheduler can now recover from lost map outputs
2011-05-20 00:19:53 -07:00
Mosharaf Chowdhury
60d1121343
Refactoring: daemonThreadFactories have all been moved to the Utils
...
object instead of having multiple copies in Broadcast and Shuffle
objects.
2011-04-27 22:13:01 -07:00
Matei Zaharia
9e59afd710
More work on new RDD design
2011-02-27 19:15:52 -08:00
Matei Zaharia
e5c4cd8a5e
Made examples and core subprojects
2011-02-01 15:11:08 -08:00