Matei Zaharia
deedb9e7b7
Fix further issues with tests and broadcast.
...
The broadcast fix is to store values as MEMORY_ONLY_DESER instead of
MEMORY_ONLY, which will save substantial time on serialization.
2012-08-23 20:31:49 -07:00
Denny
4f4a34c025
Stlystic changes
...
Conflicts:
core/src/test/scala/spark/MesosSchedulerSuite.scala
2012-07-23 16:32:20 -07:00
Denny
866e6949df
Always destroy SparkContext in after block for the unit tests.
...
Conflicts:
core/src/test/scala/spark/ShuffleSuite.scala
2012-07-23 16:29:17 -07:00
Matei Zaharia
f58da6164e
Merge branch 'master' into dev
2012-06-15 23:47:11 -07:00
Matei Zaharia
a96558caa3
Performance improvements to shuffle operations: in particular, preserve
...
RDD partitioning in more cases where it's possible, and use iterators
instead of materializing collections when doing joins.
2012-06-09 14:44:18 -07:00
Matei Zaharia
63051dd2bc
Merge in engine improvements from the Spark Streaming project, developed
...
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
2012-06-07 12:45:38 -07:00
Reynold Xin
968f75f6af
Added an option (spark.closure.serializer) to specify the serializer for
...
closures. This enables using Kryo as the closure serializer.
2012-04-09 21:59:56 -07:00
Ankur Dave
c5be7d2b22
Update Bagel unit tests to reflect API change
2011-11-08 19:56:44 +00:00
Ankur Dave
ab3889f627
Implement standalone WikipediaPageRank with custom serializer
2011-10-09 16:53:10 -07:00
Ankur Dave
cbdc01eecd
Update WikipediaPageRank to reflect Bagel API changes
2011-10-09 16:19:34 -07:00
Ankur Dave
6d707f6b63
Remove ShortestPath for now
2011-10-09 16:19:34 -07:00
Ankur Dave
0028caf3a4
Simplify and genericize type parameters in Bagel
2011-10-09 15:58:39 -07:00
Ismael Juma
0fba22b3d2
Fix issue #65 : Change @serializable to extends Serializable in 2.9 branch
...
Note that we use scala.Serializable introduced in Scala 2.9 instead of
java.io.Serializable. Also, case classes inherit from scala.Serializable by
default.
2011-08-02 10:16:33 +01:00
Matei Zaharia
969644df8e
Cleaned up a few issues to do with default parallelism levels. Also
...
renamed HadoopFileWriter to HadoopWriter (since it's not only for files)
and fixed a bug for lookup().
2011-07-14 12:40:56 -04:00
Matei Zaharia
4db50e26c7
Fixed unit tests by making them clean up the SparkContext after use and
...
thus clean up the various singletons (RDDCache, MapOutputTracker, etc).
This isn't perfect yet (ideally we shouldn't use singleton objects at
all) but we can fix that later.
2011-05-13 12:03:58 -07:00
Ankur Dave
f40a0898a7
Rename bagel to spark.bagel and Pregel to Bagel
2011-05-09 15:23:21 -07:00
Ankur Dave
c1104058c6
Move shortest path and PageRank to bagel.examples
2011-05-03 18:53:58 -07:00
Ankur Dave
563c5e717c
Refactor and add aggregator support
...
Refactored out the agg() and comp() methods from Pregel.run.
Defined an implicit conversion to allow applications that don't use
aggregators to avoid including a null argument for the result of the
aggregator in the compute function.
2011-05-03 15:40:45 -07:00
Ankur Dave
c18fa3ebc6
Package combiner functions into a trait
2011-05-03 15:40:41 -07:00
Ankur Dave
1c8ca0ebe1
Add Bagel test suite
...
Note: This test suite currently fails for the same reason that the
Spark Core test suite fails: Spark currently seems to have a bug where
any test after the first one fails.
2011-05-03 15:40:31 -07:00
Ankur Dave
c5b3ea755f
Clean up Bagel source and interface
2011-05-03 15:40:01 -07:00
Ankur Dave
19122af787
Update ShortestPath to work with controllable partitioning
2011-05-03 15:39:39 -07:00
Ankur Dave
62ef620354
Clean up Pregel.run, add logging
2011-05-03 15:38:01 -07:00
Ankur Dave
c0736f6f68
Add Bagel, an implementation of Pregel on Spark
2011-05-03 15:37:08 -07:00