Matei Zaharia
2fb6e7d71e
Initial framework to get a master and web UI up.
2012-06-30 14:45:55 -07:00
Matei Zaharia
c53670b9bf
Various code style fixes, mostly from IntelliJ IDEA
2012-06-29 18:47:12 -07:00
Matei Zaharia
3920189932
Upgraded to Akka 2 and fixed test execution (which was still parallel
...
across projects).
2012-06-28 23:51:28 -07:00
Tathagata Das
e896a505e2
Added testcase for ByteBufferInputStream bugs.
2012-06-17 16:11:12 -07:00
Matei Zaharia
f58da6164e
Merge branch 'master' into dev
2012-06-15 23:47:11 -07:00
Tathagata Das
c6156da9e2
Multiple bug fixes to pass the testsuites ShuffleSuite and BlockManagerSuite.
2012-06-13 16:26:49 -04:00
Matei Zaharia
e75b1b5cb4
Change the default broadcast implementation to a simple HTTP-based
...
broadcast. Fixes #139 .
2012-06-09 15:58:07 -07:00
Matei Zaharia
a96558caa3
Performance improvements to shuffle operations: in particular, preserve
...
RDD partitioning in more cases where it's possible, and use iterators
instead of materializing collections when doing joins.
2012-06-09 14:44:18 -07:00
Matei Zaharia
c2c7299d7a
Added BlockManagerSuite, which I'd forgotten to merge.
2012-06-07 13:47:10 -07:00
Matei Zaharia
63051dd2bc
Merge in engine improvements from the Spark Streaming project, developed
...
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
2012-06-07 12:45:38 -07:00
Matei Zaharia
6ae2746d1e
Handle arrays that contain the same element many times better in
...
SizeEstimator. Also added a test for SizeEstimator. Fixes #136 .
2012-06-06 16:13:02 -07:00
Matei Zaharia
0a617958d1
Some refactoring to make BoundedMemoryCache test similar to others
2012-06-06 16:12:08 -07:00
Matei Zaharia
e141f644ca
Merge pull request #132 from Benky/rb-first-iteration
...
Little refactoring and unit tests for CacheTrackerActor
2012-05-26 13:15:06 -07:00
Richard Benkovsky
ae64920337
MesosScheduler refactoring
2012-05-22 11:04:54 +02:00
Richard Benkovsky
3a1bcd4028
Added tests for CacheTrackerActor
2012-05-22 11:04:54 +02:00
Richard Benkovsky
518506a7c5
Added tests for Utils.copyStream
2012-05-22 11:04:51 +02:00
Richard Benkovsky
565245871f
BoundedMemoryCache.put fails when estimated size of 'value' is larger than cache capacity
2012-05-20 22:13:35 +02:00
Reynold Xin
16461e2eda
Updated Cache's put method to use a case class for response. Previously
...
it was pretty ugly that put() should return -1 for failures.
2012-05-15 00:31:52 -07:00
Reynold Xin
019e48833f
Added the capacity to report cache usage status back to the cache
...
trackor. This is essential for building a dashboard to see the status of
caches on all slaves.
2012-05-14 18:39:04 -07:00
Reynold Xin
761ea65a98
Added a test for the previous commit (failing to serialize task results
...
would throw an exception for local tasks).
2012-04-24 15:14:35 -07:00
Reynold Xin
e601b3b9e5
Added the ability to set environmental variables in piped rdd.
2012-04-17 16:40:56 -07:00
Matei Zaharia
c7af538ac1
Some fixes to sorting for when the RDD has fewer elements than the
...
number of partitions we ask to partition it into. Also, removed a test
that was taking way too long to run.
2012-03-17 13:08:36 -07:00
Matei Zaharia
1e10df0a46
Merge pull request #111 from alupher/master
...
Adding sorting to RDDs
2012-02-24 15:50:14 -08:00
Antonio
0d93d95bcf
Removed unnecessary import
2012-02-21 19:57:12 -08:00
Antonio
2990298f71
Added sorting testing suite
2012-02-21 19:54:21 -08:00
Matei Zaharia
aa04f87cd2
Added support for parallel execution of jobs in DAGScheduler.
2012-02-19 22:50:23 -08:00
Matei Zaharia
a766780f4c
Added some tests for multithreaded access to Spark.
2012-02-09 22:27:53 -08:00
Matei Zaharia
43a3335090
Simplifying test
2012-02-05 22:46:51 -08:00
Matei Zaharia
eb05154b7a
Fixed a failure recovery bug and added some tests for fault recovery.
2012-01-13 19:08:25 -08:00
Matei Zaharia
e269f6f7ea
Register RDDs with the MapOutputTracker even if they have no partitions.
...
Fixes #105 .
2012-01-05 15:59:20 -05:00
Matei Zaharia
735843a049
Merge remote-tracking branch 'origin/charles-newhadoop'
2011-12-02 21:59:30 -08:00
Charles Reiss
66f05f383e
Add new Hadoop API reading support.
2011-12-01 14:02:10 -08:00
Charles Reiss
02d43e6986
Add new Hadoop API writing support.
2011-12-01 14:01:28 -08:00
Matei Zaharia
22b8fcf632
Added fold() and aggregate() operations that reuse an object to
...
merge results into rather than requiring a new object allocation
for each element merged. Fixes #95 .
2011-11-30 11:37:47 -08:00
Matei Zaharia
9e4c79a4d3
Closure cleaner unit test
2011-11-08 00:40:15 -08:00
Matei Zaharia
c2b7fd6899
Make parallelize() work efficiently for ranges of Long, Double, etc
...
(splitting them into sub-ranges). Fixes #87 .
2011-11-02 15:16:02 -07:00
Matei Zaharia
d12122502b
Various improvements to Kryo serializer:
...
- Replaced modified Kryo version with the standard one augmented with
the kryo-serializers package, which includes support for classes with
no-arg constructors (that was why we had a modified Kryo before)
- The kryo-serializers version also fixes issue #72 .
- Added a bunch of tests.
- Serialize maps and a few other common types properly by default.
2011-07-21 22:09:33 -07:00
Matei Zaharia
e4c3402d2d
Renamed ParallelArray to ParallelCollection
2011-07-14 14:47:01 -04:00
Matei Zaharia
2604939f64
Simplified and documented code a little and added test
2011-07-14 00:19:00 -04:00
Matei Zaharia
9c0069188b
Updated save code to allow non-file-based OutputFormats and added a test
...
for file-related stuff
2011-07-13 23:04:06 -04:00
Matei Zaharia
842e14d567
Added mapPartitions operation and a bunch of tests for RDD ops
2011-07-13 00:19:52 -04:00
Olivier Grisel
2e3531d8bf
Implemented RDD.leftOuterJoin and RDD.rightOuterJoin
2011-06-24 11:00:51 +02:00
Olivier Grisel
005d1605a4
add missing test for RDD.groupWith
2011-06-23 02:10:52 +02:00
Ismael Juma
1396678baa
Move REPL classes to separate module.
2011-05-27 11:22:50 +01:00
Matei Zaharia
4db50e26c7
Fixed unit tests by making them clean up the SparkContext after use and
...
thus clean up the various singletons (RDDCache, MapOutputTracker, etc).
This isn't perfect yet (ideally we shouldn't use singleton objects at
all) but we can fix that later.
2011-05-13 12:03:58 -07:00
Matei Zaharia
e5c4cd8a5e
Made examples and core subprojects
2011-02-01 15:11:08 -08:00