Imran Rashid
13cc72cfb5
Accumulator now inherits from Accumulable, whcih simplifies a bunch of other things (eg., no +:=)
2012-07-12 12:40:10 -07:00
Imran Rashid
73935629a1
improve scaladoc
2012-07-12 09:58:06 -07:00
Imran Rashid
30480e6dae
add Accumulatable, add corresponding docs & tests for accumulators
2012-07-12 09:37:42 -07:00
Matei Zaharia
4259d37f84
Formatting
2012-07-10 11:16:34 -07:00
Matei Zaharia
6980b67557
Added more methods for loading/saving with new Hadoop API
2012-07-10 11:11:35 -07:00
Matei Zaharia
800fcbfbca
Revert "Fixed HttpBroadcast to work with this branch's Serializer."
...
This reverts commit b3eeac55b8
2012-06-17 14:29:39 -07:00
Matei Zaharia
94d77f83d3
Revert "Merge branch 'master' into dev"
...
This reverts commit f58da6164e4449eb9783
2012-06-17 14:27:45 -07:00
Matei Zaharia
0e84d620e1
Revert "Various fixes to get unit tests running. In particular, shut down"
...
This reverts commit 2893b30550
2012-06-17 14:27:30 -07:00
Matei Zaharia
4749ec063c
Revert "Fixed nasty corner case bug in ByteBufferInputStream. Could not add a test case for this as I could not figure out how to deterministically reproduce the bug in a short testcase."
...
This reverts commit 40536e3668
2012-06-17 14:27:18 -07:00
Tathagata Das
40536e3668
Fixed nasty corner case bug in ByteBufferInputStream. Could not add a test case for this as I could not figure out how to deterministically reproduce the bug in a short testcase.
2012-06-17 13:28:41 -07:00
Matei Zaharia
2893b30550
Various fixes to get unit tests running. In particular, shut down
...
ConnectionManager and DAGScheduler properly, plus a fix to
LocalScheduler that was not merged in from 0.5 and was actually caught
by one of the tests.
2012-06-17 00:28:45 -07:00
Matei Zaharia
b3eeac55b8
Fixed HttpBroadcast to work with this branch's Serializer.
2012-06-15 23:54:38 -07:00
Matei Zaharia
f58da6164e
Merge branch 'master' into dev
2012-06-15 23:47:11 -07:00
Tathagata Das
5f54bdf98b
Added shutdown for akka to SparkContext.stop(). Helps a little, but many testsuites still fail.
2012-06-13 20:49:00 -04:00
Tathagata Das
c6156da9e2
Multiple bug fixes to pass the testsuites ShuffleSuite and BlockManagerSuite.
2012-06-13 16:26:49 -04:00
Matei Zaharia
879bc0bece
Merge branch 'master' into mesos-0.9
2012-06-09 16:24:16 -07:00
Matei Zaharia
4b05798c06
Further bug fix to HttpBroadcast
2012-06-09 16:24:03 -07:00
Matei Zaharia
587a16a7ef
Merge branch 'master' into mesos-0.9
2012-06-09 16:17:07 -07:00
Matei Zaharia
8ed662862e
Bug fix to HttpBroadcast
2012-06-09 16:16:55 -07:00
Matei Zaharia
2fd9f994ae
Merge branch 'master' into mesos-0.9
2012-06-09 15:58:35 -07:00
Matei Zaharia
e75b1b5cb4
Change the default broadcast implementation to a simple HTTP-based
...
broadcast. Fixes #139 .
2012-06-09 15:58:07 -07:00
Matei Zaharia
a96558caa3
Performance improvements to shuffle operations: in particular, preserve
...
RDD partitioning in more cases where it's possible, and use iterators
instead of materializing collections when doing joins.
2012-06-09 14:44:18 -07:00
Matei Zaharia
c2c7299d7a
Added BlockManagerSuite, which I'd forgotten to merge.
2012-06-07 13:47:10 -07:00
Matei Zaharia
63051dd2bc
Merge in engine improvements from the Spark Streaming project, developed
...
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
2012-06-07 12:45:38 -07:00
Matei Zaharia
7e1c97fc4b
Merge branch 'master' into mesos-0.9
2012-06-06 16:48:59 -07:00
Matei Zaharia
048276799a
Commit task outputs to Hadoop-supported storage systems in parallel on the
...
cluster instead of on the master. Fixes #110 .
2012-06-06 16:46:53 -07:00
Matei Zaharia
6888bc7191
Merge branch 'master' into mesos-0.9
2012-06-06 16:14:19 -07:00
Matei Zaharia
6ae2746d1e
Handle arrays that contain the same element many times better in
...
SizeEstimator. Also added a test for SizeEstimator. Fixes #136 .
2012-06-06 16:13:02 -07:00
Matei Zaharia
0a617958d1
Some refactoring to make BoundedMemoryCache test similar to others
2012-06-06 16:12:08 -07:00
Matei Zaharia
dbc3c86ae3
Merge branch 'master' into mesos-0.9
...
Conflicts:
core/src/main/scala/spark/Executor.scala
2012-06-03 17:44:04 -07:00
Matei Zaharia
e141f644ca
Merge pull request #132 from Benky/rb-first-iteration
...
Little refactoring and unit tests for CacheTrackerActor
2012-05-26 13:15:06 -07:00
Richard Benkovsky
ae64920337
MesosScheduler refactoring
2012-05-22 11:04:54 +02:00
Richard Benkovsky
3a1bcd4028
Added tests for CacheTrackerActor
2012-05-22 11:04:54 +02:00
Richard Benkovsky
8f2f736d53
Little refactoring
2012-05-22 11:04:54 +02:00
Richard Benkovsky
518506a7c5
Added tests for Utils.copyStream
2012-05-22 11:04:51 +02:00
Richard Benkovsky
f162fc2beb
Formating fixed
2012-05-22 09:45:38 +02:00
Richard Benkovsky
565245871f
BoundedMemoryCache.put fails when estimated size of 'value' is larger than cache capacity
2012-05-20 22:13:35 +02:00
Richard Benkovsky
822a4be37d
Utils.memoryBytesToString fixed
2012-05-19 15:13:20 +02:00
Reynold Xin
d0c6e9f639
Made some RDD dependencies transient to reduce the amount of data needed
...
to be serialized in closure serialization. This can significantly reduce
the task setup time in Shark when the query involves a large number of
(Hive) partitions.
2012-05-16 14:16:55 -07:00
Reynold Xin
16461e2eda
Updated Cache's put method to use a case class for response. Previously
...
it was pretty ugly that put() should return -1 for failures.
2012-05-15 00:31:52 -07:00
Reynold Xin
019e48833f
Added the capacity to report cache usage status back to the cache
...
trackor. This is essential for building a dashboard to see the status of
caches on all slaves.
2012-05-14 18:39:04 -07:00
Matei Zaharia
f48742683a
Made caches dataset-aware so that they won't cyclically evict partitions
...
from the same dataset.
2012-05-06 20:14:40 -07:00
Matei Zaharia
bd2ab635a7
Fixed the way the JAR server is created after finding issue at Twitter
2012-05-05 20:05:15 -07:00
Matei Zaharia
32a4f4623c
Merge pull request #129 from mesos/rxin
...
Force serialize/deserialize task results in local execution mode.
2012-04-24 16:18:39 -07:00
Reynold Xin
761ea65a98
Added a test for the previous commit (failing to serialize task results
...
would throw an exception for local tasks).
2012-04-24 15:14:35 -07:00
Reynold Xin
9821cd4d42
Force serialize/deserialize task results in local execution mode.
2012-04-24 14:55:28 -07:00
Antonio
3e48818993
Removed commented-out System.exit call
2012-04-23 11:42:58 -07:00
Antonio
39d99168dc
Added exception handling instead of just exiting in LocalScheduler for tasks that throw exceptions
2012-04-20 14:46:43 -07:00
Reynold Xin
e601b3b9e5
Added the ability to set environmental variables in piped rdd.
2012-04-17 16:40:56 -07:00
Matei Zaharia
3b745176e0
Bug fix to pluggable closure serialization change
2012-04-12 17:53:02 +00:00
