Imran Rashid
0384be3467
tasks cannot access value of accumulator
2012-07-26 12:38:51 -07:00
Imran Rashid
7f43ba7ffa
one more minor cleanup to scaladoc
2012-07-16 18:26:48 -07:00
Imran Rashid
913d42c6a0
fix up scaladoc, naming of type parameters
2012-07-16 18:25:15 -07:00
Imran Rashid
85940a7d71
rename addToAccum to addAccumulator
2012-07-16 18:17:13 -07:00
Imran Rashid
86024ca74d
add some functionality to Vector, delete copy in AccumulatorSuite
2012-07-12 13:08:03 -07:00
Imran Rashid
42ce879486
move Vector class into core and spark.util package
2012-07-12 13:08:03 -07:00
Imran Rashid
13cc72cfb5
Accumulator now inherits from Accumulable, whcih simplifies a bunch of other things (eg., no +:=)
2012-07-12 12:40:10 -07:00
Imran Rashid
73935629a1
improve scaladoc
2012-07-12 09:58:06 -07:00
Imran Rashid
30480e6dae
add Accumulatable, add corresponding docs & tests for accumulators
2012-07-12 09:37:42 -07:00
Matei Zaharia
4259d37f84
Formatting
2012-07-10 11:16:34 -07:00
Matei Zaharia
6980b67557
Added more methods for loading/saving with new Hadoop API
2012-07-10 11:11:35 -07:00
Matei Zaharia
cbe88465a9
Merge pull request #142 from rrmckinley/patch-1
...
Scalacheck groupId has changed https://github.com/rickynils/scalacheck/i ...
2012-06-29 16:40:57 -07:00
Matei Zaharia
3a326c0ddd
Increase the default wait time for EC2 clusters to 2 minutes.
2012-06-29 16:25:06 -07:00
rrmckinley
25972b52cd
Scalacheck groupId has changed https://github.com/rickynils/scalacheck/issues/24 . Necessary to build with scalaVersion 2.9.2. Works with 2.9.1 too.
2012-06-29 12:00:23 -07:00
Matei Zaharia
800fcbfbca
Revert "Fixed HttpBroadcast to work with this branch's Serializer."
...
This reverts commit b3eeac55b8
.
2012-06-17 14:29:39 -07:00
Matei Zaharia
94d77f83d3
Revert "Merge branch 'master' into dev"
...
This reverts commit f58da6164e
, reversing
changes made to 4449eb9783
.
2012-06-17 14:27:45 -07:00
Matei Zaharia
f46e867249
Revert "Update version number for dev branch"
...
This reverts commit 08579ffa11
.
2012-06-17 14:27:32 -07:00
Matei Zaharia
0e84d620e1
Revert "Various fixes to get unit tests running. In particular, shut down"
...
This reverts commit 2893b30550
.
2012-06-17 14:27:30 -07:00
Matei Zaharia
4749ec063c
Revert "Fixed nasty corner case bug in ByteBufferInputStream. Could not add a test case for this as I could not figure out how to deterministically reproduce the bug in a short testcase."
...
This reverts commit 40536e3668
.
2012-06-17 14:27:18 -07:00
Tathagata Das
40536e3668
Fixed nasty corner case bug in ByteBufferInputStream. Could not add a test case for this as I could not figure out how to deterministically reproduce the bug in a short testcase.
2012-06-17 13:28:41 -07:00
Matei Zaharia
2893b30550
Various fixes to get unit tests running. In particular, shut down
...
ConnectionManager and DAGScheduler properly, plus a fix to
LocalScheduler that was not merged in from 0.5 and was actually caught
by one of the tests.
2012-06-17 00:28:45 -07:00
Matei Zaharia
08579ffa11
Update version number for dev branch
2012-06-15 23:55:43 -07:00
Matei Zaharia
b3eeac55b8
Fixed HttpBroadcast to work with this branch's Serializer.
2012-06-15 23:54:38 -07:00
Matei Zaharia
f58da6164e
Merge branch 'master' into dev
2012-06-15 23:47:11 -07:00
Matei Zaharia
b5cf47cda3
Merge branch 'dev' of github.com:mesos/spark into dev
2012-06-15 23:36:30 -07:00
Matei Zaharia
4449eb9783
Changed version in master branch to 0.5.1-SNAPSHOT for further
...
development.
2012-06-13 22:26:14 -04:00
Tathagata Das
5f54bdf98b
Added shutdown for akka to SparkContext.stop(). Helps a little, but many testsuites still fail.
2012-06-13 20:49:00 -04:00
Tathagata Das
c6156da9e2
Multiple bug fixes to pass the testsuites ShuffleSuite and BlockManagerSuite.
2012-06-13 16:26:49 -04:00
Matei Zaharia
0472cf8e0d
Update version in SBT
2012-06-12 14:30:49 -04:00
Matei Zaharia
4971e0f547
Updated version number to 0.5.0
2012-06-12 13:41:57 -04:00
Matei Zaharia
08c50ad1fc
Added script for launching Spark on EC2 from Mesos, to make it easier
...
for new users to get up and running on EC2.
2012-06-10 23:06:15 -07:00
Matei Zaharia
879bc0bece
Merge branch 'master' into mesos-0.9
2012-06-09 16:24:16 -07:00
Matei Zaharia
4b05798c06
Further bug fix to HttpBroadcast
2012-06-09 16:24:03 -07:00
Matei Zaharia
587a16a7ef
Merge branch 'master' into mesos-0.9
2012-06-09 16:17:07 -07:00
Matei Zaharia
8ed662862e
Bug fix to HttpBroadcast
2012-06-09 16:16:55 -07:00
Matei Zaharia
2fd9f994ae
Merge branch 'master' into mesos-0.9
2012-06-09 15:58:35 -07:00
Matei Zaharia
e75b1b5cb4
Change the default broadcast implementation to a simple HTTP-based
...
broadcast. Fixes #139 .
2012-06-09 15:58:07 -07:00
Matei Zaharia
a96558caa3
Performance improvements to shuffle operations: in particular, preserve
...
RDD partitioning in more cases where it's possible, and use iterators
instead of materializing collections when doing joins.
2012-06-09 14:44:18 -07:00
Matei Zaharia
8b0bd5dd8d
Update SBT to version 0.11.3-2.
2012-06-07 23:05:58 -07:00
Matei Zaharia
c2c7299d7a
Added BlockManagerSuite, which I'd forgotten to merge.
2012-06-07 13:47:10 -07:00
Matei Zaharia
63051dd2bc
Merge in engine improvements from the Spark Streaming project, developed
...
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
2012-06-07 12:45:38 -07:00
Matei Zaharia
7e1c97fc4b
Merge branch 'master' into mesos-0.9
2012-06-06 16:48:59 -07:00
Matei Zaharia
048276799a
Commit task outputs to Hadoop-supported storage systems in parallel on the
...
cluster instead of on the master. Fixes #110 .
2012-06-06 16:46:53 -07:00
Matei Zaharia
6888bc7191
Merge branch 'master' into mesos-0.9
2012-06-06 16:14:19 -07:00
Matei Zaharia
6ae2746d1e
Handle arrays that contain the same element many times better in
...
SizeEstimator. Also added a test for SizeEstimator. Fixes #136 .
2012-06-06 16:13:02 -07:00
Matei Zaharia
0a617958d1
Some refactoring to make BoundedMemoryCache test similar to others
2012-06-06 16:12:08 -07:00
Matei Zaharia
1fa0da5dd7
Merge branch 'master' into mesos-0.9
2012-06-05 23:31:46 -07:00
Matei Zaharia
28fed4ce3b
Add System.exit(0) at the end of all the example programs.
2012-06-05 23:31:28 -07:00
Matei Zaharia
dbc3c86ae3
Merge branch 'master' into mesos-0.9
...
Conflicts:
core/src/main/scala/spark/Executor.scala
2012-06-03 17:44:04 -07:00
Reynold Xin
1dd7d3dfff
Merge branch 'master' of github.com:mesos/spark
2012-05-30 18:41:07 -07:00