Matei Zaharia
4e2fe0bdaf
Miscellaneous bug fixes
2012-07-06 16:33:40 -07:00
Matei Zaharia
e72afdb817
Some refactoring to make cluster scheduler pluggable.
2012-07-06 15:23:26 -07:00
Matei Zaharia
5d1a887bed
Further updates to run processes on cluster.
2012-07-01 17:13:31 -07:00
Matei Zaharia
51c46eaca0
More work on standalone deploy system.
2012-07-01 01:05:59 -07:00
Matei Zaharia
a6eb9fda61
Detect connection and disconnection of slaves
2012-06-30 17:46:56 -07:00
Matei Zaharia
408b5a1332
More work on deploy code (adding Worker class)
2012-06-30 16:45:57 -07:00
Matei Zaharia
2fb6e7d71e
Initial framework to get a master and web UI up.
2012-06-30 14:45:55 -07:00
Matei Zaharia
c53670b9bf
Various code style fixes, mostly from IntelliJ IDEA
2012-06-29 18:47:12 -07:00
rrmckinley
697b0bee2c
Scalacheck groupId has changed https://github.com/rickynils/scalacheck/issues/24 . Necessary to build with scalaVersion 2.9.2. Works with 2.9.1 too.
2012-06-29 16:42:05 -07:00
Matei Zaharia
cbe88465a9
Merge pull request #142 from rrmckinley/patch-1
...
Scalacheck groupId has changed https://github.com/rickynils/scalacheck/i ...
2012-06-29 16:40:57 -07:00
Matei Zaharia
8a5bd4e765
Increase the default wait time for EC2 clusters to 2 minutes.
2012-06-29 16:25:40 -07:00
Matei Zaharia
3a326c0ddd
Increase the default wait time for EC2 clusters to 2 minutes.
2012-06-29 16:25:06 -07:00
Matei Zaharia
c6be4ffbf9
Fixes to CoarseMesosScheduler
2012-06-29 16:18:51 -07:00
Matei Zaharia
3a58efa5a5
Allow binding to a free port and change Akka logging to use SLF4J. Also
...
fixes various bugs in the previous code when running on Mesos.
2012-06-29 16:02:21 -07:00
rrmckinley
25972b52cd
Scalacheck groupId has changed https://github.com/rickynils/scalacheck/issues/24 . Necessary to build with scalaVersion 2.9.2. Works with 2.9.1 too.
2012-06-29 12:00:23 -07:00
Matei Zaharia
14bfad1183
Ignore some IDE files
2012-06-28 23:52:33 -07:00
Matei Zaharia
3920189932
Upgraded to Akka 2 and fixed test execution (which was still parallel
...
across projects).
2012-06-28 23:51:28 -07:00
Tathagata Das
ede615d719
Fixed issues duplicate class issues in sbt assembly.
2012-06-22 15:03:09 -07:00
root
6ad3e1f1b4
Various fixes when running on Mesos
2012-06-20 06:48:26 +00:00
Tathagata Das
e896a505e2
Added testcase for ByteBufferInputStream bugs.
2012-06-17 16:11:12 -07:00
Matei Zaharia
3e0396c953
Update SBT and SBT-Eclipse version
2012-06-17 14:37:18 -07:00
Matei Zaharia
800fcbfbca
Revert "Fixed HttpBroadcast to work with this branch's Serializer."
...
This reverts commit b3eeac55b8
.
2012-06-17 14:29:39 -07:00
Matei Zaharia
94d77f83d3
Revert "Merge branch 'master' into dev"
...
This reverts commit f58da6164e
, reversing
changes made to 4449eb9783
.
2012-06-17 14:27:45 -07:00
Matei Zaharia
f46e867249
Revert "Update version number for dev branch"
...
This reverts commit 08579ffa11
.
2012-06-17 14:27:32 -07:00
Matei Zaharia
0e84d620e1
Revert "Various fixes to get unit tests running. In particular, shut down"
...
This reverts commit 2893b30550
.
2012-06-17 14:27:30 -07:00
Matei Zaharia
4749ec063c
Revert "Fixed nasty corner case bug in ByteBufferInputStream. Could not add a test case for this as I could not figure out how to deterministically reproduce the bug in a short testcase."
...
This reverts commit 40536e3668
.
2012-06-17 14:27:18 -07:00
Tathagata Das
40536e3668
Fixed nasty corner case bug in ByteBufferInputStream. Could not add a test case for this as I could not figure out how to deterministically reproduce the bug in a short testcase.
2012-06-17 13:28:41 -07:00
Matei Zaharia
2893b30550
Various fixes to get unit tests running. In particular, shut down
...
ConnectionManager and DAGScheduler properly, plus a fix to
LocalScheduler that was not merged in from 0.5 and was actually caught
by one of the tests.
2012-06-17 00:28:45 -07:00
Matei Zaharia
08579ffa11
Update version number for dev branch
2012-06-15 23:55:43 -07:00
Matei Zaharia
b3eeac55b8
Fixed HttpBroadcast to work with this branch's Serializer.
2012-06-15 23:54:38 -07:00
Matei Zaharia
f58da6164e
Merge branch 'master' into dev
2012-06-15 23:47:11 -07:00
Matei Zaharia
b5cf47cda3
Merge branch 'dev' of github.com:mesos/spark into dev
2012-06-15 23:36:30 -07:00
Matei Zaharia
4449eb9783
Changed version in master branch to 0.5.1-SNAPSHOT for further
...
development.
2012-06-13 22:26:14 -04:00
Tathagata Das
5f54bdf98b
Added shutdown for akka to SparkContext.stop(). Helps a little, but many testsuites still fail.
2012-06-13 20:49:00 -04:00
Tathagata Das
c6156da9e2
Multiple bug fixes to pass the testsuites ShuffleSuite and BlockManagerSuite.
2012-06-13 16:26:49 -04:00
Matei Zaharia
0472cf8e0d
Update version in SBT
2012-06-12 14:30:49 -04:00
Matei Zaharia
4971e0f547
Updated version number to 0.5.0
2012-06-12 13:41:57 -04:00
Matei Zaharia
08c50ad1fc
Added script for launching Spark on EC2 from Mesos, to make it easier
...
for new users to get up and running on EC2.
2012-06-10 23:06:15 -07:00
Matei Zaharia
879bc0bece
Merge branch 'master' into mesos-0.9
2012-06-09 16:24:16 -07:00
Matei Zaharia
4b05798c06
Further bug fix to HttpBroadcast
2012-06-09 16:24:03 -07:00
Matei Zaharia
587a16a7ef
Merge branch 'master' into mesos-0.9
2012-06-09 16:17:07 -07:00
Matei Zaharia
8ed662862e
Bug fix to HttpBroadcast
2012-06-09 16:16:55 -07:00
Matei Zaharia
2fd9f994ae
Merge branch 'master' into mesos-0.9
2012-06-09 15:58:35 -07:00
Matei Zaharia
e75b1b5cb4
Change the default broadcast implementation to a simple HTTP-based
...
broadcast. Fixes #139 .
2012-06-09 15:58:07 -07:00
Matei Zaharia
a96558caa3
Performance improvements to shuffle operations: in particular, preserve
...
RDD partitioning in more cases where it's possible, and use iterators
instead of materializing collections when doing joins.
2012-06-09 14:44:18 -07:00
Matei Zaharia
8b0bd5dd8d
Update SBT to version 0.11.3-2.
2012-06-07 23:05:58 -07:00
Matei Zaharia
c2c7299d7a
Added BlockManagerSuite, which I'd forgotten to merge.
2012-06-07 13:47:10 -07:00
Matei Zaharia
63051dd2bc
Merge in engine improvements from the Spark Streaming project, developed
...
jointly with Tathagata Das and Haoyuan Li. This commit imports the changes
and ports them to Mesos 0.9, but does not yet pass unit tests due to
various classes not supporting a graceful stop() yet.
2012-06-07 12:45:38 -07:00
Matei Zaharia
7e1c97fc4b
Merge branch 'master' into mesos-0.9
2012-06-06 16:48:59 -07:00
Matei Zaharia
048276799a
Commit task outputs to Hadoop-supported storage systems in parallel on the
...
cluster instead of on the master. Fixes #110 .
2012-06-06 16:46:53 -07:00