Tathagata Das
|
c97ebf6437
|
Fixed bug in the number of splits in RDD after checkpointing. Modified reduceByKeyAndWindow (naive) computation from window+reduceByKey to reduceByKey+window+reduceByKey.
|
2012-11-19 23:22:07 +00:00 |
|
Tathagata Das
|
3fd7b8319b
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-11-17 17:27:07 -08:00 |
|
Tathagata Das
|
10c1abcb6a
|
Fixed checkpointing bug in CoGroupedRDD. CoGroupSplits kept around the RDD splits of its parent RDDs, thus checkpointing its parents did not release the references to the parent splits.
|
2012-11-17 17:27:00 -08:00 |
|
Patrick Wendell
|
efa93fd0e6
|
Merge pull request #4 from radlab/streaming-example
A "streaming page view" example.
|
2012-11-16 20:40:27 -08:00 |
|
Patrick Wendell
|
720cb0f467
|
A "streaming page view" example.
|
2012-11-16 12:11:22 -08:00 |
|
Patrick Wendell
|
9563f7aba9
|
Merge pull request #3 from radlab/streaming-docs
Streaming programming guide. STREAMING-2 #resolve
|
2012-11-14 22:00:48 -08:00 |
|
Patrick Wendell
|
d39ac5fbc1
|
Streaming programming guide. STREAMING-2 #resolve
|
2012-11-13 21:19:58 -08:00 |
|
Tathagata Das
|
26fec8f0b8
|
Fixed bug in MappedValuesRDD, and set default graph checkpoint interval to be batch duration.
|
2012-11-13 11:05:57 -08:00 |
|
Tathagata Das
|
c3ccd14cf8
|
Replaced StateRDD in StateDStream with MapPartitionsRDD.
|
2012-11-13 02:43:03 -08:00 |
|
Tathagata Das
|
8a25d530ed
|
Optimized checkpoint writing by reusing FileSystem object. Fixed bug in updating of checkpoint data in DStream where the checkpointed RDDs, upon recovery, were not recognized as checkpointed RDDs and therefore deleted from HDFS. Made InputStreamsSuite more robust to timing delays.
|
2012-11-13 02:16:28 -08:00 |
|
Tathagata Das
|
564dd8c3f4
|
Speeded up CheckpointSuite
|
2012-11-12 14:22:05 -08:00 |
|
Tathagata Das
|
b9bfd1456f
|
Changed default level on calling DStream.persist() to be MEMORY_ONLY_SER. Also changed the persist level of StateDStream to be MEMORY_ONLY_SER.
|
2012-11-12 21:51:42 +00:00 |
|
Tathagata Das
|
ae61ebaee6
|
Fixed bugs in RawNetworkInputDStream and in its examples. Made the ReducedWindowedDStream persist RDDs to MEMOERY_SER_ONLY by default. Removed unncessary examples. Added streaming-env.sh.template to add recommended setting for streaming.
|
2012-11-12 21:45:16 +00:00 |
|
tdas
|
052d0b800f
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-11-11 22:56:14 +00:00 |
|
Tathagata Das
|
46222dc56d
|
Fixed bug in FileInputDStream that allowed it to miss new files. Added tests in the InputStreamsSuite to test checkpointing of file and network streams.
|
2012-11-11 13:20:09 -08:00 |
|
Tathagata Das
|
04e9e9d93c
|
Refactored BlockManagerMaster (not BlockManagerMasterActor) to simplify the code and fix live lock problem in unlimited attempts to contact the master. Also added testcases in the BlockManagerSuite to test BlockManagerMaster methods getPeers and getLocations.
|
2012-11-11 08:54:21 -08:00 |
|
Tathagata Das
|
62af376863
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-11-09 16:29:11 -08:00 |
|
Tathagata Das
|
355c8e4b17
|
Fixed deadlock in BlockManager.
|
2012-11-09 16:28:45 -08:00 |
|
tdas
|
52d21cb682
|
Removed unnecessary files.
|
2012-11-08 11:35:40 +00:00 |
|
tdas
|
cc2a65f547
|
Fixed bug in InputStreamsSuite
|
2012-11-08 11:17:57 +00:00 |
|
Tathagata Das
|
fc3d0b602a
|
Added FailureTestsuite for testing multiple, repeated master failures.
|
2012-11-06 17:23:31 -08:00 |
|
Tathagata Das
|
f8bb719cd2
|
Added a few more comments to the checkpoint-related functions.
|
2012-11-05 17:53:56 -08:00 |
|
Tathagata Das
|
395167f2b2
|
Made more bug fixes for checkpointing.
|
2012-11-05 16:11:50 -08:00 |
|
Tathagata Das
|
72b2303f99
|
Fixed major bugs in checkpointing.
|
2012-11-05 11:41:36 -08:00 |
|
Tathagata Das
|
d154238789
|
Made checkpointing of dstream graph to work with checkpointing of RDDs. For streams requiring checkpointing of its RDD, the default checkpoint interval is set to 10 seconds.
|
2012-11-04 12:12:06 -08:00 |
|
Tathagata Das
|
596154eabe
|
Merge branch 'dev-checkpoint' into dev
|
2012-11-02 17:05:22 -07:00 |
|
Tathagata Das
|
3fb5c9ee24
|
Fixed serialization bug in countByWindow, added countByKey and countByKeyAndWindow, and added testcases for them.
|
2012-11-02 12:12:25 -07:00 |
|
Tathagata Das
|
34e569f40e
|
Added 'synchronized' to RDD serialization to ensure checkpoint-related changes are reflected atomically in the task closure. Added to tests to ensure that jobs running on an RDD on which checkpointing is in progress does hurt the result of the job.
|
2012-10-31 00:56:40 -07:00 |
|
Tathagata Das
|
0dcd770fdc
|
Added checkpointing support to all RDDs, along with CheckpointSuite to test checkpointing in them.
|
2012-10-30 16:09:37 -07:00 |
|
Tathagata Das
|
ac12abc17f
|
Modified RDD API to make dependencies a var (therefore can be changed to checkpointed hadoop rdd) and othere references to parent RDDs either through dependencies or through a weak reference (to allow finalizing when dependencies do not refer to it any more).
|
2012-10-29 11:55:27 -07:00 |
|
Tathagata Das
|
1b900183c8
|
Added save operations to DStreams.
|
2012-10-27 18:55:50 -07:00 |
|
Tathagata Das
|
650d717544
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-10-25 13:03:18 -07:00 |
|
Matei Zaharia
|
863a55ae42
|
Merge remote-tracking branch 'public/master' into dev
Conflicts:
core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
core/src/main/scala/spark/KryoSerializer.scala
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/executor/Executor.scala
core/src/main/scala/spark/network/Connection.scala
core/src/main/scala/spark/network/ConnectionManagerTest.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/NewHadoopRDD.scala
core/src/main/scala/spark/scheduler/ShuffleMapTask.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockMessage.scala
core/src/main/scala/spark/storage/BlockStore.scala
core/src/main/scala/spark/storage/StorageLevel.scala
core/src/main/scala/spark/util/AkkaUtils.scala
project/SparkBuild.scala
run
|
2012-10-24 23:21:00 -07:00 |
|
Tathagata Das
|
926e05b030
|
Added tests for the file input stream.
|
2012-10-24 23:14:37 -07:00 |
|
Matei Zaharia
|
f63a40fd99
|
Strip leading mesos:// in URLs passed to Mesos
|
2012-10-24 21:52:13 -07:00 |
|
Tathagata Das
|
ed71df46cd
|
Minor fixes.
|
2012-10-24 16:49:40 -07:00 |
|
Tathagata Das
|
1ef6ea2513
|
Added tests for testing network input stream.
|
2012-10-24 14:44:20 -07:00 |
|
Matei Zaharia
|
d290e964ea
|
Merge pull request #281 from rxin/memreport
Added a method to report slave memory status; force serialize accumulator update in local mode.
|
2012-10-23 22:04:35 -07:00 |
|
Matei Zaharia
|
0bd20c63e2
|
Merge remote-tracking branch 'JoshRosen/shuffle_refactoring' into dev
Conflicts:
core/src/main/scala/spark/Dependency.scala
core/src/main/scala/spark/rdd/CoGroupedRDD.scala
core/src/main/scala/spark/rdd/ShuffledRDD.scala
|
2012-10-23 22:01:45 -07:00 |
|
Matei Zaharia
|
7849216bba
|
Merge pull request #286 from JoshRosen/ec2-error-handling
Allow EC2 script to stop/destroy cluster after master/slave failures
|
2012-10-23 21:15:43 -07:00 |
|
Matei Zaharia
|
46b87dfc3a
|
Merge pull request #292 from tomdz/tweaked-run-file
Tweaked run file to live more happily with typesafe's debian package
|
2012-10-23 21:14:06 -07:00 |
|
Tathagata Das
|
020d643484
|
Renamed the streaming testsuites.
|
2012-10-23 16:24:05 -07:00 |
|
Tathagata Das
|
0e5d9be4df
|
Renamed APIs to create queueStream and fileStream.
|
2012-10-23 15:17:05 -07:00 |
|
Tathagata Das
|
c2731dd3ef
|
Updated StateDStream api to use Options instead of nulls.
|
2012-10-23 15:10:27 -07:00 |
|
Tathagata Das
|
19191d178d
|
Renamed the network input streams.
|
2012-10-23 14:40:24 -07:00 |
|
Tathagata Das
|
a6de5758f1
|
Modified API of NetworkInputDStreams and got ObjectInputDStream and RawInputDStream working.
|
2012-10-23 01:41:13 -07:00 |
|
Tathagata Das
|
2c87c853ba
|
Renamed examples
|
2012-10-22 15:31:19 -07:00 |
|
Thomas Dudziak
|
f595bb53d1
|
Tweaked run file to live more happily with typesafe's debian package
|
2012-10-22 13:11:05 -07:00 |
|
Matei Zaharia
|
0967e71a00
|
Bump up version to 0.7.0-SNAPSHOT for master branch
|
2012-10-22 11:49:42 -07:00 |
|
Matei Zaharia
|
902a608187
|
Update version to 0.6.1-SNAPSHOT to show this is in development
|
2012-10-22 11:43:57 -07:00 |
|