Patrick Wendell
|
0d0bab25bd
|
Reduce tests
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
22a8c7be9a
|
Adding more tests
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
91b3d41448
|
Better equality test (thanks Josh)
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
b0974e6c1d
|
Remving main method from tests
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
867a7455e2
|
Adding some initial tests to streaming API.
|
2013-01-14 09:42:36 -08:00 |
|
Tathagata Das
|
6cc8592f26
|
Fixed bug
|
2013-01-13 21:20:49 -08:00 |
|
Tathagata Das
|
365506fb03
|
Changed variable name form ***Time to ***Duration to keep things consistent.
|
2013-01-09 14:29:25 -08:00 |
|
Tathagata Das
|
4719e6d8fe
|
Changed locations for unit test logs.
|
2013-01-07 16:06:07 -08:00 |
|
Tathagata Das
|
e60514d79e
|
Fixed bug
|
2013-01-07 15:16:16 -08:00 |
|
Tathagata Das
|
237bac36e9
|
Renamed examples and added documentation.
|
2013-01-07 14:37:21 -08:00 |
|
Tathagata Das
|
7e0271b438
|
Refactored a whole lot to push all DStreams into the spark.streaming.dstream package.
|
2012-12-30 15:19:55 -08:00 |
|
Tathagata Das
|
9e644402c1
|
Improved jekyll and scala docs. Made many classes and method private to remove them from scala docs.
|
2012-12-29 18:31:51 -08:00 |
|
Tathagata Das
|
0bc0a60d30
|
Modifications to make sure LocalScheduler terminate cleanly without errors when SparkContext is shutdown, to minimize spurious exception during master failure tests.
|
2012-12-27 15:37:33 -08:00 |
|
Patrick Wendell
|
3ff9710265
|
Adding Flume InputDStream
|
2012-12-07 16:42:39 -08:00 |
|
Denny
|
15df4b0e52
|
Merge branch 'dev' into kafka
Conflicts:
streaming/src/main/scala/spark/streaming/DStream.scala
|
2012-12-05 10:16:56 -08:00 |
|
Tathagata Das
|
0fe2fc4d5e
|
Merged branch mesos/master to branch dev.
|
2012-11-26 13:16:59 -08:00 |
|
Tathagata Das
|
fd11d23bb3
|
Modified StreamingContext API to make constructor accept the batch size (since it is always needed, Patrick's suggestion). Added description to DStream and StreamingContext.
|
2012-11-19 19:04:39 -08:00 |
|
Denny
|
2aceae25be
|
Merge branch 'dev' into kafka
Conflicts:
streaming/src/main/scala/spark/streaming/DStream.scala
|
2012-11-13 13:16:18 -08:00 |
|
Tathagata Das
|
8a25d530ed
|
Optimized checkpoint writing by reusing FileSystem object. Fixed bug in updating of checkpoint data in DStream where the checkpointed RDDs, upon recovery, were not recognized as checkpointed RDDs and therefore deleted from HDFS. Made InputStreamsSuite more robust to timing delays.
|
2012-11-13 02:16:28 -08:00 |
|
Denny
|
255b3e44c1
|
Merge branch 'dev' into kafka
|
2012-11-12 19:39:29 -08:00 |
|
Tathagata Das
|
564dd8c3f4
|
Speeded up CheckpointSuite
|
2012-11-12 14:22:05 -08:00 |
|
Tathagata Das
|
46222dc56d
|
Fixed bug in FileInputDStream that allowed it to miss new files. Added tests in the InputStreamsSuite to test checkpointing of file and network streams.
|
2012-11-11 13:20:09 -08:00 |
|
Denny
|
d006109e95
|
Kafka Stream comments.
|
2012-11-11 11:06:49 -08:00 |
|
tdas
|
cc2a65f547
|
Fixed bug in InputStreamsSuite
|
2012-11-08 11:17:57 +00:00 |
|
Tathagata Das
|
fc3d0b602a
|
Added FailureTestsuite for testing multiple, repeated master failures.
|
2012-11-06 17:23:31 -08:00 |
|
Tathagata Das
|
395167f2b2
|
Made more bug fixes for checkpointing.
|
2012-11-05 16:11:50 -08:00 |
|
Tathagata Das
|
72b2303f99
|
Fixed major bugs in checkpointing.
|
2012-11-05 11:41:36 -08:00 |
|
Tathagata Das
|
d154238789
|
Made checkpointing of dstream graph to work with checkpointing of RDDs. For streams requiring checkpointing of its RDD, the default checkpoint interval is set to 10 seconds.
|
2012-11-04 12:12:06 -08:00 |
|
Tathagata Das
|
3fb5c9ee24
|
Fixed serialization bug in countByWindow, added countByKey and countByKeyAndWindow, and added testcases for them.
|
2012-11-02 12:12:25 -07:00 |
|
Tathagata Das
|
650d717544
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-10-25 13:03:18 -07:00 |
|
Matei Zaharia
|
863a55ae42
|
Merge remote-tracking branch 'public/master' into dev
Conflicts:
core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
core/src/main/scala/spark/KryoSerializer.scala
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/executor/Executor.scala
core/src/main/scala/spark/network/Connection.scala
core/src/main/scala/spark/network/ConnectionManagerTest.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/NewHadoopRDD.scala
core/src/main/scala/spark/scheduler/ShuffleMapTask.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockMessage.scala
core/src/main/scala/spark/storage/BlockStore.scala
core/src/main/scala/spark/storage/StorageLevel.scala
core/src/main/scala/spark/util/AkkaUtils.scala
project/SparkBuild.scala
run
|
2012-10-24 23:21:00 -07:00 |
|
Tathagata Das
|
926e05b030
|
Added tests for the file input stream.
|
2012-10-24 23:14:37 -07:00 |
|
Tathagata Das
|
ed71df46cd
|
Minor fixes.
|
2012-10-24 16:49:40 -07:00 |
|
Tathagata Das
|
1ef6ea2513
|
Added tests for testing network input stream.
|
2012-10-24 14:44:20 -07:00 |
|
Tathagata Das
|
020d643484
|
Renamed the streaming testsuites.
|
2012-10-23 16:24:05 -07:00 |
|
Tathagata Das
|
c2731dd3ef
|
Updated StateDStream api to use Options instead of nulls.
|
2012-10-23 15:10:27 -07:00 |
|
Tathagata Das
|
d85c66636b
|
Added MapValueDStream, FlatMappedValuesDStream and CoGroupedDStream, and therefore DStream operations mapValue, flatMapValues, cogroup, and join. Also, added tests for DStream operations filter, glom, mapPartitions, groupByKey, mapValues, flatMapValues, cogroup, and join.
|
2012-10-21 17:40:08 -07:00 |
|
Tathagata Das
|
c4a2b6f636
|
Fixed some bugs in tests for forgetting RDDs, and made sure that use of manual clock leads to a zeroTime of 0 in the DStreams (more intuitive).
|
2012-10-21 10:41:25 -07:00 |
|
Tathagata Das
|
6d5eb4b40c
|
Added functionality to forget RDDs from DStreams.
|
2012-10-19 12:11:44 -07:00 |
|
Tathagata Das
|
b760d6426a
|
Minor modifications.
|
2012-10-15 12:26:44 -07:00 |
|
Tathagata Das
|
3f1aae5c71
|
Refactored DStreamSuiteBase to create CheckpointSuite- testsuite for testing checkpointing under different operations.
|
2012-10-14 21:39:30 -07:00 |
|
Tathagata Das
|
b08708e6fc
|
Fixed bugs in the streaming testsuites.
|
2012-10-13 21:02:24 -07:00 |
|
Tathagata Das
|
4a7bde6865
|
Fixed bugs and added testcases for naive reduceByKeyAndWindow.
|
2012-09-06 19:06:59 -07:00 |
|
Tathagata Das
|
babb7e3ce2
|
Re-implemented ReducedWindowedDSteam to simplify and fix bugs. Added slice operator to DStream. Also, refactored DStream testsuites and added tests for reduceByKeyAndWindow.
|
2012-09-06 05:28:29 -07:00 |
|
Tathagata Das
|
7c09ad0e04
|
Changed DStream member access permissions from private to protected. Updated StateDStream to checkpoint RDDs and forget lineage.
|
2012-09-04 19:11:49 -07:00 |
|
Tathagata Das
|
7419d2c7ea
|
Added transformRDD DStream operation and TransformedDStream. Added sbt assembly option for streaming project.
|
2012-09-02 02:35:17 -07:00 |
|
Tathagata Das
|
2d01d38a41
|
Added StateDStream, corresponding stateful stream operations, and testcases. Also refactored few PairDStreamFunctions methods.
|
2012-08-31 03:47:34 -07:00 |
|
Tathagata Das
|
43e66146f7
|
Merge branch 'dev' of github.com/radlab/spark into dev
|
2012-08-28 13:51:05 -07:00 |
|
Tathagata Das
|
b5b93a621c
|
Added capabllity to take streaming input from network. Renamed SparkStreamContext to StreamingContext.
|
2012-08-28 12:35:19 -07:00 |
|
Matei Zaharia
|
e7a5cbb543
|
Reduce log4j verbosity for streaming
|
2012-08-24 16:45:01 -07:00 |
|