Denny
|
deb2c4df72
|
Add comment.
|
2012-11-11 11:11:49 -08:00 |
|
Denny
|
d006109e95
|
Kafka Stream comments.
|
2012-11-11 11:06:49 -08:00 |
|
Denny
|
2e8f2ee4ad
|
Merge branch 'dev' of github.com:radlab/spark into kafka
Conflicts:
streaming/src/main/scala/spark/streaming/DStream.scala
|
2012-11-09 12:26:17 -08:00 |
|
Denny
|
e5a0936787
|
Kafka Stream.
|
2012-11-09 12:23:46 -08:00 |
|
Tathagata Das
|
fc3d0b602a
|
Added FailureTestsuite for testing multiple, repeated master failures.
|
2012-11-06 17:23:31 -08:00 |
|
Denny
|
485803d740
|
Merge branch 'dev' of github.com:radlab/spark into kafka
|
2012-11-06 09:41:45 -08:00 |
|
Denny
|
0c1de43fc7
|
Working on kafka.
|
2012-11-06 09:41:42 -08:00 |
|
Tathagata Das
|
f8bb719cd2
|
Added a few more comments to the checkpoint-related functions.
|
2012-11-05 17:53:56 -08:00 |
|
Tathagata Das
|
395167f2b2
|
Made more bug fixes for checkpointing.
|
2012-11-05 16:11:50 -08:00 |
|
Tathagata Das
|
72b2303f99
|
Fixed major bugs in checkpointing.
|
2012-11-05 11:41:36 -08:00 |
|
Tathagata Das
|
d154238789
|
Made checkpointing of dstream graph to work with checkpointing of RDDs. For streams requiring checkpointing of its RDD, the default checkpoint interval is set to 10 seconds.
|
2012-11-04 12:12:06 -08:00 |
|
Tathagata Das
|
3fb5c9ee24
|
Fixed serialization bug in countByWindow, added countByKey and countByKeyAndWindow, and added testcases for them.
|
2012-11-02 12:12:25 -07:00 |
|
Tathagata Das
|
1b900183c8
|
Added save operations to DStreams.
|
2012-10-27 18:55:50 -07:00 |
|
Tathagata Das
|
650d717544
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-10-25 13:03:18 -07:00 |
|
Matei Zaharia
|
863a55ae42
|
Merge remote-tracking branch 'public/master' into dev
Conflicts:
core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
core/src/main/scala/spark/KryoSerializer.scala
core/src/main/scala/spark/MapOutputTracker.scala
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/executor/Executor.scala
core/src/main/scala/spark/network/Connection.scala
core/src/main/scala/spark/network/ConnectionManagerTest.scala
core/src/main/scala/spark/rdd/BlockRDD.scala
core/src/main/scala/spark/rdd/NewHadoopRDD.scala
core/src/main/scala/spark/scheduler/ShuffleMapTask.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/main/scala/spark/storage/BlockManager.scala
core/src/main/scala/spark/storage/BlockMessage.scala
core/src/main/scala/spark/storage/BlockStore.scala
core/src/main/scala/spark/storage/StorageLevel.scala
core/src/main/scala/spark/util/AkkaUtils.scala
project/SparkBuild.scala
run
|
2012-10-24 23:21:00 -07:00 |
|
Tathagata Das
|
926e05b030
|
Added tests for the file input stream.
|
2012-10-24 23:14:37 -07:00 |
|
Tathagata Das
|
ed71df46cd
|
Minor fixes.
|
2012-10-24 16:49:40 -07:00 |
|
Tathagata Das
|
1ef6ea2513
|
Added tests for testing network input stream.
|
2012-10-24 14:44:20 -07:00 |
|
Tathagata Das
|
0e5d9be4df
|
Renamed APIs to create queueStream and fileStream.
|
2012-10-23 15:17:05 -07:00 |
|
Tathagata Das
|
c2731dd3ef
|
Updated StateDStream api to use Options instead of nulls.
|
2012-10-23 15:10:27 -07:00 |
|
Tathagata Das
|
19191d178d
|
Renamed the network input streams.
|
2012-10-23 14:40:24 -07:00 |
|
Tathagata Das
|
a6de5758f1
|
Modified API of NetworkInputDStreams and got ObjectInputDStream and RawInputDStream working.
|
2012-10-23 01:41:13 -07:00 |
|
Tathagata Das
|
2c87c853ba
|
Renamed examples
|
2012-10-22 15:31:19 -07:00 |
|
Tathagata Das
|
d85c66636b
|
Added MapValueDStream, FlatMappedValuesDStream and CoGroupedDStream, and therefore DStream operations mapValue, flatMapValues, cogroup, and join. Also, added tests for DStream operations filter, glom, mapPartitions, groupByKey, mapValues, flatMapValues, cogroup, and join.
|
2012-10-21 17:40:08 -07:00 |
|
Tathagata Das
|
c4a2b6f636
|
Fixed some bugs in tests for forgetting RDDs, and made sure that use of manual clock leads to a zeroTime of 0 in the DStreams (more intuitive).
|
2012-10-21 10:41:25 -07:00 |
|
Tathagata Das
|
6d5eb4b40c
|
Added functionality to forget RDDs from DStreams.
|
2012-10-19 12:11:44 -07:00 |
|
Tathagata Das
|
b760d6426a
|
Minor modifications.
|
2012-10-15 12:26:44 -07:00 |
|
Tathagata Das
|
3f1aae5c71
|
Refactored DStreamSuiteBase to create CheckpointSuite- testsuite for testing checkpointing under different operations.
|
2012-10-14 21:39:30 -07:00 |
|
Tathagata Das
|
e95ff45b53
|
Implemented checkpointing of StreamingContext and DStream graph.
|
2012-10-13 20:10:49 -07:00 |
|
Tathagata Das
|
6d1fe02685
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-17 14:26:06 -07:00 |
|
Tathagata Das
|
86d420478f
|
Allowed StreamingContext to be created from existing SparkContext
|
2012-09-17 14:25:48 -07:00 |
|
Tathagata Das
|
3cbc72ff1d
|
Minor tweaks
|
2012-09-14 07:00:30 +00:00 |
|
Tathagata Das
|
0269792c17
|
Merge branch 'dev' of github.com:radlab/spark into dev
Conflicts:
streaming/src/main/scala/spark/streaming/Scheduler.scala
|
2012-09-07 20:18:30 +00:00 |
|
Tathagata Das
|
b5750726ff
|
Fixed bugs in streaming Scheduler and optimized QueueInputDStream.
|
2012-09-07 20:16:21 +00:00 |
|
haoyuan
|
381e2c7ac4
|
add warmup code for TopKWordCountRaw.scala
|
2012-09-06 20:54:52 -07:00 |
|
haoyuan
|
0681bbc5d9
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-07 02:18:33 +00:00 |
|
haoyuan
|
db08a362aa
|
commit opt for grep scalibility test.
|
2012-09-07 02:17:52 +00:00 |
|
Tathagata Das
|
4a7bde6865
|
Fixed bugs and added testcases for naive reduceByKeyAndWindow.
|
2012-09-06 19:06:59 -07:00 |
|
Tathagata Das
|
203ac8fa8b
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-06 05:29:06 -07:00 |
|
Tathagata Das
|
babb7e3ce2
|
Re-implemented ReducedWindowedDSteam to simplify and fix bugs. Added slice operator to DStream. Also, refactored DStream testsuites and added tests for reduceByKeyAndWindow.
|
2012-09-06 05:28:29 -07:00 |
|
root
|
019de4562c
|
Less warmup in word count
|
2012-09-06 02:50:41 +00:00 |
|
root
|
4a5d0d249e
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-05 08:23:09 +00:00 |
|
root
|
b7ad291ac5
|
Tuning Akka for more connections
|
2012-09-05 07:08:07 +00:00 |
|
root
|
fc186dc18a
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-05 05:53:18 +00:00 |
|
root
|
4ea032a142
|
Some changes to make important log output visible even if we set the logging to WARNING
|
2012-09-05 05:53:07 +00:00 |
|
Tathagata Das
|
25fd684b89
|
Merge branch 'dev' of github.com:radlab/spark into dev
|
2012-09-04 20:44:14 -07:00 |
|
Tathagata Das
|
7c09ad0e04
|
Changed DStream member access permissions from private to protected. Updated StateDStream to checkpoint RDDs and forget lineage.
|
2012-09-04 19:11:49 -07:00 |
|
haoyuan
|
96a1f2277d
|
fix the compile error in TopKWordCountRaw.scala
|
2012-09-04 18:03:34 -07:00 |
|
haoyuan
|
2ff72f60ac
|
add TopKWordCountRaw.scala
|
2012-09-04 17:55:55 -07:00 |
|
Tathagata Das
|
389a78722c
|
Updated the return types of PairDStreamFunctions to return DStreams instead of ShuffleDStreams for cleaner abstraction.
|
2012-09-04 15:37:46 -07:00 |
|