ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1474 commits 6 branches 181 tags 397 MiB
Commit graph

91 commits

Author SHA1 Message Date
Tathagata Das b9bfd1456f Changed default level on calling DStream.persist() to be MEMORY_ONLY_SER. Also changed the persist level of StateDStream to be MEMORY_ONLY_SER. 2012-11-12 21:51:42 +00:00
Tathagata Das ae61ebaee6 Fixed bugs in RawNetworkInputDStream and in its examples. Made the ReducedWindowedDStream persist RDDs to MEMOERY_SER_ONLY by default. Removed unncessary examples. Added streaming-env.sh.template to add recommended setting for streaming. 2012-11-12 21:45:16 +00:00
tdas 052d0b800f Merge branch 'dev' of github.com:radlab/spark into dev 2012-11-11 22:56:14 +00:00
Tathagata Das 46222dc56d Fixed bug in FileInputDStream that allowed it to miss new files. Added tests in the InputStreamsSuite to test checkpointing of file and network streams. 2012-11-11 13:20:09 -08:00
tdas 52d21cb682 Removed unnecessary files. 2012-11-08 11:35:40 +00:00
tdas cc2a65f547 Fixed bug in InputStreamsSuite 2012-11-08 11:17:57 +00:00
Tathagata Das fc3d0b602a Added FailureTestsuite for testing multiple, repeated master failures. 2012-11-06 17:23:31 -08:00
Tathagata Das f8bb719cd2 Added a few more comments to the checkpoint-related functions. 2012-11-05 17:53:56 -08:00
Tathagata Das 395167f2b2 Made more bug fixes for checkpointing. 2012-11-05 16:11:50 -08:00
Tathagata Das 72b2303f99 Fixed major bugs in checkpointing. 2012-11-05 11:41:36 -08:00
Tathagata Das d154238789 Made checkpointing of dstream graph to work with checkpointing of RDDs. For streams requiring checkpointing of its RDD, the default checkpoint interval is set to 10 seconds. 2012-11-04 12:12:06 -08:00
Tathagata Das 3fb5c9ee24 Fixed serialization bug in countByWindow, added countByKey and countByKeyAndWindow, and added testcases for them. 2012-11-02 12:12:25 -07:00
Tathagata Das 1b900183c8 Added save operations to DStreams. 2012-10-27 18:55:50 -07:00
Tathagata Das 650d717544 Merge branch 'dev' of github.com:radlab/spark into dev 2012-10-25 13:03:18 -07:00
Matei Zaharia 863a55ae42 Merge remote-tracking branch 'public/master' into dev 
Conflicts:
	core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
	core/src/main/scala/spark/KryoSerializer.scala
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/executor/Executor.scala
	core/src/main/scala/spark/network/Connection.scala
	core/src/main/scala/spark/network/ConnectionManagerTest.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/NewHadoopRDD.scala
	core/src/main/scala/spark/scheduler/ShuffleMapTask.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockMessage.scala
	core/src/main/scala/spark/storage/BlockStore.scala
	core/src/main/scala/spark/storage/StorageLevel.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	project/SparkBuild.scala
	run
 2012-10-24 23:21:00 -07:00
Tathagata Das 926e05b030 Added tests for the file input stream. 2012-10-24 23:14:37 -07:00
Tathagata Das ed71df46cd Minor fixes. 2012-10-24 16:49:40 -07:00
Tathagata Das 1ef6ea2513 Added tests for testing network input stream. 2012-10-24 14:44:20 -07:00
Tathagata Das 020d643484 Renamed the streaming testsuites. 2012-10-23 16:24:05 -07:00
Tathagata Das 0e5d9be4df Renamed APIs to create queueStream and fileStream. 2012-10-23 15:17:05 -07:00
Tathagata Das c2731dd3ef Updated StateDStream api to use Options instead of nulls. 2012-10-23 15:10:27 -07:00
Tathagata Das 19191d178d Renamed the network input streams. 2012-10-23 14:40:24 -07:00
Tathagata Das a6de5758f1 Modified API of NetworkInputDStreams and got ObjectInputDStream and RawInputDStream working. 2012-10-23 01:41:13 -07:00
Tathagata Das 2c87c853ba Renamed examples 2012-10-22 15:31:19 -07:00
Tathagata Das d85c66636b Added MapValueDStream, FlatMappedValuesDStream and CoGroupedDStream, and therefore DStream operations mapValue, flatMapValues, cogroup, and join. Also, added tests for DStream operations filter, glom, mapPartitions, groupByKey, mapValues, flatMapValues, cogroup, and join. 2012-10-21 17:40:08 -07:00
Tathagata Das c4a2b6f636 Fixed some bugs in tests for forgetting RDDs, and made sure that use of manual clock leads to a zeroTime of 0 in the DStreams (more intuitive). 2012-10-21 10:41:25 -07:00
Tathagata Das 6d5eb4b40c Added functionality to forget RDDs from DStreams. 2012-10-19 12:11:44 -07:00
Tathagata Das b760d6426a Minor modifications. 2012-10-15 12:26:44 -07:00
Tathagata Das 3f1aae5c71 Refactored DStreamSuiteBase to create CheckpointSuite- testsuite for testing checkpointing under different operations. 2012-10-14 21:39:30 -07:00
Tathagata Das b08708e6fc Fixed bugs in the streaming testsuites. 2012-10-13 21:02:24 -07:00
Tathagata Das e95ff45b53 Implemented checkpointing of StreamingContext and DStream graph. 2012-10-13 20:10:49 -07:00
Tathagata Das 6d1fe02685 Merge branch 'dev' of github.com:radlab/spark into dev 2012-09-17 14:26:06 -07:00
Tathagata Das 86d420478f Allowed StreamingContext to be created from existing SparkContext 2012-09-17 14:25:48 -07:00
Tathagata Das 3cbc72ff1d Minor tweaks 2012-09-14 07:00:30 +00:00
Tathagata Das 0269792c17 Merge branch 'dev' of github.com:radlab/spark into dev 
Conflicts:
	streaming/src/main/scala/spark/streaming/Scheduler.scala
 2012-09-07 20:18:30 +00:00
Tathagata Das b5750726ff Fixed bugs in streaming Scheduler and optimized QueueInputDStream. 2012-09-07 20:16:21 +00:00
haoyuan 381e2c7ac4 add warmup code for TopKWordCountRaw.scala 2012-09-06 20:54:52 -07:00
haoyuan 0681bbc5d9 Merge branch 'dev' of github.com:radlab/spark into dev 2012-09-07 02:18:33 +00:00
haoyuan db08a362aa commit opt for grep scalibility test. 2012-09-07 02:17:52 +00:00
Tathagata Das 4a7bde6865 Fixed bugs and added testcases for naive reduceByKeyAndWindow. 2012-09-06 19:06:59 -07:00
Tathagata Das 203ac8fa8b Merge branch 'dev' of github.com:radlab/spark into dev 2012-09-06 05:29:06 -07:00
Tathagata Das babb7e3ce2 Re-implemented ReducedWindowedDSteam to simplify and fix bugs. Added slice operator to DStream. Also, refactored DStream testsuites and added tests for reduceByKeyAndWindow. 2012-09-06 05:28:29 -07:00
root 019de4562c Less warmup in word count 2012-09-06 02:50:41 +00:00
root 4a5d0d249e Merge branch 'dev' of github.com:radlab/spark into dev 2012-09-05 08:23:09 +00:00
root b7ad291ac5 Tuning Akka for more connections 2012-09-05 07:08:07 +00:00
root fc186dc18a Merge branch 'dev' of github.com:radlab/spark into dev 2012-09-05 05:53:18 +00:00
root 4ea032a142 Some changes to make important log output visible even if we set the logging to WARNING 2012-09-05 05:53:07 +00:00
Tathagata Das 25fd684b89 Merge branch 'dev' of github.com:radlab/spark into dev 2012-09-04 20:44:14 -07:00
Tathagata Das 7c09ad0e04 Changed DStream member access permissions from private to protected. Updated StateDStream to checkpoint RDDs and forget lineage. 2012-09-04 19:11:49 -07:00
haoyuan 96a1f2277d fix the compile error in TopKWordCountRaw.scala 2012-09-04 18:03:34 -07:00