Tathagata Das
|
39addd3803
|
Changed scheduler and file input stream to fix bugs in the driver fault tolerance. Added MasterFailureTest to rigorously test master fault tolerance with file input stream.
|
2013-02-13 12:17:45 -08:00 |
|
Tathagata Das
|
fd90daf850
|
Fixed bugs in FileInputDStream and Scheduler that occasionally failed to reprocess old files after recovering from master failure. Completely modified spark.streaming.FailureTest to test multiple master failures using file input stream.
|
2013-02-10 19:48:42 -08:00 |
|
Tathagata Das
|
99a5fc498a
|
Added an initial spark job to ensure worker nodes are initialized.
|
2013-02-09 15:18:05 -08:00 |
|
Tathagata Das
|
4cc223b478
|
Merge branch 'mesos-master' into streaming
|
2013-02-07 13:59:31 -08:00 |
|
Tathagata Das
|
d55e3aa467
|
Updated JavaStreamingContext with updated kafkaStream API.
|
2013-02-07 13:59:18 -08:00 |
|
Tathagata Das
|
c6b2f765d3
|
Merge branch 'mesos-streaming' into streaming
|
2013-02-07 13:13:53 -08:00 |
|
Tathagata Das
|
12300758cc
|
Merge pull request #372 from Reinvigorate/sm-kafka
Removing offset management code that is non-existent in kafka 0.7.0+
|
2013-02-07 12:41:07 -08:00 |
|
Tathagata Das
|
915d9931fe
|
Merge pull request #373 from Reinvigorate/sm-updateStateByKey
StateDStream changes to give updateStateByKey consistent behavior
|
2013-02-07 11:59:19 -08:00 |
|
Patrick Wendell
|
7eea64aa4c
|
Streaming constructor which takes JavaSparkContext
It's sometimes helpful to directly pass a JavaSparkContext,
and take advantage of the various constructors available for that.
|
2013-02-05 11:43:16 -08:00 |
|
Matei Zaharia
|
9ae11603b4
|
Merge pull request #415 from stephenh/driver
Replace old 'master' term with 'driver'.
|
2013-01-29 10:41:42 -08:00 |
|
Matei Zaharia
|
b29599e5cf
|
Fix code that depended on metadata cleaner interval being in minutes
|
2013-01-28 22:24:47 -08:00 |
|
Stephen Haberman
|
7dfb82a992
|
Replace old 'master' term with 'driver'.
|
2013-01-25 11:03:00 -06:00 |
|
Tathagata Das
|
666ce431aa
|
Added support for rescheduling unprocessed batches on master failure.
|
2013-01-23 03:15:36 -08:00 |
|
Tathagata Das
|
fad2b82fc8
|
Added support for saving input files of FileInputDStream to graph checkpoints. Modified 'file input stream with checkpoint' testcase to test recovery of pre-master-failure input files.
|
2013-01-22 18:10:00 -08:00 |
|
Tathagata Das
|
364cdb679c
|
Refactored DStreamCheckpointData.
|
2013-01-22 00:43:31 -08:00 |
|
Josh Rosen
|
551a47a620
|
Refactor daemon thread pool creation.
|
2013-01-21 23:31:00 -08:00 |
|
seanm
|
c0694291c8
|
Splitting StreamingContext.queueStream into two methods
|
2013-01-20 12:09:45 -07:00 |
|
Tathagata Das
|
4f8fe58b25
|
Merge branch 'mesos-streaming' into streaming
Conflicts:
core/src/main/scala/spark/api/java/JavaRDDLike.scala
core/src/main/scala/spark/api/java/JavaSparkContext.scala
core/src/test/scala/spark/JavaAPISuite.java
|
2013-01-20 01:13:56 -08:00 |
|
Tathagata Das
|
214345ceac
|
Fixed issue https://spark-project.atlassian.net/browse/STREAMING-29, along with updates to doc comments in SparkContext.checkpoint().
|
2013-01-19 23:50:17 -08:00 |
|
seanm
|
d3064fe707
|
kafkaStream API cleanup. A quorum of zookeepers can now be specified
|
2013-01-18 21:34:29 -07:00 |
|
seanm
|
56b7fbafa2
|
further KafkaInputDStream cleanup (removing unused and commented out code relating to offset management)
|
2013-01-18 21:15:54 -07:00 |
|
Patrick Wendell
|
e0165bf714
|
Adding queueStream and some slight refactoring
|
2013-01-17 21:25:49 -08:00 |
|
Patrick Wendell
|
70ba994d6d
|
Import fixup
|
2013-01-17 18:41:59 -08:00 |
|
Patrick Wendell
|
2261e62ee5
|
Style cleanup
|
2013-01-17 18:41:59 -08:00 |
|
Patrick Wendell
|
82b8707c6b
|
Checkpointing in Streaming java API
|
2013-01-17 18:41:58 -08:00 |
|
Patrick Wendell
|
61b877c688
|
Adding flatMap
|
2013-01-17 18:41:58 -08:00 |
|
Patrick Wendell
|
8e6cbbc6c7
|
Adding other updateState functions
|
2013-01-17 18:41:58 -08:00 |
|
Patrick Wendell
|
2a872335c5
|
Bug fix and test cleanup
|
2013-01-17 18:41:58 -08:00 |
|
seanm
|
a4639400ea
|
Merge branch 'streaming' into sm-updateStateByKey
|
2013-01-15 20:29:01 -07:00 |
|
Tathagata Das
|
1638fcb0dc
|
Fixed updateStateByKey to work with primitive types.
|
2013-01-14 17:18:39 -08:00 |
|
seanm
|
c203a29296
|
StateDStream changes to give updateStateByKey consistent behavior
|
2013-01-14 17:22:03 -07:00 |
|
seanm
|
b61a4ec773
|
Removing offset management code that is non-existent in kafka 0.7.0+
|
2013-01-14 17:13:10 -07:00 |
|
Patrick Wendell
|
a0013beb03
|
Stash
|
2013-01-14 15:15:02 -08:00 |
|
Patrick Wendell
|
8ad6220bd3
|
Bugfix
|
2013-01-14 14:56:36 -08:00 |
|
Patrick Wendell
|
38d9a3a863
|
Remove AnyRef constraint in updateState
|
2013-01-14 13:31:49 -08:00 |
|
Patrick Wendell
|
ae5290f4a2
|
Bug fix
|
2013-01-14 13:31:32 -08:00 |
|
Patrick Wendell
|
6069446356
|
Making comments consistent w/ Spark style
|
2013-01-14 10:42:05 -08:00 |
|
Patrick Wendell
|
d182a57cae
|
Two changes:
- Updating countByX() types based on bug fix
- Porting new documentation to Java
|
2013-01-14 10:03:55 -08:00 |
|
Patrick Wendell
|
a292ed8d8a
|
Some style cleanup
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
3461cd99b7
|
Flume example and bug fix
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
5bcb048167
|
More work on InputStreams
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
280b6d0186
|
Porting to new Duration class
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
c2537057f9
|
Fixing issue with <Long> types
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
b36c4f7cce
|
More work on StreamingContext
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
5004eec37c
|
Import Cleanup
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
560c312c60
|
Docs, some tests, and work on
StreamingContext
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
7e1049d8f1
|
Squashing a few TODOs
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
74182010a4
|
Style cleanup and moving functions
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
056f5efc55
|
More pair functions
|
2013-01-14 09:42:36 -08:00 |
|
Patrick Wendell
|
6e514a8d35
|
PairDStream and DStreamLike
|
2013-01-14 09:42:36 -08:00 |
|