ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2153 commits 6 branches 181 tags 397 MiB
Commit graph

217 commits

Author SHA1 Message Date
Tathagata Das 99a5fc498a Added an initial spark job to ensure worker nodes are initialized. 2013-02-09 15:18:05 -08:00
Tathagata Das 4cc223b478 Merge branch 'mesos-master' into streaming 2013-02-07 13:59:31 -08:00
Tathagata Das d55e3aa467 Updated JavaStreamingContext with updated kafkaStream API. 2013-02-07 13:59:18 -08:00
Tathagata Das c6b2f765d3 Merge branch 'mesos-streaming' into streaming 2013-02-07 13:13:53 -08:00
Tathagata Das 12300758cc Merge pull request #372 from Reinvigorate/sm-kafka 
Removing offset management code that is non-existent in kafka 0.7.0+
 2013-02-07 12:41:07 -08:00
Tathagata Das 915d9931fe Merge pull request #373 from Reinvigorate/sm-updateStateByKey 
StateDStream changes to give updateStateByKey consistent behavior
 2013-02-07 11:59:19 -08:00
Patrick Wendell 7eea64aa4c Streaming constructor which takes JavaSparkContext 
It's sometimes helpful to directly pass a JavaSparkContext,
and take advantage of the various constructors available for that.
 2013-02-05 11:43:16 -08:00
Mikhail Bautin fe3eceab57 Remove activation of profiles by default 
See the discussion at https://github.com/mesos/spark/pull/355 for why
default profile activation is a problem.
 2013-01-31 13:30:41 -08:00
Matei Zaharia 9ae11603b4 Merge pull request #415 from stephenh/driver 
Replace old 'master' term with 'driver'.
 2013-01-29 10:41:42 -08:00
Matei Zaharia b29599e5cf Fix code that depended on metadata cleaner interval being in minutes 2013-01-28 22:24:47 -08:00
Stephen Haberman 7dfb82a992 Replace old 'master' term with 'driver'. 2013-01-25 11:03:00 -06:00
Tathagata Das 9c8ff1e55f Fixed checkpoint testcases 2013-01-23 07:31:49 -08:00
Tathagata Das 666ce431aa Added support for rescheduling unprocessed batches on master failure. 2013-01-23 03:15:36 -08:00
Tathagata Das fad2b82fc8 Added support for saving input files of FileInputDStream to graph checkpoints. Modified 'file input stream with checkpoint' testcase to test recovery of pre-master-failure input files. 2013-01-22 18:10:00 -08:00
Tathagata Das 364cdb679c Refactored DStreamCheckpointData. 2013-01-22 00:43:31 -08:00
Josh Rosen 551a47a620 Refactor daemon thread pool creation. 2013-01-21 23:31:00 -08:00
Stephen Haberman e5ca241335 Move JavaAPISuite into spark.streaming. 2013-01-21 16:06:58 -06:00
Matei Zaharia 6e3754bf47 Add Maven build file for streaming, and fix some issues in SBT file 
As part of this, changed our Scala 2.9.2 Kafka library to be available
as a local Maven repository, following the example in
(http://blog.dub.podval.org/2010/01/maven-in-project-repository.html)
 2013-01-20 19:22:24 -08:00
seanm c0694291c8 Splitting StreamingContext.queueStream into two methods 2013-01-20 12:09:45 -07:00
seanm ea739251eb adding updateStateByKey object lifecycle test 2013-01-20 11:29:21 -07:00
Tathagata Das 33bad85bb9 Fixed streaming testsuite bugs 2013-01-20 03:51:11 -08:00
Tathagata Das 4f8fe58b25 Merge branch 'mesos-streaming' into streaming 
Conflicts:
	core/src/main/scala/spark/api/java/JavaRDDLike.scala
	core/src/main/scala/spark/api/java/JavaSparkContext.scala
	core/src/test/scala/spark/JavaAPISuite.java
 2013-01-20 01:13:56 -08:00
Tathagata Das 214345ceac Fixed issue https://spark-project.atlassian.net/browse/STREAMING-29, along with updates to doc comments in SparkContext.checkpoint(). 2013-01-19 23:50:17 -08:00
seanm d3064fe707 kafkaStream API cleanup. A quorum of zookeepers can now be specified 2013-01-18 21:34:29 -07:00
seanm 56b7fbafa2 further KafkaInputDStream cleanup (removing unused and commented out code relating to offset management) 2013-01-18 21:15:54 -07:00
Patrick Wendell c46dd2de78 Moving tests to appropriate directory 2013-01-17 21:43:17 -08:00
Patrick Wendell e0165bf714 Adding queueStream and some slight refactoring 2013-01-17 21:25:49 -08:00
Patrick Wendell ee0314c3b3 Merge branch 'streaming' into streaming-java-api 2013-01-17 18:43:00 -08:00
Patrick Wendell 70ba994d6d Import fixup 2013-01-17 18:41:59 -08:00
Patrick Wendell 2261e62ee5 Style cleanup 2013-01-17 18:41:59 -08:00
Patrick Wendell 82b8707c6b Checkpointing in Streaming java API 2013-01-17 18:41:58 -08:00
Patrick Wendell 61b877c688 Adding flatMap 2013-01-17 18:41:58 -08:00
Patrick Wendell 8e6cbbc6c7 Adding other updateState functions 2013-01-17 18:41:58 -08:00
Patrick Wendell 2a872335c5 Bug fix and test cleanup 2013-01-17 18:41:58 -08:00
seanm a4639400ea Merge branch 'streaming' into sm-updateStateByKey 2013-01-15 20:29:01 -07:00
Tathagata Das 1638fcb0dc Fixed updateStateByKey to work with primitive types. 2013-01-14 17:18:39 -08:00
seanm c203a29296 StateDStream changes to give updateStateByKey consistent behavior 2013-01-14 17:22:03 -07:00
seanm b61a4ec773 Removing offset management code that is non-existent in kafka 0.7.0+ 2013-01-14 17:13:10 -07:00
Patrick Wendell a0013beb03 Stash 2013-01-14 15:15:02 -08:00
Patrick Wendell 8ad6220bd3 Bugfix 2013-01-14 14:56:36 -08:00
Patrick Wendell 38d9a3a863 Remove AnyRef constraint in updateState 2013-01-14 13:31:49 -08:00
Patrick Wendell ae5290f4a2 Bug fix 2013-01-14 13:31:32 -08:00
Patrick Wendell 6069446356 Making comments consistent w/ Spark style 2013-01-14 10:42:05 -08:00
Patrick Wendell d182a57cae Two changes: 
- Updating countByX() types based on bug fix
- Porting new documentation to Java
 2013-01-14 10:03:55 -08:00
Patrick Wendell a292ed8d8a Some style cleanup 2013-01-14 09:42:36 -08:00
Patrick Wendell 3461cd99b7 Flume example and bug fix 2013-01-14 09:42:36 -08:00
Patrick Wendell 5bcb048167 More work on InputStreams 2013-01-14 09:42:36 -08:00
Patrick Wendell 280b6d0186 Porting to new Duration class 2013-01-14 09:42:36 -08:00
Patrick Wendell c2537057f9 Fixing issue with <Long> types 2013-01-14 09:42:36 -08:00
Patrick Wendell b36c4f7cce More work on StreamingContext 2013-01-14 09:42:36 -08:00