Matei Zaharia
7151e1e4c8
Rename "jobs" to "applications" in the standalone cluster
2013-02-17 23:23:08 -08:00
Stephen Haberman
ae2234687d
Make CoGroupedRDDs explicitly have the same key type.
2013-02-16 13:10:31 -06:00
Patrick Wendell
3f3e77f28b
STREAMING-50: Support transform workaround in JavaPairDStream
...
This ports a useful workaround (the `transform` function) to
JavaPairDStream. It is necessary to do things like sorting which
are not supported yet in the core streaming API.
2013-02-12 14:02:32 -08:00
Patrick Wendell
d09c36065c
Using tuple swap()
2013-02-11 10:45:45 -08:00
Patrick Wendell
04786d0739
small fix
2013-02-11 10:05:49 -08:00
Patrick Wendell
c65988bdc1
Fix for MapPartitions
2013-02-11 10:03:37 -08:00
Patrick Wendell
20cf770545
Fix for flatmap
2013-02-11 10:03:37 -08:00
Patrick Wendell
314d87a038
Indentation fix
2013-02-11 10:03:37 -08:00
Patrick Wendell
f0b68c623c
Initial cut at replacing K, V in Java files
2013-02-11 10:03:37 -08:00
Patrick Wendell
7eea64aa4c
Streaming constructor which takes JavaSparkContext
...
It's sometimes helpful to directly pass a JavaSparkContext,
and take advantage of the various constructors available for that.
2013-02-05 11:43:16 -08:00
Mikhail Bautin
fe3eceab57
Remove activation of profiles by default
...
See the discussion at https://github.com/mesos/spark/pull/355 for why
default profile activation is a problem.
2013-01-31 13:30:41 -08:00
Matei Zaharia
9ae11603b4
Merge pull request #415 from stephenh/driver
...
Replace old 'master' term with 'driver'.
2013-01-29 10:41:42 -08:00
Matei Zaharia
b29599e5cf
Fix code that depended on metadata cleaner interval being in minutes
2013-01-28 22:24:47 -08:00
Stephen Haberman
7dfb82a992
Replace old 'master' term with 'driver'.
2013-01-25 11:03:00 -06:00
Josh Rosen
551a47a620
Refactor daemon thread pool creation.
2013-01-21 23:31:00 -08:00
Stephen Haberman
e5ca241335
Move JavaAPISuite into spark.streaming.
2013-01-21 16:06:58 -06:00
Matei Zaharia
6e3754bf47
Add Maven build file for streaming, and fix some issues in SBT file
...
As part of this, changed our Scala 2.9.2 Kafka library to be available
as a local Maven repository, following the example in
(http://blog.dub.podval.org/2010/01/maven-in-project-repository.html )
2013-01-20 19:22:24 -08:00
Tathagata Das
33bad85bb9
Fixed streaming testsuite bugs
2013-01-20 03:51:11 -08:00
Tathagata Das
4f8fe58b25
Merge branch 'mesos-streaming' into streaming
...
Conflicts:
core/src/main/scala/spark/api/java/JavaRDDLike.scala
core/src/main/scala/spark/api/java/JavaSparkContext.scala
core/src/test/scala/spark/JavaAPISuite.java
2013-01-20 01:13:56 -08:00
Tathagata Das
214345ceac
Fixed issue https://spark-project.atlassian.net/browse/STREAMING-29 , along with updates to doc comments in SparkContext.checkpoint().
2013-01-19 23:50:17 -08:00
Patrick Wendell
c46dd2de78
Moving tests to appropriate directory
2013-01-17 21:43:17 -08:00
Patrick Wendell
e0165bf714
Adding queueStream and some slight refactoring
2013-01-17 21:25:49 -08:00
Patrick Wendell
ee0314c3b3
Merge branch 'streaming' into streaming-java-api
2013-01-17 18:43:00 -08:00
Patrick Wendell
70ba994d6d
Import fixup
2013-01-17 18:41:59 -08:00
Patrick Wendell
2261e62ee5
Style cleanup
2013-01-17 18:41:59 -08:00
Patrick Wendell
82b8707c6b
Checkpointing in Streaming java API
2013-01-17 18:41:58 -08:00
Patrick Wendell
61b877c688
Adding flatMap
2013-01-17 18:41:58 -08:00
Patrick Wendell
8e6cbbc6c7
Adding other updateState functions
2013-01-17 18:41:58 -08:00
Patrick Wendell
2a872335c5
Bug fix and test cleanup
2013-01-17 18:41:58 -08:00
Tathagata Das
1638fcb0dc
Fixed updateStateByKey to work with primitive types.
2013-01-14 17:18:39 -08:00
Patrick Wendell
a0013beb03
Stash
2013-01-14 15:15:02 -08:00
Patrick Wendell
8ad6220bd3
Bugfix
2013-01-14 14:56:36 -08:00
Patrick Wendell
38d9a3a863
Remove AnyRef constraint in updateState
2013-01-14 13:31:49 -08:00
Patrick Wendell
ae5290f4a2
Bug fix
2013-01-14 13:31:32 -08:00
Patrick Wendell
6069446356
Making comments consistent w/ Spark style
2013-01-14 10:42:05 -08:00
Patrick Wendell
d182a57cae
Two changes:
...
- Updating countByX() types based on bug fix
- Porting new documentation to Java
2013-01-14 10:03:55 -08:00
Patrick Wendell
a292ed8d8a
Some style cleanup
2013-01-14 09:42:36 -08:00
Patrick Wendell
3461cd99b7
Flume example and bug fix
2013-01-14 09:42:36 -08:00
Patrick Wendell
5bcb048167
More work on InputStreams
2013-01-14 09:42:36 -08:00
Patrick Wendell
280b6d0186
Porting to new Duration class
2013-01-14 09:42:36 -08:00
Patrick Wendell
c2537057f9
Fixing issue with <Long> types
2013-01-14 09:42:36 -08:00
Patrick Wendell
b36c4f7cce
More work on StreamingContext
2013-01-14 09:42:36 -08:00
Patrick Wendell
5004eec37c
Import Cleanup
2013-01-14 09:42:36 -08:00
Patrick Wendell
2fe39a4468
Some docs for the JavaTestUtils
2013-01-14 09:42:36 -08:00
Patrick Wendell
560c312c60
Docs, some tests, and work on
...
StreamingContext
2013-01-14 09:42:36 -08:00
Patrick Wendell
7e1049d8f1
Squashing a few TODOs
2013-01-14 09:42:36 -08:00
Patrick Wendell
74182010a4
Style cleanup and moving functions
2013-01-14 09:42:36 -08:00
Patrick Wendell
056f5efc55
More pair functions
2013-01-14 09:42:36 -08:00
Patrick Wendell
6e514a8d35
PairDStream and DStreamLike
2013-01-14 09:42:36 -08:00
Patrick Wendell
f144e0413a
Adding transform and union
2013-01-14 09:42:36 -08:00