Stephen Haberman
cec87a0653
Merge branch 'master' into subtract
2013-02-23 23:27:55 -06:00
Tathagata Das
d853aa9658
Change spark.cleaner.delay to spark.cleaner.ttl. Updated docs.
2013-02-23 17:42:26 -08:00
Tathagata Das
41285eaae3
Fixed differences in APIs of StreamingContext and JavaStreamingContext. Change rawNetworkStream to rawSocketStream, and added twitter, actor, zeroMQ streams to JavaStreamingContext. Also added them to JavaAPISuite.
2013-02-23 16:25:07 -08:00
Matei Zaharia
8a0ca6f05c
Merge pull request #491 from pwendell/java-vector
...
Adding Java-accessible methods to Vector.scala
2013-02-23 15:55:06 -08:00
Patrick Wendell
931f439be9
Responding to code review
2013-02-23 15:40:41 -08:00
Patrick Wendell
f51b0f93f2
Adding Java-accessible methods to Vector.scala
...
This is needed for the Strata machine learning tutorial (and
also is generally helpful).
2013-02-23 13:26:59 -08:00
Matei Zaharia
d942d39072
Handle exceptions in RecordReader.close() better (suggested by Jim
...
Donahue)
2013-02-23 11:19:07 -08:00
Matei Zaharia
c89824046a
Merge pull request #490 from woggling/conn-death
...
Detect when SendingConnections disconnect even if we aren't sending to them
2013-02-22 22:58:19 -08:00
Tathagata Das
d8cee52d52
Merge branch 'mesos-streaming' into streaming
2013-02-22 18:25:34 -08:00
Tathagata Das
24c0cd6168
Fixed resolver for akka-zeromq
2013-02-22 18:23:29 -08:00
Charles Reiss
50cf8c8b79
Add fault tolerance test that uses replicated RDDs.
2013-02-22 16:11:53 -08:00
Charles Reiss
c8a7886921
Detect when SendingConnections drop by trying to read them.
...
Comment fix
2013-02-22 16:11:52 -08:00
Matei Zaharia
d4d7993bf5
Several fixes to the work to log when no resources can be used by a job.
...
Fixed some of the messages as well as code style.
2013-02-22 15:51:37 -08:00
Matei Zaharia
f33662c133
Merge remote-tracking branch 'pwendell/starvation-check'
...
Also fixed a bug where master was offering executors on dead workers
Conflicts:
core/src/main/scala/spark/deploy/master/Master.scala
2013-02-22 15:27:41 -08:00
Matei Zaharia
7341de0d48
Merge pull request #475 from JoshRosen/spark-668
...
Remove hack workaround for SPARK-668
2013-02-22 14:56:18 -08:00
Matei Zaharia
fcf7fa8448
Merge pull request #489 from pwendell/SPARK-702
...
SPARK-702: Replace Function --> JFunction in JavaAPI Suite.
2013-02-22 14:55:46 -08:00
Patrick Wendell
f8c3a03d55
SPARK-702: Replace Function --> JFunction in JavaAPI Suite.
...
In a few places the Scala (rather than Java) function class is used.
2013-02-22 12:54:15 -08:00
Tathagata Das
cfa65ebff1
Merge pull request #480 from MLnick/streaming-eg-algebird
...
[Streaming] Examples using Twitter's Algebird library
2013-02-22 12:29:04 -08:00
Tathagata Das
688e62718f
Merge pull request #479 from ScrapCodes/zeromq-streaming
...
Zeromq streaming
2013-02-22 12:17:17 -08:00
Tathagata Das
208edaac1b
Fixed condition in InputDStream isTimeValid.
2013-02-21 15:22:26 -08:00
Nick Pentreath
d9bdae8cc2
Adding documentation for HLL and CMS examples. More efficient and clear use of the monoids.
2013-02-21 12:31:31 +02:00
Nick Pentreath
718474b9c6
Bumping Algebird to 0.1.9
2013-02-21 12:11:31 +02:00
Nick Pentreath
16d456742e
Merge remote-tracking branch 'upstream/streaming' into streaming-eg-algebird
2013-02-21 09:33:08 +02:00
haitao.yao
8215b95547
Merge branch 'mesos'
2013-02-21 10:07:24 +08:00
Tathagata Das
2921fa7d81
Merge pull request #487 from tdas/streaming
...
Merged latest master changes to streaming
2013-02-20 11:06:57 -08:00
Tathagata Das
972fe7714f
Merge branch 'mesos-streaming' into streaming
...
Conflicts:
streaming/src/test/java/spark/streaming/JavaAPISuite.java
2013-02-20 11:06:01 -08:00
Tathagata Das
334ab92441
Fixed bug in CheckpointSuite
2013-02-20 10:26:36 -08:00
Tathagata Das
1cb725e417
Merge branch 'mesos-master' into streaming
2013-02-20 09:55:35 -08:00
Tathagata Das
fb9956256d
Merge branch 'mesos-master' into streaming
...
Conflicts:
core/src/main/scala/spark/rdd/CheckpointRDD.scala
streaming/src/main/scala/spark/streaming/dstream/ReducedWindowedDStream.scala
2013-02-20 09:01:29 -08:00
Prashant Sharma
ee885611ff
removed document on zeromq
2013-02-20 19:18:24 +05:30
Prashant Sharma
4e5b09664c
fixes corresponding to review feedback at pull request #479
2013-02-20 19:14:52 +05:30
Tathagata Das
991d3342fe
Merge pull request #486 from ScrapCodes/akka-example-bug-fix
...
A bug fix to AkkaWordCount example.
2013-02-20 02:06:36 -08:00
Prashant Sharma
05dc385649
A bug fix post merge, following changes to AkkaUtils
2013-02-20 15:28:12 +05:30
Matei Zaharia
05bc02e80b
Merge pull request #482 from woggling/shutdown-exceptions
...
Don't call System.exit over uncaught exceptions from shutdown hooks
2013-02-19 20:56:15 -08:00
haitao.yao
6a3d44c673
Merge branch 'mesos'
2013-02-20 10:23:58 +08:00
Charles Reiss
092c631fa8
Pull detection of being in a shutdown hook into utility function.
2013-02-19 17:49:55 -08:00
Matei Zaharia
8a992226bd
Merge pull request #484 from andyk/master
...
Fixes a broken link in documentation to issue tracker
2013-02-19 17:07:24 -08:00
Matei Zaharia
a3e86b2b1f
Merge pull request #483 from rxin/splitpruningrdd2
...
Added a method to create PartitionPruningRDD.
2013-02-19 17:07:00 -08:00
Andy Konwinski
ecd137a72d
Fixes link to issue tracker in documentation page "Contributing to Spark".
2013-02-19 16:58:02 -08:00
Reynold Xin
130f704baf
Added a method to create PartitionPruningRDD.
2013-02-19 16:03:52 -08:00
Charles Reiss
d0588bd6d7
Catch/log errors deleting temp dirs
2013-02-19 13:04:06 -08:00
Charles Reiss
687581c3ec
Paranoid uncaught exception handling for exceptions during shutdown
2013-02-19 13:03:02 -08:00
Tathagata Das
b0565ae396
Merge pull request #481 from pwendell/stream-rdd-type-streaming
...
STREAMING-51: Add RDD type as a type parameter in JavaDStreamLike Edit (streaming/ version)
2013-02-19 10:25:36 -08:00
Patrick Wendell
041c19e5f0
Small changes that were missing in merge
2013-02-19 08:44:20 -08:00
Patrick Wendell
fed1122d74
Use RDD type for slice
operator in Java.
...
This commit uses the RDD type in `slice`, making it available to both normal
and pair RDD's in java. It also updates the signature for `slice` to match
changes in the Scala API.
2013-02-19 08:32:38 -08:00
Patrick Wendell
35880de42e
Use RDD type for transform
operator in Java.
...
This is an improved implementation of the `transform` operator in Java.
The main difference is that this allows all four possible types of
transform functions
1. JavaRDD -> JavaRDD
2. JavaRDD -> JavaPairRDD
3. JavaPairRDD -> JavaPairRDD
4. JavaPairRDD -> JavaRDD
whereas previously only (1) and (3) were possible.
Conflicts:
streaming/src/test/java/spark/streaming/JavaAPISuite.java
2013-02-19 08:31:58 -08:00
Patrick Wendell
9d49a6b03f
Use RDD type for foreach
operator in Java.
2013-02-19 08:30:32 -08:00
Nick Pentreath
8a281399f9
Streaming example using Twitter Algebird's Count Min Sketch monoid
2013-02-19 17:56:02 +02:00
Nick Pentreath
d8ee184d95
Dependencies and refactoring for streaming HLL example, and using context.twitterStream method
2013-02-19 17:42:57 +02:00
Prashant Sharma
8d44480d84
example for demonstrating ZeroMQ stream
2013-02-19 19:42:14 +05:30