Imran Rashid
|
8f17387d97
|
remove bogus comment
|
2013-02-25 10:31:06 -08:00 |
|
Matei Zaharia
|
6ae9a22c3e
|
Get spark.default.paralellism on each call to defaultPartitioner,
instead of only once, in case the user changes it across Spark uses
|
2013-02-25 10:28:08 -08:00 |
|
Matei Zaharia
|
d6e6abece3
|
Merge pull request #459 from stephenh/bettersplits
Change defaultPartitioner to use upstream split size.
|
2013-02-25 09:22:04 -08:00 |
|
Matei Zaharia
|
fb76250598
|
Merge pull request #497 from ScrapCodes/dep-resolution-fix
Moving akka dependency resolver to shared.
|
2013-02-25 00:16:52 -08:00 |
|
Prashant Sharma
|
254acb1666
|
Moving akka dependency resolver to shared.
|
2013-02-25 13:37:07 +05:30 |
|
Stephen Haberman
|
c44ccf2862
|
Use default parallelism if its set.
|
2013-02-24 23:54:03 -06:00 |
|
Stephen Haberman
|
44032bc476
|
Merge branch 'master' into bettersplits
Conflicts:
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
core/src/test/scala/spark/ShuffleSuite.scala
|
2013-02-24 22:08:14 -06:00 |
|
Matei Zaharia
|
22f68b7e6b
|
Merge pull request #494 from mesos/streaming
Final set of Spark Streaming changes
|
2013-02-24 17:48:17 -08:00 |
|
Tathagata Das
|
f9881b10c6
|
Merge pull request #495 from tdas/streaming
Changed class paths and dependencies based on Matei's comments
|
2013-02-24 16:39:04 -08:00 |
|
Tathagata Das
|
abb5471865
|
Removing duplicate doc.
|
2013-02-24 16:32:44 -08:00 |
|
Tathagata Das
|
5ab37be983
|
Fixed class paths and dependencies based on Matei's comments.
|
2013-02-24 16:24:52 -08:00 |
|
Tathagata Das
|
32f1a2d540
|
Merge pull request #493 from tdas/streaming
Streaming bug fixes
|
2013-02-24 14:46:19 -08:00 |
|
Matei Zaharia
|
066ce0ffa4
|
Merge pull request #492 from JoshRosen/pyspark-split-rename
Change numSplits to numPartitions in PySpark
|
2013-02-24 14:11:07 -08:00 |
|
Josh Rosen
|
2c966c98fb
|
Change numSplits to numPartitions in PySpark.
|
2013-02-24 13:25:09 -08:00 |
|
Tathagata Das
|
28f8b721f6
|
Added back the initial spark job before starting streaming receivers
|
2013-02-24 13:01:54 -08:00 |
|
Tathagata Das
|
f282bc4960
|
Changed Algebird from 0.1.9 to 0.1.8
|
2013-02-24 12:44:12 -08:00 |
|
Tathagata Das
|
dff53d1b94
|
Merge branch 'mesos-master' into streaming
|
2013-02-24 12:17:22 -08:00 |
|
Tathagata Das
|
68c7934b1a
|
Fixed missing dependencies in streaming/pom.xml
|
2013-02-24 11:51:45 -08:00 |
|
Tathagata Das
|
c1a040db3a
|
Fixed bugs in examples.
|
2013-02-24 11:00:30 -08:00 |
|
Tathagata Das
|
b4eb24de96
|
Updated streaming programming guide with Java API info, and comments from Patrick.
|
2013-02-23 23:59:45 -08:00 |
|
Matei Zaharia
|
3b9f929467
|
Merge pull request #468 from haitaoyao/master
support customized java options for Master, Worker, Executor, and Repl
|
2013-02-23 23:38:15 -08:00 |
|
Matei Zaharia
|
beb79dbbfd
|
Merge pull request #473 from stephenh/subtract
Add RDD.subtract.
|
2013-02-23 23:25:27 -08:00 |
|
Stephen Haberman
|
37c7a71f9c
|
Add subtract to JavaRDD, JavaDoubleRDD, and JavaPairRDD.
|
2013-02-24 00:27:53 -06:00 |
|
Stephen Haberman
|
f442e7d83c
|
Update for split->partition rename.
|
2013-02-24 00:27:14 -06:00 |
|
Stephen Haberman
|
cec87a0653
|
Merge branch 'master' into subtract
|
2013-02-23 23:27:55 -06:00 |
|
Tathagata Das
|
d853aa9658
|
Change spark.cleaner.delay to spark.cleaner.ttl. Updated docs.
|
2013-02-23 17:42:26 -08:00 |
|
Tathagata Das
|
41285eaae3
|
Fixed differences in APIs of StreamingContext and JavaStreamingContext. Change rawNetworkStream to rawSocketStream, and added twitter, actor, zeroMQ streams to JavaStreamingContext. Also added them to JavaAPISuite.
|
2013-02-23 16:25:07 -08:00 |
|
Matei Zaharia
|
8a0ca6f05c
|
Merge pull request #491 from pwendell/java-vector
Adding Java-accessible methods to Vector.scala
|
2013-02-23 15:55:06 -08:00 |
|
Patrick Wendell
|
931f439be9
|
Responding to code review
|
2013-02-23 15:40:41 -08:00 |
|
Patrick Wendell
|
f51b0f93f2
|
Adding Java-accessible methods to Vector.scala
This is needed for the Strata machine learning tutorial (and
also is generally helpful).
|
2013-02-23 13:26:59 -08:00 |
|
Matei Zaharia
|
d942d39072
|
Handle exceptions in RecordReader.close() better (suggested by Jim
Donahue)
|
2013-02-23 11:19:07 -08:00 |
|
Matei Zaharia
|
c89824046a
|
Merge pull request #490 from woggling/conn-death
Detect when SendingConnections disconnect even if we aren't sending to them
|
2013-02-22 22:58:19 -08:00 |
|
Tathagata Das
|
d8cee52d52
|
Merge branch 'mesos-streaming' into streaming
|
2013-02-22 18:25:34 -08:00 |
|
Tathagata Das
|
24c0cd6168
|
Fixed resolver for akka-zeromq
|
2013-02-22 18:23:29 -08:00 |
|
Charles Reiss
|
50cf8c8b79
|
Add fault tolerance test that uses replicated RDDs.
|
2013-02-22 16:11:53 -08:00 |
|
Charles Reiss
|
c8a7886921
|
Detect when SendingConnections drop by trying to read them.
Comment fix
|
2013-02-22 16:11:52 -08:00 |
|
Matei Zaharia
|
d4d7993bf5
|
Several fixes to the work to log when no resources can be used by a job.
Fixed some of the messages as well as code style.
|
2013-02-22 15:51:37 -08:00 |
|
Matei Zaharia
|
f33662c133
|
Merge remote-tracking branch 'pwendell/starvation-check'
Also fixed a bug where master was offering executors on dead workers
Conflicts:
core/src/main/scala/spark/deploy/master/Master.scala
|
2013-02-22 15:27:41 -08:00 |
|
Matei Zaharia
|
7341de0d48
|
Merge pull request #475 from JoshRosen/spark-668
Remove hack workaround for SPARK-668
|
2013-02-22 14:56:18 -08:00 |
|
Matei Zaharia
|
fcf7fa8448
|
Merge pull request #489 from pwendell/SPARK-702
SPARK-702: Replace Function --> JFunction in JavaAPI Suite.
|
2013-02-22 14:55:46 -08:00 |
|
Patrick Wendell
|
f8c3a03d55
|
SPARK-702: Replace Function --> JFunction in JavaAPI Suite.
In a few places the Scala (rather than Java) function class is used.
|
2013-02-22 12:54:15 -08:00 |
|
Tathagata Das
|
cfa65ebff1
|
Merge pull request #480 from MLnick/streaming-eg-algebird
[Streaming] Examples using Twitter's Algebird library
|
2013-02-22 12:29:04 -08:00 |
|
Tathagata Das
|
688e62718f
|
Merge pull request #479 from ScrapCodes/zeromq-streaming
Zeromq streaming
|
2013-02-22 12:17:17 -08:00 |
|
Imran Rashid
|
0f37b43b40
|
make the ShuffleFetcher responsible for collecting shuffle metrics, which gives us metrics for CoGroupedRDD and ShuffledRDD
|
2013-02-21 16:56:28 -08:00 |
|
Imran Rashid
|
9230617f23
|
add cleanup iterator
|
2013-02-21 16:55:14 -08:00 |
|
Tathagata Das
|
208edaac1b
|
Fixed condition in InputDStream isTimeValid.
|
2013-02-21 15:22:26 -08:00 |
|
Imran Rashid
|
81bd07da26
|
sparkListeners should be a val
|
2013-02-21 15:21:45 -08:00 |
|
Imran Rashid
|
796e934d31
|
add some docs & some cleanup
|
2013-02-21 15:19:34 -08:00 |
|
Imran Rashid
|
394d3acc3e
|
store taskInfo & metrics together in a tuple
|
2013-02-21 15:19:34 -08:00 |
|
Imran Rashid
|
7960927cf4
|
get rid of a bunch of boilerplate; more formatting happens in Listener, not StageInfo
|
2013-02-21 15:19:34 -08:00 |
|