ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2591 commits 6 branches 181 tags 397 MiB
Commit graph

2591 commits

This branch
This branch
All branches
Author SHA1 Message Date
Patrick Wendell f8c3a03d55 SPARK-702: Replace Function --> JFunction in JavaAPI Suite. 
In a few places the Scala (rather than Java) function class is used.
 2013-02-22 12:54:15 -08:00
Tathagata Das cfa65ebff1 Merge pull request #480 from MLnick/streaming-eg-algebird 
[Streaming] Examples using Twitter's Algebird library
 2013-02-22 12:29:04 -08:00
Tathagata Das 688e62718f Merge pull request #479 from ScrapCodes/zeromq-streaming 
Zeromq streaming
 2013-02-22 12:17:17 -08:00
Imran Rashid 0f37b43b40 make the ShuffleFetcher responsible for collecting shuffle metrics, which gives us metrics for CoGroupedRDD and ShuffledRDD 2013-02-21 16:56:28 -08:00
Imran Rashid 9230617f23 add cleanup iterator 2013-02-21 16:55:14 -08:00
Tathagata Das 208edaac1b Fixed condition in InputDStream isTimeValid. 2013-02-21 15:22:26 -08:00
Imran Rashid 81bd07da26 sparkListeners should be a val 2013-02-21 15:21:45 -08:00
Imran Rashid 796e934d31 add some docs & some cleanup 2013-02-21 15:19:34 -08:00
Imran Rashid 394d3acc3e store taskInfo & metrics together in a tuple 2013-02-21 15:19:34 -08:00
Imran Rashid 7960927cf4 get rid of a bunch of boilerplate; more formatting happens in Listener, not StageInfo 2013-02-21 15:19:34 -08:00
Imran Rashid d0bfac3eed taskInfo tracks if a task is run on a preferred host 2013-02-21 15:19:34 -08:00
Imran Rashid 6f62a57858 add runtime breakdowns 2013-02-21 15:19:34 -08:00
Imran Rashid 176cb20703 add task result size; better formatting for time interval distributions; cleanup distribution formatting 2013-02-21 15:19:33 -08:00
Imran Rashid f2fcabf2ea add timing around parts of executor & track result size 2013-02-21 15:19:33 -08:00
Imran Rashid ff127cfcd3 Merge branch 'master' into stageInfo 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/storage/BlockManager.scala
 2013-02-21 15:16:21 -08:00
Imran Rashid 69f9a7035f fully revert change to addOnCompleteCallback -- missed this in e9f53ec 2013-02-21 15:07:46 -08:00
Imran Rashid baab23abdf TaskContext does not hold a reference to Task; instead, it has a shared instance of TaskMetrics with Task 2013-02-21 14:13:01 -08:00
Nick Pentreath d9bdae8cc2 Adding documentation for HLL and CMS examples. More efficient and clear use of the monoids. 2013-02-21 12:31:31 +02:00
Nick Pentreath 718474b9c6 Bumping Algebird to 0.1.9 2013-02-21 12:11:31 +02:00
Nick Pentreath 16d456742e Merge remote-tracking branch 'upstream/streaming' into streaming-eg-algebird 2013-02-21 09:33:08 +02:00
haitao.yao 8215b95547 Merge branch 'mesos' 2013-02-21 10:07:24 +08:00
Tathagata Das 2921fa7d81 Merge pull request #487 from tdas/streaming 
Merged latest master changes to streaming
 2013-02-20 11:06:57 -08:00
Tathagata Das 972fe7714f Merge branch 'mesos-streaming' into streaming 
Conflicts:
	streaming/src/test/java/spark/streaming/JavaAPISuite.java
 2013-02-20 11:06:01 -08:00
Tathagata Das 334ab92441 Fixed bug in CheckpointSuite 2013-02-20 10:26:36 -08:00
Tathagata Das 1cb725e417 Merge branch 'mesos-master' into streaming 2013-02-20 09:55:35 -08:00
Tathagata Das fb9956256d Merge branch 'mesos-master' into streaming 
Conflicts:
	core/src/main/scala/spark/rdd/CheckpointRDD.scala
	streaming/src/main/scala/spark/streaming/dstream/ReducedWindowedDStream.scala
 2013-02-20 09:01:29 -08:00
Prashant Sharma ee885611ff removed document on zeromq 2013-02-20 19:18:24 +05:30
Prashant Sharma 4e5b09664c fixes corresponding to review feedback at pull request #479 2013-02-20 19:14:52 +05:30
Tathagata Das 991d3342fe Merge pull request #486 from ScrapCodes/akka-example-bug-fix 
A bug fix to AkkaWordCount example.
 2013-02-20 02:06:36 -08:00
Prashant Sharma 05dc385649 A bug fix post merge, following changes to AkkaUtils 2013-02-20 15:28:12 +05:30
Matei Zaharia 05bc02e80b Merge pull request #482 from woggling/shutdown-exceptions 
Don't call System.exit over uncaught exceptions from shutdown hooks
 2013-02-19 20:56:15 -08:00
haitao.yao 6a3d44c673 Merge branch 'mesos' 2013-02-20 10:23:58 +08:00
Charles Reiss 092c631fa8 Pull detection of being in a shutdown hook into utility function. 2013-02-19 17:49:55 -08:00
Matei Zaharia 8a992226bd Merge pull request #484 from andyk/master 
Fixes a broken link in documentation to issue tracker
 2013-02-19 17:07:24 -08:00
Matei Zaharia a3e86b2b1f Merge pull request #483 from rxin/splitpruningrdd2 
Added a method to create PartitionPruningRDD.
 2013-02-19 17:07:00 -08:00
Andy Konwinski ecd137a72d Fixes link to issue tracker in documentation page "Contributing to Spark". 2013-02-19 16:58:02 -08:00
Reynold Xin 130f704baf Added a method to create PartitionPruningRDD. 2013-02-19 16:03:52 -08:00
Charles Reiss d0588bd6d7 Catch/log errors deleting temp dirs 2013-02-19 13:04:06 -08:00
Charles Reiss 687581c3ec Paranoid uncaught exception handling for exceptions during shutdown 2013-02-19 13:03:02 -08:00
Tathagata Das b0565ae396 Merge pull request #481 from pwendell/stream-rdd-type-streaming 
STREAMING-51: Add RDD type as a type parameter in JavaDStreamLike Edit (streaming/ version)
 2013-02-19 10:25:36 -08:00
Patrick Wendell 041c19e5f0 Small changes that were missing in merge 2013-02-19 08:44:20 -08:00
Patrick Wendell fed1122d74 Use RDD type for slice operator in Java. 
This commit uses the RDD type in `slice`, making it available to both normal
and pair RDD's in java. It also updates the signature for `slice` to match
changes in the Scala API.
 2013-02-19 08:32:38 -08:00
Patrick Wendell 35880de42e Use RDD type for transform operator in Java. 
This is an improved implementation of the `transform` operator in Java.
The main difference is that this allows all four possible types of
transform functions

1. JavaRDD -> JavaRDD
2. JavaRDD -> JavaPairRDD
3. JavaPairRDD -> JavaPairRDD
4. JavaPairRDD -> JavaRDD

whereas previously only (1) and (3) were possible.

Conflicts:

	streaming/src/test/java/spark/streaming/JavaAPISuite.java
 2013-02-19 08:31:58 -08:00
Patrick Wendell 9d49a6b03f Use RDD type for foreach operator in Java. 2013-02-19 08:30:32 -08:00
Nick Pentreath 8a281399f9 Streaming example using Twitter Algebird's Count Min Sketch monoid 2013-02-19 17:56:02 +02:00
Nick Pentreath d8ee184d95 Dependencies and refactoring for streaming HLL example, and using context.twitterStream method 2013-02-19 17:42:57 +02:00
Prashant Sharma 8d44480d84 example for demonstrating ZeroMQ stream 2013-02-19 19:42:14 +05:30
Prashant Sharma f7d3e309cb ZeroMQ stream as receiver 2013-02-19 19:32:52 +05:30
Nick Pentreath 315ea069e8 Merge remote-tracking branch 'upstream/streaming' into streaming-eg-algebird 
Conflicts:
	project/SparkBuild.scala
 2013-02-19 13:58:05 +02:00
Nick Pentreath 015893f0e8 Adding streaming HyperLogLog example using Algebird 2013-02-19 13:21:33 +02:00