ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3119 commits 6 branches 181 tags 397 MiB
Commit graph

3119 commits

This branch
This branch
All branches
Author SHA1 Message Date
Giovanni Delussu aceae029f7 CoalescedRDD changed to work with a big number of partitions both in the original and the new coalesced RDD. 
The limitation was in the range that Scala.Int can represent.
 2013-03-19 11:25:45 +01:00
Stephen Haberman fb34967815 Remove try/catch block that can't be hit. 2013-03-18 01:55:50 -05:00
Andy Konwinski ad7f0452ab Adds page to docs about building using Maven. 
Adds links to new instructions in:
* The main Spark project README.md
* The docs nav menu called "More"
* The docs Overview page under the "Building" and "Where to Go from Here" sections
 2013-03-17 15:02:40 -07:00
Nick Pentreath b990caeb80 Changes to more closely match line length limit style 2013-03-17 20:03:27 +02:00
Mark Hamstra ab33e27cc9 constructorOfA -> constructA in doc comments 2013-03-16 15:29:15 -07:00
Mark Hamstra 9784fc1fcd fix wayward comma in doc comment 2013-03-16 15:25:02 -07:00
Mark Hamstra 32979b5e7d whitespace 2013-03-16 13:36:46 -07:00
Mark Hamstra ca9f81e8fc refactor foldByKey to use combineByKey 2013-03-16 13:31:01 -07:00
Mark Hamstra 1fb192ef40 Merge branch 'master' of https://github.com/mesos/spark into foldByKey 2013-03-16 12:17:13 -07:00
Mark Hamstra 80fc8c82ed _With[Matei] 2013-03-16 12:16:29 -07:00
Mark Hamstra 38454c4aed Merge branch 'master' of https://github.com/mesos/spark into WithThing 2013-03-16 11:54:44 -07:00
Matei Zaharia c1e9cdc49f Merge pull request #525 from stephenh/subtractByKey 
Add PairRDDFunctions.subtractByKey.
 2013-03-16 11:47:45 -07:00
seanm d61978d0ab keeping JavaStreamingContext in sync with StreamingContext + adding comments for better clarity 2013-03-15 23:36:52 -06:00
Mark Hamstra ef75be3bf7 Merge branch 'master' of https://github.com/mesos/spark into foldByKey 2013-03-15 21:41:24 -07:00
Andrew xia 5892393140 refactor fair scheduler implementation 
1.Chage "pool" properties to be the memeber of ActiveJob
2.Abstract the Schedulable of Pool and TaskSetManager
3.Abstract the FIFO and FS comparator algorithm
4.Miscellaneous changing of class define and construction
 2013-03-16 11:13:38 +08:00
Matei Zaharia cdbfd1e196 Merge pull request #516 from squito/fix_local_metrics 
Fix local metrics
 2013-03-15 15:13:28 -07:00
Matei Zaharia f9fa2add5c Merge pull request #530 from mbautin/master-update-log4j-and-make-compile-in-IntelliJ 
Add a log4j compile dependency to fix build in IntelliJ
 2013-03-15 15:12:43 -07:00
Mikhail Bautin 7fd2708eda Add a log4j compile dependency to fix build in IntelliJ 
Also rename parent project to spark-parent (otherwise it shows up as
"parent" in IntelliJ, which is very confusing).
 2013-03-15 11:41:51 -07:00
Mark Hamstra 1a4070477d whitespace cleanup 2013-03-15 11:28:28 -07:00
Mark Hamstra 857010392b Fuller implementation of foldByKey 2013-03-15 10:56:05 -07:00
Nick Pentreath 13757b1198 Adding Java versions of Pi and LogQuery 2013-03-15 10:52:01 +02:00
seanm 33fa1e7e4a removing dependency on ZookeeperConsumerConnector + purging last relic of kafka reliability that never solidified (ie- setOffsets) 2013-03-15 00:10:13 -06:00
seanm d069283211 fixing memory leak in kafka MessageHandler 2013-03-14 23:45:33 -06:00
seanm cfa8e769a8 KafkaInputDStream improvements. Allows more Kafka configurability 2013-03-14 23:45:19 -06:00
Mark Hamstra 16a4ca4537 restrict V type of foldByKey in order to retain ClassManifest; added foldByKey to Java API and test 2013-03-14 13:58:37 -07:00
Mark Hamstra b1422cbdd5 added foldByKey 2013-03-14 12:59:58 -07:00
Stephen Haberman 7786881f47 Fix tabs that snuck in. 2013-03-14 14:57:12 -05:00
Stephen Haberman 7d8bb4df3a Allow subtractByKey's other argument to have a different value type. 2013-03-14 14:44:15 -05:00
Stephen Haberman 4632c45af1 Finished subtractByKeys. 2013-03-14 10:35:34 -05:00
Matei Zaharia 4032beba49 Merge pull request #521 from stephenh/earlyclose 
Close the reader in HadoopRDD as soon as iteration end.
 2013-03-13 19:29:46 -07:00
Matei Zaharia 3c97276a4a Merge pull request #524 from andyk/master 
Fix broken link to YARN documentation
 2013-03-13 19:25:08 -07:00
Matei Zaharia 1c3d98197b Merge pull request #517 from Reinvigorate/sm-build-fixes 
Build fixes for streaming /w SBT
 2013-03-13 19:23:48 -07:00
Stephen Haberman 63fe225587 Simplify SubtractedRDD in preparation from subtractByKey. 2013-03-13 17:17:34 -05:00
Andy Konwinski c9097628fc Fix broken link to YARN documentation. 2013-03-13 14:51:13 -07:00
Mark Hamstra cd5b947cf6 Merge branch 'master' of https://github.com/mesos/spark into WithThing 2013-03-13 13:16:14 -07:00
seanm 42822cf95d changing streaming resolver for akka 2013-03-13 11:40:42 -06:00
Stephen Haberman e7f1a69c6b Add a test for NextIterator. 2013-03-13 10:46:33 -05:00
Stephen Haberman 1a175d13b9 Add NextIterator.closeIfNeeded. 2013-03-13 10:17:39 -05:00
Matei Zaharia 2d477fd36f Merge pull request #523 from andyk/master 
Fix broken link in Quick Start
 2013-03-13 06:49:16 -07:00
Andy Konwinski cf73fbd305 Fix another broken link in quick start. 2013-03-13 02:23:44 -07:00
Andy Konwinski b63109763b Fix broken link in Quick Start. 2013-03-13 02:02:34 -07:00
Matei Zaharia 00c4d238dd Merge pull request #518 from woggling/long-bm-sizes 
Send block sizes as longs in BlockManager updates
 2013-03-12 22:19:00 -07:00
Stephen Haberman 8f00d23598 Remove NextIterator.close default implementation. 2013-03-12 12:30:10 -05:00
Harold Lim 0b64e5f1ac Removed some commented code 2013-03-12 13:31:27 +08:00
Harold Lim f5b1fecb9f Cleaned up the code 2013-03-12 13:31:27 +08:00
Harold Lim b5325182a3 Updated/Refactored the Fair Task Scheduler. It does not inherit ClusterScheduler anymore. Rather, ClusterScheduler internally uses TaskSetQueuesManager that handles the scheduling of taskset queues. This is the class that should be extended to support other scheduling policies 2013-03-12 13:31:27 +08:00
Harold Lim 54ed7c4af4 Changed the name of the system property to set the allocation xml 2013-03-12 13:31:27 +08:00
Harold Lim c07087364b Made changes to the SparkContext to have a DynamicVariable for setting local properties that can be passed down the stack. Added an implementation of the fair scheduler 2013-03-12 13:31:27 +08:00
Stephen Haberman 0cf320485d Forgot equals. 2013-03-12 00:05:35 -05:00
Stephen Haberman 9e68f48625 More quickly call close in HadoopRDD. 
This also refactors out the common "gotNext" iterator pattern into
a shared utility class.
 2013-03-11 23:59:17 -05:00