ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2871 commits 6 branches 181 tags 397 MiB
Commit graph

2871 commits

This branch
This branch
All branches
Author SHA1 Message Date
Nick Pentreath 7a443bb7d2 Java indentation 2 --> 4 spaces 2013-03-20 09:49:57 +02:00
Reynold Xin d48ee7e55e Merge branch 'master' of github.com:mesos/spark into cogroup 2013-03-20 14:00:28 +08:00
Reynold Xin 00a11304fd Added mapSideCombine flag to CoGroupedRDD. Added unit test for 
CoGroupedRDD.
 2013-03-20 13:49:51 +08:00
Matei Zaharia 945d1e720e Merge pull request #536 from sasurfer/master 
CoalescedRDD for many partitions
 2013-03-19 21:59:06 -07:00
Matei Zaharia 1cbbe94ac1 Merge pull request #534 from stephenh/removetrycatch 
Remove try/catch block that can't be hit.
 2013-03-19 21:34:34 -07:00
Matei Zaharia 71e53f8894 Merge pull request #537 from wishbear/configurableInputFormat 
call setConf from input format if it is Configurable
 2013-03-19 21:31:41 -07:00
Nick Pentreath 9fa47a2039 A few cosmetic changes for JavaKMeans 2013-03-19 15:31:03 +02:00
Nick Pentreath 568ddf7330 Adding Java K-Means example 2013-03-19 15:29:22 +02:00
Andrey Kouznetsov bd167f83b0 call setConf from input format if it is Configurable 2013-03-19 17:15:15 +04:00
Giovanni Delussu aceae029f7 CoalescedRDD changed to work with a big number of partitions both in the original and the new coalesced RDD. 
The limitation was in the range that Scala.Int can represent.
 2013-03-19 11:25:45 +01:00
Reynold Xin ba9d00c44a Merge branch 'master' into graph 
Conflicts:
	run2.cmd
 2013-03-18 18:30:14 +08:00
Stephen Haberman fb34967815 Remove try/catch block that can't be hit. 2013-03-18 01:55:50 -05:00
Andy Konwinski ad7f0452ab Adds page to docs about building using Maven. 
Adds links to new instructions in:
* The main Spark project README.md
* The docs nav menu called "More"
* The docs Overview page under the "Building" and "Where to Go from Here" sections
 2013-03-17 15:02:40 -07:00
Nick Pentreath b990caeb80 Changes to more closely match line length limit style 2013-03-17 20:03:27 +02:00
Mark Hamstra ab33e27cc9 constructorOfA -> constructA in doc comments 2013-03-16 15:29:15 -07:00
Mark Hamstra 9784fc1fcd fix wayward comma in doc comment 2013-03-16 15:25:02 -07:00
Mark Hamstra 32979b5e7d whitespace 2013-03-16 13:36:46 -07:00
Mark Hamstra ca9f81e8fc refactor foldByKey to use combineByKey 2013-03-16 13:31:01 -07:00
Mark Hamstra 1fb192ef40 Merge branch 'master' of https://github.com/mesos/spark into foldByKey 2013-03-16 12:17:13 -07:00
Mark Hamstra 80fc8c82ed _With[Matei] 2013-03-16 12:16:29 -07:00
Mark Hamstra 38454c4aed Merge branch 'master' of https://github.com/mesos/spark into WithThing 2013-03-16 11:54:44 -07:00
Matei Zaharia c1e9cdc49f Merge pull request #525 from stephenh/subtractByKey 
Add PairRDDFunctions.subtractByKey.
 2013-03-16 11:47:45 -07:00
Mark Hamstra ef75be3bf7 Merge branch 'master' of https://github.com/mesos/spark into foldByKey 2013-03-15 21:41:24 -07:00
Andrew xia 5892393140 refactor fair scheduler implementation 
1.Chage "pool" properties to be the memeber of ActiveJob
2.Abstract the Schedulable of Pool and TaskSetManager
3.Abstract the FIFO and FS comparator algorithm
4.Miscellaneous changing of class define and construction
 2013-03-16 11:13:38 +08:00
Matei Zaharia cdbfd1e196 Merge pull request #516 from squito/fix_local_metrics 
Fix local metrics
 2013-03-15 15:13:28 -07:00
Matei Zaharia f9fa2add5c Merge pull request #530 from mbautin/master-update-log4j-and-make-compile-in-IntelliJ 
Add a log4j compile dependency to fix build in IntelliJ
 2013-03-15 15:12:43 -07:00
Mikhail Bautin 7fd2708eda Add a log4j compile dependency to fix build in IntelliJ 
Also rename parent project to spark-parent (otherwise it shows up as
"parent" in IntelliJ, which is very confusing).
 2013-03-15 11:41:51 -07:00
Mark Hamstra 1a4070477d whitespace cleanup 2013-03-15 11:28:28 -07:00
Mark Hamstra 857010392b Fuller implementation of foldByKey 2013-03-15 10:56:05 -07:00
Nick Pentreath 13757b1198 Adding Java versions of Pi and LogQuery 2013-03-15 10:52:01 +02:00
Mark Hamstra 16a4ca4537 restrict V type of foldByKey in order to retain ClassManifest; added foldByKey to Java API and test 2013-03-14 13:58:37 -07:00
Mark Hamstra b1422cbdd5 added foldByKey 2013-03-14 12:59:58 -07:00
Stephen Haberman 7786881f47 Fix tabs that snuck in. 2013-03-14 14:57:12 -05:00
Stephen Haberman 7d8bb4df3a Allow subtractByKey's other argument to have a different value type. 2013-03-14 14:44:15 -05:00
Stephen Haberman 4632c45af1 Finished subtractByKeys. 2013-03-14 10:35:34 -05:00
Matei Zaharia 4032beba49 Merge pull request #521 from stephenh/earlyclose 
Close the reader in HadoopRDD as soon as iteration end.
 2013-03-13 19:29:46 -07:00
Matei Zaharia 3c97276a4a Merge pull request #524 from andyk/master 
Fix broken link to YARN documentation
 2013-03-13 19:25:08 -07:00
Stephen Haberman 63fe225587 Simplify SubtractedRDD in preparation from subtractByKey. 2013-03-13 17:17:34 -05:00
Andy Konwinski c9097628fc Fix broken link to YARN documentation. 2013-03-13 14:51:13 -07:00
Mark Hamstra cd5b947cf6 Merge branch 'master' of https://github.com/mesos/spark into WithThing 2013-03-13 13:16:14 -07:00
Stephen Haberman e7f1a69c6b Add a test for NextIterator. 2013-03-13 10:46:33 -05:00
Stephen Haberman 1a175d13b9 Add NextIterator.closeIfNeeded. 2013-03-13 10:17:39 -05:00
Matei Zaharia 2d477fd36f Merge pull request #523 from andyk/master 
Fix broken link in Quick Start
 2013-03-13 06:49:16 -07:00
Andy Konwinski cf73fbd305 Fix another broken link in quick start. 2013-03-13 02:23:44 -07:00
Andy Konwinski b63109763b Fix broken link in Quick Start. 2013-03-13 02:02:34 -07:00
Matei Zaharia 00c4d238dd Merge pull request #518 from woggling/long-bm-sizes 
Send block sizes as longs in BlockManager updates
 2013-03-12 22:19:00 -07:00
Stephen Haberman 8f00d23598 Remove NextIterator.close default implementation. 2013-03-12 12:30:10 -05:00
Harold Lim 0b64e5f1ac Removed some commented code 2013-03-12 13:31:27 +08:00
Harold Lim f5b1fecb9f Cleaned up the code 2013-03-12 13:31:27 +08:00
Harold Lim b5325182a3 Updated/Refactored the Fair Task Scheduler. It does not inherit ClusterScheduler anymore. Rather, ClusterScheduler internally uses TaskSetQueuesManager that handles the scheduling of taskset queues. This is the class that should be extended to support other scheduling policies 2013-03-12 13:31:27 +08:00