Matei Zaharia
b8949cab88
Merge pull request #505 from stephenh/volatile
...
Make Executor fields volatile since they're read from the thread pool.
2013-03-23 07:19:34 -07:00
Matei Zaharia
fd53f2fc7b
Merge pull request #510 from markhamstra/WithThing
...
mapWith, flatMapWith and filterWith
2013-03-23 07:13:21 -07:00
Andrew xia
d1d9bdaabe
Just update typo and comments
2013-03-23 07:25:30 +08:00
Stephen Haberman
00170eb0b9
Fix are/our typo.
2013-03-22 12:59:08 -05:00
Stephen Haberman
1c67c7dfd1
Add a shuffle parameter to coalesce.
...
This is useful for when you want just 1 output file (part-00000) but
still up the upstream RDD to be computed in parallel.
2013-03-22 08:54:44 -05:00
Christoph Grothaus
445f387ef4
Bugfix: WorkerWebUI must respect workDirPath from Worker
2013-03-22 11:08:40 +01:00
Matei Zaharia
4c5efcf600
Merge pull request #532 from andyk/master
...
SPARK-715: Adds instructions for building with Maven to documentation
2013-03-20 19:29:23 -07:00
Matei Zaharia
35588490cb
Merge pull request #538 from rxin/cogroup
...
Added mapSideCombine flag to CoGroupedRDD. Added unit test for CoGroupedRDD.
2013-03-20 19:27:47 -07:00
Andy Konwinski
446b801b3b
Fixing typos pointed out by Matei
2013-03-20 17:30:31 -07:00
Stephen Haberman
4f4215311a
Merge branch 'master' into volatile
2013-03-20 15:37:10 -05:00
Matei Zaharia
ca4d083ec8
Merge pull request #528 from MLnick/java-examples
...
[SPARK-707] Adding Java versions of Pi, LogQuery and K-Means examples
2013-03-20 11:22:36 -07:00
Matei Zaharia
b812e6b7bb
Merge pull request #526 from markhamstra/foldByKey
...
Add foldByKey
2013-03-20 11:21:02 -07:00
Nick Pentreath
5a378a1945
Merge branch 'java-examples' of https://github.com/MLnick/spark into java-examples
2013-03-20 09:57:00 +02:00
Nick Pentreath
52398cc1a3
Java indentation 4 --> 2 spaces
2013-03-20 09:55:42 +02:00
Nick Pentreath
7a443bb7d2
Java indentation 2 --> 4 spaces
2013-03-20 09:49:57 +02:00
Reynold Xin
d48ee7e55e
Merge branch 'master' of github.com:mesos/spark into cogroup
2013-03-20 14:00:28 +08:00
Reynold Xin
00a11304fd
Added mapSideCombine flag to CoGroupedRDD. Added unit test for
...
CoGroupedRDD.
2013-03-20 13:49:51 +08:00
Matei Zaharia
945d1e720e
Merge pull request #536 from sasurfer/master
...
CoalescedRDD for many partitions
2013-03-19 21:59:06 -07:00
Matei Zaharia
1cbbe94ac1
Merge pull request #534 from stephenh/removetrycatch
...
Remove try/catch block that can't be hit.
2013-03-19 21:34:34 -07:00
Matei Zaharia
71e53f8894
Merge pull request #537 from wishbear/configurableInputFormat
...
call setConf from input format if it is Configurable
2013-03-19 21:31:41 -07:00
Nick Pentreath
9fa47a2039
A few cosmetic changes for JavaKMeans
2013-03-19 15:31:03 +02:00
Nick Pentreath
568ddf7330
Adding Java K-Means example
2013-03-19 15:29:22 +02:00
Andrey Kouznetsov
bd167f83b0
call setConf from input format if it is Configurable
2013-03-19 17:15:15 +04:00
Giovanni Delussu
aceae029f7
CoalescedRDD changed to work with a big number of partitions both in the original and the new coalesced RDD.
...
The limitation was in the range that Scala.Int can represent.
2013-03-19 11:25:45 +01:00
Stephen Haberman
fb34967815
Remove try/catch block that can't be hit.
2013-03-18 01:55:50 -05:00
Andy Konwinski
ad7f0452ab
Adds page to docs about building using Maven.
...
Adds links to new instructions in:
* The main Spark project README.md
* The docs nav menu called "More"
* The docs Overview page under the "Building" and "Where to Go from Here" sections
2013-03-17 15:02:40 -07:00
Nick Pentreath
b990caeb80
Changes to more closely match line length limit style
2013-03-17 20:03:27 +02:00
Mark Hamstra
ab33e27cc9
constructorOfA -> constructA in doc comments
2013-03-16 15:29:15 -07:00
Mark Hamstra
9784fc1fcd
fix wayward comma in doc comment
2013-03-16 15:25:02 -07:00
Mark Hamstra
32979b5e7d
whitespace
2013-03-16 13:36:46 -07:00
Mark Hamstra
ca9f81e8fc
refactor foldByKey to use combineByKey
2013-03-16 13:31:01 -07:00
Mark Hamstra
1fb192ef40
Merge branch 'master' of https://github.com/mesos/spark into foldByKey
2013-03-16 12:17:13 -07:00
Mark Hamstra
80fc8c82ed
_With[Matei]
2013-03-16 12:16:29 -07:00
Mark Hamstra
38454c4aed
Merge branch 'master' of https://github.com/mesos/spark into WithThing
2013-03-16 11:54:44 -07:00
Matei Zaharia
c1e9cdc49f
Merge pull request #525 from stephenh/subtractByKey
...
Add PairRDDFunctions.subtractByKey.
2013-03-16 11:47:45 -07:00
Mark Hamstra
ef75be3bf7
Merge branch 'master' of https://github.com/mesos/spark into foldByKey
2013-03-15 21:41:24 -07:00
Andrew xia
5892393140
refactor fair scheduler implementation
...
1.Chage "pool" properties to be the memeber of ActiveJob
2.Abstract the Schedulable of Pool and TaskSetManager
3.Abstract the FIFO and FS comparator algorithm
4.Miscellaneous changing of class define and construction
2013-03-16 11:13:38 +08:00
Matei Zaharia
cdbfd1e196
Merge pull request #516 from squito/fix_local_metrics
...
Fix local metrics
2013-03-15 15:13:28 -07:00
Matei Zaharia
f9fa2add5c
Merge pull request #530 from mbautin/master-update-log4j-and-make-compile-in-IntelliJ
...
Add a log4j compile dependency to fix build in IntelliJ
2013-03-15 15:12:43 -07:00
Mikhail Bautin
7fd2708eda
Add a log4j compile dependency to fix build in IntelliJ
...
Also rename parent project to spark-parent (otherwise it shows up as
"parent" in IntelliJ, which is very confusing).
2013-03-15 11:41:51 -07:00
Mark Hamstra
1a4070477d
whitespace cleanup
2013-03-15 11:28:28 -07:00
Mark Hamstra
857010392b
Fuller implementation of foldByKey
2013-03-15 10:56:05 -07:00
Nick Pentreath
13757b1198
Adding Java versions of Pi and LogQuery
2013-03-15 10:52:01 +02:00
Mark Hamstra
16a4ca4537
restrict V type of foldByKey in order to retain ClassManifest; added foldByKey to Java API and test
2013-03-14 13:58:37 -07:00
Mark Hamstra
b1422cbdd5
added foldByKey
2013-03-14 12:59:58 -07:00
Stephen Haberman
7786881f47
Fix tabs that snuck in.
2013-03-14 14:57:12 -05:00
Stephen Haberman
7d8bb4df3a
Allow subtractByKey's other argument to have a different value type.
2013-03-14 14:44:15 -05:00
Stephen Haberman
4632c45af1
Finished subtractByKeys.
2013-03-14 10:35:34 -05:00
Matei Zaharia
4032beba49
Merge pull request #521 from stephenh/earlyclose
...
Close the reader in HadoopRDD as soon as iteration end.
2013-03-13 19:29:46 -07:00
Matei Zaharia
3c97276a4a
Merge pull request #524 from andyk/master
...
Fix broken link to YARN documentation
2013-03-13 19:25:08 -07:00