Matei Zaharia
a64c107449
Make ShuffledRDD.prev transient
2013-04-15 16:41:51 -04:00
Matei Zaharia
c35d530bcf
Fix compile error
2013-04-13 12:43:12 -04:00
Andrew Ash
29d3440efb
Add details when BlockManager heartbeats time out
...
Makes it more clear what the threshold was for tuning spark.storage.blockManagerSlaveTimeoutMs
Before:
WARN "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats
After:
WARN "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats: 19216ms exceeds 15000ms
2013-04-11 01:54:02 -03:00
Matei Zaharia
65caa8f711
Merge remote-tracking branch 'jey/bump-development-version-to-0.8.0'
...
Conflicts:
docs/_config.yml
project/SparkBuild.scala
2013-04-08 12:43:17 -04:00
Matei Zaharia
054feb6448
Fixed a bug with zip
2013-04-07 21:15:21 -04:00
Matei Zaharia
b5900d47b1
Fix compile warning
2013-04-07 20:55:42 -04:00
Matei Zaharia
6962d40b44
Fix deprecated warning
2013-04-07 20:27:33 -04:00
Mark Hamstra
e215f67923
Correct sense of 'filter out' in comment.
2013-03-31 08:00:13 -07:00
Mark Hamstra
8bcdc64005
Fixed broken filter in getWritableClass[T]
2013-03-30 22:09:52 -07:00
Matei Zaharia
9831bc1a09
Merge pull request #539 from cgrothaus/fix-webui-workdirpath
...
Bugfix: WorkerWebUI must respect workDirPath from Worker
2013-03-29 22:16:22 -07:00
Matei Zaharia
3cc8ab6e29
Merge pull request #541 from stephenh/shufflecoalesce
...
Add a shuffle parameter to coalesce.
2013-03-29 22:14:07 -07:00
Jey Kottalam
bc8ba222ff
Bump development version to 0.8.0
2013-03-28 15:42:01 -07:00
Holden Karau
f5df729b12
Explicitly catch all throwables (warning in 2.10)
2013-03-24 16:15:32 -07:00
Stephen Haberman
dd854d5b9f
Use Boolean in the Java API, and != for assert.
2013-03-23 11:49:45 -05:00
Stephen Haberman
4ca273edc4
Merge branch 'master' into shufflecoalesce
...
Conflicts:
core/src/test/scala/spark/RDDSuite.scala
2013-03-23 11:45:45 -05:00
Matei Zaharia
b8949cab88
Merge pull request #505 from stephenh/volatile
...
Make Executor fields volatile since they're read from the thread pool.
2013-03-23 07:19:34 -07:00
Matei Zaharia
fd53f2fc7b
Merge pull request #510 from markhamstra/WithThing
...
mapWith, flatMapWith and filterWith
2013-03-23 07:13:21 -07:00
Stephen Haberman
00170eb0b9
Fix are/our typo.
2013-03-22 12:59:08 -05:00
Stephen Haberman
1c67c7dfd1
Add a shuffle parameter to coalesce.
...
This is useful for when you want just 1 output file (part-00000) but
still up the upstream RDD to be computed in parallel.
2013-03-22 08:54:44 -05:00
Christoph Grothaus
445f387ef4
Bugfix: WorkerWebUI must respect workDirPath from Worker
2013-03-22 11:08:40 +01:00
Matei Zaharia
35588490cb
Merge pull request #538 from rxin/cogroup
...
Added mapSideCombine flag to CoGroupedRDD. Added unit test for CoGroupedRDD.
2013-03-20 19:27:47 -07:00
Stephen Haberman
4f4215311a
Merge branch 'master' into volatile
2013-03-20 15:37:10 -05:00
Matei Zaharia
b812e6b7bb
Merge pull request #526 from markhamstra/foldByKey
...
Add foldByKey
2013-03-20 11:21:02 -07:00
Reynold Xin
d48ee7e55e
Merge branch 'master' of github.com:mesos/spark into cogroup
2013-03-20 14:00:28 +08:00
Reynold Xin
00a11304fd
Added mapSideCombine flag to CoGroupedRDD. Added unit test for
...
CoGroupedRDD.
2013-03-20 13:49:51 +08:00
Matei Zaharia
945d1e720e
Merge pull request #536 from sasurfer/master
...
CoalescedRDD for many partitions
2013-03-19 21:59:06 -07:00
Matei Zaharia
1cbbe94ac1
Merge pull request #534 from stephenh/removetrycatch
...
Remove try/catch block that can't be hit.
2013-03-19 21:34:34 -07:00
Andrey Kouznetsov
bd167f83b0
call setConf from input format if it is Configurable
2013-03-19 17:15:15 +04:00
Giovanni Delussu
aceae029f7
CoalescedRDD changed to work with a big number of partitions both in the original and the new coalesced RDD.
...
The limitation was in the range that Scala.Int can represent.
2013-03-19 11:25:45 +01:00
Stephen Haberman
fb34967815
Remove try/catch block that can't be hit.
2013-03-18 01:55:50 -05:00
Mark Hamstra
ab33e27cc9
constructorOfA -> constructA in doc comments
2013-03-16 15:29:15 -07:00
Mark Hamstra
9784fc1fcd
fix wayward comma in doc comment
2013-03-16 15:25:02 -07:00
Mark Hamstra
32979b5e7d
whitespace
2013-03-16 13:36:46 -07:00
Mark Hamstra
ca9f81e8fc
refactor foldByKey to use combineByKey
2013-03-16 13:31:01 -07:00
Mark Hamstra
1fb192ef40
Merge branch 'master' of https://github.com/mesos/spark into foldByKey
2013-03-16 12:17:13 -07:00
Mark Hamstra
80fc8c82ed
_With[Matei]
2013-03-16 12:16:29 -07:00
Mark Hamstra
38454c4aed
Merge branch 'master' of https://github.com/mesos/spark into WithThing
2013-03-16 11:54:44 -07:00
Matei Zaharia
c1e9cdc49f
Merge pull request #525 from stephenh/subtractByKey
...
Add PairRDDFunctions.subtractByKey.
2013-03-16 11:47:45 -07:00
Mark Hamstra
ef75be3bf7
Merge branch 'master' of https://github.com/mesos/spark into foldByKey
2013-03-15 21:41:24 -07:00
Matei Zaharia
cdbfd1e196
Merge pull request #516 from squito/fix_local_metrics
...
Fix local metrics
2013-03-15 15:13:28 -07:00
Mikhail Bautin
7fd2708eda
Add a log4j compile dependency to fix build in IntelliJ
...
Also rename parent project to spark-parent (otherwise it shows up as
"parent" in IntelliJ, which is very confusing).
2013-03-15 11:41:51 -07:00
Mark Hamstra
1a4070477d
whitespace cleanup
2013-03-15 11:28:28 -07:00
Mark Hamstra
857010392b
Fuller implementation of foldByKey
2013-03-15 10:56:05 -07:00
Mark Hamstra
16a4ca4537
restrict V type of foldByKey in order to retain ClassManifest; added foldByKey to Java API and test
2013-03-14 13:58:37 -07:00
Mark Hamstra
b1422cbdd5
added foldByKey
2013-03-14 12:59:58 -07:00
Stephen Haberman
7786881f47
Fix tabs that snuck in.
2013-03-14 14:57:12 -05:00
Stephen Haberman
7d8bb4df3a
Allow subtractByKey's other argument to have a different value type.
2013-03-14 14:44:15 -05:00
Stephen Haberman
4632c45af1
Finished subtractByKeys.
2013-03-14 10:35:34 -05:00
Matei Zaharia
4032beba49
Merge pull request #521 from stephenh/earlyclose
...
Close the reader in HadoopRDD as soon as iteration end.
2013-03-13 19:29:46 -07:00
Stephen Haberman
63fe225587
Simplify SubtractedRDD in preparation from subtractByKey.
2013-03-13 17:17:34 -05:00