Mridul Muralidharan
|
8faf5c51c3
|
Patch from Thomas Graves to improve the YARN Client, and move to more production ready hadoop yarn branch
|
2013-04-24 02:31:57 +05:30 |
|
|
Mridul Muralidharan
|
7acab3ab45
|
Fix review comments, add a new api to SparkHadoopUtil to create appropriate Configuration. Modify an example to show how to use SplitInfo
|
2013-04-22 08:01:13 +05:30 |
|
|
Mridul Muralidharan
|
ac2e8e8720
|
Add some basic documentation
|
2013-04-19 00:13:19 +05:30 |
|
|
Mridul Muralidharan
|
5ee2f5c483
|
Cache pattern, add (commented out) alternatives for check* apis
|
2013-04-17 23:13:34 +05:30 |
|
|
Mridul Muralidharan
|
f07961060d
|
Add a small note on spark.tasks.schedule.aggression
|
2013-04-17 23:13:02 +05:30 |
|
|
Mridul Muralidharan
|
02dffd2eb0
|
Ensure all ask/await block for spark.akka.askTimeout - so that it is controllable : instead of arbitrary timeouts spread across codebase. In our tests, we use 30 seconds, though default of 10 is maintained
|
2013-04-17 05:52:57 +05:30 |
|
|
Mridul Muralidharan
|
a402b23bcd
|
Fudge order of classpath - so that our jars take precedence over what is in CLASSPATH variable. Sounds logical, hope there is no issue cos of it
|
2013-04-17 05:52:00 +05:30 |
|
|
Mridul Muralidharan
|
bcdde331c3
|
Move from master to driver
|
2013-04-17 04:12:18 +05:30 |
|
|
Mridul Muralidharan
|
ad80f68eb5
|
remove spurious debug statements
|
2013-04-16 22:15:34 +05:30 |
|
|
Mridul Muralidharan
|
f7969f72ee
|
Fix exception when checkpoint path does not exist (no data in rdd which is being checkpointed for example)
|
2013-04-16 21:51:38 +05:30 |
|
|
Mridul Muralidharan
|
323ab8ff3b
|
Scala does not prevent variable shadowing ! Sick error due to it ...
|
2013-04-16 17:05:10 +05:30 |
|
|
Mridul Muralidharan
|
59c380d69a
|
Fix npe
|
2013-04-16 03:29:38 +05:30 |
|
|
Mridul Muralidharan
|
dd2b64ec97
|
Fix bug with atomic update
|
2013-04-16 03:19:24 +05:30 |
|
|
Mridul Muralidharan
|
5540ab8243
|
Use hostname instead of hostport for executor, fix creation of workdir
|
2013-04-16 02:57:43 +05:30 |
|
|
Mridul Muralidharan
|
eb7e95e833
|
Commit job to persist files
|
2013-04-16 02:56:36 +05:30 |
|
|
Mridul Muralidharan
|
19652a44be
|
Fix issue with FileSuite failing
|
2013-04-15 19:16:36 +05:30 |
|
|
Mridul Muralidharan
|
54b3d45b81
|
Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues
|
2013-04-15 18:26:50 +05:30 |
|
|
Mridul Muralidharan
|
d90d2af103
|
Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues
|
2013-04-15 18:12:11 +05:30 |
|
|
Mridul Muralidharan
|
6798a09df8
|
Add support for building against hadoop2-yarn : adding new maven profile for it
|
2013-04-07 17:47:38 +05:30 |
|
Matei Zaharia
|
9831bc1a09
|
Merge pull request #539 from cgrothaus/fix-webui-workdirpath
Bugfix: WorkerWebUI must respect workDirPath from Worker
|
2013-03-29 22:16:22 -07:00 |
|
|
Matei Zaharia
|
3cc8ab6e29
|
Merge pull request #541 from stephenh/shufflecoalesce
Add a shuffle parameter to coalesce.
|
2013-03-29 22:14:07 -07:00 |
|
Holden Karau
|
f5df729b12
|
Explicitly catch all throwables (warning in 2.10)
|
2013-03-24 16:15:32 -07:00 |
|
Stephen Haberman
|
dd854d5b9f
|
Use Boolean in the Java API, and != for assert.
|
2013-03-23 11:49:45 -05:00 |
|
|
Stephen Haberman
|
4ca273edc4
|
Merge branch 'master' into shufflecoalesce
Conflicts:
core/src/test/scala/spark/RDDSuite.scala
|
2013-03-23 11:45:45 -05:00 |
|
|
Matei Zaharia
|
b8949cab88
|
Merge pull request #505 from stephenh/volatile
Make Executor fields volatile since they're read from the thread pool.
|
2013-03-23 07:19:34 -07:00 |
|
|
Matei Zaharia
|
fd53f2fc7b
|
Merge pull request #510 from markhamstra/WithThing
mapWith, flatMapWith and filterWith
|
2013-03-23 07:13:21 -07:00 |
|
|
Stephen Haberman
|
00170eb0b9
|
Fix are/our typo.
|
2013-03-22 12:59:08 -05:00 |
|
|
Stephen Haberman
|
1c67c7dfd1
|
Add a shuffle parameter to coalesce.
This is useful for when you want just 1 output file (part-00000) but
still up the upstream RDD to be computed in parallel.
|
2013-03-22 08:54:44 -05:00 |
|
Christoph Grothaus
|
445f387ef4
|
Bugfix: WorkerWebUI must respect workDirPath from Worker
|
2013-03-22 11:08:40 +01:00 |
|
|
Matei Zaharia
|
35588490cb
|
Merge pull request #538 from rxin/cogroup
Added mapSideCombine flag to CoGroupedRDD. Added unit test for CoGroupedRDD.
|
2013-03-20 19:27:47 -07:00 |
|
|
Stephen Haberman
|
4f4215311a
|
Merge branch 'master' into volatile
|
2013-03-20 15:37:10 -05:00 |
|
|
Matei Zaharia
|
b812e6b7bb
|
Merge pull request #526 from markhamstra/foldByKey
Add foldByKey
|
2013-03-20 11:21:02 -07:00 |
|
Reynold Xin
|
d48ee7e55e
|
Merge branch 'master' of github.com:mesos/spark into cogroup
|
2013-03-20 14:00:28 +08:00 |
|
|
Reynold Xin
|
00a11304fd
|
Added mapSideCombine flag to CoGroupedRDD. Added unit test for
CoGroupedRDD.
|
2013-03-20 13:49:51 +08:00 |
|
|
Matei Zaharia
|
945d1e720e
|
Merge pull request #536 from sasurfer/master
CoalescedRDD for many partitions
|
2013-03-19 21:59:06 -07:00 |
|
|
Matei Zaharia
|
1cbbe94ac1
|
Merge pull request #534 from stephenh/removetrycatch
Remove try/catch block that can't be hit.
|
2013-03-19 21:34:34 -07:00 |
|
Andrey Kouznetsov
|
bd167f83b0
|
call setConf from input format if it is Configurable
|
2013-03-19 17:15:15 +04:00 |
|
Giovanni Delussu
|
aceae029f7
|
CoalescedRDD changed to work with a big number of partitions both in the original and the new coalesced RDD.
The limitation was in the range that Scala.Int can represent.
|
2013-03-19 11:25:45 +01:00 |
|
|
Stephen Haberman
|
fb34967815
|
Remove try/catch block that can't be hit.
|
2013-03-18 01:55:50 -05:00 |
|
Mark Hamstra
|
ab33e27cc9
|
constructorOfA -> constructA in doc comments
|
2013-03-16 15:29:15 -07:00 |
|
|
Mark Hamstra
|
9784fc1fcd
|
fix wayward comma in doc comment
|
2013-03-16 15:25:02 -07:00 |
|
|
Mark Hamstra
|
32979b5e7d
|
whitespace
|
2013-03-16 13:36:46 -07:00 |
|
|
Mark Hamstra
|
ca9f81e8fc
|
refactor foldByKey to use combineByKey
|
2013-03-16 13:31:01 -07:00 |
|
|
Mark Hamstra
|
1fb192ef40
|
Merge branch 'master' of https://github.com/mesos/spark into foldByKey
|
2013-03-16 12:17:13 -07:00 |
|
|
Mark Hamstra
|
80fc8c82ed
|
_With[Matei]
|
2013-03-16 12:16:29 -07:00 |
|
|
Mark Hamstra
|
38454c4aed
|
Merge branch 'master' of https://github.com/mesos/spark into WithThing
|
2013-03-16 11:54:44 -07:00 |
|
|
Matei Zaharia
|
c1e9cdc49f
|
Merge pull request #525 from stephenh/subtractByKey
Add PairRDDFunctions.subtractByKey.
|
2013-03-16 11:47:45 -07:00 |
|
|
Mark Hamstra
|
ef75be3bf7
|
Merge branch 'master' of https://github.com/mesos/spark into foldByKey
|
2013-03-15 21:41:24 -07:00 |
|
|
Matei Zaharia
|
cdbfd1e196
|
Merge pull request #516 from squito/fix_local_metrics
Fix local metrics
|
2013-03-15 15:13:28 -07:00 |
|
Mikhail Bautin
|
7fd2708eda
|
Add a log4j compile dependency to fix build in IntelliJ
Also rename parent project to spark-parent (otherwise it shows up as
"parent" in IntelliJ, which is very confusing).
|
2013-03-15 11:41:51 -07:00 |
|