Matei Zaharia
c1e9cdc49f
Merge pull request #525 from stephenh/subtractByKey
...
Add PairRDDFunctions.subtractByKey.
2013-03-16 11:47:45 -07:00
Mark Hamstra
ef75be3bf7
Merge branch 'master' of https://github.com/mesos/spark into foldByKey
2013-03-15 21:41:24 -07:00
Matei Zaharia
cdbfd1e196
Merge pull request #516 from squito/fix_local_metrics
...
Fix local metrics
2013-03-15 15:13:28 -07:00
Matei Zaharia
f9fa2add5c
Merge pull request #530 from mbautin/master-update-log4j-and-make-compile-in-IntelliJ
...
Add a log4j compile dependency to fix build in IntelliJ
2013-03-15 15:12:43 -07:00
Mikhail Bautin
7fd2708eda
Add a log4j compile dependency to fix build in IntelliJ
...
Also rename parent project to spark-parent (otherwise it shows up as
"parent" in IntelliJ, which is very confusing).
2013-03-15 11:41:51 -07:00
Mark Hamstra
1a4070477d
whitespace cleanup
2013-03-15 11:28:28 -07:00
Mark Hamstra
857010392b
Fuller implementation of foldByKey
2013-03-15 10:56:05 -07:00
Nick Pentreath
13757b1198
Adding Java versions of Pi and LogQuery
2013-03-15 10:52:01 +02:00
Mark Hamstra
16a4ca4537
restrict V type of foldByKey in order to retain ClassManifest; added foldByKey to Java API and test
2013-03-14 13:58:37 -07:00
Mark Hamstra
b1422cbdd5
added foldByKey
2013-03-14 12:59:58 -07:00
Stephen Haberman
7786881f47
Fix tabs that snuck in.
2013-03-14 14:57:12 -05:00
Stephen Haberman
7d8bb4df3a
Allow subtractByKey's other argument to have a different value type.
2013-03-14 14:44:15 -05:00
Stephen Haberman
4632c45af1
Finished subtractByKeys.
2013-03-14 10:35:34 -05:00
Matei Zaharia
4032beba49
Merge pull request #521 from stephenh/earlyclose
...
Close the reader in HadoopRDD as soon as iteration end.
2013-03-13 19:29:46 -07:00
Matei Zaharia
3c97276a4a
Merge pull request #524 from andyk/master
...
Fix broken link to YARN documentation
2013-03-13 19:25:08 -07:00
Stephen Haberman
63fe225587
Simplify SubtractedRDD in preparation from subtractByKey.
2013-03-13 17:17:34 -05:00
Andy Konwinski
c9097628fc
Fix broken link to YARN documentation.
2013-03-13 14:51:13 -07:00
Mark Hamstra
cd5b947cf6
Merge branch 'master' of https://github.com/mesos/spark into WithThing
2013-03-13 13:16:14 -07:00
Stephen Haberman
e7f1a69c6b
Add a test for NextIterator.
2013-03-13 10:46:33 -05:00
Stephen Haberman
1a175d13b9
Add NextIterator.closeIfNeeded.
2013-03-13 10:17:39 -05:00
Matei Zaharia
2d477fd36f
Merge pull request #523 from andyk/master
...
Fix broken link in Quick Start
2013-03-13 06:49:16 -07:00
Andy Konwinski
cf73fbd305
Fix another broken link in quick start.
2013-03-13 02:23:44 -07:00
Andy Konwinski
b63109763b
Fix broken link in Quick Start.
2013-03-13 02:02:34 -07:00
Matei Zaharia
00c4d238dd
Merge pull request #518 from woggling/long-bm-sizes
...
Send block sizes as longs in BlockManager updates
2013-03-12 22:19:00 -07:00
Stephen Haberman
8f00d23598
Remove NextIterator.close default implementation.
2013-03-12 12:30:10 -05:00
Stephen Haberman
0cf320485d
Forgot equals.
2013-03-12 00:05:35 -05:00
Stephen Haberman
9e68f48625
More quickly call close in HadoopRDD.
...
This also refactors out the common "gotNext" iterator pattern into
a shared utility class.
2013-03-11 23:59:17 -05:00
Charles Reiss
769d399674
Send block sizes as longs.
2013-03-11 14:17:05 -07:00
Mark Hamstra
a6de80218a
Merge branch 'master' of https://github.com/mesos/spark into WithThing
2013-03-11 00:43:29 -07:00
Matei Zaharia
cbf8f0d4dd
Merge pull request #513 from MLnick/bagel-caching
...
Adds choice of persistence level to Bagel.
2013-03-11 00:23:57 -07:00
Nick Pentreath
8dd943fc3e
Fix doc style
2013-03-11 09:14:00 +02:00
Mark Hamstra
562893bea3
deleted excess curly braces
2013-03-10 22:43:08 -07:00
Imran Rashid
8a11ac3dc7
increase sleep time
2013-03-10 22:31:44 -07:00
Imran Rashid
9f97f2f9d8
add a small wait to one task to make sure some task runtime really is non-zero
2013-03-10 22:30:18 -07:00
Mark Hamstra
1289e7176b
refactored _With API and added foreachPartition
2013-03-10 22:27:13 -07:00
Mark Hamstra
b57df1f5e3
Merge branch 'master' of https://github.com/mesos/spark into WithThing
2013-03-10 16:56:31 -07:00
Matei Zaharia
2e1bbc4e7e
Merge remote-tracking branch 'woggling/dag-sched-driver-port'
...
Conflicts:
core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
2013-03-10 16:52:54 -07:00
Matei Zaharia
d4e29ea878
Update kryo-serializers version in pom.xml to match previous commit
2013-03-10 15:49:11 -07:00
Matei Zaharia
91a9d093bd
Merge pull request #512 from patelh/fix-kryo-serializer
...
Fix reference bug in Kryo serializer, add test, update version
2013-03-10 15:48:23 -07:00
Matei Zaharia
557cfd0f4d
Merge pull request #515 from woggling/deploy-app-death
...
Notify standalone deploy client of application death.
2013-03-10 15:44:57 -07:00
Matei Zaharia
a59cc6060f
Merge remote-tracking branch 'stephenh/nomocks'
...
Conflicts:
core/src/main/scala/spark/storage/BlockManagerMaster.scala
core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
2013-03-10 13:39:10 -07:00
Imran Rashid
20f01a0a1b
enable task metrics in local mode, add tests
2013-03-09 21:17:31 -08:00
Imran Rashid
ec30188a2a
rename remoteFetchWaitTime to fetchWaitTime, since it also includes time from local fetches
2013-03-09 21:16:53 -08:00
Charles Reiss
b0983c5762
Notify standalone deploy client of application death.
...
Usually, this isn't necessary since the application will be removed
as a result of the deploy client disconnecting, but occassionally, the
standalone deploy master removes an application otherwise.
Also mark applications as FAILED instead of FINISHED when they are
killed as a result of their executors failing too many times.
2013-03-09 11:29:45 -08:00
Charles Reiss
d0216cb38b
Prevent DAGSchedulerSuite from corrupting driver.port.
...
Use the LocalSparkContext abstraction to properly manage clearing
spark.driver.port.
2013-03-09 10:49:02 -08:00
Nick Pentreath
d35c5a5176
Adding test for non-default persistence level
2013-03-09 12:52:16 +02:00
Nick Pentreath
1e981d8b26
Added choice of persitance level to Bagel. Also added documentation.
2013-03-09 12:40:48 +02:00
Hiral Patel
664e5fd24b
Fix reference bug in Kryo serializer, add test, update version
2013-03-07 22:16:11 -08:00
Mark Hamstra
5ff0810b11
refactor mapWith, flatMapWith and filterWith to each use two parameter lists
2013-03-05 12:25:44 -08:00
Mark Hamstra
d046d8ad32
whitespace formatting
2013-03-05 00:48:13 -08:00