Gavin Li
4508089fc3
refine comments and add sc.clean
2013-06-17 05:23:46 +00:00
Gavin Li
728665d34b
Merge branch 'enhance_pipe' of ssh://github.com/lyogavin/spark into enhance_pipe
2013-06-17 00:23:29 +00:00
Yu L Li
33c55ec713
Merge remote branch 'upstream/master' into enhance_pipe
2013-06-17 00:21:26 +00:00
Gavin Li
e6ae049283
Merge remote-tracking branch 'upstream1/master' into enhance_pipe
2013-06-16 22:53:39 +00:00
Gavin Li
fb6d733fa8
update according to comments
2013-06-16 22:32:55 +00:00
Matei Zaharia
f961aac8b2
Merge pull request #649 from ryanlecompte/master
...
Add top K method to RDD using a bounded priority queue
2013-06-15 00:53:41 -07:00
ryanlecompte
e8801d4490
use delegation for BoundedPriorityQueue, add Java API
2013-06-14 23:39:05 -07:00
Matei Zaharia
6602d94973
Merge pull request #651 from rxin/groupbykey
...
SPARK-772 / SPARK-774: groupByKey and cogroup should disable map side combine
2013-06-14 10:41:31 -07:00
Reynold Xin
2cc188fd54
SPARK-774: cogroup should also disable map side combine by default
2013-06-14 00:10:54 -07:00
Reynold Xin
6738178d0d
SPARK-772: groupByKey should disable map side combine.
2013-06-13 23:59:42 -07:00
ryanlecompte
93b3f5e535
drop unneeded ClassManifest implicit
2013-06-13 16:26:35 -07:00
ryanlecompte
44b8dbaede
use Iterator.single(elem) instead of Iterator(elem) for improved performance based on scaladocs
2013-06-13 16:23:15 -07:00
Matei Zaharia
d93851aedf
Merge pull request #645 from pwendell/compression
...
Adding compression to Hadoop save functions
2013-06-13 13:38:45 -07:00
Matei Zaharia
f1da591a8a
Merge pull request #646 from markhamstra/jvmArgs
...
Fixed jvmArgs in maven build.
2013-06-12 17:55:08 -07:00
Mark Hamstra
3f96c6f27b
Fixed jvmArgs in maven build.
2013-06-12 17:24:22 -07:00
ryanlecompte
db5bca08ff
add a new top K method to RDD using a bounded priority queue
2013-06-12 10:54:16 -07:00
Matei Zaharia
dc4073654b
Revert "Fix start-slave not passing instance number to spark-daemon."
...
This reverts commit a674d67c0a
2013-06-11 00:08:02 -04:00
Matei Zaharia
0e94b734be
Merge pull request #625 from stephenh/fix-start-slave
...
Fix start-slave not passing instance number to spark-daemon.
2013-06-10 13:00:31 -07:00
Patrick Wendell
fd6148c8b2
Removing print statement
2013-06-10 10:27:25 -07:00
Patrick Wendell
ef14dc2e77
Adding Java-API version of compression codec
2013-06-09 18:09:46 -07:00
Patrick Wendell
df592192e7
Monads FTW
2013-06-09 18:09:24 -07:00
Patrick Wendell
083a3485ab
Clean extra whitespace
2013-06-09 11:49:33 -07:00
Patrick Wendell
d1bbcebae5
Adding compression to Hadoop save functions
2013-06-09 11:39:35 -07:00
Matei Zaharia
74b91d53bc
Merge pull request #629 from c0s/master
...
Sometime Maven build runs out of PermGen space.
2013-06-08 01:19:40 -07:00
Matei Zaharia
b58a29295b
Small formatting and style fixes
2013-06-07 22:51:28 -07:00
Matei Zaharia
c8fc423bc2
Merge pull request #631 from jerryshao/master
...
Fix block manager UI display issue when enable spark.cleaner.ttl
2013-06-07 22:43:18 -07:00
Matei Zaharia
c9ca0a4a58
Small code style fix to SchedulingAlgorithm.scala
2013-06-07 22:40:44 -07:00
Matei Zaharia
1ae60bcb36
Merge pull request #634 from xiajunluan/master
...
[Spark-753] Fix ClusterSchedulSuite unit test failed
2013-06-07 22:39:06 -07:00
Gavin Li
e179ff8a32
update according to comments
2013-06-05 22:41:05 +00:00
Matei Zaharia
fff3728552
Merge pull request #640 from pwendell/timeout-update
...
Fixing bug in BlockManager timeout
2013-06-04 16:09:50 -07:00
Patrick Wendell
061fd3ae36
Fixing bug in BlockManager timeout
2013-06-04 19:02:44 -04:00
Matei Zaharia
f420d4f228
Merge pull request #639 from pwendell/timeout-update
...
Bump akka and blockmanager timeouts to 60 seconds
2013-06-04 15:25:58 -07:00
Patrick Wendell
8bd4e12104
Bump akka and blockmanager timeouts to 60 seconds
2013-06-04 18:14:24 -04:00
Konstantin Boudnik
d1286231e0
Sometime Maven build runs out of PermGen space.
2013-06-03 15:55:44 -07:00
Andrew xia
606bb1b450
Fix schedulingAlgorithm bugs for unit test
2013-06-03 10:29:23 +08:00
Gavin Li
4a9913d66a
add ut for pipe enhancement
2013-06-02 23:21:09 +00:00
Gavin Li
9f84315c05
enhance pipe to support what we can do in hadoop streaming
2013-06-01 00:26:10 +00:00
Matei Zaharia
84530ba6d9
Merge pull request #636 from rxin/unpersist
...
Unpersist More block manager cleanup.
2013-05-31 17:06:13 -07:00
Reynold Xin
de1167bf2c
Incorporated Charles' feedback to put rdd metadata removal in
...
BlockManagerMasterActor.
2013-05-31 15:54:57 -07:00
Reynold Xin
ba5e544461
More block manager cleanup.
...
Implemented a removeRdd method in BlockManager, and use that to
implement RDD.unpersist. Previously, unpersist needs to send B akka
messages, where B = number of blocks. Now unpersist only needs to send W
akka messages, where W = the number of workers.
2013-05-31 01:48:16 -07:00
jerryshao
926f41cc52
fix block manager UI display issue when enable spark.cleaner.ttl
2013-05-31 09:32:52 +08:00
Reynold Xin
f6ad3781b1
Fixed the flaky unpersist test in RDDSuite.
2013-05-30 16:28:08 -07:00
Reynold Xin
bed1b08169
Do not create symlink for local add file. Instead, copy the file.
...
This prevents Spark from changing the original file's permission, and
also allow add file to work on non-posix operating systems.
2013-05-30 16:21:49 -07:00
Reynold Xin
ef77bb73c6
Merge pull request #627 from shivaram/master
...
Netty and shuffle bug fixes
2013-05-30 14:50:06 -07:00
Shivaram Venkataraman
3b0cd17343
Merge branch 'master' of git://github.com/mesos/spark
...
Conflicts:
core/src/test/scala/spark/ShuffleSuite.scala
2013-05-30 14:36:24 -07:00
Reynold Xin
8cb817820f
Merge pull request #628 from shivaram/zero-block-size
...
Skip fetching zero-sized blocks in NIO.
2013-05-30 14:17:44 -07:00
Shivaram Venkataraman
19fd6d54c0
Also flush serializer in revertPartialWrites
2013-05-29 17:29:34 -07:00
Shivaram Venkataraman
618c8cae1e
Skip fetching zero-sized blocks in OIO.
...
Also unify splitLocalRemoteBlocks for netty/nio and add a test case
2013-05-29 13:18:54 -07:00
Matei Zaharia
6ed71390d9
Merge pull request #626 from stephenh/remove-add-if-no-port
...
Remove unused addIfNoPort.
2013-05-29 10:14:22 -07:00
Shivaram Venkataraman
b79b10a6d6
Flush serializer to fix zero-size kryo blocks bug.
...
Also convert the local-cluster test case to check for non-zero block sizes
2013-05-29 00:52:55 -07:00
