ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2987 commits 6 branches 181 tags 397 MiB
Commit graph

2987 commits

This branch
This branch
All branches
Author SHA1 Message Date
Patrick Wendell df592192e7 Monads FTW 2013-06-09 18:09:24 -07:00
Patrick Wendell 083a3485ab Clean extra whitespace 2013-06-09 11:49:33 -07:00
Patrick Wendell d1bbcebae5 Adding compression to Hadoop save functions 2013-06-09 11:39:35 -07:00
Mingfei ade822011d not check return value of eventQueue.take 2013-06-08 16:26:45 +08:00
Matei Zaharia 74b91d53bc Merge pull request #629 from c0s/master 
Sometime Maven build runs out of PermGen space.
 2013-06-08 01:19:40 -07:00
Mingfei 4fd86e0e10 delete test code for joblogger in SparkContext 2013-06-08 15:45:47 +08:00
Mingfei 362f0f93ac Merge branch 'master' of https://github.com/mesos/spark 2013-06-08 15:20:13 +08:00
Mingfei 1a4d93c025 modify to pass job annotation by localProperties and use daeamon thread to do joblogger's work 2013-06-08 14:23:39 +08:00
Matei Zaharia b58a29295b Small formatting and style fixes 2013-06-07 22:51:28 -07:00
Matei Zaharia c8fc423bc2 Merge pull request #631 from jerryshao/master 
Fix block manager UI display issue when enable spark.cleaner.ttl
 2013-06-07 22:43:18 -07:00
Matei Zaharia c9ca0a4a58 Small code style fix to SchedulingAlgorithm.scala 2013-06-07 22:40:44 -07:00
Matei Zaharia 1ae60bcb36 Merge pull request #634 from xiajunluan/master 
[Spark-753] Fix ClusterSchedulSuite unit test failed
 2013-06-07 22:39:06 -07:00
Shivaram Venkataraman ac480fd977 Clean up variables and counters in BlockFetcherIterator 2013-06-06 16:34:27 -07:00
Gavin Li e179ff8a32 update according to comments 2013-06-05 22:41:05 +00:00
Shivaram Venkataraman cb2f5046ee Pass in bufferSize to BufferedOutputStream 2013-06-05 15:09:02 -07:00
Shivaram Venkataraman c851957fe4 Don't write zero block files with java serializer 2013-06-05 14:28:38 -07:00
Christopher Nguyen 9d35904357 In the current code, when both partitions happen to have zero-length, the return mean will be NaN. 
Consequently, the result of mean after reducing over all partitions will also be NaN,
which is not correct if there are partitions with non-zero length. This patch fixes this issue.
 2013-06-04 22:12:47 -07:00
Matei Zaharia fff3728552 Merge pull request #640 from pwendell/timeout-update 
Fixing bug in BlockManager timeout
 2013-06-04 16:09:50 -07:00
Patrick Wendell 061fd3ae36 Fixing bug in BlockManager timeout 2013-06-04 19:02:44 -04:00
Matei Zaharia f420d4f228 Merge pull request #639 from pwendell/timeout-update 
Bump akka and blockmanager timeouts to 60 seconds
 2013-06-04 15:25:58 -07:00
Patrick Wendell 8bd4e12104 Bump akka and blockmanager timeouts to 60 seconds 2013-06-04 18:14:24 -04:00
Konstantin Boudnik d1286231e0 Sometime Maven build runs out of PermGen space. 2013-06-03 15:55:44 -07:00
Shivaram Venkataraman 96943a1cc0 var to val 2013-06-03 12:29:38 -07:00
Shivaram Venkataraman cd347f547a Reuse the file object as it is valid after delete 2013-06-03 12:27:51 -07:00
Shivaram Venkataraman a058b0acf3 Delete a file for a block if it already exists. 2013-06-03 12:10:00 -07:00
Rohit Rai b104c7f5c7 Example to write the output to cassandra 2013-06-03 15:15:52 +05:30
Rohit Rai 56c64c4033 A better way to read column value if you are sure the column exists in every row. 2013-06-03 12:48:35 +05:30
Andrew xia 606bb1b450 Fix schedulingAlgorithm bugs for unit test 2013-06-03 10:29:23 +08:00
Gavin Li 4a9913d66a add ut for pipe enhancement 2013-06-02 23:21:09 +00:00
Rohit Rai 6d8423fd1b Adding deps to examples/pom.xml 
Fixing exclusion in examples deps in SparkBuild.scala
 2013-06-02 13:03:45 +05:30
Rohit Rai 81c2adc15c Removing infix call 2013-06-02 12:51:15 +05:30
Rohit Rai 3be7bdcefd Adding example to make Spark RDD from Cassandra 2013-06-01 19:32:17 +05:30
Shivaram Venkataraman 038cfc1a9a Make connect timeout configurable 2013-05-31 23:32:18 -07:00
Shivaram Venkataraman 91aca92249 Another round of Netty fixes. 
1. Avoid race condition between stop and copier completion
2. Handle socket exceptions by reporting them and filling in a failed
FetchResult
 2013-05-31 23:21:38 -07:00
Gavin Li 9f84315c05 enhance pipe to support what we can do in hadoop streaming 2013-06-01 00:26:10 +00:00
Matei Zaharia 84530ba6d9 Merge pull request #636 from rxin/unpersist 
Unpersist More block manager cleanup.
 2013-05-31 17:06:13 -07:00
Reynold Xin de1167bf2c Incorporated Charles' feedback to put rdd metadata removal in 
BlockManagerMasterActor.
 2013-05-31 15:54:57 -07:00
Reynold Xin ba5e544461 More block manager cleanup. 
Implemented a removeRdd method in BlockManager, and use that to
implement RDD.unpersist. Previously, unpersist needs to send B akka
messages, where B = number of blocks. Now unpersist only needs to send W
akka messages, where W = the number of workers.
 2013-05-31 01:48:16 -07:00
jerryshao 926f41cc52 fix block manager UI display issue when enable spark.cleaner.ttl 2013-05-31 09:32:52 +08:00
Reynold Xin f6ad3781b1 Fixed the flaky unpersist test in RDDSuite. 2013-05-30 16:28:08 -07:00
Reynold Xin bed1b08169 Do not create symlink for local add file. Instead, copy the file. 
This prevents Spark from changing the original file's permission, and
also allow add file to work on non-posix operating systems.
 2013-05-30 16:21:49 -07:00
Reynold Xin ef77bb73c6 Merge pull request #627 from shivaram/master 
Netty and shuffle  bug fixes
 2013-05-30 14:50:06 -07:00
Shivaram Venkataraman 3b0cd17343 Merge branch 'master' of git://github.com/mesos/spark 
Conflicts:
	core/src/test/scala/spark/ShuffleSuite.scala
 2013-05-30 14:36:24 -07:00
Reynold Xin 8cb817820f Merge pull request #628 from shivaram/zero-block-size 
Skip fetching zero-sized blocks in NIO.
 2013-05-30 14:17:44 -07:00
Andrew xia c3db3ea554 1. Add unit test for local scheduler 
2. Move localTaskSetManager to a new file
 2013-05-30 20:49:40 +08:00
Andrew xia ecceb101d3 implement FIFO and fair scheduler for spark local mode 2013-05-30 10:43:01 +08:00
Shivaram Venkataraman 19fd6d54c0 Also flush serializer in revertPartialWrites 2013-05-29 17:29:34 -07:00
Shivaram Venkataraman 618c8cae1e Skip fetching zero-sized blocks in OIO. 
Also unify splitLocalRemoteBlocks for netty/nio and add a test case
 2013-05-29 13:18:54 -07:00
Matei Zaharia 6ed71390d9 Merge pull request #626 from stephenh/remove-add-if-no-port 
Remove unused addIfNoPort.
 2013-05-29 10:14:22 -07:00
Shivaram Venkataraman b79b10a6d6 Flush serializer to fix zero-size kryo blocks bug. 
Also convert the local-cluster test case to check for non-zero block sizes
 2013-05-29 00:52:55 -07:00