ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2860 commits 6 branches 181 tags 397 MiB
Commit graph

372 commits

Author SHA1 Message Date
Gavin Li e6ae049283 Merge remote-tracking branch 'upstream1/master' into enhance_pipe 2013-06-16 22:53:39 +00:00
Gavin Li fb6d733fa8 update according to comments 2013-06-16 22:32:55 +00:00
Matei Zaharia f961aac8b2 Merge pull request #649 from ryanlecompte/master 
Add top K method to RDD using a bounded priority queue
 2013-06-15 00:53:41 -07:00
ryanlecompte db5bca08ff add a new top K method to RDD using a bounded priority queue 2013-06-12 10:54:16 -07:00
Patrick Wendell fd6148c8b2 Removing print statement 2013-06-10 10:27:25 -07:00
Patrick Wendell ef14dc2e77 Adding Java-API version of compression codec 2013-06-09 18:09:46 -07:00
Patrick Wendell 083a3485ab Clean extra whitespace 2013-06-09 11:49:33 -07:00
Patrick Wendell d1bbcebae5 Adding compression to Hadoop save functions 2013-06-09 11:39:35 -07:00
Matei Zaharia 1ae60bcb36 Merge pull request #634 from xiajunluan/master 
[Spark-753] Fix ClusterSchedulSuite unit test failed
 2013-06-07 22:39:06 -07:00
Gavin Li e179ff8a32 update according to comments 2013-06-05 22:41:05 +00:00
Andrew xia 606bb1b450 Fix schedulingAlgorithm bugs for unit test 2013-06-03 10:29:23 +08:00
Gavin Li 4a9913d66a add ut for pipe enhancement 2013-06-02 23:21:09 +00:00
Reynold Xin ba5e544461 More block manager cleanup. 
Implemented a removeRdd method in BlockManager, and use that to
implement RDD.unpersist. Previously, unpersist needs to send B akka
messages, where B = number of blocks. Now unpersist only needs to send W
akka messages, where W = the number of workers.
 2013-05-31 01:48:16 -07:00
Reynold Xin f6ad3781b1 Fixed the flaky unpersist test in RDDSuite. 2013-05-30 16:28:08 -07:00
Shivaram Venkataraman 3b0cd17343 Merge branch 'master' of git://github.com/mesos/spark 
Conflicts:
	core/src/test/scala/spark/ShuffleSuite.scala
 2013-05-30 14:36:24 -07:00
Shivaram Venkataraman 618c8cae1e Skip fetching zero-sized blocks in OIO. 
Also unify splitLocalRemoteBlocks for netty/nio and add a test case
 2013-05-29 13:18:54 -07:00
Shivaram Venkataraman b79b10a6d6 Flush serializer to fix zero-size kryo blocks bug. 
Also convert the local-cluster test case to check for non-zero block sizes
 2013-05-29 00:52:55 -07:00
Shivaram Venkataraman fbc1ab3468 Couple of Netty fixes 
a. Fix the port number by reading it from the bound channel
b. Fix the shutdown sequence to make sure we actually block on the channel
c. Fix the unit test to use two JVMs.
 2013-05-28 16:27:16 -07:00
Matei Zaharia e8d4b6c296 Merge pull request #529 from xiajunluan/master 
[SPARK-663]Implement Fair Scheduler in Spark Cluster Scheduler
 2013-05-25 21:09:03 -07:00
Reynold Xin 6bbbe01287 Fixed a stupid mistake that NonJavaSerializableClass was made Java 
serializable.
 2013-05-24 16:51:45 -07:00
Reynold Xin 26962c9340 Automatically configure Netty port. This makes unit tests using 
local-cluster pass. Previously they were failing because Netty was
trying to bind to the same port for all processes.

Pair programmed with @shivaram.
 2013-05-24 16:39:33 -07:00
Reynold Xin 6ea085169d Fixed the bug that shuffle serializer is ignored by the new shuffle 
block iterators for local blocks. Also added a unit test for that.
 2013-05-24 14:08:37 -07:00
Charles Reiss 786c97b87c DistributedSuite: remove dead test code 2013-05-21 11:35:49 -07:00
Andrew xia ecd6d75c6a fix bug of unit tests 2013-05-21 06:49:23 +08:00
Reynold Xin 5912cc4967 Merge pull request #610 from JoshRosen/spark-747 
Throw exception if TaskResult exceeds Akka frame size
 2013-05-17 19:58:40 -07:00
Andrew xia 3d4672eaa9 Merge branch 'master' into xiajunluan 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/scheduler/cluster/ClusterScheduler.scala
	core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
 2013-05-18 07:28:03 +08:00
Andrew xia d19753b9c7 expose TaskSetManager type to resourceOffer function in ClusterScheduler 2013-05-18 06:45:19 +08:00
Andrew xia c6e2770bfe Fix ClusterScheduler bug to avoid allocating tasks to same slave 2013-05-17 05:10:38 +08:00
Josh Rosen b8e46b6074 Abort job if result exceeds Akka frame size; add test. 2013-05-16 01:57:57 -07:00
Reynold Xin f3491cb89b Merge branch 'master' of github.com:mesos/spark into shufflemerge 
Conflicts:
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/test/scala/spark/DistributedSuite.scala
	project/SparkBuild.scala
 2013-05-15 00:31:52 -07:00
Reynold Xin 81ad2fa331 Merge branch 'jdbc' of github.com:koeninger/spark 
Conflicts:
	project/SparkBuild.scala
 2013-05-14 23:12:00 -07:00
Cody Koeninger b16c4896f6 add test for JdbcRDD using embedded derby, per rxin suggestion 2013-05-14 23:44:04 -05:00
Matei Zaharia 016ac86830 Merge pull request #601 from rxin/emptyrdd-master 
EmptyRDD (master branch 0.8)
 2013-05-13 21:45:36 -07:00
Matei Zaharia 4b354e0a08 Merge pull request #589 from mridulm/master 
Add support for instance local scheduling
 2013-05-13 17:39:19 -07:00
Patrick Wendell a5c28bb888 Removing unnecessary map 2013-05-11 14:20:39 -07:00
Patrick Wendell 0345954530 SPARK-738: Spark should detect and squash nonserializable exceptions 2013-05-11 14:17:09 -07:00
Reynold Xin 9cafacf32d Added test for Netty suite. 2013-05-07 22:42:37 -07:00
Reynold Xin 64d4d2b036 Added tests for joins, cogroups, and unions for EmptyRDD. 2013-05-06 16:30:46 -07:00
Reynold Xin 0fd84965f6 Added EmptyRDD. 2013-05-06 15:40:34 -07:00
Reynold Xin 0a2bed356b Fixed flaky unpersist test in DistributedSuite. 2013-05-04 21:50:08 -07:00
Reynold Xin 62a077cd08 Merge branch 'unpersist-test' of github.com:shivaram/spark into blockmanager 2013-05-04 21:49:50 -07:00
Mridul Muralidharan 5b011d18d7 Merge from master 2013-05-04 20:41:27 +05:30
Shivaram Venkataraman 2274ad0786 Fix flaky test by changing catch and adding sleep 2013-05-03 16:35:35 -07:00
Shivaram Venkataraman bb8a434f9d Add zipPartitions to Java API. 2013-05-03 15:14:02 -07:00
Mridul Muralidharan ea2a6f91d3 pull from master 2013-05-04 00:35:59 +05:30
Reynold Xin 98df9d2853 Added removeRdd function in BlockManager. 2013-05-01 20:17:09 -07:00
Reynold Xin 3227ec8edd Cleaned up Ram's code. Moved SparkContext.remove to RDD.unpersist. 
Also updated unit tests to make sure they are properly testing for
concurrency.
 2013-05-01 16:07:44 -07:00
harshars 8481562731 Merged Ram's commit on removing RDDs. 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
 2013-05-01 14:42:17 -07:00
Mridul Muralidharan d960e7e0f8 a) Add support for hyper local scheduling - specific to a host + port - before trying host local scheduling. 
b) Add some fixes to test code to ensure it passes (and fixes some other issues).

c) Fix bug in task scheduling which incorrectly used availableCores instead of all cores on the node.
 2013-05-01 20:24:00 +05:30
Matei Zaharia f708dda81e Merge pull request #585 from pwendell/listener-perf 
[Fix SPARK-742] Task Metrics should not employ per-record timing by default
 2013-04-30 07:51:40 -07:00