ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2914 commits 6 branches 181 tags 397 MiB
Commit graph

1484 commits

Author SHA1 Message Date
Matei Zaharia 1ef5d0d2c9 Merge pull request #644 from shimingfei/joblogger 
add Joblogger to Spark (on new Spark code)
 2013-06-22 09:35:57 -07:00
Jey Kottalam 1ba3c17303 use parens when calling method with side-effects 2013-06-21 12:14:16 -04:00
Jey Kottalam edb18ca928 Rename PythonWorker to PythonWorkerFactory 2013-06-21 12:14:16 -04:00
Jey Kottalam 62c4781400 Add tests and fixes for Python daemon shutdown 2013-06-21 12:14:16 -04:00
Jey Kottalam c79a6078c3 Prefork Python worker processes 2013-06-21 12:14:16 -04:00
Jey Kottalam 40afe0d2a5 Add Python timing instrumentation 2013-06-21 12:14:16 -04:00
Mingfei 2fc794a6c7 small modify in DAGScheduler 2013-06-21 18:21:35 +08:00
Mingfei 4b9862ac9c small format modification 2013-06-21 17:55:32 +08:00
Mingfei aa7aa587be some format modification 2013-06-21 17:48:41 +08:00
Mingfei 5240795154 edit according to comments 2013-06-21 17:38:23 +08:00
Matei Zaharia 71030ba3eb Merge pull request #654 from lyogavin/enhance_pipe 
fix typo and coding style in #638
 2013-06-19 15:21:03 -07:00
Matei Zaharia 7902baddc7 Update ASM to version 4.0 2013-06-19 13:34:30 +02:00
Gavin Li 0a2a9bce1e fix typo and coding style 2013-06-18 21:30:13 +00:00
jerryshao 1e9269c3ee reduce ZippedPartitionsRDD's getPreferredLocations complexity 2013-06-18 09:49:06 +08:00
Matei Zaharia db42451a52 Merge pull request #643 from adatao/master 
Bug fix: Zero-length partitions result in NaN for overall mean & variance
 2013-06-17 15:26:36 -07:00
Matei Zaharia e82a2ffcc9 Merge pull request #653 from rxin/logging 
SPARK-781: Log the temp directory path when Spark says "Failed to create temp directory."
 2013-06-17 15:13:15 -07:00
Matei Zaharia ec193c7d89 Merge remote-tracking branch 'xiajunluan/xiajunluan' 
Conflicts:
	core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
 2013-06-18 00:11:50 +02:00
Reynold Xin be3c406edf Fixed the typo pointed out by Matei. 2013-06-17 17:07:51 -04:00
Reynold Xin 1450296797 SPARK-781: Log the temp directory path when Spark says "Failed to create 
temp directory".
 2013-06-17 16:58:23 -04:00
Gavin Li 4508089fc3 refine comments and add sc.clean 2013-06-17 05:23:46 +00:00
Gavin Li e6ae049283 Merge remote-tracking branch 'upstream1/master' into enhance_pipe 2013-06-16 22:53:39 +00:00
Gavin Li fb6d733fa8 update according to comments 2013-06-16 22:32:55 +00:00
Christopher Nguyen f91195cc15 Import just scala.math.abs rather than scala.math._ 2013-06-16 01:29:53 -07:00
Christopher Nguyen 5c886194e4 Move zero-length partition testing from JavaAPISuite.java to PartitioningSuite.scala 2013-06-16 01:23:48 -07:00
Christopher Nguyen 479442a9b9 Add zeroLengthPartitions() test to make sure, e.g., StatCounter.scala can handle empty partitions without incorrectly returning NaN 2013-06-15 17:35:55 -07:00
Matei Zaharia f961aac8b2 Merge pull request #649 from ryanlecompte/master 
Add top K method to RDD using a bounded priority queue
 2013-06-15 00:53:41 -07:00
ryanlecompte e8801d4490 use delegation for BoundedPriorityQueue, add Java API 2013-06-14 23:39:05 -07:00
Andrew xia 53add598f2 Update LocalSchedulerSuite to avoid using sleep for task launch 2013-06-15 01:46:13 +08:00
Reynold Xin 2cc188fd54 SPARK-774: cogroup should also disable map side combine by default 2013-06-14 00:10:54 -07:00
Reynold Xin 6738178d0d SPARK-772: groupByKey should disable map side combine. 2013-06-13 23:59:42 -07:00
ryanlecompte 93b3f5e535 drop unneeded ClassManifest implicit 2013-06-13 16:26:35 -07:00
ryanlecompte 44b8dbaede use Iterator.single(elem) instead of Iterator(elem) for improved performance based on scaladocs 2013-06-13 16:23:15 -07:00
Mingfei 967a6a699d modify sparklister function interface according to comments 2013-06-13 14:36:07 +08:00
ryanlecompte db5bca08ff add a new top K method to RDD using a bounded priority queue 2013-06-12 10:54:16 -07:00
Patrick Wendell fd6148c8b2 Removing print statement 2013-06-10 10:27:25 -07:00
Andrew xia 190ec61799 change code style and debug info 2013-06-10 15:27:02 +08:00
Patrick Wendell ef14dc2e77 Adding Java-API version of compression codec 2013-06-09 18:09:46 -07:00
Patrick Wendell df592192e7 Monads FTW 2013-06-09 18:09:24 -07:00
Patrick Wendell 083a3485ab Clean extra whitespace 2013-06-09 11:49:33 -07:00
Patrick Wendell d1bbcebae5 Adding compression to Hadoop save functions 2013-06-09 11:39:35 -07:00
Mingfei ade822011d not check return value of eventQueue.take 2013-06-08 16:26:45 +08:00
Mingfei 4fd86e0e10 delete test code for joblogger in SparkContext 2013-06-08 15:45:47 +08:00
Mingfei 362f0f93ac Merge branch 'master' of https://github.com/mesos/spark 2013-06-08 15:20:13 +08:00
Mingfei 1a4d93c025 modify to pass job annotation by localProperties and use daeamon thread to do joblogger's work 2013-06-08 14:23:39 +08:00
Matei Zaharia b58a29295b Small formatting and style fixes 2013-06-07 22:51:28 -07:00
Matei Zaharia c8fc423bc2 Merge pull request #631 from jerryshao/master 
Fix block manager UI display issue when enable spark.cleaner.ttl
 2013-06-07 22:43:18 -07:00
Matei Zaharia c9ca0a4a58 Small code style fix to SchedulingAlgorithm.scala 2013-06-07 22:40:44 -07:00
Matei Zaharia 1ae60bcb36 Merge pull request #634 from xiajunluan/master 
[Spark-753] Fix ClusterSchedulSuite unit test failed
 2013-06-07 22:39:06 -07:00
Gavin Li e179ff8a32 update according to comments 2013-06-05 22:41:05 +00:00
Christopher Nguyen 9d35904357 In the current code, when both partitions happen to have zero-length, the return mean will be NaN. 
Consequently, the result of mean after reducing over all partitions will also be NaN,
which is not correct if there are partitions with non-zero length. This patch fixes this issue.
 2013-06-04 22:12:47 -07:00