ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3248 commits 6 branches 181 tags 397 MiB
Commit graph

429 commits

Author SHA1 Message Date
Prashant Sharma b59152a7c3 Changed to master version of the test, messed up during merge. 2013-07-15 12:09:17 +05:30
Prashant Sharma a3494d405d Merge branch 'master' of github.com:mesos/spark into scala-2.10 
Conflicts:
	core/src/main/scala/spark/Utils.scala
	core/src/test/scala/spark/ui/UISuite.scala
	project/SparkBuild.scala
	run
 2013-07-15 11:15:55 +05:30
Matei Zaharia 5a7835c152 Merge pull request #691 from karenfeng/logpaging 
Create log pages
 2013-07-12 20:28:21 -07:00
Karen Feng 73984b96a8 Removed unit test of nonexistent function Utils.lastNBytes 2013-07-12 14:26:56 -07:00
Prashant Sharma e86d5dbaad Merge branch 'master' into master-merge 
Conflicts:
	README.md
	core/pom.xml
	core/src/main/scala/spark/deploy/JsonProtocol.scala
	core/src/main/scala/spark/deploy/LocalSparkCluster.scala
	core/src/main/scala/spark/deploy/master/Master.scala
	core/src/main/scala/spark/deploy/master/MasterWebUI.scala
	core/src/main/scala/spark/deploy/worker/Worker.scala
	core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
	core/src/main/scala/spark/storage/BlockManagerUI.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	pom.xml
	project/SparkBuild.scala
	streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
 2013-07-12 14:49:16 +05:30
seanm a2c915fba8 giving order to top and making tests more clear 2013-07-11 18:55:00 -07:00
Karen Feng f5f3b272f8 Fixed mixup of start/end, moved more import files 2013-07-10 14:52:29 -07:00
Karen Feng dbe948d9a2 Moved appropriate import files from UISuite to UtilsSuite 2013-07-10 14:15:41 -07:00
Karen Feng 5f8a20b4a8 Moved unit tests for Utils from UISuite to UtilsSuite 2013-07-10 13:53:39 -07:00
Karen Feng 04263e4d46 Made some minor style changes 2013-07-10 13:15:42 -07:00
Karen Feng cfb6447ac4 Fixed for nonexistent bytes, added unit tests, changed stdout-page to stdout 2013-07-10 11:47:57 -07:00
seanm 24705d0f46 adding takeOrdered() to RDD 2013-07-10 10:33:11 -07:00
Shivaram Venkataraman d362d0f411 Ignore stderr when calling cat on a non-existing file 2013-07-07 04:09:46 -07:00
Shivaram Venkataraman 7d6d9e6ab2 Set DriverSuite log level to WARN 2013-07-07 04:09:15 -07:00
Shivaram Venkataraman a948f06725 Suppress log messages in sbt test with two changes: 
1. Set akka log level to ERROR before shutting down the actorSystem.
This avoids akka log messages (like Spray) from falling back to INFO
on the Stdout logger
2. Initialize netty to use SLF4J in LocalSparkContext. This ensures that
stack trace thrown during shutdown is handled by SLF4J instead of stdout
 2013-07-07 04:09:08 -07:00
Matei Zaharia 1ffadb2d9e Merge remote-tracking branch 'pwendell/ui-updates' 
Conflicts:
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	pom.xml
 2013-07-06 15:51:41 -07:00
Matei Zaharia 2a36e5449b Merge pull request #673 from xiajunluan/master 
Add config template file for fair scheduler feature
 2013-07-06 12:43:21 -07:00
Matei Zaharia 399bd65ef5 Fixed compile error due to merge 2013-07-05 11:27:06 -07:00
Matei Zaharia 652ea0f1d8 Allow RDD.takeSample to give samples bigger than the RDD 
Before, when withReplacement was set to true, we would not get a sample
bigger than the RDD's count().

Conflicts:
	core/src/main/scala/spark/RDD.scala
	core/src/test/scala/spark/RDDSuite.scala
 2013-07-05 11:15:13 -07:00
Andrew xia 6ccfb73ca9 Add fair scheduler config template file 2013-07-04 19:19:44 +08:00
Prashant Sharma a5f1f6a907 Merge branch 'master' into master-merge 
Conflicts:
	core/pom.xml
	core/src/main/scala/spark/MapOutputTracker.scala
	core/src/main/scala/spark/RDD.scala
	core/src/main/scala/spark/RDDCheckpointData.scala
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/Utils.scala
	core/src/main/scala/spark/api/python/PythonRDD.scala
	core/src/main/scala/spark/deploy/client/Client.scala
	core/src/main/scala/spark/deploy/master/MasterWebUI.scala
	core/src/main/scala/spark/deploy/worker/Worker.scala
	core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala
	core/src/main/scala/spark/rdd/BlockRDD.scala
	core/src/main/scala/spark/rdd/ZippedRDD.scala
	core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
	core/src/main/scala/spark/storage/BlockManager.scala
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/main/scala/spark/storage/BlockManagerMasterActor.scala
	core/src/main/scala/spark/storage/BlockManagerUI.scala
	core/src/main/scala/spark/util/AkkaUtils.scala
	core/src/test/scala/spark/SizeEstimatorSuite.scala
	pom.xml
	project/SparkBuild.scala
	repl/src/main/scala/spark/repl/SparkILoop.scala
	repl/src/test/scala/spark/repl/ReplSuite.scala
	streaming/src/main/scala/spark/streaming/StreamingContext.scala
	streaming/src/main/scala/spark/streaming/api/java/JavaStreamingContext.scala
	streaming/src/main/scala/spark/streaming/dstream/KafkaInputDStream.scala
	streaming/src/main/scala/spark/streaming/util/MasterFailureTest.scala
 2013-07-03 11:43:26 +05:30
Patrick Wendell 8ca1cc1786 Adding truncation for log files 2013-07-02 16:10:50 -07:00
Patrick Wendell 8688689387 Various formatting changes 2013-07-01 13:40:12 -07:00
Matei Zaharia 50ca17635a Merge pull request #664 from pwendell/test-fix 
Removing incorrect test statement
 2013-06-27 22:24:52 -07:00
Patrick Wendell c767e74370 Removing incorrect test statement 2013-06-27 21:48:58 -07:00
Patrick Wendell 362d996c81 Handful of changes based on matei's review 
- Avoid exception when no tasks have finished for a stage
- Adding DOCTYPE so css renders properly
- Adding progress slider
 2013-06-27 19:14:28 -07:00
Patrick Wendell 2cbaa0734b Making all new classes package private 2013-06-26 08:44:55 -07:00
Matei Zaharia 9f0d913295 Refactored tests to share SparkContexts in some of them 
Creating these seems to take a while and clutters the output with Akka
stuff, so it would be nice to share them.
 2013-06-25 19:18:30 -04:00
Matei Zaharia 15b00914c5 Some fixes to the launch-java-directly change: 
- Split SPARK_JAVA_OPTS into multiple command-line arguments if it
  contains spaces; this splitting follows quoting rules in bash
- Add the Scala JARs to the classpath if they're not in the CLASSPATH
  variable because the ExecutorRunner is launched with "scala" (this can
  happen when using local-cluster URLs in spark-shell)
 2013-06-25 17:17:27 -04:00
Matei Zaharia 7680ce0bd6 Fixed deprecated use of expect in SizeEstimatorSuite 2013-06-25 16:11:44 -04:00
Patrick Wendell d66bd6f885 Adding another unit test to Web UI suite 2013-06-24 17:12:55 -07:00
Patrick Wendell 42157027f2 A few bug fixes and a unit test 2013-06-24 16:25:05 -07:00
Matei Zaharia 78ffe164b3 Clone the zero value for each key in foldByKey 
The old version reused the object within each task, leading to
overwriting of the object when a mutable type is used, which is expected
to be common in fold.

Conflicts:

	core/src/test/scala/spark/ShuffleSuite.scala
 2013-06-23 10:26:53 -07:00
Matei Zaharia 3e61beff7b Merge pull request #648 from shivaram/netty-dbg 
Shuffle fixes and cleanup
 2013-06-22 16:22:47 -07:00
Matei Zaharia 1ef5d0d2c9 Merge pull request #644 from shimingfei/joblogger 
add Joblogger to Spark (on new Spark code)
 2013-06-22 09:35:57 -07:00
Mingfei 5240795154 edit according to comments 2013-06-21 17:38:23 +08:00
Matei Zaharia db42451a52 Merge pull request #643 from adatao/master 
Bug fix: Zero-length partitions result in NaN for overall mean & variance
 2013-06-17 15:26:36 -07:00
Matei Zaharia ec193c7d89 Merge remote-tracking branch 'xiajunluan/xiajunluan' 
Conflicts:
	core/src/main/scala/spark/scheduler/cluster/TaskSetManager.scala
 2013-06-18 00:11:50 +02:00
Gavin Li e6ae049283 Merge remote-tracking branch 'upstream1/master' into enhance_pipe 2013-06-16 22:53:39 +00:00
Gavin Li fb6d733fa8 update according to comments 2013-06-16 22:32:55 +00:00
Christopher Nguyen f91195cc15 Import just scala.math.abs rather than scala.math._ 2013-06-16 01:29:53 -07:00
Christopher Nguyen 5c886194e4 Move zero-length partition testing from JavaAPISuite.java to PartitioningSuite.scala 2013-06-16 01:23:48 -07:00
Christopher Nguyen 479442a9b9 Add zeroLengthPartitions() test to make sure, e.g., StatCounter.scala can handle empty partitions without incorrectly returning NaN 2013-06-15 17:35:55 -07:00
Matei Zaharia f961aac8b2 Merge pull request #649 from ryanlecompte/master 
Add top K method to RDD using a bounded priority queue
 2013-06-15 00:53:41 -07:00
Andrew xia 53add598f2 Update LocalSchedulerSuite to avoid using sleep for task launch 2013-06-15 01:46:13 +08:00
Shivaram Venkataraman 1d9f0df065 Fix some comments and style 2013-06-13 14:46:25 -07:00
Mingfei 967a6a699d modify sparklister function interface according to comments 2013-06-13 14:36:07 +08:00
Shivaram Venkataraman 5da4287b1d Merge branch 'netty-dbg' of github.com:shivaram/spark into netty-dbg 2013-06-12 16:38:37 -07:00
ryanlecompte db5bca08ff add a new top K method to RDD using a bounded priority queue 2013-06-12 10:54:16 -07:00
Patrick Wendell fd6148c8b2 Removing print statement 2013-06-10 10:27:25 -07:00