ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2458 commits 6 branches 181 tags 397 MiB
Commit graph

2458 commits

This branch
This branch
All branches
Author SHA1 Message Date
Andrew xia d1d9bdaabe Just update typo and comments 2013-03-23 07:25:30 +08:00
Andrew xia 5892393140 refactor fair scheduler implementation 
1.Chage "pool" properties to be the memeber of ActiveJob
2.Abstract the Schedulable of Pool and TaskSetManager
3.Abstract the FIFO and FS comparator algorithm
4.Miscellaneous changing of class define and construction
 2013-03-16 11:13:38 +08:00
Harold Lim 0b64e5f1ac Removed some commented code 2013-03-12 13:31:27 +08:00
Harold Lim f5b1fecb9f Cleaned up the code 2013-03-12 13:31:27 +08:00
Harold Lim b5325182a3 Updated/Refactored the Fair Task Scheduler. It does not inherit ClusterScheduler anymore. Rather, ClusterScheduler internally uses TaskSetQueuesManager that handles the scheduling of taskset queues. This is the class that should be extended to support other scheduling policies 2013-03-12 13:31:27 +08:00
Harold Lim 54ed7c4af4 Changed the name of the system property to set the allocation xml 2013-03-12 13:31:27 +08:00
Harold Lim c07087364b Made changes to the SparkContext to have a DynamicVariable for setting local properties that can be passed down the stack. Added an implementation of the fair scheduler 2013-03-12 13:31:27 +08:00
Matei Zaharia cbf8f0d4dd Merge pull request #513 from MLnick/bagel-caching 
Adds choice of persistence level to Bagel.
 2013-03-11 00:23:57 -07:00
Nick Pentreath 8dd943fc3e Fix doc style 2013-03-11 09:14:00 +02:00
Matei Zaharia 2e1bbc4e7e Merge remote-tracking branch 'woggling/dag-sched-driver-port' 
Conflicts:
	core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
 2013-03-10 16:52:54 -07:00
Matei Zaharia d4e29ea878 Update kryo-serializers version in pom.xml to match previous commit 2013-03-10 15:49:11 -07:00
Matei Zaharia 91a9d093bd Merge pull request #512 from patelh/fix-kryo-serializer 
Fix reference bug in Kryo serializer, add test, update version
 2013-03-10 15:48:23 -07:00
Matei Zaharia 557cfd0f4d Merge pull request #515 from woggling/deploy-app-death 
Notify standalone deploy client of application death.
 2013-03-10 15:44:57 -07:00
Matei Zaharia a59cc6060f Merge remote-tracking branch 'stephenh/nomocks' 
Conflicts:
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
 2013-03-10 13:39:10 -07:00
Charles Reiss b0983c5762 Notify standalone deploy client of application death. 
Usually, this isn't necessary since the application will be removed
as a result of the deploy client disconnecting, but occassionally, the
standalone deploy master removes an application otherwise.

Also mark applications as FAILED instead of FINISHED when they are
killed as a result of their executors failing too many times.
 2013-03-09 11:29:45 -08:00
Charles Reiss d0216cb38b Prevent DAGSchedulerSuite from corrupting driver.port. 
Use the LocalSparkContext abstraction to properly manage clearing
spark.driver.port.
 2013-03-09 10:49:02 -08:00
Nick Pentreath d35c5a5176 Adding test for non-default persistence level 2013-03-09 12:52:16 +02:00
Nick Pentreath 1e981d8b26 Added choice of persitance level to Bagel. Also added documentation. 2013-03-09 12:40:48 +02:00
Hiral Patel 664e5fd24b Fix reference bug in Kryo serializer, add test, update version 2013-03-07 22:16:11 -08:00
Matei Zaharia 9f0dc829cb Fix TaskMetrics not being serializable 2013-03-04 12:08:31 -08:00
Matei Zaharia 04fb81ffe5 Merge pull request #506 from rxin/spark-706 
Fixed SPARK-706: Failures in block manager put leads to read task hanging.
 2013-03-03 17:20:07 -08:00
Matei Zaharia 6cf4be4b39 Merge pull request #462 from squito/stageInfo 
Track assorted metrics for each task, report summaries to user at stage completion
 2013-03-03 17:16:22 -08:00
Imran Rashid 0bd1d00c2a minor cleanup based on feedback in review request 2013-03-03 16:46:45 -08:00
Imran Rashid f1006b99ff change CleanupIterator to CompletionIterator 2013-03-03 16:39:05 -08:00
Imran Rashid 8fef5b9c5f refactoring of TaskMetrics 2013-03-03 16:34:04 -08:00
Imran Rashid d36abdb053 Merge branch 'master' into stageInfo 2013-03-03 15:20:46 -08:00
Matei Zaharia 6bfc7cad6b Merge pull request #504 from mosharaf/master 
Worker address was getting removed when removing an app.
 2013-03-02 22:14:49 -08:00
Matei Zaharia 94b3db136d Merge pull request #508 from markhamstra/TestServerInUse 
Avoid bind failure in InputStreamsSuite
 2013-03-02 22:13:52 -08:00
Mark Hamstra b409073102 Instead of failing to bind to a fixed, already-in-use port, let the OS choose an available port for TestServer. 2013-03-01 15:05:07 -08:00
Matei Zaharia 25c71d3e12 Merge pull request #507 from markhamstra/poms271 
bump version to 0.7.1-SNAPSHOT in the subproject poms
 2013-03-01 08:00:18 -08:00
Mark Hamstra 8b06b359da bump version to 0.7.1-SNAPSHOT in the subproject poms to keep the maven build building. 2013-02-28 23:34:34 -08:00
Reynold Xin 44134e12bb Fixed SPARK-706: Failures in block manager put leads to read task 
hanging.
 2013-02-28 15:14:59 -08:00
Mosharaf Chowdhury 4ab387bcdb Fixed master datastructure updates after removing an application; and a typo. 2013-02-27 13:52:44 -08:00
Matei Zaharia db9b90fdbd Change version to 0.7.1-SNAPSHOT for development branch 2013-02-27 09:15:26 -08:00
Matei Zaharia 4f840f4e54 Added commented-out Google analytics code for website docs 2013-02-27 09:14:11 -08:00
Matei Zaharia baa30fcd99 Use new Spark EC2 scripts by default 2013-02-26 23:38:50 -08:00
Matei Zaharia fadeb1ddea More doc tweaks 2013-02-26 23:20:49 -08:00
Matei Zaharia 22334eafd9 Some tweaks to docs 2013-02-26 22:52:38 -08:00
Matei Zaharia 434a1ce773 Small hack to work around multiple JARs being built by sbt package 2013-02-26 12:24:18 -08:00
Matei Zaharia ece3edfffa Fix a problem with no hosts being counted as alive in the first job 2013-02-26 12:11:03 -08:00
Matei Zaharia 73697e2891 Fix overly large thread names in PySpark 2013-02-26 12:07:59 -08:00
Stephen Haberman db957e5bd7 Fix MapOutputTrackerSuite. 2013-02-26 01:38:50 -06:00
Matei Zaharia 9a046e30ac Switch docs to use Akka repo instead of Typesafe 2013-02-25 22:18:47 -08:00
Stephen Haberman a65aa549ff Override DAGScheduler.runLocally so we can remove the Thread.sleep. 2013-02-25 23:49:32 -06:00
Stephen Haberman a4adeb255c Merge branch 'master' into nomocks 
Conflicts:
	core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
 2013-02-25 23:48:52 -06:00
Matei Zaharia 7e67c626ee Change version number to 0.7.0 2013-02-25 20:30:47 -08:00
Matei Zaharia c6380b1749 Merge branch 'master' of https://github.com/mesos/spark 2013-02-25 20:23:41 -08:00
Matei Zaharia 108cfaa5b4 Fix Windows script for finding examples JAR 2013-02-25 20:23:36 -08:00
Matei Zaharia 4223be3aa4 Merge pull request #503 from pwendell/bug-fix 
createNewSparkContext should use sparkHome/jars/environment.
 2013-02-25 19:43:05 -08:00
Patrick Wendell 284ba90958 createNewSparkContext should use sparkHome/jars/environment. 
This fixes a bug introduced by Matei's recent change.
 2013-02-25 19:40:52 -08:00