ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2876 commits 6 branches 181 tags 397 MiB
Commit graph

2876 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 00c4d238dd Merge pull request #518 from woggling/long-bm-sizes 
Send block sizes as longs in BlockManager updates
 2013-03-12 22:19:00 -07:00
Stephen Haberman 8f00d23598 Remove NextIterator.close default implementation. 2013-03-12 12:30:10 -05:00
Harold Lim 0b64e5f1ac Removed some commented code 2013-03-12 13:31:27 +08:00
Harold Lim f5b1fecb9f Cleaned up the code 2013-03-12 13:31:27 +08:00
Harold Lim b5325182a3 Updated/Refactored the Fair Task Scheduler. It does not inherit ClusterScheduler anymore. Rather, ClusterScheduler internally uses TaskSetQueuesManager that handles the scheduling of taskset queues. This is the class that should be extended to support other scheduling policies 2013-03-12 13:31:27 +08:00
Harold Lim 54ed7c4af4 Changed the name of the system property to set the allocation xml 2013-03-12 13:31:27 +08:00
Harold Lim c07087364b Made changes to the SparkContext to have a DynamicVariable for setting local properties that can be passed down the stack. Added an implementation of the fair scheduler 2013-03-12 13:31:27 +08:00
Stephen Haberman 0cf320485d Forgot equals. 2013-03-12 00:05:35 -05:00
Stephen Haberman 9e68f48625 More quickly call close in HadoopRDD. 
This also refactors out the common "gotNext" iterator pattern into
a shared utility class.
 2013-03-11 23:59:17 -05:00
Charles Reiss 769d399674 Send block sizes as longs. 2013-03-11 14:17:05 -07:00
Mark Hamstra a6de80218a Merge branch 'master' of https://github.com/mesos/spark into WithThing 2013-03-11 00:43:29 -07:00
Matei Zaharia cbf8f0d4dd Merge pull request #513 from MLnick/bagel-caching 
Adds choice of persistence level to Bagel.
 2013-03-11 00:23:57 -07:00
Nick Pentreath 8dd943fc3e Fix doc style 2013-03-11 09:14:00 +02:00
Mark Hamstra 562893bea3 deleted excess curly braces 2013-03-10 22:43:08 -07:00
Imran Rashid 8a11ac3dc7 increase sleep time 2013-03-10 22:31:44 -07:00
Imran Rashid 9f97f2f9d8 add a small wait to one task to make sure some task runtime really is non-zero 2013-03-10 22:30:18 -07:00
Mark Hamstra 1289e7176b refactored _With API and added foreachPartition 2013-03-10 22:27:13 -07:00
Mark Hamstra b57df1f5e3 Merge branch 'master' of https://github.com/mesos/spark into WithThing 2013-03-10 16:56:31 -07:00
Matei Zaharia 2e1bbc4e7e Merge remote-tracking branch 'woggling/dag-sched-driver-port' 
Conflicts:
	core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
 2013-03-10 16:52:54 -07:00
Matei Zaharia d4e29ea878 Update kryo-serializers version in pom.xml to match previous commit 2013-03-10 15:49:11 -07:00
Matei Zaharia 91a9d093bd Merge pull request #512 from patelh/fix-kryo-serializer 
Fix reference bug in Kryo serializer, add test, update version
 2013-03-10 15:48:23 -07:00
Matei Zaharia 557cfd0f4d Merge pull request #515 from woggling/deploy-app-death 
Notify standalone deploy client of application death.
 2013-03-10 15:44:57 -07:00
Matei Zaharia a59cc6060f Merge remote-tracking branch 'stephenh/nomocks' 
Conflicts:
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
 2013-03-10 13:39:10 -07:00
Imran Rashid 20f01a0a1b enable task metrics in local mode, add tests 2013-03-09 21:17:31 -08:00
Imran Rashid ec30188a2a rename remoteFetchWaitTime to fetchWaitTime, since it also includes time from local fetches 2013-03-09 21:16:53 -08:00
Charles Reiss b0983c5762 Notify standalone deploy client of application death. 
Usually, this isn't necessary since the application will be removed
as a result of the deploy client disconnecting, but occassionally, the
standalone deploy master removes an application otherwise.

Also mark applications as FAILED instead of FINISHED when they are
killed as a result of their executors failing too many times.
 2013-03-09 11:29:45 -08:00
Charles Reiss d0216cb38b Prevent DAGSchedulerSuite from corrupting driver.port. 
Use the LocalSparkContext abstraction to properly manage clearing
spark.driver.port.
 2013-03-09 10:49:02 -08:00
Nick Pentreath d35c5a5176 Adding test for non-default persistence level 2013-03-09 12:52:16 +02:00
Nick Pentreath 1e981d8b26 Added choice of persitance level to Bagel. Also added documentation. 2013-03-09 12:40:48 +02:00
Hiral Patel 664e5fd24b Fix reference bug in Kryo serializer, add test, update version 2013-03-07 22:16:11 -08:00
Mark Hamstra 5ff0810b11 refactor mapWith, flatMapWith and filterWith to each use two parameter lists 2013-03-05 12:25:44 -08:00
Mark Hamstra d046d8ad32 whitespace formatting 2013-03-05 00:48:13 -08:00
Mark Hamstra 9148b968cf mapWith, flatMapWith and filterWith 2013-03-04 15:48:47 -08:00
Matei Zaharia 9f0dc829cb Fix TaskMetrics not being serializable 2013-03-04 12:08:31 -08:00
Matei Zaharia 04fb81ffe5 Merge pull request #506 from rxin/spark-706 
Fixed SPARK-706: Failures in block manager put leads to read task hanging.
 2013-03-03 17:20:07 -08:00
Matei Zaharia 6cf4be4b39 Merge pull request #462 from squito/stageInfo 
Track assorted metrics for each task, report summaries to user at stage completion
 2013-03-03 17:16:22 -08:00
Imran Rashid 0bd1d00c2a minor cleanup based on feedback in review request 2013-03-03 16:46:45 -08:00
Imran Rashid f1006b99ff change CleanupIterator to CompletionIterator 2013-03-03 16:39:05 -08:00
Imran Rashid 8fef5b9c5f refactoring of TaskMetrics 2013-03-03 16:34:04 -08:00
Imran Rashid d36abdb053 Merge branch 'master' into stageInfo 2013-03-03 15:20:46 -08:00
Matei Zaharia 6bfc7cad6b Merge pull request #504 from mosharaf/master 
Worker address was getting removed when removing an app.
 2013-03-02 22:14:49 -08:00
Matei Zaharia 94b3db136d Merge pull request #508 from markhamstra/TestServerInUse 
Avoid bind failure in InputStreamsSuite
 2013-03-02 22:13:52 -08:00
Mark Hamstra b409073102 Instead of failing to bind to a fixed, already-in-use port, let the OS choose an available port for TestServer. 2013-03-01 15:05:07 -08:00
Matei Zaharia 25c71d3e12 Merge pull request #507 from markhamstra/poms271 
bump version to 0.7.1-SNAPSHOT in the subproject poms
 2013-03-01 08:00:18 -08:00
Mark Hamstra 8b06b359da bump version to 0.7.1-SNAPSHOT in the subproject poms to keep the maven build building. 2013-02-28 23:34:34 -08:00
Reynold Xin 44134e12bb Fixed SPARK-706: Failures in block manager put leads to read task 
hanging.
 2013-02-28 15:14:59 -08:00
Stephen Haberman 6415c2bb60 Don't create the Executor until we have everything it needs. 2013-02-28 12:38:09 -06:00
Stephen Haberman 80eecd2cb1 Make Executor fields volatile since they're read from the thread pool. 2013-02-28 10:41:07 -06:00
Mosharaf Chowdhury 4ab387bcdb Fixed master datastructure updates after removing an application; and a typo. 2013-02-27 13:52:44 -08:00
Matei Zaharia db9b90fdbd Change version to 0.7.1-SNAPSHOT for development branch 2013-02-27 09:15:26 -08:00