Matei Zaharia
00c4d238dd
Merge pull request #518 from woggling/long-bm-sizes
...
Send block sizes as longs in BlockManager updates
2013-03-12 22:19:00 -07:00
Stephen Haberman
8f00d23598
Remove NextIterator.close default implementation.
2013-03-12 12:30:10 -05:00
Harold Lim
0b64e5f1ac
Removed some commented code
2013-03-12 13:31:27 +08:00
Harold Lim
f5b1fecb9f
Cleaned up the code
2013-03-12 13:31:27 +08:00
Harold Lim
b5325182a3
Updated/Refactored the Fair Task Scheduler. It does not inherit ClusterScheduler anymore. Rather, ClusterScheduler internally uses TaskSetQueuesManager that handles the scheduling of taskset queues. This is the class that should be extended to support other scheduling policies
2013-03-12 13:31:27 +08:00
Harold Lim
54ed7c4af4
Changed the name of the system property to set the allocation xml
2013-03-12 13:31:27 +08:00
Harold Lim
c07087364b
Made changes to the SparkContext to have a DynamicVariable for setting local properties that can be passed down the stack. Added an implementation of the fair scheduler
2013-03-12 13:31:27 +08:00
Stephen Haberman
0cf320485d
Forgot equals.
2013-03-12 00:05:35 -05:00
Stephen Haberman
9e68f48625
More quickly call close in HadoopRDD.
...
This also refactors out the common "gotNext" iterator pattern into
a shared utility class.
2013-03-11 23:59:17 -05:00
Charles Reiss
769d399674
Send block sizes as longs.
2013-03-11 14:17:05 -07:00
seanm
c1c3682c98
adding checkpoint dir to .gitignore
2013-03-11 12:40:44 -06:00
seanm
4aa1205202
adding typesafe repo to streaming resolvers so that akka-zeromq is found
2013-03-11 12:37:29 -06:00
Mark Hamstra
a6de80218a
Merge branch 'master' of https://github.com/mesos/spark into WithThing
2013-03-11 00:43:29 -07:00
Matei Zaharia
cbf8f0d4dd
Merge pull request #513 from MLnick/bagel-caching
...
Adds choice of persistence level to Bagel.
2013-03-11 00:23:57 -07:00
Nick Pentreath
8dd943fc3e
Fix doc style
2013-03-11 09:14:00 +02:00
Mark Hamstra
562893bea3
deleted excess curly braces
2013-03-10 22:43:08 -07:00
Imran Rashid
8a11ac3dc7
increase sleep time
2013-03-10 22:31:44 -07:00
Imran Rashid
9f97f2f9d8
add a small wait to one task to make sure some task runtime really is non-zero
2013-03-10 22:30:18 -07:00
Mark Hamstra
1289e7176b
refactored _With API and added foreachPartition
2013-03-10 22:27:13 -07:00
Mark Hamstra
b57df1f5e3
Merge branch 'master' of https://github.com/mesos/spark into WithThing
2013-03-10 16:56:31 -07:00
Matei Zaharia
2e1bbc4e7e
Merge remote-tracking branch 'woggling/dag-sched-driver-port'
...
Conflicts:
core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
2013-03-10 16:52:54 -07:00
Matei Zaharia
d4e29ea878
Update kryo-serializers version in pom.xml to match previous commit
2013-03-10 15:49:11 -07:00
Matei Zaharia
91a9d093bd
Merge pull request #512 from patelh/fix-kryo-serializer
...
Fix reference bug in Kryo serializer, add test, update version
2013-03-10 15:48:23 -07:00
Matei Zaharia
557cfd0f4d
Merge pull request #515 from woggling/deploy-app-death
...
Notify standalone deploy client of application death.
2013-03-10 15:44:57 -07:00
Matei Zaharia
a59cc6060f
Merge remote-tracking branch 'stephenh/nomocks'
...
Conflicts:
core/src/main/scala/spark/storage/BlockManagerMaster.scala
core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
2013-03-10 13:39:10 -07:00
Imran Rashid
20f01a0a1b
enable task metrics in local mode, add tests
2013-03-09 21:17:31 -08:00
Imran Rashid
ec30188a2a
rename remoteFetchWaitTime to fetchWaitTime, since it also includes time from local fetches
2013-03-09 21:16:53 -08:00
Charles Reiss
b0983c5762
Notify standalone deploy client of application death.
...
Usually, this isn't necessary since the application will be removed
as a result of the deploy client disconnecting, but occassionally, the
standalone deploy master removes an application otherwise.
Also mark applications as FAILED instead of FINISHED when they are
killed as a result of their executors failing too many times.
2013-03-09 11:29:45 -08:00
Charles Reiss
d0216cb38b
Prevent DAGSchedulerSuite from corrupting driver.port.
...
Use the LocalSparkContext abstraction to properly manage clearing
spark.driver.port.
2013-03-09 10:49:02 -08:00
Nick Pentreath
d35c5a5176
Adding test for non-default persistence level
2013-03-09 12:52:16 +02:00
Nick Pentreath
1e981d8b26
Added choice of persitance level to Bagel. Also added documentation.
2013-03-09 12:40:48 +02:00
Hiral Patel
664e5fd24b
Fix reference bug in Kryo serializer, add test, update version
2013-03-07 22:16:11 -08:00
Mark Hamstra
5ff0810b11
refactor mapWith, flatMapWith and filterWith to each use two parameter lists
2013-03-05 12:25:44 -08:00
Mark Hamstra
d046d8ad32
whitespace formatting
2013-03-05 00:48:13 -08:00
Mark Hamstra
9148b968cf
mapWith, flatMapWith and filterWith
2013-03-04 15:48:47 -08:00
Matei Zaharia
9f0dc829cb
Fix TaskMetrics not being serializable
2013-03-04 12:08:31 -08:00
Matei Zaharia
04fb81ffe5
Merge pull request #506 from rxin/spark-706
...
Fixed SPARK-706: Failures in block manager put leads to read task hanging.
2013-03-03 17:20:07 -08:00
Matei Zaharia
6cf4be4b39
Merge pull request #462 from squito/stageInfo
...
Track assorted metrics for each task, report summaries to user at stage completion
2013-03-03 17:16:22 -08:00
Imran Rashid
0bd1d00c2a
minor cleanup based on feedback in review request
2013-03-03 16:46:45 -08:00
Imran Rashid
f1006b99ff
change CleanupIterator to CompletionIterator
2013-03-03 16:39:05 -08:00
Imran Rashid
8fef5b9c5f
refactoring of TaskMetrics
2013-03-03 16:34:04 -08:00
Imran Rashid
d36abdb053
Merge branch 'master' into stageInfo
2013-03-03 15:20:46 -08:00
Matei Zaharia
6bfc7cad6b
Merge pull request #504 from mosharaf/master
...
Worker address was getting removed when removing an app.
2013-03-02 22:14:49 -08:00
Matei Zaharia
94b3db136d
Merge pull request #508 from markhamstra/TestServerInUse
...
Avoid bind failure in InputStreamsSuite
2013-03-02 22:13:52 -08:00
Mark Hamstra
b409073102
Instead of failing to bind to a fixed, already-in-use port, let the OS choose an available port for TestServer.
2013-03-01 15:05:07 -08:00
Matei Zaharia
25c71d3e12
Merge pull request #507 from markhamstra/poms271
...
bump version to 0.7.1-SNAPSHOT in the subproject poms
2013-03-01 08:00:18 -08:00
Mark Hamstra
8b06b359da
bump version to 0.7.1-SNAPSHOT in the subproject poms to keep the maven build building.
2013-02-28 23:34:34 -08:00
Reynold Xin
44134e12bb
Fixed SPARK-706: Failures in block manager put leads to read task
...
hanging.
2013-02-28 15:14:59 -08:00
Stephen Haberman
6415c2bb60
Don't create the Executor until we have everything it needs.
2013-02-28 12:38:09 -06:00
Stephen Haberman
80eecd2cb1
Make Executor fields volatile since they're read from the thread pool.
2013-02-28 10:41:07 -06:00