ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
7933 commits 6 branches 181 tags 397 MiB
Commit graph

7933 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury 22a47a9d16 PeerChatterController implemented. Now things should start working... 2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury 86b2824101 Formatting... 2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury 53bd64afe7 - Added TalkToPeer class which will allow peers to communicate between them. 
- Still need the controller class that will decide which peers to communicate with
 2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury 05bca235a7 Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT 2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury e70755c2a2 - Implemented TalkToGuide class that talks to the Guide periodically 
- Changed listOfSources from Vector to ListBuffer
 2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury b1c1e80026 - Added BitSet of pieces to SourceInfo. 
- Streamlined code after removing GuideInfo.
- Minor refactoring.
 2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury b45dd8b8a4 Using SourceInfo instead of a separate GuideInfo case class. 2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury 365012f586 - Replaced pqOfSources with a vector: listOfSources 
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
 2010-10-19 17:02:55 -07:00
Matei Zaharia 787faf0d0e Fixed a bug with scheduling of tasks that have no locality preferences. 
These tasks were being subjected to delay scheduling but then counted as
having been launched on a preferred node. The solution is to have a
separate queue for them and treat them as preferred during scheduling.
 2010-10-19 16:07:58 -07:00
Mosharaf Chowdhury 905745707c Removed ReplicaID from SourceInfo. 2010-10-19 15:35:54 -07:00
Mosharaf Chowdhury 4ad6c5218e Removed some commented sort/compare funtions from SourceInfo 2010-10-19 15:31:27 -07:00
Mosharaf Chowdhury 5deb23df96 Starting major changes to make it more like BitTorrent. 2010-10-19 15:28:36 -07:00
Matei Zaharia 0e0ec83570 Undid some changes that Mosharaf inadvertedly committed to master. 2010-10-19 13:58:52 -07:00
Mosharaf Chowdhury 744d49bc40 Receivers now send their hasBlocksBitVector to the sender. 2010-10-18 14:55:35 -07:00
Mosharaf Chowdhury 0791e20bfc Added hasBlocksBitVector to keep track of which blocks a peer actually have. 2010-10-18 14:34:33 -07:00
Mosharaf Chowdhury b02ad04560 Updated log4j.properties to print milliseconds. 2010-10-18 14:06:48 -07:00
Mosharaf Chowdhury 1af0cdbac7 Removed all SplitStream-related things. 2010-10-18 11:40:21 -07:00
Mosharaf Chowdhury bf7055decf Merge branch 'master' of git@github.com:mesos/spark 
Conflicts:
	src/scala/spark/SparkContext.scala

Using the latest one from Matei.
 2010-10-18 11:08:45 -07:00
Matei Zaharia b940164db3 Less hacky way of preventing config files from being overwritten when a template file changes 2010-10-16 22:01:05 -07:00
Matei Zaharia e5fb280ec8 Changed the config files that were included in git to templates which 
are used to create an initial copy of each config file if the user does
not have one. This way, users won't accidentally commit their changes to
config files to git.
 2010-10-16 21:51:25 -07:00
Matei Zaharia 023ed194b4 Fixed some whitespace 2010-10-16 21:21:16 -07:00
Matei Zaharia 74bbfa91c2 Added support for generic Hadoop InputFormats and refactored textFile to 
use this. Closes #12.
 2010-10-16 19:03:33 -07:00
Matei Zaharia 03238cb7c1 Renamed HdfsFile to HadoopFile 2010-10-16 17:25:09 -07:00
Matei Zaharia 0e2adecdab Simplified UnionRDD slightly and added a SparkContext.union method for efficiently union-ing a large number of RDDs 2010-10-16 17:13:52 -07:00
Matei Zaharia 166d9f9125 Removed setSparkHome method on SparkContext in favor of having an 
optional constructor parameter, so that the scheduler is guaranteed that
a Spark home has been set when it first builds its executor arg.
 2010-10-16 16:19:47 -07:00
Matei Zaharia 1c082ad5fb Added the ability to specify a list of JAR files when creating a 
SparkContext and have the master node serve those to workers.
 2010-10-16 16:14:13 -07:00
Matei Zaharia c0b856a056 Set absolute path for SPARK_HOME 2010-10-16 12:18:02 -07:00
Matei Zaharia 7da569e8a5 Keep track of tasks in each job so that they can be removed when the job exits 2010-10-16 12:11:19 -07:00
Matei Zaharia bf21bb28f3 Further clarified some code 2010-10-16 11:57:36 -07:00
Matei Zaharia c21f840a80 Fixed some log messages 2010-10-16 10:40:42 -07:00
Matei Zaharia dbdd7682eb Bug fixes and improvements for MesosScheduler and SimpleJob 2010-10-16 10:38:56 -07:00
Matei Zaharia a4953c5051 Moved Spark home detection to SparkContext and added a setSparkHome 
method for setting it programatically.
 2010-10-16 10:02:22 -07:00
Matei Zaharia 47b38fd207 Bug fix in passing env vars to executors 2010-10-16 09:21:43 -07:00
Matei Zaharia 6c1dee2e42 Added code so that Spark jobs can be launched from outside the Spark 
directory by setting SPARK_HOME and locating the executor relative to
that. Entries on SPARK_CLASSPATH and SPARK_LIBRARY_PATH are also passed
along to worker nodes.
 2010-10-15 19:42:26 -07:00
Matei Zaharia ecb1af576e Moved ClassServer out of repl packaged and renamed it to HttpServer. 2010-10-15 19:04:18 -07:00
Matei Zaharia a768cf417b Increased default memory for alltests 2010-10-15 16:17:43 -07:00
Matei Zaharia aa8ccec315 Abort jobs if a task fails more than a limited number of times 2010-10-15 15:57:26 -07:00
Matei Zaharia 57a778426c Updated guava to version r07 2010-10-15 15:55:58 -07:00
Matei Zaharia 31b5b8b4a6 A couple of improvements to ReplSuite: 
- Use collect instead of toArray
- Disable the "running on Mesos" test when MESOS_HOME is not set
 2010-10-15 15:37:14 -07:00
Matei Zaharia 28d6f23196 Made locality scheduling constant-time and added support for changing 
CPU and memory requested per task.
 2010-10-15 15:36:40 -07:00
Mosharaf Chowdhury a4c0281902 sendObject now takes parameters instead of relying on class 
variables.
 2010-10-14 15:36:23 -07:00
Mosharaf Chowdhury a137ca75da Got rid pf dualMode. 2010-10-13 17:01:00 -07:00
Mosharaf Chowdhury 38194e5731 - Changed guidePort to GuideInfo that now contains the hostAddress 
as well as the port. This will allow anyone other than the master
to be a guide.
- The GuideInfo object now contains the constants related to
tracker response.
 2010-10-13 16:26:18 -07:00
Mosharaf Chowdhury 8690be8f5a Cleared up some formatting. 
Branching out from here to work on BT.
 2010-10-13 11:40:03 -07:00
Mosharaf Chowdhury 0d67bc1cee multi-tracker branch now compiles and runs; but it crashes right before the 
end. The same problem is seen also in the master branch (in the
ChainedStreaming implementation)
 2010-10-12 15:39:53 -07:00
Mosharaf Chowdhury 4fdd48295b Added mesos.jar. Still not working. Major changes required. 2010-10-12 13:10:31 -07:00
Mosharaf Chowdhury e73a5f3491 Now compiles with Scala 2.8.0, but doesn't run with nexus.jar 
Must update it to use mesos.jar
 2010-10-12 13:05:32 -07:00
Mosharaf Chowdhury ad7a9c5a36 Minor cleanup in Broadcast.scala. 
Changed BroadcastTest.scala to have multiple broadcasts.
 2010-10-12 12:55:43 -07:00
Matei Zaharia a9098ad5d4 Moved Job and SimpleJob to new files 2010-10-07 18:27:26 -07:00
Matei Zaharia a5155206a1 Merge branch 'master' into matei-scheduling 2010-10-07 17:18:32 -07:00