Mosharaf Chowdhury
88264a33cf
Peers actually exchange information between themselves.
...
Don't exchange info about neighbors yet :(
2010-10-21 18:12:30 -07:00
Mosharaf Chowdhury
8e6ed77724
Put synchronized blocks around BitSets.
...
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
2010-10-21 17:58:42 -07:00
Mosharaf Chowdhury
22a47a9d16
PeerChatterController implemented. Now things should start working...
2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury
86b2824101
Formatting...
2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury
53bd64afe7
- Added TalkToPeer class which will allow peers to communicate between them.
...
- Still need the controller class that will decide which peers to communicate with
2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury
05bca235a7
Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT
2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury
e70755c2a2
- Implemented TalkToGuide class that talks to the Guide periodically
...
- Changed listOfSources from Vector to ListBuffer
2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury
b1c1e80026
- Added BitSet of pieces to SourceInfo.
...
- Streamlined code after removing GuideInfo.
- Minor refactoring.
2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury
b45dd8b8a4
Using SourceInfo instead of a separate GuideInfo case class.
2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury
365012f586
- Replaced pqOfSources with a vector: listOfSources
...
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
2010-10-19 17:02:55 -07:00
Matei Zaharia
787faf0d0e
Fixed a bug with scheduling of tasks that have no locality preferences.
...
These tasks were being subjected to delay scheduling but then counted as
having been launched on a preferred node. The solution is to have a
separate queue for them and treat them as preferred during scheduling.
2010-10-19 16:07:58 -07:00
Mosharaf Chowdhury
905745707c
Removed ReplicaID from SourceInfo.
2010-10-19 15:35:54 -07:00
Mosharaf Chowdhury
4ad6c5218e
Removed some commented sort/compare funtions from SourceInfo
2010-10-19 15:31:27 -07:00
Mosharaf Chowdhury
5deb23df96
Starting major changes to make it more like BitTorrent.
2010-10-19 15:28:36 -07:00
Matei Zaharia
0e0ec83570
Undid some changes that Mosharaf inadvertedly committed to master.
2010-10-19 13:58:52 -07:00
Mosharaf Chowdhury
744d49bc40
Receivers now send their hasBlocksBitVector to the sender.
2010-10-18 14:55:35 -07:00
Mosharaf Chowdhury
0791e20bfc
Added hasBlocksBitVector to keep track of which blocks a peer actually have.
2010-10-18 14:34:33 -07:00
Mosharaf Chowdhury
b02ad04560
Updated log4j.properties to print milliseconds.
2010-10-18 14:06:48 -07:00
Mosharaf Chowdhury
1af0cdbac7
Removed all SplitStream-related things.
2010-10-18 11:40:21 -07:00
Mosharaf Chowdhury
bf7055decf
Merge branch 'master' of git@github.com:mesos/spark
...
Conflicts:
src/scala/spark/SparkContext.scala
Using the latest one from Matei.
2010-10-18 11:08:45 -07:00
Matei Zaharia
b940164db3
Less hacky way of preventing config files from being overwritten when a template file changes
2010-10-16 22:01:05 -07:00
Matei Zaharia
e5fb280ec8
Changed the config files that were included in git to templates which
...
are used to create an initial copy of each config file if the user does
not have one. This way, users won't accidentally commit their changes to
config files to git.
2010-10-16 21:51:25 -07:00
Matei Zaharia
023ed194b4
Fixed some whitespace
2010-10-16 21:21:16 -07:00
Matei Zaharia
74bbfa91c2
Added support for generic Hadoop InputFormats and refactored textFile to
...
use this. Closes #12 .
2010-10-16 19:03:33 -07:00
Matei Zaharia
03238cb7c1
Renamed HdfsFile to HadoopFile
2010-10-16 17:25:09 -07:00
Matei Zaharia
0e2adecdab
Simplified UnionRDD slightly and added a SparkContext.union method for efficiently union-ing a large number of RDDs
2010-10-16 17:13:52 -07:00
Matei Zaharia
166d9f9125
Removed setSparkHome method on SparkContext in favor of having an
...
optional constructor parameter, so that the scheduler is guaranteed that
a Spark home has been set when it first builds its executor arg.
2010-10-16 16:19:47 -07:00
Matei Zaharia
1c082ad5fb
Added the ability to specify a list of JAR files when creating a
...
SparkContext and have the master node serve those to workers.
2010-10-16 16:14:13 -07:00
Matei Zaharia
c0b856a056
Set absolute path for SPARK_HOME
2010-10-16 12:18:02 -07:00
Matei Zaharia
7da569e8a5
Keep track of tasks in each job so that they can be removed when the job exits
2010-10-16 12:11:19 -07:00
Matei Zaharia
bf21bb28f3
Further clarified some code
2010-10-16 11:57:36 -07:00
Matei Zaharia
c21f840a80
Fixed some log messages
2010-10-16 10:40:42 -07:00
Matei Zaharia
dbdd7682eb
Bug fixes and improvements for MesosScheduler and SimpleJob
2010-10-16 10:38:56 -07:00
Matei Zaharia
a4953c5051
Moved Spark home detection to SparkContext and added a setSparkHome
...
method for setting it programatically.
2010-10-16 10:02:22 -07:00
Matei Zaharia
47b38fd207
Bug fix in passing env vars to executors
2010-10-16 09:21:43 -07:00
Matei Zaharia
6c1dee2e42
Added code so that Spark jobs can be launched from outside the Spark
...
directory by setting SPARK_HOME and locating the executor relative to
that. Entries on SPARK_CLASSPATH and SPARK_LIBRARY_PATH are also passed
along to worker nodes.
2010-10-15 19:42:26 -07:00
Matei Zaharia
ecb1af576e
Moved ClassServer out of repl packaged and renamed it to HttpServer.
2010-10-15 19:04:18 -07:00
Matei Zaharia
a768cf417b
Increased default memory for alltests
2010-10-15 16:17:43 -07:00
Matei Zaharia
aa8ccec315
Abort jobs if a task fails more than a limited number of times
2010-10-15 15:57:26 -07:00
Matei Zaharia
57a778426c
Updated guava to version r07
2010-10-15 15:55:58 -07:00
Matei Zaharia
31b5b8b4a6
A couple of improvements to ReplSuite:
...
- Use collect instead of toArray
- Disable the "running on Mesos" test when MESOS_HOME is not set
2010-10-15 15:37:14 -07:00
Matei Zaharia
28d6f23196
Made locality scheduling constant-time and added support for changing
...
CPU and memory requested per task.
2010-10-15 15:36:40 -07:00
Mosharaf Chowdhury
a4c0281902
sendObject now takes parameters instead of relying on class
...
variables.
2010-10-14 15:36:23 -07:00
Mosharaf Chowdhury
a137ca75da
Got rid pf dualMode.
2010-10-13 17:01:00 -07:00
Mosharaf Chowdhury
38194e5731
- Changed guidePort to GuideInfo that now contains the hostAddress
...
as well as the port. This will allow anyone other than the master
to be a guide.
- The GuideInfo object now contains the constants related to
tracker response.
2010-10-13 16:26:18 -07:00
Mosharaf Chowdhury
8690be8f5a
Cleared up some formatting.
...
Branching out from here to work on BT.
2010-10-13 11:40:03 -07:00
Mosharaf Chowdhury
0d67bc1cee
multi-tracker branch now compiles and runs; but it crashes right before the
...
end. The same problem is seen also in the master branch (in the
ChainedStreaming implementation)
2010-10-12 15:39:53 -07:00
Mosharaf Chowdhury
4fdd48295b
Added mesos.jar. Still not working. Major changes required.
2010-10-12 13:10:31 -07:00
Mosharaf Chowdhury
e73a5f3491
Now compiles with Scala 2.8.0, but doesn't run with nexus.jar
...
Must update it to use mesos.jar
2010-10-12 13:05:32 -07:00
Mosharaf Chowdhury
ad7a9c5a36
Minor cleanup in Broadcast.scala.
...
Changed BroadcastTest.scala to have multiple broadcasts.
2010-10-12 12:55:43 -07:00