Mosharaf Chowdhury
878d157ce3
Graceful shutdown after a single transmission in the swarm is over.
...
There might still be a problem with the Tracker shutdown. It must be done explicitly by SparkContext.
2010-11-04 22:09:14 -07:00
Mosharaf Chowdhury
10fc66b1c4
Print reception messages for duplicate blocks as well...
2010-11-01 14:00:40 -07:00
Mosharaf Chowdhury
03d73dea9f
Refactoring...
2010-10-28 15:09:58 -07:00
Mosharaf Chowdhury
681dba1373
Removed some deprecated speed related code.
2010-10-28 14:40:49 -07:00
Mosharaf Chowdhury
bcc7d38ace
Added SpeedTracker to keep track of reception time per block from each source a peer has ever talked to.
2010-10-28 14:37:59 -07:00
Mosharaf Chowdhury
44eb79f64a
- Measuring approximate time to receive a block from the sending peer.
...
- Added some Exception handling/suppression code.
2010-10-27 17:41:37 -07:00
Mosharaf Chowdhury
ae2c93a54b
Fixed some configuration bugs
2010-10-27 16:18:26 -07:00
Mosharaf Chowdhury
e4dc7847ba
- Receivers now send back the latest hasBlocksBitVector after every reception
...
- Will probably settle for the push model. Still not removing the pull related codes
- Refactoring...
2010-10-27 15:36:58 -07:00
Mosharaf Chowdhury
059b6ae475
Added data structures to turn transmission to a pull model from a push one.
2010-10-27 15:05:29 -07:00
Mosharaf Chowdhury
f7364111bb
TalkToGuide breaks once the peer has all the blocks.
2010-10-26 21:26:45 -07:00
Mosharaf Chowdhury
8587353b3e
Added some tuning parameters.
2010-10-26 21:16:37 -07:00
Mosharaf Chowdhury
d70f3549ff
MaxTxPeers and MaxRxPeers set upper limit on concurrent sending and receiving threads for every peer.
...
Relevant timeout code is also in. There is still an exception to take care of.
2010-10-26 17:53:03 -07:00
Mosharaf Chowdhury
1877f1c38d
No one is waiting on hasBlocksLock. Removed...
2010-10-26 16:18:13 -07:00
Mosharaf Chowdhury
bc1c51817c
Beautified code...
2010-10-26 16:08:22 -07:00
Mosharaf Chowdhury
d673be0e06
Minor changes. Still need to run on Mesos+EC2
2010-10-26 15:07:03 -07:00
Mosharaf Chowdhury
3f96c9b7e8
Resolved some bugs. Apparently, objects deep inside other objects could be passed as references. Bad Scala!
2010-10-26 12:56:27 -07:00
Mosharaf Chowdhury
e659efca88
Added a synchronized block around numCopiesSent manipulation.
2010-10-26 08:52:56 -07:00
Mosharaf Chowdhury
dba92f7dbe
- Fixed bugs with storing received blocks in appropriate indices.
...
- receiveBroadcast now returns, but still no good way to stop threads from running.
2010-10-25 17:34:25 -07:00
Mosharaf Chowdhury
96797cc997
Replaced nested variable.synchronized blocks with a single top level synchronized { }
2010-10-25 14:53:56 -07:00
Mosharaf Chowdhury
2099aadbbd
Implemented rarest first policy. This isn't working as expected (i.e., not distributing load as expected), probably due to some issues with how we pick peers to talk to.
2010-10-25 14:48:11 -07:00
Mosharaf Chowdhury
7a7123b525
Running in local Mesos. Multiple things have to be fixed though. Go through the TODOs...
2010-10-22 23:43:11 -07:00
Mosharaf Chowdhury
27362c6523
Fixed ordering of function calls and thread creations.
...
Changed Slaves to Workers :)
2010-10-22 12:03:40 -07:00
Mosharaf Chowdhury
88264a33cf
Peers actually exchange information between themselves.
...
Don't exchange info about neighbors yet :(
2010-10-21 18:12:30 -07:00
Mosharaf Chowdhury
8e6ed77724
Put synchronized blocks around BitSets.
...
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
2010-10-21 17:58:42 -07:00
Mosharaf Chowdhury
22a47a9d16
PeerChatterController implemented. Now things should start working...
2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury
86b2824101
Formatting...
2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury
53bd64afe7
- Added TalkToPeer class which will allow peers to communicate between them.
...
- Still need the controller class that will decide which peers to communicate with
2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury
05bca235a7
Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT
2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury
e70755c2a2
- Implemented TalkToGuide class that talks to the Guide periodically
...
- Changed listOfSources from Vector to ListBuffer
2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury
b1c1e80026
- Added BitSet of pieces to SourceInfo.
...
- Streamlined code after removing GuideInfo.
- Minor refactoring.
2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury
b45dd8b8a4
Using SourceInfo instead of a separate GuideInfo case class.
2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury
365012f586
- Replaced pqOfSources with a vector: listOfSources
...
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
2010-10-19 17:02:55 -07:00
Mosharaf Chowdhury
905745707c
Removed ReplicaID from SourceInfo.
2010-10-19 15:35:54 -07:00
Mosharaf Chowdhury
4ad6c5218e
Removed some commented sort/compare funtions from SourceInfo
2010-10-19 15:31:27 -07:00
Mosharaf Chowdhury
5deb23df96
Starting major changes to make it more like BitTorrent.
2010-10-19 15:28:36 -07:00
Mosharaf Chowdhury
744d49bc40
Receivers now send their hasBlocksBitVector to the sender.
2010-10-18 14:55:35 -07:00
Mosharaf Chowdhury
0791e20bfc
Added hasBlocksBitVector to keep track of which blocks a peer actually have.
2010-10-18 14:34:33 -07:00
Mosharaf Chowdhury
b02ad04560
Updated log4j.properties to print milliseconds.
2010-10-18 14:06:48 -07:00
Mosharaf Chowdhury
1af0cdbac7
Removed all SplitStream-related things.
2010-10-18 11:40:21 -07:00
Mosharaf Chowdhury
a4c0281902
sendObject now takes parameters instead of relying on class
...
variables.
2010-10-14 15:36:23 -07:00
Mosharaf Chowdhury
a137ca75da
Got rid pf dualMode.
2010-10-13 17:01:00 -07:00
Mosharaf Chowdhury
38194e5731
- Changed guidePort to GuideInfo that now contains the hostAddress
...
as well as the port. This will allow anyone other than the master
to be a guide.
- The GuideInfo object now contains the constants related to
tracker response.
2010-10-13 16:26:18 -07:00
Mosharaf Chowdhury
8690be8f5a
Cleared up some formatting.
...
Branching out from here to work on BT.
2010-10-13 11:40:03 -07:00
Mosharaf Chowdhury
0d67bc1cee
multi-tracker branch now compiles and runs; but it crashes right before the
...
end. The same problem is seen also in the master branch (in the
ChainedStreaming implementation)
2010-10-12 15:39:53 -07:00
Mosharaf Chowdhury
4fdd48295b
Added mesos.jar. Still not working. Major changes required.
2010-10-12 13:10:31 -07:00
Mosharaf Chowdhury
e73a5f3491
Now compiles with Scala 2.8.0, but doesn't run with nexus.jar
...
Must update it to use mesos.jar
2010-10-12 13:05:32 -07:00
Matei Zaharia
0da5b00d6e
Merge branch 'master' into multi-tracker
...
Conflicts:
Makefile
run
src/scala/spark/Broadcast.scala
src/scala/spark/HdfsFile.scala
src/scala/spark/NexusScheduler.scala
src/scala/spark/SparkContext.scala
src/test/spark/repl/ReplSuite.scala
third_party/nexus.jar
2010-06-27 22:25:56 -07:00
Matei Zaharia
7d0eae17e3
Merge branch 'dev'
...
Conflicts:
src/scala/spark/HdfsFile.scala
src/scala/spark/NexusScheduler.scala
src/test/spark/repl/ReplSuite.scala
2010-06-27 15:21:54 -07:00
root
6aacaa6870
Made Spark shell class directory configurable.
2010-06-18 23:24:18 +00:00
Matei Zaharia
323571a177
Initial work on union operation.
2010-06-18 12:54:33 -07:00