Mosharaf Chowdhury
49a2db09fb
Graceful shutdown is working with dualMode=false. Probably will have to remove dualMode completely.
...
Made BroadcastCS code more consistent with BT branches.
2010-11-06 19:27:46 -07:00
Mosharaf Chowdhury
c37c74919f
Bug fix + minor changes
2010-11-06 16:54:55 -07:00
Mosharaf Chowdhury
cbb29fae1a
Updated log outputs for consistency with BT branches.
...
Code formatting.
2010-11-06 16:52:50 -07:00
Matei Zaharia
d9ea6d69a5
Create output files one by one instead of at the same time in the map
...
phase of DfsShuffle.
2010-11-06 10:53:57 -07:00
Mosharaf Chowdhury
878d157ce3
Graceful shutdown after a single transmission in the swarm is over.
...
There might still be a problem with the Tracker shutdown. It must be done explicitly by SparkContext.
2010-11-04 22:09:14 -07:00
Matei Zaharia
16ff4dc0be
Merge branch 'matei-shuffle' of github.com:mesos/spark into matei-shuffle
2010-11-04 14:40:36 -07:00
Matei Zaharia
d984b8ab23
Properly set the number of output splits in DFS shuffle
2010-11-04 14:39:55 -07:00
root
4cc0984b43
Fixed a small bug in DFS shuffle -- the number of reduce tasks was not being set based on numOutputSplits
2010-11-04 21:34:55 +00:00
Matei Zaharia
96f0be935a
Added groupBy function in RDD
2010-11-03 23:58:53 -07:00
Matei Zaharia
72ec298cd4
Added reduceByKey, groupByKey and join operations based on combine, as
...
well as versions of the shuffle operations that set the number of splits
automatically.
2010-11-03 23:51:11 -07:00
Matei Zaharia
d947cb9778
Fixed a bug with negative hashcodes
2010-11-03 22:52:41 -07:00
Matei Zaharia
44530c310b
Made DFS shuffle's "reduce tasks" fetch inputs in a random order so they
...
don't all hit the same nodes at the same time.
2010-11-03 22:45:44 -07:00
Matei Zaharia
820dac5afe
Initial work towards a simple HDFS-based shuffle.
2010-11-03 21:27:24 -07:00
Matei Zaharia
648f42933a
Made alltests write test output as XML in build/test_results
2010-11-02 12:53:38 -07:00
Matei Zaharia
6f93baa463
'Running on Mesos' test is now only run when MESOS_HOME is set
2010-11-02 12:51:22 -07:00
Mosharaf Chowdhury
10fc66b1c4
Print reception messages for duplicate blocks as well...
2010-11-01 14:00:40 -07:00
Mosharaf Chowdhury
03d73dea9f
Refactoring...
2010-10-28 15:09:58 -07:00
Mosharaf Chowdhury
681dba1373
Removed some deprecated speed related code.
2010-10-28 14:40:49 -07:00
Mosharaf Chowdhury
bcc7d38ace
Added SpeedTracker to keep track of reception time per block from each source a peer has ever talked to.
2010-10-28 14:37:59 -07:00
Mosharaf Chowdhury
44eb79f64a
- Measuring approximate time to receive a block from the sending peer.
...
- Added some Exception handling/suppression code.
2010-10-27 17:41:37 -07:00
Mosharaf Chowdhury
ae2c93a54b
Fixed some configuration bugs
2010-10-27 16:18:26 -07:00
Mosharaf Chowdhury
e4dc7847ba
- Receivers now send back the latest hasBlocksBitVector after every reception
...
- Will probably settle for the push model. Still not removing the pull related codes
- Refactoring...
2010-10-27 15:36:58 -07:00
Mosharaf Chowdhury
059b6ae475
Added data structures to turn transmission to a pull model from a push one.
2010-10-27 15:05:29 -07:00
Mosharaf Chowdhury
f7364111bb
TalkToGuide breaks once the peer has all the blocks.
2010-10-26 21:26:45 -07:00
Mosharaf Chowdhury
8587353b3e
Added some tuning parameters.
2010-10-26 21:16:37 -07:00
Mosharaf Chowdhury
d70f3549ff
MaxTxPeers and MaxRxPeers set upper limit on concurrent sending and receiving threads for every peer.
...
Relevant timeout code is also in. There is still an exception to take care of.
2010-10-26 17:53:03 -07:00
Mosharaf Chowdhury
1877f1c38d
No one is waiting on hasBlocksLock. Removed...
2010-10-26 16:18:13 -07:00
Mosharaf Chowdhury
bc1c51817c
Beautified code...
2010-10-26 16:08:22 -07:00
Mosharaf Chowdhury
d673be0e06
Minor changes. Still need to run on Mesos+EC2
2010-10-26 15:07:03 -07:00
Mosharaf Chowdhury
3f96c9b7e8
Resolved some bugs. Apparently, objects deep inside other objects could be passed as references. Bad Scala!
2010-10-26 12:56:27 -07:00
Mosharaf Chowdhury
e659efca88
Added a synchronized block around numCopiesSent manipulation.
2010-10-26 08:52:56 -07:00
Mosharaf Chowdhury
dba92f7dbe
- Fixed bugs with storing received blocks in appropriate indices.
...
- receiveBroadcast now returns, but still no good way to stop threads from running.
2010-10-25 17:34:25 -07:00
Mosharaf Chowdhury
96797cc997
Replaced nested variable.synchronized blocks with a single top level synchronized { }
2010-10-25 14:53:56 -07:00
Mosharaf Chowdhury
2099aadbbd
Implemented rarest first policy. This isn't working as expected (i.e., not distributing load as expected), probably due to some issues with how we pick peers to talk to.
2010-10-25 14:48:11 -07:00
Matei Zaharia
dd7c5d8e34
Added initial attempt at a BoundedMemoryCache
2010-10-24 19:14:35 -07:00
Matei Zaharia
edf86fdb27
Added SizeEstimator class for use by caches
2010-10-24 18:03:49 -07:00
Matei Zaharia
a481e23761
Made caching pluggable and added soft reference and weak reference caches.
2010-10-23 17:54:25 -07:00
Matei Zaharia
93a200bc7e
Renamed aggregateSplit() to splitRdd(), plus some style fixes
2010-10-23 15:34:03 -07:00
Mosharaf Chowdhury
7a7123b525
Running in local Mesos. Multiple things have to be fixed though. Go through the TODOs...
2010-10-22 23:43:11 -07:00
Mosharaf Chowdhury
27362c6523
Fixed ordering of function calls and thread creations.
...
Changed Slaves to Workers :)
2010-10-22 12:03:40 -07:00
Mosharaf Chowdhury
88264a33cf
Peers actually exchange information between themselves.
...
Don't exchange info about neighbors yet :(
2010-10-21 18:12:30 -07:00
Mosharaf Chowdhury
8e6ed77724
Put synchronized blocks around BitSets.
...
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
2010-10-21 17:58:42 -07:00
Mosharaf Chowdhury
22a47a9d16
PeerChatterController implemented. Now things should start working...
2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury
86b2824101
Formatting...
2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury
53bd64afe7
- Added TalkToPeer class which will allow peers to communicate between them.
...
- Still need the controller class that will decide which peers to communicate with
2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury
05bca235a7
Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT
2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury
e70755c2a2
- Implemented TalkToGuide class that talks to the Guide periodically
...
- Changed listOfSources from Vector to ListBuffer
2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury
b1c1e80026
- Added BitSet of pieces to SourceInfo.
...
- Streamlined code after removing GuideInfo.
- Minor refactoring.
2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury
b45dd8b8a4
Using SourceInfo instead of a separate GuideInfo case class.
2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury
365012f586
- Replaced pqOfSources with a vector: listOfSources
...
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
2010-10-19 17:02:55 -07:00