Mosharaf Chowdhury
|
cbce44304c
|
Added exception handling to 'sendStopBroadcastNotifications'
|
2010-11-06 19:37:27 -07:00 |
|
Mosharaf Chowdhury
|
49a2db09fb
|
Graceful shutdown is working with dualMode=false. Probably will have to remove dualMode completely.
Made BroadcastCS code more consistent with BT branches.
|
2010-11-06 19:27:46 -07:00 |
|
Mosharaf Chowdhury
|
c37c74919f
|
Bug fix + minor changes
|
2010-11-06 16:54:55 -07:00 |
|
Mosharaf Chowdhury
|
cbb29fae1a
|
Updated log outputs for consistency with BT branches.
Code formatting.
|
2010-11-06 16:52:50 -07:00 |
|
Matei Zaharia
|
d9ea6d69a5
|
Create output files one by one instead of at the same time in the map
phase of DfsShuffle.
|
2010-11-06 10:53:57 -07:00 |
|
Mosharaf Chowdhury
|
878d157ce3
|
Graceful shutdown after a single transmission in the swarm is over.
There might still be a problem with the Tracker shutdown. It must be done explicitly by SparkContext.
|
2010-11-04 22:09:14 -07:00 |
|
Matei Zaharia
|
16ff4dc0be
|
Merge branch 'matei-shuffle' of github.com:mesos/spark into matei-shuffle
|
2010-11-04 14:40:36 -07:00 |
|
Matei Zaharia
|
d984b8ab23
|
Properly set the number of output splits in DFS shuffle
|
2010-11-04 14:39:55 -07:00 |
|
root
|
4cc0984b43
|
Fixed a small bug in DFS shuffle -- the number of reduce tasks was not being set based on numOutputSplits
|
2010-11-04 21:34:55 +00:00 |
|
Matei Zaharia
|
96f0be935a
|
Added groupBy function in RDD
|
2010-11-03 23:58:53 -07:00 |
|
Matei Zaharia
|
72ec298cd4
|
Added reduceByKey, groupByKey and join operations based on combine, as
well as versions of the shuffle operations that set the number of splits
automatically.
|
2010-11-03 23:51:11 -07:00 |
|
Matei Zaharia
|
d947cb9778
|
Fixed a bug with negative hashcodes
|
2010-11-03 22:52:41 -07:00 |
|
Matei Zaharia
|
44530c310b
|
Made DFS shuffle's "reduce tasks" fetch inputs in a random order so they
don't all hit the same nodes at the same time.
|
2010-11-03 22:45:44 -07:00 |
|
Matei Zaharia
|
820dac5afe
|
Initial work towards a simple HDFS-based shuffle.
|
2010-11-03 21:27:24 -07:00 |
|
Matei Zaharia
|
648f42933a
|
Made alltests write test output as XML in build/test_results
|
2010-11-02 12:53:38 -07:00 |
|
Matei Zaharia
|
6f93baa463
|
'Running on Mesos' test is now only run when MESOS_HOME is set
|
2010-11-02 12:51:22 -07:00 |
|
Mosharaf Chowdhury
|
10fc66b1c4
|
Print reception messages for duplicate blocks as well...
|
2010-11-01 14:00:40 -07:00 |
|
Mosharaf Chowdhury
|
03d73dea9f
|
Refactoring...
|
2010-10-28 15:09:58 -07:00 |
|
Mosharaf Chowdhury
|
681dba1373
|
Removed some deprecated speed related code.
|
2010-10-28 14:40:49 -07:00 |
|
Mosharaf Chowdhury
|
bcc7d38ace
|
Added SpeedTracker to keep track of reception time per block from each source a peer has ever talked to.
|
2010-10-28 14:37:59 -07:00 |
|
Mosharaf Chowdhury
|
44eb79f64a
|
- Measuring approximate time to receive a block from the sending peer.
- Added some Exception handling/suppression code.
|
2010-10-27 17:41:37 -07:00 |
|
Mosharaf Chowdhury
|
ae2c93a54b
|
Fixed some configuration bugs
|
2010-10-27 16:18:26 -07:00 |
|
Mosharaf Chowdhury
|
e4dc7847ba
|
- Receivers now send back the latest hasBlocksBitVector after every reception
- Will probably settle for the push model. Still not removing the pull related codes
- Refactoring...
|
2010-10-27 15:36:58 -07:00 |
|
Mosharaf Chowdhury
|
059b6ae475
|
Added data structures to turn transmission to a pull model from a push one.
|
2010-10-27 15:05:29 -07:00 |
|
Mosharaf Chowdhury
|
f7364111bb
|
TalkToGuide breaks once the peer has all the blocks.
|
2010-10-26 21:26:45 -07:00 |
|
Mosharaf Chowdhury
|
8587353b3e
|
Added some tuning parameters.
|
2010-10-26 21:16:37 -07:00 |
|
Mosharaf Chowdhury
|
d70f3549ff
|
MaxTxPeers and MaxRxPeers set upper limit on concurrent sending and receiving threads for every peer.
Relevant timeout code is also in. There is still an exception to take care of.
|
2010-10-26 17:53:03 -07:00 |
|
Mosharaf Chowdhury
|
1877f1c38d
|
No one is waiting on hasBlocksLock. Removed...
|
2010-10-26 16:18:13 -07:00 |
|
Mosharaf Chowdhury
|
bc1c51817c
|
Beautified code...
|
2010-10-26 16:08:22 -07:00 |
|
Mosharaf Chowdhury
|
d673be0e06
|
Minor changes. Still need to run on Mesos+EC2
|
2010-10-26 15:07:03 -07:00 |
|
Mosharaf Chowdhury
|
3f96c9b7e8
|
Resolved some bugs. Apparently, objects deep inside other objects could be passed as references. Bad Scala!
|
2010-10-26 12:56:27 -07:00 |
|
Mosharaf Chowdhury
|
e659efca88
|
Added a synchronized block around numCopiesSent manipulation.
|
2010-10-26 08:52:56 -07:00 |
|
Mosharaf Chowdhury
|
dba92f7dbe
|
- Fixed bugs with storing received blocks in appropriate indices.
- receiveBroadcast now returns, but still no good way to stop threads from running.
|
2010-10-25 17:34:25 -07:00 |
|
Mosharaf Chowdhury
|
96797cc997
|
Replaced nested variable.synchronized blocks with a single top level synchronized { }
|
2010-10-25 14:53:56 -07:00 |
|
Mosharaf Chowdhury
|
2099aadbbd
|
Implemented rarest first policy. This isn't working as expected (i.e., not distributing load as expected), probably due to some issues with how we pick peers to talk to.
|
2010-10-25 14:48:11 -07:00 |
|
Matei Zaharia
|
dd7c5d8e34
|
Added initial attempt at a BoundedMemoryCache
|
2010-10-24 19:14:35 -07:00 |
|
Matei Zaharia
|
edf86fdb27
|
Added SizeEstimator class for use by caches
|
2010-10-24 18:03:49 -07:00 |
|
Matei Zaharia
|
a481e23761
|
Made caching pluggable and added soft reference and weak reference caches.
|
2010-10-23 17:54:25 -07:00 |
|
Matei Zaharia
|
93a200bc7e
|
Renamed aggregateSplit() to splitRdd(), plus some style fixes
|
2010-10-23 15:34:03 -07:00 |
|
Mosharaf Chowdhury
|
7a7123b525
|
Running in local Mesos. Multiple things have to be fixed though. Go through the TODOs...
|
2010-10-22 23:43:11 -07:00 |
|
Mosharaf Chowdhury
|
27362c6523
|
Fixed ordering of function calls and thread creations.
Changed Slaves to Workers :)
|
2010-10-22 12:03:40 -07:00 |
|
Mosharaf Chowdhury
|
88264a33cf
|
Peers actually exchange information between themselves.
Don't exchange info about neighbors yet :(
|
2010-10-21 18:12:30 -07:00 |
|
Mosharaf Chowdhury
|
8e6ed77724
|
Put synchronized blocks around BitSets.
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
|
2010-10-21 17:58:42 -07:00 |
|
Mosharaf Chowdhury
|
22a47a9d16
|
PeerChatterController implemented. Now things should start working...
|
2010-10-21 17:06:25 -07:00 |
|
Mosharaf Chowdhury
|
86b2824101
|
Formatting...
|
2010-10-21 13:30:52 -07:00 |
|
Mosharaf Chowdhury
|
53bd64afe7
|
- Added TalkToPeer class which will allow peers to communicate between them.
- Still need the controller class that will decide which peers to communicate with
|
2010-10-20 17:27:02 -07:00 |
|
Mosharaf Chowdhury
|
05bca235a7
|
Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT
|
2010-10-20 14:37:20 -07:00 |
|
Mosharaf Chowdhury
|
e70755c2a2
|
- Implemented TalkToGuide class that talks to the Guide periodically
- Changed listOfSources from Vector to ListBuffer
|
2010-10-20 14:14:44 -07:00 |
|
Mosharaf Chowdhury
|
b1c1e80026
|
- Added BitSet of pieces to SourceInfo.
- Streamlined code after removing GuideInfo.
- Minor refactoring.
|
2010-10-19 17:53:44 -07:00 |
|
Mosharaf Chowdhury
|
b45dd8b8a4
|
Using SourceInfo instead of a separate GuideInfo case class.
|
2010-10-19 17:18:20 -07:00 |
|