Matei Zaharia
72ec298cd4
Added reduceByKey, groupByKey and join operations based on combine, as
...
well as versions of the shuffle operations that set the number of splits
automatically.
2010-11-03 23:51:11 -07:00
Matei Zaharia
d947cb9778
Fixed a bug with negative hashcodes
2010-11-03 22:52:41 -07:00
Matei Zaharia
44530c310b
Made DFS shuffle's "reduce tasks" fetch inputs in a random order so they
...
don't all hit the same nodes at the same time.
2010-11-03 22:45:44 -07:00
Matei Zaharia
820dac5afe
Initial work towards a simple HDFS-based shuffle.
2010-11-03 21:27:24 -07:00
Matei Zaharia
648f42933a
Made alltests write test output as XML in build/test_results
2010-11-02 12:53:38 -07:00
Matei Zaharia
6f93baa463
'Running on Mesos' test is now only run when MESOS_HOME is set
2010-11-02 12:51:22 -07:00
Mosharaf Chowdhury
10fc66b1c4
Print reception messages for duplicate blocks as well...
2010-11-01 14:00:40 -07:00
Mosharaf Chowdhury
03d73dea9f
Refactoring...
2010-10-28 15:09:58 -07:00
Mosharaf Chowdhury
681dba1373
Removed some deprecated speed related code.
2010-10-28 14:40:49 -07:00
Mosharaf Chowdhury
bcc7d38ace
Added SpeedTracker to keep track of reception time per block from each source a peer has ever talked to.
2010-10-28 14:37:59 -07:00
Mosharaf Chowdhury
44eb79f64a
- Measuring approximate time to receive a block from the sending peer.
...
- Added some Exception handling/suppression code.
2010-10-27 17:41:37 -07:00
Mosharaf Chowdhury
ae2c93a54b
Fixed some configuration bugs
2010-10-27 16:18:26 -07:00
Mosharaf Chowdhury
e4dc7847ba
- Receivers now send back the latest hasBlocksBitVector after every reception
...
- Will probably settle for the push model. Still not removing the pull related codes
- Refactoring...
2010-10-27 15:36:58 -07:00
Mosharaf Chowdhury
059b6ae475
Added data structures to turn transmission to a pull model from a push one.
2010-10-27 15:05:29 -07:00
Mosharaf Chowdhury
f7364111bb
TalkToGuide breaks once the peer has all the blocks.
2010-10-26 21:26:45 -07:00
Mosharaf Chowdhury
8587353b3e
Added some tuning parameters.
2010-10-26 21:16:37 -07:00
Mosharaf Chowdhury
d70f3549ff
MaxTxPeers and MaxRxPeers set upper limit on concurrent sending and receiving threads for every peer.
...
Relevant timeout code is also in. There is still an exception to take care of.
2010-10-26 17:53:03 -07:00
Mosharaf Chowdhury
1877f1c38d
No one is waiting on hasBlocksLock. Removed...
2010-10-26 16:18:13 -07:00
Mosharaf Chowdhury
bc1c51817c
Beautified code...
2010-10-26 16:08:22 -07:00
Mosharaf Chowdhury
d673be0e06
Minor changes. Still need to run on Mesos+EC2
2010-10-26 15:07:03 -07:00
Mosharaf Chowdhury
3f96c9b7e8
Resolved some bugs. Apparently, objects deep inside other objects could be passed as references. Bad Scala!
2010-10-26 12:56:27 -07:00
Mosharaf Chowdhury
e659efca88
Added a synchronized block around numCopiesSent manipulation.
2010-10-26 08:52:56 -07:00
Mosharaf Chowdhury
dba92f7dbe
- Fixed bugs with storing received blocks in appropriate indices.
...
- receiveBroadcast now returns, but still no good way to stop threads from running.
2010-10-25 17:34:25 -07:00
Mosharaf Chowdhury
96797cc997
Replaced nested variable.synchronized blocks with a single top level synchronized { }
2010-10-25 14:53:56 -07:00
Mosharaf Chowdhury
2099aadbbd
Implemented rarest first policy. This isn't working as expected (i.e., not distributing load as expected), probably due to some issues with how we pick peers to talk to.
2010-10-25 14:48:11 -07:00
Matei Zaharia
dd7c5d8e34
Added initial attempt at a BoundedMemoryCache
2010-10-24 19:14:35 -07:00
Matei Zaharia
edf86fdb27
Added SizeEstimator class for use by caches
2010-10-24 18:03:49 -07:00
Matei Zaharia
a481e23761
Made caching pluggable and added soft reference and weak reference caches.
2010-10-23 17:54:25 -07:00
Matei Zaharia
93a200bc7e
Renamed aggregateSplit() to splitRdd(), plus some style fixes
2010-10-23 15:34:03 -07:00
Mosharaf Chowdhury
7a7123b525
Running in local Mesos. Multiple things have to be fixed though. Go through the TODOs...
2010-10-22 23:43:11 -07:00
Mosharaf Chowdhury
27362c6523
Fixed ordering of function calls and thread creations.
...
Changed Slaves to Workers :)
2010-10-22 12:03:40 -07:00
Mosharaf Chowdhury
88264a33cf
Peers actually exchange information between themselves.
...
Don't exchange info about neighbors yet :(
2010-10-21 18:12:30 -07:00
Mosharaf Chowdhury
8e6ed77724
Put synchronized blocks around BitSets.
...
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
2010-10-21 17:58:42 -07:00
Mosharaf Chowdhury
22a47a9d16
PeerChatterController implemented. Now things should start working...
2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury
86b2824101
Formatting...
2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury
53bd64afe7
- Added TalkToPeer class which will allow peers to communicate between them.
...
- Still need the controller class that will decide which peers to communicate with
2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury
05bca235a7
Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT
2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury
e70755c2a2
- Implemented TalkToGuide class that talks to the Guide periodically
...
- Changed listOfSources from Vector to ListBuffer
2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury
b1c1e80026
- Added BitSet of pieces to SourceInfo.
...
- Streamlined code after removing GuideInfo.
- Minor refactoring.
2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury
b45dd8b8a4
Using SourceInfo instead of a separate GuideInfo case class.
2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury
365012f586
- Replaced pqOfSources with a vector: listOfSources
...
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
2010-10-19 17:02:55 -07:00
Matei Zaharia
787faf0d0e
Fixed a bug with scheduling of tasks that have no locality preferences.
...
These tasks were being subjected to delay scheduling but then counted as
having been launched on a preferred node. The solution is to have a
separate queue for them and treat them as preferred during scheduling.
2010-10-19 16:07:58 -07:00
Mosharaf Chowdhury
905745707c
Removed ReplicaID from SourceInfo.
2010-10-19 15:35:54 -07:00
Mosharaf Chowdhury
4ad6c5218e
Removed some commented sort/compare funtions from SourceInfo
2010-10-19 15:31:27 -07:00
Mosharaf Chowdhury
5deb23df96
Starting major changes to make it more like BitTorrent.
2010-10-19 15:28:36 -07:00
Matei Zaharia
0e0ec83570
Undid some changes that Mosharaf inadvertedly committed to master.
2010-10-19 13:58:52 -07:00
Mosharaf Chowdhury
744d49bc40
Receivers now send their hasBlocksBitVector to the sender.
2010-10-18 14:55:35 -07:00
Mosharaf Chowdhury
0791e20bfc
Added hasBlocksBitVector to keep track of which blocks a peer actually have.
2010-10-18 14:34:33 -07:00
Mosharaf Chowdhury
b02ad04560
Updated log4j.properties to print milliseconds.
2010-10-18 14:06:48 -07:00
Mosharaf Chowdhury
1af0cdbac7
Removed all SplitStream-related things.
2010-10-18 11:40:21 -07:00
