Mosharaf Chowdhury
|
eb70f6a36e
|
Receivers explicitly ask for blocks to send.
The Random generator must be updated so that everyone does not ask for the same block during the initial phase.
|
2010-11-10 16:47:45 -08:00 |
|
|
Mosharaf Chowdhury
|
d7579e743a
|
Style update: all functions that return void should have a Unit return type.
|
2010-11-10 10:32:29 -08:00 |
|
|
Mosharaf Chowdhury
|
642098c211
|
Bug fixes and/or minor optimization.
|
2010-11-09 23:59:06 -08:00 |
|
|
Mosharaf Chowdhury
|
b58bbadbfb
|
All daemon => Fast shutdown :)
|
2010-11-09 17:05:19 -08:00 |
|
|
Mosharaf Chowdhury
|
76610e842b
|
Removed some unnecessary stuff.
|
2010-11-09 16:59:32 -08:00 |
|
|
Mosharaf Chowdhury
|
50612ab3a9
|
Every thread is a daemon thread => Program shuts down ASAP once its done.
Some log messages added to blank exception handlers.
|
2010-11-09 16:44:42 -08:00 |
|
Matei Zaharia
|
7b25ab87af
|
Added options for using an external HTTP server with LocalFileShuffle
|
2010-11-09 13:46:30 -08:00 |
|
|
Mosharaf Chowdhury
|
1820634dbf
|
Bug fix: java-opts had wrong (non-capitalized) variable names.
|
2010-11-08 18:32:17 -08:00 |
|
|
Matei Zaharia
|
504f839c65
|
Removed unnecessary collectAsMap
|
2010-11-08 08:49:42 -08:00 |
|
|
Matei Zaharia
|
9d3f05a990
|
Made shuffle algorithm pluggable and added LocalFileShuffle.
|
2010-11-08 00:46:12 -08:00 |
|
|
Mosharaf Chowdhury
|
6cb76d9708
|
Commented out some cluttering log messages.
|
2010-11-07 19:16:03 -08:00 |
|
|
Mosharaf Chowdhury
|
cbce44304c
|
Added exception handling to 'sendStopBroadcastNotifications'
|
2010-11-06 19:37:27 -07:00 |
|
|
Mosharaf Chowdhury
|
49a2db09fb
|
Graceful shutdown is working with dualMode=false. Probably will have to remove dualMode completely.
Made BroadcastCS code more consistent with BT branches.
|
2010-11-06 19:27:46 -07:00 |
|
|
Mosharaf Chowdhury
|
c37c74919f
|
Bug fix + minor changes
|
2010-11-06 16:54:55 -07:00 |
|
|
Mosharaf Chowdhury
|
cbb29fae1a
|
Updated log outputs for consistency with BT branches.
Code formatting.
|
2010-11-06 16:52:50 -07:00 |
|
|
Matei Zaharia
|
d9ea6d69a5
|
Create output files one by one instead of at the same time in the map
phase of DfsShuffle.
|
2010-11-06 10:53:57 -07:00 |
|
|
Mosharaf Chowdhury
|
878d157ce3
|
Graceful shutdown after a single transmission in the swarm is over.
There might still be a problem with the Tracker shutdown. It must be done explicitly by SparkContext.
|
2010-11-04 22:09:14 -07:00 |
|
|
Matei Zaharia
|
16ff4dc0be
|
Merge branch 'matei-shuffle' of github.com:mesos/spark into matei-shuffle
|
2010-11-04 14:40:36 -07:00 |
|
|
Matei Zaharia
|
d984b8ab23
|
Properly set the number of output splits in DFS shuffle
|
2010-11-04 14:39:55 -07:00 |
|
root
|
4cc0984b43
|
Fixed a small bug in DFS shuffle -- the number of reduce tasks was not being set based on numOutputSplits
|
2010-11-04 21:34:55 +00:00 |
|
|
Matei Zaharia
|
96f0be935a
|
Added groupBy function in RDD
|
2010-11-03 23:58:53 -07:00 |
|
|
Matei Zaharia
|
72ec298cd4
|
Added reduceByKey, groupByKey and join operations based on combine, as
well as versions of the shuffle operations that set the number of splits
automatically.
|
2010-11-03 23:51:11 -07:00 |
|
|
Matei Zaharia
|
d947cb9778
|
Fixed a bug with negative hashcodes
|
2010-11-03 22:52:41 -07:00 |
|
|
Matei Zaharia
|
44530c310b
|
Made DFS shuffle's "reduce tasks" fetch inputs in a random order so they
don't all hit the same nodes at the same time.
|
2010-11-03 22:45:44 -07:00 |
|
|
Matei Zaharia
|
820dac5afe
|
Initial work towards a simple HDFS-based shuffle.
|
2010-11-03 21:27:24 -07:00 |
|
|
Matei Zaharia
|
648f42933a
|
Made alltests write test output as XML in build/test_results
|
2010-11-02 12:53:38 -07:00 |
|
|
Matei Zaharia
|
6f93baa463
|
'Running on Mesos' test is now only run when MESOS_HOME is set
|
2010-11-02 12:51:22 -07:00 |
|
|
Mosharaf Chowdhury
|
10fc66b1c4
|
Print reception messages for duplicate blocks as well...
|
2010-11-01 14:00:40 -07:00 |
|
|
Mosharaf Chowdhury
|
03d73dea9f
|
Refactoring...
|
2010-10-28 15:09:58 -07:00 |
|
|
Mosharaf Chowdhury
|
681dba1373
|
Removed some deprecated speed related code.
|
2010-10-28 14:40:49 -07:00 |
|
|
Mosharaf Chowdhury
|
bcc7d38ace
|
Added SpeedTracker to keep track of reception time per block from each source a peer has ever talked to.
|
2010-10-28 14:37:59 -07:00 |
|
|
Mosharaf Chowdhury
|
44eb79f64a
|
- Measuring approximate time to receive a block from the sending peer.
- Added some Exception handling/suppression code.
|
2010-10-27 17:41:37 -07:00 |
|
|
Mosharaf Chowdhury
|
ae2c93a54b
|
Fixed some configuration bugs
|
2010-10-27 16:18:26 -07:00 |
|
|
Mosharaf Chowdhury
|
e4dc7847ba
|
- Receivers now send back the latest hasBlocksBitVector after every reception
- Will probably settle for the push model. Still not removing the pull related codes
- Refactoring...
|
2010-10-27 15:36:58 -07:00 |
|
|
Mosharaf Chowdhury
|
059b6ae475
|
Added data structures to turn transmission to a pull model from a push one.
|
2010-10-27 15:05:29 -07:00 |
|
|
Mosharaf Chowdhury
|
f7364111bb
|
TalkToGuide breaks once the peer has all the blocks.
|
2010-10-26 21:26:45 -07:00 |
|
|
Mosharaf Chowdhury
|
8587353b3e
|
Added some tuning parameters.
|
2010-10-26 21:16:37 -07:00 |
|
|
Mosharaf Chowdhury
|
d70f3549ff
|
MaxTxPeers and MaxRxPeers set upper limit on concurrent sending and receiving threads for every peer.
Relevant timeout code is also in. There is still an exception to take care of.
|
2010-10-26 17:53:03 -07:00 |
|
|
Mosharaf Chowdhury
|
1877f1c38d
|
No one is waiting on hasBlocksLock. Removed...
|
2010-10-26 16:18:13 -07:00 |
|
|
Mosharaf Chowdhury
|
bc1c51817c
|
Beautified code...
|
2010-10-26 16:08:22 -07:00 |
|
|
Mosharaf Chowdhury
|
d673be0e06
|
Minor changes. Still need to run on Mesos+EC2
|
2010-10-26 15:07:03 -07:00 |
|
|
Mosharaf Chowdhury
|
3f96c9b7e8
|
Resolved some bugs. Apparently, objects deep inside other objects could be passed as references. Bad Scala!
|
2010-10-26 12:56:27 -07:00 |
|
|
Mosharaf Chowdhury
|
e659efca88
|
Added a synchronized block around numCopiesSent manipulation.
|
2010-10-26 08:52:56 -07:00 |
|
|
Mosharaf Chowdhury
|
dba92f7dbe
|
- Fixed bugs with storing received blocks in appropriate indices.
- receiveBroadcast now returns, but still no good way to stop threads from running.
|
2010-10-25 17:34:25 -07:00 |
|
|
Mosharaf Chowdhury
|
96797cc997
|
Replaced nested variable.synchronized blocks with a single top level synchronized { }
|
2010-10-25 14:53:56 -07:00 |
|
|
Mosharaf Chowdhury
|
2099aadbbd
|
Implemented rarest first policy. This isn't working as expected (i.e., not distributing load as expected), probably due to some issues with how we pick peers to talk to.
|
2010-10-25 14:48:11 -07:00 |
|
|
Matei Zaharia
|
dd7c5d8e34
|
Added initial attempt at a BoundedMemoryCache
|
2010-10-24 19:14:35 -07:00 |
|
|
Matei Zaharia
|
edf86fdb27
|
Added SizeEstimator class for use by caches
|
2010-10-24 18:03:49 -07:00 |
|
|
Matei Zaharia
|
a481e23761
|
Made caching pluggable and added soft reference and weak reference caches.
|
2010-10-23 17:54:25 -07:00 |
|
|
Matei Zaharia
|
93a200bc7e
|
Renamed aggregateSplit() to splitRdd(), plus some style fixes
|
2010-10-23 15:34:03 -07:00 |
|