ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
9475 commits 6 branches 181 tags 397 MiB
Commit graph

9475 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury 49a2db09fb Graceful shutdown is working with dualMode=false. Probably will have to remove dualMode completely. 
Made BroadcastCS code more consistent with BT branches.
 2010-11-06 19:27:46 -07:00
Mosharaf Chowdhury c37c74919f Bug fix + minor changes 2010-11-06 16:54:55 -07:00
Mosharaf Chowdhury cbb29fae1a Updated log outputs for consistency with BT branches. 
Code formatting.
 2010-11-06 16:52:50 -07:00
Matei Zaharia d9ea6d69a5 Create output files one by one instead of at the same time in the map 
phase of DfsShuffle.
 2010-11-06 10:53:57 -07:00
Mosharaf Chowdhury 878d157ce3 Graceful shutdown after a single transmission in the swarm is over. 
There might still be a problem with the Tracker shutdown. It must be done explicitly by SparkContext.
 2010-11-04 22:09:14 -07:00
Matei Zaharia 16ff4dc0be Merge branch 'matei-shuffle' of github.com:mesos/spark into matei-shuffle 2010-11-04 14:40:36 -07:00
Matei Zaharia d984b8ab23 Properly set the number of output splits in DFS shuffle 2010-11-04 14:39:55 -07:00
root 4cc0984b43 Fixed a small bug in DFS shuffle -- the number of reduce tasks was not being set based on numOutputSplits 2010-11-04 21:34:55 +00:00
Matei Zaharia 96f0be935a Added groupBy function in RDD 2010-11-03 23:58:53 -07:00
Matei Zaharia 72ec298cd4 Added reduceByKey, groupByKey and join operations based on combine, as 
well as versions of the shuffle operations that set the number of splits
automatically.
 2010-11-03 23:51:11 -07:00
Matei Zaharia d947cb9778 Fixed a bug with negative hashcodes 2010-11-03 22:52:41 -07:00
Matei Zaharia 44530c310b Made DFS shuffle's "reduce tasks" fetch inputs in a random order so they 
don't all hit the same nodes at the same time.
 2010-11-03 22:45:44 -07:00
Matei Zaharia 820dac5afe Initial work towards a simple HDFS-based shuffle. 2010-11-03 21:27:24 -07:00
Matei Zaharia 648f42933a Made alltests write test output as XML in build/test_results 2010-11-02 12:53:38 -07:00
Matei Zaharia 6f93baa463 'Running on Mesos' test is now only run when MESOS_HOME is set 2010-11-02 12:51:22 -07:00
Mosharaf Chowdhury 10fc66b1c4 Print reception messages for duplicate blocks as well... 2010-11-01 14:00:40 -07:00
Mosharaf Chowdhury 03d73dea9f Refactoring... 2010-10-28 15:09:58 -07:00
Mosharaf Chowdhury 681dba1373 Removed some deprecated speed related code. 2010-10-28 14:40:49 -07:00
Mosharaf Chowdhury bcc7d38ace Added SpeedTracker to keep track of reception time per block from each source a peer has ever talked to. 2010-10-28 14:37:59 -07:00
Mosharaf Chowdhury 44eb79f64a - Measuring approximate time to receive a block from the sending peer. 
- Added some Exception handling/suppression code.
 2010-10-27 17:41:37 -07:00
Mosharaf Chowdhury ae2c93a54b Fixed some configuration bugs 2010-10-27 16:18:26 -07:00
Mosharaf Chowdhury e4dc7847ba - Receivers now send back the latest hasBlocksBitVector after every reception 
- Will probably settle for the push model. Still not removing the pull related codes
 - Refactoring...
 2010-10-27 15:36:58 -07:00
Mosharaf Chowdhury 059b6ae475 Added data structures to turn transmission to a pull model from a push one. 2010-10-27 15:05:29 -07:00
Mosharaf Chowdhury f7364111bb TalkToGuide breaks once the peer has all the blocks. 2010-10-26 21:26:45 -07:00
Mosharaf Chowdhury 8587353b3e Added some tuning parameters. 2010-10-26 21:16:37 -07:00
Mosharaf Chowdhury d70f3549ff MaxTxPeers and MaxRxPeers set upper limit on concurrent sending and receiving threads for every peer. 
Relevant timeout code is also in. There is still an exception to take care of.
 2010-10-26 17:53:03 -07:00
Mosharaf Chowdhury 1877f1c38d No one is waiting on hasBlocksLock. Removed... 2010-10-26 16:18:13 -07:00
Mosharaf Chowdhury bc1c51817c Beautified code... 2010-10-26 16:08:22 -07:00
Mosharaf Chowdhury d673be0e06 Minor changes. Still need to run on Mesos+EC2 2010-10-26 15:07:03 -07:00
Mosharaf Chowdhury 3f96c9b7e8 Resolved some bugs. Apparently, objects deep inside other objects could be passed as references. Bad Scala! 2010-10-26 12:56:27 -07:00
Mosharaf Chowdhury e659efca88 Added a synchronized block around numCopiesSent manipulation. 2010-10-26 08:52:56 -07:00
Mosharaf Chowdhury dba92f7dbe - Fixed bugs with storing received blocks in appropriate indices. 
- receiveBroadcast now returns, but still no good way to stop threads from running.
 2010-10-25 17:34:25 -07:00
Mosharaf Chowdhury 96797cc997 Replaced nested variable.synchronized blocks with a single top level synchronized { } 2010-10-25 14:53:56 -07:00
Mosharaf Chowdhury 2099aadbbd Implemented rarest first policy. This isn't working as expected (i.e., not distributing load as expected), probably due to some issues with how we pick peers to talk to. 2010-10-25 14:48:11 -07:00
Matei Zaharia dd7c5d8e34 Added initial attempt at a BoundedMemoryCache 2010-10-24 19:14:35 -07:00
Matei Zaharia edf86fdb27 Added SizeEstimator class for use by caches 2010-10-24 18:03:49 -07:00
Matei Zaharia a481e23761 Made caching pluggable and added soft reference and weak reference caches. 2010-10-23 17:54:25 -07:00
Matei Zaharia 93a200bc7e Renamed aggregateSplit() to splitRdd(), plus some style fixes 2010-10-23 15:34:03 -07:00
Mosharaf Chowdhury 7a7123b525 Running in local Mesos. Multiple things have to be fixed though. Go through the TODOs... 2010-10-22 23:43:11 -07:00
Mosharaf Chowdhury 27362c6523 Fixed ordering of function calls and thread creations. 
Changed Slaves to Workers :)
 2010-10-22 12:03:40 -07:00
Mosharaf Chowdhury 88264a33cf Peers actually exchange information between themselves. 
Don't exchange info about neighbors yet :(
 2010-10-21 18:12:30 -07:00
Mosharaf Chowdhury 8e6ed77724 Put synchronized blocks around BitSets. 
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
 2010-10-21 17:58:42 -07:00
Mosharaf Chowdhury 22a47a9d16 PeerChatterController implemented. Now things should start working... 2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury 86b2824101 Formatting... 2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury 53bd64afe7 - Added TalkToPeer class which will allow peers to communicate between them. 
- Still need the controller class that will decide which peers to communicate with
 2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury 05bca235a7 Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT 2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury e70755c2a2 - Implemented TalkToGuide class that talks to the Guide periodically 
- Changed listOfSources from Vector to ListBuffer
 2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury b1c1e80026 - Added BitSet of pieces to SourceInfo. 
- Streamlined code after removing GuideInfo.
- Minor refactoring.
 2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury b45dd8b8a4 Using SourceInfo instead of a separate GuideInfo case class. 2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury 365012f586 - Replaced pqOfSources with a vector: listOfSources 
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
 2010-10-19 17:02:55 -07:00