Commit graph

22156 commits

Author SHA1 Message Date
Mosharaf Chowdhury 44eb79f64a - Measuring approximate time to receive a block from the sending peer.
- Added some Exception handling/suppression code.
2010-10-27 17:41:37 -07:00
Mosharaf Chowdhury ae2c93a54b Fixed some configuration bugs 2010-10-27 16:18:26 -07:00
Mosharaf Chowdhury e4dc7847ba - Receivers now send back the latest hasBlocksBitVector after every reception
- Will probably settle for the push model. Still not removing the pull related codes
 - Refactoring...
2010-10-27 15:36:58 -07:00
Mosharaf Chowdhury 059b6ae475 Added data structures to turn transmission to a pull model from a push one. 2010-10-27 15:05:29 -07:00
Mosharaf Chowdhury f7364111bb TalkToGuide breaks once the peer has all the blocks. 2010-10-26 21:26:45 -07:00
Mosharaf Chowdhury 8587353b3e Added some tuning parameters. 2010-10-26 21:16:37 -07:00
Mosharaf Chowdhury d70f3549ff MaxTxPeers and MaxRxPeers set upper limit on concurrent sending and receiving threads for every peer.
Relevant timeout code is also in. There is still an exception to take care of.
2010-10-26 17:53:03 -07:00
Mosharaf Chowdhury 1877f1c38d No one is waiting on hasBlocksLock. Removed... 2010-10-26 16:18:13 -07:00
Mosharaf Chowdhury bc1c51817c Beautified code... 2010-10-26 16:08:22 -07:00
Mosharaf Chowdhury d673be0e06 Minor changes. Still need to run on Mesos+EC2 2010-10-26 15:07:03 -07:00
Mosharaf Chowdhury 3f96c9b7e8 Resolved some bugs. Apparently, objects deep inside other objects could be passed as references. Bad Scala! 2010-10-26 12:56:27 -07:00
Mosharaf Chowdhury e659efca88 Added a synchronized block around numCopiesSent manipulation. 2010-10-26 08:52:56 -07:00
Mosharaf Chowdhury dba92f7dbe - Fixed bugs with storing received blocks in appropriate indices.
- receiveBroadcast now returns, but still no good way to stop threads from running.
2010-10-25 17:34:25 -07:00
Mosharaf Chowdhury 96797cc997 Replaced nested variable.synchronized blocks with a single top level synchronized { } 2010-10-25 14:53:56 -07:00
Mosharaf Chowdhury 2099aadbbd Implemented rarest first policy. This isn't working as expected (i.e., not distributing load as expected), probably due to some issues with how we pick peers to talk to. 2010-10-25 14:48:11 -07:00
Matei Zaharia dd7c5d8e34 Added initial attempt at a BoundedMemoryCache 2010-10-24 19:14:35 -07:00
Matei Zaharia edf86fdb27 Added SizeEstimator class for use by caches 2010-10-24 18:03:49 -07:00
Matei Zaharia a481e23761 Made caching pluggable and added soft reference and weak reference caches. 2010-10-23 17:54:25 -07:00
Matei Zaharia 93a200bc7e Renamed aggregateSplit() to splitRdd(), plus some style fixes 2010-10-23 15:34:03 -07:00
Mosharaf Chowdhury 7a7123b525 Running in local Mesos. Multiple things have to be fixed though. Go through the TODOs... 2010-10-22 23:43:11 -07:00
Mosharaf Chowdhury 27362c6523 Fixed ordering of function calls and thread creations.
Changed Slaves to Workers :)
2010-10-22 12:03:40 -07:00
Mosharaf Chowdhury 88264a33cf Peers actually exchange information between themselves.
Don't exchange info about neighbors yet :(
2010-10-21 18:12:30 -07:00
Mosharaf Chowdhury 8e6ed77724 Put synchronized blocks around BitSets.
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
2010-10-21 17:58:42 -07:00
Mosharaf Chowdhury 22a47a9d16 PeerChatterController implemented. Now things should start working... 2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury 86b2824101 Formatting... 2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury 53bd64afe7 - Added TalkToPeer class which will allow peers to communicate between them.
- Still need the controller class that will decide which peers to communicate with
2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury 05bca235a7 Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT 2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury e70755c2a2 - Implemented TalkToGuide class that talks to the Guide periodically
- Changed listOfSources from Vector to ListBuffer
2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury b1c1e80026 - Added BitSet of pieces to SourceInfo.
- Streamlined code after removing GuideInfo.
- Minor refactoring.
2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury b45dd8b8a4 Using SourceInfo instead of a separate GuideInfo case class. 2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury 365012f586 - Replaced pqOfSources with a vector: listOfSources
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
2010-10-19 17:02:55 -07:00
Matei Zaharia 787faf0d0e Fixed a bug with scheduling of tasks that have no locality preferences.
These tasks were being subjected to delay scheduling but then counted as
having been launched on a preferred node. The solution is to have a
separate queue for them and treat them as preferred during scheduling.
2010-10-19 16:07:58 -07:00
Mosharaf Chowdhury 905745707c Removed ReplicaID from SourceInfo. 2010-10-19 15:35:54 -07:00
Mosharaf Chowdhury 4ad6c5218e Removed some commented sort/compare funtions from SourceInfo 2010-10-19 15:31:27 -07:00
Mosharaf Chowdhury 5deb23df96 Starting major changes to make it more like BitTorrent. 2010-10-19 15:28:36 -07:00
Matei Zaharia 0e0ec83570 Undid some changes that Mosharaf inadvertedly committed to master. 2010-10-19 13:58:52 -07:00
Mosharaf Chowdhury 744d49bc40 Receivers now send their hasBlocksBitVector to the sender. 2010-10-18 14:55:35 -07:00
Mosharaf Chowdhury 0791e20bfc Added hasBlocksBitVector to keep track of which blocks a peer actually have. 2010-10-18 14:34:33 -07:00
Mosharaf Chowdhury b02ad04560 Updated log4j.properties to print milliseconds. 2010-10-18 14:06:48 -07:00
Mosharaf Chowdhury 1af0cdbac7 Removed all SplitStream-related things. 2010-10-18 11:40:21 -07:00
Mosharaf Chowdhury bf7055decf Merge branch 'master' of git@github.com:mesos/spark
Conflicts:
	src/scala/spark/SparkContext.scala

Using the latest one from Matei.
2010-10-18 11:08:45 -07:00
Matei Zaharia b940164db3 Less hacky way of preventing config files from being overwritten when a template file changes 2010-10-16 22:01:05 -07:00
Matei Zaharia e5fb280ec8 Changed the config files that were included in git to templates which
are used to create an initial copy of each config file if the user does
not have one. This way, users won't accidentally commit their changes to
config files to git.
2010-10-16 21:51:25 -07:00
Matei Zaharia 023ed194b4 Fixed some whitespace 2010-10-16 21:21:16 -07:00
Matei Zaharia 74bbfa91c2 Added support for generic Hadoop InputFormats and refactored textFile to
use this. Closes #12.
2010-10-16 19:03:33 -07:00
Matei Zaharia 03238cb7c1 Renamed HdfsFile to HadoopFile 2010-10-16 17:25:09 -07:00
Matei Zaharia 0e2adecdab Simplified UnionRDD slightly and added a SparkContext.union method for efficiently union-ing a large number of RDDs 2010-10-16 17:13:52 -07:00
Matei Zaharia 166d9f9125 Removed setSparkHome method on SparkContext in favor of having an
optional constructor parameter, so that the scheduler is guaranteed that
a Spark home has been set when it first builds its executor arg.
2010-10-16 16:19:47 -07:00
Matei Zaharia 1c082ad5fb Added the ability to specify a list of JAR files when creating a
SparkContext and have the master node serve those to workers.
2010-10-16 16:14:13 -07:00
Matei Zaharia c0b856a056 Set absolute path for SPARK_HOME 2010-10-16 12:18:02 -07:00