ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
68 commits 6 branches 181 tags 397 MiB
Commit graph

68 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury bc1c51817c Beautified code... 2010-10-26 16:08:22 -07:00
Mosharaf Chowdhury d673be0e06 Minor changes. Still need to run on Mesos+EC2 2010-10-26 15:07:03 -07:00
Mosharaf Chowdhury 3f96c9b7e8 Resolved some bugs. Apparently, objects deep inside other objects could be passed as references. Bad Scala! 2010-10-26 12:56:27 -07:00
Mosharaf Chowdhury e659efca88 Added a synchronized block around numCopiesSent manipulation. 2010-10-26 08:52:56 -07:00
Mosharaf Chowdhury dba92f7dbe - Fixed bugs with storing received blocks in appropriate indices. 
- receiveBroadcast now returns, but still no good way to stop threads from running.
 2010-10-25 17:34:25 -07:00
Mosharaf Chowdhury 96797cc997 Replaced nested variable.synchronized blocks with a single top level synchronized { } 2010-10-25 14:53:56 -07:00
Mosharaf Chowdhury 2099aadbbd Implemented rarest first policy. This isn't working as expected (i.e., not distributing load as expected), probably due to some issues with how we pick peers to talk to. 2010-10-25 14:48:11 -07:00
Mosharaf Chowdhury 7a7123b525 Running in local Mesos. Multiple things have to be fixed though. Go through the TODOs... 2010-10-22 23:43:11 -07:00
Mosharaf Chowdhury 27362c6523 Fixed ordering of function calls and thread creations. 
Changed Slaves to Workers :)
 2010-10-22 12:03:40 -07:00
Mosharaf Chowdhury 88264a33cf Peers actually exchange information between themselves. 
Don't exchange info about neighbors yet :(
 2010-10-21 18:12:30 -07:00
Mosharaf Chowdhury 8e6ed77724 Put synchronized blocks around BitSets. 
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
 2010-10-21 17:58:42 -07:00
Mosharaf Chowdhury 22a47a9d16 PeerChatterController implemented. Now things should start working... 2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury 86b2824101 Formatting... 2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury 53bd64afe7 - Added TalkToPeer class which will allow peers to communicate between them. 
- Still need the controller class that will decide which peers to communicate with
 2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury 05bca235a7 Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT 2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury e70755c2a2 - Implemented TalkToGuide class that talks to the Guide periodically 
- Changed listOfSources from Vector to ListBuffer
 2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury b1c1e80026 - Added BitSet of pieces to SourceInfo. 
- Streamlined code after removing GuideInfo.
- Minor refactoring.
 2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury b45dd8b8a4 Using SourceInfo instead of a separate GuideInfo case class. 2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury 365012f586 - Replaced pqOfSources with a vector: listOfSources 
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
 2010-10-19 17:02:55 -07:00
Mosharaf Chowdhury 905745707c Removed ReplicaID from SourceInfo. 2010-10-19 15:35:54 -07:00
Mosharaf Chowdhury 4ad6c5218e Removed some commented sort/compare funtions from SourceInfo 2010-10-19 15:31:27 -07:00
Mosharaf Chowdhury 5deb23df96 Starting major changes to make it more like BitTorrent. 2010-10-19 15:28:36 -07:00
Mosharaf Chowdhury 744d49bc40 Receivers now send their hasBlocksBitVector to the sender. 2010-10-18 14:55:35 -07:00
Mosharaf Chowdhury 0791e20bfc Added hasBlocksBitVector to keep track of which blocks a peer actually have. 2010-10-18 14:34:33 -07:00
Mosharaf Chowdhury b02ad04560 Updated log4j.properties to print milliseconds. 2010-10-18 14:06:48 -07:00
Mosharaf Chowdhury 1af0cdbac7 Removed all SplitStream-related things. 2010-10-18 11:40:21 -07:00
Mosharaf Chowdhury a4c0281902 sendObject now takes parameters instead of relying on class 
variables.
 2010-10-14 15:36:23 -07:00
Mosharaf Chowdhury a137ca75da Got rid pf dualMode. 2010-10-13 17:01:00 -07:00
Mosharaf Chowdhury 38194e5731 - Changed guidePort to GuideInfo that now contains the hostAddress 
as well as the port. This will allow anyone other than the master
to be a guide.
- The GuideInfo object now contains the constants related to
tracker response.
 2010-10-13 16:26:18 -07:00
Mosharaf Chowdhury 8690be8f5a Cleared up some formatting. 
Branching out from here to work on BT.
 2010-10-13 11:40:03 -07:00
Mosharaf Chowdhury 0d67bc1cee multi-tracker branch now compiles and runs; but it crashes right before the 
end. The same problem is seen also in the master branch (in the
ChainedStreaming implementation)
 2010-10-12 15:39:53 -07:00
Mosharaf Chowdhury 4fdd48295b Added mesos.jar. Still not working. Major changes required. 2010-10-12 13:10:31 -07:00
Mosharaf Chowdhury e73a5f3491 Now compiles with Scala 2.8.0, but doesn't run with nexus.jar 
Must update it to use mesos.jar
 2010-10-12 13:05:32 -07:00
Matei Zaharia 0da5b00d6e Merge branch 'master' into multi-tracker 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/NexusScheduler.scala
	src/scala/spark/SparkContext.scala
	src/test/spark/repl/ReplSuite.scala
	third_party/nexus.jar
 2010-06-27 22:25:56 -07:00
Matei Zaharia 7d0eae17e3 Merge branch 'dev' 
Conflicts:
	src/scala/spark/HdfsFile.scala
	src/scala/spark/NexusScheduler.scala
	src/test/spark/repl/ReplSuite.scala
 2010-06-27 15:21:54 -07:00
root 6aacaa6870 Made Spark shell class directory configurable. 2010-06-18 23:24:18 +00:00
Matei Zaharia 323571a177 Initial work on union operation. 2010-06-18 12:54:33 -07:00
Matei Zaharia b54198819e Added appropriate hashCode, equals and toString to ParallelArraySplit. 2010-06-17 13:19:02 -07:00
Matei Zaharia cd247b7d86 Created common RDD superclass for distributed files and parallel arrays. 
This also means that parallel arrays now get all the functionality files
used to have (filter, map, reduce, cache, etc).
 2010-06-17 12:49:42 -07:00
Matei Zaharia 77103eab2a Fixed README 2010-06-11 14:55:23 -07:00
Matei Zaharia 0d9c51d473 Added back REPL tests 2010-06-11 10:03:01 -07:00
Matei Zaharia e58fba279d Fix junk stripper 2010-06-11 01:18:43 -07:00
Matei Zaharia 396f48e5a4 New interpreter port for Scala 2.8 interpreter 2010-06-11 01:10:03 -07:00
Matei Zaharia 4eb39e0c8a New nexus.jar 2010-06-10 22:41:23 -07:00
Matei Zaharia 1473987fb7 Fixed classpath for tests 2010-06-10 22:36:45 -07:00
Matei Zaharia 359e84c585 Use new Nexus API 2010-06-10 22:09:13 -07:00
Matei Zaharia 92246c843b Initial work on 2.8 port 2010-06-10 21:50:55 -07:00
Matei Zaharia c177a546a5 Ignore .DS_Store 2010-06-10 18:08:59 -07:00
Mosharaf Chowdhury 7ab703117a Added timers around BroadcastTest and sendBroadcast. 
Turned OFF saving to HDFS for now for stress tests.
pqOfSources is ordered by least leechers again.
 2010-05-18 20:49:59 -07:00
Mosharaf Chowdhury 2d381c974e Added flush calls after all writeObject calls as well as after creating every 
ObjectOutputStream object.
 2010-05-16 17:06:05 -07:00