ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
61 commits 6 branches 181 tags 397 MiB
Commit graph

61 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury 7a7123b525 Running in local Mesos. Multiple things have to be fixed though. Go through the TODOs... 2010-10-22 23:43:11 -07:00
Mosharaf Chowdhury 27362c6523 Fixed ordering of function calls and thread creations. 
Changed Slaves to Workers :)
 2010-10-22 12:03:40 -07:00
Mosharaf Chowdhury 88264a33cf Peers actually exchange information between themselves. 
Don't exchange info about neighbors yet :(
 2010-10-21 18:12:30 -07:00
Mosharaf Chowdhury 8e6ed77724 Put synchronized blocks around BitSets. 
Added getLocalSourceInfo method that returns a SourceInfo based on local info.
 2010-10-21 17:58:42 -07:00
Mosharaf Chowdhury 22a47a9d16 PeerChatterController implemented. Now things should start working... 2010-10-21 17:06:25 -07:00
Mosharaf Chowdhury 86b2824101 Formatting... 2010-10-21 13:30:52 -07:00
Mosharaf Chowdhury 53bd64afe7 - Added TalkToPeer class which will allow peers to communicate between them. 
- Still need the controller class that will decide which peers to communicate with
 2010-10-20 17:27:02 -07:00
Mosharaf Chowdhury 05bca235a7 Changed ChainedStreamingBroadcast to BitTorrentBroadcast and BroadcastCS to BroadcastBT 2010-10-20 14:37:20 -07:00
Mosharaf Chowdhury e70755c2a2 - Implemented TalkToGuide class that talks to the Guide periodically 
- Changed listOfSources from Vector to ListBuffer
 2010-10-20 14:14:44 -07:00
Mosharaf Chowdhury b1c1e80026 - Added BitSet of pieces to SourceInfo. 
- Streamlined code after removing GuideInfo.
- Minor refactoring.
 2010-10-19 17:53:44 -07:00
Mosharaf Chowdhury b45dd8b8a4 Using SourceInfo instead of a separate GuideInfo case class. 2010-10-19 17:18:20 -07:00
Mosharaf Chowdhury 365012f586 - Replaced pqOfSources with a vector: listOfSources 
- Guide now returns back multiple Sources instead of just one
- Receiver part is not updated though. So it wont work
- Added BroadcastCS.MaxPeersInGuideResponse
- Changed Source selection to simple round-robin
 2010-10-19 17:02:55 -07:00
Mosharaf Chowdhury 905745707c Removed ReplicaID from SourceInfo. 2010-10-19 15:35:54 -07:00
Mosharaf Chowdhury 4ad6c5218e Removed some commented sort/compare funtions from SourceInfo 2010-10-19 15:31:27 -07:00
Mosharaf Chowdhury 5deb23df96 Starting major changes to make it more like BitTorrent. 2010-10-19 15:28:36 -07:00
Mosharaf Chowdhury 744d49bc40 Receivers now send their hasBlocksBitVector to the sender. 2010-10-18 14:55:35 -07:00
Mosharaf Chowdhury 0791e20bfc Added hasBlocksBitVector to keep track of which blocks a peer actually have. 2010-10-18 14:34:33 -07:00
Mosharaf Chowdhury b02ad04560 Updated log4j.properties to print milliseconds. 2010-10-18 14:06:48 -07:00
Mosharaf Chowdhury 1af0cdbac7 Removed all SplitStream-related things. 2010-10-18 11:40:21 -07:00
Mosharaf Chowdhury a4c0281902 sendObject now takes parameters instead of relying on class 
variables.
 2010-10-14 15:36:23 -07:00
Mosharaf Chowdhury a137ca75da Got rid pf dualMode. 2010-10-13 17:01:00 -07:00
Mosharaf Chowdhury 38194e5731 - Changed guidePort to GuideInfo that now contains the hostAddress 
as well as the port. This will allow anyone other than the master
to be a guide.
- The GuideInfo object now contains the constants related to
tracker response.
 2010-10-13 16:26:18 -07:00
Mosharaf Chowdhury 8690be8f5a Cleared up some formatting. 
Branching out from here to work on BT.
 2010-10-13 11:40:03 -07:00
Mosharaf Chowdhury 0d67bc1cee multi-tracker branch now compiles and runs; but it crashes right before the 
end. The same problem is seen also in the master branch (in the
ChainedStreaming implementation)
 2010-10-12 15:39:53 -07:00
Mosharaf Chowdhury 4fdd48295b Added mesos.jar. Still not working. Major changes required. 2010-10-12 13:10:31 -07:00
Mosharaf Chowdhury e73a5f3491 Now compiles with Scala 2.8.0, but doesn't run with nexus.jar 
Must update it to use mesos.jar
 2010-10-12 13:05:32 -07:00
Matei Zaharia 0da5b00d6e Merge branch 'master' into multi-tracker 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/NexusScheduler.scala
	src/scala/spark/SparkContext.scala
	src/test/spark/repl/ReplSuite.scala
	third_party/nexus.jar
 2010-06-27 22:25:56 -07:00
Matei Zaharia 7d0eae17e3 Merge branch 'dev' 
Conflicts:
	src/scala/spark/HdfsFile.scala
	src/scala/spark/NexusScheduler.scala
	src/test/spark/repl/ReplSuite.scala
 2010-06-27 15:21:54 -07:00
root 6aacaa6870 Made Spark shell class directory configurable. 2010-06-18 23:24:18 +00:00
Matei Zaharia 323571a177 Initial work on union operation. 2010-06-18 12:54:33 -07:00
Matei Zaharia b54198819e Added appropriate hashCode, equals and toString to ParallelArraySplit. 2010-06-17 13:19:02 -07:00
Matei Zaharia cd247b7d86 Created common RDD superclass for distributed files and parallel arrays. 
This also means that parallel arrays now get all the functionality files
used to have (filter, map, reduce, cache, etc).
 2010-06-17 12:49:42 -07:00
Matei Zaharia 77103eab2a Fixed README 2010-06-11 14:55:23 -07:00
Matei Zaharia 0d9c51d473 Added back REPL tests 2010-06-11 10:03:01 -07:00
Matei Zaharia e58fba279d Fix junk stripper 2010-06-11 01:18:43 -07:00
Matei Zaharia 396f48e5a4 New interpreter port for Scala 2.8 interpreter 2010-06-11 01:10:03 -07:00
Matei Zaharia 4eb39e0c8a New nexus.jar 2010-06-10 22:41:23 -07:00
Matei Zaharia 1473987fb7 Fixed classpath for tests 2010-06-10 22:36:45 -07:00
Matei Zaharia 359e84c585 Use new Nexus API 2010-06-10 22:09:13 -07:00
Matei Zaharia 92246c843b Initial work on 2.8 port 2010-06-10 21:50:55 -07:00
Matei Zaharia c177a546a5 Ignore .DS_Store 2010-06-10 18:08:59 -07:00
Mosharaf Chowdhury 7ab703117a Added timers around BroadcastTest and sendBroadcast. 
Turned OFF saving to HDFS for now for stress tests.
pqOfSources is ordered by least leechers again.
 2010-05-18 20:49:59 -07:00
Mosharaf Chowdhury 2d381c974e Added flush calls after all writeObject calls as well as after creating every 
ObjectOutputStream object.
 2010-05-16 17:06:05 -07:00
Mosharaf Chowdhury e85bb3f04d All ObjectOutputStream objects created before OInputS objects. 2010-05-16 16:24:48 -07:00
Mosharaf Chowdhury 520b594bdf "if (!local) { sendBroadcast }" must be called after all the variables have 
been created /initialized
 2010-05-16 16:20:00 -07:00
Mosharaf Chowdhury 4f0b7eb02d SplitStream was not working in EC2. 
We have turned OFF SSB for now.
 2010-05-16 13:00:47 -07:00
Mosharaf Chowdhury 53a2367c9c - SplitStream working in local machine for single variable broadcast 
- Removed delays before publishing and receiving.
- Commented out some prints.
 2010-05-04 15:32:30 -07:00
root 1c90a32621 Fix native build to use build directory 2010-04-30 22:41:21 +00:00
Mosharaf Chowdhury d0a92571dd - Should work, but not tested yet. 
- Right now, each variable has to come one after another. Within a single
  variable, blocks can come out-of-order.
 2010-04-21 19:48:25 -07:00
Mosharaf Chowdhury c0117f9473 Added flesh to publish/deliver functions. 2010-04-21 02:01:34 -07:00