ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
8764 commits 6 branches 181 tags 397 MiB
Commit graph

8764 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 15829df799 Updated simple skewed group-by to generate multiple keys with each 
hashcode so that Mosharaf's current block implementation works
 2011-01-05 22:31:50 -05:00
Mosharaf Chowdhury 6908fa6c31 Updated BalanceRemaining strategy for a better faster-reducers identification mechanism. 2011-01-05 19:19:52 -08:00
Mosharaf Chowdhury 69845d5648 trackerStrategy can now only be accessed by one thread at a time. 
Previously, synchronized were around individual methods which were working on shared variables and two such methods could be accessed by different threads simultaneously. A source of possible concurrency issues.
 2011-01-05 12:18:02 -08:00
Mosharaf Chowdhury a2c7caeac8 Bug fixes, optimizations, changes to ShuffleClient/Tracker interface and updated BalanceRemainingShuffleTrackerStrategy. 
There is still 1 or 10 bugs in tracker implementations. They are normally found in the last remaining clients.
 - Sometimes ShuffleClients just don't do anything for a while.
 - Sometimes the last ShuffleClient or 2 block on reading block size from ShuffleServer and the program does not proceed at all. This happens for larger shuffle (500000 keys)
 2011-01-05 01:58:37 -08:00
Matei Zaharia c251b5e4d6 Added a skewed reduce test where one reducer gets more keys than the 
others.
 2011-01-04 17:20:13 -05:00
Mosharaf Chowdhury b35ea63c2e Updated BalanceRemainingShuffleTrackerStrategy. Better divisioning of slower and faster reducers, instead of dividing between the fastest and the rest. 2011-01-03 23:12:46 -08:00
Mosharaf Chowdhury 7eb334d97c Bug fix: splitsInRequestBitVector(splitIndex) was wrongly set to false after receiving just one block in Blocked implementations that receive multiple blocks at a time. 2011-01-03 19:05:52 -08:00
Mosharaf Chowdhury 07e778d7fa - Updated Reducer-Tracker communication protocol. 
- Implemented a new tracker strategy for shuffle where if a reducer is too fast its stalled until other catchup. Basic version is working, but more work is necessary.
 2011-01-02 23:49:56 -08:00
Mosharaf Chowdhury 33d59fb206 TrackedCustomBlockedLocalFileShuffle has also been updated. 2010-12-30 18:24:12 -08:00
Mosharaf Chowdhury a30f03eae6 CustomBlockedInMemoryShuffle is receiving multiple blocks after connecting to a mapper instead of just one. 2010-12-30 18:12:32 -08:00
Mosharaf Chowdhury b566be47d7 Bug fix/update: All the shuffle implementations are using consistent config parameters. 2010-12-30 17:52:01 -08:00
Mosharaf Chowdhury 4545df67cf Consumption is delayed until everything has been received. Otherwise it interferes with network performance. 2010-12-30 17:10:20 -08:00
Mosharaf Chowdhury 1e26fb3953 CustomBlockedLocalFileShuffle: reducers are reading multiple blocks per connections instead of just one. 
Sometimes ShuffleServer fails to start for small shuffle data with small block size in local VM. No problem with large block size.
 2010-12-30 13:33:34 -08:00
Mosharaf Chowdhury fb51df0b5b Added a skewed shuffle test example. 
Output per mapper is distributed from 1/numMappers to 1 of numKVPairs.
 2010-12-29 13:50:43 -08:00
Mosharaf Chowdhury 2fefbe17e4 TreeBroadcast is an extended version of ChainedBroadcast with customizable maxDegree per node. maxDegree = 1 is ChainedBroadcast. 2010-12-28 20:18:49 -08:00
Mosharaf Chowdhury 5540a99ab7 ChainedBroadcast is also reading masterHostAddress from config file until #42 is resolved. 2010-12-28 18:53:59 -08:00
Mosharaf Chowdhury b23d337c92 Updating reception stats before consuming. Can create trouble if there is any exception during consumption (less likely,) but this frees up splits that threads can connect to instead of idling around. 2010-12-28 16:08:40 -08:00
Mosharaf Chowdhury 5074e8500a - Implemented TrackedCustomBlockedLocalFileShuffle. 
- Fixed several bugs. (Copy-paste is the bane of coding :|)
 2010-12-28 15:28:43 -08:00
Mosharaf Chowdhury a8411eb59e - Moving common stuff to a separate Shuffle object. 
- Moved ShuffleTrackerStrategy to a separate file.
 2010-12-28 12:19:07 -08:00
Mosharaf Chowdhury 1bc10ba64c Formatting. 2010-12-28 00:47:08 -08:00
Mosharaf Chowdhury 95ebd58d0f - Made sure only one Leaving notification goes to the tracker per ShuffleClient. 
- Why ShuffleClient and ShuffleServerThread are crashing is still unknown.
 2010-12-27 03:25:50 -08:00
Mosharaf Chowdhury 7ac3463ab8 Bug fix: tracker (running in Spark master) wasn't initializing Shuffle object and was using inconsistent config values. 2010-12-27 02:55:45 -08:00
Mosharaf Chowdhury 44c8fb0873 Fixed closing order of of some of the i/o streams. Bugs remain. 2010-12-27 02:08:03 -08:00
Mosharaf Chowdhury f859941062 Merge branch 'mos-shuffle-parallel' into mos-shuffle-tracked 
Conflicts:
	conf/java-opts
 2010-12-25 23:12:59 -08:00
Mosharaf Chowdhury 6c8e9cb2f9 Consolidated shuffle options. 2010-12-25 23:08:53 -08:00
Mosharaf Chowdhury 20eec59f04 Bug fix + formatting. 2010-12-25 22:54:46 -08:00
Mosharaf Chowdhury 90e467206d Tracker framework is in place that supports pluggable tracker strategy. There are several bugs along with performance problems. 
- For larger data shuffle ShuffleServerThread gets "Broken Pipe" and ShuffleClient gets "Connection Reset"
 - There is a bug in the accounting counters of BalanceConnectionsShuffleTrackerStrategy. Some of them go below zero while decrementing which is not supposed to happen.
 2010-12-25 22:45:50 -08:00
Mosharaf Chowdhury ba71b61e40 Reading masterHostAddress from config file until #42 has been resolved. 2010-12-25 10:22:22 -08:00
Mosharaf Chowdhury c1ff210387 Fixed some comments. 2010-12-24 20:05:00 -08:00
Mosharaf Chowdhury 8dc44bfa96 CustomBlockedInMemoryShuffle is an in- memroy implementation of CustomBlockedLFS 2010-12-22 21:06:03 -08:00
Mosharaf Chowdhury a064835808 CustomBlockedLocalFileShuffle has been added. This is essentially ManualBlockedLocalFileShuffle with our servers. 2010-12-22 19:02:20 -08:00
Mosharaf Chowdhury 3447f903da Renamed CustomBlockedLocalFileShuffle to ManualBlockedLocalFileShuffle. 
There will be a new CustomBlockedLocalFileShuffle where 'Custom' will mean ManualBlockedLocalFileShuffle with custom server instead of jetty.
 2010-12-22 17:17:33 -08:00
Mosharaf Chowdhury c484b735bb Bug squashed. CustomParallelInMemoryShuffle is rocking! 
We were serializing one (the wrong) thing, trying to deserialize another (the right thing).
 2010-12-22 17:03:31 -08:00
Mosharaf Chowdhury 23586d3bef Added an in-memory implementation of CustomParalleLFS. There is a serialization/deserialization bug in the implementation. 2010-12-22 16:45:26 -08:00
Mosharaf Chowdhury c4c8f72e98 Fixed an indexing bug in HttpBlockedLocalFileShuffle. It still doesn't work on EC2 with >5 nodes cluster. 2010-12-22 12:48:11 -08:00
Mosharaf Chowdhury a5a8b7048d CustomBlockedLocalFileShuffle has separate consumer thread. 2010-12-22 12:04:12 -08:00
Mosharaf Chowdhury 92d2a9a13a Removed unncessary stuff from HttpParallelLocalFileShuffle 2010-12-22 11:28:50 -08:00
Mosharaf Chowdhury 4ab268ee36 HttpParallelLocalFileShuffle also has a consuming thread. It works on EC2. 2010-12-21 23:50:02 -08:00
Mosharaf Chowdhury 5f7bfbc70e HttpBlockedLocalFileShuffle has also been converted to have per-reducer consumption thread. Works in local mesos, but NOT on EC2 :| 2010-12-21 23:05:32 -08:00
Mosharaf Chowdhury 5f0cdabd40 Added a separate thread to deserialize (1 thread per reducer) in CustomParallelLocalFileShuffle 
Upside: No synchronized blocking on "combiners" variable. 3x faster :)
Downside: Inefficient implementation. Requiring too much temporary data. Approx. 2x increase in memory requirement :( Should be fixed at some point.
 2010-12-21 21:52:37 -08:00
Mosharaf Chowdhury f4d0e917a2 Added all the options to the java-opts file. Tired of writing them for separate runs :| 2010-12-21 18:59:51 -08:00
Mosharaf Chowdhury 6ef17e918b Fixed logging. Again. 2010-12-21 18:49:35 -08:00
Mosharaf Chowdhury f47fb44479 - Divided maxConnections to max[Rx|Tx]Connections. 
- Fixed config param loading bug in CustomParallelLFS
 2010-12-21 17:34:51 -08:00
Mosharaf Chowdhury d92b067350 Fixed log message in CustomParallelLocalFileShuffle that was giving some problem in log processing. 2010-12-21 13:12:15 -08:00
Mosharaf Chowdhury 3b21a5fb26 Code formatting... 2010-12-19 18:03:20 -08:00
Mosharaf Chowdhury 81f78282e1 All shuffle implementations are now in the same place. Time to work on new things. 2010-12-19 14:32:40 -08:00
Mosharaf Chowdhury 272c72b405 Merge branch 'mos-shuffle' into mos-shuffle-parallel 
Conflicts:
	conf/java-opts
	src/scala/spark/BasicLocalFileShuffle.scala
 2010-12-19 14:25:13 -08:00
Mosharaf Chowdhury ca37e7b33d Renamed CustomParallelLocalFileShuffle 2010-12-19 14:22:05 -08:00
Mosharaf Chowdhury 864d202cda Merge branch 'mos-shuffle-parallel-http' into mos-shuffle 
Conflicts:
	conf/java-opts
	src/scala/spark/BlockedLocalFileShuffle.scala
	src/scala/spark/CustomBlockedLocalFileShuffle.scala
	src/scala/spark/HttpBlockedLocalFileShuffle.scala
 2010-12-19 14:08:39 -08:00
Mosharaf Chowdhury 89172fcd69 Renamed this version of BlockedLocalFileShuffle to CustomBlockedLocalFileShuffle. 2010-12-19 14:05:35 -08:00