Mosharaf Chowdhury
fd3fd37383
In-memory version of tracker+blocked shuffle checked in.
2011-01-10 17:13:52 -08:00
Mosharaf Chowdhury
d7081a927f
Updated conf/java-opts
2011-01-09 17:19:26 -08:00
Mosharaf Chowdhury
de676428ac
Merge branch 'mos-shuffle-tracked' of git@github.com:mesos/spark into mos-shuffle-tracked
2011-01-07 01:44:32 -08:00
Mosharaf Chowdhury
ba74169162
Dummy log line added in SparkContext for #45
2011-01-06 21:21:35 -08:00
Mosharaf Chowdhury
e76b4f7155
Turned OFF some log messages due to #45 .
2011-01-06 17:07:14 -08:00
Mosharaf Chowdhury
f5ba0b49c6
Bug fix + default config param values update.
...
Updated some default config parameters for broadcast parameters.
Fixed a concurrency bug in the variable that keeps track of how many receivers have finished.
2011-01-06 15:44:24 -08:00
Matei Zaharia
15829df799
Updated simple skewed group-by to generate multiple keys with each
...
hashcode so that Mosharaf's current block implementation works
2011-01-05 22:31:50 -05:00
Mosharaf Chowdhury
6908fa6c31
Updated BalanceRemaining strategy for a better faster-reducers identification mechanism.
2011-01-05 19:19:52 -08:00
Mosharaf Chowdhury
69845d5648
trackerStrategy can now only be accessed by one thread at a time.
...
Previously, synchronized were around individual methods which were working on shared variables and two such methods could be accessed by different threads simultaneously. A source of possible concurrency issues.
2011-01-05 12:18:02 -08:00
Mosharaf Chowdhury
a2c7caeac8
Bug fixes, optimizations, changes to ShuffleClient/Tracker interface and updated BalanceRemainingShuffleTrackerStrategy.
...
There is still 1 or 10 bugs in tracker implementations. They are normally found in the last remaining clients.
- Sometimes ShuffleClients just don't do anything for a while.
- Sometimes the last ShuffleClient or 2 block on reading block size from ShuffleServer and the program does not proceed at all. This happens for larger shuffle (500000 keys)
2011-01-05 01:58:37 -08:00
Matei Zaharia
c251b5e4d6
Added a skewed reduce test where one reducer gets more keys than the
...
others.
2011-01-04 17:20:13 -05:00
Mosharaf Chowdhury
b35ea63c2e
Updated BalanceRemainingShuffleTrackerStrategy. Better divisioning of slower and faster reducers, instead of dividing between the fastest and the rest.
2011-01-03 23:12:46 -08:00
Mosharaf Chowdhury
7eb334d97c
Bug fix: splitsInRequestBitVector(splitIndex) was wrongly set to false after receiving just one block in Blocked implementations that receive multiple blocks at a time.
2011-01-03 19:05:52 -08:00
Mosharaf Chowdhury
07e778d7fa
- Updated Reducer-Tracker communication protocol.
...
- Implemented a new tracker strategy for shuffle where if a reducer is too fast its stalled until other catchup. Basic version is working, but more work is necessary.
2011-01-02 23:49:56 -08:00
Mosharaf Chowdhury
33d59fb206
TrackedCustomBlockedLocalFileShuffle has also been updated.
2010-12-30 18:24:12 -08:00
Mosharaf Chowdhury
a30f03eae6
CustomBlockedInMemoryShuffle is receiving multiple blocks after connecting to a mapper instead of just one.
2010-12-30 18:12:32 -08:00
Mosharaf Chowdhury
b566be47d7
Bug fix/update: All the shuffle implementations are using consistent config parameters.
2010-12-30 17:52:01 -08:00
Mosharaf Chowdhury
4545df67cf
Consumption is delayed until everything has been received. Otherwise it interferes with network performance.
2010-12-30 17:10:20 -08:00
Mosharaf Chowdhury
1e26fb3953
CustomBlockedLocalFileShuffle: reducers are reading multiple blocks per connections instead of just one.
...
Sometimes ShuffleServer fails to start for small shuffle data with small block size in local VM. No problem with large block size.
2010-12-30 13:33:34 -08:00
Mosharaf Chowdhury
fb51df0b5b
Added a skewed shuffle test example.
...
Output per mapper is distributed from 1/numMappers to 1 of numKVPairs.
2010-12-29 13:50:43 -08:00
Mosharaf Chowdhury
2fefbe17e4
TreeBroadcast is an extended version of ChainedBroadcast with customizable maxDegree per node. maxDegree = 1 is ChainedBroadcast.
2010-12-28 20:18:49 -08:00
Mosharaf Chowdhury
5540a99ab7
ChainedBroadcast is also reading masterHostAddress from config file until #42 is resolved.
2010-12-28 18:53:59 -08:00
Mosharaf Chowdhury
b23d337c92
Updating reception stats before consuming. Can create trouble if there is any exception during consumption (less likely,) but this frees up splits that threads can connect to instead of idling around.
2010-12-28 16:08:40 -08:00
Mosharaf Chowdhury
5074e8500a
- Implemented TrackedCustomBlockedLocalFileShuffle.
...
- Fixed several bugs. (Copy-paste is the bane of coding :|)
2010-12-28 15:28:43 -08:00
Mosharaf Chowdhury
a8411eb59e
- Moving common stuff to a separate Shuffle object.
...
- Moved ShuffleTrackerStrategy to a separate file.
2010-12-28 12:19:07 -08:00
Mosharaf Chowdhury
1bc10ba64c
Formatting.
2010-12-28 00:47:08 -08:00
Mosharaf Chowdhury
95ebd58d0f
- Made sure only one Leaving notification goes to the tracker per ShuffleClient.
...
- Why ShuffleClient and ShuffleServerThread are crashing is still unknown.
2010-12-27 03:25:50 -08:00
Mosharaf Chowdhury
7ac3463ab8
Bug fix: tracker (running in Spark master) wasn't initializing Shuffle object and was using inconsistent config values.
2010-12-27 02:55:45 -08:00
Mosharaf Chowdhury
44c8fb0873
Fixed closing order of of some of the i/o streams. Bugs remain.
2010-12-27 02:08:03 -08:00
Mosharaf Chowdhury
f859941062
Merge branch 'mos-shuffle-parallel' into mos-shuffle-tracked
...
Conflicts:
conf/java-opts
2010-12-25 23:12:59 -08:00
Mosharaf Chowdhury
6c8e9cb2f9
Consolidated shuffle options.
2010-12-25 23:08:53 -08:00
Mosharaf Chowdhury
20eec59f04
Bug fix + formatting.
2010-12-25 22:54:46 -08:00
Mosharaf Chowdhury
90e467206d
Tracker framework is in place that supports pluggable tracker strategy. There are several bugs along with performance problems.
...
- For larger data shuffle ShuffleServerThread gets "Broken Pipe" and ShuffleClient gets "Connection Reset"
- There is a bug in the accounting counters of BalanceConnectionsShuffleTrackerStrategy. Some of them go below zero while decrementing which is not supposed to happen.
2010-12-25 22:45:50 -08:00
Mosharaf Chowdhury
ba71b61e40
Reading masterHostAddress from config file until #42 has been resolved.
2010-12-25 10:22:22 -08:00
Mosharaf Chowdhury
c1ff210387
Fixed some comments.
2010-12-24 20:05:00 -08:00
Mosharaf Chowdhury
8dc44bfa96
CustomBlockedInMemoryShuffle is an in- memroy implementation of CustomBlockedLFS
2010-12-22 21:06:03 -08:00
Mosharaf Chowdhury
a064835808
CustomBlockedLocalFileShuffle has been added. This is essentially ManualBlockedLocalFileShuffle with our servers.
2010-12-22 19:02:20 -08:00
Mosharaf Chowdhury
3447f903da
Renamed CustomBlockedLocalFileShuffle to ManualBlockedLocalFileShuffle.
...
There will be a new CustomBlockedLocalFileShuffle where 'Custom' will mean ManualBlockedLocalFileShuffle with custom server instead of jetty.
2010-12-22 17:17:33 -08:00
Mosharaf Chowdhury
c484b735bb
Bug squashed. CustomParallelInMemoryShuffle is rocking!
...
We were serializing one (the wrong) thing, trying to deserialize another (the right thing).
2010-12-22 17:03:31 -08:00
Mosharaf Chowdhury
23586d3bef
Added an in-memory implementation of CustomParalleLFS. There is a serialization/deserialization bug in the implementation.
2010-12-22 16:45:26 -08:00
Mosharaf Chowdhury
c4c8f72e98
Fixed an indexing bug in HttpBlockedLocalFileShuffle. It still doesn't work on EC2 with >5 nodes cluster.
2010-12-22 12:48:11 -08:00
Mosharaf Chowdhury
a5a8b7048d
CustomBlockedLocalFileShuffle has separate consumer thread.
2010-12-22 12:04:12 -08:00
Mosharaf Chowdhury
92d2a9a13a
Removed unncessary stuff from HttpParallelLocalFileShuffle
2010-12-22 11:28:50 -08:00
Mosharaf Chowdhury
4ab268ee36
HttpParallelLocalFileShuffle also has a consuming thread. It works on EC2.
2010-12-21 23:50:02 -08:00
Mosharaf Chowdhury
5f7bfbc70e
HttpBlockedLocalFileShuffle has also been converted to have per-reducer consumption thread. Works in local mesos, but NOT on EC2 :|
2010-12-21 23:05:32 -08:00
Mosharaf Chowdhury
5f0cdabd40
Added a separate thread to deserialize (1 thread per reducer) in CustomParallelLocalFileShuffle
...
Upside: No synchronized blocking on "combiners" variable. 3x faster :)
Downside: Inefficient implementation. Requiring too much temporary data. Approx. 2x increase in memory requirement :( Should be fixed at some point.
2010-12-21 21:52:37 -08:00
Mosharaf Chowdhury
f4d0e917a2
Added all the options to the java-opts file. Tired of writing them for separate runs :|
2010-12-21 18:59:51 -08:00
Mosharaf Chowdhury
6ef17e918b
Fixed logging. Again.
2010-12-21 18:49:35 -08:00
Mosharaf Chowdhury
f47fb44479
- Divided maxConnections to max[Rx|Tx]Connections.
...
- Fixed config param loading bug in CustomParallelLFS
2010-12-21 17:34:51 -08:00
Mosharaf Chowdhury
d92b067350
Fixed log message in CustomParallelLocalFileShuffle that was giving some problem in log processing.
2010-12-21 13:12:15 -08:00