Mosharaf Chowdhury
07e778d7fa
- Updated Reducer-Tracker communication protocol.
...
- Implemented a new tracker strategy for shuffle where if a reducer is too fast its stalled until other catchup. Basic version is working, but more work is necessary.
2011-01-02 23:49:56 -08:00
Mosharaf Chowdhury
33d59fb206
TrackedCustomBlockedLocalFileShuffle has also been updated.
2010-12-30 18:24:12 -08:00
Mosharaf Chowdhury
a30f03eae6
CustomBlockedInMemoryShuffle is receiving multiple blocks after connecting to a mapper instead of just one.
2010-12-30 18:12:32 -08:00
Mosharaf Chowdhury
4545df67cf
Consumption is delayed until everything has been received. Otherwise it interferes with network performance.
2010-12-30 17:10:20 -08:00
Mosharaf Chowdhury
1e26fb3953
CustomBlockedLocalFileShuffle: reducers are reading multiple blocks per connections instead of just one.
...
Sometimes ShuffleServer fails to start for small shuffle data with small block size in local VM. No problem with large block size.
2010-12-30 13:33:34 -08:00
Mosharaf Chowdhury
b23d337c92
Updating reception stats before consuming. Can create trouble if there is any exception during consumption (less likely,) but this frees up splits that threads can connect to instead of idling around.
2010-12-28 16:08:40 -08:00
Mosharaf Chowdhury
5074e8500a
- Implemented TrackedCustomBlockedLocalFileShuffle.
...
- Fixed several bugs. (Copy-paste is the bane of coding :|)
2010-12-28 15:28:43 -08:00
Mosharaf Chowdhury
7ac3463ab8
Bug fix: tracker (running in Spark master) wasn't initializing Shuffle object and was using inconsistent config values.
2010-12-27 02:55:45 -08:00
Mosharaf Chowdhury
f859941062
Merge branch 'mos-shuffle-parallel' into mos-shuffle-tracked
...
Conflicts:
conf/java-opts
2010-12-25 23:12:59 -08:00
Mosharaf Chowdhury
6c8e9cb2f9
Consolidated shuffle options.
2010-12-25 23:08:53 -08:00
Mosharaf Chowdhury
90e467206d
Tracker framework is in place that supports pluggable tracker strategy. There are several bugs along with performance problems.
...
- For larger data shuffle ShuffleServerThread gets "Broken Pipe" and ShuffleClient gets "Connection Reset"
- There is a bug in the accounting counters of BalanceConnectionsShuffleTrackerStrategy. Some of them go below zero while decrementing which is not supposed to happen.
2010-12-25 22:45:50 -08:00
Mosharaf Chowdhury
c1ff210387
Fixed some comments.
2010-12-24 20:05:00 -08:00
Mosharaf Chowdhury
8dc44bfa96
CustomBlockedInMemoryShuffle is an in- memroy implementation of CustomBlockedLFS
2010-12-22 21:06:03 -08:00
Mosharaf Chowdhury
a064835808
CustomBlockedLocalFileShuffle has been added. This is essentially ManualBlockedLocalFileShuffle with our servers.
2010-12-22 19:02:20 -08:00
Mosharaf Chowdhury
3447f903da
Renamed CustomBlockedLocalFileShuffle to ManualBlockedLocalFileShuffle.
...
There will be a new CustomBlockedLocalFileShuffle where 'Custom' will mean ManualBlockedLocalFileShuffle with custom server instead of jetty.
2010-12-22 17:17:33 -08:00
Mosharaf Chowdhury
23586d3bef
Added an in-memory implementation of CustomParalleLFS. There is a serialization/deserialization bug in the implementation.
2010-12-22 16:45:26 -08:00
Mosharaf Chowdhury
c4c8f72e98
Fixed an indexing bug in HttpBlockedLocalFileShuffle. It still doesn't work on EC2 with >5 nodes cluster.
2010-12-22 12:48:11 -08:00
Mosharaf Chowdhury
a5a8b7048d
CustomBlockedLocalFileShuffle has separate consumer thread.
2010-12-22 12:04:12 -08:00
Mosharaf Chowdhury
92d2a9a13a
Removed unncessary stuff from HttpParallelLocalFileShuffle
2010-12-22 11:28:50 -08:00
Mosharaf Chowdhury
4ab268ee36
HttpParallelLocalFileShuffle also has a consuming thread. It works on EC2.
2010-12-21 23:50:02 -08:00
Mosharaf Chowdhury
5f7bfbc70e
HttpBlockedLocalFileShuffle has also been converted to have per-reducer consumption thread. Works in local mesos, but NOT on EC2 :|
2010-12-21 23:05:32 -08:00
Mosharaf Chowdhury
5f0cdabd40
Added a separate thread to deserialize (1 thread per reducer) in CustomParallelLocalFileShuffle
...
Upside: No synchronized blocking on "combiners" variable. 3x faster :)
Downside: Inefficient implementation. Requiring too much temporary data. Approx. 2x increase in memory requirement :( Should be fixed at some point.
2010-12-21 21:52:37 -08:00
Mosharaf Chowdhury
f4d0e917a2
Added all the options to the java-opts file. Tired of writing them for separate runs :|
2010-12-21 18:59:51 -08:00
Mosharaf Chowdhury
89172fcd69
Renamed this version of BlockedLocalFileShuffle to CustomBlockedLocalFileShuffle.
2010-12-19 14:05:35 -08:00
Mosharaf Chowdhury
ce96d8a7d3
First version of BlockedLocalFileShuffle is in. It works!
2010-12-16 15:15:51 -08:00
Mosharaf Chowdhury
77a4017585
Fixed config param naming in ParallelLocalFileShuffle
2010-12-16 11:42:37 -08:00
Mosharaf Chowdhury
c5483e39f9
- ParallelLocalFileShuffle does NOT use HttpPipelining at all.
...
- Config option related to pipelining has been removed.
- Summary: Basic -> Pipelining / Parallel -> NO pipelining
2010-12-15 22:08:34 -08:00
Mosharaf Chowdhury
56d8a2afa1
- Updated java-opts file of this branch.
...
- Renamed some ParallelLocalFileShuffle config options for clarity.
2010-12-15 20:56:22 -08:00
Mosharaf Chowdhury
7e2d72c328
Multiple connections created at a time. No upper limit on the server side though.
2010-12-04 18:55:55 -08:00
Mosharaf Chowdhury
540a41163f
UseHttpPipelining is 'true' by default.
2010-12-02 19:56:17 -08:00
Mosharaf Chowdhury
0de859fbe2
Enabling/disabling HTTP pipelining is a config option now. Performance tradeoffs are not obvious yet.
2010-12-02 02:32:44 -08:00
Mosharaf Chowdhury
8494b3a4f9
- Added log messages for benchmarking.
...
- Added GroupByTest.scala for benchmarking.
2010-11-27 23:51:43 -08:00
Matei Zaharia
e5fb280ec8
Changed the config files that were included in git to templates which
...
are used to create an initial copy of each config file if the user does
not have one. This way, users won't accidentally commit their changes to
config files to git.
2010-10-16 21:51:25 -07:00
Matei Zaharia
0b195927b6
Improved README and added blank templates for config files.
2010-08-13 18:54:32 -07:00
