spark-instrumented-optimizer/common
Zhang, Liye 96941b12f8 [SPARK-14242][CORE][NETWORK] avoid copy in compositeBuffer for frame decoder
## What changes were proposed in this pull request?
In this patch, we set the initial `maxNumComponents` to `Integer.MAX_VALUE` instead of the default size ( which is 16) when allocating `compositeBuffer` in `TransportFrameDecoder` because `compositeBuffer` will introduce too many memory copies underlying if `compositeBuffer` is with default `maxNumComponents` when the frame size is large (which result in many transport messages). For details, please refer to [SPARK-14242](https://issues.apache.org/jira/browse/SPARK-14242).

## How was this patch tested?
spark unit tests and manual tests.
For manual tests, we can reproduce the performance issue with following code:
`sc.parallelize(Array(1,2,3),3).mapPartitions(a=>Array(new Array[Double](1024 * 1024 * 50)).iterator).reduce((a,b)=> a).length`
It's easy to see the performance gain, both from the running time and CPU usage.

Author: Zhang, Liye <liye.zhang@intel.com>

Closes #12038 from liyezhang556520/spark-14242.
2016-03-31 20:17:52 -07:00
..
network-common [SPARK-14242][CORE][NETWORK] avoid copy in compositeBuffer for frame decoder 2016-03-31 20:17:52 -07:00
network-shuffle [SPARK-14011][CORE][SQL] Enable LineLength Java checkstyle rule 2016-03-21 07:58:57 +00:00
network-yarn [SPARK-13622][YARN] Issue creating level db for YARN shuffle service 2016-03-28 14:10:25 -07:00
sketch [SPARK-14011][CORE][SQL] Enable LineLength Java checkstyle rule 2016-03-21 07:58:57 +00:00
tags [SPARK-13548][BUILD] Move tags and unsafe modules into common 2016-03-01 15:39:13 -08:00
unsafe [SPARK-12181] Check Cached unaligned-access capability before using Unsafe 2016-03-29 17:16:53 -07:00