spark-instrumented-optimizer/common
Zhang, Liye c4bb02abf2 [SPARK-14290][CORE][NETWORK] avoid significant memory copy in netty's transferTo
## What changes were proposed in this pull request?
When netty transfer data that is not `FileRegion`, data will be in format of `ByteBuf`, If the data is large, there will occur significant performance issue because there is memory copy underlying in `sun.nio.ch.IOUtil.write`, the CPU is 100% used, and network is very low.

In this PR, if data size is large, we will split it into small chunks to call `WritableByteChannel.write()`, so that avoid wasting of memory copy. Because the data can't be written within a single write, and it will call `transferTo` multiple times.

## How was this patch tested?
Spark unit test and manual test.
Manual test:
`sc.parallelize(Array(1,2,3),3).mapPartitions(a=>Array(new Array[Double](1024 * 1024 * 50)).iterator).reduce((a,b)=> a).length`

For more details, please refer to [SPARK-14290](https://issues.apache.org/jira/browse/SPARK-14290)

Author: Zhang, Liye <liye.zhang@intel.com>

Closes #12083 from liyezhang556520/spark-14290.
2016-04-06 16:11:59 -07:00
..
network-common [SPARK-14290][CORE][NETWORK] avoid significant memory copy in netty's transferTo 2016-04-06 16:11:59 -07:00
network-shuffle [SPARK-14355][BUILD] Fix typos in Exception/Testcase/Comments and static analysis results 2016-04-03 18:14:16 -07:00
network-yarn [SPARK-13622][YARN] Issue creating level db for YARN shuffle service 2016-03-28 14:10:25 -07:00
sketch [SPARK-14011][CORE][SQL] Enable LineLength Java checkstyle rule 2016-03-21 07:58:57 +00:00
tags [SPARK-13548][BUILD] Move tags and unsafe modules into common 2016-03-01 15:39:13 -08:00
unsafe [SPARK-13992] Add support for off-heap caching 2016-04-01 14:34:59 -07:00