Imran Rashid 7e847646d1 [SPARK-24307][CORE] Support reading remote cached partitions > 2gb ... (1) Netty's ByteBuf cannot support data > 2gb. So to transfer data from a ChunkedByteBuffer over the network, we use a custom version of FileRegion which is backed by the ChunkedByteBuffer. (2) On the receiving end, we need to expose all the data in a FileSegmentManagedBuffer as a ChunkedByteBuffer. We do that by memory mapping the entire file in chunks. Added unit tests. Ran the randomized test a couple of hundred times on my laptop. Tests cover the equivalent of SPARK-24107 for the ChunkedByteBufferFileRegion. Also tested on a cluster with remote cache reads >2gb (in memory and on disk). Author: Imran Rashid <irashid@cloudera.com> Closes #21440 from squito/chunked_bb_file_region.		2018-07-20 11:16:53 +08:00
..
java	[SPARK-24564][TEST] Add test suite for RecordBinaryComparator	2018-06-28 14:19:50 +08:00
resources	[SPARK-23189][CORE][WEB UI] Reflect stage level blacklisting on executor tab	2018-02-13 09:54:52 -06:00
scala/org/apache	[SPARK-24307][CORE] Support reading remote cached partitions > 2gb	2018-07-20 11:16:53 +08:00