[SPARK-24938][CORE] Prevent Netty from using onheap memory for headers without regard for configuration
## What changes were proposed in this pull request? In MessageEncoder.java, the header would always be allocated on onheap memory regardless of whether netty was configured to use/prefer onheap or offheap. By default this made netty allocate 16mb of onheap memory for a tiny header message. It would be more practical to use preallocated buffers. Using a memory monitor tool on a simple spark application, the following services currently allocate 16 mb of onheap memory: netty-rpc-client netty-blockTransfer-client netty-external-shuffle-client With this change, the memory monitor tool reports all three of these services as using 0 b of onheap memory. The offheap memory allocation does not increase, but more of the already-allocated space is used. ## How was this patch tested? Manually tested change using spark-memory-tool https://github.com/squito/spark-memory Closes #22114 from NiharS/nettybuffer. Lead-authored-by: Nihar Sheth <niharrsheth@gmail.com> Co-authored-by: Nihar Sheth <nsheth@cloudera.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
This commit is contained in:
parent
7bf0794651
commit
055bf8ea1f
|
@ -79,7 +79,7 @@ public final class MessageEncoder extends MessageToMessageEncoder<Message> {
|
||||||
// sent.
|
// sent.
|
||||||
int headerLength = 8 + msgType.encodedLength() + in.encodedLength();
|
int headerLength = 8 + msgType.encodedLength() + in.encodedLength();
|
||||||
long frameLength = headerLength + (isBodyInFrame ? bodyLength : 0);
|
long frameLength = headerLength + (isBodyInFrame ? bodyLength : 0);
|
||||||
ByteBuf header = ctx.alloc().heapBuffer(headerLength);
|
ByteBuf header = ctx.alloc().buffer(headerLength);
|
||||||
header.writeLong(frameLength);
|
header.writeLong(frameLength);
|
||||||
msgType.encode(header);
|
msgType.encode(header);
|
||||||
in.encode(header);
|
in.encode(header);
|
||||||
|
|
Loading…
Reference in a new issue