spark-instrumented-optimizer/common
Misha Dmitriev 094aa59715 [SPARK-24801][CORE] Avoid memory waste by empty byte[] arrays in SaslEncryption$EncryptedMessage
## What changes were proposed in this pull request?

Initialize SaslEncryption$EncryptedMessage.byteChannel lazily,
so that empty, not yet used instances of ByteArrayWritableChannel
referenced by this field don't use up memory.

I analyzed a heap dump from Yarn Node Manager where this code is used, and found that there are over 40,000 of the above objects in memory, each with a big empty byte[] array. The reason they are all there is because of Netty queued up a large number of messages in memory  before transferTo() is called. There is a small number of netty ChannelOutboundBuffer objects, and then collectively , via linked lists starting from their flushedEntry data fields, they end up referencing over 40K ChannelOutboundBuffer$Entry objects, which ultimately reference EncryptedMessage objects.

## How was this patch tested?

Ran all the tests locally.

Author: Misha Dmitriev <misha@cloudera.com>

Closes #21811 from countmdm/misha/spark-24801.
2018-07-26 22:15:12 -05:00
..
kvstore [SPARK-23103][CORE] Ensure correct sort order for negative values in LevelDB. 2018-01-19 13:32:20 -06:00
network-common [SPARK-24801][CORE] Avoid memory waste by empty byte[] arrays in SaslEncryption$EncryptedMessage 2018-07-26 22:15:12 -05:00
network-shuffle [SPARK-24356][CORE] Duplicate strings in File.path managed by FileSegmentManagedBuffer 2018-06-02 23:07:39 -05:00
network-yarn [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT 2018-01-13 00:37:59 +08:00
sketch [SPARK-23381][CORE] Murmur3 hash generates a different value from other implementations 2018-02-16 17:17:55 -08:00
tags [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT 2018-01-13 00:37:59 +08:00
unsafe [SPARK-23976][CORE] Detect length overflow in UTF8String.concat()/ByteArray.concat() 2018-05-02 10:41:34 +02:00