spark-instrumented-optimizer/common
Kazuaki Ishizaki 9215ee7a16 [SPARK-23976][CORE] Detect length overflow in UTF8String.concat()/ByteArray.concat()
## What changes were proposed in this pull request?

This PR detects length overflow if total elements in inputs are not acceptable.

For example, when the three inputs has `0x7FFF_FF00`, `0x7FFF_FF00`, and `0xE00`, we should detect length overflow since we cannot allocate such a large structure on `byte[]`.
On the other hand, the current algorithm can allocate the result structure with `0x1000`-byte length due to integer sum overflow.

## How was this patch tested?

Existing UTs.
If we would create UTs, we need large heap (6-8GB). It may make test environment unstable.
If it is necessary to create UTs, I will create them.

Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com>

Closes #21064 from kiszk/SPARK-23976.
2018-05-02 10:41:34 +02:00
..
kvstore [SPARK-23103][CORE] Ensure correct sort order for negative values in LevelDB. 2018-01-19 13:32:20 -06:00
network-common [SPARK-24029][CORE] Follow up: set SO_REUSEADDR on the server socket. 2018-04-24 09:10:29 +08:00
network-shuffle [SPARK-23289][CORE] OneForOneBlockFetcher.DownloadCallback.onData should write the buffer fully 2018-02-01 21:00:47 +08:00
network-yarn [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT 2018-01-13 00:37:59 +08:00
sketch [SPARK-23381][CORE] Murmur3 hash generates a different value from other implementations 2018-02-16 17:17:55 -08:00
tags [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT 2018-01-13 00:37:59 +08:00
unsafe [SPARK-23976][CORE] Detect length overflow in UTF8String.concat()/ByteArray.concat() 2018-05-02 10:41:34 +02:00