spark-instrumented-optimizer/common
Imran Rashid 16f2c3ea46 [SPARK-6237][NETWORK] Network-layer changes to allow stream upload.
These changes allow an RPCHandler to receive an upload as a stream of
data, without having to buffer the entire message in the FrameDecoder.
The primary use case is for replicating large blocks.  By itself, this change is adding dead-code that is not being used -- it is a step towards SPARK-24296.

Added unit tests for handling streaming data, including successfully sending data, and failures in reading the stream with concurrent requests.

Summary of changes:

* Introduce a new UploadStream RPC which is sent to push a large payload as a stream (in contrast, the pre-existing StreamRequest and StreamResponse RPCs are used for pull-based streaming).
* Generalize RpcHandler.receive() to support requests which contain streams.
* Generalize StreamInterceptor to handle both request and response messages (previously it only handled responses).
* Introduce StdChannelListener to abstract away common logging logic in ChannelFuture listeners.

Author: Imran Rashid <irashid@cloudera.com>

Closes #21346 from squito/upload_stream.
2018-06-26 15:56:58 -07:00
..
kvstore [SPARK-23103][CORE] Ensure correct sort order for negative values in LevelDB. 2018-01-19 13:32:20 -06:00
network-common [SPARK-6237][NETWORK] Network-layer changes to allow stream upload. 2018-06-26 15:56:58 -07:00
network-shuffle [SPARK-24356][CORE] Duplicate strings in File.path managed by FileSegmentManagedBuffer 2018-06-02 23:07:39 -05:00
network-yarn [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT 2018-01-13 00:37:59 +08:00
sketch [SPARK-23381][CORE] Murmur3 hash generates a different value from other implementations 2018-02-16 17:17:55 -08:00
tags [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT 2018-01-13 00:37:59 +08:00
unsafe [SPARK-23976][CORE] Detect length overflow in UTF8String.concat()/ByteArray.concat() 2018-05-02 10:41:34 +02:00