spark-instrumented-optimizer/project
Imran Rashid 16f2c3ea46 [SPARK-6237][NETWORK] Network-layer changes to allow stream upload.
These changes allow an RPCHandler to receive an upload as a stream of
data, without having to buffer the entire message in the FrameDecoder.
The primary use case is for replicating large blocks.  By itself, this change is adding dead-code that is not being used -- it is a step towards SPARK-24296.

Added unit tests for handling streaming data, including successfully sending data, and failures in reading the stream with concurrent requests.

Summary of changes:

* Introduce a new UploadStream RPC which is sent to push a large payload as a stream (in contrast, the pre-existing StreamRequest and StreamResponse RPCs are used for pull-based streaming).
* Generalize RpcHandler.receive() to support requests which contain streams.
* Generalize StreamInterceptor to handle both request and response messages (previously it only handled responses).
* Introduce StdChannelListener to abstract away common logging logic in ChannelFuture listeners.

Author: Imran Rashid <irashid@cloudera.com>

Closes #21346 from squito/upload_stream.
2018-06-26 15:56:58 -07:00
..
build.properties [SPARK-24419][BUILD] Upgrade SBT to 0.13.17 with Scala 2.10.7 for JDK9+ 2018-05-30 05:18:18 -07:00
MimaBuild.scala [SPARK-23070] Bump previousSparkVersion in MimaBuild.scala to be 2.2.0 2018-01-15 22:32:38 +08:00
MimaExcludes.scala [SPARK-6237][NETWORK] Network-layer changes to allow stream upload. 2018-06-26 15:56:58 -07:00
plugins.sbt [SPARK-22269][BUILD] Run Java linter via SBT for Jenkins 2018-05-24 14:19:32 +08:00
SparkBuild.scala [SPARK-23010][BUILD][FOLLOWUP] Fix java checkstyle failure of kubernetes-integration-tests 2018-06-12 15:57:43 -07:00