ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/external
History
Takeshi YAMAMURO 256a3a8013 [SPARK-18020][STREAMING][KINESIS] Checkpoint SHARD_END to finish reading closed shards 
## What changes were proposed in this pull request?
This pr is to fix an issue occurred when resharding Kinesis streams; the resharding makes the KCL throw an exception because Spark does not checkpoint `SHARD_END` when finishing reading closed shards in `KinesisRecordProcessor#shutdown`. This bug finally leads to stopping subscribing new split (or merged) shards.

## How was this patch tested?
Added a test in `KinesisStreamSuite` to check if it works well when splitting/merging shards.

Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>

Closes #16213 from maropu/SPARK-18020.
2017-01-25 17:38:48 -08:00
..
docker [SPARK-13595][BUILD] Move docker, extras modules into external 2016-03-09 18:27:44 +00:00
docker-integration-tests [SPARK-14536][SQL] fix to handle null value in array type column for postgres. 2017-01-20 19:23:20 -08:00
flume [SPARK-17807][CORE] split test-tags into test-JAR 2016-12-21 16:37:20 -08:00
flume-assembly [SPARK-18695] Bump master branch version to 2.2.0-SNAPSHOT 2016-12-02 21:09:37 -08:00
flume-sink [MINOR][DOCS] Remove consecutive duplicated words/typo in Spark Repo 2017-01-04 15:07:29 +00:00
java8-tests [SPARK-17807][CORE] split test-tags into test-JAR 2016-12-21 16:37:20 -08:00
kafka-0-8 [SPARK-18922][SQL][CORE][STREAMING][TESTS] Fix all identified tests failed due to path and resource-not-closed problems on Windows 2017-01-10 13:19:21 +00:00
kafka-0-8-assembly [SPARK-18695] Bump master branch version to 2.2.0-SNAPSHOT 2016-12-02 21:09:37 -08:00
kafka-0-10 [SPARK-19227][SPARK-19251] remove unused imports and outdated comments 2017-01-18 09:44:32 +00:00
kafka-0-10-assembly [SPARK-18695] Bump master branch version to 2.2.0-SNAPSHOT 2016-12-02 21:09:37 -08:00
kafka-0-10-sql [SPARK-19227][SPARK-19251] remove unused imports and outdated comments 2017-01-18 09:44:32 +00:00
kinesis-asl [SPARK-18020][STREAMING][KINESIS] Checkpoint SHARD_END to finish reading closed shards 2017-01-25 17:38:48 -08:00
kinesis-asl-assembly [SPARK-18695] Bump master branch version to 2.2.0-SNAPSHOT 2016-12-02 21:09:37 -08:00
spark-ganglia-lgpl [SPARK-18695] Bump master branch version to 2.2.0-SNAPSHOT 2016-12-02 21:09:37 -08:00