Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes#6369 from tdas/SPARK-7838 and squashes the following commits:
87d1c7f [Tathagata Das] Addressed comment
37775d8 [Tathagata Das] set scope for kinesis stream
(cherry picked from commit baa89838cc)
Signed-off-by: Andrew Or <andrew@databricks.com>
KinesisReceiver calls worker.run() which is a blocking call (while loop) as per source code of kinesis-client library - https://github.com/awslabs/amazon-kinesis-client/blob/v1.2.1/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java.
This results in infinite loop while calling sparkStreamingContext.stop(stopSparkContext = false, stopGracefully = true) perhaps because ReceiverTracker is never able to register the receiver (it's receiverInfo field is a empty map) causing it to be stuck in infinite loop while waiting for running flag to be set to false.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes#6348 from tdas/SPARK-7788 and squashes the following commits:
2584683 [Tathagata Das] Added receiver id in thread name
6cf1cd4 [Tathagata Das] Made KinesisReceiver.onStart non-blocking
(cherry picked from commit 1c388a9985)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes#6325 from tdas/SPARK-7722 and squashes the following commits:
9ab35b2 [Tathagata Das] Fixed styles in Kinesis
(cherry picked from commit 311fab6f1b)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Lack of default constructor causes deserialization to fail. This occurs only when the AWS credentials are explicitly specified through KinesisUtils.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes#6316 from tdas/SPARK-7787 and squashes the following commits:
248ca5c [Tathagata Das] Fixed serializability
(cherry picked from commit 4b7ff3092c)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
- Updated Kinesis examples to use stable API
- Cleaned up comments, etc.
- Renamed KinesisWordCountProducerASL to KinesisWordProducerASL
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes#6249 from tdas/kinesis-examples and squashes the following commits:
7cc307b [Tathagata Das] More tweaks
f080872 [Tathagata Das] More cleanup
841987f [Tathagata Das] Small update
011cbe2 [Tathagata Das] More fixes
b0d74f9 [Tathagata Das] Updated examples.
(cherry picked from commit 3a6003866a)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>