ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/extras
History
Tathagata Das 130ec219aa [SPARK-7788] Made KinesisReceiver.onStart() non-blocking 
KinesisReceiver calls worker.run() which is a blocking call (while loop) as per source code of kinesis-client library - https://github.com/awslabs/amazon-kinesis-client/blob/v1.2.1/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java.
This results in infinite loop while calling sparkStreamingContext.stop(stopSparkContext = false, stopGracefully = true) perhaps because ReceiverTracker is never able to register the receiver (it's receiverInfo field is a empty map) causing it to be stuck in infinite loop while waiting for running flag to be set to false.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #6348 from tdas/SPARK-7788 and squashes the following commits:

2584683 [Tathagata Das] Added receiver id in thread name
6cf1cd4 [Tathagata Das] Made KinesisReceiver.onStart non-blocking

(cherry picked from commit 1c388a9985)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
2015-05-22 17:39:09 -07:00
..
java8-tests Preparing development version 1.4.0-SNAPSHOT 2015-05-20 17:29:00 -07:00
kinesis-asl [SPARK-7788] Made KinesisReceiver.onStart() non-blocking 2015-05-22 17:39:09 -07:00
spark-ganglia-lgpl Preparing development version 1.4.0-SNAPSHOT 2015-05-20 17:29:00 -07:00
README.md [java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs 2014-03-03 22:31:30 -08:00

README.md

This directory contains build components not included by default in Spark's build.