2b89e4aa2e
## What changes were proposed in this pull request? Add `spark.streaming.backpressure.initialRate` to direct Kafka Streams for Kafka 0.8 and 0.10 This is required in order to be able to use backpressure with huge lags, which cannot be processed at once. Without this parameter `DirectKafkaInputDStream` with backpressure enabled would try to get all the possible data from Kafka before adjusting consumption rate ## How was this patch tested? - Tests added to `org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala` and `org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala` - Manual tests on YARN cluster Author: akonopko <alex.konopko@teamaol.com> Author: Alexander Konopko <alexander.konopko@gmail.com> Closes #19431 from akonopko/SPARK-18580-initialrate. |
||
|---|---|---|
| .. | ||
| docker | ||
| docker-integration-tests | ||
| flume | ||
| flume-assembly | ||
| flume-sink | ||
| kafka-0-8 | ||
| kafka-0-8-assembly | ||
| kafka-0-10 | ||
| kafka-0-10-assembly | ||
| kafka-0-10-sql | ||
| kinesis-asl | ||
| kinesis-asl-assembly | ||
| spark-ganglia-lgpl | ||