3641c3dd69
### What changes were proposed in this pull request? Kafka producers are now closed when `spark.kafka.producer.cache.timeout` reached which could be significant problem when processing big SQL queries. The workaround was to increase `spark.kafka.producer.cache.timeout` to a number where the biggest SQL query can be finished. In this PR I've adapted similar solution which already exists on the consumer side, namely applies Apache Commons Pool on the producer side as well. Main advantages choosing this solution: * Producers are not closed until they're in use * No manual reference counting needed (which may be error prone) * Thread-safe by design * Provides jmx connection to the pool where metrics can be fetched What this PR contains: * Introduced producer side parameters to configure pool * Renamed `InternalKafkaConsumerPool` to `InternalKafkaConnectorPool` and made it abstract * Created 2 implementations from it: `InternalKafkaConsumerPool` and `InternalKafkaProducerPool` * Adapted `CachedKafkaProducer` to use `InternalKafkaProducerPool` * Changed `KafkaDataWriter` and `KafkaDataWriteTask` to release producer even in failure scenario * Added several new tests * Extended `KafkaTest` to clear not only producers but consumers as well * Renamed `InternalKafkaConsumerPoolSuite` to `InternalKafkaConnectorPoolSuite` where only consumer tests are checking the behavior (please see comment for reasoning) What this PR not yet contains(but intended when the main concept is stable): * User facing documentation ### Why are the changes needed? Kafka producer closed after 10 minutes (with default settings). ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing + additional unit tests. Cluster tests being started. Closes #25853 from gaborgsomogyi/SPARK-21869. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> |
||
|---|---|---|
| .. | ||
| avro | ||
| docker | ||
| docker-integration-tests | ||
| kafka-0-10 | ||
| kafka-0-10-assembly | ||
| kafka-0-10-sql | ||
| kafka-0-10-token-provider | ||
| kinesis-asl | ||
| kinesis-asl-assembly | ||
| spark-ganglia-lgpl | ||