8bd8f492ea
### What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-29500 `KafkaRowWriter` now supports setting the Kafka partition by reading a "partition" column in the input dataframe. Code changes in commit nr. 1. Test changes in commit nr. 2. Doc changes in commit nr. 3. tcondie dongjinleekr srowen ### Why are the changes needed? While it is possible to configure a custom Kafka Partitioner with `.option("kafka.partitioner.class", "my.custom.Partitioner")`, this is not enough for certain use cases. See the Jira issue. ### Does this PR introduce any user-facing change? No, as this behaviour is optional. ### How was this patch tested? Two new UT were added and one was updated. Closes #26153 from redsk/feature/SPARK-29500. Authored-by: redsk <nicola.bova@gmail.com> Signed-off-by: Sean Owen <sean.owen@databricks.com> |
||
|---|---|---|
| .. | ||
| avro | ||
| docker | ||
| docker-integration-tests | ||
| kafka-0-10 | ||
| kafka-0-10-assembly | ||
| kafka-0-10-sql | ||
| kafka-0-10-token-provider | ||
| kinesis-asl | ||
| kinesis-asl-assembly | ||
| spark-ganglia-lgpl | ||