[SPARK-28464][DOC][SS] Document Kafka source minPartitions option
Adding doc for the kafka source minPartitions option to "Structured Streaming + Kafka Integration Guide" The text is based on the content in https://docs.databricks.com/spark/latest/structured-streaming/kafka.html#configuration Closes #25219 from arunpandianp/SPARK-28464. Authored-by: Arun Pandian <apandian@groupon.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
parent
6e65d39576
commit
a0a58cf2ef
|
@ -388,6 +388,16 @@ The following configurations are optional:
|
|||
<td>streaming and batch</td>
|
||||
<td>Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>minPartitions</td>
|
||||
<td>int</td>
|
||||
<td>none</td>
|
||||
<td>streaming and batch</td>
|
||||
<td>Minimum number of partitions to read from Kafka.
|
||||
By default, Spark has a 1-1 mapping of topicPartitions to Spark partitions consuming from Kafka.
|
||||
If you set this option to a value greater than your topicPartitions, Spark will divvy up large
|
||||
Kafka partitions to smaller pieces.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>groupIdPrefix</td>
|
||||
<td>string</td>
|
||||
|
|
Loading…
Reference in a new issue