ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-28464][DOC][SS] Document Kafka source minPartitions option

Browse source 
Adding doc for the kafka source minPartitions option to "Structured Streaming + Kafka Integration Guide"

The text is based on the content in  https://docs.databricks.com/spark/latest/structured-streaming/kafka.html#configuration

Closes #25219 from arunpandianp/SPARK-28464.

Authored-by: Arun Pandian <apandian@groupon.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
Arun Pandian 2019-07-21 13:07:22 -07:00 committed by Dongjoon Hyun
parent 6e65d39576
commit a0a58cf2ef
1 changed files with 10 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
docs/structured-streaming-kafka-integration.md
View file

@ -388,6 +388,16 @@ The following configurations are optional:
<td>streaming and batch</td> <td>streaming and batch</td>
<td>Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume.</td> <td>Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume.</td>
</tr> </tr>
<tr>
<td>minPartitions</td>
<td>int</td>
<td>none</td>
<td>streaming and batch</td>
<td>Minimum number of partitions to read from Kafka.
By default, Spark has a 1-1 mapping of topicPartitions to Spark partitions consuming from Kafka.
If you set this option to a value greater than your topicPartitions, Spark will divvy up large
Kafka partitions to smaller pieces.</td>
</tr>
<tr> <tr>
<td>groupIdPrefix</td> <td>groupIdPrefix</td>
<td>string</td> <td>string</td>