ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/external
History
Yuanjian Li 36c24a03bd [SPARK-35312][SS][FOLLOW-UP] More documents and checking logic for the new options 
### What changes were proposed in this pull request?
Add more documents and checking logic for the new options `minOffsetPerTrigger` and `maxTriggerDelay`.

### Why are the changes needed?
Have a clear description of the behavior introduced in SPARK-35312

### Does this PR introduce _any_ user-facing change?
Yes.
If the user set minOffsetsPerTrigger > maxOffsetsPerTrigger, the new code will throw an AnalysisException. The original behavior is to ignore the maxOffsetsPerTrigger silenctly.

### How was this patch tested?
Existing tests.

Closes #33792 from xuanyuanking/SPARK-35312-follow.

Authored-by: Yuanjian Li <yuanjian.li@databricks.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
(cherry picked from commit a0b24019ed)
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
2021-08-20 10:41:54 +09:00
..
avro Preparing development version 3.2.1-SNAPSHOT 2021-08-19 14:08:32 +00:00
docker [SPARK-32353][TEST] Update docker/spark-test and clean up unused stuff 2020-07-17 12:05:45 -07:00
docker-integration-tests Preparing development version 3.2.1-SNAPSHOT 2021-08-19 14:08:32 +00:00
kafka-0-10 Preparing development version 3.2.1-SNAPSHOT 2021-08-19 14:08:32 +00:00
kafka-0-10-assembly Preparing development version 3.2.1-SNAPSHOT 2021-08-19 14:08:32 +00:00
kafka-0-10-sql [SPARK-35312][SS][FOLLOW-UP] More documents and checking logic for the new options 2021-08-20 10:41:54 +09:00
kafka-0-10-token-provider Preparing development version 3.2.1-SNAPSHOT 2021-08-19 14:08:32 +00:00
kinesis-asl Preparing development version 3.2.1-SNAPSHOT 2021-08-19 14:08:32 +00:00
kinesis-asl-assembly Preparing development version 3.2.1-SNAPSHOT 2021-08-19 14:08:32 +00:00
spark-ganglia-lgpl Preparing development version 3.2.1-SNAPSHOT 2021-08-19 14:08:32 +00:00