928d631625
We will do checkpoint when generating a batch and completing a batch. When the processing time of a batch is greater than the batch interval, checkpointing for completing an old batch may run after checkpointing for generating a new batch. If this happens, checkpoint of an old batch actually has the latest information, so we want to recovery from it. This PR will use the latest checkpoint time as the file name, so that we can always recovery from the latest checkpoint file. Author: Shixiong Zhu <shixiong@databricks.com> Closes #9707 from zsxwing/fix-checkpoint. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| context.py | ||
| dstream.py | ||
| flume.py | ||
| kafka.py | ||
| kinesis.py | ||
| listener.py | ||
| mqtt.py | ||
| tests.py | ||
| util.py | ||