14d7c1c3e9
## What changes were proposed in this pull request? This builds on top of SPARK-24748 to report 'offset lag' as a custom metrics for Kafka structured streaming source. This lag is the difference between the latest offsets in Kafka the time the metrics is reported (just after a micro-batch completes) and the latest offset Spark has processed. It can be 0 (or close to 0) if spark keeps up with the rate at which messages are ingested into Kafka topics in steady state. This measures how far behind the spark source has fallen behind (per partition) and can aid in tuning the application. ## How was this patch tested? Existing and new unit tests Please review http://spark.apache.org/contributing.html before opening a pull request. Closes #21819 from arunmahadevan/SPARK-24863. Authored-by: Arun Mahadevan <arunm@apache.org> Signed-off-by: hyukjinkwon <gurwls223@apache.org> |
||
|---|---|---|
| .. | ||
| src | ||
| pom.xml | ||