ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-35311][SS][UI][DOCS] Structured Streaming Web UI state information documentation

Browse source 
### What changes were proposed in this pull request?
In this PR I'm adding Structured Streaming Web UI state information documentation.

### Why are the changes needed?
Missing documentation.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
```
cd docs/
SKIP_API=1 bundle exec jekyll build
```
Manual webpage check.

Closes #32433 from gaborgsomogyi/SPARK-35311.

Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
This commit is contained in:
Gabor Somogyi 2021-05-14 10:40:12 +09:00 committed by Jungtaek Lim
parent 8fa739fb9d
commit b6a0a7ea53
2 changed files with 6 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
docs/img/webui-structured-streaming-detail2.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 353 KiB

6
docs/web-ui.md
View file

@ -415,6 +415,7 @@ click a "run id" in the tables.
<p style="text-align: center;">
<img src="img/webui-structured-streaming-detail.png" title="Structured Streaming Query Statistics" alt="Structured Streaming Query Statistics">
<img src="img/webui-structured-streaming-detail2.png">
</p>
The statistics page displays some useful metrics for insight into the status of your streaming
@ -431,6 +432,11 @@ The tracked operations are listed as follows.
* latestOffset & getOffset: Time taken to query the maximum available offset for this source.
* queryPlanning: Time taken to generates the execution plan.
* walCommit: Time taken to write the offsets to the metadata log.
* **Global Watermark Gap.** The gap between batch timestamp and global watermark for the batch.
* **Aggregated Number Of Total State Rows.** The aggregated number of total state rows.
* **Aggregated Number Of Updated State Rows.** The aggregated number of updated state rows.
* **Aggregated State Memory Used In Bytes.** The aggregated state memory used in bytes.
* **Aggregated Number Of State Rows Dropped By Watermark.** The aggregated number of state rows dropped by watermark.
As an early-release version, the statistics page is still under development and will be improved in
future releases.