[DOCS] Added important updateStateByKey details
Runs for *all* existing keys and returning "None" will remove the key-value pair. Author: Michael Vogiatzis <michaelvogiatzis@gmail.com> Closes #7229 from mvogiatzis/patch-1 and squashes the following commits: e7a2946 [Michael Vogiatzis] Updated updateStateByKey text 00283ed [Michael Vogiatzis] Removed space c2656f9 [Michael Vogiatzis] Moved description farther up 0a42551 [Michael Vogiatzis] Added important updateStateByKey details
This commit is contained in:
parent
1903641e68
commit
d538919cc4
|
@ -854,6 +854,8 @@ it with new information. To use this, you will have to do two steps.
|
|||
1. Define the state update function - Specify with a function how to update the state using the
|
||||
previous state and the new values from an input stream.
|
||||
|
||||
In every batch, Spark will apply the state update function for all existing keys, regardless of whether they have new data in a batch or not. If the update function returns `None` then the key-value pair will be eliminated.
|
||||
|
||||
Let's illustrate this with an example. Say you want to maintain a running count of each word
|
||||
seen in a text data stream. Here, the running count is the state and it is an integer. We
|
||||
define the update function as:
|
||||
|
|
Loading…
Reference in a new issue