47485a3c2d
### What changes were proposed in this pull request? This PR aims to add support for specifying a user defined initial state for arbitrary structured streaming stateful processing using [flat]MapGroupsWithState operator. ### Why are the changes needed? Users can load previous state of their stateful processing as an initial state instead of redoing the entire processing once again. ### Does this PR introduce _any_ user-facing change? Yes this PR introduces new API ``` def mapGroupsWithState[S: Encoder, U: Encoder]( timeoutConf: GroupStateTimeout, initialState: KeyValueGroupedDataset[K, S])( func: (K, Iterator[V], GroupState[S]) => U): Dataset[U] def flatMapGroupsWithState[S: Encoder, U: Encoder]( outputMode: OutputMode, timeoutConf: GroupStateTimeout, initialState: KeyValueGroupedDataset[K, S])( func: (K, Iterator[V], GroupState[S]) => Iterator[U]) ``` ### How was this patch tested? Through unit tests in FlatMapGroupsWithStateSuite Closes #33093 from rahulsmahadev/flatMapGroupsWithState. Authored-by: Rahul Mahadev <rahul.mahadev@databricks.com> Signed-off-by: Gengliang Wang <gengliang@apache.org> |
||
---|---|---|
.. | ||
benchmarks | ||
src | ||
pom.xml |