fceabe2372
### What changes were proposed in this pull request? This PR implemented the proposal per [design doc](https://docs.google.com/document/d/1RfFn2e9o_1uHJ8jFGsSakp-BZMizX1uRrJSybMe2a6M) for SPARK-35779. ### Why are the changes needed? Spark supports dynamic partition filtering that enables reusing parts of the query to skip unnecessary partitions in the larger table during joins. This optimization has proven to be beneficial for star-schema queries which are common in the industry. Unfortunately, dynamic pruning is currently limited to partition pruning during joins and is only supported for built-in v1 sources. As more and more Spark users migrate to Data Source V2, it is important to generalize dynamic filtering and expose it to all v2 connectors. Please, see the design doc for more information on this effort. ### Does this PR introduce _any_ user-facing change? Yes, this PR adds a new optional mix-in interface for `Scan` in Data Source V2. ### How was this patch tested? This PR comes with tests. Closes #32921 from aokolnychyi/dynamic-filtering-wip. Authored-by: Anton Okolnychyi <aokolnychyi@apple.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||