94fc0e3235
### What changes were proposed in this pull request? File source V2: support partition pruning. Note: subquery predicates are not pushed down for partition pruning even after this PR, due to the limitation for the current data source V2 API and framework. The rule `PlanSubqueries` requires the subquery expression to be in the children or class parameters in `SparkPlan`, while the condition is not satisfied for `BatchScanExec`. ### Why are the changes needed? It's important for reading performance. ### Does this PR introduce any user-facing change? No ### How was this patch tested? New unit tests for all the V2 file sources Closes #27112 from gengliangwang/PartitionPruningInFileScan. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||