6506616b97
## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/22104 , we create the python-eval nodes at the end of the optimization phase, which causes a problem. After the main optimization batch, Filter and Project nodes are usually pushed to the bottom, near the scan node. However, if we extract Python UDFs from Filter/Project, and create a python-eval node under Filter/Project, it will break column pruning/filter pushdown of the scan node. There are some hacks in the `ExtractPythonUDFs` rule, to duplicate the column pruning and filter pushdown logic. However, it has some bugs as demonstrated in the new test case(only column pruning is broken). This PR removes the hacks and re-apply the column pruning and filter pushdown rules explicitly. **Before:** ``` ... == Analyzed Logical Plan == a: bigint Project [a#168L] +- Filter dummyUDF(a#168L) +- Relation[a#168L,b#169L] parquet == Optimized Logical Plan == Project [a#168L] +- Project [a#168L, b#169L] +- Filter pythonUDF0#174: boolean +- BatchEvalPython [dummyUDF(a#168L)], [a#168L, b#169L, pythonUDF0#174] +- Relation[a#168L,b#169L] parquet == Physical Plan == *(2) Project [a#168L] +- *(2) Project [a#168L, b#169L] +- *(2) Filter pythonUDF0#174: boolean +- BatchEvalPython [dummyUDF(a#168L)], [a#168L, b#169L, pythonUDF0#174] +- *(1) FileScan parquet [a#168L,b#169L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/spark-798bae3c-a2..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint> ``` **After:** ``` ... == Analyzed Logical Plan == a: bigint Project [a#168L] +- Filter dummyUDF(a#168L) +- Relation[a#168L,b#169L] parquet == Optimized Logical Plan == Project [a#168L] +- Filter pythonUDF0#174: boolean +- BatchEvalPython [dummyUDF(a#168L)], [pythonUDF0#174] +- Project [a#168L] +- Relation[a#168L,b#169L] parquet == Physical Plan == *(2) Project [a#168L] +- *(2) Filter pythonUDF0#174: boolean +- BatchEvalPython [dummyUDF(a#168L)], [pythonUDF0#174] +- *(1) FileScan parquet [a#168L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/spark-9500cafb-78..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint> ``` ## How was this patch tested? new test Closes #24675 from cloud-fan/python. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||