6506616b97
## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/22104 , we create the python-eval nodes at the end of the optimization phase, which causes a problem. After the main optimization batch, Filter and Project nodes are usually pushed to the bottom, near the scan node. However, if we extract Python UDFs from Filter/Project, and create a python-eval node under Filter/Project, it will break column pruning/filter pushdown of the scan node. There are some hacks in the `ExtractPythonUDFs` rule, to duplicate the column pruning and filter pushdown logic. However, it has some bugs as demonstrated in the new test case(only column pruning is broken). This PR removes the hacks and re-apply the column pruning and filter pushdown rules explicitly. **Before:** ``` ... == Analyzed Logical Plan == a: bigint Project [a#168L] +- Filter dummyUDF(a#168L) +- Relation[a#168L,b#169L] parquet == Optimized Logical Plan == Project [a#168L] +- Project [a#168L, b#169L] +- Filter pythonUDF0#174: boolean +- BatchEvalPython [dummyUDF(a#168L)], [a#168L, b#169L, pythonUDF0#174] +- Relation[a#168L,b#169L] parquet == Physical Plan == *(2) Project [a#168L] +- *(2) Project [a#168L, b#169L] +- *(2) Filter pythonUDF0#174: boolean +- BatchEvalPython [dummyUDF(a#168L)], [a#168L, b#169L, pythonUDF0#174] +- *(1) FileScan parquet [a#168L,b#169L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/spark-798bae3c-a2..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint> ``` **After:** ``` ... == Analyzed Logical Plan == a: bigint Project [a#168L] +- Filter dummyUDF(a#168L) +- Relation[a#168L,b#169L] parquet == Optimized Logical Plan == Project [a#168L] +- Filter pythonUDF0#174: boolean +- BatchEvalPython [dummyUDF(a#168L)], [pythonUDF0#174] +- Project [a#168L] +- Relation[a#168L,b#169L] parquet == Physical Plan == *(2) Project [a#168L] +- *(2) Filter pythonUDF0#174: boolean +- BatchEvalPython [dummyUDF(a#168L)], [pythonUDF0#174] +- *(1) FileScan parquet [a#168L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/_1/bzcp960d0hlb988k90654z2w0000gp/T/spark-9500cafb-78..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint> ``` ## How was this patch tested? new test Closes #24675 from cloud-fan/python. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> |
||
|---|---|---|
| .. | ||
| catalyst | ||
| core | ||
| hive | ||
| hive-thriftserver | ||
| create-docs.sh | ||
| gen-sql-markdown.py | ||
| mkdocs.yml | ||
| README.md | ||
Spark SQL
This module provides support for executing relational queries expressed in either SQL or the DataFrame/Dataset API.
Spark SQL is broken up into four subprojects:
- Catalyst (sql/catalyst) - An implementation-agnostic framework for manipulating trees of relational operators and expressions.
- Execution (sql/core) - A query planner / execution engine for translating Catalyst's logical query plans into Spark RDDs. This component also includes a new public interface, SQLContext, that allows users to execute SQL or LINQ statements against existing RDDs and Parquet files.
- Hive Support (sql/hive) - Includes an extension of SQLContext called HiveContext that allows users to write queries using a subset of HiveQL and access data from a Hive Metastore using Hive SerDes. There are also wrappers that allow users to run queries that include Hive UDFs, UDAFs, and UDTFs.
- HiveServer and CLI support (sql/hive-thriftserver) - Includes support for the SQL CLI (bin/spark-sql) and a HiveServer2 (for JDBC/ODBC) compatible server.
Running sql/create-docs.sh generates SQL documentation for built-in functions under sql/site.