209b9361ac
## What changes were proposed in this pull request? This change uses Arrow to optimize the creation of a Spark DataFrame from a Pandas DataFrame. The input df is sliced according to the default parallelism. The optimization is enabled with the existing conf "spark.sql.execution.arrow.enabled" and is disabled by default. ## How was this patch tested? Added new unit test to create DataFrame with and without the optimization enabled, then compare results. Author: Bryan Cutler <cutlerb@gmail.com> Author: Takuya UESHIN <ueshin@databricks.com> Closes #19459 from BryanCutler/arrow-createDataFrame-from_pandas-SPARK-20791. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| catalog.py | ||
| column.py | ||
| conf.py | ||
| context.py | ||
| dataframe.py | ||
| functions.py | ||
| group.py | ||
| readwriter.py | ||
| session.py | ||
| streaming.py | ||
| tests.py | ||
| types.py | ||
| utils.py | ||
| window.py | ||