8e2a0bdce7
### What changes were proposed in this pull request? This change adds MapType support for PySpark with Arrow, if using pyarrow >= 2.0.0. ### Why are the changes needed? MapType was previous unsupported with Arrow. ### Does this PR introduce _any_ user-facing change? User can now enable MapType for `createDataFrame()`, `toPandas()` with Arrow optimization, and with Pandas UDFs. ### How was this patch tested? Added new PySpark tests for createDataFrame(), toPandas() and Scalar Pandas UDFs. Closes #30393 from BryanCutler/arrow-add-MapType-SPARK-24554. Authored-by: Bryan Cutler <cutlerb@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> |
||
|---|---|---|
| .. | ||
| _typing | ||
| __init__.py | ||
| conversion.py | ||
| conversion.pyi | ||
| functions.py | ||
| functions.pyi | ||
| group_ops.py | ||
| group_ops.pyi | ||
| map_ops.py | ||
| map_ops.pyi | ||
| serializers.py | ||
| typehints.py | ||
| types.py | ||
| utils.py | ||