ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/python/pyspark/sql
History
itholic 03e6de2abe [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame 
### What changes were proposed in this pull request?

This PR proposes move `to_pandas_on_spark` function from `pyspark.pandas.frame` to `pyspark.sql.dataframe`, and added the related tests to the PySpark DataFrame tests.

### Why are the changes needed?

Because now the Koalas is ported into PySpark, so we don't need to Spark auto-patch anymore.
And also `to_pandas_on_spark` is belongs to the pandas-on-Spark DataFrame doesn't look make sense.

### Does this PR introduce _any_ user-facing change?

No, it's kinda internal refactoring stuff.

### How was this patch tested?

Added the related tests and manually check they're passed.

Closes #33054 from itholic/SPARK-35605.

Authored-by: itholic <haejoon.lee@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
2021-06-28 11:47:09 +09:00
..
avro [SPARK-34300][PYSPARK][DOCS][MINOR] Fix some typos and syntax issues in docstrings and output of dev/lint-python 2021-02-02 09:30:50 +09:00
pandas [SPARK-34803][PYSPARK] Pass the raised ImportError if pandas or pyarrow fail to import 2021-03-22 23:29:28 +09:00
tests [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame 2021-06-28 11:47:09 +09:00
__init__.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
__init__.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
_typing.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
catalog.py [SPARK-33730][PYTHON] Standardize warning types 2021-01-18 09:32:55 +09:00
catalog.pyi [SPARK-35019][PYTHON][SQL] Fix type hints mismatches in pyspark.sql.* 2021-04-13 11:21:13 +09:00
column.py [SPARK-33730][PYTHON] Standardize warning types 2021-01-18 09:32:55 +09:00
column.pyi [SPARK-34630][PYTHON][SQL] Added typehint for pyspark.sql.Column.contains 2021-03-24 15:21:19 +01:00
conf.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
conf.pyi [SPARK-35019][PYTHON][SQL] Fix type hints mismatches in pyspark.sql.* 2021-04-13 11:21:13 +09:00
context.py [MINOR][DOCS] Avoid some python docs where first sentence has "e.g." or similar 2021-05-12 10:38:59 +09:00
context.pyi [SPARK-35019][PYTHON][SQL] Fix type hints mismatches in pyspark.sql.* 2021-04-13 11:21:13 +09:00
dataframe.py [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame 2021-06-28 11:47:09 +09:00
dataframe.pyi [SPARK-35605][PYTHON] Move to_pandas_on_spark to the Spark DataFrame 2021-06-28 11:47:09 +09:00
functions.py [SPARK-35643][PYTHON] Fix ambiguous reference in functions.py column() 2021-06-05 12:40:39 +09:00
functions.pyi [SPARK-35418][SQL] Add sentences function to functions.{scala,py} 2021-05-19 20:07:28 +09:00
group.py [SPARK-33250][PYTHON][DOCS] Migration to NumPy documentation style in SQL (pyspark.sql.*) 2020-11-03 10:00:49 +09:00
group.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
readwriter.py [SPARK-35583][DOCS] Move JDBC data source options from Python and Scala into a single page 2021-06-02 14:21:16 +09:00
readwriter.pyi [SPARK-33566][CORE][SQL][SS][PYTHON] Make unescapedQuoteHandling option configurable when read CSV 2020-11-27 15:47:39 +09:00
session.py [MINOR][DOCS] Avoid some python docs where first sentence has "e.g." or similar 2021-05-12 10:38:59 +09:00
session.pyi [SPARK-33457][PYTHON] Adjust mypy configuration 2020-11-25 09:27:04 +09:00
streaming.py [SPARK-35433][DOCS] Move CSV data source options from Python and Scala into a single page 2021-06-01 10:58:49 +09:00
streaming.pyi [SPARK-33836][SS][PYTHON] Expose DataStreamReader.table and DataStreamWriter.toTable 2020-12-21 19:42:59 +09:00
types.py [SPARK-32194][PYTHON] Use proper exception classes instead of plain Exception 2021-05-26 11:54:40 +09:00
types.pyi [SPARK-33457][PYTHON] Adjust mypy configuration 2020-11-25 09:27:04 +09:00
udf.py [SPARK-34408][PYTHON] Refactor spark.udf.register to share the same path to generate UDF instance 2021-02-11 10:57:02 +09:00
udf.pyi [SPARK-33457][PYTHON] Adjust mypy configuration 2020-11-25 09:27:04 +09:00
utils.py Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
window.py [SPARK-33250][PYTHON][DOCS] Migration to NumPy documentation style in SQL (pyspark.sql.*) 2020-11-03 10:00:49 +09:00
window.pyi [SPARK-33250][PYTHON][DOCS] Migration to NumPy documentation style in SQL (pyspark.sql.*) 2020-11-03 10:00:49 +09:00