spark-instrumented-optimizer/python/pyspark/sql
Bryan Cutler d8e825e3bc [SPARK-22106][PYSPARK][SQL] Disable 0-parameter pandas_udf and add doctests
## What changes were proposed in this pull request?

This change disables the use of 0-parameter pandas_udfs due to the API being overly complex and awkward, and can easily be worked around by using an index column as an input argument.  Also added doctests for pandas_udfs which revealed bugs for handling empty partitions and using the pandas_udf decorator.

## How was this patch tested?

Reworked existing 0-parameter test to verify error is raised, added doctest for pandas_udf, added new tests for empty partition and decorator usage.

Author: Bryan Cutler <cutlerb@gmail.com>

Closes #19325 from BryanCutler/arrow-pandas_udf-0-param-remove-SPARK-22106.
2017-09-26 10:54:00 +09:00
..
__init__.py [SPARK-16772][PYTHON][DOCS] Fix API doc references to UDFRegistration + Update "important classes" 2016-08-06 05:02:59 +01:00
catalog.py [SPARK-18777][PYTHON][SQL] Return UDF from udf.register 2017-05-06 22:28:42 -07:00
column.py [SPARK-19165][PYTHON][SQL] PySpark APIs using columns as arguments should validate input types for column 2017-08-24 20:29:03 +09:00
conf.py [SPARK-15464][ML][MLLIB][SQL][TESTS] Replace SQLContext and SparkContext with SparkSession using builder pattern in python test code 2016-05-23 18:14:48 -07:00
context.py [SPARK-20586][SQL] Add deterministic to ScalaUDF 2017-07-25 17:19:44 -07:00
dataframe.py [SPARK-22100][SQL] Make percentile_approx support date/timestamp type and change the output type to be the same as input type 2017-09-25 09:28:42 -07:00
functions.py [SPARK-22106][PYSPARK][SQL] Disable 0-parameter pandas_udf and add doctests 2017-09-26 10:54:00 +09:00
group.py [MINOR][PYSPARK][DOC] Fix wrongly formatted examples in PySpark documentation 2016-07-06 10:45:51 -07:00
readwriter.py [SPARK-21839][SQL] Support SQL config for ORC compression 2017-08-31 08:16:58 +09:00
session.py [SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type dispatch in schema verification and improve exception message 2017-07-04 20:45:58 +08:00
streaming.py [SPARK-21756][SQL] Add JSON option to allow unquoted control characters 2017-08-25 10:18:03 -07:00
tests.py [SPARK-22106][PYSPARK][SQL] Disable 0-parameter pandas_udf and add doctests 2017-09-26 10:54:00 +09:00
types.py [SPARK-21190][PYSPARK] Python Vectorized UDFs 2017-09-22 16:17:50 +08:00
utils.py [MINOR][DOCS] Remove consecutive duplicated words/typo in Spark Repo 2017-01-04 15:07:29 +00:00
window.py [SPARK-18690][PYTHON][SQL] Backward compatibility of unbounded frames 2016-12-02 17:39:28 -08:00