ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36617][PYTHON] Fix type hints for approxQuantile to support multi-column version

Browse source 
### What changes were proposed in this pull request?
Update both `DataFrame.approxQuantile` and `DataFrameStatFunctions.approxQuantile` to support overloaded definitions when multiple columns are supplied.

### Why are the changes needed?
The current type hints don't support the multi-column signature, a form that was added in Spark 2.2 (see [the approxQuantile docs](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.approxQuantile.html).) This change was also introduced to pyspark-stubs (https://github.com/zero323/pyspark-stubs/pull/552). zero323 asked me to open a PR for the upstream change.

### Does this PR introduce _any_ user-facing change?
This change only affects type hints - it brings the `approxQuantile` type hints up to date with the actual code.

### How was this patch tested?
Ran `./dev/lint-python`.

Closes #33880 from carylee/master.

Authored-by: Cary Lee <cary@amperity.com>
Signed-off-by: zero323 <mszymkiewicz@gmail.com>
This commit is contained in:
Cary Lee 2021-09-02 15:02:40 +02:00 committed by zero323
parent 94c306284a
commit 37f5ab07fa
No known key found for this signature in database
GPG key ID: A30CEF0C31A501EC
1 changed files with 23 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

27
python/pyspark/sql/dataframe.pyi
View file

@ -240,12 +240,20 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
value: OptionalPrimitiveType,
subset: Optional[List[str]] = ...,
) -> DataFrame: ...
@overload
def approxQuantile(
self,
col: Union[str, Tuple[str, ...], List[str]],
probabilities: Union[List[float], Tuple[float, ...]],
relativeError: float
col: str,
probabilities: Union[List[float], Tuple[float]],
relativeError: float,
) -> List[float]: ...
@overload
def approxQuantile(
self,
col: Union[List[str], Tuple[str]],
probabilities: Union[List[float], Tuple[float]],
relativeError: float,
) -> List[List[float]]: ...
def corr(self, col1: str, col2: str, method: Optional[str] = ...) -> float: ...
def cov(self, col1: str, col2: str) -> float: ...
def crosstab(self, col1: str, col2: str) -> DataFrame: ...
@ -318,9 +326,20 @@ class DataFrameNaFunctions:
class DataFrameStatFunctions:
df: DataFrame
def __init__(self, df: DataFrame) -> None: ...
@overload
def approxQuantile(
self, col: str, probabilities: List[float], relativeError: float
self,
col: str,
probabilities: Union[List[float], Tuple[float]],
relativeError: float,
) -> List[float]: ...
@overload
def approxQuantile(
self,
col: Union[List[str], Tuple[str]],
probabilities: Union[List[float], Tuple[float]],
relativeError: float,
) -> List[List[float]]: ...
def corr(self, col1: str, col2: str, method: Optional[str] = ...) -> float: ...
def cov(self, col1: str, col2: str) -> float: ...
def crosstab(self, col1: str, col2: str) -> DataFrame: ...