ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/python/pyspark/pandas
History
Takuya UESHIN a9ebfc5374 [SPARK-35466][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.data_type_ops.* 
### What changes were proposed in this pull request?

Adds more type annotations in the files `python/pyspark/pandas/data_type_ops/*.py` and fixes the mypy check failures.

### Why are the changes needed?

We should enable more disallow_untyped_defs mypy checks.

### Does this PR introduce _any_ user-facing change?

Yes.
This PR adds more type annotations in pandas APIs on Spark module, which can impact interaction with development tools for users.

### How was this patch tested?

The mypy check with a new configuration and existing tests should pass.

Closes #33094 from ueshin/issues/SPARK-35466/disallow_untyped_defs_data_ops.

Authored-by: Takuya UESHIN <ueshin@databricks.com>
Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
2021-06-25 18:16:25 -07:00
..
data_type_ops [SPARK-35466][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.data_type_ops.* 2021-06-25 18:16:25 -07:00
indexes [SPARK-35476][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.series 2021-06-24 19:32:33 +09:00
missing [SPARK-35071][PYTHON] Rename Koalas to pandas-on-Spark in main codes 2021-04-15 12:48:59 +09:00
plot [SPARK-35738][PYTHON] Support 'y' properly in DataFrame with non-numeric columns with plots 2021-06-12 14:36:46 +09:00
spark [SPARK-35470][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.base 2021-06-22 11:25:16 +09:00
tests [SPARK-35812][PYTHON] Throw ValueError if version and timestamp are used together in to_delta 2021-06-23 19:04:45 +09:00
typedef [SPARK-35638][PYTHON] Introduce InternalField to manage dtypes and StructFields 2021-06-08 11:57:28 +09:00
usage_logging [SPARK-35499][PYTHON] Apply black to pandas API on Spark codes 2021-06-06 17:30:07 -07:00
__init__.py [SPARK-35539][PYTHON] Restore to_koalas to keep the backward compatibility 2021-06-02 10:39:24 +09:00
accessors.py [SPARK-35469][PYTHON] Fix disallow_untyped_defs mypy checks 2021-06-18 20:43:59 +09:00
base.py [SPARK-35476][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.series 2021-06-24 19:32:33 +09:00
categorical.py [SPARK-35465][PYTHON] Set up the mypy configuration to enable disallow_untyped_defs check for pandas APIs on Spark module 2021-05-21 11:03:35 -07:00
config.py [SPARK-35499][PYTHON] Apply black to pandas API on Spark codes 2021-06-06 17:30:07 -07:00
datetimes.py [SPARK-35453][PYTHON] Move Koalas accessor to pandas_on_spark accessor 2021-06-01 10:33:10 +09:00
exceptions.py [SPARK-35465][PYTHON] Set up the mypy configuration to enable disallow_untyped_defs check for pandas APIs on Spark module 2021-05-21 11:03:35 -07:00
extensions.py [SPARK-35499][PYTHON] Apply black to pandas API on Spark codes 2021-06-06 17:30:07 -07:00
frame.py [SPARK-35471][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.frame 2021-06-25 14:41:58 +09:00
generic.py [SPARK-35471][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.frame 2021-06-25 14:41:58 +09:00
groupby.py [SPARK-35471][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.frame 2021-06-25 14:41:58 +09:00
indexing.py [SPARK-35474] Enable disallow_untyped_defs mypy check for pyspark.pandas.indexing 2021-06-09 22:35:12 -07:00
internal.py [SPARK-35476][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.series 2021-06-24 19:32:33 +09:00
ml.py [SPARK-35465][PYTHON] Set up the mypy configuration to enable disallow_untyped_defs check for pandas APIs on Spark module 2021-05-21 11:03:35 -07:00
mlflow.py [SPARK-35638][PYTHON] Introduce InternalField to manage dtypes and StructFields 2021-06-08 11:57:28 +09:00
namespace.py [SPARK-35471][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.frame 2021-06-25 14:41:58 +09:00
numpy_compat.py [SPARK-35470][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.base 2021-06-22 11:25:16 +09:00
series.py [SPARK-35471][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.frame 2021-06-25 14:41:58 +09:00
sql_processor.py [SPARK-35465][PYTHON] Set up the mypy configuration to enable disallow_untyped_defs check for pandas APIs on Spark module 2021-05-21 11:03:35 -07:00
strings.py [SPARK-35761][PYTHON] Use type-annotation based pandas_udf or avoid specifying udf types to suppress warnings 2021-06-15 11:17:56 +09:00
utils.py [SPARK-35471][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.frame 2021-06-25 14:41:58 +09:00
version.py [SPARK-34890][PYTHON] Port/integrate Koalas main codes into PySpark 2021-04-06 12:42:39 +09:00
window.py [SPARK-35478][PYTHON][FOLLOWUP] Fix Jenkins' linter 2021-06-18 13:52:54 -07:00