caf04f9b77
### What changes were proposed in this pull request? As a first step of [SPARK-34849](https://issues.apache.org/jira/browse/SPARK-34849), this PR proposes porting the Koalas main code into PySpark. This PR contains minimal changes to the existing Koalas code as follows: 1. `databricks.koalas` -> `pyspark.pandas` 2. `from databricks import koalas as ks` -> `from pyspark import pandas as pp` 3. `ks.xxx -> pp.xxx` Other than them: 1. Added a line to `python/mypy.ini` in order to ignore the mypy test. See related issue at [SPARK-34941](https://issues.apache.org/jira/browse/SPARK-34941). 2. Added a comment to several lines in several files to ignore the flake8 F401. See related issue at [SPARK-34943](https://issues.apache.org/jira/browse/SPARK-34943). When this PR is merged, all the features that were previously used in [Koalas](https://github.com/databricks/koalas) will be available in PySpark as well. Users can access to the pandas API in PySpark as below: ```python >>> from pyspark import pandas as pp >>> ppdf = pp.DataFrame({"A": [1, 2, 3], "B": [15, 20, 25]}) >>> ppdf A B 0 1 15 1 2 20 2 3 25 ``` The existing "options and settings" in Koalas are also available in the same way: ```python >>> from pyspark.pandas.config import set_option, reset_option, get_option >>> ppser1 = pp.Series([1, 2, 3]) >>> ppser2 = pp.Series([3, 4, 5]) >>> ppser1 + ppser2 Traceback (most recent call last): ... ValueError: Cannot combine the series or dataframe because it comes from a different dataframe. In order to allow this operation, enable 'compute.ops_on_diff_frames' option. >>> set_option("compute.ops_on_diff_frames", True) >>> ppser1 + ppser2 0 4 1 6 2 8 dtype: int64 ``` Please also refer to the [API Reference](https://koalas.readthedocs.io/en/latest/reference/index.html) and [Options and Settings](https://koalas.readthedocs.io/en/latest/user_guide/options.html) for more detail. **NOTE** that this PR intentionally ports the main codes of Koalas first almost as are with minimal changes because: - Koalas project is fairly large. Making some changes together for PySpark will make it difficult to review the individual change. Koalas dev includes multiple Spark committers who will review. By doing this, the committers will be able to more easily and effectively review and drive the development. - Koalas tests and documentation require major changes to make it look great together with PySpark whereas main codes do not require. - We lately froze the Koalas codebase, and plan to work together on the initial porting. By porting the main codes first as are, it unblocks the Koalas dev to work on other items in parallel. I promise and will make sure on: - Rename Koalas to PySpark pandas APIs and/or pandas-on-Spark accordingly in documentation, and the docstrings and comments in the main codes. - Triage APIs to remove that don’t make sense when Koalas is in PySpark The documentation changes will be tracked in [SPARK-34885](https://issues.apache.org/jira/browse/SPARK-34885), the test code changes will be tracked in [SPARK-34886](https://issues.apache.org/jira/browse/SPARK-34886). ### Why are the changes needed? Please refer to: - [[DISCUSS] Support pandas API layer on PySpark](http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Support-pandas-API-layer-on-PySpark-td30945.html) - [[VOTE] SPIP: Support pandas API layer on PySpark](http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-SPIP-Support-pandas-API-layer-on-PySpark-td30996.html) ### Does this PR introduce _any_ user-facing change? Yes, now users can use the pandas APIs on Spark ### How was this patch tested? Manually tested for exposed major APIs and options as described above. ### Koalas contributors Koalas would not have been possible without the following contributors: ueshin HyukjinKwon rxin xinrong-databricks RainFung charlesdong1991 harupy floscha beobest2 thunterdb garawalid LucasG0 shril deepyaman gioa fwani 90jam thoo AbdealiJK abishekganesh72 gliptak DumbMachine dvgodoy stbof nitlev hjoo gatorsmile tomspur icexelloss awdavidson guyao akhilputhiry scook12 patryk-oleniuk tracek dennyglee athena15 gstaubli WeichenXu123 hsubbaraj lfdversluis ktksq shengjh margaret-databricks LSturtew sllynn manuzhang jijosg sadikovi Closes #32036 from itholic/SPARK-34890. Authored-by: itholic <haejoon.lee@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
133 lines
2.9 KiB
INI
133 lines
2.9 KiB
INI
;
|
|
; Licensed to the Apache Software Foundation (ASF) under one or more
|
|
; contributor license agreements. See the NOTICE file distributed with
|
|
; this work for additional information regarding copyright ownership.
|
|
; The ASF licenses this file to You under the Apache License, Version 2.0
|
|
; (the "License"); you may not use this file except in compliance with
|
|
; the License. You may obtain a copy of the License at
|
|
;
|
|
; http://www.apache.org/licenses/LICENSE-2.0
|
|
;
|
|
; Unless required by applicable law or agreed to in writing, software
|
|
; distributed under the License is distributed on an "AS IS" BASIS,
|
|
; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
; See the License for the specific language governing permissions and
|
|
; limitations under the License.
|
|
;
|
|
|
|
[mypy]
|
|
strict_optional = True
|
|
no_implicit_optional = True
|
|
disallow_untyped_defs = True
|
|
|
|
; Allow untyped def in internal modules and tests
|
|
|
|
[mypy-pyspark.daemon]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.find_spark_home]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark._globals]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.install]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.java_gateway]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.join]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.ml.tests.*]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.mllib.tests.*]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.rddsampler]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.resource.tests.*]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.serializers]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.shuffle]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.streaming.tests.*]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.streaming.util]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.sql.tests.*]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.sql.pandas.serializers]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.sql.pandas.types]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.sql.pandas.typehints]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.sql.pandas.utils]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.sql.pandas._typing.protocols.*]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.sql.utils]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.tests.*]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.testing.*]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.traceback_utils]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.util]
|
|
disallow_untyped_defs = False
|
|
|
|
[mypy-pyspark.worker]
|
|
disallow_untyped_defs = False
|
|
|
|
; Ignore errors in embedded third party code
|
|
|
|
no_implicit_optional = True
|
|
|
|
[mypy-pyspark.cloudpickle.*]
|
|
ignore_errors = True
|
|
|
|
; Ignore missing imports for external untyped packages
|
|
|
|
[mypy-py4j.*]
|
|
ignore_missing_imports = True
|
|
|
|
[mypy-numpy]
|
|
ignore_missing_imports = True
|
|
|
|
[mypy-scipy.*]
|
|
ignore_missing_imports = True
|
|
|
|
[mypy-pandas.*]
|
|
ignore_missing_imports = True
|
|
|
|
[mypy-pyarrow.*]
|
|
ignore_missing_imports = True
|
|
|
|
[mypy-psutil.*]
|
|
ignore_missing_imports = True
|
|
|
|
# TODO(SPARK-34941): Enable mypy for pandas-on-Spark
|
|
[mypy-pyspark.pandas.*]
|
|
ignore_errors = True
|