### What changes were proposed in this pull request? Currently, there is no migration section for PySpark, SparkCore and Structured Streaming. It is difficult for users to know what to do when they upgrade. This PR proposes to create create a "Migration Guide" tap at Spark documentation. ![Screen Shot 2019-09-11 at 7 02 05 PM](https://user-images.githubusercontent.com/6477701/64688126-ad712f80-d4c6-11e9-8672-9a2c56c05bf8.png) ![Screen Shot 2019-09-11 at 7 27 15 PM](https://user-images.githubusercontent.com/6477701/64689915-389ff480-d4ca-11e9-8c54-7f46095d0d23.png) This page will contain migration guides for Spark SQL, PySpark, SparkR, MLlib, Structured Streaming and Core. Basically it is a refactoring. There are some new information added, which I will leave a comment inlined for easier review. 1. **MLlib** Merge [ml-guide.html#migration-guide](https://spark.apache.org/docs/latest/ml-guide.html#migration-guide) and [ml-migration-guides.html](https://spark.apache.org/docs/latest/ml-migration-guides.html) ``` 'docs/ml-guide.md' ↓ Merge new/old migration guides 'docs/ml-migration-guide.md' ``` 2. **PySpark** Extract PySpark specific items from https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html ``` 'docs/sql-migration-guide-upgrade.md' ↓ Extract PySpark specific items 'docs/pyspark-migration-guide.md' ``` 3. **SparkR** Move [sparkr.html#migration-guide](https://spark.apache.org/docs/latest/sparkr.html#migration-guide) into a separate file, and extract from [sql-migration-guide-upgrade.html](https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html) ``` 'docs/sparkr.md' 'docs/sql-migration-guide-upgrade.md' Move migration guide section ↘ ↙ Extract SparkR specific items docs/sparkr-migration-guide.md ``` 4. **Core** Newly created at `'docs/core-migration-guide.md'`. I skimmed resolved JIRAs at 3.0.0 and found some items to note. 5. **Structured Streaming** Newly created at `'docs/ss-migration-guide.md'`. I skimmed resolved JIRAs at 3.0.0 and found some items to note. 6. **SQL** Merged [sql-migration-guide-upgrade.html](https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html) and [sql-migration-guide-hive-compatibility.html](https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) ``` 'docs/sql-migration-guide-hive-compatibility.md' 'docs/sql-migration-guide-upgrade.md' Move Hive compatibility section ↘ ↙ Left over after filtering PySpark and SparkR items 'docs/sql-migration-guide.md' ``` ### Why are the changes needed? In order for users in production to effectively migrate to higher versions, and detect behaviour or breaking changes before upgrading and/or migrating. ### Does this PR introduce any user-facing change? Yes, this changes Spark's documentation at https://spark.apache.org/docs/latest/index.html. ### How was this patch tested? Manually build the doc. This can be verified as below: ```bash cd docs SKIP_API=1 jekyll build open _site/index.html ``` Closes #25757 from HyukjinKwon/migration-doc. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
6.2 KiB
layout | title | displayTitle | license |
---|---|---|---|
global | Migration Guide: PySpark (Python on Spark) | Migration Guide: PySpark (Python on Spark) | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. |
- Table of contents {:toc}
Note that this migration guide describes the items specific to PySpark. Many items of SQL migration can be applied when migrating PySpark to higher versions. Please refer Migration Guide: SQL, Datasets and DataFrame.
Upgrading from PySpark 2.4 to 3.0
-
Since Spark 3.0, PySpark requires a Pandas version of 0.23.2 or higher to use Pandas related functionality, such as
toPandas
,createDataFrame
from Pandas DataFrame, etc. -
Since Spark 3.0, PySpark requires a PyArrow version of 0.12.1 or higher to use PyArrow related functionality, such as
pandas_udf
,toPandas
andcreateDataFrame
with "spark.sql.execution.arrow.enabled=true", etc. -
In PySpark, when creating a
SparkSession
withSparkSession.builder.getOrCreate()
, if there is an existingSparkContext
, the builder was trying to update theSparkConf
of the existingSparkContext
with configurations specified to the builder, but theSparkContext
is shared by allSparkSession
s, so we should not update them. Since 3.0, the builder comes to not update the configurations. This is the same behavior as Java/Scala API in 2.3 and above. If you want to update them, you need to update them prior to creating aSparkSession
. -
In PySpark, when Arrow optimization is enabled, if Arrow version is higher than 0.11.0, Arrow can perform safe type conversion when converting Pandas.Series to Arrow array during serialization. Arrow will raise errors when detecting unsafe type conversion like overflow. Setting
spark.sql.execution.pandas.arrowSafeTypeConversion
to true can enable it. The default setting is false. PySpark's behavior for Arrow versions is illustrated in the table below:PyArrow version Integer Overflow Floating Point Truncation version < 0.11.0 Raise error Silently allows version > 0.11.0, arrowSafeTypeConversion=false Silent overflow Silently allows version > 0.11.0, arrowSafeTypeConversion=true Raise error Raise error -
Since Spark 3.0,
createDataFrame(..., verifySchema=True)
validatesLongType
as well in PySpark. Previously,LongType
was not verified and resulted inNone
in case the value overflows. To restore this behavior,verifySchema
can be set toFalse
to disable the validation.
Upgrading from PySpark 2.3 to 2.4
- In PySpark, when Arrow optimization is enabled, previously
toPandas
just failed when Arrow optimization is unable to be used whereascreateDataFrame
from Pandas DataFrame allowed the fallback to non-optimization. Now, bothtoPandas
andcreateDataFrame
from Pandas DataFrame allow the fallback by default, which can be switched off byspark.sql.execution.arrow.fallback.enabled
.
Upgrading from PySpark 2.3.0 to 2.3.1 and above
- As of version 2.3.1 Arrow functionality, including
pandas_udf
andtoPandas()
/createDataFrame()
withspark.sql.execution.arrow.enabled
set toTrue
, has been marked as experimental. These are still evolving and not currently recommended for use in production.
Upgrading from PySpark 2.2 to 2.3
-
In PySpark, now we need Pandas 0.19.2 or upper if you want to use Pandas related functionalities, such as
toPandas
,createDataFrame
from Pandas DataFrame, etc. -
In PySpark, the behavior of timestamp values for Pandas related functionalities was changed to respect session timezone. If you want to use the old behavior, you need to set a configuration
spark.sql.execution.pandas.respectSessionTimeZone
toFalse
. See SPARK-22395 for details. -
In PySpark,
na.fill()
orfillna
also accepts boolean and replaces nulls with booleans. In prior Spark versions, PySpark just ignores it and returns the original Dataset/DataFrame. -
In PySpark,
df.replace
does not allow to omitvalue
whento_replace
is not a dictionary. Previously,value
could be omitted in the other cases and hadNone
by default, which is counterintuitive and error-prone.
Upgrading from PySpark 1.4 to 1.5
-
Resolution of strings to columns in Python now supports using dots (
.
) to qualify the column or access nested values. For exampledf['table.column.nestedField']
. However, this means that if your column name contains any dots you must now escape them using backticks (e.g.,table.`column.with.dots`.nested
). -
DataFrame.withColumn method in PySpark supports adding a new column or replacing existing columns of the same name.
Upgrading from PySpark 1.0-1.2 to 1.3
Python DataTypes No Longer Singletons
{:.no_toc}
When using DataTypes in Python you will need to construct them (i.e. StringType()
) instead of
referencing a singleton.