### What changes were proposed in this pull request? This is a follow-up of #29278. This PR changes the config name to switch allow/disallow `SparkContext` in executors as per the comment https://github.com/apache/spark/pull/29278#pullrequestreview-460256338. ### Why are the changes needed? The config name `spark.executor.allowSparkContext` is more reasonable. ### Does this PR introduce _any_ user-facing change? Yes, the config name is changed. ### How was this patch tested? Updated tests. Closes #29340 from ueshin/issues/SPARK-32160/change_config_name. Authored-by: Takuya UESHIN <ueshin@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
3 KiB
layout | title | displayTitle | license |
---|---|---|---|
global | Migration Guide: Spark Core | Migration Guide: Spark Core | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. |
- Table of contents {:toc}
Upgrading from Core 3.0 to 3.1
- In Spark 3.0 and below,
SparkContext
can be created in executors. Since Spark 3.1, an exception will be thrown when creatingSparkContext
in executors. You can allow it by setting the configurationspark.executor.allowSparkContext
when creatingSparkContext
in executors.
Upgrading from Core 2.4 to 3.0
-
The
org.apache.spark.ExecutorPlugin
interface and related configuration has been replaced withorg.apache.spark.api.plugin.SparkPlugin
, which adds new functionality. Plugins using the old interface must be modified to extend the new interfaces. Check the Monitoring guide for more details. -
Deprecated method
TaskContext.isRunningLocally
has been removed. Local execution was removed and it always has returnedfalse
. -
Deprecated method
shuffleBytesWritten
,shuffleWriteTime
andshuffleRecordsWritten
inShuffleWriteMetrics
have been removed. Instead, usebytesWritten
,writeTime
andrecordsWritten
respectively. -
Deprecated method
AccumulableInfo.apply
have been removed because creatingAccumulableInfo
is disallowed. -
Deprecated accumulator v1 APIs have been removed and please use v2 APIs instead.
-
Event log file will be written as UTF-8 encoding, and Spark History Server will replay event log files as UTF-8 encoding. Previously Spark wrote the event log file as default charset of driver JVM process, so Spark History Server of Spark 2.x is needed to read the old event log files in case of incompatible encoding.
-
A new protocol for fetching shuffle blocks is used. It's recommended that external shuffle services be upgraded when running Spark 3.0 apps. You can still use old external shuffle services by setting the configuration
spark.shuffle.useOldFetchProtocol
totrue
. Otherwise, Spark may run into errors with messages likeIllegalArgumentException: Unexpected message type: <number>
. -
SPARK_WORKER_INSTANCES
is deprecated in Standalone mode. It's recommended to launch multiple executors in one worker and launch one worker per node instead of launching multiple workers per node and launching one executor per worker.