ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-35938][PYTHON] Add deprecation warning for Python 3.6

Browse source 
### What changes were proposed in this pull request?

Add deprecation warning for Python 3.6.

### Why are the changes needed?

According to https://endoflife.date/python, Python 3.6 will be EOL on 23 Dec, 2021.
We should prepare for the deprecation of Python 3.6 support in Spark in advance.

### Does this PR introduce _any_ user-facing change?

N/A.

### How was this patch tested?

Manual tests.

Closes #33139 from xinrong-databricks/deprecate3.6_warn.

Authored-by: Xinrong Meng <xinrong.meng@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
Xinrong Meng 2021-07-01 09:32:25 +09:00 committed by Hyukjin Kwon
parent 9e39415f3a
commit 5ad12611ec
1 changed files with 8 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
python/pyspark/context.py
View file

@ -230,6 +230,14 @@ class SparkContext(object):
self.pythonExec = os.environ.get("PYSPARK_PYTHON", 'python3')
self.pythonVer = "%d.%d" % sys.version_info[:2]
if sys.version_info[:2] < (3, 7):
with warnings.catch_warnings():
warnings.simplefilter("once")
warnings.warn(
"Python 3.6 support is deprecated in Spark 3.2.",
FutureWarning
)
# Broadcast's __reduce__ method stores Broadcast instances here.
# This allows other code to determine which Broadcast instances have
# been pickled, so it can determine which Java broadcast objects to