spark-instrumented-optimizer/python/pyspark/cloudpickle/__init__.py

from __future__ import absolute_import


from pyspark.cloudpickle.cloudpickle import *  # noqa
from pyspark.cloudpickle.cloudpickle_fast import CloudPickler, dumps, dump  # noqa

# Conform to the convention used by python serialization libraries, which
# expose their Pickler subclass at top-level under the  "Pickler" name.
Pickler = CloudPickler

__version__ = '1.6.0'
[SPARK-32094][PYTHON] Update cloudpickle to v1.5.0 ### What changes were proposed in this pull request? This PR aims to upgrade PySpark's embedded cloudpickle to the latest cloudpickle v1.5.0 (See https://github.com/cloudpipe/cloudpickle/blob/v1.5.0/cloudpickle/cloudpickle.py) ### Why are the changes needed? There are many bug fixes. For example, the bug described in the JIRA: dill unpickling fails because they define `types.ClassType`, which is undefined in dill. This results in the following error: ``` Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py", line 279, in loads return dill.loads(s) File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 317, in loads return load(file, ignore) File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 305, in load obj = pik.load() File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 577, in _load_type return _reverse_typemap[name] KeyError: 'ClassType' ``` See also https://github.com/cloudpipe/cloudpickle/issues/82. This was fixed for cloudpickle 1.3.0+ (https://github.com/cloudpipe/cloudpickle/pull/337), but PySpark's cloudpickle.py doesn't have this change yet. More notably, now it supports C pickle implementation with Python 3.8 which hugely improve performance. This is already adopted in another project such as Ray. ### Does this PR introduce _any_ user-facing change? Yes, as described above, the bug fixes. Internally, users also could leverage the fast cloudpickle backed by C pickle. ### How was this patch tested? Jenkins will test it out. Closes #29114 from HyukjinKwon/SPARK-32094. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org> 2020-07-16 22:49:18 -04:00			`from __future__ import absolute_import`


			`from pyspark.cloudpickle.cloudpickle import * # noqa`
			`from pyspark.cloudpickle.cloudpickle_fast import CloudPickler, dumps, dump # noqa`

[SPARK-33983][PYTHON] Update cloudpickle to v1.6.0 ### What changes were proposed in this pull request? This PR proposes to upgrade cloudpickle from 1.5.0 to 1.6.0. It virtually contains one fix: https://github.com/cloudpipe/cloudpickle/commit/4510be850d55bc60decf86953324f98bc3199f9e From a cursory look, this isn't a regression, and not even properly supported in Python: ```python >>> import pickle >>> pickle.dumps({}.keys()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: cannot pickle 'dict_keys' object ``` So it seems fine not to backport. ### Why are the changes needed? To leverage bug fixes from the cloudpickle upstream. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Jenkins build and GitHub actions build will test it out. Closes #31007 from HyukjinKwon/cloudpickle-upgrade. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> 2021-01-04 13:36:31 -05:00			`# Conform to the convention used by python serialization libraries, which`
			`# expose their Pickler subclass at top-level under the "Pickler" name.`
			`Pickler = CloudPickler`

			`__version__ = '1.6.0'`