811d563fbf
### What changes were proposed in this pull request? Inline cloudpickle in PySpark to cloudpickle 1.1.1. See https://github.com/cloudpipe/cloudpickle/blob/v1.1.1/cloudpickle/cloudpickle.py https://github.com/cloudpipe/cloudpickle/pull/269 was added for Python 3.8 support (fixed from 1.1.0). Using 1.2.2 seems breaking PyPy 2 due to cloudpipe/cloudpickle#278 so this PR currently uses 1.1.1. Once we drop Python 2, we can switch to the highest version. ### Why are the changes needed? positional-only arguments was newly introduced from Python 3.8 (see https://docs.python.org/3/whatsnew/3.8.html#positional-only-parameters) Particularly the newly added argument to `types.CodeType` was the problem (https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-python-api): > `types.CodeType` has a new parameter in the second position of the constructor (posonlyargcount) to support positional-only arguments defined in **PEP 570**. The first argument (argcount) now represents the total number of positional arguments (including positional-only arguments). The new `replace()` method of `types.CodeType` can be used to make the code future-proof. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Manually tested. Note that the optional dependency PyArrow looks not yet supporting Python 3.8; therefore, it was not tested. See "Details" below. <details> <p> ```bash cd python ./run-tests --python-executables=python3.8 ``` ``` Running PySpark tests. Output is in /Users/hyukjin.kwon/workspace/forked/spark/python/unit-tests.log Will test against the following Python executables: ['python3.8'] Will test the following Python modules: ['pyspark-core', 'pyspark-ml', 'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming'] Starting test(python3.8): pyspark.ml.tests.test_algorithms Starting test(python3.8): pyspark.ml.tests.test_feature Starting test(python3.8): pyspark.ml.tests.test_base Starting test(python3.8): pyspark.ml.tests.test_evaluation Finished test(python3.8): pyspark.ml.tests.test_base (12s) Starting test(python3.8): pyspark.ml.tests.test_image Finished test(python3.8): pyspark.ml.tests.test_evaluation (14s) Starting test(python3.8): pyspark.ml.tests.test_linalg Finished test(python3.8): pyspark.ml.tests.test_feature (23s) Starting test(python3.8): pyspark.ml.tests.test_param Finished test(python3.8): pyspark.ml.tests.test_image (22s) Starting test(python3.8): pyspark.ml.tests.test_persistence Finished test(python3.8): pyspark.ml.tests.test_param (25s) Starting test(python3.8): pyspark.ml.tests.test_pipeline Finished test(python3.8): pyspark.ml.tests.test_linalg (37s) Starting test(python3.8): pyspark.ml.tests.test_stat Finished test(python3.8): pyspark.ml.tests.test_pipeline (7s) Starting test(python3.8): pyspark.ml.tests.test_training_summary Finished test(python3.8): pyspark.ml.tests.test_stat (21s) Starting test(python3.8): pyspark.ml.tests.test_tuning Finished test(python3.8): pyspark.ml.tests.test_persistence (45s) Starting test(python3.8): pyspark.ml.tests.test_wrapper Finished test(python3.8): pyspark.ml.tests.test_algorithms (83s) Starting test(python3.8): pyspark.mllib.tests.test_algorithms Finished test(python3.8): pyspark.ml.tests.test_training_summary (32s) Starting test(python3.8): pyspark.mllib.tests.test_feature Finished test(python3.8): pyspark.ml.tests.test_wrapper (20s) Starting test(python3.8): pyspark.mllib.tests.test_linalg Finished test(python3.8): pyspark.mllib.tests.test_feature (32s) Starting test(python3.8): pyspark.mllib.tests.test_stat Finished test(python3.8): pyspark.mllib.tests.test_algorithms (70s) Starting test(python3.8): pyspark.mllib.tests.test_streaming_algorithms Finished test(python3.8): pyspark.mllib.tests.test_stat (37s) Starting test(python3.8): pyspark.mllib.tests.test_util Finished test(python3.8): pyspark.mllib.tests.test_linalg (70s) Starting test(python3.8): pyspark.sql.tests.test_arrow Finished test(python3.8): pyspark.sql.tests.test_arrow (1s) ... 53 tests were skipped Starting test(python3.8): pyspark.sql.tests.test_catalog Finished test(python3.8): pyspark.mllib.tests.test_util (15s) Starting test(python3.8): pyspark.sql.tests.test_column Finished test(python3.8): pyspark.sql.tests.test_catalog (24s) Starting test(python3.8): pyspark.sql.tests.test_conf Finished test(python3.8): pyspark.sql.tests.test_column (21s) Starting test(python3.8): pyspark.sql.tests.test_context Finished test(python3.8): pyspark.ml.tests.test_tuning (125s) Starting test(python3.8): pyspark.sql.tests.test_dataframe Finished test(python3.8): pyspark.sql.tests.test_conf (9s) Starting test(python3.8): pyspark.sql.tests.test_datasources Finished test(python3.8): pyspark.sql.tests.test_context (29s) Starting test(python3.8): pyspark.sql.tests.test_functions Finished test(python3.8): pyspark.sql.tests.test_datasources (32s) Starting test(python3.8): pyspark.sql.tests.test_group Finished test(python3.8): pyspark.sql.tests.test_dataframe (39s) ... 3 tests were skipped Starting test(python3.8): pyspark.sql.tests.test_pandas_udf Finished test(python3.8): pyspark.sql.tests.test_pandas_udf (1s) ... 6 tests were skipped Starting test(python3.8): pyspark.sql.tests.test_pandas_udf_cogrouped_map Finished test(python3.8): pyspark.sql.tests.test_pandas_udf_cogrouped_map (0s) ... 14 tests were skipped Starting test(python3.8): pyspark.sql.tests.test_pandas_udf_grouped_agg Finished test(python3.8): pyspark.sql.tests.test_pandas_udf_grouped_agg (1s) ... 15 tests were skipped Starting test(python3.8): pyspark.sql.tests.test_pandas_udf_grouped_map Finished test(python3.8): pyspark.sql.tests.test_pandas_udf_grouped_map (1s) ... 20 tests were skipped Starting test(python3.8): pyspark.sql.tests.test_pandas_udf_scalar Finished test(python3.8): pyspark.sql.tests.test_pandas_udf_scalar (1s) ... 49 tests were skipped Starting test(python3.8): pyspark.sql.tests.test_pandas_udf_window Finished test(python3.8): pyspark.sql.tests.test_pandas_udf_window (1s) ... 14 tests were skipped Starting test(python3.8): pyspark.sql.tests.test_readwriter Finished test(python3.8): pyspark.sql.tests.test_functions (29s) Starting test(python3.8): pyspark.sql.tests.test_serde Finished test(python3.8): pyspark.sql.tests.test_group (20s) Starting test(python3.8): pyspark.sql.tests.test_session Finished test(python3.8): pyspark.mllib.tests.test_streaming_algorithms (126s) Starting test(python3.8): pyspark.sql.tests.test_streaming Finished test(python3.8): pyspark.sql.tests.test_serde (25s) Starting test(python3.8): pyspark.sql.tests.test_types Finished test(python3.8): pyspark.sql.tests.test_readwriter (38s) Starting test(python3.8): pyspark.sql.tests.test_udf Finished test(python3.8): pyspark.sql.tests.test_session (32s) Starting test(python3.8): pyspark.sql.tests.test_utils Finished test(python3.8): pyspark.sql.tests.test_utils (17s) Starting test(python3.8): pyspark.streaming.tests.test_context Finished test(python3.8): pyspark.sql.tests.test_types (45s) Starting test(python3.8): pyspark.streaming.tests.test_dstream Finished test(python3.8): pyspark.sql.tests.test_udf (44s) Starting test(python3.8): pyspark.streaming.tests.test_kinesis Finished test(python3.8): pyspark.streaming.tests.test_kinesis (0s) ... 2 tests were skipped Starting test(python3.8): pyspark.streaming.tests.test_listener Finished test(python3.8): pyspark.streaming.tests.test_context (28s) Starting test(python3.8): pyspark.tests.test_appsubmit Finished test(python3.8): pyspark.sql.tests.test_streaming (60s) Starting test(python3.8): pyspark.tests.test_broadcast Finished test(python3.8): pyspark.streaming.tests.test_listener (11s) Starting test(python3.8): pyspark.tests.test_conf Finished test(python3.8): pyspark.tests.test_conf (17s) Starting test(python3.8): pyspark.tests.test_context Finished test(python3.8): pyspark.tests.test_broadcast (39s) Starting test(python3.8): pyspark.tests.test_daemon Finished test(python3.8): pyspark.tests.test_daemon (5s) Starting test(python3.8): pyspark.tests.test_join Finished test(python3.8): pyspark.tests.test_context (31s) Starting test(python3.8): pyspark.tests.test_profiler Finished test(python3.8): pyspark.tests.test_join (9s) Starting test(python3.8): pyspark.tests.test_rdd Finished test(python3.8): pyspark.tests.test_profiler (12s) Starting test(python3.8): pyspark.tests.test_readwrite Finished test(python3.8): pyspark.tests.test_readwrite (23s) ... 3 tests were skipped Starting test(python3.8): pyspark.tests.test_serializers Finished test(python3.8): pyspark.tests.test_appsubmit (94s) Starting test(python3.8): pyspark.tests.test_shuffle Finished test(python3.8): pyspark.streaming.tests.test_dstream (110s) Starting test(python3.8): pyspark.tests.test_taskcontext Finished test(python3.8): pyspark.tests.test_rdd (42s) Starting test(python3.8): pyspark.tests.test_util Finished test(python3.8): pyspark.tests.test_serializers (11s) Starting test(python3.8): pyspark.tests.test_worker Finished test(python3.8): pyspark.tests.test_shuffle (12s) Starting test(python3.8): pyspark.accumulators Finished test(python3.8): pyspark.tests.test_util (7s) Starting test(python3.8): pyspark.broadcast Finished test(python3.8): pyspark.accumulators (8s) Starting test(python3.8): pyspark.conf Finished test(python3.8): pyspark.broadcast (8s) Starting test(python3.8): pyspark.context Finished test(python3.8): pyspark.tests.test_worker (19s) Starting test(python3.8): pyspark.ml.classification Finished test(python3.8): pyspark.conf (4s) Starting test(python3.8): pyspark.ml.clustering Finished test(python3.8): pyspark.context (22s) Starting test(python3.8): pyspark.ml.evaluation Finished test(python3.8): pyspark.tests.test_taskcontext (49s) Starting test(python3.8): pyspark.ml.feature Finished test(python3.8): pyspark.ml.clustering (43s) Starting test(python3.8): pyspark.ml.fpm Finished test(python3.8): pyspark.ml.evaluation (27s) Starting test(python3.8): pyspark.ml.image Finished test(python3.8): pyspark.ml.image (8s) Starting test(python3.8): pyspark.ml.linalg.__init__ Finished test(python3.8): pyspark.ml.linalg.__init__ (0s) Starting test(python3.8): pyspark.ml.recommendation Finished test(python3.8): pyspark.ml.classification (63s) Starting test(python3.8): pyspark.ml.regression Finished test(python3.8): pyspark.ml.fpm (23s) Starting test(python3.8): pyspark.ml.stat Finished test(python3.8): pyspark.ml.stat (30s) Starting test(python3.8): pyspark.ml.tuning Finished test(python3.8): pyspark.ml.regression (51s) Starting test(python3.8): pyspark.mllib.classification Finished test(python3.8): pyspark.ml.feature (93s) Starting test(python3.8): pyspark.mllib.clustering Finished test(python3.8): pyspark.ml.tuning (39s) Starting test(python3.8): pyspark.mllib.evaluation Finished test(python3.8): pyspark.mllib.classification (38s) Starting test(python3.8): pyspark.mllib.feature Finished test(python3.8): pyspark.mllib.evaluation (25s) Starting test(python3.8): pyspark.mllib.fpm Finished test(python3.8): pyspark.mllib.clustering (64s) Starting test(python3.8): pyspark.mllib.linalg.__init__ Finished test(python3.8): pyspark.ml.recommendation (131s) Starting test(python3.8): pyspark.mllib.linalg.distributed Finished test(python3.8): pyspark.mllib.linalg.__init__ (0s) Starting test(python3.8): pyspark.mllib.random Finished test(python3.8): pyspark.mllib.feature (36s) Starting test(python3.8): pyspark.mllib.recommendation Finished test(python3.8): pyspark.mllib.fpm (31s) Starting test(python3.8): pyspark.mllib.regression Finished test(python3.8): pyspark.mllib.random (16s) Starting test(python3.8): pyspark.mllib.stat.KernelDensity Finished test(python3.8): pyspark.mllib.stat.KernelDensity (1s) Starting test(python3.8): pyspark.mllib.stat._statistics Finished test(python3.8): pyspark.mllib.stat._statistics (25s) Starting test(python3.8): pyspark.mllib.tree Finished test(python3.8): pyspark.mllib.regression (44s) Starting test(python3.8): pyspark.mllib.util Finished test(python3.8): pyspark.mllib.recommendation (49s) Starting test(python3.8): pyspark.profiler Finished test(python3.8): pyspark.mllib.linalg.distributed (53s) Starting test(python3.8): pyspark.rdd Finished test(python3.8): pyspark.profiler (14s) Starting test(python3.8): pyspark.serializers Finished test(python3.8): pyspark.mllib.tree (30s) Starting test(python3.8): pyspark.shuffle Finished test(python3.8): pyspark.shuffle (2s) Starting test(python3.8): pyspark.sql.avro.functions Finished test(python3.8): pyspark.mllib.util (30s) Starting test(python3.8): pyspark.sql.catalog Finished test(python3.8): pyspark.serializers (17s) Starting test(python3.8): pyspark.sql.column Finished test(python3.8): pyspark.rdd (31s) Starting test(python3.8): pyspark.sql.conf Finished test(python3.8): pyspark.sql.conf (7s) Starting test(python3.8): pyspark.sql.context Finished test(python3.8): pyspark.sql.avro.functions (19s) Starting test(python3.8): pyspark.sql.dataframe Finished test(python3.8): pyspark.sql.catalog (16s) Starting test(python3.8): pyspark.sql.functions Finished test(python3.8): pyspark.sql.column (27s) Starting test(python3.8): pyspark.sql.group Finished test(python3.8): pyspark.sql.context (26s) Starting test(python3.8): pyspark.sql.readwriter Finished test(python3.8): pyspark.sql.group (52s) Starting test(python3.8): pyspark.sql.session Finished test(python3.8): pyspark.sql.dataframe (73s) Starting test(python3.8): pyspark.sql.streaming Finished test(python3.8): pyspark.sql.functions (75s) Starting test(python3.8): pyspark.sql.types Finished test(python3.8): pyspark.sql.readwriter (57s) Starting test(python3.8): pyspark.sql.udf Finished test(python3.8): pyspark.sql.types (13s) Starting test(python3.8): pyspark.sql.window Finished test(python3.8): pyspark.sql.session (32s) Starting test(python3.8): pyspark.streaming.util Finished test(python3.8): pyspark.streaming.util (1s) Starting test(python3.8): pyspark.util Finished test(python3.8): pyspark.util (0s) Finished test(python3.8): pyspark.sql.streaming (30s) Finished test(python3.8): pyspark.sql.udf (27s) Finished test(python3.8): pyspark.sql.window (22s) Tests passed in 855 seconds ``` </p> </details> Closes #26194 from HyukjinKwon/SPARK-29536. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
257 lines
11 KiB
Python
257 lines
11 KiB
Python
#!/usr/bin/env python
|
|
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
from __future__ import print_function
|
|
import glob
|
|
import os
|
|
import sys
|
|
from setuptools import setup
|
|
from shutil import copyfile, copytree, rmtree
|
|
|
|
if sys.version_info < (2, 7):
|
|
print("Python versions prior to 2.7 are not supported for pip installed PySpark.",
|
|
file=sys.stderr)
|
|
sys.exit(-1)
|
|
|
|
try:
|
|
exec(open('pyspark/version.py').read())
|
|
except IOError:
|
|
print("Failed to load PySpark version file for packaging. You must be in Spark's python dir.",
|
|
file=sys.stderr)
|
|
sys.exit(-1)
|
|
VERSION = __version__ # noqa
|
|
# A temporary path so we can access above the Python project root and fetch scripts and jars we need
|
|
TEMP_PATH = "deps"
|
|
SPARK_HOME = os.path.abspath("../")
|
|
|
|
# Provide guidance about how to use setup.py
|
|
incorrect_invocation_message = """
|
|
If you are installing pyspark from spark source, you must first build Spark and
|
|
run sdist.
|
|
|
|
To build Spark with maven you can run:
|
|
./build/mvn -DskipTests clean package
|
|
Building the source dist is done in the Python directory:
|
|
cd python
|
|
python setup.py sdist
|
|
pip install dist/*.tar.gz"""
|
|
|
|
# Figure out where the jars are we need to package with PySpark.
|
|
JARS_PATH = glob.glob(os.path.join(SPARK_HOME, "assembly/target/scala-*/jars/"))
|
|
|
|
if len(JARS_PATH) == 1:
|
|
JARS_PATH = JARS_PATH[0]
|
|
elif (os.path.isfile("../RELEASE") and len(glob.glob("../jars/spark*core*.jar")) == 1):
|
|
# Release mode puts the jars in a jars directory
|
|
JARS_PATH = os.path.join(SPARK_HOME, "jars")
|
|
elif len(JARS_PATH) > 1:
|
|
print("Assembly jars exist for multiple scalas ({0}), please cleanup assembly/target".format(
|
|
JARS_PATH), file=sys.stderr)
|
|
sys.exit(-1)
|
|
elif len(JARS_PATH) == 0 and not os.path.exists(TEMP_PATH):
|
|
print(incorrect_invocation_message, file=sys.stderr)
|
|
sys.exit(-1)
|
|
|
|
EXAMPLES_PATH = os.path.join(SPARK_HOME, "examples/src/main/python")
|
|
SCRIPTS_PATH = os.path.join(SPARK_HOME, "bin")
|
|
USER_SCRIPTS_PATH = os.path.join(SPARK_HOME, "sbin")
|
|
DATA_PATH = os.path.join(SPARK_HOME, "data")
|
|
LICENSES_PATH = os.path.join(SPARK_HOME, "licenses")
|
|
|
|
SCRIPTS_TARGET = os.path.join(TEMP_PATH, "bin")
|
|
USER_SCRIPTS_TARGET = os.path.join(TEMP_PATH, "sbin")
|
|
JARS_TARGET = os.path.join(TEMP_PATH, "jars")
|
|
EXAMPLES_TARGET = os.path.join(TEMP_PATH, "examples")
|
|
DATA_TARGET = os.path.join(TEMP_PATH, "data")
|
|
LICENSES_TARGET = os.path.join(TEMP_PATH, "licenses")
|
|
|
|
# Check and see if we are under the spark path in which case we need to build the symlink farm.
|
|
# This is important because we only want to build the symlink farm while under Spark otherwise we
|
|
# want to use the symlink farm. And if the symlink farm exists under while under Spark (e.g. a
|
|
# partially built sdist) we should error and have the user sort it out.
|
|
in_spark = (os.path.isfile("../core/src/main/scala/org/apache/spark/SparkContext.scala") or
|
|
(os.path.isfile("../RELEASE") and len(glob.glob("../jars/spark*core*.jar")) == 1))
|
|
|
|
|
|
def _supports_symlinks():
|
|
"""Check if the system supports symlinks (e.g. *nix) or not."""
|
|
return getattr(os, "symlink", None) is not None
|
|
|
|
|
|
if (in_spark):
|
|
# Construct links for setup
|
|
try:
|
|
os.mkdir(TEMP_PATH)
|
|
except:
|
|
print("Temp path for symlink to parent already exists {0}".format(TEMP_PATH),
|
|
file=sys.stderr)
|
|
sys.exit(-1)
|
|
|
|
# If you are changing the versions here, please also change ./python/pyspark/sql/utils.py
|
|
# For Arrow, you should also check ./pom.xml and ensure there are no breaking changes in the
|
|
# binary format protocol with the Java version, see ARROW_HOME/format/* for specifications.
|
|
_minimum_pandas_version = "0.23.2"
|
|
_minimum_pyarrow_version = "0.12.1"
|
|
|
|
try:
|
|
# We copy the shell script to be under pyspark/python/pyspark so that the launcher scripts
|
|
# find it where expected. The rest of the files aren't copied because they are accessed
|
|
# using Python imports instead which will be resolved correctly.
|
|
try:
|
|
os.makedirs("pyspark/python/pyspark")
|
|
except OSError:
|
|
# Don't worry if the directory already exists.
|
|
pass
|
|
copyfile("pyspark/shell.py", "pyspark/python/pyspark/shell.py")
|
|
|
|
if (in_spark):
|
|
# Construct the symlink farm - this is necessary since we can't refer to the path above the
|
|
# package root and we need to copy the jars and scripts which are up above the python root.
|
|
if _supports_symlinks():
|
|
os.symlink(JARS_PATH, JARS_TARGET)
|
|
os.symlink(SCRIPTS_PATH, SCRIPTS_TARGET)
|
|
os.symlink(USER_SCRIPTS_PATH, USER_SCRIPTS_TARGET)
|
|
os.symlink(EXAMPLES_PATH, EXAMPLES_TARGET)
|
|
os.symlink(DATA_PATH, DATA_TARGET)
|
|
os.symlink(LICENSES_PATH, LICENSES_TARGET)
|
|
else:
|
|
# For windows fall back to the slower copytree
|
|
copytree(JARS_PATH, JARS_TARGET)
|
|
copytree(SCRIPTS_PATH, SCRIPTS_TARGET)
|
|
copytree(USER_SCRIPTS_PATH, USER_SCRIPTS_TARGET)
|
|
copytree(EXAMPLES_PATH, EXAMPLES_TARGET)
|
|
copytree(DATA_PATH, DATA_TARGET)
|
|
copytree(LICENSES_PATH, LICENSES_TARGET)
|
|
else:
|
|
# If we are not inside of SPARK_HOME verify we have the required symlink farm
|
|
if not os.path.exists(JARS_TARGET):
|
|
print("To build packaging must be in the python directory under the SPARK_HOME.",
|
|
file=sys.stderr)
|
|
|
|
if not os.path.isdir(SCRIPTS_TARGET):
|
|
print(incorrect_invocation_message, file=sys.stderr)
|
|
sys.exit(-1)
|
|
|
|
# Scripts directive requires a list of each script path and does not take wild cards.
|
|
script_names = os.listdir(SCRIPTS_TARGET)
|
|
scripts = list(map(lambda script: os.path.join(SCRIPTS_TARGET, script), script_names))
|
|
# We add find_spark_home.py to the bin directory we install so that pip installed PySpark
|
|
# will search for SPARK_HOME with Python.
|
|
scripts.append("pyspark/find_spark_home.py")
|
|
|
|
# Parse the README markdown file into rst for PyPI
|
|
long_description = "!!!!! missing pandoc do not upload to PyPI !!!!"
|
|
try:
|
|
import pypandoc
|
|
long_description = pypandoc.convert('README.md', 'rst')
|
|
except ImportError:
|
|
print("Could not import pypandoc - required to package PySpark", file=sys.stderr)
|
|
except OSError:
|
|
print("Could not convert - pandoc is not installed", file=sys.stderr)
|
|
|
|
setup(
|
|
name='pyspark',
|
|
version=VERSION,
|
|
description='Apache Spark Python API',
|
|
long_description=long_description,
|
|
author='Spark Developers',
|
|
author_email='dev@spark.apache.org',
|
|
url='https://github.com/apache/spark/tree/master/python',
|
|
packages=['pyspark',
|
|
'pyspark.mllib',
|
|
'pyspark.mllib.linalg',
|
|
'pyspark.mllib.stat',
|
|
'pyspark.ml',
|
|
'pyspark.ml.linalg',
|
|
'pyspark.ml.param',
|
|
'pyspark.sql',
|
|
'pyspark.streaming',
|
|
'pyspark.bin',
|
|
'pyspark.sbin',
|
|
'pyspark.jars',
|
|
'pyspark.python.pyspark',
|
|
'pyspark.python.lib',
|
|
'pyspark.data',
|
|
'pyspark.licenses',
|
|
'pyspark.examples.src.main.python'],
|
|
include_package_data=True,
|
|
package_dir={
|
|
'pyspark.jars': 'deps/jars',
|
|
'pyspark.bin': 'deps/bin',
|
|
'pyspark.sbin': 'deps/sbin',
|
|
'pyspark.python.lib': 'lib',
|
|
'pyspark.data': 'deps/data',
|
|
'pyspark.licenses': 'deps/licenses',
|
|
'pyspark.examples.src.main.python': 'deps/examples',
|
|
},
|
|
package_data={
|
|
'pyspark.jars': ['*.jar'],
|
|
'pyspark.bin': ['*'],
|
|
'pyspark.sbin': ['spark-config.sh', 'spark-daemon.sh',
|
|
'start-history-server.sh',
|
|
'stop-history-server.sh', ],
|
|
'pyspark.python.lib': ['*.zip'],
|
|
'pyspark.data': ['*.txt', '*.data'],
|
|
'pyspark.licenses': ['*.txt'],
|
|
'pyspark.examples.src.main.python': ['*.py', '*/*.py']},
|
|
scripts=scripts,
|
|
license='http://www.apache.org/licenses/LICENSE-2.0',
|
|
install_requires=['py4j==0.10.8.1'],
|
|
setup_requires=['pypandoc'],
|
|
extras_require={
|
|
'ml': ['numpy>=1.7'],
|
|
'mllib': ['numpy>=1.7'],
|
|
'sql': [
|
|
'pandas>=%s' % _minimum_pandas_version,
|
|
'pyarrow>=%s' % _minimum_pyarrow_version,
|
|
]
|
|
},
|
|
classifiers=[
|
|
'Development Status :: 5 - Production/Stable',
|
|
'License :: OSI Approved :: Apache Software License',
|
|
'Programming Language :: Python :: 2.7',
|
|
'Programming Language :: Python :: 3',
|
|
'Programming Language :: Python :: 3.4',
|
|
'Programming Language :: Python :: 3.5',
|
|
'Programming Language :: Python :: 3.6',
|
|
'Programming Language :: Python :: 3.7',
|
|
'Programming Language :: Python :: 3.8',
|
|
'Programming Language :: Python :: Implementation :: CPython',
|
|
'Programming Language :: Python :: Implementation :: PyPy']
|
|
)
|
|
finally:
|
|
# We only cleanup the symlink farm if we were in Spark, otherwise we are installing rather than
|
|
# packaging.
|
|
if (in_spark):
|
|
# Depending on cleaning up the symlink farm or copied version
|
|
if _supports_symlinks():
|
|
os.remove(os.path.join(TEMP_PATH, "jars"))
|
|
os.remove(os.path.join(TEMP_PATH, "bin"))
|
|
os.remove(os.path.join(TEMP_PATH, "sbin"))
|
|
os.remove(os.path.join(TEMP_PATH, "examples"))
|
|
os.remove(os.path.join(TEMP_PATH, "data"))
|
|
os.remove(os.path.join(TEMP_PATH, "licenses"))
|
|
else:
|
|
rmtree(os.path.join(TEMP_PATH, "jars"))
|
|
rmtree(os.path.join(TEMP_PATH, "bin"))
|
|
rmtree(os.path.join(TEMP_PATH, "sbin"))
|
|
rmtree(os.path.join(TEMP_PATH, "examples"))
|
|
rmtree(os.path.join(TEMP_PATH, "data"))
|
|
rmtree(os.path.join(TEMP_PATH, "licenses"))
|
|
os.rmdir(TEMP_PATH)
|