d03aebbe65
## What changes were proposed in this pull request? Integrate Apache Arrow with Spark to increase performance of `DataFrame.toPandas`. This has been done by using Arrow to convert data partitions on the executor JVM to Arrow payload byte arrays where they are then served to the Python process. The Python DataFrame can then collect the Arrow payloads where they are combined and converted to a Pandas DataFrame. Data types except complex, date, timestamp, and decimal are currently supported, otherwise an `UnsupportedOperation` exception is thrown. Additions to Spark include a Scala package private method `Dataset.toArrowPayload` that will convert data partitions in the executor JVM to `ArrowPayload`s as byte arrays so they can be easily served. A package private class/object `ArrowConverters` that provide data type mappings and conversion routines. In Python, a private method `DataFrame._collectAsArrow` is added to collect Arrow payloads and a SQLConf "spark.sql.execution.arrow.enable" can be used in `toPandas()` to enable using Arrow (uses the old conversion by default). ## How was this patch tested? Added a new test suite `ArrowConvertersSuite` that will run tests on conversion of Datasets to Arrow payloads for supported types. The suite will generate a Dataset and matching Arrow JSON data, then the dataset is converted to an Arrow payload and finally validated against the JSON data. This will ensure that the schema and data has been converted correctly. Added PySpark tests to verify the `toPandas` method is producing equal DataFrames with and without pyarrow. A roundtrip test to ensure the pandas DataFrame produced by pyspark is equal to a one made directly with pandas. Author: Bryan Cutler <cutlerb@gmail.com> Author: Li Jin <ice.xelloss@gmail.com> Author: Li Jin <li.jin@twosigma.com> Author: Wes McKinney <wes.mckinney@twosigma.com> Closes #18459 from BryanCutler/toPandas_with_arrow-SPARK-13534.
78 lines
2.9 KiB
Bash
Executable file
78 lines
2.9 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
if [ -z "${SPARK_HOME}" ]; then
|
|
source "$(dirname "$0")"/find-spark-home
|
|
fi
|
|
|
|
source "${SPARK_HOME}"/bin/load-spark-env.sh
|
|
export _SPARK_CMD_USAGE="Usage: ./bin/pyspark [options]"
|
|
|
|
# In Spark 2.0, IPYTHON and IPYTHON_OPTS are removed and pyspark fails to launch if either option
|
|
# is set in the user's environment. Instead, users should set PYSPARK_DRIVER_PYTHON=ipython
|
|
# to use IPython and set PYSPARK_DRIVER_PYTHON_OPTS to pass options when starting the Python driver
|
|
# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook'). This supports full customization of the IPython
|
|
# and executor Python executables.
|
|
|
|
# Fail noisily if removed options are set
|
|
if [[ -n "$IPYTHON" || -n "$IPYTHON_OPTS" ]]; then
|
|
echo "Error in pyspark startup:"
|
|
echo "IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead."
|
|
exit 1
|
|
fi
|
|
|
|
# Default to standard python interpreter unless told otherwise
|
|
if [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then
|
|
PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"python"}"
|
|
fi
|
|
|
|
WORKS_WITH_IPYTHON=$(python -c 'import sys; print(sys.version_info >= (2, 7, 0))')
|
|
|
|
# Determine the Python executable to use for the executors:
|
|
if [[ -z "$PYSPARK_PYTHON" ]]; then
|
|
if [[ $PYSPARK_DRIVER_PYTHON == *ipython* && ! $WORKS_WITH_IPYTHON ]]; then
|
|
echo "IPython requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON" 1>&2
|
|
exit 1
|
|
else
|
|
PYSPARK_PYTHON=python
|
|
fi
|
|
fi
|
|
export PYSPARK_PYTHON
|
|
|
|
# Add the PySpark classes to the Python path:
|
|
export PYTHONPATH="${SPARK_HOME}/python/:$PYTHONPATH"
|
|
export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.6-src.zip:$PYTHONPATH"
|
|
|
|
# Load the PySpark shell.py script when ./pyspark is used interactively:
|
|
export OLD_PYTHONSTARTUP="$PYTHONSTARTUP"
|
|
export PYTHONSTARTUP="${SPARK_HOME}/python/pyspark/shell.py"
|
|
|
|
# For pyspark tests
|
|
if [[ -n "$SPARK_TESTING" ]]; then
|
|
unset YARN_CONF_DIR
|
|
unset HADOOP_CONF_DIR
|
|
export PYTHONHASHSEED=0
|
|
exec "$PYSPARK_DRIVER_PYTHON" -m "$@"
|
|
exit
|
|
fi
|
|
|
|
export PYSPARK_DRIVER_PYTHON
|
|
export PYSPARK_DRIVER_PYTHON_OPTS
|
|
exec "${SPARK_HOME}"/bin/spark-submit pyspark-shell-main --name "PySparkShell" "$@"
|