4e9b551a0b
This pull request addresses a few issues related to PySpark's IPython support: - Fix the remaining uses of the '-u' flag, which IPython doesn't support (see SPARK-3772). - Change PYSPARK_PYTHON_OPTS to PYSPARK_DRIVER_PYTHON_OPTS, so that the old name is reserved in case we ever want to allow the worker Python options to be customized (this variable was introduced in #2554 and hasn't landed in a release yet, so this doesn't break any compatibility). - Introduce a PYSPARK_DRIVER_PYTHON option that allows the driver to use `ipython` while the workers use a different Python version. - Attempt to use Python 2.7 by default if PYSPARK_PYTHON is not specified. - Retain the old semantics for IPYTHON=1 and IPYTHON_OPTS (to avoid breaking existing example programs). There are more details in a block comment in `bin/pyspark`. Author: Josh Rosen <joshrosen@apache.org> Closes #2651 from JoshRosen/SPARK-3772 and squashes the following commits: 7b8eb86 [Josh Rosen] More changes to PySpark python executable configuration: c4f5778 [Josh Rosen] [SPARK-3772] Allow ipython to be used by Pyspark workers; IPython fixes: |
||
---|---|---|
.. | ||
beeline | ||
compute-classpath.cmd | ||
compute-classpath.sh | ||
load-spark-env.sh | ||
pyspark | ||
pyspark.cmd | ||
pyspark2.cmd | ||
run-example | ||
run-example.cmd | ||
run-example2.cmd | ||
spark-class | ||
spark-class.cmd | ||
spark-class2.cmd | ||
spark-shell | ||
spark-shell.cmd | ||
spark-sql | ||
spark-submit | ||
spark-submit.cmd | ||
utils.sh |