2014-08-06 15:58:24 -04:00
|
|
|
#!/usr/bin/env bash
|
|
|
|
#
|
|
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
|
|
# this work for additional information regarding copyright ownership.
|
|
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
|
|
# (the "License"); you may not use this file except in compliance with
|
|
|
|
# the License. You may obtain a copy of the License at
|
|
|
|
#
|
|
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
#
|
|
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
# See the License for the specific language governing permissions and
|
|
|
|
# limitations under the License.
|
|
|
|
#
|
2018-11-20 15:38:40 -05:00
|
|
|
# define test binaries + versions
|
|
|
|
FLAKE8_BUILD="flake8"
|
[SPARK-34890][PYTHON] Port/integrate Koalas main codes into PySpark
### What changes were proposed in this pull request?
As a first step of [SPARK-34849](https://issues.apache.org/jira/browse/SPARK-34849), this PR proposes porting the Koalas main code into PySpark.
This PR contains minimal changes to the existing Koalas code as follows:
1. `databricks.koalas` -> `pyspark.pandas`
2. `from databricks import koalas as ks` -> `from pyspark import pandas as pp`
3. `ks.xxx -> pp.xxx`
Other than them:
1. Added a line to `python/mypy.ini` in order to ignore the mypy test. See related issue at [SPARK-34941](https://issues.apache.org/jira/browse/SPARK-34941).
2. Added a comment to several lines in several files to ignore the flake8 F401. See related issue at [SPARK-34943](https://issues.apache.org/jira/browse/SPARK-34943).
When this PR is merged, all the features that were previously used in [Koalas](https://github.com/databricks/koalas) will be available in PySpark as well.
Users can access to the pandas API in PySpark as below:
```python
>>> from pyspark import pandas as pp
>>> ppdf = pp.DataFrame({"A": [1, 2, 3], "B": [15, 20, 25]})
>>> ppdf
A B
0 1 15
1 2 20
2 3 25
```
The existing "options and settings" in Koalas are also available in the same way:
```python
>>> from pyspark.pandas.config import set_option, reset_option, get_option
>>> ppser1 = pp.Series([1, 2, 3])
>>> ppser2 = pp.Series([3, 4, 5])
>>> ppser1 + ppser2
Traceback (most recent call last):
...
ValueError: Cannot combine the series or dataframe because it comes from a different dataframe. In order to allow this operation, enable 'compute.ops_on_diff_frames' option.
>>> set_option("compute.ops_on_diff_frames", True)
>>> ppser1 + ppser2
0 4
1 6
2 8
dtype: int64
```
Please also refer to the [API Reference](https://koalas.readthedocs.io/en/latest/reference/index.html) and [Options and Settings](https://koalas.readthedocs.io/en/latest/user_guide/options.html) for more detail.
**NOTE** that this PR intentionally ports the main codes of Koalas first almost as are with minimal changes because:
- Koalas project is fairly large. Making some changes together for PySpark will make it difficult to review the individual change.
Koalas dev includes multiple Spark committers who will review. By doing this, the committers will be able to more easily and effectively review and drive the development.
- Koalas tests and documentation require major changes to make it look great together with PySpark whereas main codes do not require.
- We lately froze the Koalas codebase, and plan to work together on the initial porting. By porting the main codes first as are, it unblocks the Koalas dev to work on other items in parallel.
I promise and will make sure on:
- Rename Koalas to PySpark pandas APIs and/or pandas-on-Spark accordingly in documentation, and the docstrings and comments in the main codes.
- Triage APIs to remove that don’t make sense when Koalas is in PySpark
The documentation changes will be tracked in [SPARK-34885](https://issues.apache.org/jira/browse/SPARK-34885), the test code changes will be tracked in [SPARK-34886](https://issues.apache.org/jira/browse/SPARK-34886).
### Why are the changes needed?
Please refer to:
- [[DISCUSS] Support pandas API layer on PySpark](http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Support-pandas-API-layer-on-PySpark-td30945.html)
- [[VOTE] SPIP: Support pandas API layer on PySpark](http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-SPIP-Support-pandas-API-layer-on-PySpark-td30996.html)
### Does this PR introduce _any_ user-facing change?
Yes, now users can use the pandas APIs on Spark
### How was this patch tested?
Manually tested for exposed major APIs and options as described above.
### Koalas contributors
Koalas would not have been possible without the following contributors:
ueshin
HyukjinKwon
rxin
xinrong-databricks
RainFung
charlesdong1991
harupy
floscha
beobest2
thunterdb
garawalid
LucasG0
shril
deepyaman
gioa
fwani
90jam
thoo
AbdealiJK
abishekganesh72
gliptak
DumbMachine
dvgodoy
stbof
nitlev
hjoo
gatorsmile
tomspur
icexelloss
awdavidson
guyao
akhilputhiry
scook12
patryk-oleniuk
tracek
dennyglee
athena15
gstaubli
WeichenXu123
hsubbaraj
lfdversluis
ktksq
shengjh
margaret-databricks
LSturtew
sllynn
manuzhang
jijosg
sadikovi
Closes #32036 from itholic/SPARK-34890.
Authored-by: itholic <haejoon.lee@databricks.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
2021-04-05 23:42:39 -04:00
|
|
|
# TODO(SPARK-34943): minimum version should be 3.8+
|
2018-11-07 23:26:21 -05:00
|
|
|
MINIMUM_FLAKE8="3.5.0"
|
2020-10-19 15:50:01 -04:00
|
|
|
MYPY_BUILD="mypy"
|
2018-11-20 15:38:40 -05:00
|
|
|
PYCODESTYLE_BUILD="pycodestyle"
|
2021-04-14 11:07:05 -04:00
|
|
|
MINIMUM_PYCODESTYLE="2.7.0"
|
2014-08-06 15:58:24 -04:00
|
|
|
|
2018-11-20 15:38:40 -05:00
|
|
|
SPHINX_BUILD="sphinx-build"
|
2014-08-06 15:58:24 -04:00
|
|
|
|
2020-02-05 20:01:33 -05:00
|
|
|
PYTHON_EXECUTABLE="python3"
|
|
|
|
|
2020-03-15 00:09:35 -04:00
|
|
|
function satisfies_min_version {
|
|
|
|
local provided_version="$1"
|
|
|
|
local expected_version="$2"
|
|
|
|
echo "$(
|
|
|
|
"$PYTHON_EXECUTABLE" << EOM
|
|
|
|
from setuptools.extern.packaging import version
|
|
|
|
print(version.parse('$provided_version') >= version.parse('$expected_version'))
|
|
|
|
EOM
|
|
|
|
)"
|
|
|
|
}
|
|
|
|
|
2018-11-20 15:38:40 -05:00
|
|
|
function compile_python_test {
|
|
|
|
local COMPILE_STATUS=
|
|
|
|
local COMPILE_REPORT=
|
|
|
|
|
|
|
|
if [[ ! "$1" ]]; then
|
|
|
|
echo "No python files found! Something is very wrong -- exiting."
|
|
|
|
exit 1;
|
|
|
|
fi
|
2015-03-19 15:46:10 -04:00
|
|
|
|
2020-01-30 06:17:29 -05:00
|
|
|
# compileall: https://docs.python.org/3/library/compileall.html
|
2018-11-20 15:38:40 -05:00
|
|
|
echo "starting python compilation test..."
|
2020-02-05 20:01:33 -05:00
|
|
|
COMPILE_REPORT=$( ("$PYTHON_EXECUTABLE" -B -mcompileall -q -l -x "[/\\\\][.]git" $1) 2>&1)
|
2018-11-20 15:38:40 -05:00
|
|
|
COMPILE_STATUS=$?
|
|
|
|
|
|
|
|
if [ $COMPILE_STATUS -ne 0 ]; then
|
|
|
|
echo "Python compilation failed with the following errors:"
|
|
|
|
echo "$COMPILE_REPORT"
|
|
|
|
echo "$COMPILE_STATUS"
|
|
|
|
exit "$COMPILE_STATUS"
|
|
|
|
else
|
|
|
|
echo "python compilation succeeded."
|
|
|
|
echo
|
|
|
|
fi
|
|
|
|
}
|
2014-08-06 15:58:24 -04:00
|
|
|
|
2018-11-20 15:38:40 -05:00
|
|
|
function pycodestyle_test {
|
|
|
|
local PYCODESTYLE_STATUS=
|
|
|
|
local PYCODESTYLE_REPORT=
|
|
|
|
local RUN_LOCAL_PYCODESTYLE=
|
2020-03-15 00:09:35 -04:00
|
|
|
local PYCODESTYLE_VERSION=
|
2018-11-20 15:38:40 -05:00
|
|
|
local EXPECTED_PYCODESTYLE=
|
|
|
|
local PYCODESTYLE_SCRIPT_PATH="$SPARK_ROOT_DIR/dev/pycodestyle-$MINIMUM_PYCODESTYLE.py"
|
|
|
|
local PYCODESTYLE_SCRIPT_REMOTE_PATH="https://raw.githubusercontent.com/PyCQA/pycodestyle/$MINIMUM_PYCODESTYLE/pycodestyle.py"
|
2014-08-06 15:58:24 -04:00
|
|
|
|
2018-11-20 15:38:40 -05:00
|
|
|
if [[ ! "$1" ]]; then
|
|
|
|
echo "No python files found! Something is very wrong -- exiting."
|
|
|
|
exit 1;
|
2015-04-18 19:46:28 -04:00
|
|
|
fi
|
2018-11-20 15:38:40 -05:00
|
|
|
|
|
|
|
# check for locally installed pycodestyle & version
|
|
|
|
RUN_LOCAL_PYCODESTYLE="False"
|
|
|
|
if hash "$PYCODESTYLE_BUILD" 2> /dev/null; then
|
2020-03-15 00:09:35 -04:00
|
|
|
PYCODESTYLE_VERSION="$($PYCODESTYLE_BUILD --version)"
|
|
|
|
EXPECTED_PYCODESTYLE="$(satisfies_min_version $PYCODESTYLE_VERSION $MINIMUM_PYCODESTYLE)"
|
2018-11-20 15:38:40 -05:00
|
|
|
if [ "$EXPECTED_PYCODESTYLE" == "True" ]; then
|
|
|
|
RUN_LOCAL_PYCODESTYLE="True"
|
2018-11-07 23:26:21 -05:00
|
|
|
fi
|
2018-11-20 15:38:40 -05:00
|
|
|
fi
|
[SPARK-25270] lint-python: Add flake8 to find syntax errors and undefined names
## What changes were proposed in this pull request?
Add [flake8](http://flake8.pycqa.org) tests to find Python syntax errors and undefined names.
__E901,E999,F821,F822,F823__ are the "_showstopper_" flake8 issues that can halt the runtime with a SyntaxError, NameError, etc. Most other flake8 issues are merely "style violations" -- useful for readability but they do not effect runtime safety.
* F821: undefined name `name`
* F822: undefined name `name` in `__all__`
* F823: local variable name referenced before assignment
* E901: SyntaxError or IndentationError
* E999: SyntaxError -- failed to compile a file into an Abstract Syntax Tree
## How was this patch tested?
$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
$ __flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics__
Please review http://spark.apache.org/contributing.html before opening a pull request.
Closes #22266 from cclauss/patch-3.
Authored-by: cclauss <cclauss@bluewin.ch>
Signed-off-by: Holden Karau <holden@pigscanfly.ca>
2018-09-07 12:35:25 -04:00
|
|
|
|
2018-11-20 15:38:40 -05:00
|
|
|
# download the right version or run locally
|
|
|
|
if [ $RUN_LOCAL_PYCODESTYLE == "False" ]; then
|
|
|
|
# Get pycodestyle at runtime so that we don't rely on it being installed on the build server.
|
|
|
|
# See: https://github.com/apache/spark/pull/1744#issuecomment-50982162
|
|
|
|
# Updated to the latest official version of pep8. pep8 is formally renamed to pycodestyle.
|
|
|
|
echo "downloading pycodestyle from $PYCODESTYLE_SCRIPT_REMOTE_PATH..."
|
|
|
|
if [ ! -e "$PYCODESTYLE_SCRIPT_PATH" ]; then
|
|
|
|
curl --silent -o "$PYCODESTYLE_SCRIPT_PATH" "$PYCODESTYLE_SCRIPT_REMOTE_PATH"
|
|
|
|
local curl_status="$?"
|
|
|
|
|
|
|
|
if [ "$curl_status" -ne 0 ]; then
|
|
|
|
echo "Failed to download pycodestyle.py from $PYCODESTYLE_SCRIPT_REMOTE_PATH"
|
|
|
|
exit "$curl_status"
|
|
|
|
fi
|
2018-11-07 23:26:21 -05:00
|
|
|
fi
|
2018-11-20 15:38:40 -05:00
|
|
|
|
|
|
|
echo "starting pycodestyle test..."
|
2020-02-05 20:01:33 -05:00
|
|
|
PYCODESTYLE_REPORT=$( ("$PYTHON_EXECUTABLE" "$PYCODESTYLE_SCRIPT_PATH" --config=dev/tox.ini $1) 2>&1)
|
2018-11-20 15:38:40 -05:00
|
|
|
PYCODESTYLE_STATUS=$?
|
|
|
|
else
|
|
|
|
# we have the right version installed, so run locally
|
|
|
|
echo "starting pycodestyle test..."
|
|
|
|
PYCODESTYLE_REPORT=$( ($PYCODESTYLE_BUILD --config=dev/tox.ini $1) 2>&1)
|
|
|
|
PYCODESTYLE_STATUS=$?
|
|
|
|
fi
|
|
|
|
|
|
|
|
if [ $PYCODESTYLE_STATUS -ne 0 ]; then
|
|
|
|
echo "pycodestyle checks failed:"
|
|
|
|
echo "$PYCODESTYLE_REPORT"
|
|
|
|
exit "$PYCODESTYLE_STATUS"
|
2018-11-07 23:26:21 -05:00
|
|
|
else
|
2018-11-20 15:38:40 -05:00
|
|
|
echo "pycodestyle checks passed."
|
|
|
|
echo
|
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
2020-10-19 15:50:01 -04:00
|
|
|
function mypy_test {
|
|
|
|
local MYPY_REPORT=
|
|
|
|
local MYPY_STATUS=
|
|
|
|
|
2020-10-27 01:03:57 -04:00
|
|
|
# TODO(SPARK-32797): Install mypy on the Jenkins CI workers
|
2020-10-19 15:50:01 -04:00
|
|
|
if ! hash "$MYPY_BUILD" 2> /dev/null; then
|
|
|
|
echo "The $MYPY_BUILD command was not found. Skipping for now."
|
|
|
|
return
|
|
|
|
fi
|
|
|
|
|
|
|
|
echo "starting $MYPY_BUILD test..."
|
|
|
|
MYPY_REPORT=$( ($MYPY_BUILD --config-file python/mypy.ini python/pyspark) 2>&1)
|
|
|
|
MYPY_STATUS=$?
|
|
|
|
|
|
|
|
if [ "$MYPY_STATUS" -ne 0 ]; then
|
|
|
|
echo "mypy checks failed:"
|
|
|
|
echo "$MYPY_REPORT"
|
|
|
|
echo "$MYPY_STATUS"
|
|
|
|
exit "$MYPY_STATUS"
|
|
|
|
else
|
|
|
|
echo "mypy checks passed."
|
|
|
|
echo
|
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
2018-11-20 15:38:40 -05:00
|
|
|
function flake8_test {
|
|
|
|
local FLAKE8_VERSION=
|
|
|
|
local EXPECTED_FLAKE8=
|
|
|
|
local FLAKE8_REPORT=
|
|
|
|
local FLAKE8_STATUS=
|
|
|
|
|
|
|
|
if ! hash "$FLAKE8_BUILD" 2> /dev/null; then
|
|
|
|
echo "The flake8 command was not found."
|
2018-11-07 23:26:21 -05:00
|
|
|
echo "flake8 checks failed."
|
|
|
|
exit 1
|
|
|
|
fi
|
2018-10-27 09:20:42 -04:00
|
|
|
|
2020-03-15 00:09:35 -04:00
|
|
|
_FLAKE8_VERSION=($($FLAKE8_BUILD --version))
|
|
|
|
FLAKE8_VERSION="${_FLAKE8_VERSION[0]}"
|
|
|
|
EXPECTED_FLAKE8="$(satisfies_min_version $FLAKE8_VERSION $MINIMUM_FLAKE8)"
|
2018-11-20 15:38:40 -05:00
|
|
|
|
|
|
|
if [[ "$EXPECTED_FLAKE8" == "False" ]]; then
|
|
|
|
echo "\
|
|
|
|
The minimum flake8 version needs to be $MINIMUM_FLAKE8. Your current version is $FLAKE8_VERSION
|
|
|
|
|
|
|
|
flake8 checks failed."
|
|
|
|
exit 1
|
|
|
|
fi
|
|
|
|
|
|
|
|
echo "starting $FLAKE8_BUILD test..."
|
2020-08-08 11:51:57 -04:00
|
|
|
FLAKE8_REPORT=$( ($FLAKE8_BUILD --append-config dev/tox.ini --count --show-source --statistics .) 2>&1)
|
2018-11-20 15:38:40 -05:00
|
|
|
FLAKE8_STATUS=$?
|
|
|
|
|
|
|
|
if [ "$FLAKE8_STATUS" -ne 0 ]; then
|
|
|
|
echo "flake8 checks failed:"
|
|
|
|
echo "$FLAKE8_REPORT"
|
|
|
|
echo "$FLAKE8_STATUS"
|
|
|
|
exit "$FLAKE8_STATUS"
|
2018-10-27 09:20:42 -04:00
|
|
|
else
|
2018-11-20 15:38:40 -05:00
|
|
|
echo "flake8 checks passed."
|
|
|
|
echo
|
2018-10-27 09:20:42 -04:00
|
|
|
fi
|
2018-11-20 15:38:40 -05:00
|
|
|
}
|
|
|
|
|
|
|
|
function sphinx_test {
|
|
|
|
local SPHINX_REPORT=
|
|
|
|
local SPHINX_STATUS=
|
|
|
|
|
|
|
|
# Check that the documentation builds acceptably, skip check if sphinx is not installed.
|
|
|
|
if ! hash "$SPHINX_BUILD" 2> /dev/null; then
|
2020-07-13 22:22:44 -04:00
|
|
|
echo "The $SPHINX_BUILD command was not found. Skipping Sphinx build for now."
|
|
|
|
echo
|
|
|
|
return
|
|
|
|
fi
|
|
|
|
|
|
|
|
PYTHON_HAS_SPHINX=$("$PYTHON_EXECUTABLE" -c 'import importlib.util; print(importlib.util.find_spec("sphinx") is not None)')
|
|
|
|
if [[ "$PYTHON_HAS_SPHINX" == "False" ]]; then
|
|
|
|
echo "$PYTHON_EXECUTABLE does not have Sphinx installed. Skipping Sphinx build for now."
|
2018-11-20 15:38:40 -05:00
|
|
|
echo
|
|
|
|
return
|
2020-07-27 04:49:21 -04:00
|
|
|
fi
|
|
|
|
|
|
|
|
# TODO(SPARK-32407): Sphinx 3.1+ does not correctly index nested classes.
|
|
|
|
# See also https://github.com/sphinx-doc/sphinx/issues/7551.
|
|
|
|
PYTHON_HAS_SPHINX_3_0=$("$PYTHON_EXECUTABLE" -c 'from distutils.version import LooseVersion; import sphinx; print(LooseVersion(sphinx.__version__) < LooseVersion("3.1.0"))')
|
|
|
|
if [[ "$PYTHON_HAS_SPHINX_3_0" == "False" ]]; then
|
2021-02-01 19:30:50 -05:00
|
|
|
echo "$PYTHON_EXECUTABLE has Sphinx 3.1+ installed but it requires lower than 3.1. Skipping Sphinx build for now."
|
2020-07-27 04:49:21 -04:00
|
|
|
echo
|
|
|
|
return
|
|
|
|
fi
|
|
|
|
|
|
|
|
# TODO(SPARK-32391): Install pydata_sphinx_theme in Jenkins machines
|
|
|
|
PYTHON_HAS_THEME=$("$PYTHON_EXECUTABLE" -c 'import importlib.util; print(importlib.util.find_spec("pydata_sphinx_theme") is not None)')
|
|
|
|
if [[ "$PYTHON_HAS_THEME" == "False" ]]; then
|
|
|
|
echo "$PYTHON_EXECUTABLE does not have pydata_sphinx_theme installed. Skipping Sphinx build for now."
|
|
|
|
echo
|
|
|
|
return
|
[SPARK-32204][SPARK-32182][DOCS] Add a quickstart page with Binder integration in PySpark documentation
### What changes were proposed in this pull request?
This PR proposes to:
- add a notebook with a Binder integration which allows users to try PySpark in a live notebook. Please [try this here](https://mybinder.org/v2/gh/HyukjinKwon/spark/SPARK-32204?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart.ipynb).
- reuse this notebook as a quickstart guide in PySpark documentation.
Note that Binder turns a Git repo into a collection of interactive notebooks. It works based on Docker image. Once somebody builds, other people can reuse the image against a specific commit.
Therefore, if we run Binder with the images based on released tags in Spark, virtually all users can instantly launch the Jupyter notebooks.
<br/>
I made a simple demo to make it easier to review. Please see:
- [Main page](https://hyukjin-spark.readthedocs.io/en/stable/). Note that the link ("Live Notebook") in the main page wouldn't work since this PR is not merged yet.
- [Quickstart page](https://hyukjin-spark.readthedocs.io/en/stable/getting_started/quickstart.html)
<br/>
When reviewing the notebook file itself, please give my direct feedback which I will appreciate and address.
Another way might be:
- open [here](https://mybinder.org/v2/gh/HyukjinKwon/spark/SPARK-32204?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart.ipynb).
- edit / change / update the notebook. Please feel free to change as whatever you want. I can apply as are or slightly update more when I apply to this PR.
- download it as a `.ipynb` file:
![Screen Shot 2020-08-20 at 10 12 19 PM](https://user-images.githubusercontent.com/6477701/90774311-3e38c800-e332-11ea-8476-699a653984db.png)
- upload the `.ipynb` file here in a GitHub comment. Then, I will push a commit with that file with crediting correctly, of course.
- alternatively, push a commit into this PR right away if that's easier for you (if you're a committer).
References:
- https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html
- https://databricks.com/jp/blog/2020/03/31/10-minutes-from-pandas-to-koalas-on-apache-spark.html - my own blog post .. :-) and https://koalas.readthedocs.io/en/latest/getting_started/10min.html
### Why are the changes needed?
To improve PySpark's usability. The current quickstart for Python users are very friendly.
### Does this PR introduce _any_ user-facing change?
Yes, it will add a documentation page, and expose a live notebook to PySpark users.
### How was this patch tested?
Manually tested, and GitHub Actions builds will test.
Closes #29491 from HyukjinKwon/SPARK-32204.
Lead-authored-by: HyukjinKwon <gurwls223@apache.org>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
2020-08-25 23:23:24 -04:00
|
|
|
fi
|
|
|
|
|
|
|
|
# TODO(SPARK-32666): Install nbsphinx in Jenkins machines
|
|
|
|
PYTHON_HAS_NBSPHINX=$("$PYTHON_EXECUTABLE" -c 'import importlib.util; print(importlib.util.find_spec("nbsphinx") is not None)')
|
|
|
|
if [[ "$PYTHON_HAS_NBSPHINX" == "False" ]]; then
|
|
|
|
echo "$PYTHON_EXECUTABLE does not have nbsphinx installed. Skipping Sphinx build for now."
|
|
|
|
echo
|
|
|
|
return
|
|
|
|
fi
|
|
|
|
|
|
|
|
# TODO(SPARK-32666): Install ipython in Jenkins machines
|
2020-09-09 00:22:13 -04:00
|
|
|
PYTHON_HAS_IPYTHON=$("$PYTHON_EXECUTABLE" -c 'import importlib.util; print(importlib.util.find_spec("IPython") is not None)')
|
[SPARK-32204][SPARK-32182][DOCS] Add a quickstart page with Binder integration in PySpark documentation
### What changes were proposed in this pull request?
This PR proposes to:
- add a notebook with a Binder integration which allows users to try PySpark in a live notebook. Please [try this here](https://mybinder.org/v2/gh/HyukjinKwon/spark/SPARK-32204?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart.ipynb).
- reuse this notebook as a quickstart guide in PySpark documentation.
Note that Binder turns a Git repo into a collection of interactive notebooks. It works based on Docker image. Once somebody builds, other people can reuse the image against a specific commit.
Therefore, if we run Binder with the images based on released tags in Spark, virtually all users can instantly launch the Jupyter notebooks.
<br/>
I made a simple demo to make it easier to review. Please see:
- [Main page](https://hyukjin-spark.readthedocs.io/en/stable/). Note that the link ("Live Notebook") in the main page wouldn't work since this PR is not merged yet.
- [Quickstart page](https://hyukjin-spark.readthedocs.io/en/stable/getting_started/quickstart.html)
<br/>
When reviewing the notebook file itself, please give my direct feedback which I will appreciate and address.
Another way might be:
- open [here](https://mybinder.org/v2/gh/HyukjinKwon/spark/SPARK-32204?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart.ipynb).
- edit / change / update the notebook. Please feel free to change as whatever you want. I can apply as are or slightly update more when I apply to this PR.
- download it as a `.ipynb` file:
![Screen Shot 2020-08-20 at 10 12 19 PM](https://user-images.githubusercontent.com/6477701/90774311-3e38c800-e332-11ea-8476-699a653984db.png)
- upload the `.ipynb` file here in a GitHub comment. Then, I will push a commit with that file with crediting correctly, of course.
- alternatively, push a commit into this PR right away if that's easier for you (if you're a committer).
References:
- https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html
- https://databricks.com/jp/blog/2020/03/31/10-minutes-from-pandas-to-koalas-on-apache-spark.html - my own blog post .. :-) and https://koalas.readthedocs.io/en/latest/getting_started/10min.html
### Why are the changes needed?
To improve PySpark's usability. The current quickstart for Python users are very friendly.
### Does this PR introduce _any_ user-facing change?
Yes, it will add a documentation page, and expose a live notebook to PySpark users.
### How was this patch tested?
Manually tested, and GitHub Actions builds will test.
Closes #29491 from HyukjinKwon/SPARK-32204.
Lead-authored-by: HyukjinKwon <gurwls223@apache.org>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
2020-08-25 23:23:24 -04:00
|
|
|
if [[ "$PYTHON_HAS_IPYTHON" == "False" ]]; then
|
|
|
|
echo "$PYTHON_EXECUTABLE does not have ipython installed. Skipping Sphinx build for now."
|
|
|
|
echo
|
|
|
|
return
|
2018-11-20 15:38:40 -05:00
|
|
|
fi
|
2020-10-27 01:03:57 -04:00
|
|
|
|
|
|
|
# TODO(SPARK-33242): Install numpydoc in Jenkins machines
|
|
|
|
PYTHON_HAS_NUMPYDOC=$("$PYTHON_EXECUTABLE" -c 'import importlib.util; print(importlib.util.find_spec("numpydoc") is not None)')
|
|
|
|
if [[ "$PYTHON_HAS_NUMPYDOC" == "False" ]]; then
|
|
|
|
echo "$PYTHON_EXECUTABLE does not have numpydoc installed. Skipping Sphinx build for now."
|
|
|
|
echo
|
|
|
|
return
|
|
|
|
fi
|
2018-11-20 15:38:40 -05:00
|
|
|
|
|
|
|
echo "starting $SPHINX_BUILD tests..."
|
|
|
|
pushd python/docs &> /dev/null
|
|
|
|
make clean &> /dev/null
|
|
|
|
# Treat warnings as errors so we stop correctly
|
|
|
|
SPHINX_REPORT=$( (SPHINXOPTS="-a -W" make html) 2>&1)
|
|
|
|
SPHINX_STATUS=$?
|
|
|
|
|
|
|
|
if [ "$SPHINX_STATUS" -ne 0 ]; then
|
|
|
|
echo "$SPHINX_BUILD checks failed:"
|
|
|
|
echo "$SPHINX_REPORT"
|
|
|
|
echo
|
|
|
|
echo "re-running make html to print full warning list:"
|
|
|
|
make clean &> /dev/null
|
|
|
|
SPHINX_REPORT=$( (SPHINXOPTS="-a" make html) 2>&1)
|
|
|
|
echo "$SPHINX_REPORT"
|
|
|
|
exit "$SPHINX_STATUS"
|
|
|
|
else
|
|
|
|
echo "$SPHINX_BUILD checks passed."
|
|
|
|
echo
|
|
|
|
fi
|
|
|
|
|
|
|
|
popd &> /dev/null
|
|
|
|
}
|
|
|
|
|
|
|
|
SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
|
|
|
|
SPARK_ROOT_DIR="$(dirname "${SCRIPT_DIR}")"
|
|
|
|
|
|
|
|
pushd "$SPARK_ROOT_DIR" &> /dev/null
|
|
|
|
|
[SPARK-34433][DOCS] Lock Jekyll version by Gemfile and Bundler
### What changes were proposed in this pull request?
Improving the documentation and release process by pinning Jekyll version by Gemfile and Bundler.
Some files and their responsibilities within this PR:
- `docs/.bundle/config` is used to specify a directory "docs/.local_ruby_bundle" which will be used as destination to install the ruby packages into instead of the global one which requires root access
- `docs/Gemfile` is specifying the required Jekyll version and other top level gem versions
- `docs/Gemfile.lock` is generated by the "bundle install". This file contains the exact resolved versions of all the gems including the top level gems and all the direct and transitive dependencies of those gems. When this file is generated it contains a platform related section "PLATFORMS" (in my case after the generation it was "universal-darwin-19"). Still this file must be under version control as when the version of a gem does not fit to the one specified in `Gemfile` an error comes (i.e. if the `Gemfile.lock` was generated for Jekyll 4.1.0 and its version is updated in the `Gemfile` to 4.2.0 then it triggers the error: "The bundle currently has jekyll locked at 4.1.0."). This is solution is also suggested officially in [its documentation](https://bundler.io/rationale.html#checking-your-code-into-version-control). To get rid of the specific platform (like "universal-darwin-19") first we have to add "ruby" as platform [which means this should work on every platform where Ruby runs](https://guides.rubygems.org/what-is-a-gem/)) by running "bundle lock --add-platform ruby" then the specific platform can be removed by "bundle lock --remove-platform universal-darwin-19".
After this the correct process to update Jekyll version is the following:
1. update the version in `Gemfile`
2. run "bundle update" which updates the `Gemfile.lock`
3. commit both files
This process for version update is tested for details please check the testing section.
### Why are the changes needed?
Using different Jekyll versions can generate different output documents.
This PR standardize the process.
### Does this PR introduce _any_ user-facing change?
No, assuming the release was done via docker by using `do-release-docker.sh`.
In that case there should be no difference at all as the same Jekyll version is specified in the Gemfile.
### How was this patch tested?
#### Testing document generation
Doc generation step was triggered via the docker release:
```
$ ./do-release-docker.sh -d ~/working -n -s docs
...
========================
= Building documentation...
Command: /opt/spark-rm/release-build.sh docs
Log file: docs.log
Skipping publish step.
```
The docs.log contains the followings:
```
Building Spark docs
Fetching gem metadata from https://rubygems.org/.........
Using bundler 2.2.9
Fetching rb-fsevent 0.10.4
Fetching forwardable-extended 2.6.0
Fetching public_suffix 4.0.6
Fetching colorator 1.1.0
Fetching eventmachine 1.2.7
Fetching http_parser.rb 0.6.0
Fetching ffi 1.14.2
Fetching concurrent-ruby 1.1.8
Installing colorator 1.1.0
Installing forwardable-extended 2.6.0
Installing rb-fsevent 0.10.4
Installing public_suffix 4.0.6
Installing http_parser.rb 0.6.0 with native extensions
Installing eventmachine 1.2.7 with native extensions
Installing concurrent-ruby 1.1.8
Fetching rexml 3.2.4
Fetching liquid 4.0.3
Installing ffi 1.14.2 with native extensions
Installing rexml 3.2.4
Installing liquid 4.0.3
Fetching mercenary 0.4.0
Installing mercenary 0.4.0
Fetching rouge 3.26.0
Installing rouge 3.26.0
Fetching safe_yaml 1.0.5
Installing safe_yaml 1.0.5
Fetching unicode-display_width 1.7.0
Installing unicode-display_width 1.7.0
Fetching webrick 1.7.0
Installing webrick 1.7.0
Fetching pathutil 0.16.2
Fetching kramdown 2.3.0
Fetching terminal-table 2.0.0
Fetching addressable 2.7.0
Fetching i18n 1.8.9
Installing terminal-table 2.0.0
Installing pathutil 0.16.2
Installing i18n 1.8.9
Installing addressable 2.7.0
Installing kramdown 2.3.0
Fetching kramdown-parser-gfm 1.1.0
Installing kramdown-parser-gfm 1.1.0
Fetching rb-inotify 0.10.1
Fetching sassc 2.4.0
Fetching em-websocket 0.5.2
Installing rb-inotify 0.10.1
Installing em-websocket 0.5.2
Installing sassc 2.4.0 with native extensions
Fetching listen 3.4.1
Installing listen 3.4.1
Fetching jekyll-watch 2.2.1
Installing jekyll-watch 2.2.1
Fetching jekyll-sass-converter 2.1.0
Installing jekyll-sass-converter 2.1.0
Fetching jekyll 4.2.0
Installing jekyll 4.2.0
Fetching jekyll-redirect-from 0.16.0
Installing jekyll-redirect-from 0.16.0
Bundle complete! 4 Gemfile dependencies, 30 gems now installed.
Bundled gems are installed into `./.local_ruby_bundle`
```
#### Testing Jekyll (or other gem) update
First locally I reverted Jekyll to 4.1.0:
```
$ rm Gemfile.lock
$ rm -rf .local_ruby_bundle
# edited Gemfile to use version 4.1.0
$ cat Gemfile
source "https://rubygems.org"
gem "jekyll", "4.1.0"
gem "rouge", "3.26.0"
gem "jekyll-redirect-from", "0.16.0"
gem "webrick", "1.7"
$ bundle install
...
```
Testing Jekyll version before the update:
```
$ bundle exec jekyll --version
jekyll 4.1.0
```
Imitating Jekyll update coming from git by reverting my local changes:
```
$ git checkout Gemfile
Updated 1 path from the index
$ cat Gemfile
source "https://rubygems.org"
gem "jekyll", "4.2.0"
gem "rouge", "3.26.0"
gem "jekyll-redirect-from", "0.16.0"
gem "webrick", "1.7"
$ git checkout Gemfile.lock
Updated 1 path from the index
```
Run the install:
```
$ bundle install
...
```
Checking the updated Jekyll version:
```
$ bundle exec jekyll --version
jekyll 4.2.0
```
Closes #31559 from attilapiros/pin-jekyll-version.
Lead-authored-by: “attilapiros” <piros.attila.zsolt@gmail.com>
Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
2021-02-17 22:17:57 -05:00
|
|
|
# skipping local ruby bundle directory from the search
|
|
|
|
PYTHON_SOURCE="$(find . -path ./docs/.local_ruby_bundle -prune -false -o -name "*.py")"
|
2018-11-20 15:38:40 -05:00
|
|
|
|
|
|
|
compile_python_test "$PYTHON_SOURCE"
|
|
|
|
pycodestyle_test "$PYTHON_SOURCE"
|
|
|
|
flake8_test
|
2020-10-19 15:50:01 -04:00
|
|
|
mypy_test
|
2018-11-20 15:38:40 -05:00
|
|
|
sphinx_test
|
|
|
|
|
|
|
|
echo
|
|
|
|
echo "all lint-python tests passed!"
|
|
|
|
|
|
|
|
popd &> /dev/null
|