ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-35506][PYTHON][INFRA] Run tests with Python 3.9 in GitHub Actions

Browse source 
### What changes were proposed in this pull request?

This PR enables GitHub Actions to test PySpark with Python 3.9.

### Why are the changes needed?

To verify the support of Python 3.9.

### Does this PR introduce _any_ user-facing change?

No, test-only.

### How was this patch tested?

Existing tests should cover.

Closes #32657 from HyukjinKwon/SPARK-35506.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
Hyukjin Kwon 2021-05-26 09:25:51 +09:00
parent 4ba1db91f0
commit e47e615c0e
3 changed files with 23 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
.github/workflows/build_and_test.yml vendored
View file

@ -217,6 +217,16 @@ jobs:
run: |
python3.6 -m pip install numpy 'pyarrow<3.0.0' pandas scipy xmlrunner plotly>=4.8
python3.6 -m pip list
# TODO(SPARK-35507) Move Python 3.9 installtation to the docker image
- name: Install Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9
architecture: x64
- name: Install Python packages (Python 3.9)
run: |
python3.9 -m pip install numpy 'pyarrow<5.0.0' pandas scipy xmlrunner plotly>=4.8
python3.9 -m pip list
- name: Install Conda for pip packaging test
run: |
curl -s https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh

8
dev/run-tests.py
View file

@ -474,11 +474,11 @@ def run_python_tests(test_modules, parallelism, with_coverage=False):
command.append("--modules=%s" % ','.join(m.name for m in test_modules))
command.append("--parallelism=%i" % parallelism)
if "GITHUB_ACTIONS" in os.environ:
# See SPARK-33565. Python 3.8 was temporarily removed as its default Python executables
# to test because of Jenkins environment issue. Once Jenkins has Python 3.8 to test,
# we should remove this change back and add python3.8 into python/run-tests.py script.
# See SPARK-33565. Python 3.9 was temporarily removed as its default Python executables
# to test because of Jenkins environment issue. Once Jenkins has Python 3.9 to test,
# we should remove this change back and add python3.9 into python/run-tests.py script.
command.append("--python-executable=%s" % ','.join(
x for x in ["python3.6", "python3.8", "pypy3"] if which(x)))
x for x in ["python3.6", "python3.9", "pypy3"] if which(x)))
run_cmd(command)
if with_coverage:

17
python/pyspark/pandas/tests/test_stats.py
View file

@ -375,14 +375,15 @@ class StatsTest(PandasOnSparkTestCase, SQLTestUtils):
self.assert_eq(len(psdf.kurtosis(numeric_only=True)), len(pdf.kurtosis(numeric_only=True)))
self.assert_eq(len(psdf.skew(numeric_only=True)), len(pdf.skew(numeric_only=True)))
self.assert_eq(
len(psdf.quantile(q=0.5, numeric_only=True)),
len(pdf.quantile(q=0.5, numeric_only=True)),
)
self.assert_eq(
len(psdf.quantile(q=[0.25, 0.5, 0.75], numeric_only=True)),
len(pdf.quantile(q=[0.25, 0.5, 0.75], numeric_only=True)),
)
# TODO(SPARK-35510): This fails with Python 3.9. We should fix and reenable it.
# self.assert_eq(
# len(psdf.quantile(q=0.5, numeric_only=True)),
# len(pdf.quantile(q=0.5, numeric_only=True)),
# )
# self.assert_eq(
# len(psdf.quantile(q=[0.25, 0.5, 0.75], numeric_only=True)),
# len(pdf.quantile(q=[0.25, 0.5, 0.75], numeric_only=True)),
# )
def test_numeric_only_unsupported(self):
pdf = pd.DataFrame({"i": [0, 1, 2], "b": [False, False, True], "s": ["x", "y", "z"]})