ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36279][INFRA][PYTHON] Fix lint-python to work with Python 3.9

Browse source 
### What changes were proposed in this pull request?

Fix `lint-python` to pick `PYTHON_EXECUTABLE` from the environment variable first to switch the Python and explicitly specify `PYTHON_EXECUTABLE` to use `python3.9` in CI.

### Why are the changes needed?

Currently `lint-python` uses `python3`, but it's not the one we expect in CI.
As a result, `black` check is not working.

```
The python3 -m black command was not found. Skipping black checks for now.
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

The `black` check in `lint-python` should work.

Closes #33507 from ueshin/issues/SPARK-36279/lint-python.

Authored-by: Takuya UESHIN <ueshin@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 663cbdfbe5)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
Takuya UESHIN 2021-07-24 16:49:11 +09:00 committed by Hyukjin Kwon
parent 44cfce8548
commit c1434b1928
3 changed files with 4 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.github/workflows/build_and_test.yml vendored
View file

@ -386,7 +386,7 @@ jobs:
- name: Java linter
run: ./dev/lint-java
- name: Python linter
run: ./dev/lint-python
run: PYTHON_EXECUTABLE=python3.9 ./dev/lint-python
- name: R linter
run: ./dev/lint-r
- name: JS linter

2
dev/lint-python
View file

@ -24,7 +24,7 @@ MYPY_BUILD="mypy"
PYCODESTYLE_BUILD="pycodestyle"
MINIMUM_PYCODESTYLE="2.7.0"
PYTHON_EXECUTABLE="python3"
PYTHON_EXECUTABLE="${PYTHON_EXECUTABLE:-python3}"
BLACK_BUILD="$PYTHON_EXECUTABLE -m black"

8
python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
View file

@ -358,9 +358,7 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
for col in self.numeric_df_cols:
pser, psser = pdf[col], psdf[col]
if isinstance(psser.spark.data_type, DecimalType):
self.assertRaisesRegex(
TypeError, "< can not be applied to", lambda: psser < psser
)
self.assertRaisesRegex(TypeError, "< can not be applied to", lambda: psser < psser)
else:
self.assert_eq(pser < pser, psser < psser)
@ -380,9 +378,7 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
for col in self.numeric_df_cols:
pser, psser = pdf[col], psdf[col]
if isinstance(psser.spark.data_type, DecimalType):
self.assertRaisesRegex(
TypeError, "> can not be applied to", lambda: psser > psser
)
self.assertRaisesRegex(TypeError, "> can not be applied to", lambda: psser > psser)
else:
self.assert_eq(pser > pser, psser > psser)