ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-22302][INFRA] Remove manual backports for subprocess and print explicit message for < Python 2.7

Browse source 
## What changes were proposed in this pull request?

Seems there was a mistake - missing import for `subprocess.call`, while refactoring this script a long ago, which should be used for backports of some missing functions in `subprocess`, specifically in < Python 2.7.

Reproduction is:

```
cd dev && python2.6
```

```
>>> from sparktestsupport import shellutils
>>> shellutils.subprocess_check_call("ls")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sparktestsupport/shellutils.py", line 46, in subprocess_check_call
    retcode = call(*popenargs, **kwargs)
NameError: global name 'call' is not defined
```

For Jenkins logs, please see https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3950/console

Since we dropped the Python 2.6.x support, looks better we remove those workarounds and print out explicit error messages in order to reduce the efforts to find out the root causes for such cases, for example, `https://github.com/apache/spark/pull/19513#issuecomment-337406734`.

## How was this patch tested?

Manually tested:

```
./dev/run-tests
```

```
Python versions prior to 2.7 are not supported.
```

```
./dev/run-tests-jenkins
```

```
Python versions prior to 2.7 are not supported.
```

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #19524 from HyukjinKwon/SPARK-22302.
This commit is contained in:
hyukjinkwon 2017-10-22 02:22:35 +09:00
parent a763607e4f
commit ff8de99a1c
3 changed files with 14 additions and 30 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
dev/run-tests
View file

@ -20,4 +20,10 @@
FWDIR="$(cd "`dirname $0`"/..; pwd)"
cd "$FWDIR"
PYTHON_VERSION_CHECK=$(python -c 'import sys; print(sys.version_info < (2, 7, 0))')
if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
echo "Python versions prior to 2.7 are not supported."
exit -1
fi
exec python -u ./dev/run-tests.py "$@"

7
dev/run-tests-jenkins
View file

@ -25,5 +25,10 @@
FWDIR="$( cd "$( dirname "$0" )/.." && pwd )"
cd "$FWDIR"
export PATH=/home/anaconda/bin:$PATH
PYTHON_VERSION_CHECK=$(python -c 'import sys; print(sys.version_info < (2, 7, 0))')
if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
echo "Python versions prior to 2.7 are not supported."
exit -1
fi
exec python -u ./dev/run-tests-jenkins.py "$@"

31
dev/sparktestsupport/shellutils.py
View file

@ -21,35 +21,8 @@ import shutil
import subprocess
import sys
if sys.version_info >= (2, 7):
subprocess_check_output = subprocess.check_output
subprocess_check_call = subprocess.check_call
else:
# SPARK-8763
# backported from subprocess module in Python 2.7
def subprocess_check_output(*popenargs, **kwargs):
if 'stdout' in kwargs:
raise ValueError('stdout argument not allowed, it will be overridden.')
process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
output, unused_err = process.communicate()
retcode = process.poll()
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise subprocess.CalledProcessError(retcode, cmd, output=output)
return output
# backported from subprocess module in Python 2.7
def subprocess_check_call(*popenargs, **kwargs):
retcode = call(*popenargs, **kwargs)
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
raise CalledProcessError(retcode, cmd)
return 0
subprocess_check_output = subprocess.check_output
subprocess_check_call = subprocess.check_call
def exit_from_command_with_retcode(cmd, retcode):