[SPARK-3399][PySpark] Test for PySpark should ignore HADOOP_CONF_DIR and YARN_CONF_DIR
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #2270 from sarutak/SPARK-3399 and squashes the following commits: 7613be6 [Kousuke Saruta] Modified pyspark script to ignore environment variables YARN_CONF_DIR and HADOOP_CONF_DIR while testing
This commit is contained in:
parent
62c5576099
commit
7ff8c45d71
|
@ -85,6 +85,8 @@ export PYSPARK_SUBMIT_ARGS
|
|||
|
||||
# For pyspark tests
|
||||
if [[ -n "$SPARK_TESTING" ]]; then
|
||||
unset YARN_CONF_DIR
|
||||
unset HADOOP_CONF_DIR
|
||||
if [[ -n "$PYSPARK_DOC_TEST" ]]; then
|
||||
exec "$PYSPARK_PYTHON" -m doctest $1
|
||||
else
|
||||
|
|
Loading…
Reference in a new issue