ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/dev
History
Hyukjin Kwon 7d371d27f2 [SPARK-35393][PYTHON][INFRA][TESTS] Recover pip packaging test in Github Actions 
### What changes were proposed in this pull request?

Currently pip packaging test is being skipped:

```
========================================================================
Running PySpark packaging tests
========================================================================
Constructing virtual env for testing
Missing virtualenv & conda, skipping pip installability tests
Cleaning up temporary directory - /tmp/tmp.iILYWISPXW
```

See https://github.com/apache/spark/runs/2568923639?check_suite_focus=true

GitHub Actions's image has its default Conda installed at `/usr/share/miniconda` but seems like the image we're using for PySpark does not have it (which is legitimate).

This PR proposes to install Conda to use in pip packaging tests in GitHub Actions.

### Why are the changes needed?

To recover the test coverage.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

It was tested in my fork: https://github.com/HyukjinKwon/spark/runs/2575126882?check_suite_focus=true

```
========================================================================
Running PySpark packaging tests
========================================================================
Constructing virtual env for testing
Using conda virtual environments
Testing pip installation with python 3.6
Using /tmp/tmp.qPjTenqfGn for virtualenv
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /tmp/tmp.qPjTenqfGn/3.6

  added / updated specs:
    - numpy
    - pandas
    - pip
    - python=3.6
    - setuptools

...

Successfully ran pip sanity check
```

Closes #32537 from HyukjinKwon/SPARK-35393.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2021-05-13 10:35:56 -07:00
..
create-release [SPARK-35358][BUILD] Increase maximum Java heap used for release build to avoid OOM 2021-05-10 00:29:05 -07:00
deps [SPARK-35295][ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0 2021-05-12 08:59:36 -05:00
sparktestsupport [SPARK-35032][PYTHON] Port Koalas Index unit tests into PySpark 2021-04-16 08:53:30 +09:00
tests Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
.gitignore [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*.py to .gitignore file. 2018-01-31 00:51:00 +09:00
.rat-excludes [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
.scalafmt.conf [SPARK-35255][BUILD] Automated formatting for Scala Code for Blank Lines 2021-04-30 11:45:58 +09:00
appveyor-guide.md Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
appveyor-install-dependencies.ps1 [SPARK-33105][INFRA] Change default R arch from i386 to x64 and parametrize BINPREF 2020-10-10 13:48:26 +09:00
change-scala-version.sh [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli 2021-03-18 12:31:50 +09:00
check-license [MINOR][INFRA] Suppress warning in check-license 2020-11-23 10:38:40 +09:00
checkstyle-suppressions.xml [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+ 2019-11-03 15:13:06 -08:00
checkstyle.xml [MINOR] Fix google style guide address 2019-12-12 11:04:01 -06:00
eslint.json [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
github_jira_sync.py Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
lint-java [SPARK-23063][K8S] K8s changes for publishing scripts (and a couple of other misses) 2018-01-13 21:34:28 -08:00
lint-js [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
lint-python [SPARK-35061][BUILD] Upgrade pycodestyle from 2.6.0 to 2.7.0 2021-04-14 08:07:05 -07:00
lint-r [SPARK-29932][R][TESTS] lint-r should do non-zero exit in case of errors 2019-11-17 10:09:46 -08:00
lint-r.R [MINOR][R] small tidying of sh scripts for R 2020-04-30 16:58:05 -07:00
lint-scala [SPARK-27158][BUILD] dev/mima and dev/scalastyle support dynamic profiles 2019-03-15 08:20:42 +09:00
make-distribution.sh [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh 2020-03-11 08:22:02 -05:00
merge_spark_pr.py [MINOR] Fix usage print to guide pip3 to install jira-python library 2020-09-03 01:10:59 +09:00
mima [SPARK-33510][BUILD] Update SBT to 1.4.4 2020-11-22 22:56:59 -08:00
package-lock.json [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
package.json [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
pip-sanity-check.py [SPARK-32319][PYSPARK] Disallow the use of unused imports 2020-08-08 08:51:57 -07:00
README.md Merge pull request #565 from pwendell/dev-scripts. Closes #565. 2014-02-08 23:13:34 -08:00
requirements.txt [MINOR][INFRA] Upgrade Jira client to 2.0.0 2021-04-18 11:34:24 +09:00
run-pip-tests [SPARK-35393][PYTHON][INFRA][TESTS] Recover pip packaging test in Github Actions 2021-05-13 10:35:56 -07:00
run-tests [SPARK-29672][PYSPARK] update spark testing framework to use python3 2019-11-14 10:18:55 -08:00
run-tests-jenkins [SPARK-33535][INFRA][TESTS] Export LANG to en_US.UTF-8 in run-tests-jenkins script 2020-11-24 09:50:10 -08:00
run-tests-jenkins.py Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
run-tests.py [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources 2021-04-14 13:28:11 +09:00
sbt-checkstyle [SPARK-27158][BUILD] dev/mima and dev/scalastyle support dynamic profiles 2019-03-15 08:20:42 +09:00
scalafmt [SPARK-30570][BUILD] Update scalafmt plugin to 1.0.3 with onlyChangedFiles feature 2020-01-23 12:44:43 -08:00
scalastyle Revert "[SPARK-30534][INFRA] Use mvn in dev/scalastyle" 2020-01-21 18:23:03 +09:00
test-dependencies.sh [SPARK-20202][BUILD][SQL] Remove references to org.spark-project.hive (Hive 1.2.1) 2020-10-05 15:29:56 -07:00
tox.ini [SPARK-33749][BUILD][PYTHON] Exclude target directory in pycodestyle and flake8 2020-12-11 14:15:56 +09:00

README.md

Spark Developer Scripts

This directory contains scripts useful to developers when packaging, testing, or committing to Spark.

Many of these scripts require Apache credentials to work correctly.