807b4006ca
### What changes were proposed in this pull request? This PR proposes to update `dev/requirement.txt` file. NOTE that: - This file isn't used anywhere in Apache Spark CI. It's just for convenience - To minimize the overhead of maintenance, I removed all lowerbounds of dependencies, which means that using the latest versions of them should work in the clean environment (e.g., you can reinstall all of them). ### Why are the changes needed? To note the dependencies needed for Spark dev, and for easier env setting up. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Logically derived from setup.py, and other places like CI Closes #32780 from HyukjinKwon/SPARK-35648. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
35 lines
382 B
Plaintext
35 lines
382 B
Plaintext
# PySpark dependencies (required)
|
|
py4j
|
|
|
|
# PySpark dependencies (optional)
|
|
numpy
|
|
pyarrow
|
|
pandas
|
|
scipy
|
|
plotly
|
|
mlflow
|
|
matplotlib<3.3.0
|
|
|
|
# PySpark test dependencies
|
|
xmlrunner
|
|
|
|
# Linter
|
|
mypy
|
|
flake8
|
|
|
|
# Documentation (SQL)
|
|
mkdocs
|
|
|
|
# Documentation (Python)
|
|
pydata_sphinx_theme
|
|
ipython
|
|
nbsphinx
|
|
numpydoc
|
|
jinja2<3.0.0
|
|
sphinx<3.1.0
|
|
sphinx-plotly-directive
|
|
|
|
# Development scripts
|
|
jira
|
|
PyGithub
|