[MINOR][DOCS] Minor doc fixes related with doc build and uses script dir in SQL doc gen script
## What changes were proposed in this pull request? This PR proposes both: - Add information about Javadoc, SQL docs and few more information in `docs/README.md` and a comment in `docs/_plugins/copy_api_dirs.rb` related with Javadoc. - Adds some commands so that the script always runs the SQL docs build under `./sql` directory (for directly running `./sql/create-docs.sh` in the root directory). ## How was this patch tested? Manual tests with `jekyll build` and `./sql/create-docs.sh` in the root directory. Author: hyukjinkwon <gurwls223@gmail.com> Closes #19019 from HyukjinKwon/minor-doc-build.
This commit is contained in:
parent
522e1f80d6
commit
3b66b1c440
|
@ -9,19 +9,22 @@ documentation yourself. Why build it yourself? So that you have the docs that co
|
|||
whichever version of Spark you currently have checked out of revision control.
|
||||
|
||||
## Prerequisites
|
||||
The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala,
|
||||
Python and R.
|
||||
|
||||
The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java,
|
||||
Python, R and SQL.
|
||||
|
||||
You need to have [Ruby](https://www.ruby-lang.org/en/documentation/installation/) and
|
||||
[Python](https://docs.python.org/2/using/unix.html#getting-and-installing-the-latest-version-of-python)
|
||||
installed. Also install the following libraries:
|
||||
|
||||
```sh
|
||||
$ sudo gem install jekyll jekyll-redirect-from pygments.rb
|
||||
$ sudo pip install Pygments
|
||||
# Following is needed only for generating API docs
|
||||
$ sudo pip install sphinx pypandoc mkdocs
|
||||
$ sudo Rscript -e 'install.packages(c("knitr", "devtools", "roxygen2", "testthat", "rmarkdown"), repos="http://cran.stat.ucla.edu/")'
|
||||
$ sudo gem install jekyll jekyll-redirect-from pygments.rb
|
||||
$ sudo pip install Pygments
|
||||
# Following is needed only for generating API docs
|
||||
$ sudo pip install sphinx pypandoc mkdocs
|
||||
$ sudo Rscript -e 'install.packages(c("knitr", "devtools", "roxygen2", "testthat", "rmarkdown"), repos="http://cran.stat.ucla.edu/")'
|
||||
```
|
||||
|
||||
(Note: If you are on a system with both Ruby 1.9 and Ruby 2.0 you may need to replace gem with gem2.0)
|
||||
|
||||
## Generating the Documentation HTML
|
||||
|
@ -32,42 +35,49 @@ the source code and be captured by revision control (currently git). This way th
|
|||
includes the version of the documentation that is relevant regardless of which version or release
|
||||
you have checked out or downloaded.
|
||||
|
||||
In this directory you will find textfiles formatted using Markdown, with an ".md" suffix. You can
|
||||
read those text files directly if you want. Start with index.md.
|
||||
In this directory you will find text files formatted using Markdown, with an ".md" suffix. You can
|
||||
read those text files directly if you want. Start with `index.md`.
|
||||
|
||||
Execute `jekyll build` from the `docs/` directory to compile the site. Compiling the site with
|
||||
Jekyll will create a directory called `_site` containing index.html as well as the rest of the
|
||||
Jekyll will create a directory called `_site` containing `index.html` as well as the rest of the
|
||||
compiled files.
|
||||
|
||||
$ cd docs
|
||||
$ jekyll build
|
||||
|
||||
You can modify the default Jekyll build as follows:
|
||||
```sh
|
||||
# Skip generating API docs (which takes a while)
|
||||
$ SKIP_API=1 jekyll build
|
||||
|
||||
# Serve content locally on port 4000
|
||||
$ jekyll serve --watch
|
||||
|
||||
# Build the site with extra features used on the live page
|
||||
$ PRODUCTION=1 jekyll build
|
||||
$ cd docs
|
||||
$ jekyll build
|
||||
```
|
||||
|
||||
## API Docs (Scaladoc, Sphinx, roxygen2)
|
||||
You can modify the default Jekyll build as follows:
|
||||
|
||||
You can build just the Spark scaladoc by running `build/sbt unidoc` from the SPARK_PROJECT_ROOT directory.
|
||||
```sh
|
||||
# Skip generating API docs (which takes a while)
|
||||
$ SKIP_API=1 jekyll build
|
||||
|
||||
# Serve content locally on port 4000
|
||||
$ jekyll serve --watch
|
||||
|
||||
# Build the site with extra features used on the live page
|
||||
$ PRODUCTION=1 jekyll build
|
||||
```
|
||||
|
||||
## API Docs (Scaladoc, Javadoc, Sphinx, roxygen2, MkDocs)
|
||||
|
||||
You can build just the Spark scaladoc and javadoc by running `build/sbt unidoc` from the `SPARK_HOME` directory.
|
||||
|
||||
Similarly, you can build just the PySpark docs by running `make html` from the
|
||||
SPARK_PROJECT_ROOT/python/docs directory. Documentation is only generated for classes that are listed as
|
||||
public in `__init__.py`. The SparkR docs can be built by running SPARK_PROJECT_ROOT/R/create-docs.sh.
|
||||
`SPARK_HOME/python/docs` directory. Documentation is only generated for classes that are listed as
|
||||
public in `__init__.py`. The SparkR docs can be built by running `SPARK_HOME/R/create-docs.sh`, and
|
||||
the SQL docs can be built by running `SPARK_HOME/sql/create-docs.sh`
|
||||
after [building Spark](https://github.com/apache/spark#building-spark) first.
|
||||
|
||||
When you run `jekyll` in the `docs` directory, it will also copy over the scaladoc for the various
|
||||
When you run `jekyll build` in the `docs` directory, it will also copy over the scaladoc and javadoc for the various
|
||||
Spark subprojects into the `docs` directory (and then also into the `_site` directory). We use a
|
||||
jekyll plugin to run `build/sbt unidoc` before building the site so if you haven't run it (recently) it
|
||||
may take some time as it generates all of the scaladoc. The jekyll plugin also generates the
|
||||
PySpark docs using [Sphinx](http://sphinx-doc.org/).
|
||||
may take some time as it generates all of the scaladoc and javadoc using [Unidoc](https://github.com/sbt/sbt-unidoc).
|
||||
The jekyll plugin also generates the PySpark docs using [Sphinx](http://sphinx-doc.org/), SparkR docs
|
||||
using [roxygen2](https://cran.r-project.org/web/packages/roxygen2/index.html) and SQL docs
|
||||
using [MkDocs](http://www.mkdocs.org/).
|
||||
|
||||
NOTE: To skip the step of building and copying over the Scala, Python, R and SQL API docs, run `SKIP_API=1
|
||||
jekyll`. In addition, `SKIP_SCALADOC=1`, `SKIP_PYTHONDOC=1`, `SKIP_RDOC=1` and `SKIP_SQLDOC=1` can be used
|
||||
to skip a single step of the corresponding language.
|
||||
NOTE: To skip the step of building and copying over the Scala, Java, Python, R and SQL API docs, run `SKIP_API=1
|
||||
jekyll build`. In addition, `SKIP_SCALADOC=1`, `SKIP_PYTHONDOC=1`, `SKIP_RDOC=1` and `SKIP_SQLDOC=1` can be used
|
||||
to skip a single step of the corresponding language. `SKIP_SCALADOC` indicates skipping both the Scala and Java docs.
|
||||
|
|
|
@ -20,7 +20,7 @@ include FileUtils
|
|||
|
||||
if not (ENV['SKIP_API'] == '1')
|
||||
if not (ENV['SKIP_SCALADOC'] == '1')
|
||||
# Build Scaladoc for Java/Scala
|
||||
# Build Scaladoc for Scala and Javadoc for Java
|
||||
|
||||
puts "Moving to project root and building API docs."
|
||||
curr_dir = pwd
|
||||
|
|
|
@ -37,6 +37,8 @@ if ! hash mkdocs 2>/dev/null; then
|
|||
pip install mkdocs
|
||||
fi
|
||||
|
||||
pushd "$FWDIR" > /dev/null
|
||||
|
||||
# Now create the markdown file
|
||||
rm -fr docs
|
||||
mkdir docs
|
||||
|
@ -47,3 +49,5 @@ echo "Generating markdown files for SQL documentation."
|
|||
echo "Generating HTML files for SQL documentation."
|
||||
mkdocs build --clean
|
||||
rm -fr docs
|
||||
|
||||
popd
|
||||
|
|
Loading…
Reference in a new issue