spark-instrumented-optimizer/docs
Patrick Wendell 7801d44fd3 Organize configuration docs
This PR improves and organizes the config option page
and makes a few other changes to config docs. See a preview here:
http://people.apache.org/~pwendell/config-improvements/configuration.html

The biggest changes are:
1. The configs for the standalone master/workers were moved to the
standalone page and out of the general config doc.
2. SPARK_LOCAL_DIRS was missing from the standalone docs.
3. Expanded discussion of injecting configs with spark-submit, including an
example.
4. Config options were organized into the following categories:
- Runtime Environment
- Shuffle Behavior
- Spark UI
- Compression and Serialization
- Execution Behavior
- Networking
- Scheduling
- Security
- Spark Streaming

Author: Patrick Wendell <pwendell@gmail.com>

Closes #880 from pwendell/config-cleanup and squashes the following commits:

93f56c3 [Patrick Wendell] Feedback from Matei
6f66efc [Patrick Wendell] More feedback
16ae776 [Patrick Wendell] Adding back header section
d9c264f [Patrick Wendell] Small fix
e0c1728 [Patrick Wendell] Response to Matei's review
27d57db [Patrick Wendell] Reverting changes to index.html (covered in #896)
e230ef9 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup
a374369 [Patrick Wendell] Line wrapping fixes
fdff7fc [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup
3289ea4 [Patrick Wendell] Pulling in changes from #856
106ee31 [Patrick Wendell] Small link fix
f7e79bc [Patrick Wendell] Re-organizing config options.
54b184d [Patrick Wendell] Adding standalone configs to the standalone page
592e94a [Patrick Wendell] Stash
29b5446 [Patrick Wendell] Better discussion of spark-submit in configuration docs
2d719ef [Patrick Wendell] Small fix
4af9e07 [Patrick Wendell] Adding SPARK_LOCAL_DIRS docs
204b248 [Patrick Wendell] Small fixes
2014-05-28 15:49:54 -07:00
..
_layouts [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
_plugins [SPARK-1439, SPARK-1440] Generate unified Scaladoc across projects and Javadocs 2014-04-21 21:57:40 -07:00
css SPARK-1093: Annotate developer and experimental API's 2014-04-09 01:14:46 -07:00
img Merge pull request #497 from tdas/docs-update 2014-01-28 21:51:05 -08:00
js [SPARK-1439, SPARK-1440] Generate unified Scaladoc across projects and Javadocs 2014-04-21 21:57:40 -07:00
_config.yml SPARK-1818 Freshen Mesos documentation 2014-05-14 09:46:20 -07:00
api.md [SPARK-1439, SPARK-1440] Generate unified Scaladoc across projects and Javadocs 2014-04-21 21:57:40 -07:00
bagel-programming-guide.md SPARK-1727. Correct small compile errors, typos, and markdown issues in (primarly) MLlib docs 2014-05-06 20:07:22 -07:00
building-with-maven.md [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. 2014-05-12 19:44:14 -07:00
cluster-overview.md [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. 2014-05-12 19:44:14 -07:00
configuration.md Organize configuration docs 2014-05-28 15:49:54 -07:00
contributing-to-spark.md Work in progress: 2013-09-08 00:29:11 -07:00
ec2-scripts.md fix persistent-hdfs 2013-11-01 17:47:37 -07:00
graphx-programming-guide.md Unify GraphImpl RDDs + other graph load optimizations 2014-05-10 14:48:07 -07:00
hadoop-third-party-distributions.md [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. 2014-05-12 19:44:14 -07:00
hardware-provisioning.md Change port from 3030 to 4040 2013-09-11 10:01:38 -07:00
index.md [SPARK-1824] Remove <master> from Python examples 2014-05-16 22:36:23 -07:00
java-programming-guide.md [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. 2014-05-12 19:44:14 -07:00
job-scheduling.md SPARK-1183. Don't use "worker" to mean executor 2014-03-13 12:11:33 -07:00
mllib-basics.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
mllib-clustering.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
mllib-collaborative-filtering.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
mllib-decision-tree.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
mllib-dimensionality-reduction.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
mllib-guide.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
mllib-linear-methods.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
mllib-naive-bayes.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
mllib-optimization.md [WIP][SPARK-1871][MLLIB] Improve MLlib guide for v1.0 2014-05-18 17:00:57 -07:00
monitoring.md Modify a typo in monitoring.md 2014-05-12 19:21:06 -07:00
python-programming-guide.md [SPARK-1824] Remove <master> from Python examples 2014-05-16 22:36:23 -07:00
quick-start.md Organize configuration docs 2014-05-28 15:49:54 -07:00
README.md SPARK-1903 Document Spark's network connections 2014-05-25 17:15:47 -07:00
running-on-mesos.md Tweaks to Mesos docs 2014-05-16 17:35:05 -07:00
running-on-yarn.md [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. 2014-05-12 19:44:14 -07:00
scala-programming-guide.md Documentation: Encourage use of reduceByKey instead of groupByKey. 2014-05-14 22:24:04 -07:00
security.md SPARK-1557 Set permissions on event log files/directories 2014-04-29 09:19:48 -05:00
spark-debugger.md SPARK-1619 Launch spark-shell with spark-submit 2014-04-24 23:59:16 -07:00
spark-standalone.md Organize configuration docs 2014-05-28 15:49:54 -07:00
sql-programming-guide.md SPARK-1727. Correct small compile errors, typos, and markdown issues in (primarly) MLlib docs 2014-05-06 20:07:22 -07:00
streaming-custom-receivers.md [SPARK-1504], [SPARK-1505], [SPARK-1558] Updated Spark Streaming guide 2014-05-05 15:28:19 -07:00
streaming-programming-guide.md Fix doc about NetworkWordCount/JavaNetworkWordCount usage of spark streaming 2014-05-27 22:41:47 -07:00
tuning.md [SPARK-1439, SPARK-1440] Generate unified Scaladoc across projects and Javadocs 2014-04-21 21:57:40 -07:00

Welcome to the Spark documentation!

This readme will walk you through navigating and building the Spark documentation, which is included here with the Spark source code. You can also find documentation specific to release versions of Spark at http://spark.apache.org/documentation.html.

Read on to learn more about viewing documentation in plain text (i.e., markdown) or building the documentation yourself. Why build it yourself? So that you have the docs that corresponds to whichever version of Spark you currently have checked out of revision control.

Generating the Documentation HTML

We include the Spark documentation as part of the source (as opposed to using a hosted wiki, such as the github wiki, as the definitive documentation) to enable the documentation to evolve along with the source code and be captured by revision control (currently git). This way the code automatically includes the version of the documentation that is relevant regardless of which version or release you have checked out or downloaded.

In this directory you will find textfiles formatted using Markdown, with an ".md" suffix. You can read those text files directly if you want. Start with index.md.

The markdown code can be compiled to HTML using the Jekyll tool. To use the jekyll command, you will need to have Jekyll installed. The easiest way to do this is via a Ruby Gem, see the jekyll installation instructions. If not already installed, you need to install kramdown with sudo gem install kramdown. Execute jekyll from the docs/ directory. Compiling the site with Jekyll will create a directory called _site containing index.html as well as the rest of the compiled files.

You can modify the default Jekyll build as follows:

# Skip generating API docs (which takes a while)
$ SKIP_SCALADOC=1 jekyll build
# Serve content locally on port 4000
$ jekyll serve --watch
# Build the site with extra features used on the live page
$ PRODUCTION=1 jekyll build

Pygments

We also use pygments (http://pygments.org) for syntax highlighting in documentation markdown pages, so you will also need to install that (it requires Python) by running sudo easy_install Pygments.

To mark a block of code in your markdown to be syntax highlighted by jekyll during the compile phase, use the following sytax:

{% highlight scala %}
// Your scala code goes here, you can replace scala with many other
// supported languages too.
{% endhighlight %}

API Docs (Scaladoc and Epydoc)

You can build just the Spark scaladoc by running sbt/sbt doc from the SPARK_PROJECT_ROOT directory.

Similarly, you can build just the PySpark epydoc by running epydoc --config epydoc.conf from the SPARK_PROJECT_ROOT/pyspark directory. Documentation is only generated for classes that are listed as public in __init__.py.

When you run jekyll in the docs directory, it will also copy over the scaladoc for the various Spark subprojects into the docs directory (and then also into the _site directory). We use a jekyll plugin to run sbt/sbt doc before building the site so if you haven't run it (recently) it may take some time as it generates all of the scaladoc. The jekyll plugin also generates the PySpark docs using epydoc.

NOTE: To skip the step of building and copying over the Scala and Python API docs, run SKIP_API=1 jekyll.