2012-09-03 02:05:40 -04:00
<!DOCTYPE html>
<!-- [if lt IE 7]> <html class="no - js lt - ie9 lt - ie8 lt - ie7"> <![endif] -->
<!-- [if IE 7]> <html class="no - js lt - ie9 lt - ie8"> <![endif] -->
<!-- [if IE 8]> <html class="no - js lt - ie9"> <![endif] -->
<!-- [if gt IE 8]><! --> < html class = "no-js" > <!-- <![endif] -->
< head >
< meta charset = "utf-8" >
< meta http-equiv = "X-UA-Compatible" content = "IE=edge,chrome=1" >
2013-10-05 20:53:11 -04:00
< title > {{ page.title }} - Spark {{site.SPARK_VERSION_SHORT}} Documentation< / title >
2015-02-05 14:12:50 -05:00
{% if page.description %}
< meta name = "description" content = "{{page.description | replace: 'SPARK_VERSION_SHORT', site.SPARK_VERSION_SHORT}}" >
{% endif %}
2012-09-03 02:05:40 -04:00
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
{% if page.redirect %}
< meta http-equiv = "refresh" content = "0; url={{page.redirect}}" >
< link rel = "canonical" href = "{{page.redirect}}" / >
{% endif %}
2012-10-08 13:13:26 -04:00
< link rel = "stylesheet" href = "css/bootstrap.min.css" >
2012-09-03 02:05:40 -04:00
< style >
body {
padding-top: 60px;
padding-bottom: 40px;
}
< / style >
2012-09-13 18:35:47 -04:00
< meta name = "viewport" content = "width=device-width" >
2012-10-08 13:13:26 -04:00
< link rel = "stylesheet" href = "css/bootstrap-responsive.min.css" >
< link rel = "stylesheet" href = "css/main.css" >
2012-09-03 02:05:40 -04:00
2012-10-08 13:13:26 -04:00
< script src = "js/vendor/modernizr-2.6.1-respond-1.1.0.min.js" > < / script >
2014-01-10 03:39:08 -05:00
2012-10-08 13:13:26 -04:00
< link rel = "stylesheet" href = "css/pygments-default.css" >
2013-02-27 12:14:11 -05:00
2014-03-02 21:19:01 -05:00
{% production %}
2013-02-27 12:14:11 -05:00
<!-- Google analytics script -->
< script type = "text/javascript" >
var _gaq = _gaq || [];
2014-12-11 09:21:23 -05:00
_gaq.push(['_setAccount', 'UA-32518208-2']);
2013-02-27 12:14:11 -05:00
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
< / script >
2014-03-02 21:19:01 -05:00
{% endproduction %}
2013-02-27 12:14:11 -05:00
2012-09-03 02:05:40 -04:00
< / head >
< body >
<!-- [if lt IE 7]>
< p class = "chromeframe" > You are using an outdated browser. < a href = "http://browsehappy.com/" > Upgrade your browser today< / a > or < a href = "http://www.google.com/chromeframe/?redirect=true" > install Google Chrome Frame< / a > to better experience this site.< / p >
<![endif]-->
<!-- This code is taken from http://twitter.github.com/bootstrap/examples/hero.html -->
2012-09-26 02:26:56 -04:00
< div class = "navbar navbar-fixed-top" id = "topbar" >
2012-09-03 02:05:40 -04:00
< div class = "navbar-inner" >
< div class = "container" >
2012-10-13 22:05:11 -04:00
< div class = "brand" > < a href = "index.html" >
2013-09-08 00:41:18 -04:00
< img src = "img/spark-logo-hd.png" style = "height:50px;" / > < / a > < span class = "version" > {{site.SPARK_VERSION_SHORT}}< / span >
2012-10-13 22:05:11 -04:00
< / div >
2012-09-13 18:35:47 -04:00
< ul class = "nav" >
<!-- TODO(andyk): Add class="active" attribute to li some how. -->
2012-10-08 13:13:26 -04:00
< li > < a href = "index.html" > Overview< / a > < / li >
2012-09-16 18:28:52 -04:00
2012-09-13 18:35:47 -04:00
< li class = "dropdown" >
2012-09-25 22:31:07 -04:00
< a href = "#" class = "dropdown-toggle" data-toggle = "dropdown" > Programming Guides< b class = "caret" > < / b > < / a >
2012-09-13 18:35:47 -04:00
< ul class = "dropdown-menu" >
2012-10-08 13:13:26 -04:00
< li > < a href = "quick-start.html" > Quick Start< / a > < / li >
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
< li > < a href = "programming-guide.html" > Spark Programming Guide< / a > < / li >
2013-08-30 13:16:26 -04:00
< li class = "divider" > < / li >
2013-01-01 15:21:32 -05:00
< li > < a href = "streaming-programming-guide.html" > Spark Streaming< / a > < / li >
2015-03-09 19:16:16 -04:00
< li > < a href = "sql-programming-guide.html" > DataFrames and SQL< / a > < / li >
2013-08-31 17:21:10 -04:00
< li > < a href = "mllib-guide.html" > MLlib (Machine Learning)< / a > < / li >
2014-01-11 03:20:54 -05:00
< li > < a href = "graphx-programming-guide.html" > GraphX (Graph Processing)< / a > < / li >
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
< li > < a href = "bagel-programming-guide.html" > Bagel (Pregel on Spark)< / a > < / li >
2012-09-16 18:28:52 -04:00
< / ul >
< / li >
2014-01-10 03:39:08 -05:00
2013-01-01 15:21:32 -05:00
< li class = "dropdown" >
2013-02-25 17:23:03 -05:00
< a href = "#" class = "dropdown-toggle" data-toggle = "dropdown" > API Docs< b class = "caret" > < / b > < / a >
2013-01-01 15:21:32 -05:00
< ul class = "dropdown-menu" >
2014-12-11 09:21:23 -05:00
< li > < a href = "api/scala/index.html#org.apache.spark.package" > Scala< / a > < / li >
< li > < a href = "api/java/index.html" > Java< / a > < / li >
< li > < a href = "api/python/index.html" > Python< / a > < / li >
2013-01-01 15:21:32 -05:00
< / ul >
< / li >
2012-09-16 18:28:52 -04:00
< li class = "dropdown" >
< a href = "#" class = "dropdown-toggle" data-toggle = "dropdown" > Deploying< b class = "caret" > < / b > < / a >
< ul class = "dropdown-menu" >
2013-09-06 00:29:37 -04:00
< li > < a href = "cluster-overview.html" > Overview< / a > < / li >
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
< li > < a href = "submitting-applications.html" > Submitting Applications< / a > < / li >
< li class = "divider" > < / li >
2014-12-11 09:21:23 -05:00
< li > < a href = "spark-standalone.html" > Spark Standalone< / a > < / li >
2012-10-09 17:30:23 -04:00
< li > < a href = "running-on-mesos.html" > Mesos< / a > < / li >
< li > < a href = "running-on-yarn.html" > YARN< / a > < / li >
2014-12-11 09:21:23 -05:00
< li class = "divider" > < / li >
< li > < a href = "ec2-scripts.html" > Amazon EC2< / a > < / li >
2012-09-16 18:28:52 -04:00
< / ul >
< / li >
< li class = "dropdown" >
2012-10-08 13:13:26 -04:00
< a href = "api.html" class = "dropdown-toggle" data-toggle = "dropdown" > More< b class = "caret" > < / b > < / a >
2012-09-16 18:28:52 -04:00
< ul class = "dropdown-menu" >
< li > < a href = "configuration.html" > Configuration< / a > < / li >
2013-09-06 02:33:44 -04:00
< li > < a href = "monitoring.html" > Monitoring< / a > < / li >
2012-09-27 01:54:39 -04:00
< li > < a href = "tuning.html" > Tuning Guide< / a > < / li >
2013-09-06 00:29:37 -04:00
< li > < a href = "job-scheduling.html" > Job Scheduling< / a > < / li >
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
< li > < a href = "security.html" > Security< / a > < / li >
< li > < a href = "hardware-provisioning.html" > Hardware Provisioning< / a > < / li >
< li > < a href = "hadoop-third-party-distributions.html" > 3< sup > rd< / sup > -Party Hadoop Distros< / a > < / li >
2013-09-08 00:41:18 -04:00
< li class = "divider" > < / li >
2014-09-16 12:18:03 -04:00
< li > < a href = "building-spark.html" > Building Spark< / a > < / li >
2013-09-06 00:29:37 -04:00
< li > < a href = "https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark" > Contributing to Spark< / a > < / li >
2014-09-16 16:46:06 -04:00
< li > < a href = "https://cwiki.apache.org/confluence/display/SPARK/Supplemental+Spark+Projects" > Supplemental Projects< / a > < / li >
2012-09-13 18:35:47 -04:00
< / ul >
< / li >
< / ul >
2013-10-05 20:53:11 -04:00
<!-- <p class="navbar - text pull - right"><span class="version - text">v{{site.SPARK_VERSION_SHORT}}</span></p> -->
2012-09-03 02:05:40 -04:00
< / div >
< / div >
< / div >
2012-09-13 12:56:27 -04:00
< div class = "container" id = "content" >
2014-05-18 20:00:57 -04:00
{% if page.displayTitle %}
< h1 class = "title" > {{ page.displayTitle }}< / h1 >
{% else %}
< h1 class = "title" > {{ page.title }}< / h1 >
{% endif %}
2012-09-12 22:47:31 -04:00
2012-09-03 02:05:40 -04:00
{{ content }}
< / div > <!-- /container -->
2012-09-26 02:26:56 -04:00
< script src = "js/vendor/jquery-1.8.0.min.js" > < / script >
2012-09-03 02:05:40 -04:00
< script src = "js/vendor/bootstrap.min.js" > < / script >
< script src = "js/main.js" > < / script >
2014-01-10 03:39:08 -05:00
2014-04-22 00:57:40 -04:00
<!-- MathJax Section -->
< script type = "text/x-mathjax-config" >
2014-06-24 02:18:47 -04:00
MathJax.Hub.Config({
2014-04-22 00:57:40 -04:00
TeX: { equationNumbers: { autoNumber: "AMS" } }
2014-06-24 02:18:47 -04:00
});
< / script >
2012-09-26 02:26:56 -04:00
< script >
2014-06-24 02:18:47 -04:00
// Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS.
// We could use "//cdn.mathjax...", but that won't support "file://".
(function(d, script) {
script = d.createElement('script');
script.type = 'text/javascript';
script.async = true;
script.onload = function(){
MathJax.Hub.Config({
tex2jax: {
inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ],
2014-09-16 16:46:06 -04:00
displayMath: [ ["$$","$$"], ["\\[", "\\]"] ],
2014-06-24 02:18:47 -04:00
processEscapes: true,
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
}
});
};
script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') +
'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
d.getElementsByTagName('head')[0].appendChild(script);
}(document));
2012-09-26 02:26:56 -04:00
< / script >
2012-09-03 02:05:40 -04:00
< / body >
< / html >