b004150adb
Important updates to the streaming programming guide - Make the fault-tolerance properties easier to understand, with information about write ahead logs - Update the information about deploying the spark streaming app with information about Driver HA - Update Receiver guide to discuss reliable vs unreliable receivers. Author: Tathagata Das <tathagata.das1565@gmail.com> Author: Josh Rosen <joshrosen@databricks.com> Author: Josh Rosen <rosenville@gmail.com> Closes #3653 from tdas/streaming-doc-update-1.2 and squashes the following commits: f53154a [Tathagata Das] Addressed Josh's comments. ce299e4 [Tathagata Das] Minor update. ca19078 [Tathagata Das] Minor change f746951 [Tathagata Das] Mentioned performance problem with WAL 7787209 [Tathagata Das] Merge branch 'streaming-doc-update-1.2' of github.com:tdas/spark into streaming-doc-update-1.2 2184729 [Tathagata Das] Updated Kafka and Flume guides with reliability information. 2f3178c [Tathagata Das] Added more information about writing reliable receivers in the custom receiver guide. 91aa5aa [Tathagata Das] Improved API Docs menu 5707581 [Tathagata Das] Added Pythn API badge b9c8c24 [Tathagata Das] Merge pull request #26 from JoshRosen/streaming-programming-guide b8c8382 [Josh Rosen] minor fixes a4ef126 [Josh Rosen] Restructure parts of the fault-tolerance section to read a bit nicer when skipping over the headings 65f66cd [Josh Rosen] Fix broken link to fault-tolerance semantics section. f015397 [Josh Rosen] Minor grammar / pluralization fixes. 3019f3a [Josh Rosen] Fix minor Markdown formatting issues aa8bb87 [Tathagata Das] Small update. 195852c [Tathagata Das] Updated based on Josh's comments, updated receiver reliability and deploying section, and also updated configuration. 17b99fb [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into streaming-doc-update-1.2 a0217c0 [Tathagata Das] Changed Deploying menu layout 67fcffc [Tathagata Das] Added cluster mode + supervise example to submitting application guide. e45453b [Tathagata Das] Update streaming guide, added deploying section. 192c7a7 [Tathagata Das] Added more info about Python API, and rewrote the checkpointing section.
169 lines
8.7 KiB
HTML
Executable file
169 lines
8.7 KiB
HTML
Executable file
<!DOCTYPE html>
|
|
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
|
|
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
|
|
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
|
|
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
|
|
<title>{{ page.title }} - Spark {{site.SPARK_VERSION_SHORT}} Documentation</title>
|
|
<meta name="description" content="">
|
|
|
|
{% if page.redirect %}
|
|
<meta http-equiv="refresh" content="0; url={{page.redirect}}">
|
|
<link rel="canonical" href="{{page.redirect}}" />
|
|
{% endif %}
|
|
|
|
<link rel="stylesheet" href="css/bootstrap.min.css">
|
|
<style>
|
|
body {
|
|
padding-top: 60px;
|
|
padding-bottom: 40px;
|
|
}
|
|
</style>
|
|
<meta name="viewport" content="width=device-width">
|
|
<link rel="stylesheet" href="css/bootstrap-responsive.min.css">
|
|
<link rel="stylesheet" href="css/main.css">
|
|
|
|
<script src="js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script>
|
|
|
|
<link rel="stylesheet" href="css/pygments-default.css">
|
|
|
|
{% production %}
|
|
<!-- Google analytics script -->
|
|
<script type="text/javascript">
|
|
var _gaq = _gaq || [];
|
|
_gaq.push(['_setAccount', 'UA-32518208-2']);
|
|
_gaq.push(['_trackPageview']);
|
|
|
|
(function() {
|
|
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
|
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
|
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
|
})();
|
|
</script>
|
|
{% endproduction %}
|
|
|
|
</head>
|
|
<body>
|
|
<!--[if lt IE 7]>
|
|
<p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/">Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true">install Google Chrome Frame</a> to better experience this site.</p>
|
|
<![endif]-->
|
|
|
|
<!-- This code is taken from http://twitter.github.com/bootstrap/examples/hero.html -->
|
|
|
|
<div class="navbar navbar-fixed-top" id="topbar">
|
|
<div class="navbar-inner">
|
|
<div class="container">
|
|
<div class="brand"><a href="index.html">
|
|
<img src="img/spark-logo-hd.png" style="height:50px;"/></a><span class="version">{{site.SPARK_VERSION_SHORT}}</span>
|
|
</div>
|
|
<ul class="nav">
|
|
<!--TODO(andyk): Add class="active" attribute to li some how.-->
|
|
<li><a href="index.html">Overview</a></li>
|
|
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Programming Guides<b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li><a href="quick-start.html">Quick Start</a></li>
|
|
<li><a href="programming-guide.html">Spark Programming Guide</a></li>
|
|
<li class="divider"></li>
|
|
<li><a href="streaming-programming-guide.html">Spark Streaming</a></li>
|
|
<li><a href="sql-programming-guide.html">Spark SQL</a></li>
|
|
<li><a href="mllib-guide.html">MLlib (Machine Learning)</a></li>
|
|
<li><a href="graphx-programming-guide.html">GraphX (Graph Processing)</a></li>
|
|
<li><a href="bagel-programming-guide.html">Bagel (Pregel on Spark)</a></li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li><a href="api/scala/index.html#org.apache.spark.package">Scala</a></li>
|
|
<li><a href="api/java/index.html">Java</a></li>
|
|
<li><a href="api/python/index.html">Python</a></li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Deploying<b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li><a href="cluster-overview.html">Overview</a></li>
|
|
<li><a href="submitting-applications.html">Submitting Applications</a></li>
|
|
<li class="divider"></li>
|
|
<li><a href="spark-standalone.html">Spark Standalone</a></li>
|
|
<li><a href="running-on-mesos.html">Mesos</a></li>
|
|
<li><a href="running-on-yarn.html">YARN</a></li>
|
|
<li class="divider"></li>
|
|
<li><a href="ec2-scripts.html">Amazon EC2</a></li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li class="dropdown">
|
|
<a href="api.html" class="dropdown-toggle" data-toggle="dropdown">More<b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li><a href="configuration.html">Configuration</a></li>
|
|
<li><a href="monitoring.html">Monitoring</a></li>
|
|
<li><a href="tuning.html">Tuning Guide</a></li>
|
|
<li><a href="job-scheduling.html">Job Scheduling</a></li>
|
|
<li><a href="security.html">Security</a></li>
|
|
<li><a href="hardware-provisioning.html">Hardware Provisioning</a></li>
|
|
<li><a href="hadoop-third-party-distributions.html">3<sup>rd</sup>-Party Hadoop Distros</a></li>
|
|
<li class="divider"></li>
|
|
<li><a href="building-spark.html">Building Spark</a></li>
|
|
<li><a href="https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark">Contributing to Spark</a></li>
|
|
<li><a href="https://cwiki.apache.org/confluence/display/SPARK/Supplemental+Spark+Projects">Supplemental Projects</a></li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<!--<p class="navbar-text pull-right"><span class="version-text">v{{site.SPARK_VERSION_SHORT}}</span></p>-->
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="container" id="content">
|
|
{% if page.displayTitle %}
|
|
<h1 class="title">{{ page.displayTitle }}</h1>
|
|
{% else %}
|
|
<h1 class="title">{{ page.title }}</h1>
|
|
{% endif %}
|
|
|
|
{{ content }}
|
|
|
|
</div> <!-- /container -->
|
|
|
|
<script src="js/vendor/jquery-1.8.0.min.js"></script>
|
|
<script src="js/vendor/bootstrap.min.js"></script>
|
|
<script src="js/main.js"></script>
|
|
|
|
<!-- MathJax Section -->
|
|
<script type="text/x-mathjax-config">
|
|
MathJax.Hub.Config({
|
|
TeX: { equationNumbers: { autoNumber: "AMS" } }
|
|
});
|
|
</script>
|
|
<script>
|
|
// Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS.
|
|
// We could use "//cdn.mathjax...", but that won't support "file://".
|
|
(function(d, script) {
|
|
script = d.createElement('script');
|
|
script.type = 'text/javascript';
|
|
script.async = true;
|
|
script.onload = function(){
|
|
MathJax.Hub.Config({
|
|
tex2jax: {
|
|
inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ],
|
|
displayMath: [ ["$$","$$"], ["\\[", "\\]"] ],
|
|
processEscapes: true,
|
|
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
|
|
}
|
|
});
|
|
};
|
|
script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') +
|
|
'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
|
|
d.getElementsByTagName('head')[0].appendChild(script);
|
|
}(document));
|
|
</script>
|
|
</body>
|
|
</html>
|