7d4eb38bbc
### What changes were proposed in this pull request? Currently, there is no migration section for PySpark, SparkCore and Structured Streaming. It is difficult for users to know what to do when they upgrade. This PR proposes to create create a "Migration Guide" tap at Spark documentation.   This page will contain migration guides for Spark SQL, PySpark, SparkR, MLlib, Structured Streaming and Core. Basically it is a refactoring. There are some new information added, which I will leave a comment inlined for easier review. 1. **MLlib** Merge [ml-guide.html#migration-guide](https://spark.apache.org/docs/latest/ml-guide.html#migration-guide) and [ml-migration-guides.html](https://spark.apache.org/docs/latest/ml-migration-guides.html) ``` 'docs/ml-guide.md' ↓ Merge new/old migration guides 'docs/ml-migration-guide.md' ``` 2. **PySpark** Extract PySpark specific items from https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html ``` 'docs/sql-migration-guide-upgrade.md' ↓ Extract PySpark specific items 'docs/pyspark-migration-guide.md' ``` 3. **SparkR** Move [sparkr.html#migration-guide](https://spark.apache.org/docs/latest/sparkr.html#migration-guide) into a separate file, and extract from [sql-migration-guide-upgrade.html](https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html) ``` 'docs/sparkr.md' 'docs/sql-migration-guide-upgrade.md' Move migration guide section ↘ ↙ Extract SparkR specific items docs/sparkr-migration-guide.md ``` 4. **Core** Newly created at `'docs/core-migration-guide.md'`. I skimmed resolved JIRAs at 3.0.0 and found some items to note. 5. **Structured Streaming** Newly created at `'docs/ss-migration-guide.md'`. I skimmed resolved JIRAs at 3.0.0 and found some items to note. 6. **SQL** Merged [sql-migration-guide-upgrade.html](https://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html) and [sql-migration-guide-hive-compatibility.html](https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) ``` 'docs/sql-migration-guide-hive-compatibility.md' 'docs/sql-migration-guide-upgrade.md' Move Hive compatibility section ↘ ↙ Left over after filtering PySpark and SparkR items 'docs/sql-migration-guide.md' ``` ### Why are the changes needed? In order for users in production to effectively migrate to higher versions, and detect behaviour or breaking changes before upgrading and/or migrating. ### Does this PR introduce any user-facing change? Yes, this changes Spark's documentation at https://spark.apache.org/docs/latest/index.html. ### How was this patch tested? Manually build the doc. This can be verified as below: ```bash cd docs SKIP_API=1 jekyll build open _site/index.html ``` Closes #25757 from HyukjinKwon/migration-doc. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
201 lines
10 KiB
HTML
Executable file
201 lines
10 KiB
HTML
Executable file
|
|
<!DOCTYPE html>
|
|
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
|
|
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
|
|
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
|
|
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
|
|
<title>{{ page.title }} - Spark {{site.SPARK_VERSION_SHORT}} Documentation</title>
|
|
{% if page.description %}
|
|
<meta name="description" content="{{page.description | replace: 'SPARK_VERSION_SHORT', site.SPARK_VERSION_SHORT}}">
|
|
{% endif %}
|
|
|
|
{% if page.redirect %}
|
|
<meta http-equiv="refresh" content="0; url={{page.redirect}}">
|
|
<link rel="canonical" href="{{page.redirect}}" />
|
|
{% endif %}
|
|
|
|
<link rel="stylesheet" href="css/bootstrap.min.css">
|
|
<style>
|
|
body {
|
|
padding-top: 60px;
|
|
padding-bottom: 40px;
|
|
}
|
|
</style>
|
|
<meta name="viewport" content="width=device-width">
|
|
<link rel="stylesheet" href="css/bootstrap-responsive.min.css">
|
|
<link rel="stylesheet" href="css/main.css">
|
|
|
|
<script src="js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script>
|
|
|
|
<link rel="stylesheet" href="css/pygments-default.css">
|
|
|
|
{% production %}
|
|
<!-- Google analytics script -->
|
|
<script type="text/javascript">
|
|
var _gaq = _gaq || [];
|
|
_gaq.push(['_setAccount', 'UA-32518208-2']);
|
|
_gaq.push(['_trackPageview']);
|
|
|
|
(function() {
|
|
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
|
|
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
|
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
|
|
})();
|
|
</script>
|
|
{% endproduction %}
|
|
|
|
</head>
|
|
<body>
|
|
<!--[if lt IE 7]>
|
|
<p class="chromeframe">You are using an outdated browser. <a href="https://browsehappy.com/">Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true">install Google Chrome Frame</a> to better experience this site.</p>
|
|
<![endif]-->
|
|
|
|
<!-- This code is taken from http://twitter.github.com/bootstrap/examples/hero.html -->
|
|
|
|
<div class="navbar navbar-fixed-top" id="topbar">
|
|
<div class="navbar-inner">
|
|
<div class="container">
|
|
<div class="brand"><a href="index.html">
|
|
<img src="img/spark-logo-hd.png" style="height:50px;"/></a><span class="version">{{site.SPARK_VERSION_SHORT}}</span>
|
|
</div>
|
|
<ul class="nav">
|
|
<!--TODO(andyk): Add class="active" attribute to li some how.-->
|
|
<li><a href="index.html">Overview</a></li>
|
|
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Programming Guides<b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li><a href="quick-start.html">Quick Start</a></li>
|
|
<li><a href="rdd-programming-guide.html">RDDs, Accumulators, Broadcasts Vars</a></li>
|
|
<li><a href="sql-programming-guide.html">SQL, DataFrames, and Datasets</a></li>
|
|
<li><a href="structured-streaming-programming-guide.html">Structured Streaming</a></li>
|
|
<li><a href="streaming-programming-guide.html">Spark Streaming (DStreams)</a></li>
|
|
<li><a href="ml-guide.html">MLlib (Machine Learning)</a></li>
|
|
<li><a href="graphx-programming-guide.html">GraphX (Graph Processing)</a></li>
|
|
<li><a href="sparkr.html">SparkR (R on Spark)</a></li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li><a href="api/scala/index.html#org.apache.spark.package">Scala</a></li>
|
|
<li><a href="api/java/index.html">Java</a></li>
|
|
<li><a href="api/python/index.html">Python</a></li>
|
|
<li><a href="api/R/index.html">R</a></li>
|
|
<li><a href="api/sql/index.html">SQL, Built-in Functions</a></li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Deploying<b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li><a href="cluster-overview.html">Overview</a></li>
|
|
<li><a href="submitting-applications.html">Submitting Applications</a></li>
|
|
<li class="divider"></li>
|
|
<li><a href="spark-standalone.html">Spark Standalone</a></li>
|
|
<li><a href="running-on-mesos.html">Mesos</a></li>
|
|
<li><a href="running-on-yarn.html">YARN</a></li>
|
|
<li><a href="running-on-kubernetes.html">Kubernetes</a></li>
|
|
</ul>
|
|
</li>
|
|
|
|
<li class="dropdown">
|
|
<a href="api.html" class="dropdown-toggle" data-toggle="dropdown">More<b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<li><a href="configuration.html">Configuration</a></li>
|
|
<li><a href="monitoring.html">Monitoring</a></li>
|
|
<li><a href="tuning.html">Tuning Guide</a></li>
|
|
<li><a href="job-scheduling.html">Job Scheduling</a></li>
|
|
<li><a href="security.html">Security</a></li>
|
|
<li><a href="hardware-provisioning.html">Hardware Provisioning</a></li>
|
|
<li><a href="migration-guide.html">Migration Guide</a></li>
|
|
<li class="divider"></li>
|
|
<li><a href="building-spark.html">Building Spark</a></li>
|
|
<li><a href="https://spark.apache.org/contributing.html">Contributing to Spark</a></li>
|
|
<li><a href="https://spark.apache.org/third-party-projects.html">Third Party Projects</a></li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<!--<p class="navbar-text pull-right"><span class="version-text">v{{site.SPARK_VERSION_SHORT}}</span></p>-->
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="container-wrapper">
|
|
|
|
{% if page.url contains "/ml" or page.url contains "/sql" or page.url contains "migration-guide.html" %}
|
|
{% if page.url contains "migration-guide.html" %}
|
|
{% include nav-left-wrapper-migration.html nav-migration=site.data.menu-migration %}
|
|
{% elsif page.url contains "/ml" %}
|
|
{% include nav-left-wrapper-ml.html nav-mllib=site.data.menu-mllib nav-ml=site.data.menu-ml %}
|
|
{% else %}
|
|
{% include nav-left-wrapper-sql.html nav-sql=site.data.menu-sql %}
|
|
{% endif %}
|
|
<input id="nav-trigger" class="nav-trigger" checked type="checkbox">
|
|
<label for="nav-trigger"></label>
|
|
<div class="content-with-sidebar" id="content">
|
|
{% if page.displayTitle %}
|
|
<h1 class="title">{{ page.displayTitle }}</h1>
|
|
{% else %}
|
|
<h1 class="title">{{ page.title }}</h1>
|
|
{% endif %}
|
|
|
|
{{ content }}
|
|
|
|
</div>
|
|
{% else %}
|
|
<div class="content" id="content">
|
|
{% if page.displayTitle %}
|
|
<h1 class="title">{{ page.displayTitle }}</h1>
|
|
{% else %}
|
|
<h1 class="title">{{ page.title }}</h1>
|
|
{% endif %}
|
|
|
|
{{ content }}
|
|
|
|
</div>
|
|
{% endif %}
|
|
<!-- /container -->
|
|
</div>
|
|
|
|
<script src="js/vendor/jquery-3.4.1.min.js"></script>
|
|
<script src="js/vendor/bootstrap.min.js"></script>
|
|
<script src="js/vendor/anchor.min.js"></script>
|
|
<script src="js/main.js"></script>
|
|
|
|
<!-- MathJax Section -->
|
|
<script type="text/x-mathjax-config">
|
|
MathJax.Hub.Config({
|
|
TeX: { equationNumbers: { autoNumber: "AMS" } }
|
|
});
|
|
</script>
|
|
<script>
|
|
// Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS.
|
|
// We could use "//cdn.mathjax...", but that won't support "file://".
|
|
(function(d, script) {
|
|
script = d.createElement('script');
|
|
script.type = 'text/javascript';
|
|
script.async = true;
|
|
script.onload = function(){
|
|
MathJax.Hub.Config({
|
|
tex2jax: {
|
|
inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ],
|
|
displayMath: [ ["$$","$$"], ["\\[", "\\]"] ],
|
|
processEscapes: true,
|
|
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
|
|
}
|
|
});
|
|
};
|
|
script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') +
|
|
'cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js' +
|
|
'?config=TeX-AMS-MML_HTMLorMML';
|
|
d.getElementsByTagName('head')[0].appendChild(script);
|
|
}(document));
|
|
</script>
|
|
</body>
|
|
</html>
|