5ffd5d3838
## What changes were proposed in this pull request? Made DataFrame-based API primary * Spark doc menu bar and other places now link to ml-guide.html, not mllib-guide.html * mllib-guide.html keeps RDD-specific list of features, with a link at the top redirecting people to ml-guide.html * ml-guide.html includes a "maintenance mode" announcement about the RDD-based API * **Reviewers: please check this carefully** * (minor) Titles for DF API no longer include "- spark.ml" suffix. Titles for RDD API have "- RDD-based API" suffix * Moved migration guide to ml-guide from mllib-guide * Also moved past guides from mllib-migration-guides to ml-migration-guides, with a redirect link on mllib-migration-guides * **Reviewers**: I did not change any of the content of the migration guides. Reorganized DataFrame-based guide: * ml-guide.html mimics the old mllib-guide.html page in terms of content: overview, migration guide, etc. * Moved Pipeline description into ml-pipeline.html and moved tuning into ml-tuning.html * **Reviewers**: I did not change the content of these guides, except some intro text. * Sidebar remains the same, but with pipeline and tuning sections added Other: * ml-classification-regression.html: Moved text about linear methods to new section in page ## How was this patch tested? Generated docs locally Author: Joseph K. Bradley <joseph@databricks.com> Closes #14213 from jkbradley/ml-guide-2.0.
87 lines
4.6 KiB
Markdown
87 lines
4.6 KiB
Markdown
---
|
|
layout: global
|
|
title: Isotonic regression - RDD-based API
|
|
displayTitle: Regression - RDD-based API
|
|
---
|
|
|
|
## Isotonic regression
|
|
[Isotonic regression](http://en.wikipedia.org/wiki/Isotonic_regression)
|
|
belongs to the family of regression algorithms. Formally isotonic regression is a problem where
|
|
given a finite set of real numbers `$Y = {y_1, y_2, ..., y_n}$` representing observed responses
|
|
and `$X = {x_1, x_2, ..., x_n}$` the unknown response values to be fitted
|
|
finding a function that minimises
|
|
|
|
`\begin{equation}
|
|
f(x) = \sum_{i=1}^n w_i (y_i - x_i)^2
|
|
\end{equation}`
|
|
|
|
with respect to complete order subject to
|
|
`$x_1\le x_2\le ...\le x_n$` where `$w_i$` are positive weights.
|
|
The resulting function is called isotonic regression and it is unique.
|
|
It can be viewed as least squares problem under order restriction.
|
|
Essentially isotonic regression is a
|
|
[monotonic function](http://en.wikipedia.org/wiki/Monotonic_function)
|
|
best fitting the original data points.
|
|
|
|
`spark.mllib` supports a
|
|
[pool adjacent violators algorithm](http://doi.org/10.1198/TECH.2010.10111)
|
|
which uses an approach to
|
|
[parallelizing isotonic regression](http://doi.org/10.1007/978-3-642-99789-1_10).
|
|
The training input is a RDD of tuples of three double values that represent
|
|
label, feature and weight in this order. Additionally IsotonicRegression algorithm has one
|
|
optional parameter called $isotonic$ defaulting to true.
|
|
This argument specifies if the isotonic regression is
|
|
isotonic (monotonically increasing) or antitonic (monotonically decreasing).
|
|
|
|
Training returns an IsotonicRegressionModel that can be used to predict
|
|
labels for both known and unknown features. The result of isotonic regression
|
|
is treated as piecewise linear function. The rules for prediction therefore are:
|
|
|
|
* If the prediction input exactly matches a training feature
|
|
then associated prediction is returned. In case there are multiple predictions with the same
|
|
feature then one of them is returned. Which one is undefined
|
|
(same as java.util.Arrays.binarySearch).
|
|
* If the prediction input is lower or higher than all training features
|
|
then prediction with lowest or highest feature is returned respectively.
|
|
In case there are multiple predictions with the same feature
|
|
then the lowest or highest is returned respectively.
|
|
* If the prediction input falls between two training features then prediction is treated
|
|
as piecewise linear function and interpolated value is calculated from the
|
|
predictions of the two closest features. In case there are multiple values
|
|
with the same feature then the same rules as in previous point are used.
|
|
|
|
### Examples
|
|
|
|
<div class="codetabs">
|
|
<div data-lang="scala" markdown="1">
|
|
Data are read from a file where each line has a format label,feature
|
|
i.e. 4710.28,500.00. The data are split to training and testing set.
|
|
Model is created using the training set and a mean squared error is calculated from the predicted
|
|
labels and real labels in the test set.
|
|
|
|
Refer to the [`IsotonicRegression` Scala docs](api/scala/index.html#org.apache.spark.mllib.regression.IsotonicRegression) and [`IsotonicRegressionModel` Scala docs](api/scala/index.html#org.apache.spark.mllib.regression.IsotonicRegressionModel) for details on the API.
|
|
|
|
{% include_example scala/org/apache/spark/examples/mllib/IsotonicRegressionExample.scala %}
|
|
</div>
|
|
<div data-lang="java" markdown="1">
|
|
Data are read from a file where each line has a format label,feature
|
|
i.e. 4710.28,500.00. The data are split to training and testing set.
|
|
Model is created using the training set and a mean squared error is calculated from the predicted
|
|
labels and real labels in the test set.
|
|
|
|
Refer to the [`IsotonicRegression` Java docs](api/java/org/apache/spark/mllib/regression/IsotonicRegression.html) and [`IsotonicRegressionModel` Java docs](api/java/org/apache/spark/mllib/regression/IsotonicRegressionModel.html) for details on the API.
|
|
|
|
{% include_example java/org/apache/spark/examples/mllib/JavaIsotonicRegressionExample.java %}
|
|
</div>
|
|
<div data-lang="python" markdown="1">
|
|
Data are read from a file where each line has a format label,feature
|
|
i.e. 4710.28,500.00. The data are split to training and testing set.
|
|
Model is created using the training set and a mean squared error is calculated from the predicted
|
|
labels and real labels in the test set.
|
|
|
|
Refer to the [`IsotonicRegression` Python docs](api/python/pyspark.mllib.html#pyspark.mllib.regression.IsotonicRegression) and [`IsotonicRegressionModel` Python docs](api/python/pyspark.mllib.html#pyspark.mllib.regression.IsotonicRegressionModel) for more details on the API.
|
|
|
|
{% include_example python/mllib/isotonic_regression_example.py %}
|
|
</div>
|
|
</div>
|