5ffd5d3838
## What changes were proposed in this pull request? Made DataFrame-based API primary * Spark doc menu bar and other places now link to ml-guide.html, not mllib-guide.html * mllib-guide.html keeps RDD-specific list of features, with a link at the top redirecting people to ml-guide.html * ml-guide.html includes a "maintenance mode" announcement about the RDD-based API * **Reviewers: please check this carefully** * (minor) Titles for DF API no longer include "- spark.ml" suffix. Titles for RDD API have "- RDD-based API" suffix * Moved migration guide to ml-guide from mllib-guide * Also moved past guides from mllib-migration-guides to ml-migration-guides, with a redirect link on mllib-migration-guides * **Reviewers**: I did not change any of the content of the migration guides. Reorganized DataFrame-based guide: * ml-guide.html mimics the old mllib-guide.html page in terms of content: overview, migration guide, etc. * Moved Pipeline description into ml-pipeline.html and moved tuning into ml-tuning.html * **Reviewers**: I did not change the content of these guides, except some intro text. * Sidebar remains the same, but with pipeline and tuning sections added Other: * ml-classification-regression.html: Moved text about linear methods to new section in page ## How was this patch tested? Generated docs locally Author: Joseph K. Bradley <joseph@databricks.com> Closes #14213 from jkbradley/ml-guide-2.0.
4.6 KiB
| layout | title | displayTitle |
|---|---|---|
| global | Isotonic regression - RDD-based API | Regression - RDD-based API |
Isotonic regression
Isotonic regression
belongs to the family of regression algorithms. Formally isotonic regression is a problem where
given a finite set of real numbers $Y = {y_1, y_2, ..., y_n}$ representing observed responses
and $X = {x_1, x_2, ..., x_n}$ the unknown response values to be fitted
finding a function that minimises
\begin{equation} f(x) = \sum_{i=1}^n w_i (y_i - x_i)^2 \end{equation}
with respect to complete order subject to
$x_1\le x_2\le ...\le x_n$ where $w_i$ are positive weights.
The resulting function is called isotonic regression and it is unique.
It can be viewed as least squares problem under order restriction.
Essentially isotonic regression is a
monotonic function
best fitting the original data points.
spark.mllib supports a
pool adjacent violators algorithm
which uses an approach to
parallelizing isotonic regression.
The training input is a RDD of tuples of three double values that represent
label, feature and weight in this order. Additionally IsotonicRegression algorithm has one
optional parameter called isotonic defaulting to true.
This argument specifies if the isotonic regression is
isotonic (monotonically increasing) or antitonic (monotonically decreasing).
Training returns an IsotonicRegressionModel that can be used to predict labels for both known and unknown features. The result of isotonic regression is treated as piecewise linear function. The rules for prediction therefore are:
- If the prediction input exactly matches a training feature then associated prediction is returned. In case there are multiple predictions with the same feature then one of them is returned. Which one is undefined (same as java.util.Arrays.binarySearch).
- If the prediction input is lower or higher than all training features then prediction with lowest or highest feature is returned respectively. In case there are multiple predictions with the same feature then the lowest or highest is returned respectively.
- If the prediction input falls between two training features then prediction is treated as piecewise linear function and interpolated value is calculated from the predictions of the two closest features. In case there are multiple values with the same feature then the same rules as in previous point are used.
Examples
Refer to the IsotonicRegression Scala docs and IsotonicRegressionModel Scala docs for details on the API.
{% include_example scala/org/apache/spark/examples/mllib/IsotonicRegressionExample.scala %}
Refer to the IsotonicRegression Java docs and IsotonicRegressionModel Java docs for details on the API.
{% include_example java/org/apache/spark/examples/mllib/JavaIsotonicRegressionExample.java %}
Refer to the IsotonicRegression Python docs and IsotonicRegressionModel Python docs for more details on the API.
{% include_example python/mllib/isotonic_regression_example.py %}