328eb49e62
JIRA link: https://issues.apache.org/jira/browse/SPARK-11729 Author: Xusen Yin <yinxusen@gmail.com> Closes #9713 from yinxusen/SPARK-11729.
5.5 KiB
| layout | title | displayTitle |
|---|---|---|
| global | Linear Methods - ML | <a href="ml-guide.html">ML</a> - Linear Methods |
\[ \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\av}{\mathbf{\alpha}} \newcommand{\bv}{\mathbf{b}} \newcommand{\N}{\mathbb{N}} \newcommand{\id}{\mathbf{I}} \newcommand{\ind}{\mathbf{1}} \newcommand{\0}{\mathbf{0}} \newcommand{\unit}{\mathbf{e}} \newcommand{\one}{\mathbf{1}} \newcommand{\zero}{\mathbf{0}} \]
In MLlib, we implement popular linear methods such as logistic
regression and linear least squares with L_1 or L_2 regularization.
Refer to the linear methods in mllib for
details. In spark.ml, we also include Pipelines API for Elastic
net, a hybrid
of L_1 and L_2 regularization proposed in Zou et al, Regularization
and variable selection via the elastic
net.
Mathematically, it is defined as a convex combination of the L_1 and
the L_2 regularization terms:
\[ \alpha \left( \lambda \|\wv\|_1 \right) + (1-\alpha) \left( \frac{\lambda}{2}\|\wv\|_2^2 \right) , \alpha \in [0, 1], \lambda \geq 0 \]
By setting \alpha properly, elastic net contains both L_1 and $L_2$
regularization as special cases. For example, if a linear
regression model is
trained with the elastic net parameter \alpha set to 1, it is
equivalent to a
Lasso model.
On the other hand, if \alpha is set to 0, the trained model reduces
to a ridge
regression model.
We implement Pipelines API for both linear regression and logistic
regression with elastic net regularization.
Example: Logistic Regression
The following example shows how to train a logistic regression model
with elastic net regularization. elasticNetParam corresponds to
\alpha and regParam corresponds to \lambda.
The spark.ml implementation of logistic regression also supports
extracting a summary of the model over the training set. Note that the
predictions and metrics which are stored as Dataframe in
BinaryLogisticRegressionSummary are annotated @transient and hence
only available on the driver.
LogisticRegressionTrainingSummary
provides a summary for a
LogisticRegressionModel.
Currently, only binary classification is supported and the
summary must be explicitly cast to
BinaryLogisticRegressionTrainingSummary.
This will likely change when multiclass classification is supported.
Continuing the earlier example:
{% include_example scala/org/apache/spark/examples/ml/LogisticRegressionSummaryExample.scala %}
Continuing the earlier example:
{% include_example java/org/apache/spark/examples/ml/JavaLogisticRegressionSummaryExample.java %}
Example: Linear Regression
The interface for working with linear regression models and model summaries is similar to the logistic regression case. The following example demonstrates training an elastic net regularized linear regression model and extracting model summary statistics.
Optimization
The optimization algorithm underlying the implementation is called Orthant-Wise Limited-memory QuasiNewton (OWL-QN). It is an extension of L-BFGS that can effectively handle L1 regularization and elastic net.