2014-02-08 14:39:13 -05:00
|
|
|
---
|
|
|
|
layout: global
|
2014-05-18 20:00:57 -04:00
|
|
|
title: Optimization - MLlib
|
|
|
|
displayTitle: <a href="mllib-guide.html">MLlib</a> - Optimization
|
2014-02-08 14:39:13 -05:00
|
|
|
---
|
|
|
|
|
|
|
|
* Table of contents
|
|
|
|
{:toc}
|
|
|
|
|
2014-02-09 18:19:50 -05:00
|
|
|
`\[
|
|
|
|
\newcommand{\R}{\mathbb{R}}
|
|
|
|
\newcommand{\E}{\mathbb{E}}
|
|
|
|
\newcommand{\x}{\mathbf{x}}
|
|
|
|
\newcommand{\y}{\mathbf{y}}
|
|
|
|
\newcommand{\wv}{\mathbf{w}}
|
|
|
|
\newcommand{\av}{\mathbf{\alpha}}
|
|
|
|
\newcommand{\bv}{\mathbf{b}}
|
|
|
|
\newcommand{\N}{\mathbb{N}}
|
|
|
|
\newcommand{\id}{\mathbf{I}}
|
|
|
|
\newcommand{\ind}{\mathbf{1}}
|
|
|
|
\newcommand{\0}{\mathbf{0}}
|
|
|
|
\newcommand{\unit}{\mathbf{e}}
|
|
|
|
\newcommand{\one}{\mathbf{1}}
|
|
|
|
\newcommand{\zero}{\mathbf{0}}
|
|
|
|
\]`
|
2014-02-08 14:39:13 -05:00
|
|
|
|
|
|
|
|
|
|
|
|
2014-04-22 14:20:47 -04:00
|
|
|
## Mathematical description
|
|
|
|
|
|
|
|
### Gradient descent
|
2014-02-09 18:19:50 -05:00
|
|
|
The simplest method to solve optimization problems of the form `$\min_{\wv \in\R^d} \; f(\wv)$`
|
|
|
|
is [gradient descent](http://en.wikipedia.org/wiki/Gradient_descent).
|
|
|
|
Such first-order optimization methods (including gradient descent and stochastic variants
|
|
|
|
thereof) are well-suited for large-scale and distributed computation.
|
|
|
|
|
|
|
|
Gradient descent methods aim to find a local minimum of a function by iteratively taking steps in
|
|
|
|
the direction of steepest descent, which is the negative of the derivative (called the
|
|
|
|
[gradient](http://en.wikipedia.org/wiki/Gradient)) of the function at the current point, i.e., at
|
|
|
|
the current parameter value.
|
|
|
|
If the objective function `$f$` is not differentiable at all arguments, but still convex, then a
|
2014-04-22 14:20:47 -04:00
|
|
|
*sub-gradient*
|
2014-02-09 18:19:50 -05:00
|
|
|
is the natural generalization of the gradient, and assumes the role of the step direction.
|
2014-04-22 14:20:47 -04:00
|
|
|
In any case, computing a gradient or sub-gradient of `$f$` is expensive --- it requires a full
|
2014-02-09 18:19:50 -05:00
|
|
|
pass through the complete dataset, in order to compute the contributions from all loss terms.
|
|
|
|
|
2014-04-22 14:20:47 -04:00
|
|
|
### Stochastic gradient descent (SGD)
|
2014-02-09 18:19:50 -05:00
|
|
|
Optimization problems whose objective function `$f$` is written as a sum are particularly
|
2014-04-22 14:20:47 -04:00
|
|
|
suitable to be solved using *stochastic gradient descent (SGD)*.
|
2014-02-09 18:19:50 -05:00
|
|
|
In our case, for the optimization formulations commonly used in <a
|
|
|
|
href="mllib-classification-regression.html">supervised machine learning</a>,
|
|
|
|
`\begin{equation}
|
|
|
|
f(\wv) :=
|
|
|
|
\lambda\, R(\wv) +
|
|
|
|
\frac1n \sum_{i=1}^n L(\wv;\x_i,y_i)
|
|
|
|
\label{eq:regPrimal}
|
|
|
|
\ .
|
|
|
|
\end{equation}`
|
|
|
|
this is especially natural, because the loss is written as an average of the individual losses
|
|
|
|
coming from each datapoint.
|
|
|
|
|
|
|
|
A stochastic subgradient is a randomized choice of a vector, such that in expectation, we obtain
|
|
|
|
a true subgradient of the original objective function.
|
|
|
|
Picking one datapoint `$i\in[1..n]$` uniformly at random, we obtain a stochastic subgradient of
|
|
|
|
`$\eqref{eq:regPrimal}$`, with respect to `$\wv$` as follows:
|
|
|
|
`\[
|
|
|
|
f'_{\wv,i} := L'_{\wv,i} + \lambda\, R'_\wv \ ,
|
|
|
|
\]`
|
|
|
|
where `$L'_{\wv,i} \in \R^d$` is a subgradient of the part of the loss function determined by the
|
|
|
|
`$i$`-th datapoint, that is `$L'_{\wv,i} \in \frac{\partial}{\partial \wv} L(\wv;\x_i,y_i)$`.
|
|
|
|
Furthermore, `$R'_\wv$` is a subgradient of the regularizer `$R(\wv)$`, i.e. `$R'_\wv \in
|
|
|
|
\frac{\partial}{\partial \wv} R(\wv)$`. The term `$R'_\wv$` does not depend on which random
|
|
|
|
datapoint is picked.
|
|
|
|
Clearly, in expectation over the random choice of `$i\in[1..n]$`, we have that `$f'_{\wv,i}$` is
|
|
|
|
a subgradient of the original objective `$f$`, meaning that `$\E\left[f'_{\wv,i}\right] \in
|
|
|
|
\frac{\partial}{\partial \wv} f(\wv)$`.
|
|
|
|
|
|
|
|
Running SGD now simply becomes walking in the direction of the negative stochastic subgradient
|
|
|
|
`$f'_{\wv,i}$`, that is
|
|
|
|
`\begin{equation}\label{eq:SGDupdate}
|
|
|
|
\wv^{(t+1)} := \wv^{(t)} - \gamma \; f'_{\wv,i} \ .
|
|
|
|
\end{equation}`
|
|
|
|
**Step-size.**
|
|
|
|
The parameter `$\gamma$` is the step-size, which in the default implementation is chosen
|
|
|
|
decreasing with the square root of the iteration counter, i.e. `$\gamma := \frac{s}{\sqrt{t}}$`
|
|
|
|
in the `$t$`-th iteration, with the input parameter `$s=$ stepSize`. Note that selecting the best
|
|
|
|
step-size for SGD methods can often be delicate in practice and is a topic of active research.
|
|
|
|
|
|
|
|
**Gradients.**
|
|
|
|
A table of (sub)gradients of the machine learning methods implemented in MLlib, is available in
|
|
|
|
the <a href="mllib-classification-regression.html">classification and regression</a> section.
|
|
|
|
|
|
|
|
|
|
|
|
**Proximal Updates.**
|
|
|
|
As an alternative to just use the subgradient `$R'(\wv)$` of the regularizer in the step
|
|
|
|
direction, an improved update for some cases can be obtained by using the proximal operator
|
|
|
|
instead.
|
|
|
|
For the L1-regularizer, the proximal operator is given by soft thresholding, as implemented in
|
2014-04-22 00:57:40 -04:00
|
|
|
[L1Updater](api/scala/index.html#org.apache.spark.mllib.optimization.L1Updater).
|
2014-02-09 18:19:50 -05:00
|
|
|
|
|
|
|
|
2014-04-22 14:20:47 -04:00
|
|
|
### Update schemes for distributed SGD
|
2014-02-09 18:19:50 -05:00
|
|
|
The SGD implementation in
|
2014-04-22 00:57:40 -04:00
|
|
|
[GradientDescent](api/scala/index.html#org.apache.spark.mllib.optimization.GradientDescent) uses
|
2014-02-09 18:19:50 -05:00
|
|
|
a simple (distributed) sampling of the data examples.
|
|
|
|
We recall that the loss part of the optimization problem `$\eqref{eq:regPrimal}$` is
|
|
|
|
`$\frac1n \sum_{i=1}^n L(\wv;\x_i,y_i)$`, and therefore `$\frac1n \sum_{i=1}^n L'_{\wv,i}$` would
|
|
|
|
be the true (sub)gradient.
|
|
|
|
Since this would require access to the full data set, the parameter `miniBatchFraction` specifies
|
|
|
|
which fraction of the full data to use instead.
|
|
|
|
The average of the gradients over this subset, i.e.
|
|
|
|
`\[
|
|
|
|
\frac1{|S|} \sum_{i\in S} L'_{\wv,i} \ ,
|
|
|
|
\]`
|
|
|
|
is a stochastic gradient. Here `$S$` is the sampled subset of size `$|S|=$ miniBatchFraction
|
|
|
|
$\cdot n$`.
|
|
|
|
|
|
|
|
In each iteration, the sampling over the distributed dataset
|
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <matei@databricks.com>
Closes #896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
2014-05-30 03:34:33 -04:00
|
|
|
([RDD](programming-guide.html#resilient-distributed-datasets-rdds)), as well as the
|
2014-02-09 18:19:50 -05:00
|
|
|
computation of the sum of the partial results from each worker machine is performed by the
|
|
|
|
standard spark routines.
|
|
|
|
|
|
|
|
If the fraction of points `miniBatchFraction` is set to 1 (default), then the resulting step in
|
|
|
|
each iteration is exact (sub)gradient descent. In this case there is no randomness and no
|
|
|
|
variance in the used step directions.
|
|
|
|
On the other extreme, if `miniBatchFraction` is chosen very small, such that only a single point
|
|
|
|
is sampled, i.e. `$|S|=$ miniBatchFraction $\cdot n = 1$`, then the algorithm is equivalent to
|
|
|
|
standard SGD. In that case, the step direction depends from the uniformly random sampling of the
|
|
|
|
point.
|
|
|
|
|
2014-05-12 22:20:24 -04:00
|
|
|
### Limited-memory BFGS (L-BFGS)
|
|
|
|
[L-BFGS](http://en.wikipedia.org/wiki/Limited-memory_BFGS) is an optimization
|
|
|
|
algorithm in the family of quasi-Newton methods to solve the optimization problems of the form
|
|
|
|
`$\min_{\wv \in\R^d} \; f(\wv)$`. The L-BFGS method approximates the objective function locally as a
|
|
|
|
quadratic without evaluating the second partial derivatives of the objective function to construct the
|
|
|
|
Hessian matrix. The Hessian matrix is approximated by previous gradient evaluations, so there is no
|
|
|
|
vertical scalability issue (the number of training features) when computing the Hessian matrix
|
|
|
|
explicitly in Newton's method. As a result, L-BFGS often achieves rapider convergence compared with
|
|
|
|
other first-order optimization.
|
2014-02-09 18:19:50 -05:00
|
|
|
|
2014-04-22 14:20:47 -04:00
|
|
|
## Implementation in MLlib
|
2014-02-09 18:19:50 -05:00
|
|
|
|
2014-05-12 22:20:24 -04:00
|
|
|
### Gradient descent and stochastic gradient descent
|
2014-02-09 18:19:50 -05:00
|
|
|
Gradient descent methods including stochastic subgradient descent (SGD) as
|
|
|
|
included as a low-level primitive in `MLlib`, upon which various ML algorithms
|
|
|
|
are developed, see the
|
2014-04-22 14:20:47 -04:00
|
|
|
<a href="mllib-linear-methods.html">linear methods</a>
|
2014-02-09 18:19:50 -05:00
|
|
|
section for example.
|
|
|
|
|
2014-06-17 02:10:36 -04:00
|
|
|
The SGD class
|
|
|
|
[GradientDescent](api/scala/index.html#org.apache.spark.mllib.optimization.GradientDescent)
|
|
|
|
sets the following parameters:
|
2014-02-09 18:19:50 -05:00
|
|
|
|
2014-05-12 22:20:24 -04:00
|
|
|
* `Gradient` is a class that computes the stochastic gradient of the function
|
2014-02-08 14:39:13 -05:00
|
|
|
being optimized, i.e., with respect to a single training example, at the
|
|
|
|
current parameter value. MLlib includes gradient classes for common loss
|
|
|
|
functions, e.g., hinge, logistic, least-squares. The gradient class takes as
|
|
|
|
input a training example, its label, and the current parameter value.
|
2014-05-12 22:20:24 -04:00
|
|
|
* `Updater` is a class that performs the actual gradient descent step, i.e.
|
2014-02-09 18:19:50 -05:00
|
|
|
updating the weights in each iteration, for a given gradient of the loss part.
|
|
|
|
The updater is also responsible to perform the update from the regularization
|
|
|
|
part. MLlib includes updaters for cases without regularization, as well as
|
2014-02-08 14:39:13 -05:00
|
|
|
L1 and L2 regularizers.
|
2014-02-09 18:19:50 -05:00
|
|
|
* `stepSize` is a scalar value denoting the initial step size for gradient
|
2014-02-08 14:39:13 -05:00
|
|
|
descent. All updaters in MLlib use a step size at the t-th step equal to
|
2014-02-09 18:19:50 -05:00
|
|
|
`stepSize $/ \sqrt{t}$`.
|
|
|
|
* `numIterations` is the number of iterations to run.
|
|
|
|
* `regParam` is the regularization parameter when using L1 or L2 regularization.
|
|
|
|
* `miniBatchFraction` is the fraction of the total data that is sampled in
|
|
|
|
each iteration, to compute the gradient direction.
|
2014-02-08 14:39:13 -05:00
|
|
|
|
|
|
|
Available algorithms for gradient descent:
|
|
|
|
|
2014-06-17 02:10:36 -04:00
|
|
|
* [GradientDescent](api/scala/index.html#org.apache.spark.mllib.optimization.GradientDescent)
|
2014-05-12 22:20:24 -04:00
|
|
|
|
|
|
|
### L-BFGS
|
|
|
|
L-BFGS is currently only a low-level optimization primitive in `MLlib`. If you want to use L-BFGS in various
|
|
|
|
ML algorithms such as Linear Regression, and Logistic Regression, you have to pass the gradient of objective
|
|
|
|
function, and updater into optimizer yourself instead of using the training APIs like
|
2014-05-18 20:00:57 -04:00
|
|
|
[LogisticRegressionWithSGD](api/scala/index.html#org.apache.spark.mllib.classification.LogisticRegressionWithSGD).
|
2014-05-12 22:20:24 -04:00
|
|
|
See the example below. It will be addressed in the next release.
|
|
|
|
|
|
|
|
The L1 regularization by using
|
2014-05-18 20:00:57 -04:00
|
|
|
[L1Updater](api/scala/index.html#org.apache.spark.mllib.optimization.L1Updater) will not work since the
|
2014-05-12 22:20:24 -04:00
|
|
|
soft-thresholding logic in L1Updater is designed for gradient descent. See the developer's note.
|
|
|
|
|
|
|
|
The L-BFGS method
|
|
|
|
[LBFGS.runLBFGS](api/scala/index.html#org.apache.spark.mllib.optimization.LBFGS)
|
|
|
|
has the following parameters:
|
|
|
|
|
|
|
|
* `Gradient` is a class that computes the gradient of the objective function
|
|
|
|
being optimized, i.e., with respect to a single training example, at the
|
|
|
|
current parameter value. MLlib includes gradient classes for common loss
|
|
|
|
functions, e.g., hinge, logistic, least-squares. The gradient class takes as
|
|
|
|
input a training example, its label, and the current parameter value.
|
|
|
|
* `Updater` is a class that computes the gradient and loss of objective function
|
|
|
|
of the regularization part for L-BFGS. MLlib includes updaters for cases without
|
|
|
|
regularization, as well as L2 regularizer.
|
|
|
|
* `numCorrections` is the number of corrections used in the L-BFGS update. 10 is
|
|
|
|
recommended.
|
|
|
|
* `maxNumIterations` is the maximal number of iterations that L-BFGS can be run.
|
|
|
|
* `regParam` is the regularization parameter when using regularization.
|
|
|
|
|
|
|
|
The `return` is a tuple containing two elements. The first element is a column matrix
|
|
|
|
containing weights for every feature, and the second element is an array containing
|
|
|
|
the loss computed for every iteration.
|
|
|
|
|
|
|
|
Here is an example to train binary logistic regression with L2 regularization using
|
|
|
|
L-BFGS optimizer.
|
2014-07-20 23:48:44 -04:00
|
|
|
|
|
|
|
<div class="codetabs">
|
|
|
|
|
|
|
|
<div data-lang="scala" markdown="1">
|
2014-05-12 22:20:24 -04:00
|
|
|
{% highlight scala %}
|
|
|
|
import org.apache.spark.SparkContext
|
|
|
|
import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
|
|
|
|
import org.apache.spark.mllib.linalg.Vectors
|
|
|
|
import org.apache.spark.mllib.util.MLUtils
|
|
|
|
import org.apache.spark.mllib.classification.LogisticRegressionModel
|
[SPARK-1484][MLLIB] Warn when running an iterative algorithm on uncached data.
Add warnings to KMeans, GeneralizedLinearAlgorithm, and computeSVD when called with input data that is not cached. KMeans is implemented iteratively, and I believe that GeneralizedLinearAlgorithm’s current optimizers are iterative and its future optimizers are also likely to be iterative. RowMatrix’s computeSVD is iterative against an RDD when run in DistARPACK mode. ALS and DecisionTree are iterative as well, but they implement RDD caching internally so do not require a warning.
I added a warning to GeneralizedLinearAlgorithm rather than inside its optimizers, where the iteration actually occurs, because internally GeneralizedLinearAlgorithm maps its input data to an uncached RDD before passing it to an optimizer. (In other words, the warning would be printed for every GeneralizedLinearAlgorithm run, regardless of whether its input is cached, if the warning were in GradientDescent or other optimizer.) I assume that use of an uncached RDD by GeneralizedLinearAlgorithm is intentional, and that the mapping there (adding label, intercepts and scaling) is a lightweight operation. Arguably a user calling an optimizer such as GradientDescent will be knowledgable enough to cache their data without needing a log warning, so lack of a warning in the optimizers may be ok.
Some of the documentation examples making use of these iterative algorithms did not cache their training RDDs (while others did). I updated the examples to always cache. I also fixed some (unrelated) minor errors in the documentation examples.
Author: Aaron Staple <aaron.staple@gmail.com>
Closes #2347 from staple/SPARK-1484 and squashes the following commits:
bd49701 [Aaron Staple] Address review comments.
ab2d4a4 [Aaron Staple] Disable warnings on python code path.
a7a0f99 [Aaron Staple] Change code comments per review comments.
7cca1dc [Aaron Staple] Change warning message text.
c77e939 [Aaron Staple] [SPARK-1484][MLLIB] Warn when running an iterative algorithm on uncached data.
3b6c511 [Aaron Staple] Minor doc example fixes.
2014-09-25 19:11:00 -04:00
|
|
|
import org.apache.spark.mllib.optimization.{LBFGS, LogisticGradient, SquaredL2Updater}
|
2014-05-12 22:20:24 -04:00
|
|
|
|
2014-07-13 22:27:43 -04:00
|
|
|
val data = MLUtils.loadLibSVMFile(sc, "data/mllib/sample_libsvm_data.txt")
|
2014-05-12 22:20:24 -04:00
|
|
|
val numFeatures = data.take(1)(0).features.size
|
|
|
|
|
|
|
|
// Split data into training (60%) and test (40%).
|
|
|
|
val splits = data.randomSplit(Array(0.6, 0.4), seed = 11L)
|
|
|
|
|
|
|
|
// Append 1 into the training data as intercept.
|
|
|
|
val training = splits(0).map(x => (x.label, MLUtils.appendBias(x.features))).cache()
|
|
|
|
|
|
|
|
val test = splits(1)
|
|
|
|
|
|
|
|
// Run training algorithm to build the model
|
|
|
|
val numCorrections = 10
|
|
|
|
val convergenceTol = 1e-4
|
|
|
|
val maxNumIterations = 20
|
|
|
|
val regParam = 0.1
|
|
|
|
val initialWeightsWithIntercept = Vectors.dense(new Array[Double](numFeatures + 1))
|
|
|
|
|
|
|
|
val (weightsWithIntercept, loss) = LBFGS.runLBFGS(
|
|
|
|
training,
|
|
|
|
new LogisticGradient(),
|
|
|
|
new SquaredL2Updater(),
|
|
|
|
numCorrections,
|
|
|
|
convergenceTol,
|
|
|
|
maxNumIterations,
|
|
|
|
regParam,
|
|
|
|
initialWeightsWithIntercept)
|
|
|
|
|
|
|
|
val model = new LogisticRegressionModel(
|
|
|
|
Vectors.dense(weightsWithIntercept.toArray.slice(0, weightsWithIntercept.size - 1)),
|
|
|
|
weightsWithIntercept(weightsWithIntercept.size - 1))
|
|
|
|
|
|
|
|
// Clear the default threshold.
|
|
|
|
model.clearThreshold()
|
|
|
|
|
|
|
|
// Compute raw scores on the test set.
|
|
|
|
val scoreAndLabels = test.map { point =>
|
|
|
|
val score = model.predict(point.features)
|
|
|
|
(score, point.label)
|
|
|
|
}
|
|
|
|
|
|
|
|
// Get evaluation metrics.
|
|
|
|
val metrics = new BinaryClassificationMetrics(scoreAndLabels)
|
|
|
|
val auROC = metrics.areaUnderROC()
|
|
|
|
|
|
|
|
println("Loss of each step in training process")
|
|
|
|
loss.foreach(println)
|
|
|
|
println("Area under ROC = " + auROC)
|
|
|
|
{% endhighlight %}
|
2014-07-20 23:48:44 -04:00
|
|
|
</div>
|
|
|
|
|
|
|
|
<div data-lang="java" markdown="1">
|
|
|
|
{% highlight java %}
|
|
|
|
import java.util.Arrays;
|
|
|
|
import java.util.Random;
|
|
|
|
|
|
|
|
import scala.Tuple2;
|
|
|
|
|
|
|
|
import org.apache.spark.api.java.*;
|
|
|
|
import org.apache.spark.api.java.function.Function;
|
|
|
|
import org.apache.spark.mllib.classification.LogisticRegressionModel;
|
|
|
|
import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics;
|
|
|
|
import org.apache.spark.mllib.linalg.Vector;
|
|
|
|
import org.apache.spark.mllib.linalg.Vectors;
|
|
|
|
import org.apache.spark.mllib.optimization.*;
|
|
|
|
import org.apache.spark.mllib.regression.LabeledPoint;
|
|
|
|
import org.apache.spark.mllib.util.MLUtils;
|
|
|
|
import org.apache.spark.SparkConf;
|
|
|
|
import org.apache.spark.SparkContext;
|
|
|
|
|
|
|
|
public class LBFGSExample {
|
|
|
|
public static void main(String[] args) {
|
|
|
|
SparkConf conf = new SparkConf().setAppName("L-BFGS Example");
|
|
|
|
SparkContext sc = new SparkContext(conf);
|
|
|
|
String path = "data/mllib/sample_libsvm_data.txt";
|
|
|
|
JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc, path).toJavaRDD();
|
|
|
|
int numFeatures = data.take(1).get(0).features().size();
|
|
|
|
|
|
|
|
// Split initial RDD into two... [60% training data, 40% testing data].
|
|
|
|
JavaRDD<LabeledPoint> trainingInit = data.sample(false, 0.6, 11L);
|
|
|
|
JavaRDD<LabeledPoint> test = data.subtract(trainingInit);
|
|
|
|
|
|
|
|
// Append 1 into the training data as intercept.
|
|
|
|
JavaRDD<Tuple2<Object, Vector>> training = data.map(
|
|
|
|
new Function<LabeledPoint, Tuple2<Object, Vector>>() {
|
|
|
|
public Tuple2<Object, Vector> call(LabeledPoint p) {
|
|
|
|
return new Tuple2<Object, Vector>(p.label(), MLUtils.appendBias(p.features()));
|
|
|
|
}
|
|
|
|
});
|
|
|
|
training.cache();
|
|
|
|
|
|
|
|
// Run training algorithm to build the model.
|
|
|
|
int numCorrections = 10;
|
|
|
|
double convergenceTol = 1e-4;
|
|
|
|
int maxNumIterations = 20;
|
|
|
|
double regParam = 0.1;
|
|
|
|
Vector initialWeightsWithIntercept = Vectors.dense(new double[numFeatures + 1]);
|
|
|
|
|
|
|
|
Tuple2<Vector, double[]> result = LBFGS.runLBFGS(
|
|
|
|
training.rdd(),
|
|
|
|
new LogisticGradient(),
|
|
|
|
new SquaredL2Updater(),
|
|
|
|
numCorrections,
|
|
|
|
convergenceTol,
|
|
|
|
maxNumIterations,
|
|
|
|
regParam,
|
|
|
|
initialWeightsWithIntercept);
|
|
|
|
Vector weightsWithIntercept = result._1();
|
|
|
|
double[] loss = result._2();
|
|
|
|
|
|
|
|
final LogisticRegressionModel model = new LogisticRegressionModel(
|
|
|
|
Vectors.dense(Arrays.copyOf(weightsWithIntercept.toArray(), weightsWithIntercept.size() - 1)),
|
|
|
|
(weightsWithIntercept.toArray())[weightsWithIntercept.size() - 1]);
|
|
|
|
|
|
|
|
// Clear the default threshold.
|
|
|
|
model.clearThreshold();
|
|
|
|
|
|
|
|
// Compute raw scores on the test set.
|
|
|
|
JavaRDD<Tuple2<Object, Object>> scoreAndLabels = test.map(
|
|
|
|
new Function<LabeledPoint, Tuple2<Object, Object>>() {
|
|
|
|
public Tuple2<Object, Object> call(LabeledPoint p) {
|
|
|
|
Double score = model.predict(p.features());
|
|
|
|
return new Tuple2<Object, Object>(score, p.label());
|
|
|
|
}
|
|
|
|
});
|
|
|
|
|
|
|
|
// Get evaluation metrics.
|
|
|
|
BinaryClassificationMetrics metrics =
|
|
|
|
new BinaryClassificationMetrics(scoreAndLabels.rdd());
|
|
|
|
double auROC = metrics.areaUnderROC();
|
|
|
|
|
|
|
|
System.out.println("Loss of each step in training process");
|
|
|
|
for (double l : loss)
|
|
|
|
System.out.println(l);
|
|
|
|
System.out.println("Area under ROC = " + auROC);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
{% endhighlight %}
|
|
|
|
</div>
|
|
|
|
</div>
|
2014-05-12 22:20:24 -04:00
|
|
|
#### Developer's note
|
|
|
|
Since the Hessian is constructed approximately from previous gradient evaluations,
|
|
|
|
the objective function can not be changed during the optimization process.
|
|
|
|
As a result, Stochastic L-BFGS will not work naively by just using miniBatch;
|
|
|
|
therefore, we don't provide this until we have better understanding.
|
|
|
|
|
|
|
|
* `Updater` is a class originally designed for gradient decent which computes
|
|
|
|
the actual gradient descent step. However, we're able to take the gradient and
|
|
|
|
loss of objective function of regularization for L-BFGS by ignoring the part of logic
|
|
|
|
only for gradient decent such as adaptive step size stuff. We will refactorize
|
|
|
|
this into regularizer to replace updater to separate the logic between
|
2014-05-18 20:00:57 -04:00
|
|
|
regularization and step update later.
|