2014-02-08 14:39:13 -05:00
---
layout: global
2014-05-18 20:00:57 -04:00
title: Clustering - MLlib
displayTitle: < a href = "mllib-guide.html" > MLlib< / a > - Clustering
2014-02-08 14:39:13 -05:00
---
Clustering is an unsupervised learning problem whereby we aim to group subsets
of entities with one another based on some notion of similarity. Clustering is
often used for exploratory analysis and/or as a component of a hierarchical
supervised learning pipeline (in which distinct classifiers or regression
2015-02-13 15:31:27 -05:00
models are trained for each cluster).
2014-04-22 14:20:47 -04:00
2015-02-06 13:26:51 -05:00
MLlib supports the following models:
2015-02-13 18:09:27 -05:00
* Table of contents
{:toc}
## K-means
2015-02-06 13:26:51 -05:00
[k-means ](http://en.wikipedia.org/wiki/K-means_clustering ) is one of the
most commonly used clustering algorithms that clusters the data points into a
2014-05-06 23:07:22 -04:00
predefined number of clusters. The MLlib implementation includes a parallelized
2014-02-08 14:39:13 -05:00
variant of the [k-means++ ](http://en.wikipedia.org/wiki/K-means%2B%2B ) method
called [kmeans|| ](http://theory.stanford.edu/~sergei/papers/vldb12-kmpar.pdf ).
2015-02-13 15:31:27 -05:00
The implementation in MLlib has the following parameters:
2014-02-08 14:39:13 -05:00
* *k* is the number of desired clusters.
* *maxIterations* is the maximum number of iterations to run.
* *initializationMode* specifies either random initialization or
initialization via k-means\|\|.
* *runs* is the number of times to run the k-means algorithm (k-means is not
guaranteed to find a globally optimal solution, and when run multiple times on
a given dataset, the algorithm returns the best clustering result).
2014-05-06 23:07:22 -04:00
* *initializationSteps* determines the number of steps in the k-means\|\| algorithm.
2015-02-13 15:31:27 -05:00
* *epsilon* determines the distance threshold within which we consider k-means to have converged.
2014-02-08 14:39:13 -05:00
2015-02-13 18:09:27 -05:00
**Examples**
2015-02-06 13:26:51 -05:00
2014-04-22 14:20:47 -04:00
< div class = "codetabs" >
< div data-lang = "scala" markdown = "1" >
2014-08-12 20:15:21 -04:00
The following code snippets can be executed in `spark-shell` .
2014-02-08 14:39:13 -05:00
2014-04-22 14:20:47 -04:00
In the following example after loading and parsing data, we use the
2014-05-18 20:00:57 -04:00
[`KMeans` ](api/scala/index.html#org.apache.spark.mllib.clustering.KMeans ) object to cluster the data
2014-02-08 14:39:13 -05:00
into two clusters. The number of desired clusters is passed to the algorithm. We then compute Within
Set Sum of Squared Error (WSSSE). You can reduce this error measure by increasing *k* . In fact the
optimal *k* is usually one where there is an "elbow" in the WSSSE graph.
{% highlight scala %}
import org.apache.spark.mllib.clustering.KMeans
[WIP] SPARK-1430: Support sparse data in Python MLlib
This PR adds a SparseVector class in PySpark and updates all the regression, classification and clustering algorithms and models to support sparse data, similar to MLlib. I chose to add this class because SciPy is quite difficult to install in many environments (more so than NumPy), but I plan to add support for SciPy sparse vectors later too, and make the methods work transparently on objects of either type.
On the Scala side, we keep Python sparse vectors sparse and pass them to MLlib. We always return dense vectors from our models.
Some to-do items left:
- [x] Support SciPy's scipy.sparse matrix objects when SciPy is available. We can easily add a function to convert these to our own SparseVector.
- [x] MLlib currently uses a vector with one extra column on the left to represent what we call LabeledPoint in Scala. Do we really want this? It may get annoying once you deal with sparse data since you must add/subtract 1 to each feature index when training. We can remove this API in 1.0 and use tuples for labeling.
- [x] Explain how to use these in the Python MLlib docs.
CC @mengxr, @joshrosen
Author: Matei Zaharia <matei@databricks.com>
Closes #341 from mateiz/py-ml-update and squashes the following commits:
d52e763 [Matei Zaharia] Remove no-longer-needed slice code and handle review comments
ea5a25a [Matei Zaharia] Fix remaining uses of copyto() after merge
b9f97a3 [Matei Zaharia] Fix test
1e1bd0f [Matei Zaharia] Add MLlib logistic regression example in Python
88bc01f [Matei Zaharia] Clean up inheritance of LinearModel in Python, and expose its parametrs
37ab747 [Matei Zaharia] Fix some examples and docs due to changes in MLlib API
da0f27e [Matei Zaharia] Added a MLlib K-means example and updated docs to discuss sparse data
c48e85a [Matei Zaharia] Added some tests for passing lists as input, and added mllib/tests.py to run-tests script.
a07ba10 [Matei Zaharia] Fix some typos and calculation of initial weights
74eefe7 [Matei Zaharia] Added LabeledPoint class in Python
889dde8 [Matei Zaharia] Support scipy.sparse matrices in all our algorithms and models
ab244d1 [Matei Zaharia] Allow SparseVectors to be initialized using a dict
a5d6426 [Matei Zaharia] Add linalg.py to run-tests script
0e7a3d8 [Matei Zaharia] Keep vectors sparse in Java when reading LabeledPoints
eaee759 [Matei Zaharia] Update regression, classification and clustering models for sparse data
2abbb44 [Matei Zaharia] Further work to get linear models working with sparse data
154f45d [Matei Zaharia] Update docs, name some magic values
881fef7 [Matei Zaharia] Added a sparse vector in Python and made Java-Python format more compact
2014-04-15 23:33:24 -04:00
import org.apache.spark.mllib.linalg.Vectors
2014-02-08 14:39:13 -05:00
// Load and parse the data
2014-07-13 22:27:43 -04:00
val data = sc.textFile("data/mllib/kmeans_data.txt")
[SPARK-1484][MLLIB] Warn when running an iterative algorithm on uncached data.
Add warnings to KMeans, GeneralizedLinearAlgorithm, and computeSVD when called with input data that is not cached. KMeans is implemented iteratively, and I believe that GeneralizedLinearAlgorithm’s current optimizers are iterative and its future optimizers are also likely to be iterative. RowMatrix’s computeSVD is iterative against an RDD when run in DistARPACK mode. ALS and DecisionTree are iterative as well, but they implement RDD caching internally so do not require a warning.
I added a warning to GeneralizedLinearAlgorithm rather than inside its optimizers, where the iteration actually occurs, because internally GeneralizedLinearAlgorithm maps its input data to an uncached RDD before passing it to an optimizer. (In other words, the warning would be printed for every GeneralizedLinearAlgorithm run, regardless of whether its input is cached, if the warning were in GradientDescent or other optimizer.) I assume that use of an uncached RDD by GeneralizedLinearAlgorithm is intentional, and that the mapping there (adding label, intercepts and scaling) is a lightweight operation. Arguably a user calling an optimizer such as GradientDescent will be knowledgable enough to cache their data without needing a log warning, so lack of a warning in the optimizers may be ok.
Some of the documentation examples making use of these iterative algorithms did not cache their training RDDs (while others did). I updated the examples to always cache. I also fixed some (unrelated) minor errors in the documentation examples.
Author: Aaron Staple <aaron.staple@gmail.com>
Closes #2347 from staple/SPARK-1484 and squashes the following commits:
bd49701 [Aaron Staple] Address review comments.
ab2d4a4 [Aaron Staple] Disable warnings on python code path.
a7a0f99 [Aaron Staple] Change code comments per review comments.
7cca1dc [Aaron Staple] Change warning message text.
c77e939 [Aaron Staple] [SPARK-1484][MLLIB] Warn when running an iterative algorithm on uncached data.
3b6c511 [Aaron Staple] Minor doc example fixes.
2014-09-25 19:11:00 -04:00
val parsedData = data.map(s => Vectors.dense(s.split(' ').map(_.toDouble))).cache()
2014-02-08 14:39:13 -05:00
// Cluster the data into two classes using KMeans
val numClusters = 2
[WIP] SPARK-1430: Support sparse data in Python MLlib
This PR adds a SparseVector class in PySpark and updates all the regression, classification and clustering algorithms and models to support sparse data, similar to MLlib. I chose to add this class because SciPy is quite difficult to install in many environments (more so than NumPy), but I plan to add support for SciPy sparse vectors later too, and make the methods work transparently on objects of either type.
On the Scala side, we keep Python sparse vectors sparse and pass them to MLlib. We always return dense vectors from our models.
Some to-do items left:
- [x] Support SciPy's scipy.sparse matrix objects when SciPy is available. We can easily add a function to convert these to our own SparseVector.
- [x] MLlib currently uses a vector with one extra column on the left to represent what we call LabeledPoint in Scala. Do we really want this? It may get annoying once you deal with sparse data since you must add/subtract 1 to each feature index when training. We can remove this API in 1.0 and use tuples for labeling.
- [x] Explain how to use these in the Python MLlib docs.
CC @mengxr, @joshrosen
Author: Matei Zaharia <matei@databricks.com>
Closes #341 from mateiz/py-ml-update and squashes the following commits:
d52e763 [Matei Zaharia] Remove no-longer-needed slice code and handle review comments
ea5a25a [Matei Zaharia] Fix remaining uses of copyto() after merge
b9f97a3 [Matei Zaharia] Fix test
1e1bd0f [Matei Zaharia] Add MLlib logistic regression example in Python
88bc01f [Matei Zaharia] Clean up inheritance of LinearModel in Python, and expose its parametrs
37ab747 [Matei Zaharia] Fix some examples and docs due to changes in MLlib API
da0f27e [Matei Zaharia] Added a MLlib K-means example and updated docs to discuss sparse data
c48e85a [Matei Zaharia] Added some tests for passing lists as input, and added mllib/tests.py to run-tests script.
a07ba10 [Matei Zaharia] Fix some typos and calculation of initial weights
74eefe7 [Matei Zaharia] Added LabeledPoint class in Python
889dde8 [Matei Zaharia] Support scipy.sparse matrices in all our algorithms and models
ab244d1 [Matei Zaharia] Allow SparseVectors to be initialized using a dict
a5d6426 [Matei Zaharia] Add linalg.py to run-tests script
0e7a3d8 [Matei Zaharia] Keep vectors sparse in Java when reading LabeledPoints
eaee759 [Matei Zaharia] Update regression, classification and clustering models for sparse data
2abbb44 [Matei Zaharia] Further work to get linear models working with sparse data
154f45d [Matei Zaharia] Update docs, name some magic values
881fef7 [Matei Zaharia] Added a sparse vector in Python and made Java-Python format more compact
2014-04-15 23:33:24 -04:00
val numIterations = 20
2014-02-08 14:39:13 -05:00
val clusters = KMeans.train(parsedData, numClusters, numIterations)
// Evaluate clustering by computing Within Set Sum of Squared Errors
val WSSSE = clusters.computeCost(parsedData)
println("Within Set Sum of Squared Errors = " + WSSSE)
{% endhighlight %}
2014-04-22 14:20:47 -04:00
< / div >
2014-02-08 14:39:13 -05:00
2014-04-22 14:20:47 -04:00
< div data-lang = "java" markdown = "1" >
2014-02-08 14:39:13 -05:00
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
2014-10-15 00:37:51 -04:00
calling `.rdd()` on your `JavaRDD` object. A self-contained application example
2014-08-12 20:15:21 -04:00
that is equivalent to the provided example in Scala is given below:
2014-07-20 23:48:44 -04:00
{% highlight java %}
import org.apache.spark.api.java.*;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.mllib.clustering.KMeans;
import org.apache.spark.mllib.clustering.KMeansModel;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
import org.apache.spark.SparkConf;
public class KMeansExample {
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("K-means Example");
JavaSparkContext sc = new JavaSparkContext(conf);
// Load and parse data
String path = "data/mllib/kmeans_data.txt";
JavaRDD< String > data = sc.textFile(path);
JavaRDD< Vector > parsedData = data.map(
new Function< String , Vector > () {
public Vector call(String s) {
String[] sarray = s.split(" ");
double[] values = new double[sarray.length];
for (int i = 0; i < sarray.length ; i + + )
values[i] = Double.parseDouble(sarray[i]);
return Vectors.dense(values);
}
}
);
[SPARK-1484][MLLIB] Warn when running an iterative algorithm on uncached data.
Add warnings to KMeans, GeneralizedLinearAlgorithm, and computeSVD when called with input data that is not cached. KMeans is implemented iteratively, and I believe that GeneralizedLinearAlgorithm’s current optimizers are iterative and its future optimizers are also likely to be iterative. RowMatrix’s computeSVD is iterative against an RDD when run in DistARPACK mode. ALS and DecisionTree are iterative as well, but they implement RDD caching internally so do not require a warning.
I added a warning to GeneralizedLinearAlgorithm rather than inside its optimizers, where the iteration actually occurs, because internally GeneralizedLinearAlgorithm maps its input data to an uncached RDD before passing it to an optimizer. (In other words, the warning would be printed for every GeneralizedLinearAlgorithm run, regardless of whether its input is cached, if the warning were in GradientDescent or other optimizer.) I assume that use of an uncached RDD by GeneralizedLinearAlgorithm is intentional, and that the mapping there (adding label, intercepts and scaling) is a lightweight operation. Arguably a user calling an optimizer such as GradientDescent will be knowledgable enough to cache their data without needing a log warning, so lack of a warning in the optimizers may be ok.
Some of the documentation examples making use of these iterative algorithms did not cache their training RDDs (while others did). I updated the examples to always cache. I also fixed some (unrelated) minor errors in the documentation examples.
Author: Aaron Staple <aaron.staple@gmail.com>
Closes #2347 from staple/SPARK-1484 and squashes the following commits:
bd49701 [Aaron Staple] Address review comments.
ab2d4a4 [Aaron Staple] Disable warnings on python code path.
a7a0f99 [Aaron Staple] Change code comments per review comments.
7cca1dc [Aaron Staple] Change warning message text.
c77e939 [Aaron Staple] [SPARK-1484][MLLIB] Warn when running an iterative algorithm on uncached data.
3b6c511 [Aaron Staple] Minor doc example fixes.
2014-09-25 19:11:00 -04:00
parsedData.cache();
2014-07-20 23:48:44 -04:00
// Cluster the data into two classes using KMeans
int numClusters = 2;
int numIterations = 20;
KMeansModel clusters = KMeans.train(parsedData.rdd(), numClusters, numIterations);
// Evaluate clustering by computing Within Set Sum of Squared Errors
double WSSSE = clusters.computeCost(parsedData.rdd());
System.out.println("Within Set Sum of Squared Errors = " + WSSSE);
}
}
{% endhighlight %}
2014-04-22 14:20:47 -04:00
< / div >
2014-02-08 14:39:13 -05:00
2014-04-22 14:20:47 -04:00
< div data-lang = "python" markdown = "1" >
2014-08-12 20:15:21 -04:00
The following examples can be tested in the PySpark shell.
2014-02-08 14:39:13 -05:00
2014-04-22 14:20:47 -04:00
In the following example after loading and parsing data, we use the KMeans object to cluster the
data into two clusters. The number of desired clusters is passed to the algorithm. We then compute
Within Set Sum of Squared Error (WSSSE). You can reduce this error measure by increasing *k* . In
fact the optimal *k* is usually one where there is an "elbow" in the WSSSE graph.
2014-02-08 14:39:13 -05:00
{% highlight python %}
from pyspark.mllib.clustering import KMeans
from numpy import array
from math import sqrt
# Load and parse the data
2014-07-13 22:27:43 -04:00
data = sc.textFile("data/mllib/kmeans_data.txt")
2014-02-08 14:39:13 -05:00
parsedData = data.map(lambda line: array([float(x) for x in line.split(' ')]))
# Build the model (cluster the data)
clusters = KMeans.train(parsedData, 2, maxIterations=10,
2014-04-22 14:20:47 -04:00
runs=10, initializationMode="random")
2014-02-08 14:39:13 -05:00
# Evaluate clustering by computing Within Set Sum of Squared Errors
def error(point):
center = clusters.centers[clusters.predict(point)]
return sqrt(sum([x**2 for x in (point - center)]))
WSSSE = parsedData.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print("Within Set Sum of Squared Error = " + str(WSSSE))
{% endhighlight %}
2014-04-22 14:20:47 -04:00
< / div >
2014-02-08 14:39:13 -05:00
2014-04-22 14:20:47 -04:00
< / div >
2014-10-15 00:37:51 -04:00
2015-02-13 18:09:27 -05:00
## Gaussian mixture
A [Gaussian Mixture Model ](http://en.wikipedia.org/wiki/Mixture_model#Multivariate_Gaussian_mixture_model )
represents a composite distribution whereby points are drawn from one of *k* Gaussian sub-distributions,
each with its own probability. The MLlib implementation uses the
[expectation-maximization ](http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm )
algorithm to induce the maximum-likelihood model given a set of samples. The implementation
has the following parameters:
* *k* is the number of desired clusters.
* *convergenceTol* is the maximum change in log-likelihood at which we consider convergence achieved.
* *maxIterations* is the maximum number of iterations to perform without reaching convergence.
* *initialModel* is an optional starting point from which to start the EM algorithm. If this parameter is omitted, a random starting point will be constructed from the data.
**Examples**
2015-02-06 13:26:51 -05:00
< div class = "codetabs" >
< div data-lang = "scala" markdown = "1" >
In the following example after loading and parsing data, we use a
2015-02-13 15:31:27 -05:00
[GaussianMixture ](api/scala/index.html#org.apache.spark.mllib.clustering.GaussianMixture )
object to cluster the data into two clusters. The number of desired clusters is passed
2015-02-06 13:26:51 -05:00
to the algorithm. We then output the parameters of the mixture model.
{% highlight scala %}
import org.apache.spark.mllib.clustering.GaussianMixture
2015-03-25 17:45:23 -04:00
import org.apache.spark.mllib.clustering.GaussianMixtureModel
2015-02-06 13:26:51 -05:00
import org.apache.spark.mllib.linalg.Vectors
// Load and parse the data
val data = sc.textFile("data/mllib/gmm_data.txt")
val parsedData = data.map(s => Vectors.dense(s.trim.split(' ').map(_.toDouble))).cache()
// Cluster the data into two classes using GaussianMixture
val gmm = new GaussianMixture().setK(2).run(parsedData)
2015-03-25 17:45:23 -04:00
// Save and load model
gmm.save(sc, "myGMMModel")
val sameModel = GaussianMixtureModel.load(sc, "myGMMModel")
2015-02-06 13:26:51 -05:00
// output parameters of max-likelihood model
for (i < - 0 until gmm . k ) {
2015-02-13 15:31:27 -05:00
println("weight=%f\nmu=%s\nsigma=\n%s\n" format
2015-02-06 13:26:51 -05:00
(gmm.weights(i), gmm.gaussians(i).mu, gmm.gaussians(i).sigma))
}
{% endhighlight %}
< / div >
< div data-lang = "java" markdown = "1" >
All of MLlib's methods use Java-friendly types, so you can import and call them there the same
way you do in Scala. The only caveat is that the methods take Scala RDD objects, while the
Spark Java API uses a separate `JavaRDD` class. You can convert a Java RDD to a Scala one by
calling `.rdd()` on your `JavaRDD` object. A self-contained application example
that is equivalent to the provided example in Scala is given below:
{% highlight java %}
import org.apache.spark.api.java.*;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.mllib.clustering.GaussianMixture;
import org.apache.spark.mllib.clustering.GaussianMixtureModel;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
import org.apache.spark.SparkConf;
public class GaussianMixtureExample {
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("GaussianMixture Example");
JavaSparkContext sc = new JavaSparkContext(conf);
// Load and parse data
String path = "data/mllib/gmm_data.txt";
JavaRDD< String > data = sc.textFile(path);
JavaRDD< Vector > parsedData = data.map(
new Function< String , Vector > () {
public Vector call(String s) {
String[] sarray = s.trim().split(" ");
double[] values = new double[sarray.length];
for (int i = 0; i < sarray.length ; i + + )
values[i] = Double.parseDouble(sarray[i]);
return Vectors.dense(values);
}
}
);
parsedData.cache();
// Cluster the data into two classes using GaussianMixture
GaussianMixtureModel gmm = new GaussianMixture().setK(2).run(parsedData.rdd());
2015-03-25 17:45:23 -04:00
// Save and load GaussianMixtureModel
gmm.save(sc, "myGMMModel")
GaussianMixtureModel sameModel = GaussianMixtureModel.load(sc, "myGMMModel")
2015-02-06 13:26:51 -05:00
// Output the parameters of the mixture model
for(int j=0; j< gmm.k ( ) ; j + + ) {
System.out.println("weight=%f\nmu=%s\nsigma=\n%s\n",
gmm.weights()[j], gmm.gaussians()[j].mu(), gmm.gaussians()[j].sigma());
}
}
}
{% endhighlight %}
< / div >
< div data-lang = "python" markdown = "1" >
In the following example after loading and parsing data, we use a
[GaussianMixture ](api/python/pyspark.mllib.html#pyspark.mllib.clustering.GaussianMixture )
2015-02-13 15:31:27 -05:00
object to cluster the data into two clusters. The number of desired clusters is passed
2015-02-06 13:26:51 -05:00
to the algorithm. We then output the parameters of the mixture model.
{% highlight python %}
from pyspark.mllib.clustering import GaussianMixture
from numpy import array
# Load and parse the data
data = sc.textFile("data/mllib/gmm_data.txt")
parsedData = data.map(lambda line: array([float(x) for x in line.strip().split(' ')]))
# Build the model (cluster the data)
gmm = GaussianMixture.train(parsedData, 2)
# output parameters of model
for i in range(2):
print ("weight = ", gmm.weights[i], "mu = ", gmm.gaussians[i].mu,
"sigma = ", gmm.gaussians[i].sigma.toArray())
{% endhighlight %}
< / div >
< / div >
2015-02-13 18:09:27 -05:00
## Power iteration clustering (PIC)
2015-02-18 19:29:32 -05:00
Power iteration clustering (PIC) is a scalable and efficient algorithm for clustering vertices of a
graph given pairwise similarties as edge properties,
described in [Lin and Cohen, Power Iteration Clustering ](http://www.icml2010.org/papers/387.pdf ).
It computes a pseudo-eigenvector of the normalized affinity matrix of the graph via
[power iteration ](http://en.wikipedia.org/wiki/Power_iteration ) and uses it to cluster vertices.
MLlib includes an implementation of PIC using GraphX as its backend.
It takes an `RDD` of `(srcId, dstId, similarity)` tuples and outputs a model with the clustering assignments.
The similarities must be nonnegative.
PIC assumes that the similarity measure is symmetric.
A pair `(srcId, dstId)` regardless of the ordering should appear at most once in the input data.
If a pair is missing from input, their similarity is treated as zero.
MLlib's PIC implementation takes the following (hyper-)parameters:
* `k` : number of clusters
* `maxIterations` : maximum number of power iterations
* `initializationMode` : initialization model. This can be either "random", which is the default,
to use a random vector as vertex properties, or "degree" to use normalized sum similarities.
2015-02-13 18:09:27 -05:00
2015-02-18 19:29:32 -05:00
**Examples**
In the following, we show code snippets to demonstrate how to use PIC in MLlib.
< div class = "codetabs" >
< div data-lang = "scala" markdown = "1" >
[`PowerIterationClustering` ](api/scala/index.html#org.apache.spark.mllib.clustering.PowerIterationClustering )
implements the PIC algorithm.
It takes an `RDD` of `(srcId: Long, dstId: Long, similarity: Double)` tuples representing the
affinity matrix.
Calling `PowerIterationClustering.run` returns a
[`PowerIterationClusteringModel` ](api/scala/index.html#org.apache.spark.mllib.clustering.PowerIterationClusteringModel ),
which contains the computed clustering assignments.
{% highlight scala %}
import org.apache.spark.mllib.clustering.PowerIterationClustering
import org.apache.spark.mllib.linalg.Vectors
val similarities: RDD[(Long, Long, Double)] = ...
val pic = new PowerIteartionClustering()
.setK(3)
.setMaxIterations(20)
val model = pic.run(similarities)
[SPARK-5900][MLLIB] make PIC and FPGrowth Java-friendly
In the previous version, PIC stores clustering assignments as an `RDD[(Long, Int)]`. This is mapped to `RDD<Tuple2<Object, Object>>` in Java and hence Java users have to cast types manually. We should either create a new method called `javaAssignments` that returns `JavaRDD[(java.lang.Long, java.lang.Int)]` or wrap the result pair in a class. I chose the latter approach in this PR. Now assignments are stored as an `RDD[Assignment]`, where `Assignment` is a class with `id` and `cluster`.
Similarly, in FPGrowth, the frequent itemsets are stored as an `RDD[(Array[Item], Long)]`, which is mapped to `RDD<Tuple2<Object, Object>>`. Though we provide a "Java-friendly" method `javaFreqItemsets` that returns `JavaRDD[(Array[Item], java.lang.Long)]`. It doesn't really work because `Array[Item]` is mapped to `Object` in Java. So in this PR I created a class `FreqItemset` to wrap the results. It has `items` and `freq`, as well as a `javaItems` method that returns `List<Item>` in Java.
I'm not certain that the names I chose are proper: `Assignment`/`id`/`cluster` and `FreqItemset`/`items`/`freq`. Please let me know if there are better suggestions.
CC: jkbradley
Author: Xiangrui Meng <meng@databricks.com>
Closes #4695 from mengxr/SPARK-5900 and squashes the following commits:
865b5ca [Xiangrui Meng] make Assignment serializable
cffa96e [Xiangrui Meng] fix test
9c0e590 [Xiangrui Meng] remove unused Tuple2
1b9db3d [Xiangrui Meng] make PIC and FPGrowth Java-friendly
2015-02-19 21:06:16 -05:00
model.assignments.foreach { a =>
println(s"${a.id} -> ${a.cluster}")
2015-02-18 19:29:32 -05:00
}
{% endhighlight %}
2015-02-13 18:09:27 -05:00
2015-02-18 19:29:32 -05:00
A full example that produces the experiment described in the PIC paper can be found under
[`examples/` ](https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/PowerIterationClusteringExample.scala ).
2015-02-13 18:09:27 -05:00
2015-02-18 19:29:32 -05:00
< / div >
< div data-lang = "java" markdown = "1" >
[`PowerIterationClustering` ](api/java/org/apache/spark/mllib/clustering/PowerIterationClustering.html )
implements the PIC algorithm.
It takes an `JavaRDD` of `(srcId: Long, dstId: Long, similarity: Double)` tuples representing the
affinity matrix.
Calling `PowerIterationClustering.run` returns a
[`PowerIterationClusteringModel` ](api/java/org/apache/spark/mllib/clustering/PowerIterationClusteringModel.html )
which contains the computed clustering assignments.
{% highlight java %}
import scala.Tuple2;
import scala.Tuple3;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.mllib.clustering.PowerIterationClustering;
import org.apache.spark.mllib.clustering.PowerIterationClusteringModel;
2015-02-13 18:09:27 -05:00
2015-02-18 19:29:32 -05:00
JavaRDD< Tuple3 < Long , Long , Double > > similarities = ...
PowerIterationClustering pic = new PowerIterationClustering()
.setK(2)
.setMaxIterations(10);
PowerIterationClusteringModel model = pic.run(similarities);
[SPARK-5900][MLLIB] make PIC and FPGrowth Java-friendly
In the previous version, PIC stores clustering assignments as an `RDD[(Long, Int)]`. This is mapped to `RDD<Tuple2<Object, Object>>` in Java and hence Java users have to cast types manually. We should either create a new method called `javaAssignments` that returns `JavaRDD[(java.lang.Long, java.lang.Int)]` or wrap the result pair in a class. I chose the latter approach in this PR. Now assignments are stored as an `RDD[Assignment]`, where `Assignment` is a class with `id` and `cluster`.
Similarly, in FPGrowth, the frequent itemsets are stored as an `RDD[(Array[Item], Long)]`, which is mapped to `RDD<Tuple2<Object, Object>>`. Though we provide a "Java-friendly" method `javaFreqItemsets` that returns `JavaRDD[(Array[Item], java.lang.Long)]`. It doesn't really work because `Array[Item]` is mapped to `Object` in Java. So in this PR I created a class `FreqItemset` to wrap the results. It has `items` and `freq`, as well as a `javaItems` method that returns `List<Item>` in Java.
I'm not certain that the names I chose are proper: `Assignment`/`id`/`cluster` and `FreqItemset`/`items`/`freq`. Please let me know if there are better suggestions.
CC: jkbradley
Author: Xiangrui Meng <meng@databricks.com>
Closes #4695 from mengxr/SPARK-5900 and squashes the following commits:
865b5ca [Xiangrui Meng] make Assignment serializable
cffa96e [Xiangrui Meng] fix test
9c0e590 [Xiangrui Meng] remove unused Tuple2
1b9db3d [Xiangrui Meng] make PIC and FPGrowth Java-friendly
2015-02-19 21:06:16 -05:00
for (PowerIterationClustering.Assignment a: model.assignments().toJavaRDD().collect()) {
System.out.println(a.id() + " -> " + a.cluster());
2015-02-18 19:29:32 -05:00
}
{% endhighlight %}
< / div >
< / div >
2015-02-13 18:09:27 -05:00
## Latent Dirichlet allocation (LDA)
[Latent Dirichlet allocation (LDA) ](http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation )
is a topic model which infers topics from a collection of text documents.
LDA can be thought of as a clustering algorithm as follows:
* Topics correspond to cluster centers, and documents correspond to examples (rows) in a dataset.
* Topics and documents both exist in a feature space, where feature vectors are vectors of word counts.
* Rather than estimating a clustering using a traditional distance, LDA uses a function based
on a statistical model of how text documents are generated.
LDA takes in a collection of documents as vectors of word counts.
It learns clustering using [expectation-maximization ](http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm )
on the likelihood function. After fitting on the documents, LDA provides:
* Topics: Inferred topics, each of which is a probability distribution over terms (words).
* Topic distributions for documents: For each document in the training set, LDA gives a probability distribution over topics.
LDA takes the following parameters:
* `k` : Number of topics (i.e., cluster centers)
* `maxIterations` : Limit on the number of iterations of EM used for learning
* `docConcentration` : Hyperparameter for prior over documents' distributions over topics. Currently must be > 1, where larger values encourage smoother inferred distributions.
* `topicConcentration` : Hyperparameter for prior over topics' distributions over terms (words). Currently must be > 1, where larger values encourage smoother inferred distributions.
* `checkpointInterval` : If using checkpointing (set in the Spark configuration), this parameter specifies the frequency with which checkpoints will be created. If `maxIterations` is large, using checkpointing can help reduce shuffle file sizes on disk and help with failure recovery.
*Note*: LDA is a new feature with some missing functionality. In particular, it does not yet
support prediction on new documents, and it does not have a Python API. These will be added in the future.
**Examples**
2015-02-09 02:40:36 -05:00
In the following example, we load word count vectors representing a corpus of documents.
We then use [LDA ](api/scala/index.html#org.apache.spark.mllib.clustering.LDA )
2015-02-13 15:31:27 -05:00
to infer three topics from the documents. The number of desired clusters is passed
2015-02-09 02:40:36 -05:00
to the algorithm. We then output the topics, represented as probability distributions over words.
< div class = "codetabs" >
< div data-lang = "scala" markdown = "1" >
{% highlight scala %}
import org.apache.spark.mllib.clustering.LDA
import org.apache.spark.mllib.linalg.Vectors
// Load and parse the data
val data = sc.textFile("data/mllib/sample_lda_data.txt")
val parsedData = data.map(s => Vectors.dense(s.trim.split(' ').map(_.toDouble)))
// Index documents with unique IDs
val corpus = parsedData.zipWithIndex.map(_.swap).cache()
// Cluster the documents into three topics using LDA
val ldaModel = new LDA().setK(3).run(corpus)
// Output topics. Each is a distribution over words (matching word count vectors)
println("Learned topics (as distributions over vocab of " + ldaModel.vocabSize + " words):")
val topics = ldaModel.topicsMatrix
for (topic < - Range ( 0 , 3 ) ) {
print("Topic " + topic + ":")
for (word < - Range ( 0 , ldaModel . vocabSize ) ) { print ( " " + topics ( word , topic ) ) ; }
println()
}
{% endhighlight %}
< / div >
< div data-lang = "java" markdown = "1" >
{% highlight java %}
import scala.Tuple2;
import org.apache.spark.api.java.*;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.mllib.clustering.DistributedLDAModel;
import org.apache.spark.mllib.clustering.LDA;
import org.apache.spark.mllib.linalg.Matrix;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
import org.apache.spark.SparkConf;
public class JavaLDAExample {
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("LDA Example");
JavaSparkContext sc = new JavaSparkContext(conf);
// Load and parse the data
String path = "data/mllib/sample_lda_data.txt";
JavaRDD< String > data = sc.textFile(path);
JavaRDD< Vector > parsedData = data.map(
new Function< String , Vector > () {
public Vector call(String s) {
String[] sarray = s.trim().split(" ");
double[] values = new double[sarray.length];
for (int i = 0; i < sarray.length ; i + + )
values[i] = Double.parseDouble(sarray[i]);
return Vectors.dense(values);
}
}
);
// Index documents with unique IDs
JavaPairRDD< Long , Vector > corpus = JavaPairRDD.fromJavaRDD(parsedData.zipWithIndex().map(
new Function< Tuple2 < Vector , Long > , Tuple2< Long , Vector > >() {
public Tuple2< Long , Vector > call(Tuple2< Vector , Long > doc_id) {
return doc_id.swap();
}
}
));
corpus.cache();
// Cluster the documents into three topics using LDA
DistributedLDAModel ldaModel = new LDA().setK(3).run(corpus);
// Output topics. Each is a distribution over words (matching word count vectors)
System.out.println("Learned topics (as distributions over vocab of " + ldaModel.vocabSize()
+ " words):");
Matrix topics = ldaModel.topicsMatrix();
for (int topic = 0; topic < 3 ; topic + + ) {
System.out.print("Topic " + topic + ":");
for (int word = 0; word < ldaModel.vocabSize ( ) ; word + + ) {
System.out.print(" " + topics.apply(word, topic));
}
System.out.println();
}
}
}
{% endhighlight %}
< / div >
< / div >
2015-02-13 18:09:27 -05:00
## Streaming k-means
2014-11-01 01:30:12 -04:00
2015-02-13 15:31:27 -05:00
When data arrive in a stream, we may want to estimate clusters dynamically,
updating them as new data arrive. MLlib provides support for streaming k-means clustering,
with parameters to control the decay (or "forgetfulness") of the estimates. The algorithm
uses a generalization of the mini-batch k-means update rule. For each batch of data, we assign
2014-11-01 01:30:12 -04:00
all points to their nearest cluster, compute new cluster centers, then update each cluster using:
`\begin{equation}
c_{t+1} = \frac{c_tn_t\alpha + x_tm_t}{n_t\alpha+m_t}
\end{equation}`
`\begin{equation}
2015-02-13 15:31:27 -05:00
n_{t+1} = n_t + m_t
2014-11-01 01:30:12 -04:00
\end{equation}`
2015-02-13 15:31:27 -05:00
Where `$c_t$` is the previous center for the cluster, `$n_t$` is the number of points assigned
to the cluster thus far, `$x_t$` is the new cluster center from the current batch, and `$m_t$`
is the number of points added to the cluster in the current batch. The decay factor `$\alpha$`
can be used to ignore the past: with `$\alpha$=1` all data will be used from the beginning;
with `$\alpha$=0` only the most recent data will be used. This is analogous to an
exponentially-weighted moving average.
2014-11-01 01:30:12 -04:00
2015-02-13 15:31:27 -05:00
The decay can be specified using a `halfLife` parameter, which determines the
2014-11-01 01:30:12 -04:00
correct decay factor `a` such that, for data acquired
at time `t` , its contribution by time `t + halfLife` will have dropped to 0.5.
The unit of time can be specified either as `batches` or `points` and the update rule
will be adjusted accordingly.
2015-02-13 18:09:27 -05:00
**Examples**
2014-11-01 01:30:12 -04:00
This example shows how to estimate clusters on streaming data.
< div class = "codetabs" >
< div data-lang = "scala" markdown = "1" >
First we import the neccessary classes.
{% highlight scala %}
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.clustering.StreamingKMeans
{% endhighlight %}
2015-02-13 15:31:27 -05:00
Then we make an input stream of vectors for training, as well as a stream of labeled data
points for testing. We assume a StreamingContext `ssc` has been created, see
[Spark Streaming Programming Guide ](streaming-programming-guide.html#initializing ) for more info.
2014-11-01 01:30:12 -04:00
{% highlight scala %}
val trainingData = ssc.textFileStream("/training/data/dir").map(Vectors.parse)
val testData = ssc.textFileStream("/testing/data/dir").map(LabeledPoint.parse)
{% endhighlight %}
We create a model with random clusters and specify the number of clusters to find
{% highlight scala %}
val numDimensions = 3
val numClusters = 2
val model = new StreamingKMeans()
.setK(numClusters)
.setDecayFactor(1.0)
.setRandomCenters(numDimensions, 0.0)
{% endhighlight %}
2015-02-13 15:31:27 -05:00
Now register the streams for training and testing and start the job, printing
2014-11-01 01:30:12 -04:00
the predicted cluster assignments on new data points as they arrive.
{% highlight scala %}
model.trainOn(trainingData)
2015-02-13 15:31:27 -05:00
model.predictOnValues(testData.map(lp => (lp.label, lp.features))).print()
2014-11-01 01:30:12 -04:00
ssc.start()
ssc.awaitTermination()
2015-02-13 15:31:27 -05:00
2014-11-01 01:30:12 -04:00
{% endhighlight %}
2015-02-13 15:31:27 -05:00
As you add new text files with data the cluster centers will update. Each training
2014-11-01 01:30:12 -04:00
point should be formatted as `[x1, x2, x3]` , and each test data point
2015-02-13 15:31:27 -05:00
should be formatted as `(y, [x1, x2, x3])` , where `y` is some useful label or identifier
(e.g. a true category assignment). Anytime a text file is placed in `/training/data/dir`
the model will update. Anytime a text file is placed in `/testing/data/dir`
2014-11-01 01:30:12 -04:00
you will see predictions. With new data, the cluster centers will change!
< / div >
< / div >