### What changes were proposed in this pull request? Change the link to the Scala API document. ``` $ git grep "#org.apache.spark.package" docs/_layouts/global.html: <li><a href="api/scala/index.html#org.apache.spark.package">Scala</a></li> docs/index.md:* [Spark Scala API (Scaladoc)](api/scala/index.html#org.apache.spark.package) docs/rdd-programming-guide.md:[Scala](api/scala/#org.apache.spark.package), [Java](api/java/), [Python](api/python/) and [R](api/R/). ``` ### Why are the changes needed? The home page link for Scala API document is incorrect after upgrade to 3.0 ### Does this PR introduce any user-facing change? Document UI change only. ### How was this patch tested? Local test, attach screenshots below: Before: ![image](https://user-images.githubusercontent.com/4833765/74335713-c2385300-4dd7-11ea-95d8-f5a3639d2578.png) After: ![image](https://user-images.githubusercontent.com/4833765/74335727-cbc1bb00-4dd7-11ea-89d9-4dcc1310e679.png) Closes #27549 from xuanyuanking/scala-doc. Authored-by: Yuanjian Li <xyliyuanjian@gmail.com> Signed-off-by: Sean Owen <srowen@gmail.com>
9.8 KiB
layout | title | displayTitle | license |
---|---|---|---|
global | Clustering | Clustering | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. |
This page describes clustering algorithms in MLlib. The guide for clustering in the RDD-based API also has relevant information about these algorithms.
Table of Contents
- This will become a table of contents (this text will be scraped). {:toc}
K-means
k-means is one of the most commonly used clustering algorithms that clusters the data points into a predefined number of clusters. The MLlib implementation includes a parallelized variant of the k-means++ method called kmeans||.
KMeans
is implemented as an Estimator
and generates a KMeansModel
as the base model.
Input Columns
Param name | Type(s) | Default | Description |
---|---|---|---|
featuresCol | Vector | "features" | Feature vector |
Output Columns
Param name | Type(s) | Default | Description |
---|---|---|---|
predictionCol | Int | "prediction" | Predicted cluster center |
Examples
{% include_example scala/org/apache/spark/examples/ml/KMeansExample.scala %}
{% include_example java/org/apache/spark/examples/ml/JavaKMeansExample.java %}
{% include_example python/ml/kmeans_example.py %}
Refer to the R API docs for more details.
{% include_example r/ml/kmeans.R %}
Latent Dirichlet allocation (LDA)
LDA
is implemented as an Estimator
that supports both EMLDAOptimizer
and OnlineLDAOptimizer
,
and generates a LDAModel
as the base model. Expert users may cast a LDAModel
generated by
EMLDAOptimizer
to a DistributedLDAModel
if needed.
Examples
Refer to the Scala API docs for more details.
{% include_example scala/org/apache/spark/examples/ml/LDAExample.scala %}
Refer to the Java API docs for more details.
{% include_example java/org/apache/spark/examples/ml/JavaLDAExample.java %}
Refer to the Python API docs for more details.
{% include_example python/ml/lda_example.py %}
Refer to the R API docs for more details.
{% include_example r/ml/lda.R %}
Bisecting k-means
Bisecting k-means is a kind of hierarchical clustering using a divisive (or "top-down") approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.
Bisecting K-means can often be much faster than regular K-means, but it will generally produce a different clustering.
BisectingKMeans
is implemented as an Estimator
and generates a BisectingKMeansModel
as the base model.
Examples
{% include_example scala/org/apache/spark/examples/ml/BisectingKMeansExample.scala %}
{% include_example java/org/apache/spark/examples/ml/JavaBisectingKMeansExample.java %}
{% include_example python/ml/bisecting_k_means_example.py %}
Refer to the R API docs for more details.
{% include_example r/ml/bisectingKmeans.R %}
Gaussian Mixture Model (GMM)
A Gaussian Mixture Model
represents a composite distribution whereby points are drawn from one of k Gaussian sub-distributions,
each with its own probability. The spark.ml
implementation uses the
expectation-maximization
algorithm to induce the maximum-likelihood model given a set of samples.
GaussianMixture
is implemented as an Estimator
and generates a GaussianMixtureModel
as the base
model.
Input Columns
Param name | Type(s) | Default | Description |
---|---|---|---|
featuresCol | Vector | "features" | Feature vector |
Output Columns
Param name | Type(s) | Default | Description |
---|---|---|---|
predictionCol | Int | "prediction" | Predicted cluster center |
probabilityCol | Vector | "probability" | Probability of each cluster |
Examples
{% include_example scala/org/apache/spark/examples/ml/GaussianMixtureExample.scala %}
{% include_example java/org/apache/spark/examples/ml/JavaGaussianMixtureExample.java %}
{% include_example python/ml/gaussian_mixture_example.py %}
Refer to the R API docs for more details.
{% include_example r/ml/gaussianMixture.R %}
Power Iteration Clustering (PIC)
Power Iteration Clustering (PIC) is a scalable graph clustering algorithm developed by Lin and Cohen. From the abstract: PIC finds a very low-dimensional embedding of a dataset using truncated power iteration on a normalized pair-wise similarity matrix of the data.
spark.ml
's PowerIterationClustering implementation takes the following parameters:
k
: the number of clusters to createinitMode
: param for the initialization algorithmmaxIter
: param for maximum number of iterationssrcCol
: param for the name of the input column for source vertex IDsdstCol
: name of the input column for destination vertex IDsweightCol
: Param for weight column name
Examples
{% include_example scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala %}
{% include_example java/org/apache/spark/examples/ml/JavaPowerIterationClusteringExample.java %}
{% include_example python/ml/power_iteration_clustering_example.py %}
Refer to the R API docs for more details.
{% include_example r/ml/powerIterationClustering.R %}