ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[MINOR][DOCS][ML] Explicit return type of array_to_vector utility function

Browse source 
There are two types of dense vectors:
* pyspark.ml.linalg.DenseVector
* pyspark.mllib.linalg.DenseVector

In spark-3.1.1, array_to_vector returns instances of pyspark.ml.linalg.DenseVector.
The documentation is ambiguous & can lead to the false conclusion that instances of
pyspark.mllib.linalg.DenseVector will be returned.
Conversion from ml versions to mllib versions can easly be achieved with
mlutils.convertVectorColumnsToML helper.

### What changes were proposed in this pull request?
Make documentation more explicit

### Why are the changes needed?
The documentation is a bit misleading and users can lose time investigating & realizing there are two DenseVector types.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
No test were run as only the documentation was changed

Closes #32255 from jlafaye/master.

Authored-by: Julien Lafaye <jlafaye@gmail.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
This commit is contained in:
Julien Lafaye 2021-04-27 09:08:26 -05:00 committed by Sean Owen
parent 16d223efee
commit 592230e47b
1 changed files with 3 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
python/pyspark/ml/functions.py
View file

@ -71,7 +71,8 @@ def vector_to_array(col, dtype="float64"):
def array_to_vector(col):
"""
Converts a column of array of numeric type into a column of dense vectors in MLlib
Converts a column of array of numeric type into a column of pyspark.ml.linalg.DenseVector
instances
.. versionadded:: 3.1.0
@ -83,7 +84,7 @@ def array_to_vector(col):
Returns
-------
:py:class:`pyspark.sql.Column`
The converted column of MLlib dense vectors.
The converted column of dense vectors.
Examples
--------