spark-instrumented-optimizer

History

Peng Meng a0345cbebe [SPARK-21680][ML][MLLIB] optimize Vector compress ## What changes were proposed in this pull request? When use Vector.compressed to change a Vector to SparseVector, the performance is very low comparing with Vector.toSparse. This is because you have to scan the value three times using Vector.compressed, but you just need two times when use Vector.toSparse. When the length of the vector is large, there is significant performance difference between this two method. ## How was this patch tested? The existing UT Author: Peng Meng <peng.meng@intel.com> Closes #18899 from mpjlu/optVectorCompress.	2017-08-16 19:05:20 +01:00
..
src	[SPARK-21680][ML][MLLIB] optimize Vector compress	2017-08-16 19:05:20 +01:00
pom.xml	[SPARK-15526][ML][FOLLOWUP] Make JPMML provided scope to avoid including unshaded JARs, and repromote to compile in MLlib	2017-07-18 09:53:51 -07:00

Peng Meng a0345cbebe [SPARK-21680][ML][MLLIB] optimize Vector compress

## What changes were proposed in this pull request?

When use Vector.compressed to change a Vector to SparseVector, the performance is very low comparing with Vector.toSparse.
This is because you have to scan the value three times using Vector.compressed, but you just need two times when use Vector.toSparse.
When the length of the vector is large, there is significant performance difference between this two method.

## How was this patch tested?

The existing UT

Author: Peng Meng <peng.meng@intel.com>

Closes #18899 from mpjlu/optVectorCompress.

2017-08-16 19:05:20 +01:00

src

[SPARK-21680][ML][MLLIB] optimize Vector compress

2017-08-16 19:05:20 +01:00

pom.xml

[SPARK-15526][ML][FOLLOWUP] Make JPMML provided scope to avoid including unshaded JARs, and repromote to compile in MLlib

2017-07-18 09:53:51 -07:00