ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/python/pyspark/ml
History
zhengruifeng 432afac07e [SPARK-32907][ML] adaptively blockify instances - revert blockify gmm 
### What changes were proposed in this pull request?
revert blockify gmm

### Why are the changes needed?
WeichenXu123  and I thought we should use memory size instead of number of rows to blockify instance; then if a buffer's size is large and determined by number of rows, we should discard it.
In GMM, we found that the pre-allocated memory maybe too large and should be discarded:
```
transient private lazy val auxiliaryPDFMat = DenseMatrix.zeros(blockSize, numFeatures)
```
We had some offline discuss and thought it is better to revert blockify GMM.

### Does this PR introduce _any_ user-facing change?
blockSize added in master branch will be removed

### How was this patch tested?
existing testsuites

Closes #29782 from zhengruifeng/unblockify_gmm.

Authored-by: zhengruifeng <ruifengz@foxmail.com>
Signed-off-by: zhengruifeng <ruifengz@foxmail.com>
2020-09-23 15:54:56 +08:00
..
linalg [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
param [SPARK-32814][PYTHON] Replace __metaclass__ field with metaclass keyword 2020-09-16 20:22:11 +09:00
tests [SPARK-32092][ML][PYSPARK][FOLLOWUP] Fixed CrossValidatorModel.copy() to copy models instead of list 2020-08-28 10:15:16 -07:00
__init__.py [SPARK-32319][PYSPARK] Disallow the use of unused imports 2020-08-08 08:51:57 -07:00
base.py [SPARK-32814][PYTHON] Replace __metaclass__ field with metaclass keyword 2020-09-16 20:22:11 +09:00
classification.py [SPARK-32933][PYTHON] Use keyword-only syntax for keyword_only methods 2020-09-23 09:28:33 +09:00
clustering.py [SPARK-32907][ML] adaptively blockify instances - revert blockify gmm 2020-09-23 15:54:56 +08:00
common.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
evaluation.py [SPARK-32933][PYTHON] Use keyword-only syntax for keyword_only methods 2020-09-23 09:28:33 +09:00
feature.py [SPARK-32933][PYTHON] Use keyword-only syntax for keyword_only methods 2020-09-23 09:28:33 +09:00
fpm.py [SPARK-32933][PYTHON] Use keyword-only syntax for keyword_only methods 2020-09-23 09:28:33 +09:00
functions.py [SPARK-30859][PYSPARK][DOCS][MINOR] Fixed docstring syntax issues preventing proper compilation of documentation 2020-02-18 16:46:45 +09:00
image.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
pipeline.py [SPARK-32933][PYTHON] Use keyword-only syntax for keyword_only methods 2020-09-23 09:28:33 +09:00
recommendation.py [SPARK-32933][PYTHON] Use keyword-only syntax for keyword_only methods 2020-09-23 09:28:33 +09:00
regression.py [SPARK-32933][PYTHON] Use keyword-only syntax for keyword_only methods 2020-09-23 09:28:33 +09:00
stat.py [SPARK-32319][PYSPARK] Disallow the use of unused imports 2020-08-08 08:51:57 -07:00
tree.py [SPARK-32719][PYTHON] Add Flake8 check missing imports 2020-08-31 11:23:31 +09:00
tuning.py [SPARK-32933][PYTHON] Use keyword-only syntax for keyword_only methods 2020-09-23 09:28:33 +09:00
util.py [SPARK-32319][PYSPARK] Disallow the use of unused imports 2020-08-08 08:51:57 -07:00
wrapper.py [SPARK-32814][PYTHON] Replace __metaclass__ field with metaclass keyword 2020-09-16 20:22:11 +09:00