ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/python/pyspark/ml
History
Weichen Xu f9f6c0d350 [SPARK-36425][PYSPARK][ML] Support CrossValidatorModel get standard deviation of metrics for each paramMap 
Signed-off-by: Weichen Xu <weichen.xudatabricks.com>

### What changes were proposed in this pull request?
Support CrossValidatorModel get standard deviation of metrics for each paramMap.

### Why are the changes needed?
So that in mlflow autologging, we can log standard deviation of metrics which is useful.

### Does this PR introduce _any_ user-facing change?
Yes.
`CrossValidatorModel` add a public attribute `stdMetrics` which are the standard deviation of metrics for each paramMap

### How was this patch tested?
Unit test.

Closes #33652 from WeichenXu123/add_std_metric.

Authored-by: Weichen Xu <weichen.xu@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
2021-08-09 10:08:52 +09:00
..
linalg [SPARK-36146][PYTHON][INFRA][TESTS] Upgrade Python version from 3.6 to 3.9 in GitHub Actions' linter/docs 2021-07-15 08:01:54 -07:00
param [SPARK-35176][PYTHON] Standardize input validation error type 2021-05-03 15:34:24 +09:00
tests [SPARK-36425][PYSPARK][ML] Support CrossValidatorModel get standard deviation of metrics for each paramMap 2021-08-09 10:08:52 +09:00
__init__.py [SPARK-32319][PYSPARK] Disallow the use of unused imports 2020-08-08 08:51:57 -07:00
_typing.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
base.py [SPARK-35176][PYTHON] Standardize input validation error type 2021-05-03 15:34:24 +09:00
base.pyi [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) 2020-11-10 09:33:48 +09:00
classification.py [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Copy local properties when starting the thread, and use inheritable thread in the current codebase 2021-06-20 11:48:38 +09:00
classification.pyi [SPARK-33520][ML][PYSPARK] make CrossValidator/TrainValidateSplit/OneVsRest Reader/Writer support Python backend estimator/evaluator 2020-12-04 08:35:50 +08:00
clustering.py [SPARK-35392][ML][PYTHON] Fix flaky tests in ml/clustering.py and ml/feature.py 2021-05-13 22:23:51 +09:00
clustering.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
common.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
common.pyi [SPARK-33457][PYTHON] Adjust mypy configuration 2020-11-25 09:27:04 +09:00
evaluation.py [SPARK-35176][PYTHON] Standardize input validation error type 2021-05-03 15:34:24 +09:00
evaluation.pyi [SPARK-33457][PYTHON] Adjust mypy configuration 2020-11-25 09:27:04 +09:00
feature.py [SPARK-35392][ML][PYTHON] Fix flaky tests in ml/clustering.py and ml/feature.py 2021-05-13 22:23:51 +09:00
feature.pyi [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector 2021-01-16 11:09:23 +08:00
fpm.py [MINOR][DOCS] Use ASCII characters when possible in PySpark documentation 2021-04-04 09:49:36 +03:00
fpm.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
functions.py [MINOR][DOCS][ML] Explicit return type of array_to_vector utility function 2021-04-27 09:08:26 -05:00
functions.pyi [SPARK-33556][ML] Add array_to_vector function for dataframe column 2020-12-01 09:52:19 +09:00
image.py [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) 2020-11-10 09:33:48 +09:00
image.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
pipeline.py [SPARK-33592] Fix: Pyspark ML Validator params in estimatorParamMaps may be lost after saving and reloading 2020-12-01 09:36:42 +08:00
pipeline.pyi [SPARK-33457][PYTHON] Adjust mypy configuration 2020-11-25 09:27:04 +09:00
recommendation.py [SPARK-35150][ML] Accelerate fallback BLAS with dev.ludovic.netlib 2021-04-27 14:00:59 -05:00
recommendation.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
regression.py [MINOR][DOCS] Avoid some python docs where first sentence has "e.g." or similar 2021-05-12 10:38:59 +09:00
regression.pyi Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
stat.py [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector 2021-01-16 11:09:23 +08:00
stat.pyi [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector 2021-01-16 11:09:23 +08:00
tree.py [SPARK-34093][ML] param maxDepth should check upper bound 2021-01-18 11:36:10 -06:00
tree.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00
tuning.py [SPARK-36425][PYSPARK][ML] Support CrossValidatorModel get standard deviation of metrics for each paramMap 2021-08-09 10:08:52 +09:00
tuning.pyi [SPARK-34415][ML] Randomization in hyperparameter optimization 2021-02-27 08:34:39 -06:00
util.py [SPARK-36225][PYTHON][DOCS] Use DataFrame in python docstrings 2021-07-24 16:58:10 +09:00
util.pyi [SPARK-33520][ML][PYSPARK] make CrossValidator/TrainValidateSplit/OneVsRest Reader/Writer support Python backend estimator/evaluator 2020-12-04 08:35:50 +08:00
wrapper.py [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) 2020-11-10 09:33:48 +09:00
wrapper.pyi [SPARK-32714][PYTHON] Initial pyspark-stubs port 2020-09-24 14:15:36 +09:00