spark-instrumented-optimizer/python/pyspark/ml
Peng 79ff853631 [SPARK-17645][MLLIB][ML] add feature selector method based on: False Discovery Rate (FDR) and Family wise error rate (FWE)
## What changes were proposed in this pull request?

Univariate feature selection works by selecting the best features based on univariate statistical tests.
FDR and FWE are a popular univariate statistical test for feature selection.
In 2005, the Benjamini and Hochberg paper on FDR was identified as one of the 25 most-cited statistical papers. The FDR uses the Benjamini-Hochberg procedure in this PR. https://en.wikipedia.org/wiki/False_discovery_rate.
In statistics, FWE is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses tests.
https://en.wikipedia.org/wiki/Family-wise_error_rate

We add  FDR and FWE methods for ChiSqSelector in this PR, like it is implemented in scikit-learn.
http://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection
## How was this patch tested?

ut will be added soon

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Author: Peng <peng.meng@intel.com>
Author: Peng, Meng <peng.meng@intel.com>

Closes #15212 from mpjlu/fdr_fwe.
2016-12-28 00:49:36 -08:00
..
linalg [SPARK-18447][DOCS] Fix the markdown for Note:/NOTE:/Note that across Python API documentation 2016-11-22 11:40:18 +00:00
param [SPARK-17057][ML] ProbabilisticClassifierModels' thresholds should have at most one 0 2016-09-24 08:15:55 +01:00
__init__.py [SPARK-14817][ML][MLLIB][DOC] Made DataFrame-based API primary in MLlib guide 2016-07-15 13:38:23 -07:00
base.py [SPARK-15364][ML][PYSPARK] Implement PySpark picklers for ml.Vector and ml.Matrix under spark.ml.python 2016-06-13 19:59:53 -07:00
classification.py [SPARK-18319][ML][QA2.1] 2.1 QA: API: Experimental, DeveloperApi, final, sealed audit 2016-11-29 18:46:59 -08:00
clustering.py [SPARK-15819][PYSPARK][ML] Add KMeanSummary in KMeans of PySpark 2016-11-29 20:51:27 -08:00
common.py [SPARK-17679] [PYSPARK] remove unnecessary Py4J ListConverter patch 2016-10-03 14:12:03 -07:00
evaluation.py [SPARK-15402][ML][PYSPARK] PySpark ml.evaluation should support save/load 2016-10-14 04:17:03 -07:00
feature.py [SPARK-17645][MLLIB][ML] add feature selector method based on: False Discovery Rate (FDR) and Family wise error rate (FWE) 2016-12-28 00:49:36 -08:00
pipeline.py [SPARK-15018][PYSPARK][ML] Improve handling of PySpark Pipeline when used without stages 2016-08-19 23:46:36 -07:00
recommendation.py [SPARK-15741][PYSPARK][ML] Pyspark cleanup of set default seed to None 2016-06-21 11:43:25 -07:00
regression.py [SPARK-18447][DOCS] Fix the markdown for Note:/NOTE:/Note that across Python API documentation 2016-11-22 11:40:18 +00:00
tests.py [SPARK-18274][ML][PYSPARK] Memory leak in PySpark JavaWrapper 2016-12-01 13:22:40 -08:00
tuning.py [SPARK-16831][PYTHON] Fixed bug in CrossValidator.avgMetrics 2016-08-03 04:18:28 -07:00
util.py [SPARK-18319][ML][QA2.1] 2.1 QA: API: Experimental, DeveloperApi, final, sealed audit 2016-11-29 18:46:59 -08:00
wrapper.py [SPARK-18274][ML][PYSPARK] Memory leak in PySpark JavaWrapper 2016-12-01 13:22:40 -08:00