spark-instrumented-optimizer/project
Liang-Chi Hsieh 83013752e3 [SPARK-23455][ML] Default Params in ML should be saved separately in metadata
## What changes were proposed in this pull request?

We save ML's user-supplied params and default params as one entity in metadata. During loading the saved models, we set all the loaded params into created ML model instances as user-supplied params.

It causes some problems, e.g., if we strictly disallow some params to be set at the same time, a default param can fail the param check because it is treated as user-supplied param after loading.

The loaded default params should not be set as user-supplied params. We should save ML default params separately in metadata.

For backward compatibility, when loading metadata, if it is a metadata file from previous Spark, we shouldn't raise error if we can't find the default param field.

## How was this patch tested?

Pass existing tests and added tests.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes #20633 from viirya/save-ml-default-params.
2018-04-24 10:40:25 -07:00
..
build.properties [SPARK-21709][BUILD] sbt 0.13.16 and some plugin updates 2017-08-12 20:01:20 +01:00
MimaBuild.scala [SPARK-23070] Bump previousSparkVersion in MimaBuild.scala to be 2.2.0 2018-01-15 22:32:38 +08:00
MimaExcludes.scala [SPARK-23455][ML] Default Params in ML should be saved separately in metadata 2018-04-24 10:40:25 -07:00
plugins.sbt [SPARK-21708][BUILD] update some sbt plugins 2017-10-31 08:16:54 +00:00
SparkBuild.scala [SPARK-22855][BUILD] Add -no-java-comments to sbt docs/scalacOptions 2017-12-21 10:08:38 -06:00