spark-instrumented-optimizer/python/pyspark/ml
Max Gekk 7eb6f45688 [SPARK-32499][SQL] Use {} in conversions maps and structs to strings
### What changes were proposed in this pull request?
Change casting of map and struct values to strings by using the `{}` brackets instead of `[]`. The behavior is controlled by the SQL config `spark.sql.legacy.castComplexTypesToString.enabled`. When it is `true`, `CAST` wraps maps and structs by `[]` in casting to strings. Otherwise, if this is `false`, which is the default, maps and structs are wrapped by `{}`.

### Why are the changes needed?
- To distinguish structs/maps from arrays.
- To make `show`'s output consistent with Hive and conversions to Hive strings.
- To display dataframe content in the same form by `spark-sql` and `show`
- To be consistent with the `*.sql` tests

### Does this PR introduce _any_ user-facing change?
Yes

### How was this patch tested?
By existing test suite `CastSuite`.

Closes #29308 from MaxGekk/show-struct-map.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
2020-08-04 14:57:09 +00:00
..
linalg [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
param [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
tests [SPARK-32310][ML][PYSPARK] ML params default value parity in feature and tuning 2020-08-03 08:50:34 -07:00
__init__.py [SPARK-29212][ML][PYSPARK] Add common classes without using JVM backend 2020-03-04 12:20:02 +08:00
base.py [SPARK-30930][ML] Remove ML/MLLIB DeveloperApi annotations 2020-03-16 12:41:22 -05:00
classification.py [SPARK-32310][ML][PYSPARK] ML params default value parity in feature and tuning 2020-08-03 08:50:34 -07:00
clustering.py [SPARK-32310][ML][PYSPARK] ML params default value parity in feature and tuning 2020-08-03 08:50:34 -07:00
common.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
evaluation.py [SPARK-31734][ML][PYSPARK] Add weight support in ClusteringEvaluator 2020-05-25 09:18:08 -05:00
feature.py [SPARK-32310][ML][PYSPARK] ML params default value parity in feature and tuning 2020-08-03 08:50:34 -07:00
fpm.py [SPARK-32310][ML][PYSPARK] ML params default value parity in feature and tuning 2020-08-03 08:50:34 -07:00
functions.py [SPARK-30859][PYSPARK][DOCS][MINOR] Fixed docstring syntax issues preventing proper compilation of documentation 2020-02-18 16:46:45 +09:00
image.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
pipeline.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
recommendation.py [SPARK-32310][ML][PYSPARK] ML params default value parity in feature and tuning 2020-08-03 08:50:34 -07:00
regression.py [SPARK-32310][ML][PYSPARK] ML params default value parity in feature and tuning 2020-08-03 08:50:34 -07:00
stat.py [SPARK-32499][SQL] Use {} in conversions maps and structs to strings 2020-08-04 14:57:09 +00:00
tree.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
tuning.py [SPARK-32310][ML][PYSPARK] ML params default value parity in feature and tuning 2020-08-03 08:50:34 -07:00
util.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00
wrapper.py [SPARK-32138] Drop Python 2.7, 3.4 and 3.5 2020-07-14 11:22:44 +09:00