ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-33299][SQL][DOCS] Don't mention schemas in JSON format in docs for from_json

Browse source 
### What changes were proposed in this pull request?
Remove the JSON formatted schema from comments for `from_json()` in Scala/Python APIs.

Closes #30201

### Why are the changes needed?
Schemas in JSON format is internal (not documented). It shouldn't be recommenced for usage.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
By linters.

Closes #30226 from MaxGekk/from_json-common-schema-parsing-2.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
Max Gekk 2020-11-02 10:10:24 -08:00 committed by Dongjoon Hyun
parent eecebd0302
commit bdabf60fb4
2 changed files with 3 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
python/pyspark/sql/functions.py
View file

@ -2847,8 +2847,7 @@ def from_json(col, schema, options={}):
:param schema: a StructType or ArrayType of StructType to use when parsing the json column. :param schema: a StructType or ArrayType of StructType to use when parsing the json column.
:param options: options to control parsing. accepts the same options as the json datasource :param options: options to control parsing. accepts the same options as the json datasource
.. note:: Since Spark 2.3, the DDL-formatted string or a JSON format string is also .. note:: Since Spark 2.3, the DDL-formatted string is also supported for ``schema``.
supported for ``schema``.
>>> from pyspark.sql.types import * >>> from pyspark.sql.types import *
>>> data = [(1, '''{"a": 1}''')] >>> data = [(1, '''{"a": 1}''')]

7
sql/core/src/main/scala/org/apache/spark/sql/functions.scala
View file

@ -4077,9 +4077,7 @@ object functions {
* Returns `null`, in the case of an unparseable string. * Returns `null`, in the case of an unparseable string.
* *
* @param e a string column containing JSON data. * @param e a string column containing JSON data.
* @param schema the schema to use when parsing the json string as a json string. In Spark 2.1, * @param schema the schema as a DDL-formatted string.
* the user-provided schema has to be in JSON format. Since Spark 2.2, the DDL
* format is also supported for the schema.
* *
* @group collection_funcs * @group collection_funcs
* @since 2.1.0 * @since 2.1.0
@ -4094,8 +4092,7 @@ object functions {
* Returns `null`, in the case of an unparseable string. * Returns `null`, in the case of an unparseable string.
* *
* @param e a string column containing JSON data. * @param e a string column containing JSON data.
* @param schema the schema to use when parsing the json string as a json string, it could be a * @param schema the schema as a DDL-formatted string.
* JSON format string or a DDL-formatted string.
* *
* @group collection_funcs * @group collection_funcs
* @since 2.3.0 * @since 2.3.0