a28728a9af
## What changes were proposed in this pull request? In previous work SPARK-21513, we has allowed `MapType` and `ArrayType` of `MapType`s convert to a json string but only for Scala API. In this follow-up PR, we will make SparkSQL support it for PySpark and SparkR, too. We also fix some little bugs and comments of the previous work in this follow-up PR. ### For PySpark ``` >>> data = [(1, {"name": "Alice"})] >>> df = spark.createDataFrame(data, ("key", "value")) >>> df.select(to_json(df.value).alias("json")).collect() [Row(json=u'{"name":"Alice")'] >>> data = [(1, [{"name": "Alice"}, {"name": "Bob"}])] >>> df = spark.createDataFrame(data, ("key", "value")) >>> df.select(to_json(df.value).alias("json")).collect() [Row(json=u'[{"name":"Alice"},{"name":"Bob"}]')] ``` ### For SparkR ``` # Converts a map into a JSON object df2 <- sql("SELECT map('name', 'Bob')) as people") df2 <- mutate(df2, people_json = to_json(df2$people)) # Converts an array of maps into a JSON array df2 <- sql("SELECT array(map('name', 'Bob'), map('name', 'Alice')) as people") df2 <- mutate(df2, people_json = to_json(df2$people)) ``` ## How was this patch tested? Add unit test cases. cc viirya HyukjinKwon Author: goldmedal <liugs963@gmail.com> Closes #19223 from goldmedal/SPARK-21513-fp-PySaprkAndSparkR. |
||
---|---|---|
.. | ||
jarTest.R | ||
packageInAJarTest.R | ||
test_binary_function.R | ||
test_binaryFile.R | ||
test_broadcast.R | ||
test_client.R | ||
test_context.R | ||
test_includePackage.R | ||
test_jvm_api.R | ||
test_mllib_classification.R | ||
test_mllib_clustering.R | ||
test_mllib_fpm.R | ||
test_mllib_recommendation.R | ||
test_mllib_regression.R | ||
test_mllib_stat.R | ||
test_mllib_tree.R | ||
test_parallelize_collect.R | ||
test_rdd.R | ||
test_Serde.R | ||
test_shuffle.R | ||
test_sparkR.R | ||
test_sparkSQL.R | ||
test_streaming.R | ||
test_take.R | ||
test_textFile.R | ||
test_utils.R | ||
test_Windows.R |