spark-instrumented-optimizer/R/pkg/tests/fulltests
hyukjinkwon 2bfd5accdc [SPARK-21266][R][PYTHON] Support schema a DDL-formatted string in dapply/gapply/from_json
## What changes were proposed in this pull request?

This PR supports schema in a DDL formatted string for `from_json` in R/Python and `dapply` and `gapply` in R, which are commonly used and/or consistent with Scala APIs.

Additionally, this PR exposes `structType` in R to allow working around in other possible corner cases.

**Python**

`from_json`

```python
from pyspark.sql.functions import from_json

data = [(1, '''{"a": 1}''')]
df = spark.createDataFrame(data, ("key", "value"))
df.select(from_json(df.value, "a INT").alias("json")).show()
```

**R**

`from_json`

```R
df <- sql("SELECT named_struct('name', 'Bob') as people")
df <- mutate(df, people_json = to_json(df$people))
head(select(df, from_json(df$people_json, "name STRING")))
```

`structType.character`

```R
structType("a STRING, b INT")
```

`dapply`

```R
dapply(createDataFrame(list(list(1.0)), "a"), function(x) {x}, "a DOUBLE")
```

`gapply`

```R
gapply(createDataFrame(list(list(1.0)), "a"), "a", function(key, x) { x }, "a DOUBLE")
```

## How was this patch tested?

Doc tests for `from_json` in Python and unit tests `test_sparkSQL.R` in R.

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #18498 from HyukjinKwon/SPARK-21266.
2017-07-10 10:40:03 -07:00
..
jarTest.R [SPARK-20877][SPARKR] refactor tests to basic tests only for CRAN 2017-06-11 00:00:33 -07:00
packageInAJarTest.R [SPARK-20877][SPARKR] refactor tests to basic tests only for CRAN 2017-06-11 00:00:33 -07:00
test_binary_function.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_binaryFile.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_broadcast.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_client.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_context.R [SPARK-21149][R] Add job description API for R 2017-06-23 09:59:24 -07:00
test_includePackage.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_jvm_api.R [SPARK-20877][SPARKR] refactor tests to basic tests only for CRAN 2017-06-11 00:00:33 -07:00
test_mllib_classification.R [SPARK-20906][SPARKR] Constrained Logistic Regression for SparkR 2017-06-21 20:42:45 -07:00
test_mllib_clustering.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_mllib_fpm.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_mllib_recommendation.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_mllib_regression.R [SPARK-20917][ML][SPARKR] SparkR supports string encoding consistent with R 2017-06-21 10:35:16 -07:00
test_mllib_stat.R [SPARK-20877][SPARKR] refactor tests to basic tests only for CRAN 2017-06-11 00:00:33 -07:00
test_mllib_tree.R [SPARK-20307][SPARKR] SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer 2017-07-07 23:51:32 -07:00
test_parallelize_collect.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_rdd.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_Serde.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_shuffle.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_sparkR.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_sparkSQL.R [SPARK-21266][R][PYTHON] Support schema a DDL-formatted string in dapply/gapply/from_json 2017-07-10 10:40:03 -07:00
test_streaming.R [SPARK-21224][R] Specify a schema by using a DDL-formatted string when reading in R 2017-06-28 19:36:00 -07:00
test_take.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_textFile.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_utils.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00
test_Windows.R [SPARK-20877][SPARKR][FOLLOWUP] clean up after test move 2017-06-11 03:00:44 -07:00