spark-instrumented-optimizer/python/pyspark/sql
Wenchen Fan d57daf1f77 [SPARK-13593] [SQL] improve the createDataFrame to accept data type string and verify the data
## What changes were proposed in this pull request?

This PR improves the `createDataFrame` method to make it also accept datatype string, then users can convert python RDD to DataFrame easily, for example, `df = rdd.toDF("a: int, b: string")`.
It also supports flat schema so users can convert an RDD of int to DataFrame directly, we will automatically wrap int to row for users.
If schema is given, now we checks if the real data matches the given schema, and throw error if it doesn't.

## How was this patch tested?

new tests in `test.py` and doc test in `types.py`

Author: Wenchen Fan <wenchen@databricks.com>

Closes #11444 from cloud-fan/pyrdd.
2016-03-08 14:00:03 -08:00
..
__init__.py [SPARK-12600][SQL] Remove deprecated methods in Spark SQL 2016-01-04 18:02:38 -08:00
column.py [SPARK-12799] Simplify various string output for expressions 2016-02-21 22:53:15 +08:00
context.py [SPARK-13593] [SQL] improve the createDataFrame to accept data type string and verify the data 2016-03-08 14:00:03 -08:00
dataframe.py [SPARK-13594][SQL] remove typed operations(e.g. map, flatMap) from python DataFrame 2016-03-02 15:26:34 -08:00
functions.py [SPARK-12720][SQL] SQL Generation Support for Cube, Rollup, and Grouping Sets 2016-03-05 19:25:03 +08:00
group.py [SPARK-12756][SQL] use hash expression in Exchange 2016-01-13 22:43:28 -08:00
readwriter.py [SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option() 2016-03-03 10:30:55 -08:00
tests.py [SPARK-13593] [SQL] improve the createDataFrame to accept data type string and verify the data 2016-03-08 14:00:03 -08:00
types.py [SPARK-13593] [SQL] improve the createDataFrame to accept data type string and verify the data 2016-03-08 14:00:03 -08:00
utils.py [SPARK-11804] [PYSPARK] Exception raise when using Jdbc predicates opt… 2015-11-18 08:18:54 -08:00
window.py [SPARK-10373] [PYSPARK] move @since into pyspark from sql 2015-09-08 20:56:22 -07:00