spark-instrumented-optimizer/python/test_support/sql
hyukjinkwon 7e5359be5c [SPARK-19610][SQL] Support parsing multiline CSV files
## What changes were proposed in this pull request?

This PR proposes the support for multiple lines for CSV by resembling the multiline supports in JSON datasource (in case of JSON, per file).

So, this PR introduces `wholeFile` option which makes the format not splittable and reads each whole file. Since Univocity parser can produces each row from a stream, it should be capable of parsing very large documents when the internal rows are fix in the memory.

## How was this patch tested?

Unit tests in `CSVSuite` and `tests.py`

Manual tests with a single 9GB CSV file in local file system, for example,

```scala
spark.read.option("wholeFile", true).option("inferSchema", true).csv("tmp.csv").count()
```

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #16976 from HyukjinKwon/SPARK-19610.
2017-02-28 13:34:33 -08:00
..
orc_partitioned [SPARK-10716] [BUILD] spark-1.5.0-bin-hadoop2.6.tgz file doesn't uncompress on OS X due to hidden file 2015-09-21 23:29:59 -07:00
parquet_partitioned [SPARK-8060] Improve DataFrame Python test coverage and documentation. 2015-06-03 00:23:34 -07:00
streaming [SPARK-14555] First cut of Python API for Structured Streaming 2016-04-20 10:32:01 -07:00
ages.csv [SPARK-13509][SPARK-13507][SQL] Support for writing CSV with a single function call 2016-02-29 09:44:29 -08:00
ages_newlines.csv [SPARK-19610][SQL] Support parsing multiline CSV files 2017-02-28 13:34:33 -08:00
people.json [SPARK-8060] Improve DataFrame Python test coverage and documentation. 2015-06-03 00:23:34 -07:00
people1.json [SPARK-10185] [SQL] Feat sql comma separated paths 2015-10-17 14:56:24 -07:00
people_array.json [SPARK-18352][SQL] Support parsing multiline json files 2017-02-16 20:51:19 -08:00
text-test.txt [SPARK-11292] [SQL] Python API for text data source 2015-10-28 14:28:38 -07:00