Maxim Gekk 13bedc05c2 [SPARK-24329][SQL] Test for skipping multi-space lines ... ## What changes were proposed in this pull request? The PR is a continue of https://github.com/apache/spark/pull/21380 . It checks cases that are handled by the code: e3de6ab30d/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala (L303-L304) Basically the code skips lines with one or many whitespaces, and lines with comments (see [filterCommentAndEmpty](e3de6ab30d/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVUtils.scala (L47))) ```scala iter.filter { line => line.trim.nonEmpty && !line.startsWith(options.comment.toString) } ``` Closes #21380 ## How was this patch tested? Added a test for the case described above. Author: Maxim Gekk <maxim.gekk@databricks.com> Author: Maxim Gekk <max.gekk@gmail.com> Closes #21394 from MaxGekk/test-for-multi-space-lines.		2018-05-24 22:18:58 +08:00
..
benchmarks	[SPARK-17335][SQL] Fix ArrayType and MapType CatalogString.	2016-09-03 19:02:20 +02:00
src	[SPARK-24329][SQL] Test for skipping multi-space lines	2018-05-24 22:18:58 +08:00
pom.xml	[SPARK-17916][SQL] Fix empty string being parsed as null when nullValue is set.	2018-05-14 10:01:06 +08:00