spark-instrumented-optimizer/sql/hive
Tathagata Das f7b7ef4166 [SPARK-14997][SQL] Fixed FileCatalog to return correct set of files when there is no partitioning scheme in the given paths
## What changes were proposed in this pull request?
Lets says there are json files in the following directories structure
```
xyz/file0.json
xyz/subdir1/file1.json
xyz/subdir2/file2.json
xyz/subdir1/subsubdir1/file3.json
```
`sqlContext.read.json("xyz")` should read only file0.json according to behavior in Spark 1.6.1. However in current master, all the 4 files are read.

The fix is to make FileCatalog return only the children files of the given path if there is not partitioning detected (instead of all the recursive list of files).

Closes #12774

## How was this patch tested?

unit tests

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #12856 from tdas/SPARK-14997.
2016-05-06 15:04:16 -07:00
..
compatibility/src/test/scala/org/apache/spark/sql/hive/execution [SPARK-14414][SQL] Make DDL exceptions more consistent 2016-05-03 18:07:53 -07:00
src [SPARK-14997][SQL] Fixed FileCatalog to return correct set of files when there is no partitioning scheme in the given paths 2016-05-06 15:04:16 -07:00
pom.xml Revert "[SPARK-14613][ML] Add @Since into the matrix and vector classes in spark-mllib-local" 2016-04-28 19:57:41 -07:00