Burak Yavuz 85d609cf25 [SPARK-17613] S3A base paths with no '/' at the end return empty DataFrames ... ## What changes were proposed in this pull request? Consider you have a bucket as `s3a://some-bucket` and under it you have files: ``` s3a://some-bucket/file1.parquet s3a://some-bucket/file2.parquet ``` Getting the parent path of `s3a://some-bucket/file1.parquet` yields `s3a://some-bucket/` and the ListingFileCatalog uses this as the key in the hash map. When catalog.allFiles is called, we use `s3a://some-bucket` (no slash at the end) to get the list of files, and we're left with an empty list! This PR fixes this by adding a `/` at the end of the `URI` iff the given `Path` doesn't have a parent, i.e. is the root. This is a no-op if the path already had a `/` at the end, and is handled through the Hadoop Path, path merging semantics. ## How was this patch tested? Unit test in `FileCatalogSuite`. Author: Burak Yavuz <brkyvz@gmail.com> Closes #15169 from brkyvz/SPARK-17613.		2016-09-22 13:05:41 -07:00
..
java/org/apache/spark/sql	[SPARK-17354] [SQL] Partitioning by dates/timestamps should work with Parquet vectorized reader	2016-09-09 14:23:05 -07:00
resources	[SPARK-16031] Add debug-only socket source in Structured Streaming	2016-06-19 21:27:04 -07:00
scala/org/apache/spark/sql	[SPARK-17613] S3A base paths with no '/' at the end return empty DataFrames	2016-09-22 13:05:41 -07:00