spark-instrumented-optimizer/R/pkg
Wenchen Fan 3a1bc6f478 [SPARK-17470][SQL] unify path for data source table and locationUri for hive serde table
## What changes were proposed in this pull request?

Due to a limitation of hive metastore(table location must be directory path, not file path), we always store `path` for data source table in storage properties, instead of the `locationUri` field. However, we should not expose this difference to `CatalogTable` level, but just treat it as a hack in `HiveExternalCatalog`, like we store table schema of data source table in table properties.

This PR unifies `path` and `locationUri` outside of `HiveExternalCatalog`, both data source table and hive serde table should use the `locationUri` field.

This PR also unifies the way we handle default table location for managed table. Previously, the default table location of hive serde managed table is set by external catalog, but the one of data source table is set by command. After this PR, we follow the hive way and the default table location is always set by external catalog.

For managed non-file-based tables, we will assign a default table location and create an empty directory for it, the table location will be removed when the table is dropped. This is reasonable as metastore doesn't care about whether a table is file-based or not, and an empty table directory has no harm.
For external non-file-based tables, ideally we can omit the table location, but due to a hive metastore issue, we will assign a random location to it, and remove it right after the table is created. See SPARK-15269 for more details. This is fine as it's well isolated in `HiveExternalCatalog`.

To keep the existing behaviour of the `path` option, in this PR we always add the `locationUri` to storage properties using key `path`, before passing storage properties to `DataSource` as data source options.
## How was this patch tested?

existing tests.

Author: Wenchen Fan <wenchen@databricks.com>

Closes #15024 from cloud-fan/path.
2016-11-02 18:05:14 -07:00
..
inst [SPARK-17470][SQL] unify path for data source table and locationUri for hive serde table 2016-11-02 18:05:14 -07:00
R [SPARK-17838][SPARKR] Check named arguments for options and use formatted R friendly message from JVM exception message 2016-11-01 22:14:53 -07:00
src-native [SPARK-6811] Copy SparkR lib in make-distribution.sh 2015-05-23 00:04:01 -07:00
tests [SPARK-12034][SPARKR] Eliminate warnings in SparkR test cases. 2015-12-07 10:38:17 -08:00
vignettes [SPARKR][DOC] minor formatting and output cleanup for R vignettes 2016-10-04 09:22:26 -07:00
.lintr [SPARK-12327][SPARKR] fix code for lintr warning for commented code 2016-01-03 20:53:35 +05:30
.Rbuildignore [SPARK-16507][SPARKR] Add a CRAN checker, fix Rd aliases 2016-07-16 17:06:44 -07:00
DESCRIPTION [SPARK-16581][SPARKR] Make JVM backend calling functions public 2016-08-29 12:55:32 -07:00
NAMESPACE [SPARK-16137][SPARKR] randomForest for R 2016-10-30 16:19:19 -07:00