Cheng Lian
8a12580d25
[SPARK-14127][SQL] "DESC <table>": Extracts schema information from table properties for data source tables
## What changes were proposed in this pull request?
This is a follow-up of #12934 and #12844. This PR adds a set of utility methods in `DDLUtils` to help extract schema information (user-defined schema, partition columns, and bucketing information) from data source table properties. These utility methods are then used in `DescribeTableCommand` to refine output for data source tables. Before this PR, the aforementioned schema information are only shown as table properties, which are hard to read.
Sample output:
```
+----------------------------+---------------------------------------------------------+-------+
|col_name |data_type |comment|
+----------------------------+---------------------------------------------------------+-------+
|a |bigint | |
|b |bigint | |
|c |bigint | |
|d |bigint | |
|# Partition Information | | |
|# col_name | | |
|d | | |
| | | |
|# Detailed Table Information| | |
|Database: |default | |
|Owner: |lian | |
|Create Time: |Tue May 10 03:20:34 PDT 2016 | |
|Last Access Time: |Wed Dec 31 16:00:00 PST 1969 | |
|Location: |file:/Users/lian/local/src/spark/workspace-a/target/... | |
|Table Type: |MANAGED | |
|Table Parameters: | | |
| rawDataSize |-1 | |
| numFiles |1 | |
| transient_lastDdlTime |1462875634 | |
| totalSize |684 | |
| spark.sql.sources.provider|parquet | |
| EXTERNAL |FALSE | |
| COLUMN_STATS_ACCURATE |false | |
| numRows |-1 | |
| | | |
|# Storage Information | | |
|SerDe Library: |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | |
|InputFormat: |org.apache.hadoop.mapred.SequenceFileInputFormat | |
|OutputFormat: |org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat| |
|Compressed: |No | |
|Num Buckets: |2 | |
|Bucket Columns: |[b] | |
|Sort Columns: |[c] | |
|Storage Desc Parameters: | | |
| path |file:/Users/lian/local/src/spark/workspace-a/target/... | |
| serialization.format |1 | |
+----------------------------+---------------------------------------------------------+-------+
```
## How was this patch tested?
Test cases are added in `HiveDDLSuite` to check command output.
Author: Cheng Lian <lian@databricks.com>
Closes #13025 from liancheng/spark-14127-extract-schema-info.