4ecc648ad7
## What changes were proposed in this pull request?
This PR implements `DESCRIBE table PARTITION` SQL Syntax again. It was supported until Spark 1.6.2, but was dropped since 2.0.0.
**Spark 1.6.2**
```scala
scala> sql("CREATE TABLE partitioned_table (a STRING, b INT) PARTITIONED BY (c STRING, d STRING)")
res1: org.apache.spark.sql.DataFrame = [result: string]
scala> sql("ALTER TABLE partitioned_table ADD PARTITION (c='Us', d=1)")
res2: org.apache.spark.sql.DataFrame = [result: string]
scala> sql("DESC partitioned_table PARTITION (c='Us', d=1)").show(false)
+----------------------------------------------------------------+
|result |
+----------------------------------------------------------------+
|a string |
|b int |
|c string |
|d string |
| |
|# Partition Information |
|# col_name data_type comment |
| |
|c string |
|d string |
+----------------------------------------------------------------+
```
**Spark 2.0**
- **Before**
```scala
scala> sql("CREATE TABLE partitioned_table (a STRING, b INT) PARTITIONED BY (c STRING, d STRING)")
res0: org.apache.spark.sql.DataFrame = []
scala> sql("ALTER TABLE partitioned_table ADD PARTITION (c='Us', d=1)")
res1: org.apache.spark.sql.DataFrame = []
scala> sql("DESC partitioned_table PARTITION (c='Us', d=1)").show(false)
org.apache.spark.sql.catalyst.parser.ParseException:
Unsupported SQL statement
```
- **After**
```scala
scala> sql("CREATE TABLE partitioned_table (a STRING, b INT) PARTITIONED BY (c STRING, d STRING)")
res0: org.apache.spark.sql.DataFrame = []
scala> sql("ALTER TABLE partitioned_table ADD PARTITION (c='Us', d=1)")
res1: org.apache.spark.sql.DataFrame = []
scala> sql("DESC partitioned_table PARTITION (c='Us', d=1)").show(false)
+-----------------------+---------+-------+
|col_name |data_type|comment|
+-----------------------+---------+-------+
|a |string |null |
|b |int |null |
|c |string |null |
|d |string |null |
|# Partition Information| | |
|# col_name |data_type|comment|
|c |string |null |
|d |string |null |
+-----------------------+---------+-------+
scala> sql("DESC EXTENDED partitioned_table PARTITION (c='Us', d=1)").show(100,false)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-------+
|col_name |data_type|comment|
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-------+
|a |string |null |
|b |int |null |
|c |string |null |
|d |string |null |
|# Partition Information | | |
|# col_name |data_type|comment|
|c |string |null |
|d |string |null |
| | | |
|Detailed Partition Information CatalogPartition(
Partition Values: [Us, 1]
Storage(Location: file:/Users/dhyun/SPARK-17612-DESC-PARTITION/spark-warehouse/partitioned_table/c=Us/d=1, InputFormat: org.apache.hadoop.mapred.TextInputFormat, OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, Serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Properties: [serialization.format=1])
Partition Parameters:{transient_lastDdlTime=1475001066})| | |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-------+
scala> sql("DESC FORMATTED partitioned_table PARTITION (c='Us', d=1)").show(100,false)
+--------------------------------+---------------------------------------------------------------------------------------+-------+
|col_name |data_type |comment|
+--------------------------------+---------------------------------------------------------------------------------------+-------+
|a |string |null |
|b |int |null |
|c |string |null |
|d |string |null |
|# Partition Information | | |
|# col_name |data_type |comment|
|c |string |null |
|d |string |null |
| | | |
|# Detailed Partition Information| | |
|Partition Value: |[Us, 1] | |
|Database: |default | |
|Table: |partitioned_table | |
|Location: |file:/Users/dhyun/SPARK-17612-DESC-PARTITION/spark-warehouse/partitioned_table/c=Us/d=1| |
|Partition Parameters: | | |
| transient_lastDdlTime |1475001066 | |
| | | |
|# Storage Information | | |
|SerDe Library: |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | |
|InputFormat: |org.apache.hadoop.mapred.TextInputFormat | |
|OutputFormat: |org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | |
|Compressed: |No | |
|Storage Desc Parameters: | | |
| serialization.format |1 | |
+--------------------------------+---------------------------------------------------------------------------------------+-------+
```
## How was this patch tested?
Pass the Jenkins tests with a new testcase.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #15168 from dongjoon-hyun/SPARK-17612.
|
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||