890baaca50
## What changes were proposed in this pull request? The current implementation of "CREATE TEMPORARY TABLE USING datasource..." is NOT creating any intermediate temporary data directory like temporary HDFS folder, instead, it only stores a SQL string in memory. Probably we should use "TEMPORARY VIEW" instead. This PR assumes a temporary table has to link with some temporary intermediate data. It follows the definition of temporary table like this (from [hortonworks doc](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/temp-tables.html)): > A temporary table is a convenient way for an application to automatically manage intermediate data generated during a complex query **Example**: ``` scala> spark.sql("CREATE temporary view my_tab7 (c1: String, c2: String) USING org.apache.spark.sql.execution.datasources.csv.CSVFileFormat OPTIONS (PATH '/Users/seanzhong/csv/cars.csv')") scala> spark.sql("select c1, c2 from my_tab7").show() +----+-----+ | c1| c2| +----+-----+ |year| make| |2012|Tesla| ... ``` It NOW prints a **deprecation warning** if "CREATE TEMPORARY TABLE USING..." is used. ``` scala> spark.sql("CREATE temporary table my_tab7 (c1: String, c2: String) USING org.apache.spark.sql.execution.datasources.csv.CSVFileFormat OPTIONS (PATH '/Users/seanzhong/csv/cars.csv')") 16/05/31 10:39:27 WARN SparkStrategies$DDLStrategy: CREATE TEMPORARY TABLE tableName USING... is deprecated, please use CREATE TEMPORARY VIEW viewName USING... instead ``` ## How was this patch tested? Unit test. Author: Sean Zhong <seanzhong@databricks.com> Closes #13414 from clockfly/create_temp_view_using. |
||
---|---|---|
.. | ||
avro | ||
gen-java/org/apache/spark/sql/execution/datasources/parquet/test/avro | ||
java/test/org/apache/spark/sql | ||
resources | ||
scala/org/apache/spark/sql | ||
scripts | ||
thrift | ||
README.md |
Notes for Parquet compatibility tests
The following directories and files are used for Parquet compatibility tests:
.
├── README.md # This file
├── avro
│ ├── *.avdl # Testing Avro IDL(s)
│ └── *.avpr # !! NO TOUCH !! Protocol files generated from Avro IDL(s)
├── gen-java # !! NO TOUCH !! Generated Java code
├── scripts
│ ├── gen-avro.sh # Script used to generate Java code for Avro
│ └── gen-thrift.sh # Script used to generate Java code for Thrift
└── thrift
└── *.thrift # Testing Thrift schema(s)
To avoid code generation during build time, Java code generated from testing Thrift schema and Avro IDL are also checked in.
When updating the testing Thrift schema and Avro IDL, please run gen-avro.sh
and gen-thrift.sh
accordingly to update generated Java code.
Prerequisites
Please ensure avro-tools
and thrift
are installed. You may install these two on Mac OS X via:
$ brew install thrift avro-tools