spark-instrumented-optimizer/sql/hive
Cheng Lian 2d2f607bfa [SPARK-13709][SQL] Initialize deserializer with both table and partition properties when reading partitioned tables
## What changes were proposed in this pull request?

When reading partitions of a partitioned Hive SerDe table, we only initializes the deserializer using partition properties. However, for SerDes like `AvroSerDe`, essential properties (e.g. Avro schema information) may be defined in table properties. We should merge both table properties and partition properties before initializing the deserializer.

Note that an individual partition may have different properties than the one defined in the table properties (e.g. partitions within a table can have different SerDes). Thus, for any property key defined in both partition and table properties, the value set in partition properties wins.

## How was this patch tested?

New test case added in `QueryPartitionSuite`.

Author: Cheng Lian <lian@databricks.com>

Closes #13865 from liancheng/spark-13709-partitioned-avro-table.
2016-06-23 23:11:46 -07:00
..
compatibility/src/test/scala/org/apache/spark/sql/hive/execution [SPARK-15539][SQL] DROP TABLE throw exception if table doesn't exist 2016-05-26 12:04:18 -07:00
src [SPARK-13709][SQL] Initialize deserializer with both table and partition properties when reading partitioned tables 2016-06-23 23:11:46 -07:00
pom.xml [SPARK-15290][BUILD] Move annotations, like @Since / @DeveloperApi, into spark-tags 2016-05-17 09:55:53 +01:00