History

hyukjinkwon 6d7ebf2f9f [SPARK-22165][SQL] Fixes type conflicts between double, long, decimals, dates and timestamps in partition column ## What changes were proposed in this pull request? This PR proposes to add a rule that re-uses `TypeCoercion.findWiderCommonType` when resolving type conflicts in partition values. Currently, this uses numeric precedence-like comparison; therefore, it looks introducing failures for type conflicts between timestamps, dates and decimals, please see: ```scala private val upCastingOrder: Seq[DataType] = Seq(NullType, IntegerType, LongType, FloatType, DoubleType, StringType) ... literals.map(_.dataType).maxBy(upCastingOrder.indexOf(_)) ``` The codes below: ```scala val df = Seq((1, "2015-01-01"), (2, "2016-01-01 00:00:00")).toDF("i", "ts") df.write.format("parquet").partitionBy("ts").save("/tmp/foo") spark.read.load("/tmp/foo").printSchema() val df = Seq((1, "1"), (2, "1" * 30)).toDF("i", "decimal") df.write.format("parquet").partitionBy("decimal").save("/tmp/bar") spark.read.load("/tmp/bar").printSchema() ``` produces output as below: Before ``` root \|-- i: integer (nullable = true) \|-- ts: date (nullable = true) root \|-- i: integer (nullable = true) \|-- decimal: integer (nullable = true) ``` After ``` root \|-- i: integer (nullable = true) \|-- ts: timestamp (nullable = true) root \|-- i: integer (nullable = true) \|-- decimal: decimal(30,0) (nullable = true) ``` ### Type coercion table: This PR proposes the type conflict resolusion as below: Before \|InputA \ InputB\|`NullType`\|`IntegerType`\|`LongType`\|`DecimalType(38,0)`\|`DoubleType`\|`DateType`\|`TimestampType`\|`StringType`\| \|------------------------\|----------\|----------\|----------\|----------\|----------\|----------\|----------\|----------\| \|`NullType`\|`StringType`\|`IntegerType`\|`LongType`\|`StringType`\|`DoubleType`\|`StringType`\|`StringType`\|`StringType`\| \|`IntegerType`\|`IntegerType`\|`IntegerType`\|`LongType`\|`IntegerType`\|`DoubleType`\|`IntegerType`\|`IntegerType`\|`StringType`\| \|`LongType`\|`LongType`\|`LongType`\|`LongType`\|`LongType`\|`DoubleType`\|`LongType`\|`LongType`\|`StringType`\| \|`DecimalType(38,0)`\|`StringType`\|`IntegerType`\|`LongType`\|`DecimalType(38,0)`\|`DoubleType`\|`DecimalType(38,0)`\|`DecimalType(38,0)`\|`StringType`\| \|`DoubleType`\|`DoubleType`\|`DoubleType`\|`DoubleType`\|`DoubleType`\|`DoubleType`\|`DoubleType`\|`DoubleType`\|`StringType`\| \|`DateType`\|`StringType`\|`IntegerType`\|`LongType`\|`DateType`\|`DoubleType`\|`DateType`\|`DateType`\|`StringType`\| \|`TimestampType`\|`StringType`\|`IntegerType`\|`LongType`\|`TimestampType`\|`DoubleType`\|`TimestampType`\|`TimestampType`\|`StringType`\| \|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\| After \|InputA \ InputB\|`NullType`\|`IntegerType`\|`LongType`\|`DecimalType(38,0)`\|`DoubleType`\|`DateType`\|`TimestampType`\|`StringType`\| \|------------------------\|----------\|----------\|----------\|----------\|----------\|----------\|----------\|----------\| \|`NullType`\|`NullType`\|`IntegerType`\|`LongType`\|`DecimalType(38,0)`\|`DoubleType`\|`DateType`\|`TimestampType`\|`StringType`\| \|`IntegerType`\|`IntegerType`\|`IntegerType`\|`LongType`\|`DecimalType(38,0)`\|`DoubleType`\|`StringType`\|`StringType`\|`StringType`\| \|`LongType`\|`LongType`\|`LongType`\|`LongType`\|`DecimalType(38,0)`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\| \|`DecimalType(38,0)`\|`DecimalType(38,0)`\|`DecimalType(38,0)`\|`DecimalType(38,0)`\|`DecimalType(38,0)`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\| \|`DoubleType`\|`DoubleType`\|`DoubleType`\|`StringType`\|`StringType`\|`DoubleType`\|`StringType`\|`StringType`\|`StringType`\| \|`DateType`\|`DateType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`DateType`\|`TimestampType`\|`StringType`\| \|`TimestampType`\|`TimestampType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`TimestampType`\|`TimestampType`\|`StringType`\| \|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\|`StringType`\| This was produced by: ```scala test("Print out chart") { val supportedTypes: Seq[DataType] = Seq( NullType, IntegerType, LongType, DecimalType(38, 0), DoubleType, DateType, TimestampType, StringType) // Old type conflict resolution: val upCastingOrder: Seq[DataType] = Seq(NullType, IntegerType, LongType, FloatType, DoubleType, StringType) def oldResolveTypeConflicts(dataTypes: Seq[DataType]): DataType = { val topType = dataTypes.maxBy(upCastingOrder.indexOf(_)) if (topType == NullType) StringType else topType } println(s"\|InputA \\ InputB\|${supportedTypes.map(dt => s"`${dt.toString}`").mkString("\|")}\|") println(s"\|------------------------\|${supportedTypes.map(_ => "----------").mkString("\|")}\|") supportedTypes.foreach { inputA => val types = supportedTypes.map(inputB => oldResolveTypeConflicts(Seq(inputA, inputB))) println(s"\|`$inputA`\|${types.map(dt => s"`${dt.toString}`").mkString("\|")}\|") } // New type conflict resolution: def newResolveTypeConflicts(dataTypes: Seq[DataType]): DataType = { dataTypes.fold[DataType](NullType)(findWiderTypeForPartitionColumn) } println(s"\|InputA \\ InputB\|${supportedTypes.map(dt => s"`${dt.toString}`").mkString("\|")}\|") println(s"\|------------------------\|${supportedTypes.map(_ => "----------").mkString("\|")}\|") supportedTypes.foreach { inputA => val types = supportedTypes.map(inputB => newResolveTypeConflicts(Seq(inputA, inputB))) println(s"\|`$inputA`\|${types.map(dt => s"`${dt.toString}`").mkString("\|")}\|") } } ``` ## How was this patch tested? Unit tests added in `ParquetPartitionDiscoverySuite`. Author: hyukjinkwon <gurwls223@gmail.com> Closes #19389 from HyukjinKwon/partition-type-coercion.		2017-11-21 20:53:38 +01:00
..
catalyst	[SPARK-22165][SQL] Fixes type conflicts between double, long, decimals, dates and timestamps in partition column	2017-11-21 20:53:38 +01:00
core	[SPARK-22165][SQL] Fixes type conflicts between double, long, decimals, dates and timestamps in partition column	2017-11-21 20:53:38 +01:00
hive	[SPARK-22557][TEST] Use ThreadSignaler explicitly	2017-11-20 13:32:01 +09:00
hive-thriftserver	[SPARK-22487][SQL][FOLLOWUP] still keep spark.sql.hive.version	2017-11-13 13:10:13 -08:00
create-docs.sh	[MINOR][DOCS] Minor doc fixes related with doc build and uses script dir in SQL doc gen script	2017-08-26 13:56:24 +09:00
gen-sql-markdown.py	[SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples and arguments separately, and note/since in SQL built-in function documentation	2017-08-05 10:10:56 -07:00
mkdocs.yml	[SPARK-21485][SQL][DOCS] Spark SQL documentation generation for built-in functions	2017-07-26 09:38:51 -07:00
README.md	[SPARK-21485][SQL][DOCS] Spark SQL documentation generation for built-in functions	2017-07-26 09:38:51 -07:00

README.md

Spark SQL

This module provides support for executing relational queries expressed in either SQL or the DataFrame/Dataset API.

Spark SQL is broken up into four subprojects:

Catalyst (sql/catalyst) - An implementation-agnostic framework for manipulating trees of relational operators and expressions.
Execution (sql/core) - A query planner / execution engine for translating Catalyst's logical query plans into Spark RDDs. This component also includes a new public interface, SQLContext, that allows users to execute SQL or LINQ statements against existing RDDs and Parquet files.
Hive Support (sql/hive) - Includes an extension of SQLContext called HiveContext that allows users to write queries using a subset of HiveQL and access data from a Hive Metastore using Hive SerDes. There are also wrappers that allows users to run queries that include Hive UDFs, UDAFs, and UDTFs.
HiveServer and CLI support (sql/hive-thriftserver) - Includes support for the SQL CLI (bin/spark-sql) and a HiveServer2 (for JDBC/ODBC) compatible server.

Running sql/create-docs.sh generates SQL documentation for built-in functions under sql/site.