ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36775][DOCS] Add documentation for ANSI store assignment rules

Browse source 
### What changes were proposed in this pull request?

Add documentation for ANSI store assignment rules for
- the valid source/target type combinations
- runtime error will happen on numberic overflow

### Why are the changes needed?

Better  docs
### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Build docs and preview:
![image](https://user-images.githubusercontent.com/1097932/133554600-8c80c0a9-8753-4c01-94d0-994d8082e319.png)

Closes #34014 from gengliangwang/addStoreAssignDoc.

Authored-by: Gengliang Wang <gengliang@apache.org>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This commit is contained in:
Gengliang Wang 2021-09-16 15:50:40 +08:00 committed by Wenchen Fan
parent c217797297
commit ff7705ad2a
1 changed files with 27 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

29
docs/sql-ref-ansi-compliance.md
View file

@ -78,7 +78,7 @@ The `CAST` clause of Spark ANSI mode follows the syntax rules of section 6.13 "c
* MapType => String
* StructType => String
The valid combinations of target data type and source data type in a `CAST` expression are given by the following table.
The valid combinations of source and target data type in a `CAST` expression are given by the following table.
“Y” indicates that the combination is syntactically valid without restriction and “N” indicates that the combination is not valid.
| Source\Target | Numeric | String | Date | Timestamp | Interval | Boolean | Binary | Array | Map | Struct |
@ -156,7 +156,32 @@ SELECT * FROM t;
```
### Store assignment
As mentioned at the beginning, when `spark.sql.storeAssignmentPolicy` is set to `ANSI`(which is the default value), Spark SQL complies with the ANSI store assignment rules. During table insertion, Spark will throw exception on numeric value overflow or the source value can't be stored as the target type.
As mentioned at the beginning, when `spark.sql.storeAssignmentPolicy` is set to `ANSI`(which is the default value), Spark SQL complies with the ANSI store assignment rules on table insertions. The valid combinations of source and target data type in table insertions are given by the following table.
| Source\Target | Numeric | String | Date | Timestamp | Interval | Boolean | Binary | Array | Map | Struct |
|:-------------:|:-------:|:------:|:----:|:---------:|:--------:|:-------:|:------:|:-----:|:---:|:------:|
| Numeric | Y | Y | N | N | N | N | N | N | N | N |
| String | N | Y | N | N | N | N | N | N | N | N |
| Date | N | Y | Y | Y | N | N | N | N | N | N |
| Timestamp | N | Y | Y | Y | N | N | N | N | N | N |
| Interval | N | Y | N | N | N* | N | N | N | N | N |
| Boolean | N | Y | N | N | N | Y | N | N | N | N |
| Binary | N | Y | N | N | N | N | Y | N | N | N |
| Array | N | N | N | N | N | N | N | Y** | N | N |
| Map | N | N | N | N | N | N | N | N | Y** | N |
| Struct | N | N | N | N | N | N | N | N | N | Y** |
\* Spark doesn't support interval type table column.
\*\* For Array/Map/Struct types, the data type check rule applies recursively to its component elements.
During table insertion, Spark will throw exception on numeric value overflow.
```sql
CREATE TABLE test(i INT);
INSERT INTO test VALUES (2147483648L);
java.lang.ArithmeticException: Casting 2147483648 to int causes overflow
```
### Type coercion
#### Type Promotion and Precedence