spark-instrumented-optimizer/sql/core
Wenchen Fan fd2d55c991 [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
### What changes were proposed in this pull request?

When reading/writing datetime values that before the rebase switch day, from/to Avro/Parquet files, fail by default and ask users to set a config to explicitly do rebase or not.

### Why are the changes needed?

Rebase or not rebase have different behaviors and we should let users decide it explicitly. In most cases, users won't hit this exception as it only affects ancient datetime values.

### Does this PR introduce _any_ user-facing change?

Yes, now users will see an error when reading/writing dates before 1582-10-15 or timestamps before 1900-01-01 from/to Parquet/Avro files, with an error message to ask setting a config.

### How was this patch tested?

updated tests

Closes #28477 from cloud-fan/rebase.

Authored-by: Wenchen Fan <wenchen@databricks.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
2020-05-14 12:32:40 +09:00
..
benchmarks [SPARK-31630][SQL] Fix perf regression by skipping timestamps rebasing after some threshold 2020-05-05 14:11:53 +00:00
src [SPARK-31405][SQL] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files 2020-05-14 12:32:40 +09:00
v1.2/src [SPARK-31489][SQL] Fix pushing down filters with java.time.LocalDate values in ORC 2020-04-26 15:49:00 -07:00
v2.3/src [SPARK-31489][SQL] Fix pushing down filters with java.time.LocalDate values in ORC 2020-04-26 15:49:00 -07:00
pom.xml [SPARK-31272][SQL] Support DB2 Kerberos login in JDBC connector 2020-04-22 17:10:30 -07:00