def8bc5c96
### What changes were proposed in this pull request? This PR fixes an issue that `from_csv/to_csv` doesn't handle day-time intervals properly. `from_csv` throws exception if day-time interval types are given. ``` spark-sql> select from_csv("interval '1 2:3:4' day to second", "a interval day to second"); 21/07/03 04:39:13 ERROR SparkSQLDriver: Failed in [select from_csv("interval '1 2:3:4' day to second", "a interval day to second")] java.lang.Exception: Unsupported type: interval day to second at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedTypeError(QueryExecutionErrors.scala:775) at org.apache.spark.sql.catalyst.csv.UnivocityParser.makeConverter(UnivocityParser.scala:224) at org.apache.spark.sql.catalyst.csv.UnivocityParser.$anonfun$valueConverters$1(UnivocityParser.scala:134) ``` Also, `to_csv` doesn't handle day-time interval types properly though any exception is thrown. The result of `to_csv` for day-time interval types is not ANSI interval compliant form. ``` spark-sql> select to_csv(named_struct("a", interval '1 2:3:4' day to second)); 93784000000 ``` The result above should be `INTERVAL '1 02:03:04' DAY TO SECOND`. ### Why are the changes needed? Bug fix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests. Closes #33226 from sarutak/csv-dtinterval. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> |
||
---|---|---|
.. | ||
benchmarks | ||
src | ||
pom.xml |