37d2e037ed
### What changes were proposed in this pull request? Extracting millennium, century, decade, millisecond, microsecond and epoch from datetime is neither ANSI standard nor quite common in modern SQL platforms. Most of the systems listing below does not support these except PostgreSQL and redshift. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions050.htm https://prestodb.io/docs/current/functions/datetime.html https://docs.cloudera.com/documentation/enterprise/5-8-x/topics/impala_datetime_functions.html https://docs.snowflake.com/en/sql-reference/functions-date-time.html#label-supported-date-time-parts https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT This PR removes these extract fields support from extract function for date and timestamp values `isoyear` is PostgreSQL specific but `yearofweek` is more commonly used across platforms `isodow` is PostgreSQL specific but `iso` as a suffix is more commonly used across platforms so, `dow_iso` and `dayofweek_iso` is used to replace it. For historical reasons, we have [`dayofweek`, `dow`] implemented for representing a non-ISO day-of-week and a newly added `isodow` from PostgreSQL for ISO day-of-week. Many other systems only have one week-numbering system support and use either full names or abbreviations. Things in spark become a little bit complicated. 1. because of the existence of `isodow`, so we need to add iso-prefix to `dayofweek` to make a pair for it too. [`dayofweek`, `isodayofweek`, `dow` and `isodow`] 2. because there are rare `iso`-prefixed systems and more systems choose `iso`-suffixed way, so we may result in [`dayofweek`, `dayofweekiso`, `dow`, `dowiso`] 3. `dayofweekiso` looks nice and has use cases in the platforms listed above, e.g. snowflake, but `dowiso` looks weird and no use cases found. 4. with a discussion the community，we have agreed with an underscore before `iso` may look much better because `isodow` is new and there is no standard for `iso` kind of things, so this may be good for us to make it simple and clear for end-users if they are well documented too. Thus, we finally result in [`dayofweek`, `dow`] for Non-ISO day-of-week system and [`dayofweek_iso`, `dow_iso`] for ISO system ### Why are the changes needed? Remove some nonstandard and uncommon features as we can add them back if necessary ### Does this PR introduce any user-facing change? NO, we should target this to 3.0.0 and these are added during 3.0.0 ### How was this patch tested? Remove unused tests Closes #28284 from yaooqinn/SPARK-31507. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
105 lines
11 KiB
Plaintext
105 lines
11 KiB
Plaintext
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.15.4
|
|
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
|
|
Invoke extract for timestamp: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
cast to timestamp 333 345 13 30.0 33.3 1.0X
|
|
YEAR of timestamp 826 840 16 12.1 82.6 0.4X
|
|
YEAROFWEEK of timestamp 1161 1168 7 8.6 116.1 0.3X
|
|
QUARTER of timestamp 961 966 6 10.4 96.1 0.3X
|
|
MONTH of timestamp 831 838 6 12.0 83.1 0.4X
|
|
WEEK of timestamp 1155 1159 4 8.7 115.5 0.3X
|
|
DAY of timestamp 822 828 6 12.2 82.2 0.4X
|
|
DAYOFWEEK of timestamp 968 972 6 10.3 96.8 0.3X
|
|
DOW of timestamp 960 966 9 10.4 96.0 0.3X
|
|
DOW_ISO of timestamp 961 967 9 10.4 96.1 0.3X
|
|
DAYOFWEEK_ISO of timestamp 965 968 5 10.4 96.5 0.3X
|
|
DOY of timestamp 864 873 9 11.6 86.4 0.4X
|
|
HOUR of timestamp 631 647 14 15.9 63.1 0.5X
|
|
MINUTE of timestamp 638 651 12 15.7 63.8 0.5X
|
|
SECOND of timestamp 758 760 2 13.2 75.8 0.4X
|
|
|
|
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.15.4
|
|
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
|
|
Invoke date_part for timestamp: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
cast to timestamp 301 307 8 33.2 30.1 1.0X
|
|
YEAR of timestamp 797 805 7 12.6 79.7 0.4X
|
|
YEAROFWEEK of timestamp 1147 1156 8 8.7 114.7 0.3X
|
|
QUARTER of timestamp 965 974 9 10.4 96.5 0.3X
|
|
MONTH of timestamp 801 815 17 12.5 80.1 0.4X
|
|
WEEK of timestamp 1133 1157 21 8.8 113.3 0.3X
|
|
DAY of timestamp 825 828 4 12.1 82.5 0.4X
|
|
DAYOFWEEK of timestamp 929 937 8 10.8 92.9 0.3X
|
|
DOW of timestamp 933 940 11 10.7 93.3 0.3X
|
|
DOW_ISO of timestamp 944 954 11 10.6 94.4 0.3X
|
|
DAYOFWEEK_ISO of timestamp 948 965 21 10.5 94.8 0.3X
|
|
DOY of timestamp 894 898 6 11.2 89.4 0.3X
|
|
HOUR of timestamp 678 683 6 14.7 67.8 0.4X
|
|
MINUTE of timestamp 669 670 2 15.0 66.9 0.5X
|
|
SECOND of timestamp 762 771 12 13.1 76.2 0.4X
|
|
|
|
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.15.4
|
|
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
|
|
Invoke extract for date: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
cast to date 737 751 23 13.6 73.7 1.0X
|
|
YEAR of date 804 811 11 12.4 80.4 0.9X
|
|
YEAROFWEEK of date 1151 1155 4 8.7 115.1 0.6X
|
|
QUARTER of date 964 987 26 10.4 96.4 0.8X
|
|
MONTH of date 831 835 6 12.0 83.1 0.9X
|
|
WEEK of date 1162 1163 2 8.6 116.2 0.6X
|
|
DAY of date 826 833 7 12.1 82.6 0.9X
|
|
DAYOFWEEK of date 984 1003 26 10.2 98.4 0.7X
|
|
DOW of date 995 997 2 10.0 99.5 0.7X
|
|
DOW_ISO of date 985 992 8 10.2 98.5 0.7X
|
|
DAYOFWEEK_ISO of date 985 1001 19 10.1 98.5 0.7X
|
|
DOY of date 884 890 8 11.3 88.4 0.8X
|
|
HOUR of date 1630 1640 13 6.1 163.0 0.5X
|
|
MINUTE of date 1631 1636 6 6.1 163.1 0.5X
|
|
SECOND of date 1737 1739 2 5.8 173.7 0.4X
|
|
|
|
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.15.4
|
|
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
|
|
Invoke date_part for date: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
cast to date 746 753 8 13.4 74.6 1.0X
|
|
YEAR of date 841 843 1 11.9 84.1 0.9X
|
|
YEAROFWEEK of date 1169 1191 20 8.6 116.9 0.6X
|
|
QUARTER of date 1008 1012 4 9.9 100.8 0.7X
|
|
MONTH of date 838 844 6 11.9 83.8 0.9X
|
|
WEEK of date 1179 1182 3 8.5 117.9 0.6X
|
|
DAY of date 849 850 1 11.8 84.9 0.9X
|
|
DAYOFWEEK of date 984 988 4 10.2 98.4 0.8X
|
|
DOW of date 986 994 7 10.1 98.6 0.8X
|
|
DOW_ISO of date 974 992 17 10.3 97.4 0.8X
|
|
DAYOFWEEK_ISO of date 979 994 14 10.2 97.9 0.8X
|
|
DOY of date 874 878 3 11.4 87.4 0.9X
|
|
HOUR of date 1644 1649 6 6.1 164.4 0.5X
|
|
MINUTE of date 1610 1626 15 6.2 161.0 0.5X
|
|
SECOND of date 1768 1789 30 5.7 176.8 0.4X
|
|
|
|
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.15.4
|
|
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
|
|
Invoke extract for interval: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
cast to interval 989 1003 13 10.1 98.9 1.0X
|
|
YEAR of interval 980 999 16 10.2 98.0 1.0X
|
|
MONTH of interval 988 1000 15 10.1 98.8 1.0X
|
|
DAY of interval 970 983 11 10.3 97.0 1.0X
|
|
HOUR of interval 989 1000 13 10.1 98.9 1.0X
|
|
MINUTE of interval 1019 1034 13 9.8 101.9 1.0X
|
|
SECOND of interval 1084 1095 10 9.2 108.4 0.9X
|
|
|
|
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.15.4
|
|
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
|
|
Invoke date_part for interval: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
cast to interval 955 969 15 10.5 95.5 1.0X
|
|
YEAR of interval 1004 1007 3 10.0 100.4 1.0X
|
|
MONTH of interval 989 990 2 10.1 98.9 1.0X
|
|
DAY of interval 1006 1011 6 9.9 100.6 0.9X
|
|
HOUR of interval 986 997 10 10.1 98.6 1.0X
|
|
MINUTE of interval 1004 1014 15 10.0 100.4 1.0X
|
|
SECOND of interval 1083 1088 6 9.2 108.3 0.9X
|
|
|