bef5828e12
### What changes were proposed in this pull request? Skip timestamps rebasing after a global threshold when there is no difference between Julian and Gregorian calendars. This allows to avoid checking hash maps of switch points, and fixes perf regressions in `toJavaTimestamp()` and `fromJavaTimestamp()`. ### Why are the changes needed? The changes fix perf regressions of conversions to/from external type `java.sql.Timestamp`. Before (see the PR's results https://github.com/apache/spark/pull/28440): ``` ================================================================================================ Conversion from/to external types ================================================================================================ OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws Intel(R) Xeon(R) CPU E5-2670 v2 2.50GHz To/from Java's date-time: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ From java.sql.Timestamp 376 388 10 13.3 75.2 1.1X Collect java.sql.Timestamp 1878 1937 64 2.7 375.6 0.2X ``` After: ``` ================================================================================================ Conversion from/to external types ================================================================================================ OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws Intel(R) Xeon(R) CPU E5-2670 v2 2.50GHz To/from Java's date-time: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ From java.sql.Timestamp 249 264 24 20.1 49.8 1.7X Collect java.sql.Timestamp 1503 1523 24 3.3 300.5 0.3X ``` Perf improvements in average of: 1. From java.sql.Timestamp is ~ 34% 2. To java.sql.Timestamps is ~16% ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By existing test suites `DateTimeUtilsSuite` and `RebaseDateTimeSuite`. Closes #28441 from MaxGekk/opt-rebase-common-threshold. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> |
||
---|---|---|
.. | ||
AggregateBenchmark-jdk11-results.txt | ||
AggregateBenchmark-results.txt | ||
BloomFilterBenchmark-jdk11-results.txt | ||
BloomFilterBenchmark-results.txt | ||
BuiltInDataSourceWriteBenchmark-jdk11-results.txt | ||
BuiltInDataSourceWriteBenchmark-results.txt | ||
ColumnarBatchBenchmark-jdk11-results.txt | ||
ColumnarBatchBenchmark-results.txt | ||
CompressionSchemeBenchmark-jdk11-results.txt | ||
CompressionSchemeBenchmark-results.txt | ||
CSVBenchmark-jdk11-results.txt | ||
CSVBenchmark-results.txt | ||
DatasetBenchmark-jdk11-results.txt | ||
DatasetBenchmark-results.txt | ||
DataSourceReadBenchmark-jdk11-results.txt | ||
DataSourceReadBenchmark-results.txt | ||
DateTimeBenchmark-jdk11-results.txt | ||
DateTimeBenchmark-results.txt | ||
DateTimeRebaseBenchmark-jdk11-results.txt | ||
DateTimeRebaseBenchmark-results.txt | ||
ExternalAppendOnlyUnsafeRowArrayBenchmark-jdk11-results.txt | ||
ExternalAppendOnlyUnsafeRowArrayBenchmark-results.txt | ||
ExtractBenchmark-jdk11-results.txt | ||
ExtractBenchmark-results.txt | ||
FilterPushdownBenchmark-jdk11-results.txt | ||
FilterPushdownBenchmark-results.txt | ||
HashedRelationMetricsBenchmark-jdk11-results.txt | ||
HashedRelationMetricsBenchmark-results.txt | ||
InExpressionBenchmark-jdk11-results.txt | ||
InExpressionBenchmark-results.txt | ||
IntervalBenchmark-jdk11-results.txt | ||
IntervalBenchmark-results.txt | ||
JoinBenchmark-jdk11-results.txt | ||
JoinBenchmark-results.txt | ||
JsonBenchmark-jdk11-results.txt | ||
JsonBenchmark-results.txt | ||
MakeDateTimeBenchmark-jdk11-results.txt | ||
MakeDateTimeBenchmark-results.txt | ||
MetricsAggregationBenchmark-jdk11-results.txt | ||
MetricsAggregationBenchmark-results.txt | ||
MiscBenchmark-jdk11-results.txt | ||
MiscBenchmark-results.txt | ||
OrcNestedSchemaPruningBenchmark-jdk11-results.txt | ||
OrcNestedSchemaPruningBenchmark-results.txt | ||
OrcV2NestedSchemaPruningBenchmark-jdk11-results.txt | ||
OrcV2NestedSchemaPruningBenchmark-results.txt | ||
ParquetNestedPredicatePushDownBenchmark-jdk11-results.txt | ||
ParquetNestedPredicatePushDownBenchmark-results.txt | ||
ParquetNestedSchemaPruningBenchmark-jdk11-results.txt | ||
ParquetNestedSchemaPruningBenchmark-results.txt | ||
PrimitiveArrayBenchmark-jdk11-results.txt | ||
PrimitiveArrayBenchmark-results.txt | ||
RangeBenchmark-jdk11-results.txt | ||
RangeBenchmark-results.txt | ||
SortBenchmark-jdk11-results.txt | ||
SortBenchmark-results.txt | ||
TPCDSQueryBenchmark-jdk11-results.txt | ||
TPCDSQueryBenchmark-results.txt | ||
UDFBenchmark-jdk11-results.txt | ||
UDFBenchmark-results.txt | ||
UnsafeArrayDataBenchmark-jdk11-results.txt | ||
UnsafeArrayDataBenchmark-results.txt | ||
WideSchemaBenchmark-jdk11-results.txt | ||
WideSchemaBenchmark-results.txt | ||
WideTableBenchmark-jdk11-results.txt | ||
WideTableBenchmark-results.txt |