ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-26900][SQL] Simplify truncation to quarter of year

Browse source 
## What changes were proposed in this pull request?

In the PR, I propose to simplify timestamp truncation to quarter of year by using *java.time* API directly. The `LocalDate` instance can be truncation to quarter timestamp via adjusting by chrono field `IsoFields.DAY_OF_QUARTER`.

## How was this patch tested?

This was checked by existing test suite - `DateTimeUtilsSuite`.

Closes #23808 from MaxGekk/date-quarter-of-year.

Authored-by: Maxim Gekk <max.gekk@gmail.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
This commit is contained in:
Maxim Gekk 2019-02-20 08:55:08 -06:00 committed by Sean Owen
parent 74e9e1c192
commit 331ac60f28
2 changed files with 6 additions and 14 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
View file

@ -719,17 +719,9 @@ object DateTimeUtils {
daysToMillis(prevMonday, timeZone)
case TRUNC_TO_QUARTER =>
val dDays = millisToDays(millis, timeZone)
val month = getQuarter(dDays) match {
case 1 => Month.JANUARY
case 2 => Month.APRIL
case 3 => Month.JULY
case 4 => Month.OCTOBER
}
millis = daysToMillis(truncDate(dDays, TRUNC_TO_MONTH), timeZone)
val instant = Instant.ofEpochMilli(millis)
val localDateTime = LocalDateTime.ofInstant(instant, timeZone.toZoneId)
val truncated = localDateTime.withMonth(month.getValue)
truncated.atZone(timeZone.toZoneId).toInstant.toEpochMilli
val daysOfQuarter = LocalDate.ofEpochDay(dDays)
.`with`(IsoFields.DAY_OF_QUARTER, 1L).toEpochDay.toInt
daysToMillis(daysOfQuarter, timeZone)
case _ =>
// caller make sure that this should never be reached
sys.error(s"Invalid trunc level: $level")

6
sql/core/benchmarks/DateTimeBenchmark-results.txt
View file

@ -324,12 +324,12 @@ date_trunc WEEK: Best/Avg Time(ms) Rate(M/s) Per Ro
date_trunc WEEK wholestage off 794 / 797 12.6 79.4 1.0X
date_trunc WEEK wholestage on 754 / 761 13.3 75.4 1.1X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc QUARTER: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
date_trunc QUARTER wholestage off 5261 / 5271 1.9 526.1 1.0X
date_trunc QUARTER wholestage on 5145 / 5151 1.9 514.5 1.0X
date_trunc QUARTER wholestage off 1465 / 1467 6.8 146.5 1.0X
date_trunc QUARTER wholestage on 1419 / 1423 7.0 141.9 1.0X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz