ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/sql/core/benchmarks/DateTimeRebaseBenchmark-results.txt
Wenchen Fan f72220b8ab [SPARK-31606][SQL] Reduce the perf regression of vectorized parquet reader caused by datetime rebase 
### What changes were proposed in this pull request?

Push the rebase logic to the lower level of the parquet vectorized reader, to make the final code more vectorization-friendly.

### Why are the changes needed?

Parquet vectorized reader is carefully implemented, to make it more likely to be vectorized by the JVM. However, the newly added datetime rebase degrade the performance a lot, as it breaks vectorization, even if the datetime values don't need to rebase (this is very likely as dates before 1582 is rare).

### Does this PR introduce any user-facing change?

no

### How was this patch tested?

Run part of the `DateTimeRebaseBenchmark` locally. The results:
before this patch
```
[info] Load dates from parquet:                  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] after 1582, vec on, rebase off                     2677           2838         142         37.4          26.8       1.0X
[info] after 1582, vec on, rebase on                      3828           4331         805         26.1          38.3       0.7X
[info] before 1582, vec on, rebase off                    2903           2926          34         34.4          29.0       0.9X
[info] before 1582, vec on, rebase on                     4163           4197          38         24.0          41.6       0.6X

[info] Load timestamps from parquet:             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] after 1900, vec on, rebase off                     3537           3627         104         28.3          35.4       1.0X
[info] after 1900, vec on, rebase on                      6891           7010         105         14.5          68.9       0.5X
[info] before 1900, vec on, rebase off                    3692           3770          72         27.1          36.9       1.0X
[info] before 1900, vec on, rebase on                     7588           7610          30         13.2          75.9       0.5X
```

After this patch
```
[info] Load dates from parquet:                  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] after 1582, vec on, rebase off                     2758           2944         197         36.3          27.6       1.0X
[info] after 1582, vec on, rebase on                      2908           2966          51         34.4          29.1       0.9X
[info] before 1582, vec on, rebase off                    2840           2878          37         35.2          28.4       1.0X
[info] before 1582, vec on, rebase on                     3407           3433          24         29.4          34.1       0.8X

[info] Load timestamps from parquet:             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] after 1900, vec on, rebase off                     3861           4003         139         25.9          38.6       1.0X
[info] after 1900, vec on, rebase on                      4194           4283          77         23.8          41.9       0.9X
[info] before 1900, vec on, rebase off                    3849           3937          79         26.0          38.5       1.0X
[info] before 1900, vec on, rebase on                     7512           7546          55         13.3          75.1       0.5X
```

Date type is 30% faster if the values don't need to rebase, 20% faster if need to rebase.
Timestamp type is 60% faster if the values don't need to rebase, no difference if need to rebase.

Closes #28406 from cloud-fan/perf.

Lead-authored-by: Wenchen Fan <wenchen@databricks.com>
Co-authored-by: Maxim Gekk <max.gekk@gmail.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
2020-05-04 15:30:10 +09:00

95 lines
8.5 KiB
Plaintext
Raw Blame History

================================================================================================
Rebasing dates/timestamps in Parquet datasource
================================================================================================
OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 23088 23088 0 4.3 230.9 1.0X
before 1582, noop 10782 10782 0 9.3 107.8 2.1X
after 1582, rebase off 34821 34821 0 2.9 348.2 0.7X
after 1582, rebase on 35040 35040 0 2.9 350.4 0.7X
before 1582, rebase off 22151 22151 0 4.5 221.5 1.0X
before 1582, rebase on 24677 24677 0 4.1 246.8 0.9X
OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off, rebase off 13559 13650 79 7.4 135.6 1.0X
after 1582, vec off, rebase on 12942 12973 28 7.7 129.4 1.0X
after 1582, vec on, rebase off 3657 3689 29 27.3 36.6 3.7X
after 1582, vec on, rebase on 3859 3902 53 25.9 38.6 3.5X
before 1582, vec off, rebase off 12588 12607 17 7.9 125.9 1.1X
before 1582, vec off, rebase on 13396 13420 25 7.5 134.0 1.0X
before 1582, vec on, rebase off 3631 3650 19 27.5 36.3 3.7X
before 1582, vec on, rebase on 4706 4755 77 21.3 47.1 2.9X
OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1900, noop 2681 2681 0 37.3 26.8 1.0X
before 1900, noop 3051 3051 0 32.8 30.5 0.9X
after 1900, rebase off 16901 16901 0 5.9 169.0 0.2X
after 1900, rebase on 19725 19725 0 5.1 197.3 0.1X
before 1900, rebase off 16900 16900 0 5.9 169.0 0.2X
before 1900, rebase on 20381 20381 0 4.9 203.8 0.1X
OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1900, vec off, rebase off 15236 15291 62 6.6 152.4 1.0X
after 1900, vec off, rebase on 17832 18047 187 5.6 178.3 0.9X
after 1900, vec on, rebase off 4875 4901 31 20.5 48.7 3.1X
after 1900, vec on, rebase on 5354 5386 37 18.7 53.5 2.8X
before 1900, vec off, rebase off 15229 15338 108 6.6 152.3 1.0X
before 1900, vec off, rebase on 18626 18668 44 5.4 186.3 0.8X
before 1900, vec on, rebase off 4968 4975 6 20.1 49.7 3.1X
before 1900, vec on, rebase on 9913 9932 16 10.1 99.1 1.5X
================================================================================================
Rebasing dates/timestamps in ORC datasource
================================================================================================
OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, noop 22942 22942 0 4.4 229.4 1.0X
before 1582, noop 11035 11035 0 9.1 110.4 2.1X
after 1582 31341 31341 0 3.2 313.4 0.7X
before 1582 20376 20376 0 4.9 203.8 1.1X
OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off 10361 10378 29 9.7 103.6 1.0X
after 1582, vec on 3820 3828 11 26.2 38.2 2.7X
before 1582, vec off 10709 10720 13 9.3 107.1 1.0X
before 1582, vec on 4136 4153 15 24.2 41.4 2.5X
OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1900, noop 2888 2888 0 34.6 28.9 1.0X
before 1900, noop 2823 2823 0 35.4 28.2 1.0X
after 1900 19790 19790 0 5.1 197.9 0.1X
before 1900 20774 20774 0 4.8 207.7 0.1X
OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
after 1900, vec off 14649 14687 38 6.8 146.5 1.0X
after 1900, vec on 7850 7937 130 12.7 78.5 1.9X
before 1900, vec off 15354 15417 108 6.5 153.5 1.0X
before 1900, vec on 8382 8408 22 11.9 83.8 1.7X
Reference in a new issue View git blame Copy permalink