6a576161ae
### What changes were proposed in this pull request? This PR aims to add a benchmark suite for nested predicate pushdown with parquet file: Performance comparison: Nested predicate pushdown disabled vs enabled, with the following queries scenarios: 1. When predicate pushed down, parquet reader are able to filter out all the row groups without loading them. 2. When predicate pushed down, parquet reader only loads one of the row groups. 3. When predicate pushed down, parquet reader can't filter out any row group in order to see if we introduce too much overhead or not when enabling nested predicate push down. ### Why are the changes needed? No benchmark exists today for nested fields predicate pushdown performance evaluation. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Benchmark runs and reporting result. Closes #28319 from JiJiTang/SPARK-31364. Authored-by: Jian Tang <jian_tang@apple.com> Signed-off-by: DB Tsai <d_tsai@apple.com>
22 lines
1.7 KiB
Plaintext
22 lines
1.7 KiB
Plaintext
OpenJDK 64-Bit Server VM 11.0.2+9 on Mac OS X 10.14.6
|
|
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
|
|
Can skip all row groups: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
Without nested predicate Pushdown 34214 35752 NaN 3.1 326.3 1.0X
|
|
With nested predicate Pushdown 86 102 11 1216.2 0.8 396.8X
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.2+9 on Mac OS X 10.14.6
|
|
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
|
|
Can skip some row groups: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
Without nested predicate Pushdown 34211 35162 843 3.1 326.3 1.0X
|
|
With nested predicate Pushdown 3470 3514 36 30.2 33.1 9.9X
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.2+9 on Mac OS X 10.14.6
|
|
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
|
|
Can skip no row groups: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
Without nested predicate Pushdown 37533 37919 329 2.8 357.9 1.0X
|
|
With nested predicate Pushdown 37876 39132 536 2.8 361.2 1.0X
|
|
|