6a576161ae
### What changes were proposed in this pull request? This PR aims to add a benchmark suite for nested predicate pushdown with parquet file: Performance comparison: Nested predicate pushdown disabled vs enabled, with the following queries scenarios: 1. When predicate pushed down, parquet reader are able to filter out all the row groups without loading them. 2. When predicate pushed down, parquet reader only loads one of the row groups. 3. When predicate pushed down, parquet reader can't filter out any row group in order to see if we introduce too much overhead or not when enabling nested predicate push down. ### Why are the changes needed? No benchmark exists today for nested fields predicate pushdown performance evaluation. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Benchmark runs and reporting result. Closes #28319 from JiJiTang/SPARK-31364. Authored-by: Jian Tang <jian_tang@apple.com> Signed-off-by: DB Tsai <d_tsai@apple.com>
22 lines
1.7 KiB
Plaintext
22 lines
1.7 KiB
Plaintext
OpenJDK 64-Bit Server VM 1.8.0_252-b09 on Mac OS X 10.14.6
|
|
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
|
|
Can skip all row groups: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
Without nested predicate Pushdown 30687 31552 NaN 3.4 292.7 1.0X
|
|
With nested predicate Pushdown 105 150 61 999.3 1.0 292.5X
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_252-b09 on Mac OS X 10.14.6
|
|
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
|
|
Can skip some row groups: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
Without nested predicate Pushdown 30505 31828 NaN 3.4 290.9 1.0X
|
|
With nested predicate Pushdown 3156 3215 77 33.2 30.1 9.7X
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_252-b09 on Mac OS X 10.14.6
|
|
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
|
|
Can skip no row groups: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
Without nested predicate Pushdown 34475 35302 NaN 3.0 328.8 1.0X
|
|
With nested predicate Pushdown 34003 34596 567 3.1 324.3 1.0X
|
|
|