f5118f81e3
### What changes were proposed in this pull request?
In the PR, I propose to replace `.collect()`, `.count()` and `.foreach(_ => ())` in SQL benchmarks and use the `NoOp` datasource. I added an implicit class to `SqlBasedBenchmark` with the `.noop()` method. It can be used in benchmark like: `ds.noop()`. The last one is unfolded to `ds.write.format("noop").mode(Overwrite).save()`.
### Why are the changes needed?
To avoid additional overhead that `collect()` (and other actions) has. For example, `.collect()` has to convert values according to external types and pull data to the driver. This can hide actual performance regressions or improvements of benchmarked operations.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Re-run all modified benchmarks using Amazon EC2.
| Item | Description |
| ---- | ----|
| Region | us-west-2 (Oregon) |
| Instance | r3.xlarge (spot instance) |
| AMI | ami-06f2f779464715dc5 (ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1) |
| Java | OpenJDK8/10 |
- Run `TPCDSQueryBenchmark` using instructions from the PR #26049
```
# `spark-tpcds-datagen` needs this. (JDK8)
$ git clone https://github.com/apache/spark.git -b branch-2.4 --depth 1 spark-2.4
$ export SPARK_HOME=$PWD
$ ./build/mvn clean package -DskipTests
# Generate data. (JDK8)
$ git clone gitgithub.com:maropu/spark-tpcds-datagen.git
$ cd spark-tpcds-datagen/
$ build/mvn clean package
$ mkdir -p /data/tpcds
$ ./bin/dsdgen --output-location /data/tpcds/s1 // This need `Spark 2.4`
```
- Other benchmarks ran by the script:
```
#!/usr/bin/env python3
import os
from sparktestsupport.shellutils import run_cmd
benchmarks = [
['sql/test', 'org.apache.spark.sql.execution.benchmark.AggregateBenchmark'],
['avro/test', 'org.apache.spark.sql.execution.benchmark.AvroReadBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.BloomFilterBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.DataSourceReadBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.DateTimeBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.ExtractBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.FilterPushdownBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.InExpressionBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.IntervalBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.JoinBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.MakeDateTimeBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.MiscBenchmark'],
['hive/test', 'org.apache.spark.sql.execution.benchmark.ObjectHashAggregateExecBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.OrcNestedSchemaPruningBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.OrcV2NestedSchemaPruningBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.ParquetNestedSchemaPruningBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.RangeBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.UDFBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.WideSchemaBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.benchmark.WideTableBenchmark'],
['hive/test', 'org.apache.spark.sql.hive.orc.OrcReadBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.datasources.csv.CSVBenchmark'],
['sql/test', 'org.apache.spark.sql.execution.datasources.json.JsonBenchmark']
]
print('Set SPARK_GENERATE_BENCHMARK_FILES=1')
os.environ['SPARK_GENERATE_BENCHMARK_FILES'] = '1'
for b in benchmarks:
print("Run benchmark: %s" % b[1])
run_cmd(['build/sbt', '%s:runMain %s' % (b[0], b[1])])
```
Closes #27078 from MaxGekk/noop-in-benchmarks.
Lead-authored-by: Maxim Gekk <max.gekk@gmail.com>
Co-authored-by: Maxim Gekk <maxim.gekk@databricks.com>
Co-authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
144 lines
11 KiB
Plaintext
144 lines
11 KiB
Plaintext
================================================================================================
|
|
aggregate without grouping
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
agg w/o group: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
agg w/o group wholestage off 49902 52257 NaN 42.0 23.8 1.0X
|
|
agg w/o group wholestage on 1162 1171 10 1805.2 0.6 43.0X
|
|
|
|
|
|
================================================================================================
|
|
stat functions
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
stddev: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
stddev wholestage off 8203 8243 56 12.8 78.2 1.0X
|
|
stddev wholestage on 1287 1303 10 81.5 12.3 6.4X
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
kurtosis: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
kurtosis wholestage off 39557 39919 511 2.7 377.2 1.0X
|
|
kurtosis wholestage on 1398 1476 138 75.0 13.3 28.3X
|
|
|
|
|
|
================================================================================================
|
|
aggregate with linear keys
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
Aggregate w keys: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
codegen = F 11236 12182 1337 7.5 133.9 1.0X
|
|
codegen = T hashmap = F 7079 7337 250 11.9 84.4 1.6X
|
|
codegen = T hashmap = T 1278 1419 186 65.6 15.2 8.8X
|
|
|
|
|
|
================================================================================================
|
|
aggregate with randomized keys
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
Aggregate w keys: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
codegen = F 11629 11650 30 7.2 138.6 1.0X
|
|
codegen = T hashmap = F 7552 7747 169 11.1 90.0 1.5X
|
|
codegen = T hashmap = T 2414 2662 167 34.7 28.8 4.8X
|
|
|
|
|
|
================================================================================================
|
|
aggregate with string key
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
Aggregate w string key: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
codegen = F 4790 4904 162 4.4 228.4 1.0X
|
|
codegen = T hashmap = F 3439 3504 105 6.1 164.0 1.4X
|
|
codegen = T hashmap = T 2327 2365 39 9.0 111.0 2.1X
|
|
|
|
|
|
================================================================================================
|
|
aggregate with decimal key
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
Aggregate w decimal key: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
codegen = F 3260 3418 223 6.4 155.5 1.0X
|
|
codegen = T hashmap = F 2316 2325 14 9.1 110.4 1.4X
|
|
codegen = T hashmap = T 605 607 2 34.7 28.8 5.4X
|
|
|
|
|
|
================================================================================================
|
|
aggregate with multiple key types
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
Aggregate w multiple keys: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
codegen = F 7426 7473 67 2.8 354.1 1.0X
|
|
codegen = T hashmap = F 4685 4723 54 4.5 223.4 1.6X
|
|
codegen = T hashmap = T 3946 4005 83 5.3 188.2 1.9X
|
|
|
|
|
|
================================================================================================
|
|
max function bytecode size of wholestagecodegen
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
max function bytecode size: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
codegen = F 628 672 49 1.0 958.4 1.0X
|
|
codegen = T hugeMethodLimit = 10000 357 373 12 1.8 545.3 1.8X
|
|
codegen = T hugeMethodLimit = 1500 344 356 7 1.9 525.6 1.8X
|
|
|
|
|
|
================================================================================================
|
|
cube
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
cube: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
cube wholestage off 3167 3266 140 1.7 604.1 1.0X
|
|
cube wholestage on 1549 1576 29 3.4 295.4 2.0X
|
|
|
|
|
|
================================================================================================
|
|
hash and BytesToBytesMap
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
BytesToBytesMap: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------
|
|
UnsafeRowhash 326 328 2 64.3 15.5 1.0X
|
|
murmur3 hash 147 147 0 142.7 7.0 2.2X
|
|
fast hash 74 75 1 282.3 3.5 4.4X
|
|
arrayEqual 175 175 0 119.8 8.3 1.9X
|
|
Java HashMap (Long) 138 140 4 152.1 6.6 2.4X
|
|
Java HashMap (two ints) 148 154 7 141.7 7.1 2.2X
|
|
Java HashMap (UnsafeRow) 1043 1090 66 20.1 49.8 0.3X
|
|
LongToUnsafeRowMap (opt=false) 464 466 2 45.2 22.1 0.7X
|
|
LongToUnsafeRowMap (opt=true) 104 106 8 202.3 4.9 3.1X
|
|
BytesToBytesMap (off Heap) 1140 1149 12 18.4 54.4 0.3X
|
|
BytesToBytesMap (on Heap) 1002 1132 183 20.9 47.8 0.3X
|
|
Aggregate HashMap 74 74 0 281.9 3.5 4.4X
|
|
|
|
|