0ab9bd79b3
### What changes were proposed in this pull request? Change `map` in `InvokeLike.invoke` to a while loop to improve performance, following Spark [style guide](https://github.com/databricks/scala-style-guide#traversal-and-zipwithindex). ### Why are the changes needed? `InvokeLike.invoke`, which is used in non-codegen path for `Invoke` and `StaticInvoke`, currently uses `map` to evaluate arguments: ```scala val args = arguments.map(e => e.eval(input).asInstanceOf[Object]) if (needNullCheck && args.exists(_ == null)) { // return null if one of arguments is null null } else { ... ``` which is pretty expensive if the method itself is trivial. We can change it to a plain while loop. <img width="871" alt="Screen Shot 2021-05-12 at 12 19 59 AM" src="https://user-images.githubusercontent.com/506679/118055719-7f985a00-b33d-11eb-943b-cf85eab35f44.png"> Benchmark results show this can improve as much as 3x from `V2FunctionBenchmark`: Before ``` OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.4.0-1046-azure Intel(R) Xeon(R) CPU E5-2673 v3 2.40GHz scalar function (long + long) -> long, result_nullable = false codegen = false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative -------------------------------------------------------------------------------------------------------------------------------------------------------------- native_long_add 36506 36656 251 13.7 73.0 1.0X java_long_add_default 47151 47540 370 10.6 94.3 0.8X java_long_add_magic 178691 182457 1327 2.8 357.4 0.2X java_long_add_static_magic 177151 178258 1151 2.8 354.3 0.2X ``` After ``` OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.4.0-1046-azure Intel(R) Xeon(R) CPU E5-2673 v3 2.40GHz scalar function (long + long) -> long, result_nullable = false codegen = false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative -------------------------------------------------------------------------------------------------------------------------------------------------------------- native_long_add 29897 30342 568 16.7 59.8 1.0X java_long_add_default 40628 41075 664 12.3 81.3 0.7X java_long_add_magic 54553 54755 182 9.2 109.1 0.5X java_long_add_static_magic 55410 55532 127 9.0 110.8 0.5X ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests. Closes #32527 from sunchao/SPARK-35384. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
45 lines
5.4 KiB
Plaintext
45 lines
5.4 KiB
Plaintext
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
|
|
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
|
|
scalar function (long + long) -> long, result_nullable = true codegen = true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------------------------------------------
|
|
native_long_add 17138 17431 486 29.2 34.3 1.0X
|
|
java_long_add_default 47386 48316 1583 10.6 94.8 0.4X
|
|
java_long_add_magic 19409 19532 152 25.8 38.8 0.9X
|
|
java_long_add_static_magic 18257 18294 33 27.4 36.5 0.9X
|
|
scala_long_add_default 49259 49512 235 10.2 98.5 0.3X
|
|
scala_long_add_magic 18964 19025 53 26.4 37.9 0.9X
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
|
|
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
|
|
scalar function (long + long) -> long, result_nullable = false codegen = true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
-------------------------------------------------------------------------------------------------------------------------------------------------------------
|
|
native_long_add 16814 16916 99 29.7 33.6 1.0X
|
|
java_long_add_default 43725 43909 216 11.4 87.4 0.4X
|
|
java_long_add_magic 19015 19060 39 26.3 38.0 0.9X
|
|
java_long_add_static_magic 18940 18993 52 26.4 37.9 0.9X
|
|
scala_long_add_default 43804 43874 88 11.4 87.6 0.4X
|
|
scala_long_add_magic 18753 18791 34 26.7 37.5 0.9X
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
|
|
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
|
|
scalar function (long + long) -> long, result_nullable = true codegen = false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
-------------------------------------------------------------------------------------------------------------------------------------------------------------
|
|
native_long_add 42493 42830 506 11.8 85.0 1.0X
|
|
java_long_add_default 54557 54710 141 9.2 109.1 0.8X
|
|
java_long_add_magic 74409 74564 227 6.7 148.8 0.6X
|
|
java_long_add_static_magic 75081 75235 190 6.7 150.2 0.6X
|
|
scala_long_add_default 54789 54862 77 9.1 109.6 0.8X
|
|
scala_long_add_magic 73777 73886 96 6.8 147.6 0.6X
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.4.0-1046-azure
|
|
Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
|
|
scalar function (long + long) -> long, result_nullable = false codegen = false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
--------------------------------------------------------------------------------------------------------------------------------------------------------------
|
|
native_long_add 37357 37490 116 13.4 74.7 1.0X
|
|
java_long_add_default 53166 53192 23 9.4 106.3 0.7X
|
|
java_long_add_magic 70501 71258 1121 7.1 141.0 0.5X
|
|
java_long_add_static_magic 68934 69636 1115 7.3 137.9 0.5X
|
|
scala_long_add_default 53075 53146 62 9.4 106.2 0.7X
|
|
scala_long_add_magic 69838 70746 1442 7.2 139.7 0.5X
|
|
|