b2e3256256
## What changes were proposed in this pull request?
Before the code changes, I tried to run it with 8G memory:
```
build/sbt -mem 8000 "core/testOnly org.apache.spark.serializer.KryoBenchmark"
```
Still I got got OOM.
This is because the lengths of the arrays are random
669ade3a8e/core/src/test/scala/org/apache/spark/serializer/KryoBenchmark.scala (L90-L91)
And the 2D array is usually large: `10000 * Random.nextInt(0, 10000)`
This PR is to fix it and refactor it to use main method.
The benchmark result is also reason compared to the original one.
## How was this patch tested?
Run with
```
bin/spark-submit --class org.apache.spark.serializer.KryoBenchmark core/target/scala-2.11/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar
```
and
```
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "core/test:runMain org.apache.spark.serializer.KryoBenchmark"
Closes #22663 from gengliangwang/kyroBenchmark.
Authored-by: Gengliang Wang <gengliang.wang@databricks.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
30 lines
2.2 KiB
Plaintext
30 lines
2.2 KiB
Plaintext
================================================================================================
|
|
Benchmark Kryo Unsafe vs safe Serialization
|
|
================================================================================================
|
|
|
|
Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 on Mac OS X 10.13.6
|
|
Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
|
|
|
|
Benchmark Kryo Unsafe vs safe Serialization: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------
|
|
basicTypes: Int with unsafe:true 138 / 149 7.2 138.0 1.0X
|
|
basicTypes: Long with unsafe:true 168 / 173 6.0 167.7 0.8X
|
|
basicTypes: Float with unsafe:true 153 / 174 6.5 153.1 0.9X
|
|
basicTypes: Double with unsafe:true 161 / 185 6.2 161.1 0.9X
|
|
Array: Int with unsafe:true 2 / 3 409.7 2.4 56.5X
|
|
Array: Long with unsafe:true 4 / 5 232.5 4.3 32.1X
|
|
Array: Float with unsafe:true 3 / 4 367.3 2.7 50.7X
|
|
Array: Double with unsafe:true 4 / 5 228.5 4.4 31.5X
|
|
Map of string->Double with unsafe:true 38 / 45 26.5 37.8 3.7X
|
|
basicTypes: Int with unsafe:false 176 / 187 5.7 175.9 0.8X
|
|
basicTypes: Long with unsafe:false 191 / 203 5.2 191.2 0.7X
|
|
basicTypes: Float with unsafe:false 166 / 176 6.0 166.2 0.8X
|
|
basicTypes: Double with unsafe:false 174 / 190 5.7 174.3 0.8X
|
|
Array: Int with unsafe:false 19 / 26 52.9 18.9 7.3X
|
|
Array: Long with unsafe:false 27 / 31 37.7 26.5 5.2X
|
|
Array: Float with unsafe:false 8 / 10 124.3 8.0 17.2X
|
|
Array: Double with unsafe:false 12 / 13 83.6 12.0 11.5X
|
|
Map of string->Double with unsafe:false 38 / 42 26.1 38.3 3.6X
|
|
|
|
|