f642234d85
### What changes were proposed in this pull request? This PR aims to speed up `MapStatus` deserialization by 5~18% with the latest RoaringBitmap `0.9.0` and new APIs. Note that we focus on `deserialization` time because `serialization` occurs once while `deserialization` occurs many times. ### Why are the changes needed? The current version is too old. We had better upgrade it to get the performance improvement and bug fixes. Although `MapStatusesSerDeserBenchmark` is synthetic, the benchmark result is updated with this patch. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the Jenkins or GitHub Action. Closes #29233 from dongjoon-hyun/SPARK-ROAR. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
67 lines
4.3 KiB
Plaintext
67 lines
4.3 KiB
Plaintext
OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
200000 MapOutputs, 10 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
-------------------------------------------------------------------------------------------------------------------------
|
|
Serialization 175 183 12 1.1 874.1 1.0X
|
|
Deserialization 458 462 6 0.4 2288.6 0.4X
|
|
|
|
Compressed Serialized MapStatus sizes: 410 bytes
|
|
Compressed Serialized Broadcast MapStatus sizes: 2 MB
|
|
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
200000 MapOutputs, 10 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
--------------------------------------------------------------------------------------------------------------------------
|
|
Serialization 160 171 8 1.2 801.1 1.0X
|
|
Deserialization 453 484 38 0.4 2263.4 0.4X
|
|
|
|
Compressed Serialized MapStatus sizes: 2 MB
|
|
Compressed Serialized Broadcast MapStatus sizes: 0 bytes
|
|
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
200000 MapOutputs, 100 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
--------------------------------------------------------------------------------------------------------------------------
|
|
Serialization 343 346 2 0.6 1717.1 1.0X
|
|
Deserialization 492 540 59 0.4 2459.6 0.7X
|
|
|
|
Compressed Serialized MapStatus sizes: 426 bytes
|
|
Compressed Serialized Broadcast MapStatus sizes: 13 MB
|
|
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
200000 MapOutputs, 100 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
---------------------------------------------------------------------------------------------------------------------------
|
|
Serialization 297 299 2 0.7 1486.2 1.0X
|
|
Deserialization 489 535 87 0.4 2446.5 0.6X
|
|
|
|
Compressed Serialized MapStatus sizes: 13 MB
|
|
Compressed Serialized Broadcast MapStatus sizes: 0 bytes
|
|
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
200000 MapOutputs, 1000 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
---------------------------------------------------------------------------------------------------------------------------
|
|
Serialization 1641 1819 252 0.1 8204.1 1.0X
|
|
Deserialization 844 882 37 0.2 4219.7 1.9X
|
|
|
|
Compressed Serialized MapStatus sizes: 553 bytes
|
|
Compressed Serialized Broadcast MapStatus sizes: 121 MB
|
|
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 4.15.0-1044-aws
|
|
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
|
|
200000 MapOutputs, 1000 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
----------------------------------------------------------------------------------------------------------------------------
|
|
Serialization 1360 1412 73 0.1 6799.3 1.0X
|
|
Deserialization 850 859 13 0.2 4249.9 1.6X
|
|
|
|
Compressed Serialized MapStatus sizes: 121 MB
|
|
Compressed Serialized Broadcast MapStatus sizes: 0 bytes
|
|
|
|
|