7fcb127674
### What changes were proposed in this pull request? This PR aims to upgrade `zstd-jni` to 1.5.0-2, which uses `zstd` version 1.5.0. ### Why are the changes needed? Major improvements to Zstd support are targeted for the upcoming 3.2.0 release of Spark. Zstd 1.5.0 introduces significant compression (+25% to 140%) and decompression (~15%) speed improvements in benchmarks described in more detail on the releases page: - https://github.com/facebook/zstd/releases/tag/v1.5.0 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Build passes build tests, but the benchmark tests seem flaky. I am unsure if this change is responsible. The error is: ``` Running org.apache.spark.rdd.CoalescedRDDBenchmark: 21/06/08 18:53:10 ERROR SparkContext: Failed to add file:/home/runner/work/spark/spark/./core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar to Spark environment java.lang.IllegalArgumentException: requirement failed: File spark-core_2.12-3.2.0-SNAPSHOT-tests.jar was already registered with a different path (old path = /home/runner/work/spark/spark/core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar, new path = /home/runner/work/spark/spark/./core/target/scala-2.12/spark-core_2.12-3.2.0-SNAPSHOT-tests.jar ``` https://github.com/dchristle/spark/runs/2776123749?check_suite_focus=true cc: dongjoon-hyun Closes #32826 from dchristle/ZSTD150. Lead-authored-by: David Christle <dchristle@squareup.com> Co-authored-by: David Christle <dchristle@users.noreply.github.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
28 lines
2.6 KiB
Plaintext
28 lines
2.6 KiB
Plaintext
================================================================================================
|
|
Benchmark ZStandardCompressionCodec
|
|
================================================================================================
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-1033-azure
|
|
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
|
|
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
--------------------------------------------------------------------------------------------------------------------------------------
|
|
Compression 10000 times at level 1 without buffer pool 805 1103 500 0.0 80501.4 1.0X
|
|
Compression 10000 times at level 2 without buffer pool 728 744 20 0.0 72819.9 1.1X
|
|
Compression 10000 times at level 3 without buffer pool 987 995 7 0.0 98719.4 0.8X
|
|
Compression 10000 times at level 1 with buffer pool 371 377 8 0.0 37092.3 2.2X
|
|
Compression 10000 times at level 2 with buffer pool 465 473 6 0.0 46509.8 1.7X
|
|
Compression 10000 times at level 3 with buffer pool 715 738 20 0.0 71500.2 1.1X
|
|
|
|
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-1033-azure
|
|
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
|
|
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
|
|
------------------------------------------------------------------------------------------------------------------------------------------
|
|
Decompression 10000 times from level 1 without buffer pool 776 786 11 0.0 77649.5 1.0X
|
|
Decompression 10000 times from level 2 without buffer pool 787 792 5 0.0 78686.6 1.0X
|
|
Decompression 10000 times from level 3 without buffer pool 782 790 7 0.0 78195.4 1.0X
|
|
Decompression 10000 times from level 1 with buffer pool 529 551 21 0.0 52914.0 1.5X
|
|
Decompression 10000 times from level 2 with buffer pool 523 537 11 0.0 52266.2 1.5X
|
|
Decompression 10000 times from level 3 with buffer pool 519 527 10 0.0 51932.3 1.5X
|
|
|
|
|