e7af44861e
### What changes were proposed in this pull request? Apache Parquet 1.12.0 switches its ZSTD compression from Hadoop codec to its own codec. ### Why are the changes needed? **Apache Spark 3.1 (It requires libhadoop built with zstd)** ```scala scala> spark.range(10).write.option("compression", "zstd").parquet("/tmp/a") 21/03/27 08:49:38 ERROR Executor: Exception in task 11.0 in stage 0.0 (TID 11)2] java.lang.RuntimeException: native zStandard library not available: this version of libhadoop was built without zstd support. ``` **Apache Spark 3.2 (No libhadoop requirement)** ```scala scala> spark.range(10).write.option("compression", "zstd").parquet("/tmp/a") ``` ### Does this PR introduce _any_ user-facing change? Yes, this is an improvement. ### How was this patch tested? Pass the CI with the newly added test coverage. Closes #31981 from dongjoon-hyun/SPARK-34880. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> |
||
---|---|---|
.. | ||
benchmarks | ||
src | ||
pom.xml |