506ef9aad7
### What changes were proposed in this pull request? This upgrade default Hadoop version from 3.2.1 to 3.3.1. The changes here are simply update the version number and dependency file. ### Why are the changes needed? Hadoop 3.3.1 just came out, which comes with many client-side improvements such as for S3A/ABFS (20% faster when accessing S3). These are important for users who want to use Spark in a cloud environment. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Existing unit tests in Spark - Manually tested using my S3 bucket for event log dir: ``` bin/spark-shell \ -c spark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID \ -c spark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY \ -c spark.eventLog.enabled=true -c spark.eventLog.dir=s3a://<my-bucket> ``` - Manually tested against docker-based YARN dev cluster, by running `SparkPi`. Closes #30135 from sunchao/SPARK-29250. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
|---|---|---|
| .. | ||
| kubernetes | ||
| mesos | ||
| yarn | ||