[SPARK-35429][CORE] Remove commons-httpclient from Hadoop-3.2 profile due to EOL and CVEs
### What changes were proposed in this pull request?
Remove commons-httpclient as a direct dependency for Hadoop-3.2 profile.
Hadoop-2.7 profile distribution still has it, hadoop-client has a compile dependency on commons-httpclient, thus we cannot remove it for Hadoop-2.7 profile.
```
[INFO] +- org.apache.hadoop:hadoop-client:jar:2.7.4:compile
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.7.4:compile
[INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile
```
### Why are the changes needed?
Spark is pulling in commons-httpclient as a dependency directly. commons-httpclient went EOL years ago and there are most likely CVEs not being reported against it, thus we should remove it.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Existing unittests
- Checked the dependency tree before and after introducing the changes
Before:
```
./build/mvn dependency:tree -Phadoop-3.2 | grep -i "commons-httpclient"
Using `mvn` from path: /usr/bin/mvn
[INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:provided
```
After
```
./build/mvn dependency:tree | grep -i "commons-httpclient"
Using `mvn` from path: /Users/sumeet.gajjar/cloudera/upstream-spark/build/apache-maven-3.6.3/bin/mvn
```
P.S. Reopening this since [spark upgraded](463daabd5a
) its `hive.version` to `2.3.9` which does not have a dependency on `commons-httpclient`.
Closes #32912 from sumeetgajjar/SPARK-35429.
Authored-by: Sumeet Gajjar <sumeetgajjar93@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This commit is contained in:
parent
61ce8f7649
commit
864ff67746
|
@ -35,7 +35,6 @@ commons-compiler/3.1.4//commons-compiler-3.1.4.jar
|
||||||
commons-compress/1.20//commons-compress-1.20.jar
|
commons-compress/1.20//commons-compress-1.20.jar
|
||||||
commons-crypto/1.1.0//commons-crypto-1.1.0.jar
|
commons-crypto/1.1.0//commons-crypto-1.1.0.jar
|
||||||
commons-dbcp/1.4//commons-dbcp-1.4.jar
|
commons-dbcp/1.4//commons-dbcp-1.4.jar
|
||||||
commons-httpclient/3.1//commons-httpclient-3.1.jar
|
|
||||||
commons-io/2.8.0//commons-io-2.8.0.jar
|
commons-io/2.8.0//commons-io-2.8.0.jar
|
||||||
commons-lang/2.6//commons-lang-2.6.jar
|
commons-lang/2.6//commons-lang-2.6.jar
|
||||||
commons-lang3/3.12.0//commons-lang3-3.12.0.jar
|
commons-lang3/3.12.0//commons-lang3-3.12.0.jar
|
||||||
|
|
11
pom.xml
11
pom.xml
|
@ -157,8 +157,6 @@
|
||||||
<!-- org.apache.httpcomponents/httpclient-->
|
<!-- org.apache.httpcomponents/httpclient-->
|
||||||
<commons.httpclient.version>4.5.13</commons.httpclient.version>
|
<commons.httpclient.version>4.5.13</commons.httpclient.version>
|
||||||
<commons.httpcore.version>4.4.14</commons.httpcore.version>
|
<commons.httpcore.version>4.4.14</commons.httpcore.version>
|
||||||
<!-- commons-httpclient/commons-httpclient-->
|
|
||||||
<httpclient.classic.version>3.1</httpclient.classic.version>
|
|
||||||
<commons.math3.version>3.4.1</commons.math3.version>
|
<commons.math3.version>3.4.1</commons.math3.version>
|
||||||
<!-- managed up from 3.2.1 for SPARK-11652 -->
|
<!-- managed up from 3.2.1 for SPARK-11652 -->
|
||||||
<commons.collections.version>3.2.2</commons.collections.version>
|
<commons.collections.version>3.2.2</commons.collections.version>
|
||||||
|
@ -592,11 +590,6 @@
|
||||||
<artifactId>jsr305</artifactId>
|
<artifactId>jsr305</artifactId>
|
||||||
<version>${jsr305.version}</version>
|
<version>${jsr305.version}</version>
|
||||||
</dependency>
|
</dependency>
|
||||||
<dependency>
|
|
||||||
<groupId>commons-httpclient</groupId>
|
|
||||||
<artifactId>commons-httpclient</artifactId>
|
|
||||||
<version>${httpclient.classic.version}</version>
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.apache.httpcomponents</groupId>
|
<groupId>org.apache.httpcomponents</groupId>
|
||||||
<artifactId>httpclient</artifactId>
|
<artifactId>httpclient</artifactId>
|
||||||
|
@ -1810,10 +1803,6 @@
|
||||||
<groupId>commons-codec</groupId>
|
<groupId>commons-codec</groupId>
|
||||||
<artifactId>commons-codec</artifactId>
|
<artifactId>commons-codec</artifactId>
|
||||||
</exclusion>
|
</exclusion>
|
||||||
<exclusion>
|
|
||||||
<groupId>commons-httpclient</groupId>
|
|
||||||
<artifactId>commons-httpclient</artifactId>
|
|
||||||
</exclusion>
|
|
||||||
<exclusion>
|
<exclusion>
|
||||||
<groupId>org.apache.avro</groupId>
|
<groupId>org.apache.avro</groupId>
|
||||||
<artifactId>avro-mapred</artifactId>
|
<artifactId>avro-mapred</artifactId>
|
||||||
|
|
|
@ -133,10 +133,6 @@
|
||||||
<groupId>org.apache.avro</groupId>
|
<groupId>org.apache.avro</groupId>
|
||||||
<artifactId>avro-mapred</artifactId>
|
<artifactId>avro-mapred</artifactId>
|
||||||
</dependency>
|
</dependency>
|
||||||
<dependency>
|
|
||||||
<groupId>commons-httpclient</groupId>
|
|
||||||
<artifactId>commons-httpclient</artifactId>
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.apache.httpcomponents</groupId>
|
<groupId>org.apache.httpcomponents</groupId>
|
||||||
<artifactId>httpclient</artifactId>
|
<artifactId>httpclient</artifactId>
|
||||||
|
|
Loading…
Reference in a new issue