ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-33212][FOLLOW-UP][BUILD] Uses provided properties for Hadoop client dependencies in root pom

Browse source 
### What changes were proposed in this pull request?

This PR is a followup of https://github.com/apache/spark/pull/30701. It uses properties of `hadoop-client-api.artifact`, `hadoop-client-runtime.artifact` and `hadoop-client-minicluster.artifact` explicitly to set the dependencies and versions.

Otherwise, it is logically incorrect. For example, if you build with Hadoop 2, this dependency becomes `hadoop-client-api:2.7.4` internally, which does not exist in Hadoop 2 (https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client-api).

### Why are the changes needed?

- To fix the logical incorrectness.
- It fixes a potential issue: this actually caused an issue when `generate-sources` plugin is used together with Hadoop 2 by default, which attempts to pull 2.7.4 of `hadoop-client-api`, `hadoop-client-runtime` and `hadoop-client-minicluster` for whatever reason.

### Does this PR introduce _any_ user-facing change?

No for users and dev. It's more a cleanup.

### How was this patch tested?

Manually checked the dependencies are correctly placed.

Closes #31467 from HyukjinKwon/SPARK-33212.

Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
This commit is contained in:
HyukjinKwon 2021-02-04 17:31:36 +09:00
parent 42c32e8a3a
commit 6a194f1978
1 changed files with 3 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
pom.xml
View file

@ -1079,19 +1079,19 @@
<!-- Hadoop 3.x dependencies --> <!-- Hadoop 3.x dependencies -->
<dependency> <dependency>
<groupId>org.apache.hadoop</groupId> <groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-api</artifactId> <artifactId>${hadoop-client-api.artifact}</artifactId>
<version>${hadoop.version}</version> <version>${hadoop.version}</version>
<scope>${hadoop.deps.scope}</scope> <scope>${hadoop.deps.scope}</scope>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.apache.hadoop</groupId> <groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-runtime</artifactId> <artifactId>${hadoop-client-runtime.artifact}</artifactId>
<version>${hadoop.version}</version> <version>${hadoop.version}</version>
<scope>${hadoop.deps.scope}</scope> <scope>${hadoop.deps.scope}</scope>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.apache.hadoop</groupId> <groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-minicluster</artifactId> <artifactId>${hadoop-client-minicluster.artifact}</artifactId>
<version>${yarn.version}</version> <version>${yarn.version}</version>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>