[SPARK-33212][FOLLOWUP] Add hadoop-yarn-server-web-proxy for Hadoop 3.x profile
### What changes were proposed in this pull request? This adds `hadoop-yarn-server-web-proxy` as dependency for Yarn and Hadoop 3.x profile (it is already a dependency for 2.x). Also excludes some dependencies from the module which are already covered by other Hadoop jars used by Spark. ### Why are the changes needed? The class `org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter` is used by `ApplicationMaster`: ```scala private def addAmIpFilter(driver: Option[RpcEndpointRef], proxyBase: String) = { val amFilter = "org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter" val params = client.getAmIpFilterParams(yarnConf, proxyBase) driver match { case Some(d) => d.send(AddWebUIFilter(amFilter, params, proxyBase)) ... ``` and will be loaded at runtime. Therefore, without the above jar Spark Yarn app will fail with `ClassNotFoundError`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing unit tests. Also tested manually and it worked with the fix, while was failing previously. Closes #31642 from sunchao/SPARK-33212-followup-2. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This commit is contained in:
parent
1afe284ed8
commit
f494c5cff9
|
@ -136,6 +136,10 @@
|
|||
<artifactId>spark-yarn_${scala.binary.version}</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-yarn-server-web-proxy</artifactId>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</profile>
|
||||
<profile>
|
||||
|
|
|
@ -58,6 +58,7 @@ gson/2.2.4//gson-2.2.4.jar
|
|||
guava/14.0.1//guava-14.0.1.jar
|
||||
hadoop-client-api/3.2.2//hadoop-client-api-3.2.2.jar
|
||||
hadoop-client-runtime/3.2.2//hadoop-client-runtime-3.2.2.jar
|
||||
hadoop-yarn-server-web-proxy/3.2.2//hadoop-yarn-server-web-proxy-3.2.2.jar
|
||||
hive-beeline/2.3.8//hive-beeline-2.3.8.jar
|
||||
hive-cli/2.3.8//hive-cli-2.3.8.jar
|
||||
hive-common/2.3.8//hive-common-2.3.8.jar
|
||||
|
|
24
pom.xml
24
pom.xml
|
@ -1407,6 +1407,26 @@
|
|||
<version>${yarn.version}</version>
|
||||
<scope>${hadoop.deps.scope}</scope>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-yarn-server-common</artifactId>
|
||||
</exclusion>
|
||||
<exclusion>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-yarn-common</artifactId>
|
||||
</exclusion>
|
||||
<exclusion>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-yarn-api</artifactId>
|
||||
</exclusion>
|
||||
<exclusion>
|
||||
<groupId>org.bouncycastle</groupId>
|
||||
<artifactId>bcprov-jdk15on</artifactId>
|
||||
</exclusion>
|
||||
<exclusion>
|
||||
<groupId>org.bouncycastle</groupId>
|
||||
<artifactId>bcpkix-jdk15on</artifactId>
|
||||
</exclusion>
|
||||
<exclusion>
|
||||
<groupId>org.fusesource.leveldbjni</groupId>
|
||||
<artifactId>leveldbjni-all</artifactId>
|
||||
|
@ -1427,6 +1447,10 @@
|
|||
<groupId>javax.servlet</groupId>
|
||||
<artifactId>servlet-api</artifactId>
|
||||
</exclusion>
|
||||
<exclusion>
|
||||
<groupId>javax.servlet</groupId>
|
||||
<artifactId>javax.servlet-api</artifactId>
|
||||
</exclusion>
|
||||
<exclusion>
|
||||
<groupId>commons-logging</groupId>
|
||||
<artifactId>commons-logging</artifactId>
|
||||
|
|
Loading…
Reference in a new issue