f494c5cff9
### What changes were proposed in this pull request? This adds `hadoop-yarn-server-web-proxy` as dependency for Yarn and Hadoop 3.x profile (it is already a dependency for 2.x). Also excludes some dependencies from the module which are already covered by other Hadoop jars used by Spark. ### Why are the changes needed? The class `org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter` is used by `ApplicationMaster`: ```scala private def addAmIpFilter(driver: Option[RpcEndpointRef], proxyBase: String) = { val amFilter = "org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter" val params = client.getAmIpFilterParams(yarnConf, proxyBase) driver match { case Some(d) => d.send(AddWebUIFilter(amFilter, params, proxyBase)) ... ``` and will be loaded at runtime. Therefore, without the above jar Spark Yarn app will fail with `ClassNotFoundError`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing unit tests. Also tested manually and it worked with the fix, while was failing previously. Closes #31642 from sunchao/SPARK-33212-followup-2. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
---|---|---|
.. | ||
src/main/assembly | ||
pom.xml | ||
README |
This is an assembly module for Spark project. It creates a single tar.gz file that includes all needed dependency of the project except for org.apache.hadoop.* jars that are supposed to be available from the deployed Hadoop cluster. This module is off by default. To activate it specify the profile in the command line -Pbigtop-dist If you need to build an assembly for a different version of Hadoop the hadoop-version system property needs to be set as in this example: -Dhadoop.version=2.7.4