From 3c825a18b64d3330e01f2fa62604771b2019e5e6 Mon Sep 17 00:00:00 2001 From: Kousuke Saruta Date: Fri, 9 Jul 2021 15:18:52 +0900 Subject: [PATCH] [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly ### What changes were proposed in this pull request? This PR fixes an issue that `YarnClusterSuite` fails due to `NoClassDefFoundError unless `hadoop-3.2` profile is activated explicitly regardless of building with SBT or Maven. ``` build/sbt -Pyarn "yarn/testOnly org.apache.spark.deploy.yarn.YarnClusterSuite" ... [info] YarnClusterSuite: [info] org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (598 milliseconds) [info] java.lang.NoClassDefFoundError: org/bouncycastle/operator/OperatorCreationException [info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:888) [info] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) [info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1410) [info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:344) [info] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) [info] at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:359) ``` The solution is modifying `yarn/pom.xml` to activate `hadoop-3.2` profiles by default. ### Why are the changes needed? hadoop-3.2 profile should be enabled by default so `YarnClusterSuite` should also successfully finishes without `-Phadoop-3.2`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Run `YarnClusterSuite` with both SBT and Maven without `-Phadoop-3.2` and it successfully finished. ``` build/sbt -Pyarn "yarn/testOnly org.apache.spark.deploy.yarn.YarnClusterSuite" ... [info] Run completed in 5 minutes, 38 seconds. [info] Total number of tests run: 27 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. build/mvn -Pyarn -pl resource-managers/yarn test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite ... Run completed in 5 minutes, 49 seconds. Total number of tests run: 27 Suites: completed 2, aborted 0 Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #33276 from sarutak/fix-bouncy-issue. Authored-by: Kousuke Saruta Signed-off-by: Hyukjin Kwon (cherry picked from commit 9aa65a6e0efbe9d12670ce655cbf0119fc2d3173) Signed-off-by: Hyukjin Kwon --- resource-managers/yarn/pom.xml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/resource-managers/yarn/pom.xml b/resource-managers/yarn/pom.xml index 00ad7e868a..e5fef2ab2d 100644 --- a/resource-managers/yarn/pom.xml +++ b/resource-managers/yarn/pom.xml @@ -78,6 +78,9 @@ hadoop-3.2 + + true +