[SPARK-36068][BUILD][TEST] No tests in hadoop-cloud run unless hadoop-3.2 profile is activated explicitly
### What changes were proposed in this pull request? This PR fixes an issue that no tests in `hadoop-cloud` are compiled and run unless `hadoop-3.2` profile is activated explicitly. The root cause seems similar to SPARK-36067 (#33276) so the solution is to activate `hadoop-3.2` profile in `hadoop-cloud/pom.xml` by default. This PR introduced an empty profile for `hadoop-2.7`. Without this, building with `hadoop-2.7` fails. ### Why are the changes needed? `hadoop-3.2` profile should be activated by default so tests in `hadoop-cloud` also should be compiled and run without activating `hadoop-3.2` profile explicitly. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed tests in `hadoop-cloud` ran with both SBT and Maven. ``` build/sbt -Phadoop-cloud "hadoop-cloud/test" ... [info] CommitterBindingSuite: [info] - BindingParquetOutputCommitter binds to the inner committer (258 milliseconds) [info] - committer protocol can be serialized and deserialized (11 milliseconds) [info] - local filesystem instantiation (3 milliseconds) [info] - reject dynamic partitioning (1 millisecond) [info] Run completed in 1 second, 234 milliseconds. [info] Total number of tests run: 4 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. build/mvn -Phadoop-cloud -pl hadoop-cloud test ... CommitterBindingSuite: - BindingParquetOutputCommitter binds to the inner committer - committer protocol can be serialized and deserialized - local filesystem instantiation - reject dynamic partitioning Run completed in 560 milliseconds. Total number of tests run: 4 Suites: completed 2, aborted 0 Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` I also confirmed building with `-Phadoop-2.7` successfully finishes with both SBT and Maven. ``` build/sbt -Phadoop-cloud -Phadoop-2.7 "hadoop-cloud/Test/compile" build/mvn -Phadoop-cloud -Phadoop-2.7 -pl hadoop-cloud testCompile ``` Closes #33277 from sarutak/fix-hadoop-3.2-cloud. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
parent
01cf6f4c6b
commit
0f5c3a4fd6
|
@ -201,13 +201,22 @@
|
||||||
</dependencies>
|
</dependencies>
|
||||||
|
|
||||||
<profiles>
|
<profiles>
|
||||||
|
<!--
|
||||||
|
hadoop-3.2 profile is activated by default so hadoop-2.7 profile
|
||||||
|
also needs to be declared here for building with -Phadoop-2.7.
|
||||||
|
-->
|
||||||
|
<profile>
|
||||||
|
<id>hadoop-2.7</id>
|
||||||
|
</profile>
|
||||||
<!--
|
<!--
|
||||||
Hadoop 3 simplifies the classpath, and adds a new committer base class which
|
Hadoop 3 simplifies the classpath, and adds a new committer base class which
|
||||||
enables store-specific committers.
|
enables store-specific committers.
|
||||||
-->
|
-->
|
||||||
<profile>
|
<profile>
|
||||||
<id>hadoop-3.2</id>
|
<id>hadoop-3.2</id>
|
||||||
|
<activation>
|
||||||
|
<activeByDefault>true</activeByDefault>
|
||||||
|
</activation>
|
||||||
<properties>
|
<properties>
|
||||||
<extra.source.dir>src/hadoop-3/main/scala</extra.source.dir>
|
<extra.source.dir>src/hadoop-3/main/scala</extra.source.dir>
|
||||||
<extra.testsource.dir>src/hadoop-3/test/scala</extra.testsource.dir>
|
<extra.testsource.dir>src/hadoop-3/test/scala</extra.testsource.dir>
|
||||||
|
|
Loading…
Reference in a new issue