16d9de815e
### What changes were proposed in this pull request? Before this pr, when we execute maven test command to test `mllib` and `kafka-0-10` module independently, there are some Java UTs failed, the key error messages are as follows: ``` java.lang.NoClassDefFoundError: scala/collection/parallel/TaskSupport ``` and ``` java.lang.NoClassDefFoundError: scala/collection/parallel/immutable/ParVector ``` The UTs need `scala-parallel-collections_2.13`, but it not in classpath when we run `mvn test -pl mllib -Pscala-2.13` and `mvn test -pl external/kafka-0-10 -Pscala-2.13`. So the main change of this pr is add `scala-2.13` profile to `mllib/pom.xml` and `external/kafka-0-10/pom.xml`, the `scala-2.13` profile include dependency on `scala-parallel-collections_2.13`, then these two modules can maven test independently. ### Why are the changes needed? Ensure mllib and kafka-0-10 module can be maven test independently in Scala 2.13 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass the GitHub Action Scala 2.13 job - Manual test: 1. Execute ``` dev/change-scala-version.sh 2.13 mvn clean install -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 ``` 2. Execute ``` mvn test -pl mllib -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 ``` **Before** 6 Java UTs failed: ``` [ERROR] Errors: [ERROR] JavaStreamingLogisticRegressionSuite.javaAPI:78 » TestFailed 20005 was not les... [ERROR] JavaStreamingKMeansSuite.javaAPI:78 » TestFailed 20040 was not less than 20000... [ERROR] JavaPrefixSpanSuite.runPrefixSpan:45 » NoClassDefFound scala/collection/parall... [ERROR] JavaPrefixSpanSuite.runPrefixSpanSaveLoad:67 » NoClassDefFound scala/collectio... [ERROR] JavaStreamingLinearRegressionSuite.javaAPI:77 » TestFailed 20014 was not less ... [ERROR] JavaStatisticsSuite.streamingTest:112 » TestFailed 20043 was not less than 200... [INFO] [ERROR] Tests run: 122, Failures: 0, Errors: 6, Skipped: 0 ``` **After** ``` [INFO] Tests run: 122, Failures: 0, Errors: 0, Skipped: 0 Run completed in 28 minutes, 32 seconds. Total number of tests run: 1654 Suites: completed 208, aborted 0 Tests: succeeded 1654, failed 0, canceled 0, ignored 7, pending 0 All tests passed. ``` 3. Execute ``` mvn test -pl external/kafka-0-10 -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 ``` **Before** 2 Java UTs failed: ``` [ERROR] Errors: [ERROR] org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite.testKafkaStream [ERROR] Run 1: JavaDirectKafkaStreamSuite.testKafkaStream:170 expected:<[topic1-1, topic1-2, topic2-1, topic1-3, topic2-2, topic2-3]> but was:<[]> [ERROR] Run 2: JavaDirectKafkaStreamSuite.tearDown:57 » NoClassDefFound scala/collection/para... [ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0 ``` **After** ``` [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 Run completed in 1 minute, 3 seconds. Total number of tests run: 21 Suites: completed 4, aborted 0 Tests: succeeded 21, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #32676 from LuciferYang/mllib-kafka-mvn-test. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> |
||
---|---|---|
.. | ||
src | ||
pom.xml |