4f51e0045e
### What changes were proposed in this pull request? To make the test suite more robust, this PR aims to add a new trait, `LocalRootDirsTest`, by refactoring `SortShuffleSuite`'s helper functions and applying it to the following: - ShuffleNettySuite - ShuffleOldFetchProtocolSuite - ExternalShuffleServiceSuite - KubernetesLocalDiskShuffleDataIOSuite - LocalDirsSuite - RDDCleanerSuite - ALSCleanerSuite In addition, this fixes a UT in `KubernetesLocalDiskShuffleDataIOSuite`. ### Why are the changes needed? `ShuffleSuite` is extended by four classes but only `SortShuffleSuite` does the clean-up correctly. ``` ShuffleSuite - SortShuffleSuite - ShuffleNettySuite - ShuffleOldFetchProtocolSuite - ExternalShuffleServiceSuite ``` Since `KubernetesLocalDiskShuffleDataIOSuite` is looking for the other storage directory, the leftover of `ShuffleSuite` causes flakiness. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/2649/testReport/junit/org.apache.spark.shuffle/KubernetesLocalDiskShuffleDataIOSuite/recompute_is_not_blocked_by_the_recovery/ ``` org.apache.spark.SparkException: Job aborted due to stage failure: task 0.0 in stage 1.0 (TID 3) had a not serializable result: org.apache.spark.ShuffleSuite$NonJavaSerializableClass ... org.apache.spark.shuffle.KubernetesLocalDiskShuffleDataIOSuite.$anonfun$new$2(KubernetesLocalDiskShuffleDataIOSuite.scala:52) ``` For the other suites, the clean-up implementation is used but not complete. So, they are refactored to use new trait. ### Does this PR introduce _any_ user-facing change? No, this is a test-only change. ### How was this patch tested? Pass the CIs. Closes #32986 from dongjoon-hyun/SPARK-35832. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
---|---|---|
.. | ||
core | ||
docker/src/main/dockerfiles/spark | ||
integration-tests |