From be6ecb6d19528cc80e3c09e6127d2a89e5b4aa13 Mon Sep 17 00:00:00 2001 From: byungsoo Date: Mon, 3 May 2021 18:06:06 +0900 Subject: [PATCH] [SPARK-35266][TESTS] Fix error in BenchmarkBase.scala that occurs when creating benchmark files in non-existent directory ### What changes were proposed in this pull request? This PR fixes an error in `BenchmarkBase.scala` that occurs when creating a benchmark file in a non-existent directory. ### Why are the changes needed? When submitting a benchmark job using `org.apache.spark.benchmark.Benchmarks` class with `SPARK_GENERATE_BENCHMARK_FILES=1` option, an exception is raised if the directory where the benchmark file will be generated does not exist. For more information, please refer to [SPARK-35266](https://issues.apache.org/jira/browse/SPARK-35266). ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? After building Spark, manually tested with the following command: ``` SPARK_GENERATE_BENCHMARK_FILES=1 bin/spark-submit --class \ org.apache.spark.benchmark.Benchmarks --jars \ "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \ "`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \ "org.apache.spark.ml.linalg.BLASBenchmark" ``` It successfully generated the benchmark result files. **Why it is sufficient:** As illustrated in the comments in `Benchmarks.scala`, the command below runs all benchmarks and generates the results: ``` SPARK_GENERATE_BENCHMARK_FILES=1 bin/spark-submit --class \ org.apache.spark.benchmark.Benchmarks --jars \ "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \ "`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \ "*" ``` Of all the benchmarks (55 benchmarks in total), only `BLASBenchmark` fails due to the proposed issue for the current code in the master branch. Thus, it is currently sufficient to test `BLASBenchmark` to validate this change. Closes #32394 from byungsoo-oh/SPARK-35266. Authored-by: byungsoo Signed-off-by: HyukjinKwon --- .../scala/org/apache/spark/benchmark/BenchmarkBase.scala | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/core/src/test/scala/org/apache/spark/benchmark/BenchmarkBase.scala b/core/src/test/scala/org/apache/spark/benchmark/BenchmarkBase.scala index cce58c35f9..9b42a2772b 100644 --- a/core/src/test/scala/org/apache/spark/benchmark/BenchmarkBase.scala +++ b/core/src/test/scala/org/apache/spark/benchmark/BenchmarkBase.scala @@ -49,7 +49,14 @@ abstract class BenchmarkBase { val resultFileName = s"${this.getClass.getSimpleName.replace("$", "")}$jdkString$suffix-results.txt" val prefix = Benchmarks.currentProjectRoot.map(_ + "/").getOrElse("") - val file = new File(s"${prefix}benchmarks/$resultFileName") + val dir = new File(s"${prefix}benchmarks/") + if (!dir.exists()) { + // scalastyle:off println + println(s"Creating ${dir.getAbsolutePath} for benchmark results.") + // scalastyle:on println + dir.mkdirs() + } + val file = new File(s"${dir}$resultFileName") if (!file.exists()) { file.createNewFile() }