ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/extras/java8-tests
History
Sean Owen 7120a2979d SPARK-1798. Tests should clean up temp files 
Three issues related to temp files that tests generate – these should be touched up for hygiene but are not urgent.

Modules have a log4j.properties which directs the unit-test.log output file to a directory like `[module]/target/unit-test.log`. But this ends up creating `[module]/[module]/target/unit-test.log` instead of former.

The `work/` directory is not deleted by "mvn clean", in the parent and in modules. Neither is the `checkpoint/` directory created under the various external modules.

Many tests create a temp directory, which is not usually deleted. This can be largely resolved by calling `deleteOnExit()` at creation and trying to call `Utils.deleteRecursively` consistently to clean up, sometimes in an `@After` method.

_If anyone seconds the motion, I can create a more significant change that introduces a new test trait along the lines of `LocalSparkContext`, which provides management of temp directories for subclasses to take advantage of._

Author: Sean Owen <sowen@cloudera.com>

Closes #732 from srowen/SPARK-1798 and squashes the following commits:

5af578e [Sean Owen] Try to consistently delete test temp dirs and files, and set deleteOnExit() for each
b21b356 [Sean Owen] Remove work/ and checkpoint/ dirs with mvn clean
bdd0f41 [Sean Owen] Remove duplicate module dir in log4j.properties output path for tests
2014-05-12 14:16:19 -07:00
..
src/test SPARK-1798. Tests should clean up temp files 2014-05-12 14:16:19 -07:00
pom.xml [java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs 2014-03-03 22:31:30 -08:00
README.md [java8API] SPARK-964 Investigate the potential for using JDK 8 lambda expressions for the Java/Scala APIs 2014-03-03 22:31:30 -08:00

README.md

Java 8 Test Suites

These tests require having Java 8 installed and are isolated from the main Spark build. If Java 8 is not your system's default Java version, you will need to point Spark's build to your Java location. The set-up depends a bit on the build system:

  • Sbt users can either set JAVA_HOME to the location of a Java 8 JDK or explicitly pass -java-home to the sbt launch script. If a Java 8 JDK is detected sbt will automatically include the Java 8 test project.

    $ JAVA_HOME=/opt/jdk1.8.0/ sbt/sbt clean "test-only org.apache.spark.Java8APISuite"

  • For Maven users,

    Maven users can also refer to their Java 8 directory using JAVA_HOME. However, Maven will not automatically detect the presence of a Java 8 JDK, so a special build profile -Pjava8-tests must be used.

    $ JAVA_HOME=/opt/jdk1.8.0/ mvn clean install -DskipTests $ JAVA_HOME=/opt/jdk1.8.0/ mvn test -Pjava8-tests -DwildcardSuites=org.apache.spark.Java8APISuite

    Note that the above command can only be run from project root directory since this module depends on core and the test-jars of core and streaming. This means an install step is required to make the test dependencies visible to the Java 8 sub-project.