[SPARK-29923][SQL][TESTS] Set io.netty.tryReflectionSetAccessible for Arrow on JDK9+
### What changes were proposed in this pull request? This PR aims to add `io.netty.tryReflectionSetAccessible=true` to the testing configuration for JDK11 because this is an officially documented requirement of Apache Arrow. Apache Arrow community documented this requirement at `0.15.0` ([ARROW-6206](https://github.com/apache/arrow/pull/5078)). > #### For java 9 or later, should set "-Dio.netty.tryReflectionSetAccessible=true". > This fixes `java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available`. thrown by netty. ### Why are the changes needed? After ARROW-3191, Arrow Java library requires the property `io.netty.tryReflectionSetAccessible` to be set to true for JDK >= 9. After https://github.com/apache/spark/pull/26133, JDK11 Jenkins job seem to fail. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/676/ - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/677/ - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/678/ ```scala Previous exception in task: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
 io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:473)
 io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:243)
 io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:233)
 io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:245)
 org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:222)
 ``` ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass the Jenkins with JDK11. Closes #26552 from dongjoon-hyun/SPARK-ARROW-JDK11. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
parent
1112fc6029
commit
f77c10de38
|
@ -23,7 +23,7 @@ FAILED=0
|
|||
LOGFILE=$FWDIR/unit-tests.out
|
||||
rm -f $LOGFILE
|
||||
|
||||
SPARK_TESTING=1 NOT_CRAN=true $FWDIR/../bin/spark-submit --driver-java-options "-Dlog4j.configuration=file:$FWDIR/log4j.properties" --conf spark.hadoop.fs.defaultFS="file:///" $FWDIR/pkg/tests/run-all.R 2>&1 | tee -a $LOGFILE
|
||||
SPARK_TESTING=1 NOT_CRAN=true $FWDIR/../bin/spark-submit --driver-java-options "-Dlog4j.configuration=file:$FWDIR/log4j.properties" --conf spark.hadoop.fs.defaultFS="file:///" --conf spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" --conf spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" $FWDIR/pkg/tests/run-all.R 2>&1 | tee -a $LOGFILE
|
||||
FAILED=$((PIPESTATUS[0]||$FAILED))
|
||||
|
||||
NUM_TEST_WARNING="$(grep -c -e 'Warnings ----------------' $LOGFILE)"
|
||||
|
|
4
pom.xml
4
pom.xml
|
@ -2326,7 +2326,7 @@
|
|||
<include>**/*Suite.java</include>
|
||||
</includes>
|
||||
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
|
||||
<argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize}</argLine>
|
||||
<argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
|
||||
<environmentVariables>
|
||||
<!--
|
||||
Setting SPARK_DIST_CLASSPATH is a simple way to make sure any child processes
|
||||
|
@ -2376,7 +2376,7 @@
|
|||
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
|
||||
<junitxml>.</junitxml>
|
||||
<filereports>SparkTestSuite.txt</filereports>
|
||||
<argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize}</argLine>
|
||||
<argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
|
||||
<stderr/>
|
||||
<environmentVariables>
|
||||
<!--
|
||||
|
|
|
@ -978,6 +978,7 @@ object TestSettings {
|
|||
javaOptions in Test += "-Dspark.unsafe.exceptionOnMemoryLeak=true",
|
||||
javaOptions in Test += "-Dsun.io.serialization.extendedDebugInfo=false",
|
||||
javaOptions in Test += "-Dderby.system.durability=test",
|
||||
javaOptions in Test += "-Dio.netty.tryReflectionSetAccessible=true",
|
||||
javaOptions in Test ++= System.getProperties.asScala.filter(_._1.startsWith("spark"))
|
||||
.map { case (k,v) => s"-D$k=$v" }.toSeq,
|
||||
javaOptions in Test += "-ea",
|
||||
|
|
|
@ -86,9 +86,10 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
|
|||
env["TMPDIR"] = tmp_dir
|
||||
|
||||
# Also override the JVM's temp directory by setting driver and executor options.
|
||||
java_options = "-Djava.io.tmpdir={0} -Dio.netty.tryReflectionSetAccessible=true".format(tmp_dir)
|
||||
spark_args = [
|
||||
"--conf", "spark.driver.extraJavaOptions=-Djava.io.tmpdir={0}".format(tmp_dir),
|
||||
"--conf", "spark.executor.extraJavaOptions=-Djava.io.tmpdir={0}".format(tmp_dir),
|
||||
"--conf", "spark.driver.extraJavaOptions='{0}'".format(java_options),
|
||||
"--conf", "spark.executor.extraJavaOptions='{0}'".format(java_options),
|
||||
"pyspark-shell"
|
||||
]
|
||||
env["PYSPARK_SUBMIT_ARGS"] = " ".join(spark_args)
|
||||
|
|
|
@ -148,7 +148,7 @@
|
|||
<groupId>org.scalatest</groupId>
|
||||
<artifactId>scalatest-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize}</argLine>
|
||||
<argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
|
|
|
@ -177,7 +177,7 @@
|
|||
<groupId>org.scalatest</groupId>
|
||||
<artifactId>scalatest-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize}</argLine>
|
||||
<argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
|
|
|
@ -244,7 +244,7 @@
|
|||
<artifactId>scalatest-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<!-- Specially disable assertions since some Hive tests fail them -->
|
||||
<argLine>-da -Xmx4g -XX:ReservedCodeCacheSize=${CodeCacheSize}</argLine>
|
||||
<argLine>-da -Xmx4g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
|
|
Loading…
Reference in a new issue