d7df7a805f
### What changes were proposed in this pull request? This PR aims to set `MaxMetaspaceSize` to `2g` because it's increasing the native memory consumption unlimitedly by default. The unlimited increasing memory causes GitHub Action flakiness. The value I observed during `hive` module test was over 1.8G and growing. - https://docs.oracle.com/javase/10/gctuning/other-considerations.htm#JSGCT-GUID-BFB89453-60C0-42AC-81CA-87D59B0ACE2E > Starting with JDK 8, the permanent generation was removed and the class metadata is allocated in native memory. The amount of native memory that can be used for class metadata is by default unlimited. Use the option -XX:MaxMetaspaceSize to put an upper limit on the amount of native memory used for class metadata. In addition, I increased the following memory limit to 4g consistently from two places. ```xml - <jvmArg>-Xms2048m</jvmArg> - <jvmArg>-Xmx2048m</jvmArg> + <jvmArg>-Xms4g</jvmArg> + <jvmArg>-Xmx4g</jvmArg> ``` ```scala - javaOptions += "-Xmx3g", + javaOptions ++= "-Xmx4g -XX:MaxMetaspaceSize=2g".split(" ").toSeq, ``` ### Why are the changes needed? This will reduce the flakiness in CI environment by limiting the memory usage explicitly. When we limit it with `1g`, Hive module fails with `OOM` like the following. ``` java.lang.OutOfMemoryError: Metaspace Error: Exception in thread "dispatcher-event-loop-110" java.lang.OutOfMemoryError: Metaspace ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes #33405 from dongjoon-hyun/SPARK-36195. Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: Kyle Bendickson <kbendickson@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> |
||
|---|---|---|
| .. | ||
| build.properties | ||
| MimaBuild.scala | ||
| MimaExcludes.scala | ||
| plugins.sbt | ||
| SparkBuild.scala | ||