From 0b5683a4d5971ddcbe738ee6a412bad3019fdb68 Mon Sep 17 00:00:00 2001 From: Gengliang Wang Date: Wed, 9 Jun 2021 19:36:29 +0800 Subject: [PATCH] [SPARK-35694][INFRA] Increase the default JVM stack size of SBT/Maven ### What changes were proposed in this pull request? The jenkins SBT/Maven build keep failing with stack overflow error: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139542 We should increase the JVM stack size to 16MB. Also, https://github.com/apache/spark/pull/32521 set the stack size to 256MB for Java 11 build, which might be too big since every thread will allocate this memory for the stack. This PR also set it as 16MB to make the config consistent. ### Why are the changes needed? Fix SBT/Maven build. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Jenkins and GA tests. Closes #32838 from gengliangwang/increaseSBTStackSize. Authored-by: Gengliang Wang Signed-off-by: Gengliang Wang --- .github/workflows/build_and_test.yml | 2 +- build/sbt | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index c5c1c12dec..4e129cac32 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -455,7 +455,7 @@ jobs: java-version: 11 - name: Build with Maven run: | - export MAVEN_OPTS="-Xss256m -Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN" + export MAVEN_OPTS="-Xss16m -Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN" export MAVEN_CLI_OPTS="--no-transfer-progress" # It uses Maven's 'install' intentionally, see https://github.com/apache/spark/pull/26414. ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=11 install diff --git a/build/sbt b/build/sbt index ae9ca93fc9..ada22387a5 100755 --- a/build/sbt +++ b/build/sbt @@ -53,7 +53,7 @@ realpath () { declare -r noshare_opts="-Dsbt.global.base=project/.sbtboot -Dsbt.boot.directory=project/.boot -Dsbt.ivy.home=project/.ivy" declare -r sbt_opts_file=".sbtopts" declare -r etc_sbt_opts_file="/etc/sbt/sbtopts" -declare -r default_sbt_opts="-Xss4m" +declare -r default_sbt_opts="-Xss16m" usage() { cat <-P:silencer:globalFilters=.*deprecated.* - -Xss4m + -Xss16m -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=${CodeCacheSize}