ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36393][BUILD] Try to raise memory for GHA

Browse source 
### What changes were proposed in this pull request?

According to the feedback from GitHub, the change causing memory issue has been rolled back. We can try to raise memory again for GA.

### Why are the changes needed?

Trying higher memory settings for GA. It could speed up the testing time.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

GA

Closes #33623 from viirya/increasing-mem-ga.

Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
This commit is contained in:
Liang-Chi Hsieh 2021-08-05 01:31:35 -07:00
parent b377ea26e2
commit 7d13ac177b
4 changed files with 10 additions and 13 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.github/workflows/build_and_test.yml vendored
View file

@ -220,7 +220,7 @@ jobs:
SPARK_LOCAL_IP: localhost
SKIP_UNIDOC: true
SKIP_MIMA: true
METASPACE_SIZE: 128m
METASPACE_SIZE: 1g
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2

6
build/sbt-launch-lib.bash
View file

@ -117,9 +117,11 @@ addDebugger () {
# so they need not be dicked around with individually.
get_mem_opts () {
local mem=${1:-$sbt_default_mem}
local codecache=128
local codecache=$(( $mem / 8 ))
(( $codecache > 128 )) || codecache=128
(( $codecache < 2048 )) || codecache=2048
echo "-Xms256m -Xmx${mem}m -XX:ReservedCodeCacheSize=${codecache}m"
echo "-Xms${mem}m -Xmx${mem}m -XX:ReservedCodeCacheSize=${codecache}m"
}
require_arg () {

7
dev/run-tests.py
View file

@ -268,12 +268,7 @@ def exec_sbt(sbt_args=()):
"""Will call SBT in the current directory with the list of mvn_args passed
in and returns the subprocess for any further processing"""
sbt_cmd = [os.path.join(SPARK_HOME, "build", "sbt")]
if "GITHUB_ACTIONS" in os.environ:
sbt_cmd = sbt_cmd + ['-mem', '2300']
sbt_cmd = sbt_cmd + sbt_args
sbt_cmd = [os.path.join(SPARK_HOME, "build", "sbt")] + sbt_args
sbt_output_filter = re.compile(b"^.*[info].*Resolving" + b"|" +
b"^.*[warn].*Merging" + b"|" +

8
pom.xml
View file

@ -2617,8 +2617,8 @@
</args>
<jvmArgs>
<jvmArg>-Xss128m</jvmArg>
<jvmArg>-Xms1024m</jvmArg>
<jvmArg>-Xmx3200m</jvmArg>
<jvmArg>-Xms4g</jvmArg>
<jvmArg>-Xmx4g</jvmArg>
<jvmArg>-XX:MaxMetaspaceSize=2g</jvmArg>
<jvmArg>-XX:ReservedCodeCacheSize=${CodeCacheSize}</jvmArg>
</jvmArgs>
@ -2668,7 +2668,7 @@
<include>**/*Suite.java</include>
</includes>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<argLine>-ea -Xmx3200m -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
<argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
<environmentVariables>
<!--
Setting SPARK_DIST_CLASSPATH is a simple way to make sure any child processes
@ -2719,7 +2719,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
<argLine>-ea -Xmx3200m -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
<argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
<stderr/>
<environmentVariables>
<!--