Using GraalVM in a Github Actions device causes building Native Image to fail due to insufficient memory

[linghengqian]

Describe the issue
A clear and concise description of the issue. We recommend using the latest snapshot builds to replicate the issue as bugs are constantly being fixed in the master branch

• Using GraalVM in a Github Actions device causes building Native Image to fail due to insufficient memory.
• I created a reproducible Github Actions file in https://github.com/linghengqian/graalvm-v2301-ci-memory-test. The essence of this file is to build Apache ShardingSphere Proxative's Native Image on an 8GB Github Actions device. For specific Error Logs, see https://github.com/linghengqian/graalvm-v2301-ci-memory-test/actions/runs/5863254462 .
• This CI is using GraalVM CE 23.0.1. On GraalVM CE 22.3.1, Github Actions does not report an out-of-memory error.
• More early investigations are in https://github.com/apache/shardingsphere/actions/runs/5857913307/job/15880819781 and Make ShardingSphere Proxy in GraalVM Native Image form available apache/shardingsphere#21347.

Steps to reproduce the issue
Please include both build steps as well as run steps

1. [e.g.: git clone --depth 1 https://git.myrepo.com/projectone ]
2. [e.g.: mvn clean package]

• The essence of this replicated CI is to download the source code of Apache ShardingSphere, collect GraalVM Reachability Metadata, and build Native Image. Refer to https://github.com/linghengqian/graalvm-v2301-ci-memory-test/actions/runs/5863254462 and linghengqian-scratches/graalvm-v2301-ci-memory-test@b399055 .

git clone -b master https://github.com/apache/shardingsphere.git
cd ./shardingsphere/
git reset --hard fd574b8148331d9d2d23a2843ef83fef07323018
./mvnw -PgenerateStandardMetadata -DskipNativeTests -B -T1C clean test
./mvnw -am -pl distribution/proxy-native -B -T1C -Prelease.native -DskipTests clean package

• An interesting thing is that the following command will not cause Github Actions to run out of memory without ingesting GraalVM Reachability Metadata. Reference https://github.com/linghengqian/graalvm-v2301-ci-memory-test/actions/runs/5861785759 and linghengqian-scratches/graalvm-v2301-ci-memory-test@0193199 .

git clone -b master https://github.com/apache/shardingsphere.git
cd ./shardingsphere/
git reset --hard fd574b8148331d9d2d23a2843ef83fef07323018
./mvnw -am -pl distribution/proxy-native -B -T1C -Prelease.native -DskipTests clean package

Describe GraalVM and your environment:

• GraalVM version (latest snapshot builds can be found here), or commit id if built from source: [e.g. EE 19.3, CE 20.0, CE 20.1.0-dev-20200304_0848] CE 17.0.8
• JDK major version: [e.g.:8] 17
• OS: [e.g. macOS Catalina] Ubuntu
• Architecture: [e.g.: AMD64] AMD64

More details
Add any other information about the problem here. Especially important are stack traces or log output. Feel free to link to gists or to screenshots if necessary.

• The following logs are from https://github.com/linghengqian/graalvm-v2301-ci-memory-test/actions/runs/5863254462/job/15896383095.

[1/8] Initializing...                                                                                   (27.7s @ 0.36GB)
 Java version: 17.0.8+7, vendor version: GraalVM CE 17.0.8+7.1
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (linux, x86_64, 11.4.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 2 user-specific feature(s)
 - com.oracle.svm.polyglot.groovy.GroovyIndyInterfaceFeature
 - com.oracle.svm.thirdparty.gson.GsonFeature
[2/8] Performing analysis...  [*******]                                                                (324.4s @ 3.00GB)
  34,237 (86.31%) of 39,667 types reachable
  54,240 (57.28%) of 94,692 fields reachable
 173,836 (59.15%) of 293,909 methods reachable
   9,189 types,   478 fields, and 5,914 methods registered for reflection
      95 types,    94 fields, and    75 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z

=== Image generator watchdog detected no activity. This can be a sign of a deadlock during image building. Dumping all stack traces. Current time: Tue Aug 15 04:14:34 UTC 2023
"main" Id=1 in WAITING on lock=java.util.concurrent.ForkJoinTask$AdaptedRunnableAction@306b61a
    at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
    at [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
    at [email protected]/java.util.concurrent.ForkJoinTask.awaitDone(ForkJoinTask.java:468)
    at [email protected]/java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:670)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.meta.UniverseBuilder.build(UniverseBuilder.java:200)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:601)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:539)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:408)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:612)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:134)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:94)

"Reference Handler" Id=2 in RUNNABLE
    at [email protected]/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
    at [email protected]/java.lang.ref.Reference.processPendingReferences(Reference.java:253)
    at [email protected]/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:215)

"Finalizer" Id=3 in WAITING on lock=java.lang.ref.ReferenceQueue$Lock@dc1480a
    at [email protected]/java.lang.Object.wait(Native Method)
    at [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
    at [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
    at [email protected]/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:172)

"Signal Dispatcher" Id=4 in RUNNABLE

"JVMCI-native CompilerThread0" Id=7 in RUNNABLE

"Common-Cleaner" Id=10 in TIMED_WAITING on lock=java.lang.ref.ReferenceQueue$Lock@9fedd6a
    at [email protected]/java.lang.Object.wait(Native Method)
    at [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
    at [email protected]/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
    at [email protected]/java.lang.Thread.run(Thread.java:833)
    at [email protected]/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:162)

"Notification Thread" Id=11 in RUNNABLE

"native-image pid watcher" Id=12 in TIMED_WAITING on lock=java.util.TaskQueue@4cfc4231
    at [email protected]/java.lang.Object.wait(Native Method)
    at [email protected]/java.util.TimerThread.mainLoop(Timer.java:563)
    at [email protected]/java.util.TimerThread.run(Timer.java:516)

"Thread-1" Id=13 in RUNNABLE
    at [email protected]/sun.management.ThreadImpl.dumpThreads0(Native Method)
    at [email protected]/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
    at [email protected]/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.DeadlockWatchdog.threadDump(DeadlockWatchdog.java:124)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.DeadlockWatchdog.watchdogThread(DeadlockWatchdog.java:81)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.DeadlockWatchdog$$Lambda$140/0x00000007c017dad0.run(Unknown Source)
    at [email protected]/java.lang.Thread.run(Thread.java:833)

"Timer-0" Id=23 in TIMED_WAITING on lock=java.util.TaskQueue@7a7c8bd2
    at [email protected]/java.lang.Object.wait(Native Method)
    at [email protected]/java.util.TimerThread.mainLoop(Timer.java:563)
    at [email protected]/java.util.TimerThread.run(Timer.java:516)

"ForkJoinPool.commonPool-worker-3" Id=30 in RUNNABLE
    at platform/jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphEncoder.writeEdges(GraphEncoder.java:462)
    at platform/jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphEncoder.encode(GraphEncoder.java:286)
    at platform/jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphEncoder.encodeSingleGraph(GraphEncoder.java:185)
    at platform/jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphEncoder.encodeSingleGraph(GraphEncoder.java:175)
    at app/org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.results.StrengthenGraphs.makeOrApplyResults(StrengthenGraphs.java:183)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.meta.UniverseBuilder.lambda$buildProfilingInformation$0(UniverseBuilder.java:375)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.meta.UniverseBuilder$$Lambda$6697/0x00000007c3dfd228.accept(Unknown Source)
    at [email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at [email protected]/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1779)
    at [email protected]/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at [email protected]/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
    at [email protected]/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
    at [email protected]/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
    at [email protected]/java.util.concurrent.ForkJoinPool$WorkQueue.helpComplete(ForkJoinPool.java:1223)
    at [email protected]/java.util.concurrent.ForkJoinPool.helpComplete(ForkJoinPool.java:1915)
    at [email protected]/java.util.concurrent.ForkJoinTask.awaitDone(ForkJoinTask.java:433)
    at [email protected]/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:687)
    at [email protected]/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
    at [email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
    at [email protected]/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at [email protected]/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
    at [email protected]/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:765)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.meta.UniverseBuilder.buildProfilingInformation(UniverseBuilder.java:371)
    at app/org.graalvm.nativeimage.builder/com.oracle.svm.hosted.meta.UniverseBuilder$$Lambda$6696/0x00000007c3dfd000.run(Unknown Source)
    at [email protected]/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1375)
    at [email protected]/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
    at [email protected]/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
    at [email protected]/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
    at [email protected]/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
    at [email protected]/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)


=== Memory statistics (in MB):
=== Used heap size: 3552
=== Free heap size: 478
=== Maximum heap size: 3552
=== Image generator watchdog is aborting image generation. To configure the watchdog, use the options -H:DeadlockWatchdogInterval=10 and -H:+DeadlockWatchdogExitOnTimeout
Error: Image build request for 'apache-shardingsphere-proxy-native' (pid: 9904, path: /home/runner/work/graalvm-v2301-ci-memory-test/graalvm-v2301-ci-memory-test/shardingsphere/distribution/proxy-native/target) failed with exit status 30
com.oracle.svm.driver.NativeImage$NativeImageError: Image build request for 'apache-shardingsphere-proxy-native' (pid: 9904, path: /home/runner/work/graalvm-v2301-ci-memory-test/graalvm-v2301-ci-memory-test/shardingsphere/distribution/proxy-native/target) failed with exit status 30
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1982)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.build(NativeImage.java:1613)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1557)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.main(NativeImage.java:1531)

• In the https://github.com/linghengqian/graalvm-v2301-ci-memory-test/actions/runs/5858871258/job/15883695274 of the same CI content, you can see another out-of-memory Error Log.

[1/8] Initializing...                                                                                   (24.7s @ 0.35GB)
 Java version: 17.0.8+7, vendor version: GraalVM CE 17.0.8+7.1
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (linux, x86_64, 11.4.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 2 user-specific feature(s)
 - com.oracle.svm.polyglot.groovy.GroovyIndyInterfaceFeature
 - com.oracle.svm.thirdparty.gson.GsonFeature
[2/8] Performing analysis...  [*******]                                                                (437.3s @ 2.85GB)
GC warning: 243.6s spent in 222 GCs during the last stage, taking up 55.67% of the time.
            Please ensure more than 4.57GB of memory is available for Native Image
            to reduce GC overhead and improve image build time.
  34,237 (86.31%) of 39,667 types reachable
  54,240 (57.26%) of 94,718 fields reachable
 173,836 (59.15%) of 293,910 methods reachable
   9,189 types,   478 fields, and 5,914 methods registered for reflection
      95 types,    94 fields, and    75 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
The Native Image build process ran out of memory.
Please make sure your build system has more memory available.
com.oracle.svm.driver.NativeImage$NativeImageError
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1982)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.build(NativeImage.java:1609)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1557)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.main(NativeImage.java:1531)

[fniephaus]

Hi @linghengqian,
the Native Image build process determined that only 4.57GB of memory is available at startup, which implies that other processes are taking up a significant amount of memory, too (Linux/Windows runners have 7GB of memory in total). I don't have any details on this (feel free to investigate yourself), but it seems the app you're trying to build is quite large.

Could you please try and increase the memory limit for the build process, for example by adding -J-Xmx7g to your build arguments? Maybe also try setting MAVEN_OPTS=-Xmx128m to limit the memory overhead by the outer Maven process.

[linghengqian]

• @fniephaus Sorry I took so long to reply because the Native Image artifact on the ShardingSphere side was interfered with by Native image generation fails when adding a dependency with io.micronaut.gcp:micronaut-gcp-tracing  micronaut-projects/micronaut-gcp#532 until Disable the implementation of Etcd during the build of ShardingSphere Proxy Native apache/shardingsphere#28194 fixed.

• I tested in Add -J-Xmx7g buildArg to fix graalvm build in GitHub Actions linghengqian/shardingsphere#1 just need to use -J-Xmx7g buildArg to fix this issue. Refer to https://github.com/linghengqian/shardingsphere/actions/runs/5922463926/job/16056465812?pr=1 . But I tested in Add -Xmx128m Arg linghengqian-scratches/graalvm-v2301-ci-memory-test#1 that just setting MAVEN_OPTS=-Xmx128m was not enough to solve the problem. Refer to https://github.com/linghengqian/graalvm-v2301-ci-memory-test/actions/runs/5920944435/job/16052863223?pr=1 .

• Looks like I should be able to close this issue?