OpenJ9 JDK 8 fastdebug gpf in JIT Compilation Thread compiling Unsafe direct-memory test under Balanced GC

[mengxy5236]

Java -version output

openjdk version "1.8.0_502-internal-fastdebug"
OpenJDK Runtime Environment (build 1.8.0_502-internal-fastdebug-_2026_06_24_18_20-b00)
Eclipse OpenJ9 VM (build 1.8.0_502-internal-_2026_06_24_18_20-b00-master-45981f54ca, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20260624_000000 (JIT enabled, AOT enabled)
OpenJ9   - 45981f54ca
OMR      - a39ef2b2c
JCL      - 0c2003e7d6 based on jdk8u502-b05)

Summary of problem

Hi, we have identified a concurrent Java GC/JIT stress test case that can cause OpenJ9 JDK 8 fastdebug to crash.

The attached package contains the reduced Java test sources, a replay Java agent, a recorded scheduling sequence, VM options, and a javacore from a reproduced crash. The Java agent is used to replay a recorded thread schedule. The fatal event occurs inside OpenJ9 while a JIT compilation thread is compiling the test method:

com/gcscheduling/generated/DirectMemoryCrash.alternateBitsUsingBytes(I)V

The crash is reproducible with OpenJ9 JDK 8 fastdebug and Balanced GC using:

-Xmx512m -Xms512m
-Xgcpolicy:balanced
-Xgcthreads16
-XX:+IdleTuningIgnoreUnrecognizedOptions

Observed crash:

Dump Event "gpf" (00002000)
Signal_Number: 00000005
Current thread: "JIT Compilation Thread"
Compiling method: com/gcscheduling/generated/DirectMemoryCrash.alternateBitsUsingBytes(I)V
Module: /lib/x86_64-linux-gnu/libc.so.6
Symbol: gsignal

From the reproduced javacore:

1XHEXCPMODULE  Compiling method: com/gcscheduling/generated/DirectMemoryCrash.alternateBitsUsingBytes(I)V
3XMTHREADINFO  "JIT Compilation Thread-001" ... current category="JIT"

Steps to Reproduce

The attached package is named bug4_submit.zip.

bug4_submit.zip

unzip bug4_submit.zip
cd bug4_submit
chmod +x run.sh
JAVA_HOME=/path/to/openj9-jdk8-fastdebug ./run.sh

The script compiles the Java sources and executes the test with replay scheduling enabled. The effective JVM command is:

$JAVA_HOME/bin/java \
  -Xgcpolicy:balanced \
  -Xmx512m -Xms512m \
  -Xgcthreads16 \
  -XX:+IdleTuningIgnoreUnrecognizedOptions \
  -javaagent:Schedule/target/Schedule-1.0-SNAPSHOT-agent.jar=mode=replay\;file=workspace/sequence-checkpoints/ConcurrentTest_20260626_045330_629_BALANCED_sequence.json\;dumpBytecode=true \
  -cp workspace/output/classes:workspace/output/classes:Schedule/target/Schedule-1.0-SNAPSHOT-agent.jar \
  com.gcscheduling.generated.ConcurrentTest_20260626_045330_629

The crash is probabilistic. In our environment, the attached reproducer triggered the crash in 3 out of 5 runs.

Note

This issue was reproduced on a self-built OpenJ9 JDK 8 fastdebug build. We are reporting it as a fastdebug crash because the same test did not reproduce in our limited runs on a release OpenJ9 JDK 8 build.

The Java agent is included only to replay the recorded concurrent schedule. The package also includes a javacore from a reproduced crash.

The package keeps the original replay layout because the crash appears to be timing-sensitive; simplifying the layout reduced reproducibility in our tests.

This may be related to #23961 because both failures involve OpenJ9 JDK 8, Balanced GC, replayed concurrent scheduling, and JIT/stack-walking related behavior. However, the crash signature is different: #23961 crashed in a GC Worker in libj9gc29.so, while this reproducer crashes in a JIT Compilation Thread while compiling DirectMemoryCrash.alternateBitsUsingBytes(I)V on a fastdebug build.

Software and Hardware Environment

OS:
Linux 1ad3273ad17c 4.15.0-196-generic #207-Ubuntu SMP Thu Oct 27 21:24:58 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Architecture:
x86_64 / amd64

Container:
Linux container
NUMA support enabled, as reported in the OpenJ9 javacore.

[pshipton]

@hzongaro @mpirvu fyi

[hzongaro]

Are you able to reproduce this only with a fastdebug build? I tried a few runs with a regular build, but wasn't able to reproduce the problem with that build. I don't have a fastdebug build handy, so it might take me a bit of time to try to reproduce this.

Out of curiosity, are you able to reproduce the problem with the -XXgc:disableVirtualLargeObjectHeap option included?

[mengxy5236]

Thanks for taking a look.

Yes, so far we have only been able to reproduce this with our self-built OpenJ9 JDK 8 fastdebug build. We also tried several runs with a regular/release OpenJ9 JDK 8 build, but were not able to reproduce the crash there.

We also tested the same reproducer with -XXgc:disableVirtualLargeObjectHeap added. The crash is still reproducible with that option enabled on the fastdebug build.

So at the moment it looks fastdebug-only from our side, and -XXgc:disableVirtualLargeObjectHeap does not avoid the crash in our environment.

[hzongaro]

The VM state in the javacore is 514FF:

java -Xjit:vmstate=0x514FF -version
vmState [0x514ff]: {J9VMSTATE_JIT} {globalValuePropagation} {Performing Optimization}
...

[hzongaro]

Do you have a system core file or a jitdump file from the crash that you are able to share?

[mengxy5236]

Yes. I have attached the jitdump file I have.

jitdump.20260701.094343.37645.0004.dmp

[hzongaro]

Thank you for the jitdump. Sorry - I should have asked if you could provide both a jitdump and a javacore from the same crash.

[mengxy5236]

No problem. I attached both the javacore and jitdump from the same crash. Please let me know if you need anything else.
javacore.20260701.094343.37645.0002.txt
jitdump.20260701.094343.37645.0004.dmp

[hzongaro]

Looking at the registers at the point of the crash, the only one that holds the address of a node is R15 — 00007FF625032B20 — which happens to be the l2i instruction in this block:

n36110n   BBStart <block_864> (freq 5613) (in loop 4)                                         [0x7ff6250306e0] bci=[114,88,452] rc=0 vc=14624 vn=2741 li=- udi=- nc=0
n36227n   istore  <auto slot 10>[#3289  Auto] [flags 0x3 0x0 ]                                [0x7ff625032b70] bci=[114,88,452] rc=0 vc=14624 vn=409 li=291 udi=116 nc=1
n36226n     l2i (X>=0 cannotOverflow )                                                        [0x7ff625032b20] bci=[114,87,452] rc=2 vc=14624 vn=409 li=- udi=- nc=1 flg=0x1100
n36225n       lload  <auto slot 6>[#430  Auto] [flags 0x4 0x0 ] (highWordZero X>=0 cannotOverflow )  [0x7ff625032ad0] bci=[114,86,452] rc=1 vc=14624 vn=144 li=- udi=1042 nc=0 flg=0x5100
n36232n   ificmplt --> block_1620 BBStart at n45768n ()                                       [0x7ff625032d00] bci=[114,94,453] rc=0 vc=14622 vn=2742 li=- udi=- nc=2 flg=0x20
n36226n     ==>l2i
n36229n     iconst 0x10000 (X!=0 X>=0 )                                                       [0x7ff625032c10] bci=[114,92,453] rc=1 vc=14622 vn=211 li=- udi=- nc=0 flg=0x104
n36115n   BBEnd </block_864> =====                                                            [0x7ff625030870] bci=[114,94,453] rc=0 vc=14622 vn=2743 li=- udi=- nc=0

So the problem could be somehow related to that node or somewhere nearby. May I ask you to try rerunning the test with these additional JVM options, and post the log files that are produced with the prefix alternatebits.log*? That should help up us further narrow this down.

-XX:+MergeCompilerOptions '-Xjit:{com/gcscheduling/generated/DirectMemoryCrash.alternateBitsUsingBytes*}(optDetails,traceGlobalVP,log=alternatebits.log)'

[mengxy5236]

I reran the test with the requested options. Two of the generated alternatebits.log* files uploaded successfully:

alternatebits.log.1.20260704.180018.1272.txt
alternatebits.log.7.20260704.180107.1272.txt

The remaining file, alternatebits.log.20260704.180017.1272, is about 780 MB, so GitHub failed to upload it.

[hzongaro]

The remaining file, alternatebits.log.20260704.180017.1272, is about 780 MB, so GitHub failed to upload it.

I think that's the one that likely contains information related to the crash. May I ask you to try compressing it with gzip or zip? Alternatively you could try rerunning the test without the optDetails option to produce shorter log files:

-XX:+MergeCompilerOptions '-Xjit:{com/gcscheduling/generated/DirectMemoryCrash.alternateBitsUsingBytes*}(traceGlobalVP,log=alternatebits.log)'