ClassLoadingTest crash vmState=0x00020002 MarkingScheme12completeScanEP18MM_EnvironmentBase

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_aarch64_linux_Personal_testList_8/9 - cent9-aarch64-5
ClassLoadingTest_special_5m_6
-Xgcpolicy:optavgpause -Xjit:count=0 -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk8_j9_special.system_aarch64_linux_Personal_testList_8/9/system_test_output.tar.gz

12:20:31  CLT stderr Type=Segmentation error vmState=0x00020002
12:20:31  CLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
12:20:31  CLT stderr Handler1=0000FFFF9E5AA3CC Handler2=0000FFFF9E50F540 InaccessibleAddress=006F005200650000 Sending_Process=00650000
12:20:31  CLT stderr x0=FFFFFFFFFFFFFFFF x1=0000FFFF9C6516F0 x2=0000FFFF9C6516E8 x3=0004000000000000
12:20:31  CLT stderr x4=0000FFFF1F691000 x5=0000000000000009 x6=0000FFFF25F3DCF0 x7=0000000000000010
12:20:31  CLT stderr x8=0000000000000000 x9=0000000000000000 x10=0000000000000000 x11=0000000000000094
12:20:31  CLT stderr x12=0000000000000010 x13=000000007FFFFFFF x14=0000000000000001 x15=0000FFFF9E712A90
12:20:31  CLT stderr x16=0000000000000001 x17=0000FFFF9EBDBCD0 x18=0000000000000010 x19=0000FFFECC0041E8
12:20:31  CLT stderr x20=0000FFFF980662A0 x21=0000FFFECC004410 x22=0000FFFF0D26B268 x23=0000FFFF0D26B270
12:20:31  CLT stderr x24=0000FFFF25F3DF30 x25=0000000099669966 x26=0000FFFF0D26B288 x27=006F005200650000
12:20:31  CLT stderr x28=0000FFFF2145A790 x29(FP)=0000FFFF0D26B380 x30(LR)=0000FFFF9CF0D404 x31(SP)=0000FFFF0D26B250
12:20:31  CLT stderr PC=0000FFFF9CF0CEE8 SP=0000FFFF0D26B250 PSTATE=0000000020001000
12:20:31  CLT stderr v0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v1=2f65636170736b72 (f: 1886612352.000000, d: 2.254787e-80)
12:20:31  CLT stderr v2=7361622f63672f72 (f: 1667706752.000000, d: 6.077214e+247)
12:20:31  CLT stderr v3=2e656d6568635367 (f: 1751339904.000000, d: 3.446825e-85)
12:20:31  CLT stderr v4=68637261615f384b (f: 1633630336.000000, d: 7.097989e+194)
12:20:31  CLT stderr v5=616e6f737265505f (f: 1919242368.000000, d: 2.139469e+161)
12:20:31  CLT stderr v6=6f737265505f7875 (f: 1348434048.000000, d: 7.370996e+228)
12:20:31  CLT stderr v7=2f657361622f6367 (f: 1647272832.000000, d: 2.261376e-80)
12:20:31  CLT stderr v8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:20:31  CLT stderr v16=00000000000000f0 (f: 240.000000, d: 1.185758e-321)
12:20:31  CLT stderr v17=0000ffff0ca5f0d0 (f: 212201680.000000, d: 1.390651e-309)
12:20:31  CLT stderr v18=0000ffff0ca5f0e0 (f: 212201696.000000, d: 1.390651e-309)
12:20:31  CLT stderr v19=0000ffff0ca5f0f0 (f: 212201712.000000, d: 1.390651e-309)
12:20:31  CLT stderr v20=0000ffff0ca5f100 (f: 212201728.000000, d: 1.390651e-309)
12:20:31  CLT stderr v21=0000ffff0ca5f110 (f: 212201744.000000, d: 1.390651e-309)
12:20:31  CLT stderr v22=0000ffff0ca5f120 (f: 212201760.000000, d: 1.390651e-309)
12:20:31  CLT stderr v23=0000ffff0ca5f130 (f: 212201776.000000, d: 1.390651e-309)
12:20:31  CLT stderr v24=0000ffff0ca5f140 (f: 212201792.000000, d: 1.390651e-309)
12:20:31  CLT stderr v25=0000ffff0ca5f150 (f: 212201808.000000, d: 1.390651e-309)
12:20:31  CLT stderr v26=0000ffff0ca5f160 (f: 212201824.000000, d: 1.390651e-309)
12:20:31  CLT stderr v27=0000ffff0ca5f170 (f: 212201840.000000, d: 1.390651e-309)
12:20:31  CLT stderr v28=0000ffff0ca5f180 (f: 212201856.000000, d: 1.390651e-309)
12:20:31  CLT stderr v29=0000000000000030 (f: 48.000000, d: 2.371515e-322)
12:20:31  CLT stderr v30=000000000000ea0b (f: 59915.000000, d: 2.960194e-319)
12:20:31  CLT stderr v31=000000000000ea0a (f: 59914.000000, d: 2.960145e-319)
12:20:31  CLT stderr Module=/home/jenkins/workspace/Test_openjdk8_j9_special.system_aarch64_linux_Personal_testList_8/jdkbinary/j2sdk-image/jre/lib/aarch64/default/libj9gc_full29.so
12:20:31  CLT stderr Module_base_address=0000FFFF9CE00000
12:20:31  CLT stderr Target=2_90_20260628_720 (Linux 5.14.0-697.el9.aarch64)
12:20:31  CLT stderr CPU=aarch64 (4 logical CPUs) (0x1daecd000 RAM)
12:20:31  CLT stderr ----------- Stack Backtrace -----------
12:20:31  CLT stderr _ZN16MM_MarkingScheme12completeScanEP18MM_EnvironmentBase+0x88 (0x0000FFFF9CF0CEE8 [libj9gc_full29.so+0x10cee8])
12:20:31  CLT stderr _ZN16MM_MarkingScheme15completeMarkingEP18MM_EnvironmentBase+0x18 (0x0000FFFF9CF0DDE8 [libj9gc_full29.so+0x10dde8])
12:20:31  CLT stderr _ZN19MM_ParallelMarkTask3runEP18MM_EnvironmentBase+0x90 (0x0000FFFF9CFAFB70 [libj9gc_full29.so+0x1afb70])
12:20:31  CLT stderr _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x1c0 (0x0000FFFF9CF25580 [libj9gc_full29.so+0x125580])
12:20:31  CLT stderr _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x11c (0x0000FFFF9CF243DC [libj9gc_full29.so+0x1243dc])
12:20:31  CLT stderr omrsig_protect+0x1e0 (0x0000FFFF9E5105C0 [libj9prt29.so+0x295c0])
12:20:31  CLT stderr dispatcher_thread_proc+0x38 (0x0000FFFF9CF240F8 [libj9gc_full29.so+0x1240f8])
12:20:31  CLT stderr thread_wrapper+0xc8 (0x0000FFFF9E4BD2E8 [libj9thr29.so+0x72e8])
12:20:31  CLT stderr start_thread+0x2e8 (0x0000FFFF9EBD6F48 [libc.so.6+0x80f48])
12:20:31  CLT stderr thread_start+0xc (0x0000FFFF9EC416DC [libc.so.6+0xeb6dc])

[pshipton]

@dmitripivkine fyi

[dmitripivkine]

GC policy is optavgpause.
The reason for this crash is an object

> !j9object 0xFFFF25F3DC30
!J9IndexableObject 0x0000FFFF25F3DC30 {
    struct J9Class* clazz = !j9arrayclass 0xFFFF981EA600   // [L
    Object flags = 0x00000000;
    U_32 size = 0x00000010;
	[0] = null
	[1] = null
	[2] = !fj9object 0xffff25f3dd40 = !j9object 0xffff25f3dd40
	[3] = !fj9object 0xffff25f3e310 = !j9object 0xffff25f3e310 <--- bad
	[4] = null
	[5] = null
	[6] = null
	[7] = null
	[8] = null
	[9] = !fj9object 0xffff25fbd918 = !j9object 0xffff25fbd918 <--- bad
	[10] = null
	[11] = null
	[12] = !fj9object 0xffff25f3df30 = !j9object 0xffff25f3df30 <--- bad, crashed
	[13] = !fj9object 0xffff25f3dcc0 = !j9object 0xffff25f3dcc0
	[14] = null
	[15] = null
}

has been discovered dead and it's references have been collected but it was resurrected later on.
This array is a hash table array:

> !j9object 0xFFFF25F3D598
!J9Object 0x0000FFFF25F3D598 {
	struct J9Class* clazz = !j9class 0xFFFF981E8A00 // java/util/HashMap
	Object flags = 0x00000000;
	J lockword = 0x0000000000000000 (offset = 0) (java/lang/Object) <hidden>
	Ljava/util/Set; keySet = !fj9object 0x0 (offset = 8) (java/util/AbstractMap)
	Ljava/util/Collection; values = !fj9object 0x0 (offset = 16) (java/util/AbstractMap)
	[Ljava/util/HashMap$Node; table = !fj9object 0xffff25f3dc30 (offset = 24) (java/util/HashMap) <----
	Ljava/util/Set; entrySet = !fj9object 0x0 (offset = 32) (java/util/HashMap)
	I size = 0x00000006 (offset = 40) (java/util/HashMap)
	I modCount = 0x00000006 (offset = 44) (java/util/HashMap)
	I threshold = 0x0000000C (offset = 48) (java/util/HashMap)
	F loadFactor = 0x3F400000 (offset = 52) (java/util/HashMap)
}

Most likely this hash table array was lost right after creation before population its address to Hash Map object (missed by GC in the native) or Write Barrier call on field update has been missed.

@hzongaro Has something been touched in JIT code to have such regression? We use to have problems like this common in the past but not for a few last years. So it can be very intermittent failure, kind of blast from the past potentially.
I will keep GC component label but add JIT component as well.

[hzongaro]

Has something been touched in JIT code to have such regression?

I took a quick look at optimizer changes, to see whether anything might stand out, but I didn't spot anything. I will try doing some grinder runs to see whether it can be more easily reproduced with recent builds than with older builds.

[dmitripivkine]

Has something been touched in JIT code to have such regression?

I took a quick look at optimizer changes, to see whether anything might stand out, but I didn't spot anything. I will try doing some grinder runs to see whether it can be more easily reproduced with recent builds than with older builds.

Thank you! Please share links to failed jobs if any for triage.

[hzongaro]

No luck reproducing this failure in a 5x50 grinder run.

[hzongaro]

Trying a 5x200 grinder run.

[hzongaro]

Still no luck reproducing this in a 5x200 grinder run. I think we'll just have to wait to see whether it happens again or more frequently.