AIX OpenXL 17.1 Agent_OnLoad not found in library jvmtitest

[pshipton]

https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk24_j9_sanity.functional_ppc64_aix_Personal_testList_0/7

15:13:28 SharedCPEntryInvokerTests_1
15:13:28 J9vmTest_1
15:13:28 J9vmTest_3
15:13:28 J9vmTest_5
15:13:28 J9vmTest_7
15:13:28 cmdLineTester_jvmtitests_0
15:13:28 cmdLineTester_jvmtitests_5
15:13:28 cmdLineTester_jvmtitests_6
15:13:28 cmdLineTester_jvmtitests_7
15:13:28 cmdLineTester_jvmtitests_8
15:13:28 cmdLineTester_jvmtitests_hcr_0
15:13:28 cmdLineTester_jvmtitests_hcr_2
15:13:28 cmdLineTester_jvmtitests_hcr_4
15:13:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_1
15:13:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_4
15:13:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_7
15:13:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_8
15:13:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_9
15:13:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_10
15:13:28 cmdLineTester_jvmtitests_debug_0
15:13:28 cmdLineTester_jvmtitests_Java11andUp_0
15:13:28 cmdLineTester_jvmtitests_Java11andUp_1
15:13:28 cmdLineTester_jvmtitests_Java11andUp_2
15:13:28 cmdLineTester_jvmtitests_Java11andUp_3
15:13:28 cmdLineTester_jvmtitests_Java21andUp_2

https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk24_j9_sanity.functional_ppc64_aix_Personal_testList_1/7

15:16:28 SharedCPEntryInvokerTests_0
15:16:28 SharedCPEntryInvokerTests_2
15:16:28 J9vmTest_0
15:16:28 J9vmTest_2
15:16:28 J9vmTest_4
15:16:28 J9vmTest_6
15:16:28 cmdLineTester_jvmtitests_1
15:16:28 cmdLineTester_jvmtitests_2
15:16:28 cmdLineTester_jvmtitests_4
15:16:28 cmdLineTester_jvmtitests_hcr_1
15:16:28 cmdLineTester_jvmtitests_hcr_3
15:16:28 cmdLineTester_jvmtitests_hcr_6
15:16:28 cmdLineTester_jvmtitests_hcr_7
15:16:28 cmdLineTester_jvmtitests_hcr_8
15:16:28 cmdLineTester_jvmtitests_hcr_9
15:16:28 cmdLineTester_jvmtitests_hcr_10
15:16:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_0
15:16:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_2
15:16:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_3
15:16:28 cmdLineTester_jvmtitests_hcr_openi9_none_SCC_6
15:16:28 cmdLineTester_jvmtitests_debug_openj9_none_SCC_0
15:16:28 cmdLineTester_decompilationTests_0
15:16:28 cmdLineTester_decompilationTests_1
15:16:28 cmdLineTester_jvmtitests_Java21andUp_0
15:16:28 cmdLineTester_jvmtitests_Java21andUp_1
15:16:28 cmdLineTester_jvmtitests_Java21andUp_3

JVMJ9TI065E Agent initialization function Agent_OnLoad not found in library jvmtitest

nm -X64 ./test-images/openj9/libjvmtitest.so
nm: ./test-images/openj9/libjvmtitest.so: 0654-204 There are no symbols in the file.

[pshipton]

@midronij pls take a look

[pshipton]

@zl-wang fyi

[midronij]

@pshipton @zl-wang here are my observations after looking into this a little more.

First of all, the reason that libjvmtitest.so appears to contain no symbols is due to this command that occurs just after the linking step:

/usr/bin/strip -X32_64 /home/jenkins/midronij/openxl_build/build/aix-ppc64-server-release/vm/runtime/libjvmtitest.so

Comparing to the xlC build, where this isn't an issue, the strip command here doesn't use the -t option, which means that more data is stripped from the jvmtitest library when building with OpenXL. I tried adding the -t option back to the OpenXL build, but it resulted in this error when trying to run the command:

0654-407 Internal error while processing file tables.

So even though before this command is run, libjvmtitest.so isn't empty, it appears that the root cause of the problem occurs before this step.

Next, looking a few steps earlier, the linking command used to create libjvmtitest.so is the following:

/opt/IBM/openxlC/17.1.3/bin/ibm-clang_r -fPIC -Wl,-bE:CMakeFiles/jvmtitest.dir/exports.exp  -m64 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -pthread -fno-strict-aliasing -O3 -g -fstack-protector -gdwarf-4 -Wimplicit -Wreturn-type  -m64 -Wl,-G,-brtl,-bernotok,-bnoentry,-bnolibpath -latomic -lperfstat -liconv  -shared -Wl,-bnoipath  -o ../../libjvmtitest.so CMakeFiles/jvmtitest.dir/fakeref.c.o CMakeFiles/jvmtitest.dir/__/__/copyright.c.o  -Wl,-blibpath:"\$ORIGIN:/opt/IBM/xlmass/10.1.3/lib:/usr/lib:/lib" ../../lib/libjvmti_test_src.a ../../lib/libjvmti_test_agent.a ../../lib/libjvmti_test_src.a ../../lib/libjvmti_test_agent.a

Which, as far as I can tell, essentially means that it is the combination of libjvmti_test_agent.a and libjvmti_test_src.a. Looking at libjvmti_test_agent.a with nm, we can see that the Agent_OnLoad symbol comes from this library:

Agent_OnLoad         D        2168          24

However, this symbol table entry is nowhere to be found in libjvmtitest.so. As a sanity check I compared against the jvmtitest library that is generated by the xlC build as well, and Agent_OnLoad was indeed there:

Agent_OnLoad         D  4563410352          24

So it appears that the Agent_OnLoad function is being dropped from the symbol table during the linkage step for libjvmtitest.so. I'm not entirely sure why this is happening myself, but this seems to be where the problem originates, at least.

Something else I observed (although I'm not sure if it's related to the issue at hand) is that all three libraries (i.e.: libjvmtitest.so and libjvmti_test_agent.a built by OpenXL AND libjvmtitest.so built by xlC) contain the .Agent_OnLoad symbol:

.Agent_OnLoad        T  4294969312

(Note the "." at the beginning of the symbol name)

For more detail, I have attached the output from nm -X64 for all three libraries:
libjvmti_test_agent.a_openxl.OUT.log
libjvmtitest_unstripped_openxl.OUT.log
libjvmtitest_unstripped_xlc.OUT.log

[midronij]

Upon further inspection, I've determined that the problem here is twofold.

First, the linker command for libjvmtitest.so is not using the correct exports file. Currently, the following command is used when OpenXL is specified as the C/C++ compiler:

/opt/IBM/openxlC/17.1.3/bin/ibm-clang_r -fPIC -Wl,-bE:CMakeFiles/jvmtitest.dir/exports.exp  -m64 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -pthread -fno-strict-aliasing -O3 -g -fstack-protector -gdwarf-4 -Wimplicit -Wreturn-type  -m64 -Wl,-G,-brtl,-bernotok,-bnoentry,-bnolibpath -latomic -lperfstat -liconv  -shared -Wl,-bnoipath  -o ../../libjvmtitest.so CMakeFiles/jvmtitest.dir/fakeref.c.o CMakeFiles/jvmtitest.dir/__/__/copyright.c.o  -Wl,-blibpath:"\$ORIGIN:/opt/IBM/xlmass/10.1.3/lib:/usr/lib:/lib" ../../lib/libjvmti_test_src.a ../../lib/libjvmti_test_agent.a ../../lib/libjvmti_test_src.a ../../lib/libjvmti_test_agent.a

Notably, the exports file to be used is specified with -Wl,-bE:CMakeFiles/jvmtitest.dir/exports.exp. On the other hand, the xlC build uses the following linker command:

/opt/IBM/xlC/16.1.0/bin/xlclang   -Wl,-bE:CMakeFiles/jvmtitest.dir/exports.exp  -g -qalias=noansi -qxflag=LTOL:LTOL0 -q64 -qxlcompatmacros -O3 "-Wc,inline(auto,noreport,600,5000)" "-Wc,list(),offset,gonumber" -qstackprotect -fno-rtti -qlanglvl=extended  -Wl,-bE:jvmtitest.exp  -q64 -Wl,-G,-brtl,-bernotok,-bnoentry,-bnolibpath  -qmkshrobj -Wl,-bnoipath  -o ../../libjvmtitest.so CMakeFiles/jvmtitest.dir/fakeref.c.o CMakeFiles/jvmtitest.dir/__/__/copyright.c.o  -Wl,-blibpath:"\$ORIGIN:/usr/lib:/lib" ../../lib/libjvmti_test_src.a ../../lib/libjvmti_test_agent.a ../../lib/libjvmti_test_src.a ../../lib/libjvmti_test_agent.a

which specifies the exports files with -Wl,-bE:CMakeFiles/jvmtitest.dir/exports.exp AND -Wl,-bE:jvmtitest.exp. If we take a look at both of these exports files, we can see that Agent_OnLoad is included only in jvmtitest.exp, which is not used by the OpenXL build. So the solution here is to simply add -Wl,-bE:jvmtitest.exp to the OpenXL linker command.

The second part of the problem is, as mentioned above, with the strip command that is used after the debug symbols are extracted from libjvmtitest.so. Originally, when building with xlC, strip -t would be used on each shared library file. However, when we switched over to OpenXL, this would fail:

strip: /home/jenkins/midronij/openxl_build/build/aix-ppc64-server-release/vm/runtime/libjvmtitest.so -- 0654-407 Internal error while processing file tables.

We were originally able to get around this by removing the -t flag, but this has resulted in too much information being stripped from libjvmtitest.so, which results in the test failures. I have opened an issue in the wyvern github regarding this issue, but until that is addressed, we can simply skip the strip step when building with OpenXL as a temporary workaround.

[midronij]

PR: eclipse-omr/omr#7732