`linux.xarch` failing on newer kernel

[bin-wang]

linux.xarch is failing on my workstation running Linux 7.0.11

Whole log

$ ctest -V -R linux.xarch
UpdateCTestConfiguration  from :/home/binwang/projects/dynamorio/build/DartConfiguration.tcl
Parse Config file:/home/binwang/projects/dynamorio/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/binwang/projects/dynamorio/build/DartConfiguration.tcl
Parse Config file:/home/binwang/projects/dynamorio/build/DartConfiguration.tcl
Test project /home/binwang/projects/dynamorio/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 32
    Start 32: linux.xarch

32: Test command: /usr/bin/ctest "-V" "--build-and-test" "/home/binwang/projects/dynamorio" "/home/binwang/projects/dynamorio/build/suite/tests/32bit" "--build-generator" "Unix Makefiles" "--build-target" "dynamorio" "--build-target" "linux.execve32" "--build-target" "linux.execve-sub32" "--build-target" "client.large_options.dll" "--build-noclean" "--build-makeprogram" "/usr/bin/gmake" "--build-options" "-DDEBUG=ON" "-DBUILD_TESTS=ON" "-DBUILD_DOCS=OFF" "-DBUILD_SAMPLES=OFF" "-DBUILD_EXT=OFF" "-DBUILD_CLIENTS=OFF" "--test-command" "/usr/bin/cmake" "-P" "/home/binwang/projects/dynamorio/build/suite/tests/linux.xarch.cmake"
32: Working Directory: /home/binwang/projects/dynamorio/build/suite/tests
32: Environment variables: 
32:  CFLAGS=-m32
32:  CXXFLAGS=-m32
32: Test timeout computed to be: 1500
32: Internal cmake changing into directory: /home/binwang/projects/dynamorio/build/suite/tests/32bit
32: ======== CMake output     ======
32: CMake version 3.28.3
32: Building for x86
32: Processor vendor is AuthenticAMD
32: Version number: 11.91.20612
32: Compiler and processor support AVX.
32: Compiler and processor support AVX2.
32: Compiler and processor support AVX-512.
32: PT related libraries only supported on Linux x86_64
32: For dbghelp, choosing among: 
32: Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.3")
32: WARNING: vera++ not found: disabling code style checks
32: Processing tests and generating expected output patterns
32: Configuring done (1.7s)
32: Generating done (0.3s)
32: Build files have been written to: /home/binwang/projects/dynamorio/build/suite/tests/32bit
32: ======== End CMake output ======
32: Change Dir: '/home/binwang/projects/dynamorio/build/suite/tests/32bit'
32: 
32: Run Build Command(s): /usr/bin/gmake -f Makefile dynamorio
32: [  0%] Built target dynamorio_ldscript
32: [  0%] Built target api_headers
32: [  0%] Built target generate_syslog
32: [  0%] Built target drmemfuncs
32: [ 11%] Built target drlibc
32: [100%] Built target dynamorio
32: 
32: Change Dir: '/home/binwang/projects/dynamorio/build/suite/tests/32bit'
32: 
32: Run Build Command(s): /usr/bin/gmake -f Makefile linux.execve32
32: [  0%] Built target generate_syslog
32: [ 50%] Built target drlibc
32: [100%] Built target tools
32: [100%] Built target linux.execve32
32: 
32: Change Dir: '/home/binwang/projects/dynamorio/build/suite/tests/32bit'
32: 
32: Run Build Command(s): /usr/bin/gmake -f Makefile linux.execve-sub32
32: [  0%] Built target generate_syslog
32: [ 50%] Built target drlibc
32: [100%] Built target tools
32: [100%] Built target linux.execve-sub32
32: 
32: Change Dir: '/home/binwang/projects/dynamorio/build/suite/tests/32bit'
32: 
32: Run Build Command(s): /usr/bin/gmake -f Makefile client.large_options.dll
32: [  0%] Built target dynamorio_ldscript
32: [  0%] Built target api_headers
32: [  0%] Built target generate_syslog
32: [  0%] Built target drmemfuncs
32: [ 11%] Built target drlibc
32: [100%] Built target dynamorio
32: [100%] Built target client.large_options.dll
32: 
32: Running test command: "/usr/bin/cmake" "-P" "/home/binwang/projects/dynamorio/build/suite/tests/linux.xarch.cmake"
32: <Application /home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/linux.execve32 (72754).  Internal Error: DynamoRIO debug check failure: /home/binwang/projects/dynamorio/core/unix/os.c:10121 iter->vm_end - iter->vm_start == PAGE_SIZE || iter->vm_end - iter->vm_start == 2 * PAGE_SIZE
32: (Error occurred @0 frags in tid 72754)
32: version 11.91.20612, custom build
32: -no_dynamic_options -client_lib '/home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so;0;"-paramA" "foo" "-paramB" "bar"' -client_lib32 '/home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so;0;"-paramA" "foo" "-paramB" "ba
32: 0xffa4aab8 0xf7bf3137
32: 0xffa4acf8 0xf7da9214
32: 0xffa4ad18 0xf7da9530
32: 0xffa4ad88 0xf7da9ca1
32: 0xffa4ae48 0xf7c8d078
32: 0xffa4aed8 0xf7b75559
32: 0xffa4b708 0xf7dcc62f
32: 0xffa4c018 0xf7d7f289
32: /home/binwang/projects/dynamorio/build/suite/tests/32bit/lib32/debug/libdynamorio.so=0xf7b44000
32: /home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so=0xf7aff000
32: /usr/lib32/libc.so.6=0xf769f000
32: /usr/lib/ld-linux.so.2=0xf7ac9000>
32: large_options passed: -paramA foo -paramB bar
32: parent is running under DynamoRIO
32: parent waiting for child
32: child is running under DynamoRIO
32: <Application /home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/linux.execve-sub32 (72756).  Internal Error: DynamoRIO debug check failure: /home/binwang/projects/dynamorio/core/unix/os.c:10121 iter->vm_end - iter->vm_start == PAGE_SIZE || iter->vm_end - iter->vm_start == 2 * PAGE_SIZE
32: (Error occurred @0 frags in tid 72756)
32: version 11.91.20612, custom build
32: -client_lib '/home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so;0;"-paramA" "foo" "-paramB" "bar"' -client_lib32 '/home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so;0;"-paramA" "foo" "-paramB" "bar"' -client_lib64 '/
32: 0xff846068 0xf7c96137
32: 0xff8462a8 0xf7e4c214
32: 0xff8462c8 0xf7e4c530
32: 0xff846338 0xf7e4cca1
32: 0xff8463f8 0xf7d30078
32: 0xff846488 0xf7c18559
32: 0xff846cb8 0xf7e6f62f
32: 0xff8475c8 0xf7e22289
32: /home/binwang/projects/dynamorio/build/suite/tests/32bit/lib32/debug/libdynamorio.so=0xf7be7000
32: /home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so=0xf7ba2000
32: /usr/lib32/libc.so.6=0xf7742000
32: /usr/lib/ld-linux.so.2=0xf7b6c000>
32: child has exited
32: large_options exiting
32: <Application /home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/linux.execve32 (72759).  Internal Error: DynamoRIO debug check failure: /home/binwang/projects/dynamorio/core/unix/os.c:10121 iter->vm_end - iter->vm_start == PAGE_SIZE || iter->vm_end - iter->vm_start == 2 * PAGE_SIZE
32: (Error occurred @0 frags in tid 72759)
32: version 11.91.20612, custom build
32: -no_dynamic_options -client_lib '/home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so;0;"-paramA" "foo" "-paramB" "bar"' -client_lib32 '/home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so;0;"-paramA" "foo" "-paramB" "ba
32: 0xff81a358 0xf7c27137
32: 0xff81a598 0xf7ddd214
32: 0xff81a5b8 0xf7ddd530
32: 0xff81a628 0xf7dddca1
32: 0xff81a6e8 0xf7cc1078
32: 0xff81a778 0xf7ba9559
32: 0xff81afa8 0xf7e0062f
32: 0xff81b8b8 0xf7db3289
32: /home/binwang/projects/dynamorio/build/suite/tests/32bit/lib32/debug/libdynamorio.so=0xf7b78000
32: /home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so=0xf7b33000
32: /usr/lib32/libc.so.6=0xf76d3000
32: /usr/lib/ld-linux.so.2=0xf7afd000>
32: large_options passed: -paramA foo -paramB bar
32: parent is running under DynamoRIO
32: parent waiting for child
32: child is running under DynamoRIO
32: <Application /home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/linux.execve-sub32 (72761).  Internal Error: DynamoRIO debug check failure: /home/binwang/projects/dynamorio/core/unix/os.c:10121 iter->vm_end - iter->vm_start == PAGE_SIZE || iter->vm_end - iter->vm_start == 2 * PAGE_SIZE
32: (Error occurred @0 frags in tid 72761)
32: version 11.91.20612, custom build
32: -client_lib '/home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so;0;"-paramA" "foo" "-paramB" "bar"' -client_lib32 '/home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so;0;"-paramA" "foo" "-paramB" "bar"' -client_lib64 '/
32: 0xffa833a8 0xf7c9e137
32: 0xffa835e8 0xf7e54214
32: 0xffa83608 0xf7e54530
32: 0xffa83678 0xf7e54ca1
32: 0xffa83738 0xf7d38078
32: 0xffa837c8 0xf7c20559
32: 0xffa83ff8 0xf7e7762f
32: 0xffa84908 0xf7e2a289
32: /home/binwang/projects/dynamorio/build/suite/tests/32bit/lib32/debug/libdynamorio.so=0xf7bef000
32: /home/binwang/projects/dynamorio/build/suite/tests/32bit/suite/tests/bin/libclient.large_options.dll.so=0xf7baa000
32: /usr/lib32/libc.so.6=0xf774a000
32: /usr/lib/ld-linux.so.2=0xf7b74000>
32: child has exited
32: large_options exiting
32: 
1/1 Test #32: linux.xarch ......................***Failed  Required regular expression not found. Regex=[.*large_options passed: -paramA foo -paramB bar
parent is running under DynamoRIO
parent waiting for child
child is running under DynamoRIO
large_options passed: -paramA foo -paramB bar
it_worked
running under DynamoRIO
large_options exiting
child has exited
large_options exiting
[ 
]*large_options passed: -paramA foo -paramB bar
parent is running under DynamoRIO
parent waiting for child
child is running under DynamoRIO
large_options passed: -paramA foo -paramB bar
it_worked
running under DynamoRIO
large_options exiting
child has exited
large_options exiting
[ 
]*large_options passed: -paramA foo -paramB bar
parent is running under DynamoRIO
parent waiting for child
child is running under DynamoRIO
large_options passed: -paramA foo -paramB bar
it_worked
running under DynamoRIO
large_options exiting
child has exited
large_options exiting
[ 
]*large_options passed: -paramA foo -paramB bar
parent is running under DynamoRIO
parent waiting for child
child is running under DynamoRIO
large_options passed: -paramA foo -paramB bar
it_worked
running under DynamoRIO
large_options exiting
child has exited
large_options exiting
[ 
]*$
]  3.18 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   3.25 sec

The following tests FAILED:
	 32 - linux.xarch (Failed)
Errors while running CTest
Output from these tests are in: /home/binwang/projects/dynamorio/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

The most relevant part is

Internal Error: DynamoRIO debug check failure: /home/binwang/projects/dynamorio/core/unix/os.c:10121 iter->vm_end - iter->vm_start == PAGE_SIZE || iter->vm_end - iter->vm_start == 2 * PAGE_SIZE
32: (Error occurred @0 frags in tid 72754)

This assertion is in found_vsyscall_page()

dynamorio/core/unix/os.c

Lines 10114 to 10123 in a9afda3

	static void
	found_vsyscall_page(memquery_iter_t *iter _IF_DEBUG(DR_PARAM_OUT const char **map_type))
	{
	#ifndef X64
	/* We assume no vsyscall page for x64; thus, checking the
	* hardcoded address shouldn't have any false positives.
	*/
	ASSERT(iter->vm_end - iter->vm_start == PAGE_SIZE ||
	/* i#1583: recent kernels have 2-page vdso */
	iter->vm_end - iter->vm_start == 2 * PAGE_SIZE);

So the vdso region is neither 1 page or 2 pages. Checking my /proc/<pid>/maps, the results are:

f7f1f000-f7f23000 r--p 00000000 00:00 0                                  [vvar]
f7f23000-f7f25000 r--p 00000000 00:00 0                                  [vvar_vclock]
f7f25000-f7f28000 r-xp 00000000 00:00 0                                  [vdso]

so it's actually 3 pages now.

[bin-wang]

BTW, in the same core/unix/os.c there is this comment

dynamorio/core/unix/os.c

Lines 433 to 434 in a9afda3

	/* i#2945: vdso is now 3 pages and vsyscall is not on the 1st page. */
	app_pc vsyscall_page_start = NULL;

But when I check #2945, the vdso region reported in that issue is actually 2 pages, seems like a typo?

[bin-wang]

With fix in #7931, the initial assertion failure goes away, but 2 out of the 4 cases still fail:

ctest -V -R linux.xarch
UpdateCTestConfiguration  from :/home/binwang/projects/dynamorio/build/DartConfiguration.tcl
Parse Config file:/home/binwang/projects/dynamorio/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/binwang/projects/dynamorio/build/DartConfiguration.tcl
Parse Config file:/home/binwang/projects/dynamorio/build/DartConfiguration.tcl
Test project /home/binwang/projects/dynamorio/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 32
    Start 32: linux.xarch

32: Test command: /usr/bin/ctest "-V" "--build-and-test" "/home/binwang/projects/dynamorio" "/home/binwang/projects/dynamorio/build/suite/tests/32bit" "--build-generator" "Unix Makefiles" "--build-target" "dynamorio" "--build-target" "linux.execve32" "--build-target" "linux.execve-sub32" "--build-target" "client.large_options.dll" "--build-noclean" "--build-makeprogram" "/usr/bin/gmake" "--build-options" "-DDEBUG=ON" "-DBUILD_TESTS=ON" "-DBUILD_DOCS=OFF" "-DBUILD_SAMPLES=OFF" "-DBUILD_EXT=OFF" "-DBUILD_CLIENTS=OFF" "--test-command" "/usr/bin/cmake" "-P" "/home/binwang/projects/dynamorio/build/suite/tests/linux.xarch.cmake"
32: Working Directory: /home/binwang/projects/dynamorio/build/suite/tests
32: Environment variables: 
32:  CFLAGS=-m32
32:  CXXFLAGS=-m32
32: Test timeout computed to be: 1500
32: Internal cmake changing into directory: /home/binwang/projects/dynamorio/build/suite/tests/32bit
32: ======== CMake output     ======
32: CMake version 3.28.3
32: Building for x86
32: Processor vendor is AuthenticAMD
32: Version number: 11.91.20613
32: Compiler and processor support AVX.
32: Compiler and processor support AVX2.
32: Compiler and processor support AVX-512.
32: PT related libraries only supported on Linux x86_64
32: For dbghelp, choosing among: 
32: Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.3")
32: WARNING: vera++ not found: disabling code style checks
32: Processing tests and generating expected output patterns
32: Configuring done (1.6s)
32: Generating done (0.3s)
32: Build files have been written to: /home/binwang/projects/dynamorio/build/suite/tests/32bit
32: ======== End CMake output ======
32: Change Dir: '/home/binwang/projects/dynamorio/build/suite/tests/32bit'
32: 
32: Run Build Command(s): /usr/bin/gmake -f Makefile dynamorio
32: [  0%] Built target dynamorio_ldscript
32: [  0%] Built target api_headers
32: [  0%] Built target generate_syslog
32: [  0%] Built target drmemfuncs
32: [ 11%] Built target drlibc
32: [ 11%] Building C object core/CMakeFiles/dynamorio.dir/arch/arch.c.o
32: [ 11%] Building C object core/CMakeFiles/dynamorio.dir/unix/os.c.o
32: [ 11%] Building C object core/CMakeFiles/dynamorio.dir/unix/loader.c.o
32: [ 11%] Linking C shared library ../lib32/debug/libdynamorio.so
32: [100%] Built target dynamorio
32: 
32: Change Dir: '/home/binwang/projects/dynamorio/build/suite/tests/32bit'
32: 
32: Run Build Command(s): /usr/bin/gmake -f Makefile linux.execve32
32: [  0%] Built target generate_syslog
32: [ 50%] Built target drlibc
32: [100%] Built target tools
32: [100%] Built target linux.execve32
32: 
32: Change Dir: '/home/binwang/projects/dynamorio/build/suite/tests/32bit'
32: 
32: Run Build Command(s): /usr/bin/gmake -f Makefile linux.execve-sub32
32: [  0%] Built target generate_syslog
32: [ 50%] Built target drlibc
32: [100%] Built target tools
32: [100%] Built target linux.execve-sub32
32: 
32: Change Dir: '/home/binwang/projects/dynamorio/build/suite/tests/32bit'
32: 
32: Run Build Command(s): /usr/bin/gmake -f Makefile client.large_options.dll
32: [  0%] Built target dynamorio_ldscript
32: [  0%] Built target api_headers
32: [  0%] Built target generate_syslog
32: [  0%] Built target drmemfuncs
32: [ 11%] Built target drlibc
32: [100%] Built target dynamorio
32: [100%] Linking C shared library bin/libclient.large_options.dll.so
32: [100%] Built target client.large_options.dll
32: 
32: Running test command: "/usr/bin/cmake" "-P" "/home/binwang/projects/dynamorio/build/suite/tests/linux.xarch.cmake"
32: large_options passed: -paramA foo -paramB bar
32: parent is running under DynamoRIO
32: parent waiting for child
32: child is running under DynamoRIO
32: it_worked
32: running natively
32: child has exited
32: large_options exiting
32: large_options passed: -paramA foo -paramB bar
32: parent is running under DynamoRIO
32: parent waiting for child
32: child is running under DynamoRIO
32: large_options passed: -paramA foo -paramB bar
32: it_worked
32: running under DynamoRIO
32: large_options exiting
32: child has exited
32: large_options exiting
32: large_options passed: -paramA foo -paramB bar
32: parent is running under DynamoRIO
32: parent waiting for child
32: child is running under DynamoRIO
32: it_worked
32: running natively
32: child has exited
32: large_options exiting
32: large_options passed: -paramA foo -paramB bar
32: parent is running under DynamoRIO
32: parent waiting for child
32: child is running under DynamoRIO
32: large_options passed: -paramA foo -paramB bar
32: it_worked
32: running under DynamoRIO
32: large_options exiting
32: child has exited
32: large_options exiting
32: 
1/1 Test #32: linux.xarch ......................***Failed  Required regular expression not found. Regex=[.*large_options passed: -paramA foo -paramB bar
parent is running under DynamoRIO
parent waiting for child
child is running under DynamoRIO
large_options passed: -paramA foo -paramB bar
it_worked
running under DynamoRIO
large_options exiting
child has exited
large_options exiting
[ 
]*large_options passed: -paramA foo -paramB bar
parent is running under DynamoRIO
parent waiting for child
child is running under DynamoRIO
large_options passed: -paramA foo -paramB bar
it_worked
running under DynamoRIO
large_options exiting
child has exited
large_options exiting
[ 
]*large_options passed: -paramA foo -paramB bar
parent is running under DynamoRIO
parent waiting for child
child is running under DynamoRIO
large_options passed: -paramA foo -paramB bar
it_worked
running under DynamoRIO
large_options exiting
child has exited
large_options exiting
[ 
]*large_options passed: -paramA foo -paramB bar
parent is running under DynamoRIO
parent waiting for child
child is running under DynamoRIO
large_options passed: -paramA foo -paramB bar
it_worked
running under DynamoRIO
large_options exiting
child has exited
large_options exiting
[ 
]*$
]  5.11 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   5.18 sec

The following tests FAILED:
	 32 - linux.xarch (Failed)
Errors while running CTest
Output from these tests are in: /home/binwang/projects/dynamorio/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

Notably the 2 failed cases have

32: it_worked
32: running natively

in the log, meaning DR lost control?