Use libasan to catch a Stack over flow bug

[shahawi-sumup]

Hello Zephyr Team,

I'm trying to use CONFIG_STACK_CANARIES=y with ztest on Zephyr v2.7.1 (Board=native_posix), the problem is in one of my unit tests i get OS Crash

DEBUG   - OUTPUT: @ WEST_TOPDIR/os/kernel/compiler_stack_protect.c:40
DEBUG   - OUTPUT: [00:00:00.000,000] <err> os: >>> ZEPHYR FATAL ERROR 2: Stack overflow on CPU 0
DEBUG   - OUTPUT: [00:00:00.000,000] <err> os: Current thread: 0x8178f80 (idle 00)
DEBUG   - OUTPUT: [00:00:00.000,000] <err> os: Halting system

The unit tests started with the following args:

        ZEPHYR_TOOLCHAIN_VARIANT="zephyr" ZEPHYR_TOOLCHAIN_DIR="/usr/" os/scripts/twister    \
            --enable-asan                                                                  \
            --enable-lsan                                                                  \
            --ninja                                                                          \
            -vv -a posix -p native_posix                                                     \
            -T test/unit/ztest                                                               \
            -O $ZTEST_OUT

tests prj.conf:

CONFIG_ENTROPY_GENERATOR=y
CONFIG_STACK_CANARIES=y
CONFIG_ASAN=y

How can i know the source of the problem ? libasan didn't catch it, unless I raise a trap signal in the fault handler:

void _StackCheckHandler(void)
{
	/* Stack canary error is a software fatal condition; treat it as such.
	 */
	__asm__ volatile("int $3");
	z_except_reason(K_ERR_STACK_CHK_FAIL);
	CODE_UNREACHABLE; /* LCOV_EXCL_LINE */
}

but that also didn't help that much

DEBUG   - OUTPUT: =================================================================
DEBUG   - OUTPUT: ==3794==ERROR: AddressSanitizer: UNKNOWN SIGNAL on unknown address 0x00000000 (pc 0x081001d3 bp 0xfffcb248 sp 0xfffcb240 T0)
DEBUG   - OUTPUT:     #0 0x81001d2 in _StackCheckHandler <zephyrpath>/kernel/compiler_stack_protect.c:39
DEBUG   - OUTPUT: AddressSanitizer can not provide additional info.
DEBUG   - OUTPUT: SUMMARY: AddressSanitizer: UNKNOWN SIGNAL <zephyrpath>/kernel/compiler_stack_protect.c:39 in _StackCheckHandler
DEBUG   - OUTPUT: ==3794==ABORTING

Without CONFIG_STACK_CANARIES tests run smoothly without any problem, but that looks to me as if the problem is hidden until it bites back in the future.

What are some suggestions to help me catch the source of the bug ?
Thanks in advance :)

[ZX41R]

The trap in _StackCheckHandler() will only show you where the corrupted canary was noticed. It usually will not show the write that crossed the stack boundary.

For this class of bug I would first switch from “catch the fatal error” to “watch the guard word”. On native_posix you can do that in gdb:

1. Get the failing thread's stack range from Zephyr's thread metadata or enable stack/thread diagnostics (CONFIG_THREAD_STACK_INFO, CONFIG_INIT_STACKS, CONFIG_STACK_SENTINEL, and if available in your tree CONFIG_THREAD_ANALYZER).
2. Break after the thread is created, identify the low/high end of that thread's stack and the sentinel/canary word.
3. Set a hardware watchpoint on that word.
4. Continue the test. The debugger should stop at the actual write that damages the guard, not at the later fatal handler.

ASan does not always help here because Zephyr is managing thread stacks itself, and the compiler stack protector check fires after the overwrite has already happened. If the bad write is through a normal instrumented object, ASan may catch it; if it is just a task stack growing into the sentinel, the RTOS check is the thing that detects it.

A few practical narrowing steps:

CONFIG_NO_OPTIMIZATIONS=y
CONFIG_INIT_STACKS=y
CONFIG_THREAD_STACK_INFO=y
CONFIG_STACK_SENTINEL=y
CONFIG_STACK_USAGE=y

Then increase stacks one thread at a time. If increasing only one test thread makes the failure disappear, inspect that thread's deepest call path and generate compiler stack-usage files (-fstack-usage) for the objects in that test. If the reported current thread is idle 00, do not assume idle is the source; it may just be the thread running when the canary check notices earlier corruption.

So the highest-signal path is: use the Zephyr fatal log to identify the damaged stack, use gdb watchpoints to catch the first write to the sentinel, then use -fstack-usage / stack analyzer output to decide whether it is legitimate stack exhaustion or an actual out-of-bounds write.