undetected SPI fault causes infinite loop during +4KV ground bounce fault injection (IDFGH-18179)

[Bryght-Labs-Richard]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

Identified on v5.1.6, code seems same on v6.0.1

Espressif SoC revision.

ESP32-S3

Operating System used.

Windows

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

CMD

Development Kit.

Custom board

Power Supply used.

USB

What is the expected behavior?

When the SPI peripheral fails, I would expect the driver to return a failure.

What is the actual behavior?

The driver locks up one core in an infinite loop, but the other core runs fine.

Steps to reproduce.

I'm afraid I cannot walk you through this step-by-step, but here's my best guess:

1. Start a small SPI transfer
2. during the SPI transferuse ESD gun to fault SPI peripheral in a way that prevents it from continuing to transfer
3. CPU then polls SPI device forever

Debug Logs.

E (51820) task_wdt: Task watchdog got triggered. The following tasks/users did not reset the watchdog in time:
E (51820) task_wdt:  - IDLE1 (CPU 1)
E (51824) task_wdt: Tasks currently running:
E (51832) task_wdt: CPU 0: websocket_task
E (51836) task_wdt: CPU 1: main
E (51840) task_wdt: Print CPU 1 (current core) backtrace


Backtrace: 0x42137706:0x3FCA9050 0x42137B28:0x3FCA9070 0x4037B859:0x3FCA90A0 0x421EB6DE:0x3FCCA770 0x42132323:0x3FCCA790 0x42132385:0x3FCCA7B0 0x420B9D57:0x3FCCA7D0 0x420B992A:0x3FCCA820 0x420B91C9:0x3FCCA850 0x420B924C:0x3FCCA880 0x420B92A6:0x3FCCA8B0 0x420BA29E:0x3FCCA8D0 0x42010397:0x3FCCA900 0x420D2778:0x3FCCA920 0x420D39A1:0x3FCCA950 0x420DEAEB:0x3FCCA9F0 0x420DEC41:0x3FCCAA20 0x42010A13:0x3FCCAA40 0x421FBF27:0x3FCCAB40 0x4038AAC5:0x3FCCAB70
--- 0x42137706: task_wdt_timeout_handling at /COMPONENT_ESP_SYSTEM_DIR/task_wdt/task_wdt.c:489
--- 0x42137b28: task_wdt_isr at /COMPONENT_ESP_SYSTEM_DIR/task_wdt/task_wdt.c:565
--- 0x4037b859: _xt_lowint1 at /COMPONENT_FREERTOS_DIR/FreeRTOS-Kernel/portable/xtensa/xtensa_vectors.S:1240
--- 0x421eb6de: spi_hal_usr_is_done at /COMPONENT_HAL_DIR/spi_hal_iram.c:197
--- 0x42132323: spi_device_polling_end at /COMPONENT_DRIVER_DIR/spi/gpspi/spi_master.c:1093
--- 0x42132385: spi_device_polling_transmit at /COMPONENT_DRIVER_DIR/spi/gpspi/spi_master.c:1124
--- 0x420b9d57: panel_io_spi_tx_param at /COMPONENT_ESP_LCD_DIR/src/esp_lcd_panel_io_spi.c:236
--- 0x420b992a: esp_lcd_panel_io_tx_param at /COMPONENT_ESP_LCD_DIR/src/esp_lcd_panel_io.c:23
--- 0x420b91c9: panel_st7789_caset at /COMPONENT_BL_LCD_PANEL_ST7789_DIR/BL_lcd_panel_st7789.c:331
--- 0x420b924c: panel_st7789_send_bitmap at /COMPONENT_BL_LCD_PANEL_ST7789_DIR/BL_lcd_panel_st7789.c:376
--- 0x420b92a6: panel_st7789_draw_bitmap at /COMPONENT_BL_LCD_PANEL_ST7789_DIR/BL_lcd_panel_st7789.c:398
--- 0x420ba29e: esp_lcd_panel_draw_bitmap at /COMPONENT_ESP_LCD_DIR/src/esp_lcd_panel_ops.c:34
--- 0x42010397: example_lvgl_flush_cb at /COMPONENT_MAIN_DIR/main.c:143
--- 0x420d2778: call_flush_cb at /COMPONENT_LVGL__LVGL_DIR/src/core/lv_refr.c:1409
--- (inlined by) draw_buf_flush at /COMPONENT_LVGL__LVGL_DIR/src/core/lv_refr.c:1373
--- 0x420d39a1: refr_invalid_areas at /COMPONENT_LVGL__LVGL_DIR/src/core/lv_refr.c:635
--- (inlined by) lv_display_refr_timer at /COMPONENT_LVGL__LVGL_DIR/src/core/lv_refr.c:403
--- 0x420deaeb: lv_timer_exec at /COMPONENT_LVGL__LVGL_DIR/src/misc/lv_timer.c:327
--- (inlined by) lv_timer_handler at /COMPONENT_LVGL__LVGL_DIR/src/misc/lv_timer.c:107
--- 0x420dec41: lv_timer_handler at /COMPONENT_LVGL__LVGL_DIR/src/misc/lv_timer.c:71
--- 0x42010a13: app_main at /COMPONENT_MAIN_DIR/main.c:656
--- 0x421fbf27: main_task at /COMPONENT_FREERTOS_DIR/app_startup.c:208
--- 0x4038aac5: vPortTaskWrapper at /COMPONENT_FREERTOS_DIR/FreeRTOS-Kernel/portable/xtensa/port.c:162

Diagnostic report archive.

No response

More Information.

Suspicious code from ESP-IDF v6.0.1:

esp_err_t SPI_MASTER_ISR_ATTR spi_device_polling_transmit(spi_device_handle_t handle, spi_transaction_t* trans_desc)
{
    esp_err_t ret;
    ret = spi_device_polling_start(handle, trans_desc, portMAX_DELAY);
    if (ret != ESP_OK) {
        return ret;
    }

    return spi_device_polling_end(handle, portMAX_DELAY);//This blocks until the end of time.
}

There is a similar issue that I have emailed about.

[wanckl]

@Bryght-Labs-Richard

... what you mean CPU then polls SPI device forever? do you mean the SPI turn to send something on bus forever? or just CPU to poll SPI forever? where is the loop

and where the fault injection on? it maybe not simply SPI issue

[Bryght-Labs-Richard]

and where the fault injection on?

Ground plane of our PCB.

... what you mean CPU then polls SPI device forever?

I mean that spi_device_polling_end(handle, portMAX_DELAY) loops forever when the SPI peripheral stops during fault injection. It looks like spi_hal_usr_is_done() is always false. Ex:

    TickType_t start = xTaskGetTickCount();
    while (!spi_hal_usr_is_done(&host->hal)) {
        TickType_t end = xTaskGetTickCount();
        if (end - start > ticks_to_wait) {
            return ESP_ERR_TIMEOUT;
        }
    }

In this particular case(a display transfer), it would be enough to return a failure, and the higher level driver could re-send.

[wanckl]

@Bryght-Labs-Richard spi_device_polling_end is able to access a timeout instead of portMAX_DELAY, you can using specific polling_start/end instead of spi_device_polling_transmit API.

it seems now maybe some memory leak if return as timeout, but don't block you temporary test.

But I interested on if the following transaction can be normal.

If following transaction fails, you may need a driver uninstall-reinstall process.

[Bryght-Labs-Richard]

If following transaction fails, you may need a driver uninstall-reinstall process.

I think this is probably a good idea either way.

I should also verify all necessary clocks are clocking - possibly we lost a point in the clock tree that feeds the SPI.

Happily & sadly, some of the hardware modifications have been working well, so I haven't been able to reproduce it recently.

[wanckl]

@Bryght-Labs-Richard Grate to sound this

If issue appear again and re-install can't resolve issue, we can open issue again