Skip to content

Silabs: fix endless-join wedge, keep orphaned end devices rejoining, arm the watchdog (#458) - #477

Open
barbalexs wants to merge 6 commits into
romasku:mainfrom
barbalexs:pr-silabs-network-recovery
Open

Silabs: fix endless-join wedge, keep orphaned end devices rejoining, arm the watchdog (#458)#477
barbalexs wants to merge 6 commits into
romasku:mainfrom
barbalexs:pr-silabs-network-recovery

Conversation

@barbalexs

Copy link
Copy Markdown
Contributor

Fixes the "Silabs device blinks forever and only a power cycle helps" failure (#458), plus the reason a hung Silabs device never recovers at all. Diagnosed and hardware-tested on two SONOFF ZBMINIL2 (EFR32MG22, no-neutral sleepy end devices) that kept dying in the field.

Root causes found

  1. Steering wedge. hal_zigbee_start_network_steering() set network_steering_in_progress = true before calling sl_zigbee_af_network_steering_start() and ignored its return. Several SDK paths return an error and never deliver sl_zigbee_af_network_steering_complete_cb — most importantly, when sl_zigbee_af_network_state() != SL_ZIGBEE_NO_NETWORK the plugin (network-steering-v2 with OPTIMIZE_SCANS) just broadcasts permit_join and returns without entering its state machine. The flag then stays true for the rest of the boot: status reads "JOINING" forever, no retry ever happens, permit_join from the coordinator does nothing because the device isn't scanning. Only a power cycle clears it.
  2. Status mapping. hal_zigbee_get_network_status() mapped only JOINED_NETWORK and JOINING_NETWORK; SL_ZIGBEE_JOINED_NETWORK_NO_PARENT (a sleepy end device that lost its parent — the everyday case) and LEAVING_NETWORK (button factory reset) fell into NOT_JOINED, so app_task started steering in exactly the states where the callback never comes → trigger for (1).
  3. Rejoin gives up forever. The SDK's end-device-support plugin does handle parent loss (secure rejoin → all-channel rejoins, 10 s apart) but stops for good after SL_ZIGBEE_AF_REJOIN_ATTEMPTS_MAX = 4 attempts, i.e. any outage longer than ~40 s orphans the device permanently.
  4. No watchdog. SL_LEGACY_HAL_DISABLE_WATCHDOG defaults to 1 in the SDK, so the WDOG was never armed on Silabs (Telink builds have one since v1.1.1). A hung device stayed hung until mains power cycle. The refresh calls already exist in the zigbee stack/app-framework ticks — they were just compiled to no-ops.

The Telink HAL already guards all of this (checks the steering result, clears the flag on every failure path, rejoins with backoff); the Silabs port had dropped those protections.

Changes

  • Map JOINED_NETWORK_NO_PARENT/S2S states to JOINED (rejoin is the SDK's job) and LEAVING_NETWORK to busy; only NO_NETWORK triggers steering.
  • Trust the steering flag only when steering_start returns SL_STATUS_OK; add a 240 s safety timeout that calls sl_zigbee_af_network_steering_stop() if the completion callback never arrives (also recovers the plugin's own stranded state machine, e.g. the silent early-return in scanResultsHandler).
  • Clear the flag on SL_STATUS_NETWORK_UP.
  • While orphaned, re-kick the end-device-support move via sl_zigbee_af_start_move_cb() with 30 s → 5 min exponential backoff, so devices always come back yet stay cheap on parasitic no-neutral supplies.
  • Steering retries in app_task now back off 5 s → 60 s instead of re-scanning every superloop pass (closes the existing TODO; continuous scanning overwhelms no-neutral supplies — we believe this is what killed our weakest-supplied unit).
  • Arm the hardware watchdog in both slcp variants (SL_LEGACY_HAL_DISABLE_WATCHDOG: 0). The WDOG is frozen in EM2 (em2Run = false default), so sleepy end devices get no spurious resets.
  • Hardening: reject zeroed poll-control NVM records (poll interval 0 = sleepy device with no wake schedule), bound cluster/attribute registration to the fixed buffers in hal_zigbee_init, widen the *_by_endpoint[10] dispatch tables to 11 (endpoints are numbered 1..10, so endpoint 10 wrote out of bounds).

Testing

On a bench-powered ZBMINIL2 (EndDevice build):

  • Before: reproducing Silabs endlessly blinks if no router devices are within reach after a while #458 in the field — after a parent/coordinator loss the device blinked "joining" forever, ignored permit_join, and needed a mains power cycle. On our weakest-supplied unit (single 5 W bulb) the same episode ended with a fully hung device (LED latched on, no reaction to the wall switch), consistent with no watchdog + the scan storm browning out the parasitic supply.
  • After: coordinator switched off for 18 minutes (well past the SDK's give-up point), device kept retrying with backoff (visible as short LED flashes with growing gaps instead of a continuous blink); coordinator switched back on → device rejoined and answered within ~40 seconds, no power cycle. Idle consumption unchanged (~76 µA average at the default 250 ms poll, EM2 sleep between polls). Both our relays now run this firmware (one flashed wired, one via OTA) with bindings and Z2M control intact.

Note: no overlap with #470 — that PR flips the debug-UART energy-mode flag (relevant for DEBUG builds; release builds already swap in iostream_dummy), this one doesn't touch that file and addresses the join/recovery logic.

🤖 Generated with Claude Code

barbalexs and others added 6 commits July 27, 2026 14:08
- Map JOINED_NETWORK_NO_PARENT/S2S to JOINED and LEAVING_NETWORK to busy:
  starting network steering in these states never delivers the completion
  callback, wedging the device in a fake joining state until power cycle.
- Trust the steering-in-progress flag only when steering_start returns OK,
  add a 240s safety timeout that force-stops a steering round whose
  callback never arrived.
- Clear the flag on NETWORK_UP.
- Re-kick end-device-support rejoin with 30s..5min backoff while orphaned:
  the SDK gives up for good after REJOIN_ATTEMPTS_MAX attempts.
- Steering retry backoff 5..60s in app_task instead of a retry every
  superloop pass: continuous scanning overwhelms parasitic no-neutral
  supplies.
- Send device announce until it actually succeeds.
- Bound cluster/attribute registration to the fixed buffers instead of
  silently overrunning them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SL_LEGACY_HAL_DISABLE_WATCHDOG=1 is the SDK default, so the WDOG was never
enabled and a hung device stayed hung until mains power cycle. The zigbee
stack/app-framework ticks already call halResetWatchdog; the WDOG is frozen
in EM2, so sleepy end devices get no spurious resets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…f-by-one

A zeroed poll-control NVM record passed validation and left a sleepy end
device with no wake schedule. Endpoints are numbered 1..10 but the
per-endpoint dispatch tables held 10 entries, so endpoint 10 wrote out of
bounds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@barbalexs

Copy link
Copy Markdown
Contributor Author

Note on the red lint job here: it is not caused by this PR — lint has been failing on main itself since #457 was merged (13bc4d1 and every push after it). All files touched by this PR pass uncrustify --check with the version CI uses (0.78.1).

The two pre-existing offenders are fixed separately in #478, so this PR should go green once that lands (or once it is rebased on a fixed main).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant