Warm-boot reliability and flash-persistence failures on RP2350B, with forensics and fixes for the reset path — from a staging-HSM deployment

[jobordu]

CORRECTED 2026-08-05 — please read this first. Several claims in this original report
have since been disproven by a reproduction on unmodified upstream firmware
(pico-hsm 1ad8444 + pico-keys-sdk 843a3dc, no patches, after a full 16 MB erase).
The corrected mechanism with full forensics is in this comment.

In short: the card does not complete INITIALIZE and then sit dead. Core0 panics during
INITIALIZE inside flash_range_erase(), while holding the multicore lockout with interrupts
disabled — which parks core1 and mutes the card. The origin is scan_region() (file.c:336)
accepting an out-of-region chain link and deriving file->data from it.

Affected sections are marked [CORRECTED] inline below. The original wording is kept so the
maintainer's review replies still make sense.

Context — how we use pico-hsm (and why it matters for this report)

We run an open-source, hardware-rooted key-custody project. Production tokens are Nitrokey
HSM 2s (genuine SmartCard-HSM). The Pico HSM is used strictly as a STAGING and
procedure-rehearsal device — it speaks the same SmartCard-HSM protocol, so ceremonies and
drills can be exercised end-to-end before any production hardware is touched. Nothing in this
report asks you to treat the Pico HSM as a primary HSM, and none of our findings bear on
production use of the firmware for everyday smartcard workloads.

What staging demands that everyday use does not is wipe-and-reprovision cycles: our central
drill is "any card can die and be replaced from the seed with no loss", which means INITIALIZE
DEVICE, DKEK domain creation, share import, and wrapped-key unwrap in a tight loop, measured
statistically. That workload surfaces a family of reliability defects in the reset path and in
the flash persistence layer. All findings below were measured on a Waveshare RP2350-PiZero
(RP2350B, 16 MB flash) with SWD forensics (Raspberry Pi Debug Probe + OpenOCD rp2350 target +
arm-none-eabi-gdb), not inferred. We are happy to provide raw logs, register dumps, and
reproduction scripts.

Environment: OpenSC sc-hsm-tool and CardContact Smart Card Shell (scsh 3.18.77) on the host;
polhenarejos/pico-hsm at 1ad8444 plus our patch series (see linked PR); Pico SDK 2.x.

Bug 1 — INITIALIZE DEVICE leaves the card dead until a physical replug (already #9) [CORRECTED]

Known upstream as polhenarejos/pico-hsm#9. After re-personalisation the card completes the
wipe but never rebuilds state; the reader reports a card but no ATR, indefinitely. Our
measurement matches: only a replug recovers. The PR resets the device after INITIALIZE (the
response is transmitted first; the reset is deferred so it never lands mid-command).

[CORRECTED] Two errors here. (1) The pico-hsm#9 attribution is wrong — you were right,
that was a 2022 inter-core queue fix and is unrelated. (2) The card does not "complete the
wipe but never rebuild state". On unmodified firmware it panics mid-command: the in-flight
UPDATE BINARY (INS D7, EF 2F03) inside sc_hsm_initialize times out after ~5 s and core0 is
already dead in panic(). A post-INITIALIZE reset therefore masks a panic rather than fixing
the cause; PR #27 and pico-hsm#136 are now drafts. Full trace: comment.

Bug 2 — warm boot wedges in ~5–10% of resets; both cores never start (fixed by deep resets)

Reproduced repeatedly via a cycle harness (sc-hsm-tool --initialize in a loop). Forensics on a
live hang: watchdog REASON.TIMER set — the watchdog did fire — then both Cortex-M33 cores
reading all-zero (PC/SP/xPSR), XIP SSI all-zero, and the bootrom's bootram diagnostics
untouched. The chip reset; the boot never started.

Reset mechanism was ruled out: identical signature with SDK watchdog_reboot() and the
bootrom's rom_reboot(). AIRCR.SYSRESETREQ was ruled out separately — on RP2040/RP2350 it
resets only the asserting core (raspberrypi/pico-feedback#329); we measured it as a complete
no-op (RAM statics survived the write).

What finally closed it: the default watchdog reset leaves the switched core domain powered.
Arming POWMAN_WDSEL = RESET_SWCORE | RESET_PSM | RESET_POWMAN_ASYNC (with PSM_WDSEL
all-ones, per the datasheet note that POWMAN ignores watchdog resets that don't select CLOCKS
or earlier) makes every watchdog-family reset power-cycle the switched core domain — "the same
effect as a power-on reset" per the RP2350 datasheet. Because RESET_POWMAN restores powman
defaults, the bits are re-armed at every boot.

Validation: 100/100 consecutive reboots with zero hangs against a ~5–10% measured wedge
rate (p ≈ 0.0007 under the old rate). The fix is in the PR.

Bug 3 — multicore launch handshake reports success while core1 never starts

Twice caught live via SWD: core1 parked in the bootrom's wait-for-launch loop (PC 0x19e,
polling SIO FIFO_ST, bootrom stack 0xf0000000), one unread word in the FIFO (VLD=1),
while card_locked_func showed the launch "succeeded" and USB enumerated (TinyUSB runs on
core0). The card enumerates but never answers an APDU. The SDK's
multicore_launch_core1_raw() echo handshake can complete against stale/misaligned FIFO data —
plausibly residue across a warm reset or a word consumed by the trampoline's drain phase.
The PR verifies the launch (a flag set by the launched thread), retries, and escalates to a
bounded chip reset; it also adds a dead-man's switch (ping via the existing queue echo
protocol) for the mid-session variant, which we also caught live.

Bug 4 — 1 ms multicore-lockout timeouts orphan core1 permanently [CORRECTED]

low_flash.c calls multicore_lockout_start/end_timeout_us(1000) around flash erases. A
timed-out END leaves core1 parked in the lockout victim handler forever (SWD: PC inside
multicore_lockout_handler, card mute, core0 healthy). 1 ms is far inside normal IRQ/entry
latency during real erase workloads. The PR raises the timeouts to 100 ms and re-syncs with a
start+end pair after an end-timeout.

[CORRECTED] You asked directly whether the END-timeout warning was ever captured. It was
not — and the reproduction shows why: core1 is not orphaned by a timeout at all. Core0
panics between lockout START and END (and after save_and_disable_interrupts()), leaving
the lockout held and interrupts off permanently. The start+end re-sync was also wrong —
lockouts are non-nestable, as you said — and has been removed. PR #29 is now a draft and
addresses a scenario we never observed.

Concurrency audit you asked for: the only in-tree lockout callers are low_flash.c:116/125/138/146,
all inside low_flash_task()'s drain loop on core0, plus multicore_lockout_victim_init() at
:243. phymarker_write() erases/programs without the lockout but is #ifdef PICO_RP2040 and
runs from low_flash_init() before core1 launches — not a concurrency hazard.

Bug 5 — hard_assert panics brick the device forever [CORRECTED: not "brick"]

A corrupted flash_pages entry (an SRAM pointer, 0x20001A00, queued after interrupted wipe
cycles) reached hard_assert(flash_offs + count <= PICO_FLASH_SIZE_BYTES) and the device sat
in panic() → __breakpoint → _exit for a day — no APDU service, no reset, physical replug
required. The PR (a) validates queued page addresses against the real fs data region before
touching flash, and (b) overrides hard_assertion_failure() to reboot via rom_reboot instead
of bricking.

[CORRECTED] "Brick" is the wrong word and you were right to push back — BOOTSEL plus a full
flash_nuke and reflash always recovers the board. The accurate statement is that the panic
leaves the running firmware permanently unresponsive until an external reset.

The reproduction also confirms this is not patch-induced: the identical corrupt-SRAM-pointer
panic occurs on unmodified upstream (0x2000D5A5, from scan_region()). Your ordering critique
was also correct — validation now happens before the first read/queue and returns an error
(PR #30) — but that is a safety net, not the fix. The panic-recovery policy has been withdrawn
from the PR series.

Bug 6 — flash persistence layer: writes do not survive, and find_free_page hard-faults on corrupted metadata — NOT FIXED, need upstream help

This is the defect we cannot fix from the outside, and the one that most needs your eyes.

Symptoms (all on the same fs, all reproducible):

1. DKEK metadata reads as corrupt: sc-hsm-tool reports DKEK key check value : 0000000000000000 for a domain whose share was just imported successfully. On a truly
   factory-clean fs (fs region fully erased) the KCV reads correctly and host/card KCVs match —
   so the "always-zero KCV" behaviour is a persistence symptom, not a device quirk.
2. unwrapKey fails SW=6400 even with host and card KCVs identical
   (292848FACCC4F1BE) — the domain key is right, the unwrap still refuses. The same unwrap
   flow passed on this exact card weeks earlier.
3. PKCS#11 reports CKR_USER_PIN_NOT_INITIALIZED minutes after the same PIN verified — the
   initialize's own writes (PIN state, PKCS#15 files) do not survive a chip reset.

[CORRECTED — attribution for symptoms 1 and 3] These are consequences of the Bug 1/5 panic,
not independent persistence failures. The EF write that would create the PKCS#15 structures is
exactly the operation that panics, so those structures are never written — hence
CKR_USER_PIN_NOT_INITIALIZED from PKCS#11 even though a raw VERIFY APDU (00 20 00 81)
returns 9000, and hence the all-zero KCV. Both reproduce on unmodified upstream firmware.
Honest note on your question 2: we have not reproduced the unwrapKey SW=6400 (symptom 2) on
unmodified firmware — that one remains unattributed.

4. HardFault in the DKEK store path. Captured live via SWD (fault escalated to a debug
   halt; register dump available):
   • backtrace: isr_hardfault ← memcpy ← find_free_page (low_flash.c:403) ← flash_program_uintptr ← flash_clear_file (file.c:675) ← flash_write_data_to_file_internal ← file_put_data ← mkek_store_file ← store_dkek_key ← save_dkek_key(id=0, key=NULL) ← sc_hsm.c:478 (deferred save at PIN verify)
   • BFAR = 0x6CB44000 (bus fault, precise, CFSR=0x8200): find_free_page received
     addr_alg = 0x6CB44000, read from a prev_addr field in the fs metadata — i.e. the
     on-flash fs structure itself held a garbage pointer.

Environment note that may matter: the fs data region on this build is partition-dependent —
measured bounds start_data_pool ≈ 0x100FF000, end_flash = 0x10400000 (≈ [1 MB, 4 MB)), not
[FLASH_SIZE/2, FLASH_SIZE) as low_flash_init's own comment suggests. Any code assuming the
latter (including, candidly, a guard we added ourselves — corrected in the PR to use
flash_addr_in_fs() against the real bounds) misclassifies every address.

Ask: review of the page-cache/lazy-drain design in low_flash.c + flash.c —
find_free_page's whole-sector memcpy(p->page, (uint8_t *)addr_alg, FLASH_SECTOR_SIZE) and
the erase-then-reprogram drain — for how interrupted operations (and resets landing while lazy
writes are queued, which our reset path now explicitly guards against) can persist garbage
prev/next links or stale sector images. We can reproduce on request and can test patches.

What the linked PR changes

• reset after INITIALIZE (deferred until the APDU response is out),
• deep resets via POWMAN_WDSEL (Bug 2),
• launch verification + dead-man's switch (Bug 3),
• lockout timeout raise + re-sync (Bug 4),
• fs-page address guard with correct bounds + panic→reboot (Bug 5),
• EV_PING handling fix (our dead-man's probe re-executed staged APDUs — included for
  completeness since it touches shared code paths),
• and, defensively, a drain-guard so a scheduled reset never lands while lazy flash writes are
  queued (one plausible contributor to Bug 6's corruption).

Bug 6 is not claimed fixed by the PR.

[polhenarejos]

Thank you for the detailed report, hardware testing, and SWD forensics. I appreciate it. These changes touch several sensitive areas (RP2350 reset behavior, multicore startup/lockout, USB servicing, and flash persistence) so I need time to review and reproduce them carefully.
We recently changed the flash-writing and cache-draining implementation. I need to determine which findings still apply to the current code and whether any symptoms may depend on the experimental patch series or specific RP2350B hardware.
In particular, I want to test:

• initialization and persistence on unmodified current firmware
• RP2040 and RP2350 builds
• reset behavior independently from flash changes
• multicore lockout and launch behavior
• filesystem corruption and invalid metadata handling

I will not merge these PRs until that review and hardware testing are complete. Because PR #135 also contains documentation and environment-specific tools, it may need to be split so each functional fix can be evaluated independently.
Please allow me some time for this investigation. I will report back with results and identify which changes should be retained, revised, or handled separately.

Bug 1

I understand the reported symptom as follows: after INITIALIZE, the USB CCID reader remains visible, but the virtual card no longer answers activation/APDU requests; a physical replug performs a full boot and restores service.
This does not appear to be the same issue as pico-hsm #9, which was an inter-core queue problem fixed in 2022. The pico-keys-sdk #9 link is unrelated.
Before adding an automatic reset, I need to reproduce this on unmodified current firmware and determine whether the failure is caused by incomplete runtime reinitialization, an asynchronous flash commit, or a core1/CCID failure. Please clarify whether the reader remains enumerated, whether card presence is reported, which operation first fails to obtain the ATR, and whether any APDU remains usable before replugging.

Bug 2

Needs deep testing. As far I can see, it seems PICO_PLATFORM macro is misused and also affects RP2040, which may not be the case.

Bug 3

The SWD snapshot confirms that core1 was in the boot-ROM launch loop while core0 and USB remained active. However, card_locked_func is assigned unconditionally after the SDK launch call and is not proof that the application entry point was reached.
The proposed launch-entry flag looks like useful, minimal defensive verification. A bounded core1 relaunch may also be reasonable.
I am not yet convinced that a stale FIFO word can complete the SDK’s full six-word echo sequence. Could you provide the value of the unread FIFO word and confirm whether it matched a launch word or the multicore lockout request ID? We also need to distinguish “core1 never launched” from “core1 launched and was reset later.”
The dead-man switch should be handled separately. As currently implemented, it competes with card_status() for card_to_usb_q and can consume normal completion or button events. Please split the launch-verification change into a small PR and report the mid-session/core1-loss behavior as a separate issue if possible.

Bug 4

I agree with retaining a bounded END timeout; blocking indefinitely is undesirable if there is an unexpected lockout race. Raising both timeouts to 100 ms is reasonable.
However, I do not think another START+END pair is safe because Pico SDK documents lockout starts as non-nestable. If END still fails after the larger timeout, normal execution cannot safely continue because core1 may remain parked. A bounded recovery reset seems preferable, provided we first confirm that all flash operations are serialized and no other flash operation can be active.
Please clarify whether the END-timeout warning was actually captured and whether any other code path can call the multicore lockout API concurrently.

Bug 5

I agree that invalid cached flash addresses must be rejected before reaching the Pico SDK flash functions. However, the proposed guard in low_flash_task() occurs after find_free_page() has already copied from the supplied address. This works for a readable SRAM address such as 0x20001A00, but does not protect against arbitrary invalid addresses.
The validation should happen before the first read or queue operation and should cover the entire aligned sector/range, including erase size. The operation should then return a flash error rather than silently dropping the queued page and allowing the commit to appear successful.
Automatically rebooting after a hard assertion may be useful for unattended recovery, but it should be considered separately because it affects all assertions and may create a reboot loop for persistent faults. Please split the address-validation fix and the global panic-recovery policy into separate changes.

I would not describe this as permanently bricking the device. A panic may leave the running firmware unresponsive until reset, while persistent filesystem corruption can make subsequent boots HardFault or loop indefinitely in scan_flash(). The board remains recoverable through BOOTSEL and a full flash_nuke, followed by reflashing.
Rebooting automatically after an assertion would not repair persistent corruption and could instead produce a reboot loop. In addition to validating addresses before queueing flash operations, scan_flash() should validate that every filesystem pointer remains within the configured region and moves strictly toward the start of the region. On detecting corruption, the firmware should stop scanning and expose a controlled recovery state rather than automatically erasing potentially valuable HSM data.

Bug 6

Regarding Bug 6, could you clarify which exact firmware revision and patch set produced each symptom? In particular:

1. Was the user PIN verified before reading the all-zero KCV?
2. Did the reported unwrapKey SW=6400 occur while the experimental EV_PING dead-man switch was enabled? The PR comments mention a separate SW=6400 caused by that probe re-executing an APDU.
3. Was the earlier hard-coded flash-region guard enabled when the PIN state failed to survive reset?
4. Have the persistence failure and corrupted prev_addr been reproduced on an unmodified upstream build after flash_nuke?
   The HardFault itself identifies a genuine robustness problem: filesystem pointers are used without sufficient validation before find_free_page() reads from them. However, the supplied evidence does not yet identify whether the original metadata corruption comes from upstream interrupted writes or from the experimental patches.

I intentionally left the linked-list pointers as integrity indicators. Preventing the resulting HardFault by silently rejecting the address does not explain why a non-flash value reached prev_addr, and may hide the original failure. Before considering such a guard, we need to distinguish physical-flash corruption from corruption of the dirty SRAM sector cache or an invalid file->data/record base. Please provide the raw physical record header, the cached-sector copy, all flash_pages entries and the original unaligned prev_addr from the same halted state. The current asynchronous, slot-ordered drain can expose a list link before the referenced record sector is durable, which is a plausible mechanism for stale payload bytes later being interpreted as pointers, but it must be confirmed from the captured state.

Final note

As a final note, this report and the associated PR cover several largely independent areas: initialization, reset behavior, multicore startup and lockout, assertion recovery, and flash persistence. To make review, testing, and future troubleshooting more manageable, could you please open a separate issue and PR for each bug where possible? Small, focused changes will be much easier to validate and merge independently. Thank you again for the detailed investigation and hardware testing.

[jobordu]

Split status (2026-08-04): Bug 1 → PR #27 + pico-hsm#136 · Bug 2 → PR #31 (now PICO_RP2350-guarded) · Bug 3 → PR #28 (launch-verify only; dead-man switch excluded, mid-session issue to follow) · Bug 4 → PR #29 (no re-sync) · Bug 5 → PR #30 (pre-read validation; panic policy withdrawn) · Bug 6 → open, needs the unmodified-upstream repro on our side.

Thank you for the careful review — splitting is done (PRs #27–#31 here, pico-hsm#136; the two large PRs are closed). Answers to your questions, with the evidence we captured:

Bug 1 — what the host actually sees after INITIALIZE (unmodified-path behaviour)

The reader remains enumerated; the card never presents an ATR, so PKCS#11 sees no token and no APDU is usable — the first failing operation is the initial card connect/ATR retrieval itself. Only a replug (POR) restores service. You are right that pico-hsm#9 is a different issue (2022 inter-core queue) — our reference to it was over-eager; the INITIALIZE-dead-card behaviour stands on its own, and PR #27 + pico-hsm#136 now carry just that fix, deferred so the response goes out before the reset fires.

Bug 3 — the FIFO word, and never-launched vs launched-then-lost

• The unread word, popped from the SIO FIFO via SWD on the wedged device: 0x00000000 — consistent with a partially consumed launch sequence ({0,0,1,vtor,sp,entry}): one leading zero eaten out of phase, the second left pending, FIFO_ST.VLD=1.
• We observed BOTH variants: (a) at-boot — card enumerates post-reset but never answers, core1 parked in the bootrom wait loop, no prior session; (b) mid-session — core1 parked (same bootrom wait loop) after it had been serving APDUs, with our launch-entry flag set and card_locked_func=apdu_thread. (b) is the case that motivated the dead-man's switch; per your guidance we've split it out — PR fix(hsm): verify core1 actually starts, recover it when it does not #28 is launch-verify + bounded relaunch only. We will file the mid-session loss as its own issue with the switch discussion.

Bug 4 — what was captured

Honest answer: the WARN: FLASH LOCKOUT END TIMEOUT line was never captured on the wire (our UART console wasn't wired at the time); the orphan state was observed directly via SWD (core1 in multicore_lockout_handler, card mute, core0 healthy). On concurrency: every flash operation in this tree funnels through low_flash.c's mtx_flash, and no other code path calls the multicore lockout API (grep-verified). PR #29 is just the 100ms timeouts + aborting the drain pass on end-timeout — the re-sync pair is gone; you're right that it's non-nestable.

Bug 6 — per-symptom attribution

1. PIN verified before the all-zero KCV? No — the zero KCV appeared in unauthenticated sc-hsm-tool output. When later queried through an authenticated scsh session, the KCV read correctly and matched the host-side derivation. So the zero-KCV may be a readout artefact of the unauthenticated path; we cannot fully attribute it without a clean-firmware retest, and we've stopped calling it a device quirk either way.
2. Was SW=6400 under EV_PING? The first 6400s were observed on builds with the ping active, but we then reproduced 6400 on a control build with no EV_PING code at all (our original 4-commit patch series). So the ping's duplicate-execution bug was real (and is fixed), but it is not the 6400's root cause.
3. Was the wrong-bounds guard active when the PIN state was lost? The guard post-dates the first 6400s, so it cannot be the original cause — but you're right to ask: with the wrong bounds it was dropping every legitimate lazy write, which plausibly aggravated later persistence symptoms (including the lost PIN state). It is corrected in PR fix(flash): validate addresses against the real fs bounds before any read or erase #30 (validate pre-read against the partition-derived bounds, return an error, never silently drop).
4. Reproduced on unmodified upstream after flash_nuke? No — that exact run has not been done, and we don't claim it has. The nearest control is our 4-patch build after an SWD erase of the fs region, which failed identically. A true unmodified-upstream-after-nuke run is the right next experiment on our side.
5. The halted-state capture, as requested: BFAR=0x6CB44000, CFSR=0x8200 (precise bus fault, BFARVALID), find_free_page's addr_alg=0x6CB44000 (so the unaligned prev_addr was 0x6CB44xxx), file=0x20080e50, backtrace isr_hardfault ← memcpy ← find_free_page ← flash_program_uintptr ← flash_clear_file ← flash_write_data_to_file_internal ← file_put_data ← mkek_store_file ← store_dkek_key ← save_dkek_key(id=0, key=NULL). flash_pages at the halt: entry 0 address=0x20001A00 (an SRAM pointer) with Thumb code as its content, entry 1 address=0x10400000, rest zeroed. We did NOT capture the raw physical record header or the cached-sector copy — if we reproduce the halted state again we'll grab both plus the unaligned prev_addr.
6. Your async-drain hypothesis (a list link exposed before the referenced record sector is durable, so stale payload bytes get read as pointers) is consistent with everything we saw — the corruption appeared after interrupted operations, and prev_addr pointing at what looks like code/payload bytes matches "stale payload interpreted as pointer". Happy to run any instrumentation you suggest against that mechanism.

Bug 2 — macro guard

Correct on PICO_PLATFORM — the arming is now under PICO_RP2350 (PR #31); POWMAN doesn't exist on RP2040.

Bug 5 — agreed, and split

Pre-read validation with an error return is PR #30. The panic→reboot policy is withdrawn from the PR series — your point about reboot loops on persistent corruption stands, and BOOTSEL+flash_nuke is indeed a recovery path (our "bricking" phrasing overstated it). A scan_flash() pointer-validation + controlled recovery state is a better design than what we proposed; we won't push the auto-reboot as a PR.

Thank you again for the depth of the review — the split series should now be independently testable, and we're glad to iterate on any of them.

[jobordu]

Reproduced on unmodified upstream after flash_nuke — with the full chain, and several corrections to our own report

You asked us to reproduce on unmodified current firmware before anything else. We did, and it produced the complete path from host APDU to device panic — including the origin of the invalid pointer, which turns out to be exactly the validation gap you predicted in your Bug 5 comment.

Scope, and the limits of this evidence

• Hardware: exactly one board — Waveshare RP2350-PiZero (RP2350B), debugged with a
  Raspberry Pi Debug Probe (SWD + UART).
• No RP2040 hardware exists on this bench. Every RP2040 statement we have made upstream
  is compile-only. The RP2040 code path has never been executed by us and we cannot
  claim any runtime behaviour for it.
• Firmware under test: pico-hsm 1ad8444 + pico-keys-sdk 843a3dc, no local patches.
  Verified by symbol check — none of our patch symbols (low_flash_quiesce,
  schedule_chip_reset, chip_reset_now, flash_addr_in_fs, low_flash_busy) are present
  in the built ELF, while all are present in our patched build.
• flash_nuke equivalent: full 16 MB chip erase over SWD (flash erase_sector 0 0 last,
  erase chunks ran through 0x00fe0000 -> 0x00ffffff), then programmed the stock ELF
  (** Verified OK **).

The trigger condition is the filesystem state

Run	fs state	Result
1	blank (immediately post-nuke)	--initialize reports Card command failed, but the card survives and answers
2	populated (by run 1)	--initialize kills the card — this is Bug 1

This is why Bug 1 looked intermittent to us: it requires a populated filesystem, i.e.
INITIALIZE has to rewrite existing records rather than write into empty space.

Host side (run 2, OPENSC_DEBUG=9)

12:41:35.429 sc_single_transmit: CLA:0, INS:D7, P1:2F, P2:3, data(23)   <- UPDATE BINARY, EF 2F03
             Outgoing APDU (28 bytes):
12:41:40.441 pcsc_detect_card_presence: returning with: 5
12:41:40.443 sc_single_transmit: unable to transmit APDU: -1107 (Transmit failed)
12:41:40.443 sc_hsm_write_ef: APDU transmit failed: -1107 (Transmit failed)

~5 s in flight, then the card stops answering — permanently.

Device side — hardware watchpoint on flash_pages[0].address

Old value = 0x103F9000        (valid flash)
New value = 0x2000D000        (SRAM)

#0 find_free_page (addr=0x2000D5A5)                       low_flash.c:316
#1 flash_program_block (addr=0x2000D5A5, data={len=2})    low_flash.c:333
#2 flash_program_halfword (data@entry=17)                 low_flash.c:354
#3 flash_write_data_to_file_internal (file=0x20056b24 <file_entries+80>,
                                      offset=0, partial=false)      flash.c:195
#4 flash_write_data_to_file (file=0x20056b24)             flash.c:257
#5 file_put_data (file=0x20056b24)                        file.c:438
#6 cmd_update_ef ()                              src/hsm/cmd_update_ef.c:134
#7 sc_hsm_process_apdu ()                        src/hsm/sc_hsm.c:944
#8 apdu_thread ()                                         apdu.c:245

file->data == 0x2000D5A5 — an odd-aligned SRAM pointer — is handed straight to the
flash write API at flash.c:195:

r = extended_size ? flash_program_word((uintptr_t)file->data + sizeof(uint16_t), len)
                  : flash_program_halfword((uintptr_t)file->data, (uint16_t)len);

find_free_page() aligns it down to 0x2000D000, memcpys 4096 bytes out of SRAM into
the sector cache, and marks the slot ready = 1. (The cached page contents are ARM code and
pointers — e.g. a 70 b5 Thumb prologue — not record data, confirming the copy source.)

The origin of the bad pointer — scan_region() accepts it

file.c:336:

for (uintptr_t base = flash_read_uintptr(endp); base >= startp; base = flash_read_uintptr(base)) {
    ...
    file->data = (uint8_t *) (base + sizeof(uintptr_t) + sizeof(uintptr_t) + sizeof(uint16_t));

The loop's only validation on a link read out of the persisted chain is base >= startp.
There is no upper bound and no alignment check. The arithmetic matches the observed value
exactly:

file->data  = base + 4 + 4 + 2 = base + 10
0x2000D5A5  = base + 10   ->   base = 0x2000D59B
0x2000D59B >= start_data_pool (0x10102000)  ->  accepted by the loop guard

So a corrupt forward link in the record chain is accepted as a record base, and its derived
file->data is stored in the in-memory file table. This is precisely the validation the
maintainer proposed in his review:

scan_flash() should validate that every filesystem pointer remains within the configured
region and moves strictly toward the start of the region.

Measured fs bounds on this build: start_data_pool=0x10102000, end_flash=0x103FE000,
start_rom_pool=0x103F9FF4, end_rom_pool=0x103FDFF4.

The panic

#4 flash_range_erase (count=4096)      pico-sdk hardware_flash/flash.c:213
       hard_assert(flash_offs + count <= PICO_FLASH_SIZE_BYTES);
       0x2000D000 - XIP_BASE = 0x1000D000 ; + 4096 > 0x1000000  -> fires
#5 low_flash_task ()                   low_flash.c:122
#6 flash_task ()                       flash.c:285
#7 core0_loop ()                       main.c:136
#8 main ()                             main.c:228

Why the card goes mute — Bug 1's mechanism, and how it ties to Bug 4

low_flash.c:116-125:

if (multicore_lockout_start_timeout_us(1000) == false) { ...; continue; }
uint32_t ints = save_and_disable_interrupts();
flash_range_erase(...);        /* <-- panics HERE */
flash_range_program(...);
restore_interrupts(ints);
if (multicore_lockout_end_timeout_us(1000) == false) { ...; continue; }

The panic lands between lockout START and END and after save_and_disable_interrupts().
Core0 ends inside panic() -> _exit() -> __breakpoint(), interrupts disabled, lockout still
held. Core1 is therefore parked forever:

Thread 2: multicore_lockout_handler ()          multicore.c:241
          <signal handler called>
          __wfe ()                              queue.c:64
          queue_remove_internal (q=0x200666a4 <usb_to_card_q>, block=true)
          queue_remove_blocking (usb_to_card_q)
          apdu_thread ()                        apdu.c:231

That is the whole "enumerated but mute" class: the USB device stays up at the host, but no
APDU can ever be serviced again.

The maintainer's Bug 1 questions, answered on unmodified firmware

1. Does the reader remain enumerated? Yes — still listed, Card: Yes.
2. Is card presence reported? PCSC reports presence (pcsc_detect_card_presence: 5), but
   every subsequent connect fails: Failed to connect to card: Card not present.
3. Which operation first fails to obtain the ATR? None — the failure precedes ATR. The
   first failure is the in-flight UPDATE BINARY (INS D7, EF 2F03) issued inside
   sc_hsm_initialize's sc_hsm_write_ef, which times out after ~5 s. Nothing activates
   afterwards.
4. Is any APDU usable before replug? No.

Downstream symptoms that follow from the same defect

• pkcs15-tool --list-pins returns nothing, and PKCS#11 C_Login fails with
  CKR_USER_PIN_NOT_INITIALIZED — while a raw VERIFY APDU (00 20 00 81) returns
  9000. The PIN works; the PKCS#15 structures were never written, because the EF write
  that writes them is exactly what fails.
• DKEK key check value : 0000000000000000 after a successful-looking share import.

Both reproduce on unmodified upstream firmware.

Corrections to our own earlier report

1. Bug 1 mechanism — our framing was wrong. The card does not complete INITIALIZE and then
   need a reset; it panics during INITIALIZE. A post-INITIALIZE reset (PR fix(hsm): defer the post-INITIALIZE reset until the response is out #27 /
   pico-hsm#136) masks a panic rather than fixing it. What most likely made our patched build
   appear to recover was our panic->reboot handler, not the scheduled reset.
2. Bug 4 — the END-timeout warning was never captured. The parked core1 is not caused by an
   END timeout; it is caused by core0 panicking between START and END. PR fix(flash): raise lockout timeouts, abort the drain pass on end-timeout #29 therefore
   addresses a scenario we never observed.
3. Bug 4 concurrency audit. In-tree, the only lockout callers are low_flash.c:116/125/138/146,
   all inside low_flash_task()'s drain loop on core0, plus multicore_lockout_victim_init()
   at :243. phymarker_write() does flash_range_erase/program without the lockout, but it
   is #ifdef PICO_RP2040 and runs from low_flash_init() before core1 launches — not a
   concurrency hazard.
4. Bug 5 — the ordering critique was correct. PR fix(flash): validate addresses against the real fs bounds before any read or erase #30 now validates before the first
   read/queue and returns an error. But this trace shows the guard is a safety net, not the
   fix: it converts the panic into a clean error return while file->data still points at
   SRAM. The actual fix is the scan_region() validation above.
5. Bug 2 — correct, and worse than the code guard. The POWMAN includes sat in the generic
   PICO_PLATFORM block, so an RP2040 build failed outright with
   fatal error: hardware/powman.h: No such file or directory. Fixed in PR feat(reset): make every watchdog-family reset a full switched-core power cycle (RP2350) #31. Both
   platforms now compile — RP2040 compile-only, never run.

[jobordu]

Follow-up measurement, in case it is useful for prioritising: same board, same sequence, full 16 MB erase before each firmware.

firmware	INITIALIZE on populated fs	result
stock 1ad8444/843a3dc, unpatched	1	card mute permanently
our branch (address validation + panic→reboot)	5	5/5 alive

Caveat stated plainly: the patched build still fails the EF write (Card command failed); it just no longer panics core0 inside the lockout. The defect itself — scan_region() accepting an out-of-region chain link and deriving file->data from it — is untouched by anything we have proposed. Details on PR #30.

[jobordu]

Correction: the root cause is not filesystem corruption — it is a write to a FILE_DATA_FUNC file's function pointer

Retracting my previous attribution

My previous comment attributed this to scan_region() (file.c:336) accepting an out-of-region
chain link. That was wrong and I am retracting it. I walked the on-flash record chain over
SWD at three points on unmodified firmware — factory-clean, after INITIALIZE #1, and at the
panic. Every link was in-region, correctly doubly-linked, and properly terminated:

=== DATA chain walk (as scan_region traverses it), factory-clean ===
head word @end_data_pool = 0x103f9fbf
[00] base=0x103f9fbf link=0x103f9f92 w1=0x103f9fec fid=0x1081 len=33
[01] base=0x103f9f92 link=0x103f9f85 w1=0x103f9fbf fid=0x1088 len=33
[02] base=0x103f9f85 link=0x103f9f78 w1=0x103f9f92 fid=0x1083 len=1
[03] base=0x103f9f78 link=0x103f9f6b w1=0x103f9f85 fid=0x108a len=1
[04] base=0x103f9f6b link=0x103f9f5e w1=0x103f9f78 fid=0x1082 len=1
[05] base=0x103f9f5e link=0x00000000 w1=0x103f9f6b fid=0x1089 len=1
walk ended after 6 records (base=0x00000000)

No filesystem corruption is involved in this failure at any point. Your instinct to be
sceptical of the corruption story was correct.

What actually happens

(gdb) printf "data=%p fid=0x%04x", ((file_t*)0x20056b24)->data, ((file_t*)0x20056b24)->fid
data=0x2000d5a5 fid=0x2f03
(gdb) info symbol ((file_t*)0x20056b24)->data
parse_token_info + 1 in section .text

0x2000D5A5 is parse_token_info + 1 — a Thumb function pointer. EF.TokenInfo (0x2F03)
is declared in pico-hsm/src/hsm/files.c as:

/*  4 */ { .fid = 0x2f03, // EF.TokenInfo
           .type = FILE_TYPE_WORKING_EF | FILE_DATA_FUNC,
           .data = (uint8_t *) parse_token_info,
           ... }

For a FILE_DATA_FUNC file, ->data is a generator function, not a flash address. Every
other consumer checks the flag before using it:

• file.c (SELECT / FCI): if ((type & FILE_DATA_FUNC) == FILE_DATA_FUNC) { call it }
• cmd_read_binary.c:137 (READ): same check, calls the handler

flash_write_data_to_file_internal() (flash.c:191) checks only if (file->data && ...). It
never checks the flag — so an UPDATE BINARY on a FILE_DATA_FUNC EF hands a .text address
straight to the flash layer.

It is fully deterministic, not a race and not state-dependent

file_get_size() does file_read_uint16(tf->data) — for one of these files that reads the
first halfword of the function's own machine code as the file size:

2000d5a4 <parse_token_info>:  01 29 01 d0 2c 20 70 47      ->  old_size = 0x2901 = 10497

The host writes a 17-byte TokenInfo blob, so the in-place branch condition
!partial && len <= old_size && (extended_size || len < FLASH_FILE_EXTENDED_LENGTH)
evaluates as 17 <= 10497 && 17 < 0xFFFF → always true. It therefore calls
flash_program_halfword((uintptr_t)file->data, 17), which matches the captured frame exactly
(flash_program_halfword (..., data@entry=17)).

From there: find_free_page() caches 4 KB of .text as a dirty sector (the cached page
contains Thumb prologues, e.g. 70 b5), and low_flash_task() calls
flash_range_erase(0x2000D000 - XIP_BASE = 0x1000D000, 4096), tripping
hard_assert(flash_offs + count <= PICO_FLASH_SIZE_BYTES).

The panic lands between multicore_lockout_start_timeout_us() and ..._end_timeout_us(),
and after save_and_disable_interrupts() — so core0 dies holding the lockout with interrupts
off, and core1 stays parked in multicore_lockout_handler forever. Reader enumerated, card
permanently mute. That is the whole of Bug 1.

Why we mistakenly reported it as filesystem-dependent

old_size comes from a compiled constant in .text, not from filesystem state. Our
"blank fs survives / populated fs dies" pattern was an artifact of what the host happens to
write on each pass, not of fs population. Apologies for sending you down that path.

Reached by an ordinary, spec-conformant host operation

OpenSC's sc_hsm_initialize writes EF.TokenInfo as part of INITIALIZE — that is the
CLA:0 INS:D7 P1:2F P2:03 APDU in the host trace. No unusual host behaviour is required.

Consequences for the PR series

• fix(hsm): defer the post-INITIALIZE reset until the response is out #27 / pico-hsm#136 (reset after INITIALIZE) — these were masking this panic. They should
  be closed; I will close them unless you would rather keep them for reference.
• fix(flash): validate addresses against the real fs bounds before any read or erase #30 (fs bounds validation) — still worth having as defence in depth (it would have turned
  this panic into a clean error), and I widened it to a full-range check per your review. But it
  is explicitly not the fix for this bug.
• fix(flash): raise lockout timeouts, abort the drain pass on end-timeout #29 (lockout timeouts) — unchanged: draft, scenario never observed.
• The scan_region() weakness you proposed fixing (base >= startp only, no upper bound and no
  alignment check) is real and still worth fixing as defence in depth — but it is not the
  cause here, and no corrupt link was ever observed on this device.

Minimal reproduction — two APDUs, measured both ways

The EF's ACL gates ACL_OP_UPDATE_ERASE, so a VERIFY is needed first; unauthenticated the
command is refused with 6982 before ever reaching the write.

00 20 00 88 08 <SO-PIN>            VERIFY SO-PIN
00 D7 2F 03 13 53 11 00*17         UPDATE BINARY EF 2F03

firmware	2nd APDU	card afterwards
stock 1ad8444/843a3dc	APDU transmit failed: Transmit failed	permanently mute (Card not present)
stock + the 8-line guard	6581	still answering

No INITIALIZE, no filesystem state, no corruption required.

The fix, and its isolation test

    /* A FILE_DATA_FUNC file's ->data is a generator FUNCTION POINTER, not a flash
     * address. The SELECT/FCI path (file.c) and the READ path (cmd_read_binary.c)
     * both check this flag before using ->data; this write path did not, so an
     * UPDATE BINARY on such an EF handed a .text address to the flash layer. */
    if ((file_get_type(file) & FILE_DATA_FUNC) == FILE_DATA_FUNC) {
        return PICOKEYS_ERR_BLOCKED;
    }

Applied to otherwise-stock firmware, nothing else changed, full 16 MB erase before each:

firmware	INITIALIZE runs	result
stock	2	mute on the 2nd, permanently
stock + this 8-line guard only	4	4/4 alive, correct state throughout

Filed as a PR against master. It makes the write fail cleanly; it does not decide what
should happen when a host writes a generated EF — whether cmd_update_ef should return a
more specific SW than the 6581 that PICOKEYS_ERR_BLOCKED maps to, or whether pico-hsm should
accept and store an override for EF.TokenInfo, is a semantic decision I have left to you.

[jobordu]

Conclusion on every point you raised

You asked for each bug to be split and each question answered. Here is the closing state of all
of them, including the ones where the answer is "we were wrong" or "we cannot test that".

Hardware scope for everything below: a single Waveshare RP2350-PiZero (RP2350B) with a
Raspberry Pi Debug Probe (SWD + UART). There is no RP2040 on this bench, so every RP2040
statement is compile-only and never executed.

---

Your five test areas

you wanted to test	conclusion
initialization and persistence on unmodified current firmware	Done. Reproduced after a full 16 MB erase and root-caused to a FILE_DATA_FUNC write, not persistence. Fix in #32.
RP2040 and RP2350 builds	RP2350: built and run. RP2040: compiles only — no hardware, never executed. The include bug you predicted was real and is fixed in #31.
reset behaviour independently from flash changes	Separated: #31 is reset-only; the INITIALIZE reset PRs are closed because the failure they addressed was a panic, not a missing reset.
multicore lockout and launch behaviour	Concluded below (Bugs 3 and 4). The lockout orphan was a consequence of the panic, not a timeout.
filesystem corruption and invalid metadata handling	No filesystem corruption exists on this device. The chain was walked over SWD at three points and is intact. The scan_region() weakness is latent only.

Bug 1 — concluded

Root cause is not a missing reset and not persistence: flash_write_data_to_file_internal()
writes a FILE_DATA_FUNC file's generator function pointer as if it were a flash address
(EF.TokenInfo 0x2F03, .data = parse_token_info). Deterministic; minimal repro is two APDUs.
Fix: #32. Your four questions:

1. Reader stays enumerated — yes.
2. Card presence reported — PCSC reports presence (pcsc_detect_card_presence: 5) but every
   connect then fails with Card not present.
3. First operation to fail — none reach ATR; the in-flight UPDATE BINARY (INS D7, EF 2F03)
   times out after ~5 s and core0 is already dead.
4. Any APDU usable before replug — no.

Your correction on pico-hsm#9 was right; we retracted it. #27 and pico-hsm#136 are closed.

Bug 2 — concluded

You were right, and it was worse than the code guard: the POWMAN includes sat in the generic
PICO_PLATFORM block, so an RP2040 build failed outright (fatal error: hardware/powman.h).
Fixed in #31; both platforms now compile. RP2040 runtime remains untested — no hardware.

Bug 3 — concluded

• Unread FIFO word: 0x00000000.
• Did it match a launch word or the lockout request ID? A launch word. The launch sequence
  is {0, 0, 1, vector_table, sp, entry} — it begins with two zero words. It is definitively
  not a lockout request ID: those derive from lockout_request_id = 0x73a8831eu. The SDK's
  own comment names this hazard: "A non-zero initialisation value is used in order to reduce
  the chance of entering the lock handler on bootup due to a 0-word being present in the FIFO."
• Never-launched vs launched-then-reset: the state we captured was never-launched — core1
  parked in the bootrom wait-for-launch loop (pc=0x19e, bootrom stack) while core0 had already
  proceeded. The launch flag in fix(hsm): verify core1 actually starts, recover it when it does not #28 distinguishes the two by construction.
• Dead-man's switch competing with card_status() for card_to_usb_q: conceded, and it is
  withdrawn from the series.
• Separate issue for mid-session core1 loss: we cannot substantiate one, so we have not filed
  it. The only two core1 states we ever actually observed are (a) the launch failure above and
  (b) parking in multicore_lockout_handler, which is now fully explained as a consequence of
  the panic. Filing a speculative issue would waste your time. The dead-man's switch therefore
  rests on an unproven premise and stays withdrawn.

Bug 4 — concluded

• Was the END-timeout warning ever captured? No. Answering plainly: it was never observed.
• What actually orphans core1: core0 panics between lockout START and END, after
  save_and_disable_interrupts() — so the lockout is never released and interrupts stay off.
  fix(fs): never write a FILE_DATA_FUNC file's ->data — it is a function pointer, not a flash address #32 removes the panic.
• Can anything else call the lockout API concurrently? No. The only in-tree callers are
  low_flash.c:116/125/138/146, all inside low_flash_task()'s drain loop on core0, plus
  multicore_lockout_victim_init() at :243.
• Are all flash operations serialised? Yes — everything goes through low_flash_task() under
  mtx_flash. The one exception, phymarker_write(), erases and programs without the lockout
  but is #ifdef PICO_RP2040 and runs from low_flash_init() before core1 launches.
• fix(flash): raise lockout timeouts, abort the drain pass on end-timeout #29 is a draft and addresses a scenario we never measured; close it if you prefer.

Bug 5 — concluded

• Validate before the first read/queue: done.
• Cover the entire aligned range including erase size: this was genuinely missing in our first
  version and you caught it. We only validated the start address, while
  flash_program_block() walks forward across sectors and flash_erase_page() erases
  page_size bytes. Now fixed with a flash_range_in_fs(addr, len) that checks the whole span
  and the sector-aligned base. fix(flash): validate addresses against the real fs bounds before any read or erase #30 updated.
• Return an error rather than silently dropping: done (PICOKEYS_ERR_MEMORY_FATAL).
• Split the panic-recovery policy out: done — withdrawn from the series entirely.
• "Bricking": your wording is correct, ours was wrong. BOOTSEL + flash_nuke always
  recovers; the accurate claim is that the firmware is unresponsive until an external reset.
• scan_flash() pointer validation with a controlled recovery state: still worth doing as
  defence in depth — scan_region() (file.c:336) validates a chain link with only
  base >= startp, no upper bound and no alignment check. But it is not the cause of this
  failure and no corrupt link was ever observed here. We have deliberately not written it,
  because the recovery-state design is yours to make.

Bug 6 — concluded

1. Was the user PIN verified before reading the all-zero KCV? No — sc-hsm-tool reads the
   KCV on an unauthenticated connection. The all-zero KCV is an artefact of the read path, not
   evidence of lost DKEK state.
2. Did unwrapKey SW=6400 occur with EV_PING enabled? See the measurement below.
3. Was the hard-coded flash-region guard active when PIN state "failed to survive reset"?
   Timeline: that guard used a FLASH_SIZE_BYTES>>1 bound and was active 2026-08-03 → 08-04.
   Because it wrongly rejected every legitimate lazy write, every persistence observation we
   made in that window is contaminated and should be discarded. That includes the original
   "writes do not survive" claim.
4. Reproduced on unmodified upstream after flash_nuke? Yes — and it produced the real
   root cause, which is not persistence at all.

Your own hypothesis — that the async slot-ordered drain could expose a list link before the
referenced record sector is durable — is not needed to explain anything we saw: no corrupt
link ever existed. It may still be a real hazard, but we have no evidence for it.

Two further results you will want

1. The fix is necessary but NOT sufficient for provisioning.

With #32 applied the card survives, but sc-hsm-tool --initialize still reports
Card command failed, and the full chain then cannot proceed:

INITIALIZE            -> Card command failed   (card SURVIVES, state set: PIN 3/15, DKEK shares 1)
create/import DKEK    -> OK, "DKEK share imported"
raw VERIFY user PIN   -> 9000                  (the PIN genuinely works)
PKCS#11 C_Login       -> CKR_USER_PIN_NOT_INITIALIZED
pkcs11-tool objects   -> only a Profile object; no PKCS#15 structures
wrap-key              -> File not found        (no key: keygen never ran)

The reason is the same defect seen from the other side: EF.DIR (0x2F00) and EF.TokenInfo
(0x2F03) are both FILE_DATA_FUNC, i.e. generated files, and OpenSC's sc_hsm_initialize
wants to write them. #32 turns that write from a device-killing panic into a clean refusal,
but the write still fails, so the PKCS#15 structures are never created and PKCS#11 concludes the
PIN is not initialized.

So there is a semantic decision for you beyond the memory-safety fix: either pico-hsm should
accept and store an override for these generated EFs, or the provisioning path needs to not
require writing them. That is the real blocker for OpenSC provisioning, and we have deliberately
not guessed at it.

This also fully explains the CKR_USER_PIN_NOT_INITIALIZED symptom in the original report — it
is a downstream consequence, not a persistence failure.

2. It is not a regression from your recent flash/cache-drain rework.

You said you needed to know which findings still apply to the current code. This one applies to
both old and new: flash_write_data_to_file_internal() was introduced in 4390947
(2026-07-16, "Support multi-sector flash files with 32-bit lengths and cache draining"), but the
code it replaced did the same thing:

uint16_t size_file_flash = file->data ? flash_read_uint16((uintptr_t) file->data) : 0;
if (file->data) {                                   // already in flash
    if (offset + len <= size_file_flash) {          // 17 <= 0x2901 -> true
        flash_program_halfword((uintptr_t) file->data, offset + len);

Same missing FILE_DATA_FUNC check, same arithmetic, same outcome. The defect predates the
rework.

Still unresolved, stated plainly

unwrapKey SW=6400 from the original report is not reproduced and remains unattributed. We
could not even reach the unwrap step, because keygen cannot run until provisioning works. Given
that our whole persistence narrative collapsed, and that the observation window overlapped the
contaminated wrong-bounds-guard period, we would treat the original 6400 report as unreliable
until it is seen again on a clean build. We are not asking you to chase it.

Filing

Split as you asked: #28 (launch verify), #29 (lockout timeouts, draft), #30 (fs range
validation), #31 (deep reset, RP2350-only), #32 (the actual root-cause fix). #27 and
pico-hsm#136 are closed. Issue #25's body carries inline [CORRECTED] notes so it is not read
as current where it was wrong.

[jobordu]

Resolved: the CKR_USER_PIN_NOT_INITIALIZED symptom was a second, independent deadlock — now root-caused and fixed

Filed as polhenarejos/pico-hsm#137.

Earlier in this thread I reported that even with the FILE_DATA_FUNC panic fixed, provisioning still failed and PKCS#11 reported CKR_USER_PIN_NOT_INITIALIZED. I attributed that to EF.TokenInfo writes being refused. That was wrong — OpenSC ignores the TokenInfo write failure (sc_hsm_initialize: returning with: 0 (Success)). The real cause is a self-provisioning deadlock:

dev_name is the CHR of EF.C_DevAut and has exactly one assignment in pico-hsm. asn1_cvc_aut() refuses to build the EE certificate without it, and EF.C_DevAut is only created inside that same INITIALIZE path — so creating EF.C_DevAut requires a value parsed from EF.C_DevAut. Once the file is lost (flash erase / flash_nuke / corruption) the device can never be initialized again. OpenSC's PKCS#15 emulator then aborts on Could not decode EF.C_DevAut (LOG_TEST_RET returns), so no PKCS#15 objects — including PIN objects — are built at all.

Measured on the card: dev_name = (nil), asn1_cvc_aut returns 0, INITIALIZE returns SW_EXEC_ERROR.

That fully explains the symptom, and it means my earlier suggestion that pico-hsm might need to accept writes to generated EFs is not required — please disregard it.

Also settled: the all-zero DKEK KCV

Instrumenting load_dkek resolves the other open question from Bug 6:

file_get_size(EF_DKEK) = 65
mkek_load_file         = -1005  (PICOKEYS_NO_LOGIN)
dkek_is_complete       = 1
load_dkek              = -1008  (WRONG_DATA)

sc-hsm-tool reads DKEK status unauthenticated, so the MKEK cannot be unwrapped; dkek_kcv() returns early leaving the buffer zeroed, and cmd_key_domain.c ignores that return value, so the card reports 0000000000000000. dkek_is_complete = 1 shows the domain is correctly established. The all-zero KCV is an artifact of the unauthenticated read path, not evidence of lost DKEK state — the original report's reading of it was wrong.

(Minor, separate observation: cmd_key_domain.c discarding dkek_kcv()'s return value is why a failure surfaces as a plausible-looking all-zero KCV rather than an error. Reporting the error would have made this much easier to diagnose. Happy to send a patch if you want one.)

[jobordu]

Status update — what changed since your review, and the current state of every PR

Two days of hardware work since the last update. The headline is that the original report's
framing was wrong in an important way, and the real root cause turned out to be something
neither of us had identified. Everything below is measured on one RP2350B (Waveshare
RP2350-PiZero) with a Raspberry Pi Debug Probe. No RP2040 hardware here — every RP2040
statement in this thread remains compile-only.

---

The actual root cause: EF.C_DevAut could never be created once lost

dev_name is the CHR of EF.C_DevAut and had exactly one assignment in pico-hsm
(sc_hsm.c, reset_puk_store). asn1_cvc_aut() refuses to build the EE certificate without it,
and EF.C_DevAut is only created inside that same INITIALIZE path, after that call. So
creating EF.C_DevAut required a value parsed from EF.C_DevAut. Once the file was lost — flash
erase, flash_nuke, corruption — the device could never be initialised again.

Measured: dev_name = (nil), asn1_cvc_aut returns 0, cmd_initialize returns SW_EXEC_ERROR.

Downstream, OpenSC's PKCS#15 emulator aborts on Could not decode EF.C_DevAut (LOG_TEST_RET
returns), so no PKCS#15 objects at all are built — which is the real explanation for the
CKR_USER_PIN_NOT_INITIALIZED in the original report, on a card whose raw VERIFY APDU returns
9000. Filed as polhenarejos/pico-hsm#137.

A second, independent bug in the same area

C.DevAut was 940 bytes: a 497-byte EE authenticated request (tag 0x67) concatenated with the
443-byte certificate (tag 7F21). asn1_cvc_aut() and asn1_cvc_cert() both append
(output->len += out_len) and the shared buffer is never reset, so EF_TERMCA is written with
the cumulative length. OpenSC decodes the first object, so what it treats as the device
certificate is actually the request. Filed as polhenarejos/pico-hsm#138.

#32 changed direction, because refusing the write breaks scsh

I originally had the FILE_DATA_FUNC guard return an error. That is not viable: the Smart Card
Shell aborts initialisation the moment such a write is refused —

GPError: Card (CARD_INVALID_SW/25985) - "Unexpected SW1/SW2=6581 received"
  at scsh/sc-hsm/SmartCardHSM.js#3279

6581 is what PICOKEYS_ERR_BLOCKED maps to. OpenSC ignores the identical failure, which is
why it only surfaced once I exercised the scsh path. #32 now accepts-and-discards: ->data is
still never written through, only the reported status changes. If you would rather pico-hsm
stored an override for generated EFs, that is a larger change and yours to make — but returning
an error is strictly worse than the status quo for scsh users.

---

Answers to your Bug 6 questions, now that they can be answered

unwrapKey SW=6400 — does not reproduce, and 6400 carried no information.
SW_EXEC_ERROR() is literally set_res_sw(0x64, 0x00), returned from dozens of sites — including
cmd_initialize on the DevAut deadlock. On a card that can actually provision, the round trip
succeeds: wrap → 367-byte blob → unwrap → Key successfully imported. The original 6400 was a
downstream consequence of a card that had never completed INITIALIZE.

The all-zero DKEK KCV is an artefact of the unauthenticated read path.

file_get_size(EF_DKEK) = 65
mkek_load_file         = -1005  (PICOKEYS_NO_LOGIN)
dkek_is_complete       = 1

sc-hsm-tool reads DKEK status without authenticating, so the MKEK cannot be unwrapped;
dkek_kcv() returns early leaving the buffer zeroed, and cmd_key_domain.c discards that
return value, so the card reports 0000000000000000 rather than an error. dkek_is_complete = 1
proves the domain is fine. Our original "zero KCV means lost DKEK state" reading was wrong.

(Small suggestion, unrelated to any PR: having cmd_key_domain.c propagate dkek_kcv()'s error
instead of reporting a plausible-looking all-zero KCV would have saved days here.)

---

Corrections to my own PRs

#28 (core1 launch verification) — evidence weakened, disclosed on the PR.
card_start() is only called lazily from the command paths, so core1 parked at bootrom
0x19e with core1_alive == 0 is the normal idle state of a card that has not yet received an
APDU. The SWD capture I cited does not by itself distinguish that from a launch failure. The
underlying gap is still real — card_locked_func is assigned unconditionally after the launch
call — but I am no longer claiming any frequency, and I will convert the PR to draft if you
prefer.

#31 (POWMAN deep reset) — two properties found while using it as a recovery tool.
It does not reliably restore the USB peripheral (six successful revivals, then the cores
restarted while the device never re-enumerated), and it clears the watchdog scratch registers
— which silently resets any bounded-retry budget stored there, including the relaunch ladder in
this same series. Details on the PR.

#27 / pico-hsm#136 — closed, as discussed: they masked a panic rather than fixing one.
Note that a post-INITIALIZE reset is still needed on this firmware, but for a different reason
than we originally gave: with it removed, INITIALIZE leaves the in-memory file table inconsistent
with flash — the card reports DKEK shares: 1 while the very next IMPORT DKEK SHARE fails with
File not found, and a reboot fixes it. That is a pico-hsm-side observation, not a reason to
reopen those PRs.

---

Current state

item	state	note
#28 core1 launch verify	open	evidence caveat posted; happy to draft it
#29 lockout timeouts	draft	scenario never observed; close if you prefer
#30 fs range validation	open	widened to full-span per your review; defence in depth, not the fix
#31 POWMAN deep reset	open	RP2350-only guard fixed; new caveats posted
#32 FILE_DATA_FUNC	open	now accept-and-discard, with scsh evidence
pico-hsm#137 DevAut bootstrap	open	the actual root cause
pico-hsm#138 C.DevAut buffer	open	independent, small

With #137, #138 and #32 applied, a card that has been fully erased provisions end to end and a
key wrap/unwrap round trip completes. Our staging battery runs 19/19 green and the break-glass
recovery drill passes, including a re-provision proof that the same key comes back and verifies
against the original public key.

No rush on any of this — I know you wanted time to test. I would rather you had the corrected
picture than the one I filed originally.

[jobordu]

Correcting something I told you in this thread, plus where the series stands

The deep-reset advice in this thread was wrong, and it was my bug. Earlier I described the
switched-core power cycle (POWMAN_WDSEL with RESET_SWCORE) as the reliable reset, on the basis
of a 100/100 hammer test. That test was on an idle card and does not generalise. Arming
POWMAN_WDSEL at boot — which is what #31 proposes — makes it global state, so every
unrelated watchdog_reboot() in the tree silently becomes a power cycle:

call site	what it is
rescue.c:490	the rescue applet's reboot command
ccid.c:465	the USB reset-to-flash request

and a deep reset does not reliably bring the USB peripheral back. The rescue applet's reboot
returned my card 5/5 on stock firmware and, with the arming in place, stopped returning it at all.

Same watchdog trigger, only that register differing:

POWMAN_WDSEL	result
0x1101 armed	off the USB bus; three consecutive POWMAN cycles failed; physical replug required
0x0000 cleared	re-enumerated and answered in 5 s

Scoping the arming to the one caller that has already given up on a warm reset
(chip_reset_now(deep=true)) fixes it. Full write-up and numbers are on #31, which I have
converted to draft — I do not think it should be merged as written, and I would rather say so
than leave it looking ready.

A measurement caution that cost me a day

The reliability numbers I was chasing turned out to be dominated by which USB port the device was
plugged into, not by firmware at all:

same firmware, root port / self-powered hub : 10/12 reboots returned unaided
same firmware, powered switchable hub port  : 22/23
(a second firmware build on that hub port)  : 25/25

If you are measuring reset reliability on RP2350B, the supply is a variable worth pinning before
concluding anything about the code. I had attributed the residual to firmware and it was mostly
cabling.

Where the series stands

item	state	note
#32 FILE_DATA_FUNC	open	accept-and-discard; scsh aborts on 6581 where OpenSC ignores it
#30 fs range validation	open	widened to full-span per your review
#31 POWMAN deep reset	draft	superseded by the scoping fix above
#28 core1 launch verify	draft	my SWD evidence did not distinguish a failed launch from normal lazy-launch idle
#29 lockout timeouts	draft	the scenario was never observed
pico-hsm#137 DevAut bootstrap	open	the actual root cause of this issue
pico-hsm#138 C.DevAut buffer	open	independent, small

With #137, #138 and #32 applied, a fully erased card provisions end to end and the wrap/unwrap
round trip completes. My staging battery's hardened-init/RRC/posture/PKA phase group has run
green repeatedly.

Three things I reported here that turned out to be wrong, consolidated so the thread is not
misleading to anyone reading it later: the scan_region() root-cause attribution, the
"EF.DIR/EF.TokenInfo writes block provisioning" claim, and the --keypairgen hang (which was my own
out-of-tree instrumentation eating CCID completion events — exactly the competition for
card_to_usb_q you predicted in review).

No rush on any of this.

[jobordu]

A watchdog reset loop that survives power cycles — root cause, with the register reads

This is a stock-code finding, not one of my patches, and I think it explains a class of "the
card never came back" that is otherwise very hard to diagnose. Worth having even if you take none
of the rest of my series.

The mechanism

rescue.c:490 reboots with:

watchdog_reboot(0, 0, 100);

watchdog_reboot() leaves the watchdog ENABLED across the reset it causes, with a 100 ms
window. Boot must therefore complete inside that window. It does not always — the flash filesystem
scan at startup grows with the number of files on the card, so a populated card boots slower than
an empty one. When the scan overruns, the watchdog fires again. And again.

The same applies to rom_reboot(): the bootrom implements it via the watchdog too, so routing a
reboot through the bootrom does not avoid this.

The evidence

Rather than clearing the state to get a run moving — which is what every previous encounter with
this did — I provoked it deliberately and read the hardware while still in it:

WATCHDOG_CTRL   = 0x40000000    ENABLE set, pause-on-debug CLEAR
WATCHDOG_LOAD   = 0x00000000    countdown expired
WATCHDOG_REASON = 0x00000001    TIMER — the last reset WAS the watchdog

REASON = TIMER is the whole finding: the chip is being reset, repeatedly, by its own watchdog.

Every symptom follows from that, and none of them made sense before:

symptom	why
cores report running but time out on every halt	they are being reset out from under the debugger
no firmware output at all (RTT)	boot never gets far enough to print
USB port reads power connect without enable	enumeration never completes
survives a POWMAN power cycle and a full VBUS cut	the firmware re-arms the trap on the way up

That last row is why this costs a physical replug: removing power does not help when the first
thing the firmware does on the way back is re-arm the same trap.

The fix, and two wrong attempts worth recording

The disarm has to run before main(). I got this wrong twice:

1. Disarm at the top of main() — measured NOT to work. The loop never reaches main(). What
   made it look right is that it does run on healthy boots, so the register reads clean whenever
   you check a working card.
2. Route rescue.c through the bootrom path instead — necessary but not sufficient, because
   rom_reboot() is itself watchdog-based.

What works is a preinit hook, before clocks and before main:

static void disarm_stale_watchdog(void) {
    hw_clear_bits(&watchdog_hw->ctrl, WATCHDOG_CTRL_ENABLE_BITS);
}
PICO_RUNTIME_INIT_FUNC_HW(disarm_stale_watchdog, PICO_RUNTIME_INIT_EARLIEST);

Safe unconditionally: by the time any C runs, the reset has happened and its reboot vector is
already consumed.

Verified by the register — in wedges captured after the change, WATCHDOG_CTRL reads 0x00000000
where it previously read 0x40000000.

What this does NOT fix, so you are not misled

A second, distinct wedge remains at roughly 1 reboot in 37: watchdog clear, POWMAN clear, flash
image intact, cores halted, USB not enumerating. It is not the same fault and I have not
root-caused it. I am reporting the watchdog loop because it is understood and independently
useful, not because the device is now reliable.

Also worth flagging for anyone measuring this: a debugger attached to the target manufactures
this symptom. OpenOCD halts the core when it re-examines after the device drops off USB, and a
halted core cannot enumerate — indistinguishable from a wedge unless you issue a reset run and
re-check first. In my last clean run, 4 of 5 apparent wedges were that, not the device.

One RP2350B (Waveshare RP2350-PiZero) with a Raspberry Pi Debug Probe. No RP2040 hardware here.

Happy to send the preinit disarm as its own small PR if it is useful to you separately from the
rest of my series.

[jobordu]

Both wedge causes now fixed and verified on hardware — with the second patch inline

Following up on the watchdog finding above. The second cause is found too, and both are verified.

Cause 2: an out-of-range chain link is dereferenced during the scan

file.c's scan walks the chain with:

for (uintptr_t base = flash_read_uintptr(endp); base >= startp; base = flash_read_uintptr(base)) {

That bounds base only from below. Any garbage above the pool passes the guard and is then
read by the flash_read_*() calls in the loop body — including addresses that are not memory.

A card that stopped enumerating, and survived reset run, POWMAN cycles, VBUS cuts and a reflash,
turned out not to be wedged hardware at all:

pc   = 0x000003ec     bootrom exception handler
lr   = 0xfffffff9     EXC_RETURN — the core is inside an exception
CFSR = 0x00008200     BFSR 0x82: precise bus fault, BFAR valid
HFSR = 0x40000000     FORCED -> HardFault
BFAR = 0x40130000     peripheral space: not flash, not SRAM

Every reset re-read the same corrupt link out of flash and faulted identically, which is why
nothing short of erasing the filesystem recovered it — and why it looked for a long time like a
hardware fault that survived power cycles.

flash_range_in_fs() already guards the WRITE and ERASE paths (low_flash.c). The READ path had
no equivalent, so #30's validation never reached this dereference.

         if (base == 0x0) { //all is empty
             break;
         }
+
+        /* A corrupt chain link must not be dereferenced: the loop guard bounds `base` only from
+         * below, so garbage above the pool is read by the flash_read_*() calls that follow. */
+        if (!flash_range_in_fs(base, 2 * sizeof(uintptr_t) + sizeof(uint16_t) + sizeof(uint32_t))) {
+            printf("SCAN: chain link %x is outside the filesystem — stopping scan\n",
+                   (unsigned int) base);
+            break;
+        }

Stopping the scan at the first bad link keeps the files found so far usable, so a corrupt tail
costs a truncated scan instead of a device only a physical BOOTSEL recovers.

Verification

The bricked card was recovered by BOOTSEL and reflashed by UF2 — which rewrites only the
program region and does not erase the filesystem. So the corrupt link that caused the fault was
still present, and the card booted straight through it. The guard was tested against the exact
structure that bricked the device.

Then, with both fixes applied:

40 reboots (rescue applet reboot):  38 direct, 0 wedges
60 reboots (debugger detached):     56 direct, 4 wedges, 4/4 recovered unattended,
                                    0 human interventions

against wedges at reboot 23 and 37 on the two runs before them. My full staging battery is now
green end to end: 22 passed, 0 failed (hardened init, RRC behaviour, PIN posture, the PKA probe,
19 scenarios, and the break-glass recovery drill).

One measurement caution, because it cost me most of a day

A debugger attached to the target manufactures this exact symptom. OpenOCD halts the core when
it re-examines after the device drops off USB, and a halted core cannot enumerate. In one 40-reboot
run, 4 of 5 apparent wedges were that. A long-lived OpenOCD also goes stale and reports the chip
confidently wrong — targets said core0 was in state reset with all memory reads failing,
persisting across reset run, while a fresh server on the same chip said halted and one
reset run returned the card in 4 s.

Anyone measuring reset reliability on RP2350B should detach the debugger or rule it out explicitly
before counting a failure.

Happy to send the scan guard as its own PR if you want it separately from #30 — it is small and
independent. Say the word and I will open it.

One RP2350B (Waveshare RP2350-PiZero) + Raspberry Pi Debug Probe. No RP2040 hardware here; the
RP2040 side stays compile-only.

[jobordu]

Correcting myself: I proposed the guard you had already argued against

Re-reading your review properly, you had addressed my scan guard before I wrote it:

I intentionally left the linked-list pointers as integrity indicators. Preventing the resulting
HardFault by silently rejecting the address does not explain why a non-flash value reached
prev_addr, and may hide the original failure.

That is exactly what I posted, and I should have caught it. Apologies — you had already given the
specification and I implemented a third of it:

scan_flash() should validate that every filesystem pointer remains within the configured region
and moves strictly toward the start of the region. On detecting corruption, the firmware
should stop scanning and expose a controlled recovery state rather than automatically erasing
potentially valuable HSM data.

your requirement	my first patch
pointer inside the configured region	done
moves strictly toward the start	missing
stop and expose a controlled recovery state	missing — it stopped silently

Both are now implemented. The direction check turns out to matter on its own: a range-only check
still admits a link that points forward or at itself, which produces an infinite scan rather
than a bus fault. Different failure, equally fatal, and my version would have concealed it.

The state is now exposed rather than swallowed (fs_corruption_detected / _addr / _prev),
nothing is erased, rewritten or skipped, and what to do about a corrupt filesystem stays an
operator decision — so the pointers keep working as the integrity indicators you intend.

I am not asking you to take this patch. Your point stands that a guard does not explain how a
non-flash value reached the link, and that question is still open.

What I still owe you

Going back through your review, these are unanswered and I do not want to pretend otherwise:

Bug 3. The value of the unread FIFO word, and whether it matched a launch word or the
multicore lockout request ID (0x73a8831e). Also distinguishing "core1 never launched" from
"core1 launched and was reset later". I have not captured this; #28 is in draft and its evidence
is weaker than I originally claimed — a core parked at bootrom 0x19e with core1_alive == 0 is
also the normal lazy-launch idle state, since card_start() only runs from the command paths.

Bug 6. You asked for the raw physical record header, the cached-sector copy, all flash_pages
entries and the original unaligned prev_addr from the same halted state. I sent fault
registers (CFSR/HFSR/BFAR) but not the flash structures, which are the part that would
actually distinguish physical-flash corruption from a corrupted dirty-sector cache or an invalid
file->data. Reproducing that state deliberately is the next thing I plan to do rather than
speculate.

Splitting. You asked twice for one issue and one PR per bug. My series is still organised the
way you found it. Given how much of what I filed I have since had to retract, I would rather not
open more PRs until you have had a chance to say which of the existing ones are worth keeping.

Answered since your review, in case it is useful

• Bug 6 (2) — yes, the unwrapKey SW=6400 was on a build carrying my out-of-tree dead-man's
  switch, which ate CCID completion events exactly as you predicted. Removed; the hang went from
  1 ok / 5 bad to 0 hangs. Not a firmware defect.
• Bug 6 (1) — the all-zero KCV is read on an unauthenticated path: mkek_load_file returns
  PICOKEYS_NO_LOGIN, dkek_kcv() returns early leaving the buffer zeroed, and cmd_key_domain.c
  discards that return value. dkek_is_complete = 1 throughout. Not lost DKEK state.
• Bug 4 — the END-timeout warning was never captured. The parked core1 I reported came from
  core0 panicking between lockout START and END, so fix(flash): raise lockout timeouts, abort the drain pass on end-timeout #29 addresses a scenario I never observed. It
  is in draft.
• Bug 2 — you were right that PICO_PLATFORM was misused. usb.c included
  hardware/powman.h under PICO_PLATFORM rather than PICO_RP2350, which broke the RP2040 build
  outright in my branch. Fixed there; your tree is unaffected.

[jobordu]

Current state of this thread, because the opening report is now misleading

This issue has twelve comments and the body at the top still leads with claims I have since
retracted. Anyone reading top-down gets a picture I no longer stand behind, so here is the state in
one place.

Retracted — do not spend time on these

original claim	what it actually was
scan_region() root-cause attribution	wrong; the real cause was a FILE_DATA_FUNC function-pointer write
"EF.DIR / EF.TokenInfo writes block provisioning"	wrong; OpenSC ignores that failure and never writes EF.DIR
unwrapKey SW=6400	my own out-of-tree dead-man's switch eating CCID completions — exactly the card_to_usb_q contention you predicted
all-zero DKEK KCV = lost state	artefact of an unauthenticated read; dkek_is_complete = 1 throughout
core1 launch failure (#28)	bootrom 0x19e is normal lazy idle, and FIFO_ST = 0x02 (VLD=0) — the stale word does not exist. PR closed
lockout END timeout (#29)	the warning was never captured; scenario never observed. PR closed
deep reset as the fix (#31)	premise inverted — see below. PR closed

Holds up, with hardware evidence

finding	evidence	PR
DevAut deadlock — dev_name needs EF.C_DevAut to create EF.C_DevAut	provisions end-to-end after the fix	pico-hsm#137
C.DevAut buffer reuse — 940 B = request + cert	CVC_ELEMENTS=1 after, 2 before	pico-hsm#138
FILE_DATA_FUNC written as a flash address	scsh aborts on 6581; OpenSC ignores it	#32
fs pointer validation before flash ops	BFAR=0x40130000 HardFault	#30
watchdog reset loop	WATCHDOG_REASON = TIMER, CTRL = 0x40000000, LOAD = 0	no PR yet
scan chain-link validation (range + strictly-decreasing + exposed state, per your spec)	same HardFault	no PR yet

The one that inverts #31

watchdog_reboot() leaves the watchdog ENABLED across the reset it causes. rescue.c uses a 100 ms
window, and the startup flash scan grows with file count — so a populated card can overrun it and
reset again, forever. rom_reboot() does not avoid this; the bootrom reboots via the watchdog too.

That is why the state survives POWMAN cycles and a full VBUS cut: the firmware re-arms the trap
on the way up. I had proposed a deeper reset for this. A bigger hammer could never have fixed it.

Fix is a disarm at PICO_RUNTIME_INIT_EARLIEST — a disarm in main() was measured NOT to work,
because the loop never reaches main().

What I owe you

• the Bug 6 flash-side forensics (raw record header, cached-sector copy, flash_pages, the original
  unaligned prev_addr from one halted state). Capture is built; waiting on a reproduction.
• separate PRs for the two fixes above. I have deliberately not opened them yet — you asked for
  time and for focused changes, and given how much of this report I have had to withdraw, I would
  rather you say which are worth having than add more to your queue.

No action needed from you on any of it. Take the time you asked for.