Skip to content

fix(hsm): make the bootstrap device name device-unique, not a shared constant - #137

Open
jobordu wants to merge 1 commit into
polhenarejos:masterfrom
jobordu:fix/devaut-bootstrap
Open

jobordu wants to merge 1 commit into
polhenarejos:masterfrom
jobordu:fix/devaut-bootstrap

Conversation

@jobordu

@jobordu jobordu commented Aug 5, 2026

Copy link
Copy Markdown

Rebased onto master and cut down to one value change. The deadlock this PR originally fixed is
now fixed by your 49616b4 — that part is yours and I have dropped it. What is left is the fallback
CHR itself.

The change

reset_puk_store() keeps exactly the control flow 49616b4 introduced. Only the value changes:

 if (!dev_name) {
-    dev_name = (const uint8_t *) "ESPICOHSMTR00001";
-    dev_name_len = (uint16_t)(strlen((const char *)dev_name));
+    dev_name = hsm_bootstrap_dev_name(&dev_name_len);
 }

where hsm_bootstrap_dev_name() formats "ESP%02X%02X%02X%02X00001" from the last four bytes of
the board serial.

Why the value matters

The fallback is not a transient placeholder — it becomes the card's permanent identity:

  1. init_sc_hsm() calls reset_puk_store() at boot, so on a card with no EF.C_DevAut,
    dev_name is already the fallback before any command runs;
  2. INITIALIZE builds the device certificate with that CHR and stores it in EF_TERMCA
    (cmd_initialize.c, asn1_cvc_cert then file_put_data);
  3. every subsequent boot reads dev_name back out of that certificate.

So whatever the fallback is on first provisioning, the card keeps forever.

OpenSC derives the PKCS#11 token serial from that CHR by stripping the last five characters
unconditionally — src/libopensc/pkcs15-sc-hsm.c:1448:

len = strnlen(devcert.chr, sizeof devcert.chr);  /* Strip last 5 digit sequence number from CHR */
if (len < 8)
    return SC_ERROR_INTERNAL;
len -= 5;

With a fixed ESPICOHSMTR00001, every device provisioned from a blank filesystem reports
token serial = ESPICOHSMTR. Identically, and permanently. Two such cards on one host cannot be
distinguished by the field PKCS#11 tooling uses to select a token.

With this change the same card reports a serial derived from its own board id — on my bench,
CHR ESP2202E14A00001 -> serial num : ESP2202E14A, read back through pkcs11-tool.

Portability

pico_serial is filled by serial_init() on every supported platform including ESP32
(serial.c has an ESP_PLATFORM branch), and PICO_UNIQUE_BOARD_ID_SIZE_BYTES is 8 or 16, so the
last four bytes always exist. The 11+5 CVC holder-reference layout is preserved, as is the
recognisable ESP prefix.

Verification

Compile-checked on this rebase: PICO_BOARD=waveshare_rp2350_pizero, arm-gnu-15.2 — clean,
0 warnings. strings on the resulting ELF shows the new format string present.

Hardware evidence for the mechanism (device-unique CHR surviving provisioning, wrap/unwrap round
trip, RRC posture) is from RP2350B, on this one board. I have no ESP32 hardware, so the ESP32
claim above is source-level only.

Not included, deliberately

cvc.c:43 still carries car = dev_name ? dev_name : "ESPICOHSMTR00001";. As far as I can tell
that branch is now unreachable — reset_puk_store() guarantees dev_name is non-NULL before any
command runs — so I left it alone to keep this diff to one value. Say the word and I will fold in
its removal, or leave it as the belt-and-braces it now is.

cmd_initialize.c:255's "ESPICOHSMTR" is a PRKD label, not a CHR, and is untouched.

@jobordu

jobordu commented Aug 5, 2026

Copy link
Copy Markdown
Author

End-to-end confirmation on hardware: with this patch a card that has been fully erased provisions and completes a key wrap/unwrap round trip, which was impossible before.

Sequence on a card after a full 16 MB erase, single INITIALIZE on the patched firmware:

token flags : login required, rng, token initialized, PIN initialized
serial num  : ESP2202E14A

wrap   key reference 3   -> 367-byte blob
unwrap into reference 7  -> "Wrapped key contains: Key blob, Private Key Description (PRKD)"
                            "Key successfully imported"

Instrumenting cmd_keypair_gen also shows the bootstrap working exactly as intended — asn1_cvc_aut now returns a 490-byte CVC where it previously returned 0:

mbedtls_ecdsa_genkey = 0     ok
cvc_pk_wrap_ec       = 0     ok
asn1_cvc_aut         = 490   ok   (was 0 -> SW_EXEC_ERROR before the patch)
store_keys           = 0     ok
reached SW_OK()

Unrelated defect found while testing, reported here only so it is not mistaken for a regression from this patch:

pkcs11-tool --keypairgen hangs indefinitely (reproduced at 120 s / 180 s / 300 s, on both prime256v1 and secp256k1, after a clean reset with a healthy token). SWD shows both cores back in the normal core0_loop/hwrng_task idle path — the card is not computing. The trace above shows the handler completed and returned SW_OK; the 490-byte response never reaches the host. The key really is stored: after a reset it can be wrapped by reference, which is how the round trip above was done.

So this looks like a response-path/chaining problem for responses beyond the short-APDU limit, independent of the deadlock this PR fixes. I have not root-caused it and am not proposing a change for it here. It also plausibly explains older reports of --keypairgen "taking 35 minutes" or returning CKR_GENERAL_ERROR — the host was blocked on a response that never arrived rather than on slow on-card maths.

Happy to open a separate issue with the trace if useful.

@jobordu

jobordu commented Aug 5, 2026

Copy link
Copy Markdown
Author

Correction to my note above about --keypairgen, so the record is accurate.

I described it as the response deterministically never reaching the host. That was overstated. With more runs the behaviour is intermittent:

  • ~1 success in 6 attempts overall. One run completed fully and printed the key pair (Key pair generated / EC_POINT 256 bits); the rest hung past 180 s.
  • I briefly suspected host-side slowdown mattered, because the one success had OPENSC_DEBUG=9 on. That is disproven — two further OPENSC_DEBUG=9 runs also hung (0/2). So debug logging is not the variable.

What is solid:

  • The card side always completes: instrumented cmd_keypair_gen shows mbedtls_ecdsa_genkey = 0, cvc_pk_wrap_ec = 0, asn1_cvc_aut = 490, store_keys = 0, and the handler reaching SW_OK().
  • Both cores return to the normal core0_loop/hwrng_task idle path — the device is not stuck computing.
  • The key is genuinely persisted: after a reset it can be wrapped by reference, which is how I ran the wrap/unwrap round trip.

So it is an intermittent failure on the response path for a >256-byte response, trigger not yet identified. I have not root-caused it and I am not proposing a change for it here — flagging it only so it is not mistaken for a regression from this PR, and correcting my earlier overstatement.

This does not affect the fix in this PR, which is verified independently: two runs from a full 16 MB erase both reach token flags: … PIN initialized with a device-unique serial.

@jobordu

jobordu commented Aug 6, 2026

Copy link
Copy Markdown
Author

Retracting the --keypairgen hang I reported above. It was my own instrumentation, not this
firmware.
It has nothing to do with this PR, and you should not spend any time on it.

My test build carried an out-of-tree core1 "dead-man's switch" in card_watchdog_task()
(pico-keys-sdk, never submitted here). It popped card_to_usb_q itself — which is precisely what
you warned about when you reviewed that series:

the dead-man switch ... competes with card_status() for card_to_usb_q and can consume normal
completion or button events

That is exactly what happened, and the mechanism is unambiguous:

  • card_status() pops EV_EXEC_FINISHEDccid_task() calls driver_exec_finished_ccid()
    the response goes to the host.
  • the watchdog task popped it first → it only cleared a local flag → the response was never
    sent
    and the host waited forever.

Both run on core0, so it was a race, which is why it looked intermittent and why I could not find
a pattern in it.

Measured on hardware, same board, same command:

build pkcs11-tool --keypairgen
with the dead-man's switch 1 ok / 5 bad, failures hanging for minutes
without it 0 hangs, successes complete in ~5 s

The switch is now removed outright. I did not replace it with anything — the liveness problem it
was supposed to catch was never independently substantiated.

Note the timestamps: I posted the hang report at 23:10 UTC on 2026-08-05 and removed the switch at
23:15 UTC the same evening, so that comment describes a build I had already stopped trusting a few
minutes later. My apologies for the noise.

What this does not change: the instrumented card-side evidence for this PR stands, because it
was always about what the card computes, not about response delivery — mbedtls_ecdsa_genkey = 0,
cvc_pk_wrap_ec = 0, asn1_cvc_aut = 490 (was 0SW_EXEC_ERROR before the patch),
store_keys = 0. The DevAut bootstrap this PR adds is unaffected either way.

To be precise about which build each earlier measurement came from: the wrap/unwrap round trip I
reported was also taken on the switch-carrying build. It is unaffected, because the switch could
only ever lose the delivery of a completion event — and those operations completed and returned
their results. It cost me nothing but the keygen hang, which is the one thing I am retracting.

@jobordu

jobordu commented Aug 7, 2026

Copy link
Copy Markdown
Author

Validation status, so this is not sitting here unqualified.

With this applied (plus the other pico-hsm patch in this pair and pico-keys-sdk#32), my staging
battery's phase group — hardened init, RRC behaviour verification, production PIN posture, the PKA
probe, and the break-glass recovery drill — has run 7 passed / 0 failed / 0 skipped on
repeated full runs, on a fully erased card each time.

Two corrections I have posted elsewhere and want visible here too, since both were mine:

  • the intermittent --keypairgen hang I reported was my own out-of-tree instrumentation eating
    CCID completion events, not this firmware;
  • an unrelated change of mine (polhenarejos/pico-keys-sdk#31, now draft) armed POWMAN_WDSEL
    globally and broke the rescue applet's reboot. If you were testing against my branch rather than
    these two patches alone, that would have muddied the results.

Nothing here needs action — recording it so the state is current.

@jobordu

jobordu commented Aug 8, 2026

Copy link
Copy Markdown
Author

Status: still current, and directly exercised on a binary with verified provenance

Unchanged. This is the bootstrap that lets a card with no EF.C_DevAut self-provision a device name rather than deadlocking.

I corrected a provenance gap on my side since posting this — my superproject pinned pico-keys-sdk five commits behind my own checkout, so a clean clone built firmware without the fixes while my source checks reported green (background in polhenarejos/pico-keys-sdk#25). Every number I quote now comes with a verified chain: device flash dump → ELF → SDK commit → superproject pin.

On that binary the device-identity path this PR covers is read straight off the card and parsed offline:

CHR=ESP2202E14A00001  CAR=ESP2202E14A00001  chain=unverified

chain=unverified is deliberate — I have no upstream CVCA public key to verify against, so I report the parse and not a trust decision. The full staging battery around it is green (19/19 scenarios, 26/26 recovery checks, 7/7 phases).

RP2350B only; no RP2040 on this bench.

@jobordu

jobordu commented Aug 10, 2026

Copy link
Copy Markdown
Author

New evidence: the deadlock scenario reproduced accidentally, and the bootstrap resolved it

Still current and unchanged. Since the last update this fix was exercised by the exact failure it
was written for, which I had not previously been able to demonstrate end to end.

While investigating an unrelated filesystem issue (#35) my card reached a state where INITIALIZE
failed outright:

sc_card_ctl(*, SC_CARDCTL_SC_HSM_INITIALIZE, *) failed with Card command failed

Recovery required erasing the filesystem region over SWD:

flash erase_address 0x103f0000 0x10000

That leaves no EF.C_DevAut at all — the precise condition in the body of this PR: dev_name
has no source, asn1_cvc_aut() refuses to build the EE certificate, and INITIALIZE cannot create
the certificate that would supply the name.

On the patched build, initialization then succeeded immediately:

sc-hsm-tool --initialize --so-pin ... --pin ... --pin-retry 3
Version              : 6.6
SO-PIN tries left    : 15
User PIN tries left  : 3

and the card went straight back to generating keys. The bootstrap path is the only thing that makes
that possible — with a blank filesystem there is nothing else for dev_name to come from.

So this is no longer only a code-reading argument about a deadlock: a card with a fully erased
filesystem self-provisioned, which is the outcome the PR claims. Verified present in the flashed
tree (src/hsm/sc_hsm.c:265, hsm_bootstrap_dev_name).

Nothing else about the PR has changed.

@polhenarejos

polhenarejos commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Possibly related with #139
Can you check it?

polhenarejos added a commit that referenced this pull request Aug 20, 2026
Closes #139 #137

Signed-off-by: Pol Henarejos <pol.henarejos@cttc.es>
…constant

49616b4 fixed the initialization deadlock on a card with no EF.C_DevAut by falling
back to a fixed CHR. This keeps that control flow exactly and changes only the value.

The fallback CHR is not transient. reset_puk_store() runs from init_sc_hsm() at boot,
so on a card with no EF.C_DevAut dev_name is already set to the fallback by the time
INITIALIZE builds the device certificate — and that certificate is then stored in
EF_TERMCA and read back as dev_name on every subsequent boot. The bootstrap name
becomes the card's permanent identity.

OpenSC derives the PKCS#11 token serial from that CHR by stripping the last five
characters unconditionally:

    src/libopensc/pkcs15-sc-hsm.c:1448
    len = strnlen(devcert.chr, sizeof devcert.chr);  /* Strip last 5 digit
                                                        sequence number from CHR */
    len -= 5;

So with a fixed "ESPICOHSMTR00001" every device that provisions without a pre-existing
EF.C_DevAut reports serial "ESPICOHSMTR" — permanently, and identically. Two such cards
on one host cannot be told apart by serial, which is how PKCS#11 tooling selects a token.

Using "ESP" + 32 bits of the board serial keeps the recognisable prefix and the 11+5
CVC holder-reference layout, while making the derived serial distinct per device.

pico_serial is filled by serial_init() on every supported platform including ESP32,
where PICO_UNIQUE_BOARD_ID_SIZE_BYTES is 8, so the last four bytes always exist.

Compile-checked: PICO_BOARD=waveshare_rp2350_pizero, arm-gnu-15.2, clean, 0 warnings.
@jobordu
jobordu force-pushed the fix/devaut-bootstrap branch from 60bfe37 to 591f554 Compare September 2, 2026 07:02
@jobordu jobordu changed the title fix(hsm): bootstrap the device name so a card with no EF.C_DevAut can self-provision fix(hsm): make the bootstrap device name device-unique, not a shared constant Sep 2, 2026
@jobordu

jobordu commented Sep 2, 2026

Copy link
Copy Markdown
Author

Yes — related, and your fix supersedes most of this PR. Rebased and cut down to what is left.

Sorry for the slow reply.

Checked. #139 and this PR are the same deadlock: on a card with no EF.C_DevAut there is no
source for dev_name, asn1_cvc_aut() refuses to build the EE certificate, and INITIALIZE
cannot create the very file that would supply the name.

Your 49616b4 fixes it, and I agree with the shape you chose over #139's. Forcing
recreate_dev_key when EF_TERMCA has no data is the right condition — #139's version would have
let a card take the fast path with an empty EF_EE_DEV/EF_TERMCA, which is the state that made
the retry look like a success. That part of my PR is now yours and I have dropped it.

What I have rebased this down to is one value. Not the control flow — that is yours, unchanged
— only what the fallback CHR contains:

 if (!dev_name) {
-    dev_name = (const uint8_t *) "ESPICOHSMTR00001";
-    dev_name_len = (uint16_t)(strlen((const char *)dev_name));
+    dev_name = hsm_bootstrap_dev_name(&dev_name_len);
 }

The reason it matters is that the fallback is not transient. init_sc_hsm() calls
reset_puk_store() at boot, so dev_name is already the fallback before INITIALIZE runs;
INITIALIZE then bakes that CHR into the device certificate and stores it in EF_TERMCA; every
later boot reads it back. The bootstrap name becomes the card's permanent identity.

And OpenSC turns that CHR into the PKCS#11 token serial by stripping the last five characters —
pkcs15-sc-hsm.c:1448,
comment and all: /* Strip last 5 digit sequence number from CHR */. So with the fixed constant,
every device provisioned from a blank filesystem reports serial = ESPICOHSMTR, forever, and two
of them on one host are indistinguishable by the field tooling uses to pick a token.

"ESP" + 32 bits of board serial + "00001" keeps your prefix and the 11+5 holder-reference layout
and makes the derived serial unique. pico_serial is filled by serial_init() on every platform
including ESP32, so this is not RP2350-specific — though I should say plainly that I have no ESP32
hardware and that claim is source-level only.

Compile-checked on the rebase: PICO_BOARD=waveshare_rp2350_pizero, arm-gnu-15.2, clean, 0
warnings. Diff is now +28/-2 in two files.

One loose end I deliberately did not touch: cvc.c:43 still has
car = dev_name ? dev_name : "ESPICOHSMTR00001";. With your fix that branch looks unreachable to
me, so I left the diff at one value rather than widening it. Happy to fold in its removal if you
would rather not keep a second copy of the constant around.

@jobordu

jobordu commented Sep 2, 2026

Copy link
Copy Markdown
Author

The red checks are not from this change — fork PRs cannot decrypt the test flash image

Flagging so you do not have to re-derive it.

test (pkcs11) / test (pytest) / test (sc-hsm-pkcs11) fail at the same point for any PR from a
fork:

env:
  PICO_HSM_MEMORY_PASSPHRASE:              <- empty
...
memory archive decrypt failed: set PICO_HSM_MEMORY_PASSPHRASE or PICO_HSM_MEMORY_PASSPHRASE_FILE

.github/workflows/test.yml:76 passes ${{ secrets.PICO_HSM_MEMORY_PASSPHRASE }}, and GitHub does
not expose repository secrets to pull_request runs originating from a fork, so the value arrives
empty and tests/memory_archive.py cannot open the flash image.

Same signature on #138test (pkcs11): failure, the other two cancelled — which you merged, and
which is green on master where the secret resolves. build, both CodeQL Analyze jobs and
GitGuardian pass here.

So nothing in this PR is exercised by those jobs, and nothing in it caused them to fail. It does
mean the emulation suite has not run against this change anywhere I can reach; if you want that
before merging, it needs a run on a branch in your repo. Compile verification I can do and did:
PICO_BOARD=waveshare_rp2350_pizero, arm-gnu-15.2, clean, 0 warnings.

(If it is worth fixing generally, pull_request_target or a fallback to an unencrypted fixture when
the secret is absent would let external contributions be validated. Entirely your call — I mention
it only because it currently applies to every PR you receive from outside.)

@jobordu

jobordu commented Sep 3, 2026

Copy link
Copy Markdown
Author

Demonstrated on two physical devices — the claim in this PR is no longer inferred from the code

A second RP2350B board joined the bench today, so the thing this PR asserts can now be shown.
Previously I had one board and could only argue device-uniqueness by reading hsm_bootstrap_dev_name().

The prediction was made before flashing. I read the OTP chip id of the new board over SWD
(mdw 0x40130000 2) while it was still running unrelated firmware, and wrote down what the patch
should produce:

board #2  OTP chip id 8625B328 41D722E2  ->  predicted CHR ESP41D722E200001  ->  serial ESP41D722E2

Then a full chip erase (flash erase_sector 0 0 last, so no EF.C_DevAut at all), flash, and a
single INITIALIZE. Result:

Slot 0 (0x0): Pol Henarejos Pico Key CCID Interface 01
  token label        : Pico-HSM
  serial num         : ESP41D722E2
  uri                : pkcs11:...;serial=ESP41D722E2;token=Pico-HSM

Slot 1 (0x4): Pol Henarejos Pico Key CCID Interface
  token label        : Pico-HSM
  serial num         : ESP2202E14A
  uri                : pkcs11:...;serial=ESP2202E14A;token=Pico-HSM

Both devices, both predictions:

device OTP chip id predicted observed
board #1 C858BA45 2202E14A ESP2202E14A ESP2202E14A
board #2 8625B328 41D722E2 ESP41D722E2 ESP41D722E2

Two SmartCard-HSMs on one host, each selectable by a pkcs11:serial= URI. On master's fixed
constant both of those URIs would read serial=ESPICOHSMTR and neither would identify a device.
That part is not a measurement and does not need to be — it is a string literal.

One incidental finding you may care about more than the above

Slot ids here are 0x0 and 0x4, not 0 and 1. Anything selecting a token by
--slot <n> treating n as an ordinal picks the wrong card as soon as there are two. Not a firmware
issue, just a trap I walked into in my own harness and thought worth passing on.

Scope, honestly

RP2350B, two boards, PICO_BOARD=waveshare_rp2350_pizero, arm-gnu-15.2. Still no ESP32 hardware
here, so the portability argument for pico_serial remains source-level only.

I have not re-measured the stock-firmware arm end to end. I flashed master onto the same board
first, and after INITIALIZE it stopped answering SELECT (valid ATR, Transmit failed), so I
never read a serial off it. That is a separate matter from this PR — it is the failure
polhenarejos/pico-keys-sdk#32 addresses, and master's SDK does not carry #32 — and I am not
claiming a mechanism for it here on a single observation. I mention it only so the gap in the
evidence is visible rather than quietly omitted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants