Skip to content

fix: mask cert serial MSB to prevent DER Overlength panic in --generate#220

Draft
ryancee wants to merge 1 commit intostr4d:mainfrom
ryancee:fix/cert-serial-overlength
Draft

fix: mask cert serial MSB to prevent DER Overlength panic in --generate#220
ryancee wants to merge 1 commit intostr4d:mainfrom
ryancee:fix/cert-serial-overlength

Conversation

@ryancee
Copy link
Copy Markdown

@ryancee ryancee commented Mar 24, 2026

Summary

--generate panics with Overlength at builder.rs:127 on approximately every other invocation.

Root cause

builder.rs generates a 20-byte random certificate serial and passes it directly to SerialNumber::new:

let mut serial = [0; 20];
OsRng.fill_bytes(&mut serial);
SerialNumber::new(&serial).expect("valid")   // panics ~50% of the time

A DER INTEGER requires a 0x00 sign-extension byte when the MSB of the first content byte is set (to indicate a positive value). For a 20-byte random array this happens ~50% of the time, producing a 21-byte encoding that exceeds the RFC 5280 §4.1.2.2 limit of 20 bytes. x509-cert 0.2 / der 0.7 returns Error { kind: Overlength, position: None } and the .expect("valid") call panics with:

thread 'main' panicked at src/builder.rs:127:40:
valid: Error { kind: Overlength, position: None }

Fix

Mask the high bit of the first byte before constructing SerialNumber to ensure the value is always non-negative and the DER encoding never exceeds 20 bytes:

serial[0] &= 0x7f;

This is the standard approach used by most X.509 certificate generators (see e.g. OpenSSL, rcgen, rustls-cert-gen). The existing TODO comment notes that SerialNumber::generate (RustCrypto/formats#1270) will eventually handle this correctly; until that lands, the manual mask is the correct fix.

Testing

Verified on YubiKey 5C NFC (serial 37153875, firmware 5.7.4) with AES256 management key. Previously --generate failed consistently on this device (panicking at builder.rs:127). With this fix applied the command completes successfully and produces a valid recipient/identity stub.

Also related to / builds on top of #219 (wrong OID in util.rs).

Note on branch base

This branch is based on the PR #190 branch (str4d:yubikey-0.8) which has not yet been merged. The fix applies equally to main once #190 merges — serial[0] &= 0x7f is a one-line change independent of the rest of the #190 changes.

@ryancee ryancee force-pushed the fix/cert-serial-overlength branch from 97684df to 6a350c8 Compare March 24, 2026 02:38
SerialNumber::new in x509-cert / der encodes the value as a DER INTEGER.
When the MSB of the first byte is set the encoder prepends a 0x00
sign-extension byte, producing a 21-byte encoding that exceeds the
RFC 5280 §4.1.2.2 limit of 20 bytes. The crate returns Overlength
and the .expect("valid") call panics.

Because the 20-byte array is random, the MSB is set ~50% of the time,
so --generate panics on roughly every other invocation.

Fix: mask serial[0] &= 0x7f before constructing the certificate to ensure
the value is always positive and the DER encoding never exceeds 20 bytes.

Tested on YubiKey 5C NFC (serial 37153875, firmware 5.7.4).
@ryancee ryancee force-pushed the fix/cert-serial-overlength branch from 6a350c8 to 15b8111 Compare March 24, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant