Skip to content

fix(#75): recompute every sub-image HMAC on revert, and stop inferring success from silence - #107

Merged
AndrewDemsDS merged 7 commits into
mainfrom
fix/75-inner-image-hmac
Jul 27, 2026
Merged

fix(#75): recompute every sub-image HMAC on revert, and stop inferring success from silence#107
AndrewDemsDS merged 7 commits into
mainfrom
fix/75-inner-image-hmac

Conversation

@AndrewDemsDS

Copy link
Copy Markdown
Owner

Closes the root cause behind #75 and hardens the tooling that let it through.

Root cause. The bootloader verifies an inner HMAC per sub-image that revert --repackage never
rebuilt. Patching the serial stales sub-image 0's trailer while the manifest signature and byte-sum
still verify, so every host check passed on an image boot_load rejects ("Hash Result Incorrect!",
QE bit cleared, hang, no fall-back to the other slot).

Confirmed on hardware 2026-07-27: a repackaged stock image booted (VID 5004 / PID 13825 / sw 2),
and it booted from FW2, so the format is slot-independent.

What is in here:

  • Sign and verify the whole sub-image chain, not just sub-image 0. --apply re-verifies the
    payload before staging (the four pre-RE bootloader slot-validity check: why re-signed images fail to boot (blocks Path 2 revert) #75 bricking payloads were still valid targets by filename),
    and --repackage refuses a carve whose serial is not stock.
  • Corrects a naming trap: u32le(img[0xE0]) is the segment size, not next_img. The code read
    the right offset so it worked, but generalising the wrong name puts the trailer at 0x4060 instead
    of 0x3d80 and every check then reports MISMATCH.
  • revert --apply no longer treats "left the fabric" as success. A healthy reverted unit and a
    rejected one are both silent, because stock joins its own network. Three-way verdict plus QE-bit
    triage, and a new read-only revert --slots where an answer is decisive and silence is not.
  • QA Layer 4 walks every archived image and dump; the flash captured off the bricked unit is kept as
    a known-mismatch fixture. Also fixes Layer 3, which a doubled path meant had never actually run.
  • Panel "77" now shows for every commissioning window, not just the remote gesture. Verified on
    hardware (v1.3.31, node 64) via the Administrator Commissioning cluster: prov_status 0 -> 1 with
    the panel lit, back to 0 with it cleared. Previously a pairable unit showed nothing, which is what
    made it look bricked.

Assisted-by: AI

The bootloader verifies a second HMAC the repackage recipe never rebuilt:
HMAC-SHA256(hash_key, img[0:L]) == img[L:L+0x20], L = next_img + 0x140.
Its span starts at offset 0, so patching the serial invalidated it while the
manifest signature and byte-sum still verified, which is why every host check
passed and the image still bricked a unit (boot_load prints "Hash Result
Incorrect!", clears the flash QE bit, and hangs with no fall-back).

revert --repackage now signs serial -> manifest sig -> inner HMAC -> byte-sum,
and the pre-build self-check verifies all three. Verified on 29 real images;
round-trip with an unchanged serial reproduces the original byte-for-byte.
Not yet confirmed on hardware.

Assisted-by: AI
…uting

revert --repackage consumes int v then bumps version.txt to v+1 (its guard
cur_version <= v is a tautology, since v is max(cur,released)+1). A separately
invoked revert --apply recomputed v+2 and died looking for a pair that was
never built, so the documented two-command flow could never work.

Also records the fleet version consumed by tonight's revert build.

Assisted-by: AI
The docs and the inner_off comment called u32le(img[0xE0]) "next_img". It is
the segment SIZE. next_img lives at H+4 and is a RELATIVE offset. The code read
the right offset so it worked, but generalising the wrong name to sub-images
1..N puts the trailer at 0x4060 instead of 0x3d80 and every check then reports
MISMATCH. Corrected in docs/10, docs/13 and the code.

Verification now walks the whole chain instead of sub-image 0 only:
- --repackage self-checks every trailer on its output and on each archived image
- --apply re-verifies the payload behind the .ota before staging (the four
  pre-#75 bricking payloads are still on disk and were valid targets by filename)
- --repackage refuses a carve whose serial is not stock (< SERIAL_BASE)

Confirmed false-refusal-free over the real archive; the v10314 payload that
bricked the office unit is now rejected by name and offset.

Also records that Path 2 is hardware-confirmed (2026-07-27) and corrects
docs/12 step 7: the Matter DCT survives a stock<->custom round trip, so a unit
with a prior custom life boots WITH fabrics and suppresses BLE advertising.

Assisted-by: AI
"Node dropped off the fabric" was treated as SUCCESS. A healthy reverted stock
unit and a bootloader-rejected module are BOTH silent on our fabric, because
stock leaves it and joins its own network. That inference mislabelled two
healthy units as bricks on 2026-07-26/27.

--apply now returns a three-way verdict (reverted / not reverted / ambiguous)
and, when ambiguous, prints the triage that actually discriminates: the flash
QE bit (cleared = the bootloader rejected it), the vendor app, and reading the
app slots from a clip dump. Adds a pre-apply summary naming the target node and
which slot the image lands in.

New read-only `revert --slots <ip>` for triage: an answer proves the custom
firmware is alive, silence proves nothing, and it says so rather than guessing.

Assisted-by: AI
Adds firmware/scripts/amebaz2_image.py (one shared implementation of the carve /
walk / verify / resign recipe) and firmware/test/test_image_chain.py, wired in as
QA Layer 4. It walks every archived image and dump and asserts the manifest
signature, every sub-image trailer and the byte-sum. The flash captured off the
bricked office unit is kept as a known-mismatch fixture so the #75 signature
stays pinned.

run_tests.sh cd's to its own directory at line 5 and then re-resolved
$(dirname "$0") for Layers 3 and 4, producing a doubled path. Layer 3 has
therefore never actually executed. Both now run.

Also hardens ota_convert_stock.sh for the DCT-survives case: a unit with a prior
custom life keeps its Matter fabrics through the conversion, so BLE advertising
is suppressed and :wipekv is required before commissioning.

Assisted-by: AI
The panel lights "77" iff our outbound 0x1E heartbeat reports prov_status=1
(payload[4] bit3, bench-proven 2026-07-09). Only recommission_open_window()
ever set it, so a window we did not open ourselves left the panel blank while
the device was in fact pairable: the autostart window Server::Init opens when
FabricCount()==0 (the first boot after :wipekv), and any window opened via the
Administrator Commissioning cluster.

That is the ambiguity that cost hours on the office unit. With nothing on the
panel, "pairable" and "bricked" look identical from outside.

Hooks CommissioningWindowManager's AppDelegate, so one delegate covers all
window sources. It only flips the existing flag; no frame is queued, the next
~1Hz 0x1E carries the bit, so nothing new goes on the sniff-confirmed bus and
the prompt-clear push stays in recommission_finish.

Builds clean at v1.3.31 (serial 11431). Not yet flashed.

Assisted-by: AI
Office unit (node 64) on v1.3.31. Window opened via the Administrator
Commissioning cluster, i.e. a source that lit nothing before this change:
prov_status 0 -> 1 with the panel showing 77, and back to 0 with the panel
clearing. Confirmed both on the panel and through the `link` diag console,
which reports the outbound bit directly so this is checkable over the network
without standing at the unit.

Assisted-by: AI
@AndrewDemsDS
AndrewDemsDS merged commit 91b1c3b into main Jul 27, 2026
7 checks passed
@AndrewDemsDS
AndrewDemsDS deleted the fix/75-inner-image-hmac branch July 27, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant