Skip to content

custodian: reconstruction repairs the first committed reference to a chunk — a duplicate, or one hidden behind an unreadable record, is neither detected nor protected #700

Description

@eduralph

Carved out of #697 during a re-plan (2026-08-07), after five rounds in which every blocking
finding traced back to this one rule. Recorded rather than folded in, because settling it needs
its own design decision — not a byproduct of a resolver-wiring patch.

The property, on origin/main @ 339da46

A repair obligation is keyed by chunk alonerepair:<chunk_id> (crates/core/src/repair.rs:32).
The committed namespace is not: two committed inode records, or one record twice, may reference the
same ChunkId, and both references address the same FragmentId { chunk, index }
(crates/traits/src/lib.rs:45-48).

reconstruction::find_chunk (crates/custodian/src/reconstruction.rs:620-646) answers with the
first committed reference in scan order (:639) and stops. So today reconstruction:

  • repairs whichever reference it met first — repointing that record's placement while
    orphan-marking a fragment the other reference still points at (repair_chunk, :599-608);
  • then drains the obligation, so the second reference is never re-assessed.

Chunk ids are allocator-minted, never content-addressed (crates/server/src/write.rs:170), so a
duplicate is always an anomaly — never legitimate dedup. That is why it has had no consequence yet,
and why nothing detects it.

The second half: a claimant hidden behind an unreadable record

#650 (GC), #651 (restore) and #695/#696/#697 all move the custodian loops from "fail closed on the
first record I cannot read" to "contain it per object, name it, and keep working". That is the right
trade for the loops that build sets. For reconstruction — the one loop that writes fragment
bytes
— it opens a question the fail-closed base never had to answer:

While the pass's reading is incomplete, a record it could not read may also reference chunk C.
May the pass rebuild C's fragment in place?

The #697 line answered it twice and neither answer survived review:

  • Round 3–4 added a may_land probe consulted on every claimed slot. It had no exit
    condition inside the window it was armed in
    : GC withholds every fragment while any object is
    unresolvable (crates/custodian/src/gc.rs:306-316), and operator surface: no supported way to repair or remove an unreadable committed record #694 says there is no operator tooling to
    repair or remove an unreadable record — so a completable repair the base would have finished could
    stall indefinitely while silently falling off the operator-visible backlog gauge.
  • Round 5 narrowed the probe to fire only while the reading is incomplete
    (nothing_stands_at). The adversarial reviewer reproduced the stall anyway inside exactly that
    window, and the round's Aborted outcome then fired after bytes had already landed —
    contradicting its own documented meaning (reconstruction.rs:239-243).

The counter-argument is on the record too, and it is not weak: a slot in missing is one the
assessment just proved holds no intact shard for this chunk id, index and committed EC tuple
(repair::intact_shard), so for those bytes to be a hidden duplicate's valid shard the hidden
object would have to share the chunk id AND the fragment index AND the server AND carry a different
EcScheme. Probing anyway means refusing to rebuild in place — stalling the common repair
(scrub finds a checksum failure, reconstruction rebuilds the shard where the record already places
it) for as long as any object in the store is unreadable.

Both sides are preserved verbatim in wyrd-pdca results/issue_697/review-rejected.md
(the reconstruction.rs:689 entry) and results/issue_697/iteration-v4/check-advisory-adversary.md.

What this issue has to settle (not prescriptive)

  1. What a duplicate committed ChunkId means, and what the loop owes it. Neither reference
    repaired, obligation kept, both objects named, no certification? Or first-wins preserved with a
    detection signal only? The narrow rule and the reporting shape are one decision, not two — round 1
    and round 5 each landed a finding on the accounting (a refusal reported once per claim rather
    than once per object, losing the original claimant from every diagnostic after the second).
  2. Whether a write may proceed under an incomplete reading, and if it may not, what ends the
    wait.
    A guard with no exit is a worse failure than the one it prevents; that is what rounds 4
    and 5 measured. Any guard proposed here needs its exit condition designed first, alongside
    operator surface: no supported way to repair or remove an unreadable committed record #694 (the operator surface that makes an unreadable record repairable at all).
  3. Where the answer lives. Reconstruction is the only loop that writes fragment bytes, but
    core,custodian: repoint_chunk — ceiling-safe placement moves in a segmented record (635.4c) #682 adds the segmented write path and inherits the same question.

Out of scope for #697

#697 (re-planned 2026-08-07) preserves find_chunk's first-match-in-key-order behaviour
unchanged and adds no claimant tracking, no ambiguity verdict and no landing guard. A review
finding on either half against that slice is to be pointed at this issue, not fixed in-PR
(AGENTS.md:204-205).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions