Skip to content

Commit bf64ddd

Browse files
eduralpheduralph
committed
site: publish from wyrd@100d72721cdf21d265b5c25ee88ff98fce012c2d getwyrd/wyrd@100d727 
1 parent 4b39f5a commit bf64ddd

1 file changed

Lines changed: 205 additions & 0 deletions

File tree

‎reviews/m4-fdb-go-no-go.html‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="utf-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1">
6+
<title>FDB go/no-go: the M4 metadata fault + contention battery (#442) — Wyrd</title>
7+
<meta name="description" content="FoundationDB was chosen partly for its simulation pedigree. But that pedigree">
8+
<meta property="og:title" content="FDB go/no-go: the M4 metadata fault + contention battery (#442) — Wyrd">
9+
<meta property="og:description" content="FoundationDB was chosen partly for its simulation pedigree. But that pedigree">
10+
<meta property="og:type" content="article">
11+
<link rel="preconnect" href="https://fonts.googleapis.com">
12+
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
13+
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Spectral:wght@300;400;500;600&family=IBM+Plex+Sans:wght@400;500;600&family=IBM+Plex+Mono:wght@400;500&display=swap">
14+
<link rel="stylesheet" href="/assets/style.css">
15+
<link rel="icon" href="/favicon.ico" sizes="any">
16+
<link rel="icon" href="/favicon.svg" type="image/svg+xml">
17+
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
18+
<meta name="theme-color" content="#211e1a">
19+
</head>
20+
<body>
21+
22+
<div class="wrap">
23+
<header class="topbar">
24+
<div class="brand"><a href="/"><svg class="mark" viewBox="0 0 48 48" aria-hidden="true"><polygon class="lz-outer" points="24,3 45,24 24,45 3,24"/><polygon class="lz-inner" points="24,14 34,24 24,34 14,24"/></svg>Wyrd</a></div>
25+
<nav class="topnav" aria-label="primary">
26+
<a href="/governance.html">governance</a>
27+
<a href="/architecture/">architecture</a>
28+
<a href="/adr/">decisions</a>
29+
<a href="https://github.com/getwyrd/wyrd">github</a>
30+
</nav>
31+
</header>
32+
</div>
33+
34+
<!-- DOC_HEADER: doc-class + optional status/meta line. render_site.py emits the
35+
whole block, or nothing when there is neither a class label nor meta. -->
36+
<div class="wrap">
37+
<header class="doc-header">
38+
<p class="doc-meta">12.07.2026</p>
39+
</header>
40+
</div>
41+
42+
<div class="wrap">
43+
<main>
44+
<!-- CONTENT: rendered Markdown body (includes its own <h1>). -->
45+
<article class="prose">
46+
<h1>FDB go/no-go: the M4 metadata fault + contention battery (#442)</h1>
47+
<blockquote>
48+
<p>The verdict artifact #442 asks for. It records what was run, what was found, and what it
49+
licenses. <strong>Verdict: GO on the FoundationDB driver's correctness under fault and
50+
contention</strong> — with two things it does <em>not</em> license, named explicitly below, because a
51+
gate that quietly overstates itself is worse than no gate.</p>
52+
</blockquote>
53+
<h2>Why this battery exists</h2>
54+
<p>FoundationDB was chosen (ADR-0042) partly <em>for</em> its simulation pedigree. But that pedigree
55+
validates <strong>their</strong> code. It says nothing about <strong>our mapping layer</strong> — precondition →
56+
read-conflict set, <code>1020 not_committed</code> → <code>Conflict</code>, the unknown-result rules, the retry
57+
policy — which is entirely ours and entirely new. This battery is a black-box check of that
58+
mapping against a real cluster being genuinely abused.</p>
59+
<h2>What was built, and the one decision that shaped it</h2>
60+
<p>#442 says to reuse #257's harness. <strong>It could not be reused</strong>: #257's Tier-1 scenario was
61+
written against the concrete <code>TikvMetadataStore</code>, and its fault-effect oracle asks <strong>PD</strong> for
62+
a store heartbeat — a concept FoundationDB does not have. Pointing it at FDB was new code, not
63+
a flag.</p>
64+
<p>So the scenario was <strong>lifted into a shared crate</strong> (<code>wyrd-metadata-fault-conformance</code>), generic
65+
over <code>MetadataStore</code>, with the two backend-shaped parts behind one seam (<code>ClusterFault</code>): <em>how
66+
you cut a node</em>, and <em>how you ask its peers whether the cut bit</em>. Everything that decides PASS
67+
or FAIL — the workload, the invariants, the signal arithmetic — is now <strong>the same code for both
68+
backends</strong>. That is the only basis on which their verdicts can be compared; a private FDB notion
69+
of &quot;pass&quot; would decide nothing. It is the discipline of the shared <code>metadata-conformance</code> suite
70+
(ADR-0016), applied to the fault battery.</p>
71+
<p>Runner: <code>WYRD_TIER1=1 cargo xtask fdb-metadata-tier1</code> — repeatable, self-contained (brings up
72+
<code>deploy/fdb-multi-replica</code>, configures <code>double ssd</code>, writes the cluster file, tears down
73+
unconditionally), and <strong>opt-in but never silently skipped</strong>: opted-in-with-Docker-missing is a
74+
hard error, because a battery that quietly did not run would be recorded as a &quot;go&quot; nobody
75+
earned.</p>
76+
<h2>The three legs, and what each found</h2>
77+
<h3>1. Contention — PASS</h3>
78+
<p>Three workloads, N=4 clients each, against the live 3-process cluster.</p>
79+
<table>
80+
<thead>
81+
<tr>
82+
<th>Workload</th>
83+
<th>The invariant</th>
84+
<th>Result</th>
85+
</tr>
86+
</thead>
87+
<tbody>
88+
<tr>
89+
<td>Rename races</td>
90+
<td>Exactly one of N concurrent renames of the same binding may win; losers are <code>Conflict</code>; the binding ends up in exactly one place — never duplicated, never lost</td>
91+
<td>pass</td>
92+
</tr>
93+
<tr>
94+
<td>Inode-allocator hot path</td>
95+
<td>N clients × 8 allocations through a CAS loop: <strong>every id unique</strong>, and the cell advances by exactly the number handed out</td>
96+
<td>pass</td>
97+
</tr>
98+
<tr>
99+
<td>Blind-batch storm</td>
100+
<td>A precondition-free batch is <strong>never</strong> <code>Conflict</code> (it has nothing to lose); no write vanishes while its caller was told it committed</td>
101+
<td>pass</td>
102+
</tr>
103+
</tbody>
104+
</table>
105+
<p>The allocator leg is the one with the sharpest teeth: a misclassified <code>Conflict</code> there — a lost
106+
race reported as success — hands <strong>two files the same inode</strong>, the worst outcome the metadata
107+
layer can produce. It did not happen.</p>
108+
<h3>2. Consistency under a symmetric cluster fault — PASS</h3>
109+
<p>The shared Tier-1 scenario, with the process holding FDB's <strong><code>master</code> role</strong> symmetrically
110+
isolated (bidirectional <code>iptables</code> DROP inside its own network namespace) while ≥2 contenders
111+
race the same compare-and-swap.</p>
112+
<p>Cutting an arbitrary node would have been <strong>outcome-neutral</strong> — FDB keeps quorum on the majority
113+
side, so a bystander cut proves nothing and every assertion goes green for free. That is the
114+
hollow flip #257's review caught in the TiKV leg. The target is therefore resolved at runtime
115+
from <code>status json</code> (the <code>master</code>), and the fault is confirmed <strong>from the peers' side</strong> (a
116+
<em>survivor's</em> <code>coordinators[].reachable</code> view of the cut node) — never by probing the dropped port
117+
ourselves, which would only prove our own packets are dropped.</p>
118+
<p>All four independent signals held: <code>read_after_commit</code>, <code>converged_once</code> (the contended CAS round
119+
advanced the version by exactly one), <code>fault_materialized</code> (the cluster provably lost the node),
120+
<code>no_lost_update</code> (exactly one contended winner; the stale-CAS probe rejected).</p>
121+
<h3>3. Mid-commit kill — PASS, and it induced real 1021s</h3>
122+
<p><code>SIGKILL</code> on the process holding the <strong><code>commit_proxy</code></strong> role — the process a commit RPC is
123+
actually sent to — fired from a background thread <strong>while commits were in flight</strong>.</p>
124+
<p><strong>The first draft of this leg was a hollow green, and its own honesty check said so</strong>: it killed
125+
the <em>master</em> at a round boundary and reported <em>&quot;the kill perturbed no commit — 40 committed, 0
126+
unknown results&quot;</em>. Killing between commits interrupts nothing, and the master is not the process
127+
holding the RPC. Both corrections were load-bearing.</p>
128+
<p>The corrected leg, over 400 rounds of a single-writer CAS chain:</p>
129+
<pre><code>398 committed, 1 settled by re-read, 2 unknown-result commit(s), 0 plain fault(s),
130+
0 phantom conflict(s); final version 399
131+
</code></pre>
132+
<p>Both unknown results were <strong>real <code>1021 commit_unknown_result</code></strong> from the live cluster, surfaced
133+
as the seam's <code>CommitUnknownResult</code> — never as <code>Conflict</code>, never silently retried. And <strong>every
134+
1021 was accounted for</strong> (#442's acceptance criterion): the writer did the one thing the contract
135+
permits — re-read — and settled each one. Of the two, <strong>one had landed and one had not</strong>, which
136+
is precisely why the contract forbids guessing.</p>
137+
<p>A second run, after the leg was hardened (below), reproduced it independently: <code>399 committed, 0 settled by re-read, 1 unknown-result commit(s), 0 phantom conflict(s); final version 399</code> — that
138+
1021's batch had <em>not</em> landed, and the accounting closes on the other side of the same rule.</p>
139+
<p><strong>An unperturbed run now FAILS.</strong> The leg originally <em>printed a note</em> when the kill perturbed no
140+
commit and passed anyway. That was the round-boundary hollow green one level up: honest in the
141+
log, wrong in the exit code — <code>xtask fdb-metadata-tier1</code> would have reported that FoundationDB
142+
&quot;passed the battery&quot; for a run in which the hard path never executed, and a gate that can record
143+
a GO on that is worth nothing. It now asserts <code>unknown_results + faults + settled_by_reread &gt; 0</code>
144+
and fails as <strong>inconclusive</strong> otherwise, telling the operator to re-run (the kill window is
145+
timing-dependent). Caught by a <code>codex review</code> of #535 — a note is not a gate.</p>
146+
<p>Three properties this leg pins that nothing else could:</p>
147+
<ul>
148+
<li><strong>No phantom <code>Conflict</code>.</strong> With a single writer, nothing can legitimately lose a race, so an
149+
<code>Ok(Conflict)</code> would be a fault or an unknown result wearing the wrong hat — telling the caller
150+
<em>nothing was written</em> when something may have been. Zero occurred.</li>
151+
<li><strong>Atomicity across the kill.</strong> Markers present were exactly <code>{0..final_version}</code> — no half-landed
152+
batch.</li>
153+
<li><strong>No double-apply.</strong> 398 + 1 = 399 = the final version. Nothing landed that no caller was told
154+
about, and nothing was applied twice — which is the failure a silently-retried non-idempotent
155+
batch would produce.</li>
156+
</ul>
157+
<h2>What this verdict does NOT license</h2>
158+
<p><strong>1. It is not a TiKV-vs-FDB comparison, because the TiKV leg is currently red — and not because
159+
of this work.</strong> The shared scenario's standing guard (TiKV must stay green in the same change)
160+
could not be discharged: <code>WYRD_TIER1=1 cargo xtask metadata-tier1</code> fails at the rename, with
161+
tikv-client's own 2 s per-RPC timeout surfacing during the forced leader election
162+
(<code>PessimisticLockError { GrpcAPI(Cancelled, &quot;Timeout expired&quot;) }</code>). This was <strong>reproduced on
163+
unmodified <code>main</code>, and again at <code>a4abb69</code> — before any of the #437/#515/#516/#517 work</strong> — so it
164+
is pre-existing and independent of the lift. It wants its own issue. Until it is green, the two
165+
backends have been held to the same <em>code</em>, but only FDB has actually passed it here.</p>
166+
<p><strong>2. It does not flip the production default.</strong> That is a separate, deliberate step: the CLI
167+
default is <code>redb</code> (<code>crates/server/src/cli.rs</code>), not TiKV, and <code>fdb</code> is an off-by-default cargo
168+
feature because it links <code>libfdb_c</code> — which ADR-0042 explicitly relies on. Flipping it is a
169+
build-system change plus a docs/canonical-stack change (<code>deploy/README.md</code>'s &quot;currently
170+
canonical&quot;), not a match arm, and it should be its own reviewable change.</p>
171+
<h2>Verdict</h2>
172+
<p><strong>GO</strong> on the question this battery actually asked: <em>does our FoundationDB mapping layer hold up
173+
under real faults and real contention?</em> It does. The commit classification is correct under a
174+
lost race, under a symmetric isolation of the master, and under a commit proxy dying with the RPC
175+
in flight; the unknown-result class is surfaced, distinguishable, never retried, and never
176+
mistaken for a <code>Conflict</code>; and no batch was ever lost, torn, duplicated, or double-applied.</p>
177+
<p>The two carve-outs above are not hedges — they are the parts a reader would otherwise assume were
178+
covered. The evidence for everything claimed here is reproducible with one command:</p>
179+
<pre><code>WYRD_TIER1=1 cargo xtask fdb-metadata-tier1
180+
</code></pre>
181+
182+
</article>
183+
</main>
184+
</div>
185+
186+
<div class="wrap">
187+
<footer>
188+
<div class="foot-row">
189+
<nav class="foot-links" aria-label="footer">
190+
<a href="https://github.com/getwyrd/wyrd">github</a>
191+
<a href="/architecture/">architecture</a>
192+
<a href="/adr/">decisions</a>
193+
<a href="/references.html">prior art</a>
194+
<a href="/name.html">the name</a>
195+
</nav>
196+
<div class="foot-meta">Apache-2.0 · built in Rust</div>
197+
</div>
198+
</footer>
199+
</div>
200+
201+
<!-- SCRIPTS: render_site.py injects the Mermaid include only on pages that
202+
contain a diagram; empty otherwise. -->
203+
204+
</body>
205+
</html>

0 commit comments

Comments
 (0)