site: publish from wyrd@f23848d8c9e30c44bbc38da56f1f2d2365d9e0d4 getwyrd/wyrd@f23848d

Author: eduralph

architecture/07-deployment-view.html

@@ -239,6 +239,53 @@ <h2>7.5 Communication paths, ports, and protocols</h2>
 <p><strong>Port honesty.</strong> Only <code>50051</code> (the D-server gRPC bind) is fixed in Wyrd's own code today (<code>crates/server/src/cli.rs:32</code>). The metadata, coordination, and telemetry ports — PD <code>2379</code> / TiKV <code>20160</code>, etcd <code>2379</code>/<code>2380</code>, OTLP <code>4317</code>/<code>4318</code> — are the <strong>upstream projects' conventional defaults</strong>, not Wyrd's to assign. The S3, SDK, management, and Prometheus-scrape <em>listen</em> addresses are <strong>operator-configured</strong>; M4 fixes the exact flag names (see the blueprint's <code>[wyrd-config]</code> markers). All internal service-to-service dials are <strong>mTLS under the provider CA with no plaintext fallback</strong> (ADR-0005, ADR-0025); a plaintext internal dial is refused (M2).</p>
 <p><strong>Two planes.</strong> Bulk <strong>fragment</strong> data flows directly client/gateway → D servers (the gRPC chunk path that scales with the fleet); the <strong>metadata commit</strong> is a separate, smaller gRPC path to TiKV. Keeping them distinct is the Colossus-class separation that lets throughput scale with D servers rather than through a metadata bottleneck. On a private-network deployment (e.g. Hetzner vSwitch) only the gateway's S3 port is exposed publicly; every other path stays on the internal network.</p>
 <p><strong>Out of single-zone scope.</strong> Cross-zone replication (L3) uses <strong>NATS JetStream</strong> (client <code>4222</code>) and appears only at the <strong>provider-fleet</strong> profile (M9+); it is not part of a single-zone deployment (ADR-0027).</p>
+<h2>7.6 FoundationDB metadata backend: packaging and version coupling</h2>
+<p>ADR-0042 chose FoundationDB (<code>fdb</code>, behind the off-by-default <code>--features fdb</code>) as the production <code>MetadataStore</code> for the Small multi-node and Provider fleet profiles (§7.1). Unlike the pure-Rust TiKV client, the <code>foundationdb</code> crate binds a <strong>shared C library</strong> (<code>libfdb_c</code>) whose wire protocol is <strong>exactly</strong> coupled to the cluster's — a client built against one FDB version cannot talk to a cluster running another, at all, ever. This section is the packaging contract that follows from that fact: how the client library reaches a Wyrd process, and what happens when it disagrees with the cluster (#441).</p>
+<h3>Container path (primary)</h3>
+<p>The primary distribution is an OCI image carrying <code>wyrd</code> built with <code>--features fdb</code> plus a matching <code>libfdb_c</code>. Building and publishing that image is <strong>#470's</strong> deliverable, not this section's; this section fixes the <em>decision</em> the image implements. The image pins one <code>libfdb_c</code> version at build time — the same version-coupling rule as bare metal (below) applies unchanged; an image is a packaging choice, not an exemption from FDB's protocol coupling.</p>
+<h3>Bare-metal path</h3>
+<p>Outside a container, the host needs FoundationDB's own <code>foundationdb-clients</code> <code>.deb</code>/<code>.rpm</code> installed (it ships <code>libfdb_c.so</code> and, optionally, <code>fdbcli</code>) — a <strong>host prerequisite</strong>, not something Wyrd's own build vendors or statically links (see &quot;The single-binary trade&quot; below). The cluster file (<code>fdb.cluster</code>) is located via <code>WYRD_FDB_CLUSTER_FILE</code> (<code>crates/metadata-fdb/src/lib.rs:386</code>), falling back to FoundationDB's own default <code>/etc/foundationdb/fdb.cluster</code> (<code>:390</code>, resolved by <code>config::cluster_file</code>, <code>:436</code>) — the same default a stock FoundationDB install already writes there, so a host that followed FoundationDB's own install instructions needs no Wyrd-specific configuration to be reached.</p>
+<h3>Version coupling and the fail-closed guard</h3>
+<p>Before #441, a version-mismatched client and a genuinely unreachable cluster produced the <strong>same</strong> symptom: a bounded but anonymous <code>1031 transaction_timed_out</code> (the per-transaction deadline #438 added, <code>crates/metadata-fdb/src/lib.rs:424</code>). An operator who mismatched their client saw exactly the error an operator with a down cluster would see. The defect was <strong>misdiagnosis</strong>, not hanging.</p>
+<p><code>FdbMetadataStore::connect()</code> (<code>crates/metadata-fdb/src/lib.rs:1250</code>) — the constructor <code>open_fdb_meta</code> calls (<code>crates/server/src/cli.rs:175</code>), i.e. every <code>wyrd … --metadata-backend fdb</code> invocation — now performs a bounded readiness probe (<code>preflight</code>, <code>:1291</code>) via <code>Database::get_client_status()</code> before returning <code>Ok</code>. The pure, non-feature-gated <code>wyrd_metadata_fdb::preflight</code> module (<code>:832</code>, sibling to <code>classify</code>/<code>config</code>) classifies the result into <code>Ready</code> / <code>VersionSkew</code> / <code>Unreachable</code>; a non-<code>Ready</code> verdict fails the connect with a message naming the <strong>cluster's</strong> protocol version and pointing at the upgrade procedure below — instead of the anonymous timeout.</p>
+<p>The discriminator is <code>Compatible == false</code> on a connection whose <code>Status</code> is <code>&quot;connected&quot;</code>. It is deliberately <strong>not</strong> &quot;zero reachable coordinators&quot; (under skew the coordinator list stays populated) and not <code>Healthy == false</code> alone (false in both failure cases). Anything the probe cannot positively identify as skew — an unparsable, novel, or late status — degrades to <code>Unreachable</code> <em>with</em> a version-coupling hint, <strong>never</strong> to a guessed <code>VersionSkew</code>: an operator whose cluster is merely down must not be sent hunting for a version mismatch.</p>
+<p>Two shapes this guard deliberately keeps:</p>
+<ul>
+<li><code>connect()</code> and its probe are <strong><code>async</code></strong>, awaited on the caller's runtime — exactly like the TiKV peer <code>open_tikv_meta</code> (<code>crates/server/src/cli.rs:147</code>). All seven <code>open_fdb_meta</code> call sites are already inside a Tokio runtime, so a probe that drove itself on a runtime of its own would panic with <em>&quot;Cannot start a runtime from within a runtime&quot;</em> on every invocation.</li>
+<li><code>FdbMetadataStore::open()</code> (<code>crates/metadata-fdb/src/lib.rs:1326</code>), used only by the cluster-file-gated test harnesses, keeps <strong>no</strong> probe — so tests that point at an unreachable coordinator on purpose (<code>tests/timeout.rs</code>) are unaffected. The probe belongs to <code>connect()</code>, the <em>operator</em> path.</li>
+</ul>
+<h3>The multi-version client upgrade dance</h3>
+<p>FoundationDB's own answer to a lockstep cluster upgrade is the <strong>multi-version client</strong>: a directory of additional <code>libfdb_c</code> versions the client loads via the <code>ExternalClientDirectory</code> network option, so one client process can speak to a cluster mid-upgrade. Wyrd exposes this as <code>WYRD_FDB_EXTERNAL_CLIENT_DIR</code> (<code>crates/metadata-fdb/src/lib.rs:398</code>), consumed by <code>ensure_network()</code> (<code>:1114</code>) when the client network boots; unset, behaviour is byte-identical to a client with no multi-version support.</p>
+<p>The upgrade sequence:</p>
+<ol>
+<li><strong>Add</strong> the cluster's <em>new</em> <code>libfdb_c</code> version to every client's external-client directory (pointed to by <code>WYRD_FDB_EXTERNAL_CLIENT_DIR</code>) — the <em>old</em> library stays in place too, so the client can still speak the version the cluster is currently running.</li>
+<li><strong>Upgrade</strong> the cluster to the new FoundationDB version. During the transition the multi-version client speaks whichever protocol the coordinators currently answer with; a rolling cluster upgrade does not require every client to be updated in the same instant.</li>
+<li><strong>Drop</strong> the old <code>libfdb_c</code> from the external-client directory only once every client in the fleet has the new one and the cluster upgrade is complete — the mirror image of step 1.</li>
+</ol>
+<p>This turns a cluster upgrade into a <strong>configuration/image change</strong> (update the external-client directory, then the cluster) rather than an architecture change — no code in Wyrd's own tree needs to move.</p>
+<h3>Manual repro of the guided error</h3>
+<p>Reproducing the guided error needs a cluster running a FoundationDB version the client was not built against. #470 automates this against the <code>wyrd:fdb</code> image it builds (its acceptance criterion 4); until then, this is the manual procedure — <strong>run and observed</strong> against a <code>libfdb_c</code> 7.3.77 client and a <code>foundationdb/foundationdb:7.1.61</code> cluster:</p>
+<pre><code class="language-sh"># 1. Bring up a cluster running an OLDER FoundationDB than the client wyrd was built
+#    against. --network host, matching deploy/fdb-single-node/docker-compose.yml's own
+#    note: a libfdb_c client on the HOST must reach the address the server ADVERTISES
+#    (127.0.0.1:4500), which a default bridge-network port mapping does not give you.
+docker run -d --name fdb71 --network host \
+  -e FDB_NETWORKING_MODE=host -e FDB_PORT=4500 -e FDB_PROCESS_CLASS=unset \
+  -e FDB_CLUSTER_FILE_CONTENTS=&quot;docker:docker@127.0.0.1:4500&quot; \
+  foundationdb/foundationdb:7.1.61
+docker exec fdb71 fdbcli --exec &quot;configure new single memory&quot;
+
+# 2. Point a cluster file at it and run a `wyrd` built with --features fdb:
+printf 'docker:docker@127.0.0.1:4500\n' &gt; /tmp/skew.cluster
+echo hello &gt; /tmp/payload
+WYRD_FDB_CLUSTER_FILE=/tmp/skew.cluster \
+  wyrd put /tmp/payload --key smoke --metadata-backend fdb --data-dir /tmp/skew-data
+</code></pre>
+<p>Observed: the connect fails in <strong>~200 ms</strong> with <strong>exit status 1</strong> (not a panic, not a hang) and a message containing <code>client/cluster protocol version mismatch</code>, the cluster's reported protocol version <code>fdb00b071010000</code>, and a pointer to the multi-version upgrade procedure above — instead of the anonymous <code>1031 transaction_timed_out</code> this produced before #441.</p>
+<p>For contrast, a cluster file naming an unreachable coordinator (<code>x:x@192.0.2.1:4500</code>, RFC 5737 TEST-NET-1) reports <code>cluster unreachable … reported as unreachable rather than a guessed version skew</code> and never claims a mismatch. Tear the skew cluster down with <code>docker rm -f fdb71</code>.</p>
+<h3>The single-binary trade</h3>
+<p>The <strong>single binary (dev)</strong> profile (§7.1) stays a true static binary with <strong>zero new demands</strong> from this section: <code>crates/metadata-fdb/Cargo.toml</code>'s <code>default = []</code> means the default build never links <code>libfdb_c</code> — indeed a machine without it could not compile this crate's real driver at all if the feature were on by default. <code>fdb</code> is only reachable via the explicit, off-by-default <code>--features fdb</code> build — a <strong>production</strong>-tier build, per ADR-0014 (<code>docs/design/adr/0014-single-binary-dev-only.md:18</code>: the single-binary profile is <em>&quot;for development and evaluation only&quot;</em>, explicitly not a supported production tier).</p>
+<p>FoundationDB itself does not support static-linking <code>libfdb_c</code>, and doing so would defeat the multi-version client above — the only sanctioned upgrade mechanism — so an <code>fdb</code>-backed Wyrd is <strong>never</strong> a single static binary; it always carries a shared-library dependency, container or bare metal. This is a property of FoundationDB's own distribution model, not a Wyrd shortcoming, and no ADR change follows from stating it: ADR-0014 already scopes single-binary to dev/eval, and <code>fdb</code> was never that profile.</p>
 
     </article>
   </main>