Every significant design choice is recorded here: the alternatives, the reasoning, and the evidence. Entries are appended as decisions are actually made, never invented ahead of the work. Copy the template below for each new entry and number entries sequentially.
## D-NNN: <short title>
- Decision:
- Date:
- Status: proposed | accepted | superseded by D-MMM
- Context:
- Alternatives considered:
- Choice:
- Reasoning:
- Consequences:
- Evidence or benchmark:
- Revisit conditions:
- Decision: read every multi-byte ITCH integer with std::memcpy into a local of the exact width, then byte swap on little-endian hosts.
- Date: 2026-07-22
- Status: accepted
- Context: ITCH fields are big-endian and frequently unaligned inside the mapped file; reads happen on the hot path.
- Alternatives considered: (1) packed structs overlaid on the buffer, (2) reinterpret_cast of unaligned integer pointers, (3) per-byte shift-and-or assembly without memcpy.
- Choice: memcpy plus swap, with std::byteswap behind its feature-test macro and a portable constexpr fallback under the C++20 baseline. The 48-bit reader composes a u16 and a u32 load so it never touches a seventh byte.
- Reasoning: packed structs and casted unaligned loads are undefined or implementation-defined behavior on strict-alignment targets, and modern compilers fold memcpy-plus-swap into one plain load and a bswap, so the well-defined form costs nothing. Per-byte assembly is also well defined but obscures the intent and optimizes less uniformly.
- Consequences: all field access goes through the readers in itch.hpp; bounds checking stays the caller's responsibility and must precede every read.
- Evidence or benchmark: tests/endian_test.cpp covers zero, maximum, asymmetric patterns, unaligned addresses, 48-bit assembly, and width-overrun poisoning, green under ASan/UBSan; codegen folding is the documented compiler idiom and will be visible in Day 5 profiles.
- Revisit conditions: if a Day 5 profile ever shows reader overhead, inspect codegen before changing the approach.
- Decision: a move-only RAII wrapper that opens read-only, maps the whole file with PROT_READ + MAP_PRIVATE, closes the descriptor immediately, and unmaps exactly once in the destructor.
- Date: 2026-07-22
- Status: accepted
- Context: the parser walks one large read-only daily file; Day 0 needs the mapping foundation without committing to parser design.
- Alternatives considered: (1) keeping the descriptor open for the mapping lifetime, (2) buffered stream reads instead of mapping, (3) error codes or status objects instead of exceptions, (4) automatic madvise inside open().
- Choice: descriptor closed after mmap since POSIX mappings outlive their descriptor; empty files are valid and yield an empty span because zero-length mmap is EINVAL by specification; non-regular files are rejected; failures throw std::system_error carrying errno and the failing operation; moved-from objects reset to empty so destruction is safe and cleanup cannot run twice; the destructor never throws and ignores munmap failure, which has no recovery; sequential-access advice is a separate explicit advise_sequential() method whose failure is ignored as purely advisory.
- Reasoning: opening happens once per run, far from the hot path, so exceptions are the simplest correct error channel; keeping the fd adds a second resource with no benefit; stream reads would add copies the zero-copy parser design forbids.
- Consequences: platform code stays isolated in parser.hpp and is POSIX-only by design; Windows fails at compile time with a clear error.
- Evidence or benchmark: tests/mapped_file_test.cpp covers contents, size, binary data, empty file, missing file, directory rejection, both move operations, moved-from destruction, and advisory safety, green under ASan/UBSan.
- Revisit conditions: if Day 5 measurements show page-fault stalls, evaluate MAP_POPULATE (Linux) or explicit readahead in the benchmark harness.
- Decision: derive the frame format and every known message size exclusively from official Nasdaq specifications, downloaded from nasdaqtrader.com and checksummed, never from memory or third-party parsers.
- Date: 2026-07-22
- Status: accepted
- Context: Day 1 needs the daily-file framing rule and a validation table of known message lengths; product_spec.md section 1 makes the official PDF the protocol authority.
- Alternatives considered: (1) transcribing sizes from other ITCH parser repositories, (2) trusting the expected values embedded in product_spec.md section 2 without verification.
- Choice: two official documents. Message layouts: "Nasdaq TotalView-ITCH 5.0" specification PDF, 36 pages, latest revision April 28, 2023 (Appendix A revision control log), fetched from https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHspecification.pdf on 2026-07-22, SHA-256 45e0531d1b4b3beb886e9618b2ab824a5aa9bda3a99c0dff03509306e68aacc3. Message tables in sections 1.1 through 1.8 (pages 4 to 20); each total length is the last field's offset plus its length. Daily-file framing: "MoldUDP64 Protocol Specification" V 1.00, last revised 2024-08-02 (Version Control table), fetched from https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf on 2026-07-22, SHA-256 96cdd02b8728a441cb970d1371a96e73c01e1edaffb2891d43667e2a0ad8add5. Its "Downstream Packet: Message Block" section (pages 3 to 4) defines the 2-byte big-endian message length that excludes its own two bytes, followed by exactly that many message-data bytes.
- Reasoning: the size table is correctness metadata for a 100M-message replay; a single transcription error would silently misclassify millions of frames. Official documents with recorded hashes make the table auditable.
- Consequences: the 23 known types and sizes are S 12, R 39, H 25, Y 20, L 26, V 35, W 12, K 28, J 35, h 21, A 36, F 40, E 31, C 36, X 23, D 19, U 35, P 44, Q 40, B 19, I 50, N 20, O 48. The PDFs are not committed (Nasdaq copyrighted documents); URLs, access dates, and checksums are recorded here instead.
- Evidence or benchmark: no discrepancy with product_spec.md section 2: all seven expected book-affecting sizes (A 36, F 40, E 31, C 36, X 23, D 19, U 35) match the official tables. Note: type O (Direct Listing with Capital Raise) was added in the April 28, 2023 revision and postdates the pinned July 30, 2019 sample day, so zero O counts are expected in that file; the table entry is harmless validation metadata.
- Revisit conditions: any future Nasdaq revision of the ITCH 5.0 document; re-download, re-checksum, and re-verify the table.
- Decision: iterate frames with a bounds-checked pointer walk across the whole mapping, delivering non-owning FrameView values to a templated handler.
- Date: 2026-07-22
- Status: accepted
- Context: the frame loop is the innermost loop of every future command; product_spec.md section 3 requires zero heap allocations and inlineable dispatch on the hot path.
- Alternatives considered: (1) buffered read() streaming into a rolling window, (2) an iterator class modeling ranges, (3) std::function or virtual handler dispatch, (4) coroutines.
- Choice: a single walk_frames template over std::span with explicit bounds checks before the prefix read and before exposing each payload; the handler is a template parameter so calls inline; frames are spans into the mapping, so nothing is copied or allocated.
- Reasoning: the mapping already presents the file as contiguous memory, so streaming adds copies and window management for no benefit; std::function and virtual dispatch defeat inlining; coroutines add frame state the loop does not need.
- Consequences: FrameView values are valid only while the mapping lives; Day 2 dispatch will build on the same loop.
- Evidence or benchmark: tests/frame_test.cpp covers every boundary condition with hand-computed expectations; all green under ASan/UBSan.
- Revisit conditions: only if Day 5 profiling shows the walk itself, not the book, dominating unexpectedly.
- Decision: the known-size lookup and the scan histogram are both fixed std::array<_, 256> tables indexed directly by the type byte.
- Date: 2026-07-22
- Status: accepted
- Context: the frame loop must classify and count every message with no allocation and no branching beyond a table load.
- Alternatives considered: (1) std::unordered_map or std::map keyed by type, (2) a switch statement over known types, (3) counting only known types.
- Choice: constexpr std::array<std::uint8_t, 256> for sizes (zero means unknown) and std::array<std::uint64_t, 256> for counts, covering all 256 possible type bytes so unknown bytes are counted rather than lost.
- Reasoning: direct indexing is one load with no hashing, no pointer chase, and no allocation; 256 entries cost 256 bytes and 2 KiB respectively, which is cache-trivial; counting all bytes preserves evidence about unexpected input.
- Consequences: unknown or future message types appear in scan output as raw byte counts instead of disappearing.
- Evidence or benchmark: table verified entry by entry against the official sizes in tests; histogram exactness tested with mixed known and unknown frames.
- Revisit conditions: none foreseeable for v1.
- Decision: scanning stops at the first structural failure (truncated prefix, truncated body, or zero-length frame) and records its offset, declared length, and remaining bytes; unknown complete types and known-type size mismatches are counted and skipped by declared length; the CLI exits 0 on success, 2 on usage error, 3 on open or mapping failure, and 4 on malformed or truncated input, while size mismatches alone remain a degraded success (exit 0).
- Date: 2026-07-22
- Status: accepted
- Context: product_spec.md section 2 requires graceful handling of unknown and truncated frames: log once, stop cleanly, never crash.
- Alternatives considered: (1) resynchronization scans after a bad frame, (2) treating size mismatches as fatal, (3) treating a zero-length frame as skippable.
- Choice: after a structural failure the length prefix can no longer be trusted, so any resynchronization is guesswork; a zero-length frame cannot carry an ITCH message (every official type is at least 12 bytes) and skipping it would loop on the same offset, so the walk halts without touching a type byte. MoldUDP64 permits zero-length message data at the transport level, but in a daily replay file it cannot be valid ITCH. Size mismatches are observational on Day 1 because nothing is decoded and the declared length still frames the stream correctly; they are reported in the counters and the exit stays 0.
- Reasoning: the scan's job is honest observation with exact evidence (offset, declared, remaining), not repair; a nonzero exit for mismatches would make a future spec revision look like an I/O failure to scripts.
- Consequences: only the first failure is printed in normal CLI output; stderr is never flooded. Day 2 decoding will revisit whether mismatched known frames should be decoded (they will not be).
- Evidence or benchmark: every branch of the policy has a dedicated test, including the exit-code contract driven end to end against the real binary.
- Revisit conditions: if a real daily file ever shows nonzero mismatch counts, investigate against the current official spec revision before changing any policy.
- Decision: sequential-access advice failure is ignored, and scan timing is a single wall-clock measurement around the whole walk, labeled preliminary.
- Date: 2026-07-22
- Status: accepted
- Context: scan prints elapsed time and messages per second on Day 1; the rigorous benchmark protocol is defined by product_spec.md section 7 and arrives on Day 5.
- Alternatives considered: (1) failing the scan when posix_madvise errors, (2) per-message timers on Day 1.
- Choice: advice is a hint by definition and never affects correctness, so its failure is silently ignored; timing wraps only the complete scan (steady_clock, one start and one stop) with no per-message instrumentation.
- Reasoning: per-message timers belong to the Day 5 latency mode with its own methodology; mixing them into scan would contaminate the framing ceiling measurement.
- Consequences: scan numbers are explicitly preliminary, single-pass, and machine-labeled; they are never published as project benchmarks.
- Evidence or benchmark: none needed on Day 1 by design.
- Revisit conditions: Day 5 replaces this with the warmup-plus-3-passes median protocol.
- Decision: the single Nasdaq sample day for development, validation, benchmarking, and findings is July 30, 2019: 07302019.NASDAQ_ITCH50.gz with its official 07302019.NASDAQ_ITCH50.gz.md5sum companion file.
- Date: 2026-07-22
- Status: accepted
- Context: v1 results must be reproducible against one immutable input; changing sample days between phases would make numbers incomparable.
- Alternatives considered: (1) newest available file, (2) different days per phase.
- Choice: one fixed historical day from Nasdaq's public sample server (https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/), stored locally under ~/nanobook-data/ and never committed.
- Reasoning: a pinned input plus recorded checksums makes every future number traceable to identical bytes.
- Consequences: scripts/fetch_data.sh documents this exact file; compressed and decompressed checksums are recorded in the Day 1 report once the file exists locally. The file predates the April 2023 type O message, so O counts of zero are expected.
- Evidence or benchmark: checksums to be recorded after the manual download; not yet downloaded as of 2026-07-22.
- Revisit conditions: only if Nasdaq removes the file from the public server.
- Decision: transcribe every field offset, width, and text convention for the nine decoded types directly from the official Nasdaq TotalView-ITCH 5.0 specification message tables, and encode that transcription as the named constant offset table in include/nanobook/itch.hpp.
- Date: 2026-07-23
- Status: accepted
- Context: Day 2 decodes S, R, A, F, E, C, X, D, U; a single wrong offset corrupts every downstream layer silently.
- Alternatives considered: (1) reusing struct layouts from other ITCH parsers, (2) trusting product_spec.md section 2 without verification.
- Choice: the same official PDF recorded in D-003 (revision of April 28, 2023, SHA-256 45e0531d1b4b3beb886e9618b2ab824a5aa9bda3a99c0dff03509306e68aacc3, re-verified before implementation), sections 1.1 (S), 1.2.1 (R), 1.3.1 (A), 1.3.2 (F), 1.4.1 (E), 1.4.2 (C), 1.4.3 (X), 1.4.4 (D), 1.4.5 (U). All messages share the header type 0/1, stock locate 1/2, tracking number 3/2, timestamp 5/6. Verified sizes: S 12, R 39, A 36, F 40, E 31, C 36, X 23, D 19, U 35. Body offsets: S event code 11. R stock 11/8, market category 19, financial status 20, round lot size 21/4, round lots only 25, issue classification 26, issue subtype 27/2, authenticity 29, short sale threshold 30, IPO flag 31, LULD tier 32, ETP flag 33, ETP leverage 34/4, inverse indicator 38. A order reference 11/8, buy/sell 19, shares 20/4, stock 24/8, price 32/4; F adds attribution 36/4. E order reference 11/8, executed shares 19/4, match number 23/8; C adds printable 31 and execution price 32/4. X order reference 11/8, canceled shares 19/4. D order reference 11/8. U original reference 11/8, new reference 19/8, shares 27/4, price 31/4.
- Reasoning: official tables with recorded provenance keep the codec auditable; third-party layouts would launder unverified claims into the project.
- Consequences: decoder and encoder share the same offset constants, so layout drift between them is structurally impossible.
- Evidence or benchmark: no ambiguity or discrepancy was found; sizes match the D-003 table and product_spec.md expected values exactly. Golden tests independently re-transcribe the layouts; the real-file decode check verified all 276,798,644 supported messages byte-exactly.
- Revisit conditions: any official revision of the specification.
- Decision: decode into plain typed structs (no packing, no overlays); preserve fixed-width text byte for byte including space padding; keep the 48-bit timestamp in the low bits of a std::uint64_t; keep prices as raw fixed-point uint32 ticks; validate structure (type, exact length) but not field semantics.
- Date: 2026-07-23
- Status: accepted
- Context: decoded messages feed the Day 3 book; representation choices set the hot-path vocabulary for the rest of v1.
- Alternatives considered: (1) packed structs overlaid on the buffer via reinterpret_cast, (2) trimming or normalizing text at decode time, (3) floating-point prices, (4) rejecting messages whose code fields (buy/sell, printable, event code) hold unexpected values.
- Choice: field-by-field decoding through the memcpy readers into plain structs with defaulted memberwise equality; raw bytes preserved for all code and text fields; no semantic validation in the decoder.
- Reasoning: overlays are undefined behavior on unaligned data and bake host endianness into the layout; trimming loses protocol bytes and breaks byte-exact round trips; floats corrupt prices (product_spec.md section 2 forbids them); semantic checks belong to the layer that acts on values (the Day 3 book decides what a bad side byte means), while the decoder's job is exact structural transport. Raw bytes stay available for diagnostics.
- Consequences: any payload of the right type and length decodes successfully, including semantically absurd ones; byte-exact re-encoding is total. Display helpers that trim text can exist later, outside the hot path.
- Evidence or benchmark: golden tests assert space padding survives; round-trip tests prove re-encoding is byte-identical even for a payload relabeled between same-size types.
- Revisit conditions: if Day 3 book application needs decoder-level rejection, add an explicit validation layer above the decoder rather than inside it.
- Decision: decode_message switches on the type byte, verifies the exact official length before reading any field, and delivers typed messages to a templated handler; P, Q, H, and every other non-Day 2 type returns unsupported_known untouched; unknown bytes return unknown_type; supported types with wrong lengths return bad_length without reading fields.
- Date: 2026-07-23
- Status: accepted
- Context: the parser must dispatch millions of messages per second on Day 3+ without knowing anything about books.
- Alternatives considered: (1) virtual handler interfaces, (2) std::variant return values, (3) std::function callbacks, (4) decoding all 23 types now.
- Choice: template-parameter handler with on_message overloads, mirroring the Day 1 walker design; deliberate four-state DecodeStatus; Day 2 scope held to the nine types the book and stats layers actually consume.
- Reasoning: templates inline where virtuals and std::function cannot; variant forces a second dispatch at every consumer; decoding types nothing consumes yet would be untested surface area.
- Consequences: unsupported types are counted and skipped safely today and can be promoted one by one when a consumer exists.
- Evidence or benchmark: decoder status contract tests cover every branch; the real-file check exercised ok (276,798,644 frames) and unsupported_known (5,431,040 frames), while unknown_type and bad_length are exercised by the unit tests.
- Revisit conditions: Day 3 (book dispatch) and Day 6 (P/Q trade stats) extend the supported set.
- Decision: encode() produces an EncodedMessage holding a fixed 42-byte-capacity buffer (2-byte prefix plus the largest supported payload, F at 40 bytes) and returns std::nullopt for timestamps that exceed 48 bits; all integers are written big-endian through writers that mirror the readers.
- Date: 2026-07-23
- Status: accepted
- Context: the encoder serves itch_synth fixtures, round-trip tests, and the per-message re-encode inside decode-check, which runs 276M times over the real file.
- Alternatives considered: (1) heap-backed byte vectors per message, (2) silently truncating oversized timestamps to 48 bits, (3) a generic serialization library.
- Choice: stack value type sized by the largest supported message; validation instead of silent masking; no dependencies.
- Reasoning: per-message allocation would dominate decode-check cost; silent truncation would let impossible fixture values masquerade as valid; a serialization framework contradicts the lean-surface rule (product_spec.md section 11).
- Consequences: encoding a future type larger than 40 bytes requires growing the constant, which a static_assert pins to the size table.
- Evidence or benchmark: overflow rejection is tested for all nine encoders; decode-check over the full day allocates nothing per message.
- Revisit conditions: Day 6 stats, if P/Q encoding is added (Q is 40 bytes, I is 50 and would grow the buffer).
- Decision: decoding is proven against hand-constructed golden payloads transcribed from the official tables (never produced by the encoder), and the encoder is proven by byte-exact decode-encode and encode-decode round trips plus framed trips through the Day 1 walker; committed fixtures are generated only by itch_synth from synthetic values and regeneration must be byte-identical.
- Date: 2026-07-23
- Status: accepted
- Context: a codec tested only against itself proves consistency, not correctness; fixtures must be safe to publish.
- Alternatives considered: (1) round-trip tests alone, (2) fixtures built from real-feed excerpts, (3) uncommitted fixtures regenerated in CI only.
- Choice: golden tests break self-consistent-but-wrong codecs (a mirrored offset error in both encoder and decoder survives round trips but not goldens); real bytes may never enter the repository in any form (product_spec.md section 0), so fixtures are synthetic by construction, small, committed for reviewability, and pinned by byte-identity tests.
- Consequences: two independent transcriptions of the official tables (itch.hpp offsets, golden bytes) must agree with each other and with 282M real messages.
- Evidence or benchmark: tests/golden_messages.hpp, tests/decode_test.cpp, tests/roundtrip_test.cpp, tests/synth_test.cpp; CI runs entirely on synthetic bytes.
- Revisit conditions: none foreseeable for v1.
- Decision: real-file decode verification is an explicit option (scan FILE --decode-check) that decodes every supported frame, re-encodes it into a stack buffer, and compares byte for byte; the default scan stays framing-only; the walk continues to EOF after a divergence (recording only the first in detail) and the process exits 5 on any decode error or mismatch; timing is a single wall-clock measurement labeled diagnostic.
- Date: 2026-07-23
- Status: accepted
- Context: Day 2 needs proof over the pinned real day without contaminating the Day 1 framing ceiling or fabricating a benchmark.
- Alternatives considered: (1) folding decoding into the default scan, (2) halting at the first divergence, (3) storing decoded messages for offline comparison, (4) publishing the decode-check rate as a performance number.
- Choice: separate opt-in mode; continue-to-EOF so one run yields a full census of divergences by type while the first failure carries the forensic detail (type, frame index, file offset, differing byte index, never raw bytes); nothing decoded is retained or written because 282M retained messages would be gigabytes serving no purpose; the rate is explicitly labeled not a benchmark and stays out of the README. When a file contains both a divergence and a framing failure, the divergence exit code (5) takes precedence over the truncation exit code (4); both conditions remain visible in the printed output.
- Consequences: the Day 1 preliminary framing measurement remains comparable across days; decode-check output is aggregate-only and safe to publish.
- Evidence or benchmark: library and CLI tests cover classification, divergence recording, exit codes, and default-scan purity; the real-file run is recorded in docs/day2_decode.md.
- Revisit conditions: Day 4 dual-engine validate subsumes much of this role; Day 5 defines the real measurement protocol.
- Decision: A and F create a live order at the back of its price level's FIFO queue (F's MPID is ignored by the book); E, C, and X reduce the referenced order cumulatively and remove it at zero, with partial reductions never changing queue position and C's execution price never moving the resting order; D removes the order outright; U preserves the original's side, retires the original reference, and appends the replacement at the back of its new price queue, losing time priority even at an unchanged price. Crossed or locked orders rest exactly as reported: this engine mirrors Nasdaq's outbound feed and never matches, never removes liquidity, and waits for the feed's own executions and removals.
- Date: 2026-07-24
- Status: accepted
- Context: Day 3 turns byte-exact decoding into book state; these rules come from the official specification's Modify Order section (cumulative effects, order dead at zero shares, replace retires the reference and cannot change side) and product_spec.md section 2.
- Alternatives considered: running a matching algorithm on crossing orders.
- Choice: strict mirroring, because ITCH reports the exchange's own actions; matching locally would double-apply the exchange's work and diverge immediately.
- Consequences: momentarily crossed books are legal states; tests assert they persist untouched.
- Evidence or benchmark: semantic golden tests cover every rule, and the real 50M-frame replay applied 48,452,372 mutations with zero semantic errors, which it could not have done with wrong semantics.
- Revisit conditions: none; these are protocol facts.
- Decision: bids in std::map<price, Level, std::greater<>> and asks in std::map<price, Level, std::less<>> so begin() is always the best level; Level holds a 64-bit total plus a std::list FIFO of order references; live orders index through std::unordered_map storing remaining shares, price, a strongly typed Side, and the list iterator for O(1) mid-queue removal. One shared insertion path serves add and replace; one shared removal path serves full execution, cancel to zero, delete, and replace. Every rejected operation (duplicate add, missing reference, zero or excessive quantity, invalid side byte, replacement reference collision, zero-share add or replace) is validated before any mutation, returns a status instead of throwing, and leaves the book unchanged.
- Date: 2026-07-24
- Status: accepted
- Context: product_spec.md section 3 fixes this reference architecture; Day 3 prioritizes obvious correctness over speed.
- Alternatives considered: sorted vectors (reserved for FastBook, Day 4); boolean returns; permissive coercion of invalid side bytes; duplicated removal logic per operation.
- Choice: as stated. Aggregate level shares are 64-bit because a level sums many 32-bit orders; stored list iterators stay valid across unrelated mutations, which std::list guarantees; the atomicity policy protects tests and synthetic edge cases even though a clean full-day feed never triggers it; the single removal path keeps the subtlest logic (level emptying, neighbor preservation) in one place.
- Consequences: MapBook is deliberately slow (two tree or hash lookups per reduction) and maximally inspectable; it becomes the Day 4 equivalence oracle and benchmark baseline.
- Evidence or benchmark: 27 semantic golden tests run invariants after every operation; a 50,000-operation fixed-seed randomized stream matches an independent reference model exactly; a deliberate-corruption test proves the invariant checker detects breakage.
- Revisit conditions: never optimized; FastBook exists for that.
- Decision: a template Engine holds books in a locate-indexed vector of unique_ptr with capacity reserved for the full 16-bit range; books construct lazily on the first order message for their locate; directory metadata lives in a separate locate-indexed vector and never creates a book; system events count without touching books (locate 0 stays system scope); ticker strings never appear on the order hot path. The book replay runs strict by default: the first semantic error stops the replay with its category, frame index, file offset, message type, and stock locate (never raw feed values) and exits 6. The full cross-book invariant scan runs once at the end of a replay (and after every operation only in small tests), failing with exit 8.
- Date: 2026-07-24
- Status: accepted
- Context: product_spec.md section 3 requires vector-by-locate routing and a registry Day 4 can reuse for FastBook without virtual dispatch.
- Alternatives considered: eager construction of 65,536 books (wasteful: the real day touches under 9,000); unordered_map keyed by locate (a hash per message for no benefit); lenient replay that counts errors and continues (would let a divergence smear across millions of messages before anyone noticed); invariant scans per message on real replays (quadratic-ish cost across millions of live orders).
- Choice: as stated; strictness makes the first real divergence loud and precisely located.
- Consequences: unit tests exercise error paths directly against the book; the real replay treats any semantic error as a stop-the-world defect.
- Evidence or benchmark: engine tests cover routing, isolation, growth, late directory attachment, and strict stopping; the 50M real replay ended with invariants ok across all 8,840 books.
- Revisit conditions: Day 4 revisits nothing here; validate reuses the registry with both engines.
- Decision: TOB output is off by default and never affects plain replay; --symbols and --tob-out are required together; selection controls output only while every symbol is always reconstructed; one CSV per selected symbol opens lazily when the ticker appears in the directory (first locate wins on duplicates); a row emits only when best bid or ask price or size changed, stamped with the causing message's timestamp; prices are written as exact integers with four implied decimals (never floating point) and a missing side leaves empty fields; output is stdio-buffered with open, write, and close failures surfaced as exit 7; symbol arguments are restricted to a safe filename charset (alphanumeric first character, then alphanumerics, '.', '-', '+', at most 8 characters), which makes path separators and traversal sequences unrepresentable.
- Date: 2026-07-24
- Status: accepted
- Context: product_spec.md section 3 specifies buffered, off-by-default TOB emission so benchmarks stay pure.
- Alternatives considered: filtering reconstruction to selected symbols (would corrupt cross-symbol totals and Day 4 comparisons); floating-point price formatting (rounding drift); emitting every message (row explosion).
- Choice: as stated.
- Consequences: the required Day 3 real replay ran without TOB, so no real top-of-book values exist anywhere in the repository; TOB is tested purely against synthetic fixtures in temporary directories.
- Evidence or benchmark: ten TOB emission tests assert exact rows, formatting, selection behavior, and failure paths.
- Revisit conditions: Day 6 stats may generalize the output layer.
- Decision: --messages=N stops after exactly N complete framed messages (not N supported messages), parses as strict unsigned 64-bit rejecting malformed, signed, and overflowing values, allows zero as a valid zero-frame replay, and reports whether the replay ended at the limit or at clean EOF. Replay timing is one wall-clock measurement labeled a diagnostic MapBook replay, never a benchmark; memory is observed with /usr/bin/time -l outside the process, never with per-message probes on the hot path.
- Date: 2026-07-24
- Status: accepted
- Context: Day 3's required proof is the first 50 million frames of the pinned day on an 8 GB machine already under memory pressure; progressive 1M and 10M runs preceded the 50M run to bound memory growth safely.
- Alternatives considered: counting only supported messages toward the limit (would make the limit depend on message mix); rejecting zero (arbitrary); in-process RSS sampling (hot-path contamination).
- Choice: as stated.
- Consequences: frame limits are reproducible byte positions in the file; Day 5 defines the real measurement protocol.
- Evidence or benchmark: CLI tests pin exact-stop, zero, beyond-EOF, and malformed-limit behavior; the 50M run recorded 1.07 GiB peak RSS and zero swaps.
- Revisit conditions: Day 5 replaces diagnostic timing entirely.
- Decision: every FastBook owns its own order pool, level pool, and order index; there is no shared cross-symbol arena and the engine registry is unchanged from Day 3.
- Date: 2026-07-25
- Status: accepted
- Context: product_spec.md section 3 requires Engine to instantiate FastBook as a drop-in book type; roughly 8,840 books exist on the pinned day.
- Alternatives considered: one global order arena shared by every book, with books holding ranges or borrowing allocation from the engine.
- Choice: self-contained books. Chunked pool growth starts small (64 order slots, 16 level slots), so per-book ownership costs a few KiB for barely-traded symbols instead of forcing a shared-arena redesign of ownership and the registry.
- Reasoning: a global arena changes the engine's ownership model for an unmeasured benefit; per-book pools also keep one hot symbol's slots in its own chunks rather than interleaved with 8,839 other books.
- Consequences: total capacity is the sum of per-book peaks, slightly above the global concurrent peak; Day 5 may measure whether a shared arena is worth its complexity.
- Evidence or benchmark: FastBookScaling tests pin the per-book floor (one 64-slot order chunk, one 16-slot level chunk under churn); the 100M dual-engine validation bounds the aggregate in practice.
- Revisit conditions: Day 5 profiling only, and only with measurements.
- Decision: Pool hands out 32-bit PoolIndex slots with kInvalidIndex = 0xFFFFFFFF as the never-valid sentinel; released slots recycle LIFO through a free list threaded through the released slot's first four bytes; storage grows in chunks whose sizes double from the small first chunk to a 65,536-slot cap and never move; a per-slot state byte rejects invalid and double releases in every build configuration; growth that would collide with the sentinel (or the max_slots test seam) throws.
- Date: 2026-07-25
- Status: accepted
- Context: product_spec.md section 3 requires pooled 32-byte order slots addressed by 32-bit index with peak memory scaling with concurrently live orders, never total daily adds (124M adds versus about 3M peak live).
- Alternatives considered: (1) one std::vector arena with reallocation, (2) per-order heap allocation (std::list or unique_ptr per order), (3) pointer-based free lists, (4) debug-only release guards.
- Choice: as stated. Chunked slabs keep slot addresses stable, so growth never copies live slots and never transiently doubles a large arena on the 8 GB machine; indices, not pointers, are the persistent references, so nothing dangles even where a vector would have reallocated. The index-to-chunk mapping is closed-form over the doubling tiers (one bit_width instruction plus one well-predicted branch for the capped region). Release guards stay on in release builds because the state byte is one store per transition; tests must be able to prove double-release rejection in every configuration.
- Reasoning: per-order heap nodes are what makes MapBook slow; a plain vector arena is acceptable per the day plan but pays a reallocation copy spike exactly when a hot book grows; the threaded free list needs no side storage. The growth policy is deterministic and deliberately easy to revisit; no optimality is claimed before Day 5 measurement.
- Consequences: released slot contents are destroyed immediately (the free-list link overwrites the first four bytes, debug builds poison the rest), so every removal path must copy what it needs before releasing; this ordering rule is part of the FastBook removal path design (D-026).
- Evidence or benchmark: pool tests cover tier boundaries (64/128/256..., the capped-region crossing with a 4-slot first chunk), address stability across growth, LIFO reuse, guard rejection, the sentinel ceiling seam, move semantics, a 200,000-operation randomized stream against a reference model, and the free-list invariant audit; all green under ASan/UBSan.
- Revisit conditions: Day 5 profiles chunk sizes and the first-chunk floor; the policy is a two-constant change.
- Decision: the pooled order slot is exactly 32 bytes with 8-byte alignment, asserted at compile time: reference u64, price u32, shares u32, prev u32, next u32, level handle u32, side u8, flags u8 (bit 0: live), reserved u16.
- Date: 2026-07-25
- Status: accepted
- Context: product_spec.md section 3 targets a 32-byte slot and lists price, shares, prev, next, level, and side plus padding; it does not list the order reference.
- Alternatives considered: (1) the spec's literal field list without the reference (24 bytes padded), (2) pointers instead of indices, (3) a separate index-to-reference side table.
- Choice: store the u64 reference first (natural alignment), landing the layout exactly on the 32-byte target. This is a recorded deviation from the spec's literal field list, forced by the spec's own contracts: the queue_at diagnostic, the canonical digest, and the invariant checkers must produce order references while walking intrusive queues, and the reference-to-index hash has no usable inverse (inverting it would be a full-table scan per queue entry). Indices rather than pointers keep every stored link valid across pool growth and make the slot layout fixed-width and position-independent.
- Reasoning: a reference-free slot would make reverse validation quadratic; a side table would spend the same 8 bytes with worse locality.
- Consequences: sizeof(OrderSlot) == 32 and alignof == 8 are static assertions; the flags byte gives the slot an explicit live bit used by the invariant checker.
- Evidence or benchmark: compile-time assertions plus the FastBook invariant and equivalence suites.
- Revisit conditions: none foreseeable for v1.
- Decision: OrderIndex maps u64 order references to u32 pool indices with power-of-two capacity, mask-based bucket selection, the splitmix64 finalizer as the hash, linear probing, an explicit per-slot state byte, tombstone deletion, and a 0.7 occupancy limit counting live entries plus tombstones. Maintenance is unconditional at the limit: the table doubles when live entries exceed half of capacity and otherwise rebuilds at the same capacity, purging tombstones. Insertion probes to the first truly empty slot before declaring a key absent, then places into the first tombstone seen. Duplicate insertion fails without touching the existing mapping; erase of a missing key returns false.
- Date: 2026-07-25
- Status: accepted
- Context: product_spec.md section 3 fixes open addressing, power-of-two capacity, linear probing, load 0.7, and tombstone deletion, and asks for the backward-shift alternative to be documented.
- Alternatives considered: (1) backward-shift deletion: erase compacts the probe chain by shifting displaced entries back, leaving no tombstones, at the cost of a short shift per delete and subtler correctness arguments around wraparound; it keeps effective load equal to live load under the 120M-delete real mix and would eliminate purge rebuilds. It remains the documented alternative for a Day 5 experiment, not the v1 design, because the specification names tombstones and the tombstone policy is simpler to prove correct. (2) Reserved key sentinels instead of state bytes: rejected because every u64 value, including 0 and UINT64_MAX, is a legal order reference. (3) A multiplicative-only hash: rejected because near-sequential real references benefit from the finalizer's avalanche. (4) std::hash: unspecified across platforms.
- Choice: as stated. Two policy details matter and were fixed during design review before implementation: maintenance must trigger on the combined live-plus-tombstone occupancy unconditionally (a policy gated on tombstones being the majority contributor has a reachable hole in which the table fills completely and unsuccessful probes never terminate), and the grow-versus-purge choice must switch on live entries exceeding half of capacity (purging at, say, 0.6 live would clear so little headroom that the next burst of inserts would rebuild again immediately, a rehash storm on the hottest books).
- Reasoning: with the unconditional trigger, occupancy never exceeds 0.7, so a truly empty slot always exists and every probe loop terminates; purging at low live load is what keeps a churning steady-state book from growing its table with total daily adds.
- Consequences: erase-heavy churn purges rather than grows (asserted by tests); a duplicate add sitting beyond a tombstone is still rejected (the probe-to-empty rule; a first-tombstone-stops probe would silently accept it and resurrect ghost orders).
- Evidence or benchmark: forced-collision chain tests through tombstones and across table wraparound, purge-not-grow churn with terminating misses, a 1,000,000-operation randomized cross-check against std::unordered_map with checkpoint sweeps and invariant audits; all green under ASan/UBSan.
- Revisit conditions: Day 5 may measure backward-shift deletion and an earlier maintenance trigger (0.55 to 0.6) against the real mix.
- Decision: FastBook price levels live in their own Pool and are ordered by a per-side sorted vector of packed {price u32, level handle u32} pairs: bids descending, asks ascending, best level first. Orders store the stable level handle; every reduction and removal reaches its level in O(1) through the handle. Binary search over the pairs serves find-or-create on adds and the erase of an emptied level; pair shifts move only the 8-byte pairs, never level records, and no order slot is rewritten because another level moved.
- Date: 2026-07-25
- Status: accepted
- Context: the level-structure triangle product_spec.md section 3 requires recording: std::map (the MapBook baseline), sorted vector (chosen for FastBook), and a flat price-tick array (fastest per symbol, memory-heavy across 8,000+ symbols; a possible post-v1 experiment on the top 10 symbols only, not built on Day 4).
- Alternatives considered: (1) Level structs directly inside the sorted vector with orders storing vector positions: rejected because vector insert/erase shifts positions and every stored position would silently dangle; (2) the same but with a level-id-to-position side table: workable, but it maintains a second mutable mapping for no gain over stable pool handles; (3) no level handle in the order, binary-searching the level on every reduction: correct and simpler, but it pays a search on the E/C/X/D-heavy real mix (about 141M delete-like operations) that the stable handle makes O(1), and the spec's own slot layout includes a level field.
- Choice: as stated; the pair vector is an ordering index over stable records, which is what makes the vector-shift problem disappear by construction.
- Reasoning: level counts per symbol are small (tens to low hundreds) and updates cluster near the touch, so contiguous 8-byte pairs binary-search and shift cheaply; level records stay put for the life of the level, so handles held by orders can never be invalidated by unrelated levels.
- Consequences: emptied levels are erased from the pair vector and their records recycled through the level-pool free list immediately (retained empty levels would scale memory with distinct prices traded per day instead of live state); recycled handles are safe because a record is only freed when no order references it.
- Evidence or benchmark: contract-suite stress cases cover better, worse, and middle level insertion on both sides, first, middle, and last level removal, live orders across hundreds of pair shifts, replaces bouncing between distant prices, and recycling waves; the invariant checker verifies strict per-side ordering, pair-to-record price agreement, and level-pool accounting on every run.
- Revisit conditions: the flat tick-array experiment stays post-v1; Day 5 may measure best-at-front against best-at-back pair ordering (the memmove cost of near-touch churn is proportional to the elements after the insertion point).
- Decision: orders at a level form an intrusive doubly linked FIFO: the level record stores head and tail pool indices, each order slot stores prev and next, kInvalidIndex terminates both ends. Add and replace append at the tail in O(1); partial reductions touch no links; removal relinks neighbors (or head/tail) in O(1).
- Date: 2026-07-25
- Status: accepted
- Context: product_spec.md section 3; MapBook's std::list FIFOs allocate a heap node per order, which is the reference design's deliberate cost.
- Alternatives considered: std::list per level (a node allocation per order), a deque per level (invalidates references on reallocation), storing queue positions (dangles on mid-queue removal).
- Choice: as stated; no queue node exists outside the pool and no allocation happens per order beyond the pooled slot itself.
- Consequences: unlink correctness at head, middle, tail, and sole-order positions is load-bearing; the invariant checker walks every queue verifying bidirectional link consistency, head/tail anchoring, length against the level's order count (which also bounds the walk, so a cycle cannot loop forever), and per-slot side, price, handle, and live-flag agreement.
- Evidence or benchmark: contract tests for front, middle, back, and sole deletions plus the deep-queue chunk-crossing test; the randomized equivalence run compares exact FIFO sequences against MapBook at every checkpoint.
- Revisit conditions: none foreseeable for v1.
- Decision: FastBook validates completely before mutating: add checks quantity then duplicate (via the hash insert itself); reductions check existence then quantity; replace checks the original, then quantity, then the new-reference collision, where inserting the new reference into the hash IS the collision check (the original is still live, so new == original collides on its own entry and a failed insert has mutated nothing). One shared removal path serves full execution, cancel to zero, delete, and replace: copy the slot out, unlink it, fix the level totals and counts, erase the emptied level's pair before releasing its record (the erase needs the level's price), erase the hash entry, and release pool slots last. Rejected operations return the same status values as MapBook in the same precedence and leave the book bit-for-bit unchanged.
- Date: 2026-07-25
- Status: accepted
- Context: D-016 fixes the atomic-rejection contract and precedence; dual-engine validation compares statuses message by message, so even a rejection-order difference with identical end states is a divergence.
- Alternatives considered: validate-as-you-mutate with rollback; rejected because rollback of a partially linked intrusive structure is exactly the kind of subtle code the reference book exists to avoid, and the free-list threading of released slots (D-021) makes stale reads after release a real hazard that a strict ordering rule eliminates.
- Choice: as stated.
- Consequences: the hash pointer returned by insert is invalidated by later inserts, so replace copies the original's pool index out before inserting the new reference; erase never moves hash slots, so the new reference's value pointer stays valid through the original's removal.
- Evidence or benchmark: the typed contract suite pins every rejection and precedence case on both books, including the zero-share duplicate and the missing-then-zero-then-collision replace ladder; the 100,000-operation randomized equivalence stream compares every status; deliberate-corruption tests prove the invariant checker detects broken totals, ordering, hash linkage, leaked and prematurely released slots.
- Revisit conditions: none; these are contract facts.
- Decision: dual-engine state comparison uses a canonical digest of two independently seeded 64-bit accumulators (FNV-1a and an xxhash-prime multiply-xor variant) folded over, per side: a side marker and the level count, then per level best-first the price, total shares, and order count, then per queue entry front-first the order reference, remaining shares, resting price, and side. Digests may stand alone at intermediate checkpoints; the final verdict always comes from an element-by-element exact canonical comparison of every created book, never from hashes alone. Both books expose the identical canonical iteration surface (for_each_level, for_each_order_at), so one generic fold serves both; nothing unstable (addresses, container order, padding) is ever folded.
- Date: 2026-07-25
- Status: accepted
- Context: product_spec.md section 6 specifies a digest of (price, total shares, order count) per level per side; Day 4 design review found two false-negative classes in that digest and strengthened it.
- Alternatives considered: (1) the spec's level-tuple digest alone: rejected because misattributing a partial reduction between two same-level orders (totals preserved) and reordering a FIFO (same set, different order) both digest identically forever; (2) a single 64-bit accumulator: rejected in favor of an independent pair per the day plan; (3) comparing raw object memory: undefined and meaningless across implementations.
- Choice: as stated; the side marker and per-side level count frame the fold so an empty side cannot make mirrored books collide, and per-order folds carry reference, shares, price, and side.
- Reasoning: the digest exists for cheap checkpoint comparison; the exact comparator exists because a claimed equivalence over 100M messages should never rest on hash absence-of-collision, however unlikely.
- Consequences: on any digest mismatch the exact comparator refines the category (level count, level summary, queue length, queue order, order state) with side and ordinal context only, never raw feed values.
- Evidence or benchmark: digest sensitivity tests (side markers, share misattribution, queue order, realignment), comparator category tests, and fault-injection runs proving each category is reported.
- Revisit conditions: none foreseeable for v1.
- Decision: nanobook validate decodes each frame once and fans it to both engines, compares the returned semantic status on every frame, runs a per-frame spot check after every applied order message (the touched order's state and the touched book's best levels must agree across both engines), and marks the locate of every order message (at dispatch, never gated on status) in a preallocated 65,536-entry bitmap. Every K complete frames (default 1,000,000; K must be positive) it compares aggregate engine counters, runs both invariant checkers over every book touched since the previous checkpoint, and compares digests per touched locate, clearing the bitmap only after a fully successful comparison. At EOF or the message limit it compares counters, book-existence sets over the whole locate range, directory metadata, system-event state, every created book exactly, and final totals. Any mismatch stops the replay immediately with aggregate-only context (frame index, file offset, message type, stock locate, checkpoint number, category, ordinals) and exits with the dedicated divergence code 9. A semantic feed error on which both engines agree exits 6, the strict-mode code, because the feed itself is broken there, not the equivalence.
- Date: 2026-07-25
- Status: accepted
- Context: product_spec.md section 6 defines checkpoint cadence in messages and demands the first divergence be loud and located; project policy forbids raw feed values in any real-data output.
- Alternatives considered: comparing only at checkpoints (a status divergence could smear a million frames before detection, and, as the Day 4 adversarial review established, a divergence whose whole lifetime fits inside one checkpoint window would be invisible to digests and even to the final exact pass once the books reconverge; the per-frame spot check closes exactly that hole for the touched order and the touch); marking touched only on applied statuses (an apply-versus-reject disagreement could skip the diverged book); clearing the bitmap before comparison (a failed checkpoint would lose the touched set that the next attempt needs); aggregate-only final comparison (weaker than the exact pass for no meaningful saving).
- Choice: as stated; timing printed by validate is wall-clock only and labeled diagnostic, never a Day 5 benchmark, and memory is observed externally with /usr/bin/time.
- Consequences: checkpoint cost scales with touched books, not total frames; the driver allocates nothing per message and retains no decoded messages.
- Evidence or benchmark: validation machinery tests including checkpoint arithmetic at limits and boundaries, fault-injection detection of every category, and the CLI exit-code suite; the required real-data proof is the 100M-message run recorded in docs/day4_fastbook.md.
- Revisit conditions: Day 5 defines the real measurement protocol; the checkpoint interval stays a CLI knob.
- Decision: record the pre-real-data adversarial review of the complete Day 4 diff (five auditors over distinct dimensions, each finding independently verified) and what was done with each confirmed finding.
- Date: 2026-07-25
- Status: accepted
- Context: the project's operating pattern reviews each day's diff adversarially before real-data proofs; findings are fixed in separate commits or declined with recorded reasoning.
- Findings fixed: (1) Checkpoint blind spot: a divergence whose whole lifetime fits inside one checkpoint window was invisible to digests and to the final exact pass once the books reconverged. Fixed by the per-frame spot check (D-028); the fault-injection suite now proves several defect classes are caught on the exact frame. (2) Raw feed values in invariant diagnostics: both books' level-total failure messages embedded share quantities via std::to_string, and those strings reach stderr on real-data replays. Fixed by making the messages value-free in both books; MapBook's observable semantics are unchanged and its tests still pass unmodified. (3) Misleading summary label: a validation run stopped by a failure printed "replay ended: at clean eof". Fixed with a three-state label ("at the first failure" / "at the frame limit" / "at clean eof"). (4) Conflicting engine selection: "--engine=map --engine=fast" silently ran the fast engine. Fixed: a second --engine option is now a usage error. (5) Undocumented pointer-stability reliance: FastBook::replace writes through the hash insert pointer after an intervening erase, which the OrderIndex comment did not explicitly permit. Fixed by documenting the guarantee (erase and find never move or invalidate slots; insert and reserve may). (6) Allocation-failure consistency: an out-of-memory or index-space exception inside insert_order could leave a hash entry without its slot. Fixed: add and replace unwind the just-inserted hash entry and rethrow; after a replace the surviving state equals a completed delete, which is internally consistent (full rollback of the removal is deliberately not attempted). (7) reserve() wraparound: an unsatisfiable reserve request could loop after size_t overflow. Fixed with an explicit throw.
- Mutation-testing findings fixed (the review's test-quality auditor built and ran candidate mutants against the full suite): the touched and spot-checked handling of E and U frames was pinned only through A messages, so new fault tests make a skipped execute and a wrong-share replace heal before EOF, catchable solely on their own frames; the digest's sell-side queue fold was unpinned, so a sell-side FIFO-order digest test was added; the pool's capped chunk region was crossed only once, so the boundary test now spans three capped chunks and pins exact chunk counts and capacities; the grow-versus-purge boundary (live plus one equal to exactly half of capacity must purge) and the validation failure's file offset are now asserted directly.
- Findings declined, with reasoning: (1) Duplicate-insert rebuild: an OrderIndex insert of an existing key at the load boundary may rebuild the table before the duplicate is discovered. Accepted: avoiding it costs a duplicate pre-probe on every insert to optimize a path real feeds never take; the pointer contract already declares any insert invalidating. Documented in hash.hpp. (2) Book-command duplicate --messages/--symbols laxity: retained last-wins/accumulate behavior for compatibility with Day 3 usage; only the newly dangerous conflicting --engine case was tightened. (3) The 1,000,000-operation hash fuzz never triggers a same-capacity purge (its reserve operations keep live load below half): purge behavior, termination among tombstones, and the boundary policy are pinned by the dedicated churn and boundary tests instead. (4) Exit code 9 and the divergence detail lines are not exercised end to end through the CLI: two correct engines cannot diverge, injecting a faulty book through the binary would require production test hooks, and the outcome-to-exit mapping is a trivial switch; divergence reporting is fully covered at the library layer.
- Evidence or benchmark: all findings and dispositions are pinned by the updated unit, CLI, and fault-injection suites; the real-data progression was re-run on the final binary after the fixes.
- Revisit conditions: none; historical record.
- Decision: every published Day 5 measurement replays the whole pinned sample day from the release build at a clean, pushed commit. Dataset: 07302019.NASDAQ_ITCH50, trading date July 30, 2019, 8,661,679,413 bytes, SHA-256 9f8634a048b8195ccdcbe618e5833a759e35e04486d74e36b79176d35172654a, 282,229,684 complete frames. Build: C++20, -O3, -DNDEBUG, the recorded host-tuning flag (-march=native on this machine), and link-time optimization, with the exact compiler executable and full version captured alongside the result.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 makes the numbers the product, so the input and the binary behind every number must be pinned and auditable.
- Alternatives considered: (1) a fixed prefix of the file (faster iteration, but a partial day is not the claim the README will make); (2) allowing any local build (a stale or modified binary could silently produce the numbers).
- Choice: as stated. scripts/run_bench.sh refuses to produce a result unless the dataset size and checksum match exactly, the working tree is clean, the release build is configured, and no tracked source file is newer than the binary. A dirty tree or battery power requires an explicit flag and marks the result provisional in the JSON itself.
- Consequences: a full-file pass costs real wall time, so the protocol is run deliberately rather than casually; latency scope is recorded per run because sample buffers scale with operation count (D-037).
- Evidence or benchmark: the gates are exercised by the committed baseline and final runs; the dataset facts are re-verified inside every run and recorded in the environment capture.
- Revisit conditions: a Linux x86-64 run would repeat the identical protocol; only the platform fields change.
- Decision: every mode runs one warmup pass that is recorded but excluded from the reported statistic, then three measured passes; the reported headline is the median of the three, and every individual pass is preserved in the committed JSON.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 fixes warmup plus three passes and median reporting.
- Alternatives considered: the mean (sensitive to one slow pass on a machine shared with other applications), the minimum (flattering and not representative), and a single pass (no dispersion evidence at all).
- Choice: median of three, with all passes published so a reader can see the spread rather than trusting the summary.
- Consequences: a slow but valid pass is never discarded (D-040); an invalid pass forces the whole mode to be rerun rather than replaced.
- Evidence or benchmark: bench_test.cpp pins the odd and even median rules and that a slow pass still participates; the JSON carries per-pass records.
- Revisit conditions: none for v1.
- Decision: each pass constructs a new engine and new books; no state, pool capacity, or hash capacity carries over from a previous pass. The process is reused, so allocator and page-cache warmth do carry over, which is recorded rather than pretended away.
- Date: 2026-07-26
- Status: accepted
- Context: reusing engine state would measure a second replay against already-populated books, which is not the workload.
- Alternatives considered: a fresh process per pass (cleaner allocator state, but it would exclude the warmup's page-cache effect that the warmup pass exists to establish, and it would prevent in-process peak-RSS capture from covering the whole run).
- Choice: fresh engine per pass inside one process, with the warmup pass responsible for warming the file mapping.
- Consequences: peak RSS is the high-water mark across the whole process, therefore across all passes of that mode, which the result labels explicitly.
- Evidence or benchmark: a test proves two consecutive passes reach the identical final state, which only holds if the second started clean.
- Revisit conditions: if allocator carryover ever looks material, add a fresh-process mode and compare.
- Decision: the timed region is exactly the frame walk. One steady_clock reading immediately before and one immediately after; inside it there are no per-message timers, no output, no progress reporting, and no invariant scanning. Final counters, totals, and the cross-book invariant scan all run after the clock stops.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 requires throughput to exclude per-message instrumentation.
- Alternatives considered: timing the whole command (would include mapping, option parsing, and reporting) or timing with invariants inside (would measure the auditor, not the engine).
- Choice: as stated, and the framing mode calls the same scan_buffer walk the scan command uses, so the framing ceiling is measured through the shipped code path rather than a benchmark-only copy.
- Consequences: reported throughput excludes process startup and the invariant audit, and the JSON says so.
- Evidence or benchmark: bench.hpp's runners; tests assert that the framing mode reports no book counters and that instrumentation does not change final state.
- Revisit conditions: none for v1.
- Decision: latency covers only book-affecting operations (A and F as add, E and C as execute, X as cancel, D as delete, U as replace), and the timed region contains exactly one book mutation call. The book is resolved from its locate before the clock starts; frame walking, field decoding, dispatch, sample storage, percentile computation, output, and validation are all outside it.
- Date: 2026-07-26
- Status: accepted
- Context: the question Day 5 answers is what one book operation costs, not what one message costs end to end.
- Alternatives considered: timing the whole message path (mixes parser cost into a book number) and timing inside the book implementations (would require touching MapBook and FastBook, which must stay untouched by measurement).
- Choice: an instrument seam on the engine (a defaulted template parameter whose no-op default forwards the call) wraps the mutation. Ordinary replay, validation, and all 324 pre-Day-5 tests are unaffected.
- Consequences: the reported latency includes the two clock reads, which is stated everywhere and quantified by the empty probe (D-038).
- Evidence or benchmark: tests prove the instrumented and uninstrumented engines reach identical final states with identical mutation counts.
- Revisit conditions: none for v1.
- Decision: latency sampling is deterministic, not random: each operation group keeps its own counter and samples every Nth operation of that group, with N = 4 by default and identical for both engines in a given comparison. Samples are 32-bit nanosecond values in per-group buffers preallocated from the warmup pass's exact per-group operation counts, so no measured pass allocates inside its replay loop.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 allows one-in-four sampling under memory pressure; this machine has 8 GB with swap largely consumed by other applications.
- Alternatives considered: (1) full sampling: at roughly 277 million book operations over the full day it would need about 1.1 GB of sample storage per engine even at 4 bytes per sample, which this machine cannot afford alongside MapBook's own footprint; (2) random sampling: not reproducible run to run; (3) one global counter across groups: a rare group (C at 135,573 messages for the day) could be under-represented or missed entirely, so per-group counters are used instead; (4) 64-bit samples: doubles memory for a range no book operation approaches.
- Choice: as stated. The warmup pass counts operations without timing any, so the measured passes' reservations are exact.
- Consequences: reported percentiles describe the sampled quarter of each group, and the sample count is published beside every percentile. The interval is a CLI option so a future machine can measure at one-in-one.
- Evidence or benchmark: sampling tests cover one-in-one, one-in-four, determinism across runs, identical per-group policy, rare-group coverage, and detection of any buffer growth during a measured pass.
- Revisit conditions: a machine with headroom for full sampling, or a Linux reference run.
- Decision: the harness selects a measurement clock at startup and reports which one it chose. On x86-64 it tries a serialized TSC path (rdtscp) only when CPUID reports invariant TSC, then calibrates against steady_clock over five 2 ms busy-wait intervals and accepts the result only if the spread across samples is at most 2 percent; on acceptance it records the measured frequency, the spread, and the sample count. Any failure (no invariant TSC, a non-positive interval, or an out-of-tolerance spread) falls back to steady_clock with the reason recorded. On Apple Silicon there is no x86 TSC at all, so the fallback is structural and the code path is not even compiled.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 prefers rdtsc where trustworthy and requires steady_clock otherwise.
- Alternatives considered: mach_absolute_time on macOS (steady_clock is already implemented on top of it, so there is nothing to gain), and assuming a nominal frequency instead of calibrating (an unverified conversion factor).
- Choice: as stated, with one hard rule: a steady_clock measurement is never labeled a TSC measurement anywhere in the output, the JSON, or the documentation.
- Consequences: on this machine every Day 5 latency number comes from steady_clock, whose measurement floor is materially higher than a TSC read would be; the empty probe quantifies that floor.
- Evidence or benchmark: clock tests assert honest self-reporting, the architecture-dependent existence of the TSC path, and the synthetic tick-to-nanosecond conversion.
- Revisit conditions: a Linux x86-64 run would exercise the TSC path and record its calibration.
- Decision: latency runs record their scope explicitly (full file or an exact frame limit) in the result JSON, and both engines in a comparison always use the identical scope and sampling interval.
- Date: 2026-07-26
- Status: accepted
- Context: sample memory scales with operation count, so on a memory-constrained machine the latency scope may need to be smaller than the throughput scope; a reader must never have to guess which.
- Alternatives considered: silently reusing the throughput scope, or omitting the scope from the result (both invite misreading).
- Choice: the scope is a recorded field, checked identical across engines before any comparison is drawn.
- Consequences: throughput and latency scopes may differ within one result document; the tables print each explicitly.
- Evidence or benchmark: the committed result files and rendered tables carry the scope for every latency engine.
- Revisit conditions: none for v1.
- Decision: an empty probe (two clock reads with nothing between them, sampled the same way and reduced with the same percentile rule) is measured for every latency run and reported separately. It is never subtracted from any operation latency, and every report states that measurement overhead is included.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 requires stating the measured empty-probe cost plainly.
- Alternatives considered: subtracting the probe median from each sample (fabricates precision the measurement does not have, can produce nonsensical near-zero or negative values, and hides that on this platform the floor is a large fraction of a fast operation).
- Choice: report both, subtract neither.
- Consequences: on this machine the reported latencies for the cheapest operations are dominated by the clock, which the tables make visible rather than hiding.
- Evidence or benchmark: the empty probe appears in every latency fragment, in the aggregated JSON, and as its own table.
- Revisit conditions: none; this is a reporting-honesty rule.
- Decision: all percentiles use the nearest-rank empirical definition: rank = ceil(percentile * sample_count), clamped to [1, sample_count], reported value = sorted_samples[rank - 1]. The same function serves p50, p90, p99, p99.9, and the empty-probe percentiles.
- Date: 2026-07-26
- Status: accepted
- Context: percentile definitions differ across tools; an unstated one makes numbers incomparable.
- Alternatives considered: linear interpolation between ranks (invents values no operation exhibited) and the exclusive rank convention (adds an edge case at small sample counts for no benefit here).
- Choice: nearest rank, computed in extended precision and clamped so no sample count can index out of range.
- Consequences: at small sample counts p99 and p99.9 can be the same observation, which the published sample counts make evident.
- Evidence or benchmark: dedicated tests at every percentile, plus single sample, duplicate samples, and an outlier that must appear at the tail.
- Revisit conditions: none for v1.
- Decision: no measured sample is discarded and no valid measured pass is dropped, however inconvenient. A pass is rerun only when it is invalid for a stated mechanical reason (process failure, semantic or invariant failure, tool failure, system sleep, or interruption), and then the whole mode is rerun rather than one pass being substituted.
- Date: 2026-07-26
- Status: accepted
- Context: selective omission is the easiest way to make a benchmark lie.
- Alternatives considered: trimming the slowest pass or clipping tail samples (both would silently improve the tail percentiles that matter most).
- Choice: as stated, with the median providing robustness instead of filtering.
- Consequences: published tails include whatever the machine did, including interference from other applications; the limitations section says so.
- Evidence or benchmark: a test asserts a slow pass still participates in the median; the JSON preserves every pass.
- Revisit conditions: none; this is a reporting-honesty rule.
- Decision: peak memory is the process high-water resident set size from getrusage(RUSAGE_SELF).ru_maxrss, normalized to bytes (the field is bytes on macOS and kilobytes on Linux). It is reported per mode as the maximum and median across that mode's passes, and it is explicitly not virtual address space, not the mapped file's page cache, and not a per-pass figure.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 asks for peak memory; RSS units and meanings differ by platform and are easy to misreport.
- Alternatives considered: macOS phys_footprint through task_info (a different and less portable definition), and parsing /usr/bin/time output (external, and inconsistent across platforms).
- Choice: getrusage in-process, normalized, with the definition stated in the result document.
- Consequences: because one process runs all passes of a mode, the figure is the peak over the whole mode, which the result labels.
- Evidence or benchmark: a test asserts the value is positive and in a plausible byte range, which fails if the unit is wrong.
- Revisit conditions: none for v1.
- Decision: every result is accompanied by a committed environment capture containing timestamps, host label, operating system, kernel, architecture, CPU model and core layout, installed and available memory, swap, power source, low-power mode, thermal state, dataset basename with size, checksum, and filesystem, Git commit, branch, and clean state, compiler executable and full version, CMake version, build type, release flags, host-tuning flag, LTO state, pinning and governor availability, perf, xctrace, and sample availability, and the platform class. It contains no absolute home path, no secrets, and no market data.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 requires the environment stored alongside results.
- Alternatives considered: recording only a summary line (insufficient for reproduction) or embedding full paths (leaks the home directory).
- Choice: as stated; the dataset is identified by basename plus checksum.
- Consequences: a reader can tell exactly which machine, build, and input produced a number, and can see which capabilities were absent.
- Evidence or benchmark: the committed capture files; a privacy sweep for home paths and secret-like strings runs over the output.
- Revisit conditions: none for v1.
- Decision: on Linux the runner pins to one core with taskset, records its identity and the CPU governor, and prefers the performance governor without ever invoking sudo: if changing it needs privilege, that is reported, not silently attempted. On macOS core pinning and the governor concept do not exist, so the capture records both as unavailable, and the platform class is recorded as the documented macOS fallback rather than the Linux x86-64 reference platform.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md sections 1 and 7 name Linux x86-64 as primary with macOS as an acceptable fallback whose limitations must be stated.
- Alternatives considered: emulating pinning with thread affinity APIs (unavailable for this purpose on macOS) or silently omitting the fields (would make a fallback run look like a reference run).
- Choice: state the absence explicitly in the capture, the result JSON's limitations, and the rendered tables.
- Consequences: Day 5 results from this machine carry more run-to-run variance than a pinned Linux run would, and they are labeled accordingly.
- Evidence or benchmark: the environment capture on this machine records pinning unavailable, governor unavailable, and perf unavailable.
- Revisit conditions: a Linux x86-64 run.
- Decision: the synthetic component microbenchmarks (pool, order index, and both books) are built everywhere including CI, never executed by CI, and reported in their own file, clearly separated from full-file replay results. No microbenchmark asserts a timing threshold or that one engine must beat the other.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 wants component microbenchmarks and a CI that stays fast and synthetic.
- Alternatives considered: running them in CI (timings on shared runners are noise, and a threshold assertion would make CI flaky), or folding their numbers into the headline results (different scope entirely).
- Choice: build in CI, run manually or through the orchestration script, publish separately.
- Consequences: microbenchmark numbers inform tuning hypotheses but never become project throughput claims.
- Evidence or benchmark: bench/CMakeLists.txt builds the target; the CI workflow runs only ctest.
- Revisit conditions: none for v1.
- Decision: the committed JSON result documents plus their environment captures are the only authority for published numbers. Derived values (speedups, parse-only versus parse-plus-book book cost) are computed by the aggregator from the committed pass data. Tables are generated from the JSON by scripts/render_bench_tables.py, and no benchmark number is ever typed by hand into a document or the README.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 0 forbids borrowed or estimated numbers and section 7 requires README tables generated from the result JSON.
- Alternatives considered: writing tables by hand from the console output (invites transcription errors and quiet drift between text and data).
- Choice: as stated. Baseline and final results are separate labeled files; neither is overwritten by the other, and the aggregator refuses to silently replace an existing file.
- Consequences: Day 7 renders the README tables from the same generator rather than transcribing anything.
- Evidence or benchmark: the committed baseline, final, and table files; regenerating the tables from the same JSON is byte-identical.
- Revisit conditions: none for v1.
- Decision: profile MapBook and FastBook over an identical fixed scope (the first 100,000,000 complete frames) with the best tool the platform offers, in this preference order: Linux perf stat with the event names the machine actually supports, then the Instruments Time Profiler through xctrace, then /usr/bin/sample. Commit only a concise hotspot summary; keep raw profiler output and trace bundles out of Git.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 7 asks for a perf table on Linux and Instruments notes on macOS.
- Alternatives considered: skipping profiling on macOS (the tuning experiment must be evidence-driven, so some evidence is required), or reporting plausible-looking counter values (fabrication).
- Choice: on this machine perf does not exist and xctrace, although present as a binary with the Command Line Tools, refuses to run without a full Xcode installation, so /usr/bin/sample was used and the deviation is recorded in the profiling report itself. Call-stack sampling is never described as hardware performance counters, and no counter value is reported at all.
- Consequences: the profile shows where wall-clock time concentrates at sampling resolution, not instruction-level costs. With link-time optimization and -O3 the book calls inline into the pass runner, so leaf attribution is partial; the surviving named frames are still informative.
- Evidence or benchmark: bench/results/-baseline-profile.md.
- Revisit conditions: installing full Xcode would enable the preferred Instruments path; a Linux run would enable perf counters.
- Decision: the single Day 5 tuning experiment changes how OrderIndex chooses between doubling and purging at a maintenance trigger, so that a purge happens only when the headroom it buys is at least the number of live entries it must rehash; otherwise the table doubles.
- Date: 2026-07-26
- Status: proposed (acceptance decided by the measurement recorded below)
- Observed bottleneck: FastBook wins every median and p99 latency by roughly two to three times, but its add-path tail is worse than the reference book's. In the frozen baseline (bench/results/2026-07-26-baseline.json) FastBook add p99.9 is 3,542 ns against MapBook's 1,708 ns, and FastBook replace p99.9 is 3,542 ns against 2,875 ns, while FastBook add p50 is 125 ns against 250 ns. A tail worse than the reference book's, on the operation that accounts for about 45 percent of all book operations, is the clearest remaining inefficiency the baseline exposes.
- Supporting evidence: the baseline profile (bench/results/2026-07-26-baseline-profile.md) shows FastBook's surviving named leaf frames as remove_order, _platform_memmove (the sorted price-level pair vector shifting), OrderIndex::insert, and insert_order. Two of those four are the add path, and OrderIndex::insert is the only one whose cost is occasionally proportional to the whole live population rather than to a level's neighborhood, which is the signature of a burst.
- Amortization analysis of the current rule: maintenance fires when live plus tombstones would exceed 0.7 of capacity. The current choice doubles when live exceeds half of capacity and otherwise purges at the same capacity. A churning steady state therefore settles with live between 0.25 and 0.5 of capacity, where a purge rehashes live entries to buy only (0.7 x capacity - live) of headroom. At the unfavorable end (live near half of capacity) that is about 0.5 x capacity of work for about 0.2 x capacity of inserts, roughly 2.5 rehashed entries per insert. The real message mix, 124.2 million adds against 120.0 million deletes, sits exactly in that churning regime.
- Hypothesis: requiring a purge to buy at least as much headroom as it costs (headroom at least the live count) makes tables settle at or below 0.35 of capacity, bounds amortized rehash work at about one entry per insert, and reduces the frequency of rebuild bursts, which should lower FastBook's add and replace tail percentiles.
- Expected effect: lower p99 and p99.9 on FastBook add and replace; a small or neutral change in median throughput, since rebuilds are a minority of add cost.
- Possible regression: higher memory. Tables run at a lower load factor, so the order index grows by up to one doubling per book. At roughly 3 million concurrently live orders across all books that is on the order of tens of megabytes of additional resident memory, which the memory table will show. A second possible outcome is no measurable change, which would mean rebuilds are not the tail driver.
- Correctness risk: low. The trigger point is unchanged, so the invariant that a truly empty slot always exists (and therefore that every probe terminates) is untouched; only the action chosen at the trigger changes. The existing hash test that pins the old boundary behavior must be updated to pin the new rule deliberately, and the full battery must stay green.
- Deciding comparison: the identical Day 5 protocol re-run in full. Accept only if the complete correctness battery stays green, the 100-million dual-engine validation stays green, FastBook add or replace tail percentiles improve by more than run-to-run noise (the baseline's own within-mode pass spread reached 4.4 percent), and neither throughput nor memory regresses materially. Otherwise revert and record the negative result.
- Evidence correction after the baseline was frozen: the tail numbers quoted above came from an earlier full run that was discarded for a metadata defect, not from the committed baseline. In the committed baseline (bench/results/2026-07-26-baseline.json) FastBook's tails are no longer worse than MapBook's: FastBook add p99.9 is 3,916 ns against MapBook's 4,000 ns, and FastBook replace p99.9 is 4,000 ns against MapBook's 8,542 ns. Comparing the two runs shows the p99.9 figures moving by thousands of nanoseconds between runs of identical code, so on this unpinned, battery-powered, shared machine the tail is dominated by system noise rather than by engine behavior. That weakens the tail-based motivation considerably and is recorded rather than quietly dropped. The amortization argument for the change stands on its own, since it is a property of the rebuild policy rather than of any measurement, so the experiment proceeds; but the acceptance decision now rests primarily on throughput and memory, with tail movement inside this noise band treated as inconclusive.
- Decision: the D-047 order-index amortization change was implemented, validated, measured, and then reverted. It is not part of the shipped engine. The negative result is kept here because it is evidence.
- Date: 2026-07-26
- Status: accepted (the experiment is closed; the change is reverted)
- What was changed: at a maintenance trigger, choose a same-capacity purge only when the headroom it buys is at least the live entries it rehashes, otherwise double. Implemented in commit bae81b2, validated in dfa1ed2, reverted afterward.
- Correctness during the experiment: fully green. 386 tests in debug, ASan/UBSan, and release (one more than the baseline suite, because the experiment added a boundary test), the 1,000,000-operation order-index cross-check, randomized MapBook versus FastBook equivalence, the synthetic divergence mutants, and the required 100-million-message real dual-engine validation: 100 checkpoints, 8,840 books compared, zero semantic errors and zero status, counter, invariant, digest, and exact-state mismatches, with identical final actives and levels in both engines and a clean final exact comparison. The change was never a correctness problem.
- Measurement 1, component microbenchmarks (5 repetitions each, medians, synthetic data): FastBook add at an existing level improved from 42.37 ns to 27.03 ns (36 percent faster), add creating a level from 50.84 ns to 43.09 ns (15 percent), delete from 18.07 ns to 14.57 ns (19 percent), while both replace cases and every direct order-index case moved by about 1 percent, which is within their repetition spread. The predicted mechanism is visible exactly where predicted: the add and delete cases churn a live population, which is the regime the rebuild policy governs.
- Measurement 2, paired full-file replay, alternating builds to cancel machine drift. At 50 million frames the tuned build was faster in three of four pairs, medians 12.478 s against 13.059 s, about 4.5 percent. At 100 million frames, the scope closest to the real working set, the alternated pairs were base 30.788, 27.252, 26.003 against tuned 26.141, 27.541, 28.313, medians 27.252 against 27.541: a difference of about 1 percent with distributions that overlap almost completely. Two earlier non-alternated 100-million pairs favored the base build by roughly 20 percent. Taken together the full-file evidence is inconclusive at 50 million and slightly negative at 100 million.
- Measurement 3, memory: peak anonymous footprint over 100 million frames rose from 243 to 244 MiB on the base build to 276 to 281 MiB on the tuned build, about 14 percent, matching the regression the hypothesis predicted. Tables settle at a lower load factor, so they occupy more memory.
- Interpretation: the change does what it was designed to do, which the microbenchmarks show clearly, but the saving does not survive at scale. A sparser table spans more cache lines, so every lookup and insert touches colder memory; at the real working set (roughly 1.8 million live orders at 100 million frames) that locality cost appears to cancel the rehash saving, while the memory cost remains. A component-level win that disappears in the full system is a useful result, not a failure of the method.
- Decision against the acceptance rule stated in D-047: accept only if the intended metric improves beyond run-to-run noise and no memory regression is hidden. Full-file throughput did not improve beyond noise at the real scale, and memory measurably regressed, so the change is reverted.
- Why the full-protocol before-and-after table cannot decide this: the tuned full protocol run (bench/results/2026-07-26-tuned-experiment.json) shows every mode slower than the baseline, including the framing scan, which contains no book and no hash and therefore cannot be affected by the change, and MapBook, whose code is identical in both runs, improved its p99.9 latencies by 25 to 40 percent in the same run. Those two controls moved by more than the effect under test, so between-run drift on this unpinned, battery-powered, shared machine dominates that comparison. The paired alternating measurements above exist precisely because that table could not answer the question. Both result files are preserved.
- Note on the preserved file: bench/results/2026-07-26-tuned-experiment.json carries the internal result_label "final" because it was produced under that label before the acceptance decision was taken. The filename reflects what it actually measures. The JSON was not edited after the fact.
- Revisit conditions: a pinned Linux x86-64 machine with hardware counters could separate rehash cost from cache-locality cost directly and might reach a different conclusion; a hybrid rule (amortized purge with a cap on table sparsity) would be the natural next experiment.
- Decision: the Day 6 full-day analysis replays through FastBook. The statistics observer is engine-neutral by construction, and that neutrality is proved by synthetic and bounded real comparisons against MapBook rather than by a second full-day run.
- Date: 2026-07-26
- Status: accepted
- Context: Day 4 proved exact MapBook and FastBook equivalence over the first 100 million real frames (docs/day4_fastbook.md), and Day 5 measured FastBook at roughly 1.85 times MapBook's replay throughput. The analysis needs two full replays of a 282,229,684-frame file, so engine choice is a wall-clock decision, not a correctness one.
- Alternatives considered: running the whole analysis through both engines and comparing every output (doubles an already long run for evidence the Day 4 proof already supplies); running through MapBook for conservatism (slower, and no more trustworthy given the Day 4 result).
- Choice: FastBook for the full-day analysis. Neutrality is established by (1) replaying every committed synthetic fixture through Engine and Engine with the identical observer and requiring byte-identical outputs, and (2) a bounded real-data aggregate comparison over the first 10,000,000 frames through both engines.
- Reasoning: the observer reads only the book contract surface both engines implement identically, so a divergence in statistics would imply a divergence in book state, which Day 4 already tests for directly.
- Consequences: the analysis manifest records the engine used. The bounded real comparison is a correctness reconciliation, never a benchmark, and no timing from it is published.
- Evidence or benchmark: three synthetic tests, which are where the file-level proof lives, plus one bounded real comparison. The tests are AggregateObserver.ProducesIdenticalAggregatesFromBothEngines in tests/stats_test.cpp (every counter, every second, every locate), AnalysisOutputs.MatchAcrossBothEngines in tests/analyze_test.cpp (byte-identical files), and CliAnalyze.ProducesIdenticalStatisticsFromBothEngines in tests/cli_analyze_test.cpp (the same through the shipped binary). The bounded real comparison over the first 10,000,000 frames of the pinned day is recorded in docs/day6_findings.md. An earlier version of this entry named a file, tests/stats_engine_neutral_test.cpp, that was never created; the tests it described were written inside the three files above instead.
- Revisit conditions: any future change to either book's public contract.
- Decision: statistics attach to the existing engine through a third template parameter (the observer), defaulting to a stateless NoObserver whose hooks compile to nothing. There is no second replay engine and no virtual dispatch on the mutation path.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 8 requires a findings layer; sections 0 and 7 forbid changing the measured code path. Day 5 already introduced an instrument seam for latency (D-035), which proved the template-parameter pattern works without disturbing ordinary replay.
- Alternatives considered: a separate statistics replay loop that decodes independently (duplicates the decoder and can silently drift from the shipped path); a virtual observer interface (adds an indirect call to every mutation and defeats inlining); post-processing a dumped event stream (would require writing hundreds of gigabytes of real feed values to disk, which the data policy forbids outright).
- Choice: Engine<Book, Instrument, Observer>. Every hook is guarded by
if constexpr (Observer::kEnabled), and the top-of-book hooks are further guarded byObserver::kWantsTopOfBook, so the default instantiation generates the same code it generated on Day 5. - Reasoning: the observer must never be able to change replay. Passing it as a type rather than a callback keeps the guarantee structural: with the default observer there is no observer object, no branch, and no call.
- Consequences: the observer may read the book only through the frozen BookContract queries. It cannot mutate, cannot change a status, and cannot suppress a message; the engine ignores every value the observer returns. scan, book, validate, and bench keep their Day 5 behavior and their Day 5 exit codes.
- Evidence or benchmark: the existing 386 tests pass unchanged, the Day 4 100-million-frame dual-engine validation is re-run after the change, and a dedicated test asserts that a replay with the statistics observer attached produces the identical final book state, mutation count, and status sequence as a replay without it.
- Revisit conditions: a hook that cannot be expressed read-only.
- Decision: time buckets are computed with integer division on the raw 48-bit nanosecond timestamp, never in floating point: second_bucket = timestamp_ns / 1000000000 and millisecond_bucket = timestamp_ns / 1000000. Every interval is half open, [start, start + width). The analyzer reads the locate and timestamp of every KNOWN message type from the common 11-byte header, including types whose bodies it does not decode.
- Date: 2026-07-26
- Status: accepted
- Context: ITCH 5.0 gives every message the same header: type at offset 0, stock locate at 1, tracking number at 3, and a 48-bit nanoseconds-since- midnight timestamp at 5. The Day 2 decode set covers S, R, A, F, E, C, X, D, and U, which is 276,798,644 of the day's 282,229,684 frames; the remaining 5,431,040 known frames (I, P, Q, Y, L, H, J, V) carry timestamps the activity series would otherwise ignore.
- Alternatives considered: restricting per-second activity to decoded messages only (would silently drop about 1.9 percent of the day's messages from an "activity" chart, which is misleading); decoding the full bodies of P, Q, and H (out of Day 6 scope, which forbids parser changes unrelated to observation).
- Choice: a header-only read in the analysis driver, guarded by the declared payload length, for known types. Unknown type bytes are counted but never timestamped, because their layout is unknown.
- Reasoning: reading a documented common header is observation, not a parser change; it adds no decode path and cannot fail a replay.
- Consequences: per-second and per-millisecond series cover every known message. The manifest records the distinction between all known messages and the decoded subset. Timestamps beyond the 86,400-second day are counted in an explicit out-of-range bucket rather than clamped, so a synthetic maximum-timestamp message cannot corrupt the day series.
- Evidence or benchmark: synthetic bucket-boundary tests, including timestamp zero, the last nanosecond of a second, and the maximum 48-bit timestamp.
- Revisit conditions: none for v1.
- Decision: prices remain unsigned 32-bit integers with four implied decimals in every calculation. Binary floating point appears only where a ratio is reported, and never in a price, spread, midpoint, duration, or count.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 2 forbids floats on the hot path and D-010 already keeps prices raw through the codec and both books.
- Alternatives considered: converting to double for spread statistics (introduces representation error into published numbers for no benefit).
- Choice: spread_raw = ask_raw - bid_raw as a signed 64-bit value, computed only when both sides exist. Weighted sums use 128-bit integer accumulation, and decimal strings are produced by integer division at output time.
- Reasoning: every published price statistic must be exactly reproducible from the integers in the feed.
- Consequences: means are reported as exact scaled integers plus a decimal rendering, not as doubles. The analysis code uses __int128 for weighted sums, which both supported toolchains provide.
- Evidence or benchmark: exact-value tests over hand-computed scenarios.
- Revisit conditions: a platform without a 128-bit integer type.
- Decision: the midpoint is stored and reported as midpoint_times_2 = best_bid_raw + best_ask_raw, an exact integer, and is never rounded internally. Decimal output renders it with five decimal places, which is exact because a half tick of 0.0001 is 0.00005.
- Date: 2026-07-26
- Status: accepted
- Context: the midpoint of two ticks is a half tick whenever the spread is odd, so any representation in whole ticks must round.
- Alternatives considered: rounding to the nearest tick (loses information and biases a time-weighted mean); a rational type (heavier than needed when a factor of two suffices); floating point (rejected by D-052).
- Choice: carry the doubled value everywhere and divide only when formatting. The CSV carries both midpoint_times_2 and the five-decimal rendering, so a reader can recompute either.
- Reasoning: doubling is the smallest exact representation for a two-value midpoint.
- Consequences: the time-weighted mean midpoint is computed as sum(midpoint_times_2 * duration) / (2 * total_duration) in 128-bit integers and reported scaled.
- Evidence or benchmark: half-unit and whole-unit midpoint tests.
- Revisit conditions: none for v1.
- Decision: every top-of-book observation is classified as normal, locked, crossed, one_sided_bid, one_sided_ask, or empty. Crossed spreads are recorded as negative values and never clamped, and no observation is discarded.
- Date: 2026-07-26
- Status: accepted
- Context: D-015 records that this engine mirrors the feed and never matches, so a resting crossed or locked book is a legitimate observed state rather than a defect. A crossed book in ITCH usually reflects the reporting order of the feed rather than an executable arbitrage.
- Alternatives considered: clamping negative spreads to zero (fabricates data); dropping crossed observations (hides real states and breaks the duration reconciliation).
- Choice: classify, count, and time-weight every state separately. Spread statistics are computed over the two-sided states (normal, locked, and crossed) and the durations of each class are reported individually so a reader can exclude any class.
- Reasoning: a statistic that silently deletes inconvenient states is not defensible.
- Consequences: the duration components sum exactly to the observed duration, which the invariant checker enforces. Charts mark locked and crossed intervals rather than interpolating across them.
- Evidence or benchmark: locked, crossed, bid-only, ask-only, and empty scenario tests with hand-computed durations.
- Revisit conditions: none for v1.
- Decision: statistics about market state over time (spread, midpoint, state occupancy) are time weighted by the duration each state was in force. Statistics about message flow (counts by type, per-second activity, busiest intervals) are event weighted. Every reported statistic is labeled with which weighting it uses.
- Date: 2026-07-26
- Status: accepted
- Context: quoting activity is extremely uneven across a trading day. An event-weighted mean spread answers "what was the spread when something happened", which is dominated by the busiest microseconds; a time-weighted mean answers "what was the spread during the day".
- Alternatives considered: reporting only event-weighted statistics (cheaper, but systematically biased toward high-activity periods); sampling the book on a fixed grid (adds an arbitrary grid and loses exact reconstruction).
- Choice: on every top-of-book change, attribute the elapsed interval to the state that was in force, then adopt the new state. Attribution starts at the symbol's first observed top-of-book state and ends at the analysis window boundary. No state is inferred before the first observation.
- Reasoning: exact interval attribution is both cheaper and more accurate than sampling.
- Consequences: the final state of a window is attributed from its start to the window end, which for the full-day window is the timestamp of the last message in the file. That treatment is stated in every summary.
- Evidence or benchmark: duration reconciliation tests and the invariant that state durations sum to the observed duration.
- Revisit conditions: none for v1.
- Decision: the regular-session window is [timestamp of the 'Q' System Event, timestamp of the 'M' System Event), taken from the feed itself. The full window is every frame in the file. Both windows are reported for the selected-symbol state analysis.
- Date: 2026-07-26
- Status: accepted
- Context: the ITCH 5.0 System Event message defines 'O' start of messages, 'S' start of system hours, 'Q' start of market hours, 'M' end of market hours, 'E' end of system hours, and 'C' end of messages. Day 1 observed exactly six System Event messages on the pinned day, which matches that set exactly.
- Alternatives considered: a hard-coded clock window such as 09:30 to 16:00 (an assumption when the feed states the answer); using only the full file (mixes pre-market and post-market quoting into a "typical spread" figure).
- Choice: derive the boundaries from the observed events, verify that exactly one 'Q' precedes exactly one 'M', and fall back to a documented fixed clock window only if that verification fails. The manifest records which source was used and the exact boundary nanosecond values.
- Reasoning: the feed is the authority for its own session.
- Consequences: statistics are labeled full or regular, never unlabeled.
- Evidence or benchmark: the observed boundary timestamps are recorded in the manifest and in docs/day6_findings.md.
- Revisit conditions: a day whose system events are incomplete.
- Decision: per-second activity uses one fixed array of 86,400 entries, indexed by second since midnight, plus a single out-of-range accumulator.
- Date: 2026-07-26
- Status: accepted
- Context: a trading day has 86,400 seconds, and the ITCH timestamp is defined as nanoseconds since midnight.
- Alternatives considered: a hash map keyed by second (allocation and hashing per message for a dense, tiny key space); a growing vector (same result with extra branches).
- Choice: a preallocated array of counter structures, about 6 MB, touched by index with no allocation after construction.
- Reasoning: dense key space, bounded size, zero per-message allocation.
- Consequences: any timestamp at or beyond 86,400 seconds lands in the out-of-range accumulator and is reported rather than clamped, which keeps the reconciliation exact for synthetic maximum-timestamp messages.
- Evidence or benchmark: per-second sums reconcile with global counts in the invariant checker.
- Revisit conditions: none for v1.
- Decision: busiest-millisecond analysis uses a sliding window of eight seconds of dense millisecond buckets (8,000 buckets) that flushes each completed second into a fixed-size top-K list per metric. It never allocates a structure proportional to the day in milliseconds.
- Date: 2026-07-26
- Status: accepted
- Context: a dense day-long millisecond structure with four metrics would need roughly 1.7 GB, which this 8 GB machine cannot afford beside a memory-mapped 8.66 GB file. ITCH timestamps are near monotonic but the code must not assume strict monotonicity (product_spec.md section 2).
- Alternatives considered: a sparse hash map keyed by millisecond (tens of millions of live entries, roughly a gigabyte, plus hashing per message); approximate sketches (unacceptable when the output is a published exact count); a dense array for the observed range only (still hundreds of megabytes per metric).
- Choice: the eight-second ring. A message whose second falls inside the window updates its bucket exactly, wherever it arrives in the stream. A message older than the window is counted in an explicit late-event counter and accumulated in a small sparse overflow map that is merged before ranking. The exactness precondition is therefore observable: with zero late events the top-K is exact by construction, and the counter is published.
- Reasoning: local reordering is handled exactly, pathological reordering is detected rather than silently absorbed, and memory stays at kilobytes.
- Consequences: ties are broken by the earlier millisecond. Results are cross-checked against an independent dense recomputation at bounded scale and, for the message metric, over the full day (D-067).
- Evidence or benchmark: synthetic tie and reordering tests plus the independent dense verifier.
- Revisit conditions: a feed with substantial out-of-order timestamps.
- Decision: per-symbol statistics accumulate into a fixed array indexed by stock locate and are joined to ticker text from the directory only when the outputs are written.
- Date: 2026-07-26
- Status: accepted
- Context: D-017 already keeps tickers off the order path; locates are dense 16-bit integers and the day uses 8,849 of them.
- Alternatives considered: keying statistics by ticker string (string hashing per message, and it breaks for locates whose directory message has not arrived yet).
- Choice: a 65,536-entry array of per-symbol counters, roughly 6 MB, joined to directory metadata at output time.
- Reasoning: identical to the engine's own routing model, so the statistics cannot disagree with the engine about which symbol a message belongs to.
- Consequences: a locate with activity but no directory entry is reported with an empty ticker rather than dropped, and is ineligible for selection.
- Evidence or benchmark: per-symbol sums reconcile with global counts.
- Revisit conditions: none for v1.
- Decision: the three symbols that receive detailed state analysis are selected mechanically: eligible symbols are those with a directory entry, a displayable ticker, and at least one observation with both a bid and an ask during the analysis window; eligible symbols are ranked by successful book mutations in the window, descending, with ties broken alphabetically by ticker and then by locate; the top three are selected.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 8 asks for the most active symbols, and section 15.4 warns against choosing a recognizable brand because it looks interesting.
- Alternatives considered: ranking by executed shares (measures trading rather than quoting activity, and the spread analysis is about quoting); ranking by total messages (would include non-book messages such as imbalance indicators, which do not exercise the book); manual selection (not defensible).
- Choice: rank by successful book mutations. The full ranking evidence, the mutation counts, and the rule itself are written into the manifest, so the selection can be recomputed from the committed symbol summary.
- Reasoning: mutations are exactly the events the top-of-book analysis observes, so the ranking metric matches the analysis subject.
- Consequences: if fewer than three symbols are eligible the command selects fewer and records why. An ineligible ticker is never silently replaced by a preferred one; the eligibility rule is applied uniformly and any exclusion is recorded.
- Evidence or benchmark: symbol_summary.csv contains every symbol's counts, so any reader can rerun the ranking.
- Revisit conditions: none for v1.
- Decision: the analysis makes two replay passes. Pass one computes global, per-second, per-millisecond, and per-symbol aggregates and selects the detailed symbols. Pass two replays again and tracks top-of-book state only for the selected symbols.
- Date: 2026-07-26
- Status: accepted
- Context: the detailed analysis needs the selected symbols before it can track them, and tracking every symbol's top-of-book history would retain hundreds of millions of state rows.
- Alternatives considered: one pass retaining state changes for all 8,849 symbols (tens of gigabytes); one pass with a guessed symbol list (defeats objective selection); one pass writing all state changes to disk and filtering afterward (writes real order-book values for the whole market to disk, which the data policy discourages and which would be far larger than the input).
- Choice: two passes, run automatically by one command.
- Reasoning: a second sequential read of a memory-mapped file is cheap relative to retaining the data, and it keeps peak memory bounded.
- Consequences: the analysis runs about two replays long. That time is diagnostic and is never compared with Day 5 benchmark numbers.
- Evidence or benchmark: the recorded wall time and peak RSS of the full-day run.
- Revisit conditions: none for v1.
- Decision: committed outputs contain aggregate statistics, per-second activity, per-symbol totals, and top-of-book state series for the three selected symbols only. They never contain order reference numbers, individual order quantities, individual execution records, queue contents, or raw message bytes.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 0 forbids committing Nasdaq data or any excerpt of it. A top-of-book series is an aggregate of a level, not a message excerpt, and the selected tickers are public identifiers.
- Alternatives considered: committing no symbol-level data at all (removes the most informative part of the findings layer); committing full order books (an excerpt of the feed in all but name).
- Choice: level aggregates for three publicly identified, objectively selected symbols. Every row is a summary of the best level: price, total shares at that level, and the derived state, and never an individual order.
- Reasoning: level totals cannot be inverted into individual orders, so the committed data is a statistic rather than a redistribution.
- Consequences: a test scans every generated output for prohibited fields and for absolute paths before the outputs may be committed.
- Revisit conditions: none for v1.
- Decision: spread percentiles are time weighted and computed by nearest rank over accumulated durations: collect distinct spread values with their total durations, sort ascending, and report the smallest value whose cumulative duration reaches ceil(p * total_duration), computed in integer arithmetic. No interpolation and no trimming.
- Date: 2026-07-26
- Status: accepted
- Context: Day 5 already fixed one percentile definition for latency (D-039); using a second, different rule for spreads would be confusing.
- Alternatives considered: event-weighted percentiles (biased toward busy moments, per D-055); expanding one sample per millisecond (turns a exact computation into an approximation with a huge memory cost); linear interpolation (invents values the book never showed).
- Choice: nearest rank over durations, the same ceil rule Day 5 uses, with durations as the weights instead of sample counts.
- Reasoning: the reported value is always a spread the book actually had.
- Consequences: percentiles are monotonic by construction, which the invariant checker asserts. The event-weighted count of each spread value is not published, to avoid two similar numbers with different meanings.
- Evidence or benchmark: hand-computed weighted percentile tests.
- Revisit conditions: none for v1.
- Decision: Day 6 reports execution events and executed shares. It does not report dollar notional.
- Date: 2026-07-26
- Status: accepted
- Context: 'E' Order Executed carries no price; the execution occurs at the resting order's price. 'C' Order Executed With Price carries an explicit execution price and a printable flag.
- Alternatives considered: using the resting order's price for 'E' (the engine has it, but that mixes a reconstructed price with a reported one and would need its own validation); reporting notional for 'C' only (a number covering 0.05 percent of execution events invites misinterpretation as day-wide volume).
- Choice: shares only, with 'E' and 'C' counted and summed separately so a later day can add notional deliberately.
- Reasoning: a partial or mixed-basis notional is worse than no notional.
- Consequences: findings speak in shares and events, never in dollars.
- Revisit conditions: a post-v1 analysis that defines and validates a price basis.
- Decision: the analysis writes every output into a staging directory beside the requested destination and renames it into place only after every pass, every invariant, and every checksum has succeeded. An existing destination is refused unless --force is given.
- Date: 2026-07-26
- Status: accepted
- Context: a multi-hundred-second analysis that fails partway must not leave a half-written results directory that looks complete, and committed results must never be a mixture of two runs.
- Alternatives considered: writing directly into the destination (leaves partial output on failure); writing to a temporary directory elsewhere and copying (a copy across filesystems is not atomic).
- Choice: stage in a sibling directory so the rename stays within one filesystem, then rename. With --force the previous directory is moved aside, the new one renamed in, and the old one removed only after the rename succeeds.
- Reasoning: the destination is either the previous complete run or the new complete run, never a blend.
- Consequences: a failed run leaves the staging directory for inspection and exits nonzero, and the exit code distinguishes an output failure from an existing-output refusal.
- Evidence or benchmark: CLI tests for refusal, forced replacement, and failure leaving no partial destination.
- Revisit conditions: none for v1.
- Decision: analysis/results/analysis.json is the authority for every Day 6 number. It carries a schema version, the dataset identity, the frozen methodology, every aggregate, the selection evidence, the invariant results, and the relative path, row count, and SHA-256 of every generated file. Documents and charts are derived from committed outputs, never typed by hand.
- Date: 2026-07-26
- Status: accepted
- Context: this mirrors D-045, which made the benchmark JSON the benchmark authority.
- Alternatives considered: writing findings prose from console output (invites transcription drift).
- Choice: as stated, with a --generated-at option so a regeneration can be compared byte for byte against a committed run. SHA-256 is implemented in the repository rather than shelled out, so the checksum is computed by the same process that wrote the bytes.
- Reasoning: a reader must be able to verify that the committed CSV files are the ones the manifest describes.
- Consequences: a test regenerates the analysis from a fixture with a fixed timestamp and requires byte identity, and another test recomputes every checksum in the manifest.
- Revisit conditions: none for v1.
- Decision: headline totals are recomputed by a path that does not share accumulator code with the analyzer. scripts/verify_analysis.py reads only the committed outputs and independently recomputes the global totals, the per-second reductions, the busiest-second rankings, the symbol ranking, and the selected-symbol row counts. A dense per-millisecond recomputation inside the C++ analysis verifies the busiest-millisecond results.
- Date: 2026-07-26
- Status: accepted
- Context: an accumulator that is wrong in the same way twice cannot be caught by reusing itself.
- Alternatives considered: reusing the analyzer's own reducers for verification (proves nothing); parsing the 8.66 GB binary in Python (slow and needlessly duplicates the decoder).
- Choice: a Python verifier over the committed aggregates, plus a dense millisecond verifier in the second C++ pass. The dense verifier counts one metric, messages per millisecond, at any scale; it is a full linear scan over a dense array covering the observed range, and it is skipped with a recorded reason when that range exceeds its memory budget. The other three millisecond metrics are checked against an independent brute-force reducer in the unit tests rather than on real data, because four dense day-length arrays would cost roughly 1.7 GB, which is the constraint D-058 exists to respect. The manifest records whether the dense recomputation ran and what it covered rather than implying that every metric was verified.
- Reasoning: verification must be cheap enough to run every time and independent enough to fail when the analyzer is wrong.
- Consequences: the manifest records the reconciliation outcome. A disagreement stops acceptance, gets a synthetic regression test, and the full-day analysis is rerun.
- Revisit conditions: none for v1.
- Decision: scripts/summarize_findings.py reads the committed outputs and emits at least five candidate findings, each with an identifier, claim, scope, supporting metrics with their source fields, and limitations. Exactly three are then selected for docs/day6_findings.md on stated grounds, and the unselected candidates stay committed.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 8 requires observations backed by numbers, and section 15.4 requires findings that state their population, denominator, and limitations.
- Alternatives considered: writing findings by hand from the tables (no mechanical traceability, and it invites choosing the conclusion first).
- Choice: generate candidates mechanically, select three, and keep the full candidate list so the selection is visible.
- Reasoning: keeping the rejected candidates is what makes the selection honest.
- Consequences: no finding may claim manipulation, inefficiency, investor intent, strategy profitability, exchange causality, or that this day is representative of other days, because a single day of one venue's feed cannot establish any of those.
- Revisit conditions: none for v1.
- Decision: scripts/render_analysis.py reads only the committed CSV and JSON outputs, never the Nasdaq file, and writes one chart per image with fixed dimensions. It exits nonzero on missing files, missing columns, or malformed rows.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 1 allows matplotlib and pandas for side-scripts; Day 6 uses the standard library plus matplotlib and does not use seaborn.
- Alternatives considered: subplots combining several series (the spec asks for one chart per image); dual axes for counts and shares (visually implies a relationship between incompatible units).
- Choice: separate images, explicit units, the dataset date in every title, and no smoothing that hides raw spikes. Where a smoothed line is drawn, the raw series is drawn underneath it and the window is stated in the caption text of the chart.
- Reasoning: a chart in a portfolio repository is a claim; it must be regenerable and must not flatter the data.
- Consequences: charts regenerate byte-comparable from committed inputs on the same matplotlib version. The Day 7 hero composition is out of scope.
- Revisit conditions: none for v1.
- Decision: nanobook analyze uses 0 success, 2 usage error, 3 file open or mapping failure, 4 framing or decoding failure, 6 semantic replay failure, 11 statistical invariant failure, 12 output failure, and 13 refusal to overwrite an existing output directory.
- Date: 2026-07-26
- Status: accepted
- Context: the CLI already assigns 0, 2, 3, 4, 5, 6, 7, 8, 9, and 10 (docs/decisions.md D-006, D-014, D-017, D-018, D-028, and the Day 5 bench code).
- Alternatives considered: reusing 8 for statistical invariants (8 already means a book invariant failure, and conflating the two would hide which layer failed).
- Choice: new codes 11, 12, and 13, with 3, 4, and 6 shared with the existing replay commands because they mean exactly the same thing.
- Reasoning: a script must be able to tell a refusal to overwrite from a real failure.
- Consequences: every code is exercised by a CLI test.
- Revisit conditions: none for v1.
- Decision: include/nanobook/stats.hpp holds the observer types and the statistical accumulators; include/nanobook/analyze.hpp holds the two-pass driver, the output writers, the manifest, and the invariant checker. include/nanobook/sha256.hpp holds the checksum primitive.
- Date: 2026-07-26
- Status: accepted
- Context: product_spec.md section 3 lists stats.hpp as the statistics header, and section 11 asks for a lean surface. The complete Day 6 layer is larger than any existing header.
- Alternatives considered: one header (roughly two thousand lines mixing measurement with file output and JSON, which is harder to test in isolation); adding .cpp files (the project is header-only by design and every consumer is a template).
- Choice: three headers, mirroring the existing separation between book_map.hpp and validate.hpp. This is a deliberate deviation from the single-header sketch in the specification and is recorded as one.
- Reasoning: the accumulators are unit-testable without any file system, and keeping output concerns out of stats.hpp preserves that.
- Consequences: stats.hpp is no longer a scaffold; the analysis driver parallels validate.hpp in structure and naming.
- Revisit conditions: none for v1.
- Decision: the committed per-symbol top-of-book series is reduced. A row is written when the top of book changes and either the state classification changed or at least one second has passed since the previous row, plus the final observation of the series. Every published statistic is still computed in C++ from every state change.
- Date: 2026-07-26
- Status: accepted
- Context: the complete series for the three selected symbols of the pinned day is 3,387,986 rows and about 363 MB. Committing that to a portfolio repository is neither reasonable nor useful; the same statistics are available exactly, and the charts reduce to one point per second anyway.
- Alternatives considered: committing the complete series (363 MB in Git, for data a reader will never scroll through); committing nothing per symbol (removes the most informative artifact of the day); Git LFS (adds a dependency and a hosting requirement for data that compresses poorly and is not needed at full resolution).
- Choice: reduce with an interval rule stated in the manifest
(
symbol_series_interval_ns,symbol_series_complete) and always emit state-classification changes, so brief locked, crossed, or one-sided episodes survive the reduction even when they last microseconds. The committed run used one second. - Reasoning: the reduction touches presentation only. The observer attributes every interval and accumulates every duration before the emission filter runs.
- Consequences: proved rather than asserted. The full-day analysis was rerun with the complete series, and every aggregate, every per-second row, every per-symbol row, and every spread statistic was identical; only the row counts differ. The independent verifier rebuilt the state durations, weighted means, and weighted quantiles from that complete series and agreed with the committed summary, which is the check the reduced series cannot support on its own.
- Evidence or benchmark: docs/day6_findings.md records both runs; scripts/verify_analysis.py checks the emission rule itself on a reduced series and the full duration reconstruction on a complete one.
- Revisit conditions: a symbol whose classification flaps so often that the reduced series stops being small.
- Decision: record the defects Day 6 surfaced, what caused them, and what now prevents them from returning.
- Date: 2026-07-26
- Status: accepted
- Defect 1, a benchmark test that could fail without a defect. The Day 5 fragment test asserted that the fixture's raw price digits never appear in the benchmark JSON. Those digit strings are not distinguishable from ordinary measurements: a one millisecond latency tail is exactly "1000000" nanoseconds. It failed once in a debug run with nothing leaked. Fixed by checking the property directly: every key in the fragment must belong to the published schema, and the ticker and field-name checks stay. No benchmark result or committed file changed.
- Defect 2, an independent verifier that was right about the wrong rule. The verifier required that no two consecutive state rows carry the same state, which is true only when every state change is emitted. On the reduced series it fired on 1,509 legitimate rows for SPY alone, where the top had changed away and back inside one interval. The first full-day run stopped on it, which is the reconciliation gate doing its job. Fixed by making the check conditional: a complete series must have no repeated state, and a reduced series must instead satisfy the emission rule itself, with no two rows identical in both timestamp and state.
- Defect 3, a verifier committed one commit ahead of the fields it reads. scripts/verify_analysis.py rebuilds the symbol ranking from symbol_summary.csv, which needs the ever_two_sided and regular_book_mutations columns and the manifest's selection_metric field. The verifier was committed in "build: add deterministic analysis rendering and reconciliation" while those fields landed in the next commit, so that one CI run went red on the artifact test, and the following push was green again. The local suite had passed because the field change was sitting uncommitted in the working tree when it ran. The lesson is the ordering rule, not a code change: when a commit reads a field, the field must be in that commit or an earlier one, and a suite run with uncommitted changes present does not prove the commit is green. The failure is disclosed here and in the Day 6 report rather than being quietly absorbed by the next commit.
- Defect 4, a test that asserted the wrong arithmetic. Three analysis tests were written with hand-computed values that were wrong, not code that was wrong: a price of 100000 raw units is 10.0000 dollars rather than 100.0000, the two-sided duration of the state-walk fixture is five seconds rather than four, and the weighted median of that fixture lands on the value whose cumulative duration first reaches the rank. All three expectations were corrected against hand recalculation; the implementation was not changed.
- Evidence or benchmark: the corresponding tests in tests/cli_bench_test.cpp, scripts/verify_analysis.py, and tests/analyze_test.cpp.
- Revisit conditions: none; historical record.
- Decision: record two properties of the pinned day that limit or support the Day 6 claims, so neither is discovered later as a surprise.
- Date: 2026-07-26
- Status: accepted
- Observation 1, no locked or crossed book at the top. Across the twenty-five most active symbols of the whole day, the time spent locked and the time spent crossed were both exactly zero nanoseconds, while all twenty-five spent measurable time one sided. The classification therefore fires on real data for the one-sided cases and is exercised for locked and crossed only by synthetic tests on this dataset. That is an honest statement about coverage and about the day: Nasdaq's own displayed book for its most active names never rested locked or crossed at the top on 2019-07-30. It is not evidence about other venues, other days, or the consolidated market.
- Observation 2, the empty tail. Each selected symbol shows about 299.92 seconds with an empty book. That is the interval between the end of system hours at 20:00:00 and the last message in the file at 20:05:00, after the exchange's own end-of-day deletions. It is reported rather than trimmed, because trimming it would mean inventing a window boundary the feed did not state.
- Evidence or benchmark: the twenty-five symbol check is a bounded rerun recorded in docs/day6_findings.md; the empty tail is visible in every selected-symbol summary and in the system-event table.
- Revisit conditions: a day whose feed does contain locked or crossed rests would exercise those paths with real data and is worth rerunning against.
- Decision: any text that comes from the feed is validated at the boundary where it leaves the process. A ticker that becomes a filename must be a single safe path component; a ticker that becomes a CSV field must contain no delimiter or quote; any byte that becomes part of a JSON string must be escaped when it is outside printable ASCII. A symbol whose ticker fails is ineligible for selection and a summary row blanks it, rather than the value being rewritten silently.
- Date: 2026-07-27
- Status: accepted
- Context: the Day 6 adversarial review found three separate instances of the same omission, each one a regression against a guard the repository had already built and documented. D-018 restricts the command-line symbol list to a safe filename charset because a symbol becomes a filename; engine.hpp applies first locate wins when two locates carry the same ticker, because one ticker means one output file. The analysis layer rebuilt filenames and per-ticker output from feed bytes without inheriting either rule.
- What was wrong: (1) ticker_is_displayable accepted every printable non-space byte, so '/' and '.' passed. A Stock Directory message carrying "../../x" produced a selected symbol whose series file was written outside the output directory, truncating whatever was there, while the run exited 0 with an incomplete published tree and the manifest recorded the traversing path. A leading '/' discarded the requested output directory entirely. Reproduced with a crafted fixture. (2) Two locates carrying the same ticker were both selectable, and both wrote to one file, producing a blended series and a manifest listing the same path twice with different row counts. (3) The JSON escaper passed bytes outside printable ASCII through unchanged, so an unknown ITCH type byte that is a control character produced a manifest no parser accepts, again at exit 0. (4) The per-symbol summary wrote the directory's ticker bytes into a CSV field with no validation, so a comma would have shifted every later column.
- Alternatives considered: sanitizing the ticker into a safe filename (turns two different symbols into one name and hides the input); reusing D-018's charset from the command line (36 eligible tickers on the pinned day contain '=', '^', or '*', so that charset would drop real symbols and change published counts).
- Choice: reject at the eligibility predicate, deduplicate tickers in the one shared ranking step both selection paths now call, escape non-ASCII bytes in the shared JSON writer, and blank an unsafe ticker in the summary row. Two new statistical invariants make a regression take the refuse-to-publish path rather than republishing: selected tickers must be unique, and each must be a safe single path component.
- Consequences: none of the committed outputs change. No ticker among the 8,850 on the pinned day contains a path separator, a comma, or a quote, no two selected symbols share a ticker, and the committed manifest is already pure printable ASCII. The full-day analysis was rerun after the fixes and every committed file is byte-identical.
- Evidence or benchmark: SymbolSelection.RejectsTickersThatWouldEscapeTheOutputDirectory, SymbolSelection.RejectsTickersThatWouldBreakACsvRow, SymbolSelection.SelectsOneLocateWhenTwoShareATicker, and AnalysisManifest.StaysParseableWhenAnUnknownTypeByteIsNotPrintable in tests/analyze_test.cpp.
- Revisit conditions: any new output that embeds feed-derived text.
- Decision: record the review of the complete Day 6 diff and what was done with each confirmed finding, in the same form as D-029 did for Day 4.
- Date: 2026-07-27
- Status: accepted
- Method: six reviewers over distinct dimensions (statistical arithmetic, observer isolation, millisecond exactness, outputs and privacy, test quality, and documentation honesty), each finding then handed to a separate skeptic instructed to refute it by reading the code and the committed data. Twenty-four findings were raised and ten survived verification; several refutations were themselves substantiated by compiling the reviewer's scenario against the real headers.
- Findings fixed, published numbers first: (1) The regular-session message total was summed over the wrong second range: [34201, 57600] rather than [34200, 57599], giving 268,859,208 instead of 268,389,796. Fixed, and the report now states that a second-aligned sum cannot be exact because both session boundaries fall inside a second, and points at the exactly bounded per-message figure the manifest already publishes. (2) The published reconciliation count, 174,316, did not reproduce: the committed tree yields 174,323, because the run_analysis.sh pipeline verifies before summarize_findings.py writes findings_candidates.json, and that file adds seven checks. Both numbers and the reason are now stated. (3) D-049 cited tests/stats_engine_neutral_test.cpp, a file that was never created; the tests it described live in three other files. The citation now names them, and the bounded 10,000,000-frame real comparison it referred to is recorded in the Day 6 report with its actual result. (4) D-067 described a four-metric dense millisecond verifier. Only the message metric is implemented at any scale. The entry now describes what exists and why. (5) through (8) The four feed-derived text defects recorded in D-075.
- Test gaps closed: no test made a statistical invariant fail, so the refuse-to-publish path had no coverage at either layer; p90 and p99 were equal in every test distribution, so transposing them survived the suite; and nothing pinned which millisecond metric each argument fed. All three now have tests, including a CLI test for exit code 11.
- Findings recorded rather than fixed: (1) verify_analysis.py skips the percentile recomputation on a reduced series, so on the committed dataset the spread percentiles are checked in process and by the full-resolution rerun, not by the external verifier over the committed files. Moving its summary-to-manifest cross-check above that early exit is a script behavior change and belongs to a later day. (2) run_analysis.sh verifies before it generates the finding candidates, which is why the check count differs between the pipeline run and a manual rerun. Reordering it would change the number again, so the ordering is documented instead.
- Evidence or benchmark: every fix carries a test named in D-075 or above, the full-day analysis was rerun after the fixes with byte-identical results, and the 100-million-frame dual-engine validation was re-run green.
- Revisit conditions: none; historical record.
- Decision: the README and the release documentation present measured evidence first, state exactly what was measured and where, and make no claim the committed artifacts do not support. Every number shown is either generated from a committed artifact by a script in this repository or checked against one automatically.
- Date: 2026-07-27
- Status: accepted
- Context: product_spec.md sections 0, 9, and 15.2 set the standard: the deliverable is defensible evidence, repository polish is weak evidence on its own, and no number may be borrowed or estimated. Day 7 is presentation only, so the risk it introduces is overstatement rather than a wrong computation.
- The central claim, and the only one the repository makes: a single threaded C++20 Nasdaq TotalView-ITCH 5.0 replay and limit-order-book engine with safe binary framing, typed decoding and encoding, a correctness-first reference book, a separately designed optimized book, exact dual-engine validation over real data, controlled performance measurement, and reproducible market analysis.
- Claims that are permitted, because a committed artifact carries them: the frame and message counts of the pinned day, byte-exact round trips over the supported real messages, the Day 3 and Day 4 replay results, exact MapBook and FastBook equivalence over 100 million real frames, the provisional macOS speedup, the full-day analysis totals, the independent reconciliation count, and the three Day 6 findings with their scope and limitations.
- Claims that are forbidden regardless of how they are phrased: production-ready, exchange-certified, a trading strategy, a matching engine, low-latency trading infrastructure, lock-free, multithreaded, live-market capable, nanosecond accurate end to end, representative of all Nasdaq trading days, or faster than any named third-party system. Nothing may assert profitability, manipulation, market inefficiency, investor behavior, exchange causality, hidden-liquidity reconstruction, consolidated-market coverage, or a final Linux performance result.
- Benchmark labeling: every performance figure carries the provisional label and the platform, because the machine moved to battery power during the Day 5 run and the gate refused to call the results final (D-030, D-043). The reverted tuning experiment is presented, not hidden.
- Analysis labeling: every spread statistic states that it is time weighted and computed from the complete state-change series, while the committed per-symbol CSV files are a documented one-second reduction for inspection and charting (D-072).
- No AI authorship breakdown, usage percentage, prompt log, or development diary appears in the repository. The authorities stay what they have been since Day 0: the official protocol documents, the tests, the exact validation runs, the committed benchmark and analysis data, and the reproduction commands (product_spec.md section 15.5).
- Enforcement: scripts/check_docs.py fails when a link is broken, when a generated table is stale, when a README number disagrees with the committed source it claims, when a forbidden claim appears, when the provisional or reduced-series labels are missing, or when an em dash or an absolute path reaches tracked text. It runs in CI.
- Revisit conditions: a Linux reference benchmark would replace the provisional labeling with a stated platform result; nothing else.
- Decision: generated presentation tables live under docs/generated, diagram sources under docs/diagrams with rendered SVG under docs/images, and analysis charts stay under analysis/charts beside the data they are rendered from. The empty docs/img scaffold from Day 0 is removed.
- Date: 2026-07-27
- Status: accepted
- Context: product_spec.md section 3 sketched docs/img for committed PNGs, before the analysis directory existed. Day 6 wrote charts into analysis/charts because they are regenerated from analysis/results by one script and belong with it, which left docs/img empty.
- Alternatives considered: copying charts into docs/img for the README (two copies of the same bytes, and the copy would go stale); rendering diagrams into analysis/charts (mixes hand-authored diagrams with generated data charts).
- Choice: as stated, and recorded here as a deliberate deviation from the section 3 sketch rather than a drift.
- Diagram tooling: no Mermaid or Graphviz renderer exists on this machine and neither may become a project dependency, so scripts/render_diagrams.py emits both the Mermaid source and a standards-compliant SVG from one specification using the standard library alone. Committing both means a reader can render the source with any Mermaid tool and a viewer with no tooling still sees the diagram, and a test proves the committed files are what the script produces.
- Consequences: every presentation artifact in the repository is generated by a committed script from committed data, and check_docs.py fails when one is stale.
- Revisit conditions: none for v1.
- Decision: record the adversarial review of the Day 7 presentation layer and what was done with each confirmed finding, in the same form as D-029 and D-076.
- Date: 2026-07-27
- Status: accepted
- Method: eight reviewers over distinct reader lenses (C++ systems, microstructure, benchmark methodology, reproducibility, security and privacy, hiring manager, hostile skeptic, and documentation accessibility), each required to inspect the committed artifacts rather than the prose. Four lenses reported before the run stalled and was stopped; their 32 findings were then verified one by one against the artifacts by hand, which is how every fix below was justified. The four lenses that did not report are a gap in this review, recorded rather than glossed: reproducibility, security and privacy, hostile skeptic, and accessibility were covered by the separate clean-clone verification, the release safety sweep, and scripts/check_docs.py instead.
- Findings fixed, published claims first: (1) The README's opening sentence claimed the full day was reconstructed twice and proven to agree. The exact dual-engine comparison covers 100,000,000 of 282,229,684 frames and 8,840 of 8,841 active books. Corrected to state the measured scope. (2) The benchmark caveat said the machine "moved to" battery power. The environment capture recorded it already on battery at 26 percent and discharging. It also omitted the largest documented uncontrolled factor, a shared machine with heavily used swap. Both corrected. (3) The generated throughput table labeled its memory column "Peak RSS" while printing the maximum across passes, and the per-operation table labeled a column "Max" while printing the median of the three pass maxima. For FastBook add, that median is 2,092,417 ns against an observed maximum of 13,960,875 ns. Both columns now say what they are. (4) The before-and-after tuning table presented a percent-change column between two runs whose only source difference is the measurement harness, because the experiment was applied and reverted in between. The table now states that its column is between-run drift, and points at the paired alternating measurements in D-048 that actually decided the experiment. (5) The latency tails were published without the caveat that the empty probe's own maximum, 14,458 ns, exceeds every reported p99.9, so nothing in those columns can be attributed to book work. Added to the generated table and to the README. (6) Finding 3 was headlined "spreads are far wider outside regular hours", which names a window the analysis never measured: the ratio is full day over regular session, and the full day contains the regular session. Retitled and the containment stated. The finding now also says that one cent is the minimum quoting increment at these prices, so a one-cent median is the spread on its floor, and that zero locked and zero crossed time is closer to structural than empirical for a single venue's own displayed book. (7) Finding 1 did not say which messages "added orders" and "execution message" count. It now names them and says that replacements also place an order and that the day's trade messages are counted but not decoded. (8) Finding 2 quoted the closing second without saying that 916,875 of its 982,255 messages were deletions, and set its multiple against a baseline that includes the quiet extended hours. Both stated. Its millisecond sentence, which read as a sharper spike inside the same second, was wrong on both counts and now says what the data shows: the busiest millisecond is 35 seconds earlier, and the closing second is sustained rather than spiky. (9) The limitation "hidden and non-displayed liquidity is not in this feed" was false. Non-displayed liquidity is never quoted, so it never enters the reconstructed book, but its executions are reported by trade messages this project counts and does not decode. Corrected in the README, the Day 6 report, and the manifest generator, which required regenerating the analysis; every data file came back byte-identical. (10) "The two share BookContract and nothing else" was false: both also expose the canonical iteration that validation walks. The accurate claim, a shared public surface and no shared implementation, replaced it. (11) The README said validation attaches through a template seam, which only benchmarking and analysis do, and counted four commands where there are five. (12) The book-cost figures were quoted to the millisecond as differences of medians whose own passes span up to 14 percent, with no dispersion stated. (13) The tuning summary generalized a three-case microbenchmark improvement to all of them and called bounded paired replays full-file. (14) The architecture diagram fed bench and analyze from FastBook alone, though both accept either engine, and the FastBook slot diagram listed fields summing to 30 bytes while labeling the slot 32. (15) The headline "174,323 independent checks" did not say that 174,196 of them are two per-row assertions over the 87,098 committed series rows.
- Findings recorded rather than fixed: the Evidence and Correctness tables repeat four rows, which is deliberate for a reader who scans one and reads the other, and the quickstart sits below the evidence and the diagrams, which is the structure this presentation deliberately chose. A pointer to the quickstart now sits in the opening paragraph, the duplicated ASCII pipeline was removed, and the synthetic suite, which had appeared only as a badge, is now a row in the correctness table.
- Evidence or benchmark: every fix was verified against the artifact it concerns before being made, no measured value changed, the generated tables regenerate with identical numbers, and scripts/check_docs.py enforces the numeric claims from here on.
- Revisit conditions: none; historical record.