bench: add answer prompt v2 and cmd/replay for offline evaluation

Introduces answer prompt v2, which relaxes the abstention rule and
resolves relative dates against source timestamps. This fixes two
failures that cost ~0.10 answerable Judge at the oracle.

Adds cmd/replay to re-answer questions from existing prediction dumps.
Since the answer prompt cannot affect retrieval, replaying against a
dump allows measuring prompt and model changes without a full re-run.

- Registers v1 and v2 answer prompts for reproducibility.
- Centralizes scoring logic in bench.Score to ensure parity between
  live runs, the oracle, and offline replays.
- Updates RESULTS.md with the stage decomposition and v2 findings.

Author: AccursedGalaxy

CHANGELOG.md

@@ -14,6 +14,32 @@ benchmark validating the retrieval and consolidation levers as a set) is the
 gate for cutting it — see `bench/RESULTS.md` and `PLAN-v2.md` §10.
 
 ### Added
+- **Temporal grounding: `Message.Timestamp` and `Fact.ObservedAt`.** A message
+  can now carry when it was said, and the fact extracted from it records that
+  time — distinct from `CreatedAt`, which stays ingestion time. The extractor's
+  date arithmetic is anchored on the conversation's own timestamp instead of on
+  today, so "I went yesterday" resolves against when it was said rather than when
+  it was ingested. Both fields are optional and default to the previous behavior
+  (fall back to the clock; `ObservedAt` stays zero), so no existing caller
+  changes. SQLite stores get an `observed_at` column, added by an idempotent
+  migration on `Open` — existing databases upgrade in place and their rows read
+  back with a zero `ObservedAt`, which is the truth: their source time was never
+  recorded.
+
+  `ObservedAt` is **say-time, not event-time**: for "I went yesterday" said on 8
+  May it is 8 May, and the event's own date (7 May) is what the extractor resolves
+  into the fact text. Do not answer "when did X happen" from the field directly.
+
+  Under `Consolidate`, an `UPDATE` takes the reconciling conversation's
+  observation time; if that conversation carried no timestamp, the fact keeps the
+  date it already had rather than being blanked — losing a known date is worse
+  than not learning a new one. `Store.Update` enforces this, so no caller can
+  erase a date by passing a partially-populated record.
+
+  This fixes a silent corruption. Without a grounding date, `gemini-2.5-flash-lite`
+  wrote *"Caroline attended an LGBTQ support group on 2026-07-11"* — the ingestion
+  date — for a May 2023 event, and nothing downstream could detect it. See
+  `bench/RESULTS.md`.
 - **Consolidation write strategy** (`WithStrategy(Consolidate)`): a second LLM
   call per `Add` reconciles newly extracted facts against existing ones —
   ADD / UPDATE / DELETE / NONE — so a changed fact replaces the stale one
@@ -47,6 +73,18 @@ gate for cutting it — see `bench/RESULTS.md` and `PLAN-v2.md` §10.
 - **Transient-failure retries** in the OpenAI-compatible client: network errors,
   429/5xx, and empty/garbled 2xx bodies are retried with exponential backoff (up
   to 5 attempts), so one gateway blip no longer aborts a long ingestion or bench.
+- **Benchmark answer prompt v2** (`cmd/bench -answer-version`, now the default).
+  The source oracle showed the answer stage lost more score than retrieval did:
+  v1 abstained on questions whose answer sat verbatim in the evidence, and echoed
+  relative dates ("last week") back instead of resolving them. v2 fixes both,
+  worth +0.097 answerable Judge at the oracle. v1 stays registered so past runs
+  remain reproducible, and an unknown version is now an error rather than a
+  silent fallback to the default.
+- **`cmd/replay`**: re-runs the answer step of a finished run against a different
+  answer prompt or answer model, reading its prediction dump and writing a new one
+  for `cmd/rescore` to compare. The answer prompt cannot affect retrieval, so
+  replaying a dump is exact, not approximate — an answer-stage change now costs
+  one call per question instead of a full re-ingest.
 
 ### Changed
 - File-backed SQLite stores open in WAL mode for better concurrent-read behavior.
@@ -58,6 +96,11 @@ gate for cutting it — see `bench/RESULTS.md` and `PLAN-v2.md` §10.
   README config table, and `examples/basic` updated to match. See
   `bench/RESULTS.md`.
 
+  **Superseded.** Those figures predate the adversarial-scoring fix, and the
+  recommendation itself no longer holds: flash's edge was almost entirely
+  temporal date normalization, which `Message.Timestamp` now supplies to any
+  extraction model for free. See `bench/RESULTS.md` for the current numbers.
+
 ### Fixed
 - Consolidation detects no-op `UPDATE`s against a target that was concurrently
   deleted, instead of reporting a phantom write.

README.md

@@ -63,6 +63,27 @@ func main() {
 
 A runnable version is in [`examples/basic`](./examples/basic).
 
+### Timestamp your messages
+
+`Message.Timestamp` is optional, and setting it is the cheapest accuracy win
+available:
+
+```go
+m.Add(ctx, []mneme.Message{
+	{Role: "user", Content: "I went to the support group yesterday.",
+		Timestamp: time.Date(2023, 5, 8, 13, 56, 0, 0, time.UTC)},
+}, scope)
+```
+
+It anchors the extractor's date arithmetic on **when the conversation happened**
+rather than on when you ingested it, and it lands on the resulting fact as
+`Fact.ObservedAt`. Leave it out and the pipeline falls back to its clock, which
+means "yesterday" resolves against today — so a conversation ingested months late
+produces facts with confidently wrong dates that nothing downstream can detect.
+
+`Fact.CreatedAt` is when the fact was written; `Fact.ObservedAt` is when it was
+said. They are different questions and the store now keeps both.
+
 ## Configuration (env)
 
 `mneme.New()` builds providers from the environment (override any of them with

bench/RESULTS.md

@@ -131,6 +131,64 @@ v2's behavior would have changed nothing upstream anyway (it cannot — retrieva
 answer prompt). And the adversarial delta was measured on the real pipeline dumps, not the
 oracle, because the oracle hands adversarial questions no evidence by construction.
 
+## Temporal grounding: `ObservedAt` (the fix the section above pointed at)
+
+The section above ended by saying facts throw away the source timestamp, and that giving them one
+was the highest-value memory change on the board. It was. `Message.Timestamp` now flows into
+`Fact.ObservedAt`, the extractor's OBSERVATION DATE is anchored on when the conversation happened
+rather than on today, and a dated fact renders as `(said on 8 May, 2023) …` to the answer model.
+
+Full re-run, flash-lite extraction, answer prompt v2:
+
+| run | answerable Judge | paired Δ vs baseline [95% CI] | abstains |
+|---|---|---|---|
+| additive · 3-small (v1 baseline) | 0.341 | — | 40.0% |
+| rawturns (v2) | 0.446 | +0.048 → see below | 30.6% |
+| additive · extract=flash (v1) | 0.398 | +0.057 [+0.024, +0.083] | 39.6% |
+| **additive · flash-lite + ObservedAt (v2)** | **0.426** | **+0.085 [+0.062, +0.109]** | **29.2%** |
+
+**The cheap extractor now beats the expensive one.** `flash-lite + ObservedAt` scores 0.426 against
+`extract_flash`'s 0.398 — a real regression for flash (−0.028 [−0.052, −0.005] paired) at 5× the
+per-fact cost. Almost all of flash's edge was doing relative-date arithmetic correctly at extraction
+time. The source timestamp supplies that to *any* extraction model, for free, and more reliably.
+
+The gain is where the mechanism predicts, and nowhere else:
+
+| category | baseline | + ObservedAt |
+|---|---|---|
+| **temporal** | 0.237 | **0.533** |
+| single_hop | 0.426 | 0.476 |
+| open_domain | 0.094 | 0.125 |
+| multi_hop | 0.291 | 0.259 |
+
+Temporal more than doubles, and clears flash extraction's 0.49 using the model flash extraction was
+brought in to replace. Note this is not the answer prompt doing the work: replaying v2 against the
+*untimestamped* additive dump was worth −0.004. v2 and `ObservedAt` are worth nothing apart and
++0.085 together — the prompt supplies the rule, the record supplies the date, and neither is any use
+without the other.
+
+### The hybrid store is now the measured conclusion, not a hypothesis
+
+Under the same answer prompt, the fact store and raw turns have converged — and they win different
+categories:
+
+| | rawturns (v2) | additive + ObservedAt (v2) |
+|---|---|---|
+| answerable | 0.446 | 0.426 (Δ +0.020 [−0.019, +0.051] — **indistinguishable**) |
+| **single_hop** | **0.555** | 0.476 |
+| **temporal** | 0.414 | **0.533** |
+| multi_hop | 0.262 | 0.259 |
+| open_domain | 0.125 | 0.125 |
+
+Raw turns keep the wording, so they win single-hop recall. Facts normalize the dates, so they win
+temporal. Neither dominates, the aggregate difference is noise, and the per-category split is a
+clean statement of what each representation is *for*. That is the hybrid store's case, made with
+numbers rather than argument: keep the episodes and the derived facts, retrieve over both, and
+expect roughly single-hop-from-episodes plus temporal-from-facts.
+
+Two caveats. The `-0.03` multi-hop dip is within noise but consistent across both v2 runs, and is
+worth a look before v2 is called free. And every number here is still ten conversations.
+
 ## What this settles (measured, not guessed)
 
 1. **The system abstains correctly: adversarial 0.96.** LoCoMo's adversarial questions are unanswerable traps, and mneme declines them almost every time. The answer prompt's explicit "say I don't know" instruction is doing its job. Worth keeping, but it is not a lever, and averaging it into a headline meant to guide work only obscures the rows that differ.
@@ -149,11 +207,11 @@ oracle, because the oracle hands adversarial questions no evidence by constructi
 
 **Recommendation (v1 answer prompt): use `gemini-2.5-flash` for extraction, not flash-lite.** Roughly 5× the per-fact cost ($0.0017 vs $0.00034) for +0.057 [+0.026, +0.084] answerable Judge, most of it temporal. Boosters stay opt-in and undefaulted, pending a combined run and more conversations.
 
-> **This recommendation does not survive answer prompt v2.** Almost all of flash's edge was
-> temporal date normalization, and v2 gets that from the source timestamp instead — for free, and
-> more reliably. Under v2, rawturns (0.451, no extraction model at all) beats flash extraction
-> (0.387). If you are paying 5× for extraction, pay it for something the raw turn cannot give you.
-> See [The answer stage](#the-answer-stage-two-prompt-bugs-worth-010).
+> **Superseded — do not follow this.** Almost all of flash's edge was temporal date normalization,
+> which `Message.Timestamp`/`Fact.ObservedAt` now supply to any extraction model for free. Measured
+> head to head, **flash-lite + ObservedAt (0.426) beats flash extraction (0.398)** — flash is now a
+> real regression at 5× the per-fact cost. Use flash-lite and timestamp your messages. See
+> [Temporal grounding](#temporal-grounding-observedat-the-fix-the-section-above-pointed-at).
 
 ## Extraction-model A/B (eval fixtures)
 
@@ -176,25 +234,23 @@ the first item in.
 1. ~~The answer stage, not extraction.~~ **Done: answer prompt v2**, worth +0.097 at the oracle
    and +0.063 on rawturns. It also raised the ceiling to 0.642, which *re-opens* memory headroom
    rather than closing it.
-2. **Persist source timestamps on facts.** This is now the highest-value memory change on the
-   board, and it is small. A fact is text + hash + scope + ingest time, so the v2 date rule has
-   nothing to bite on and flash-lite's ingest-date corruption has nothing to correct it. Give
-   `types.Fact` an `ObservedAt` (the source turn's timestamp) and render it into the memory block
-   the way rawturns already does. This is the cheap half of structured attribution — do it before
-   subjects, predicates, and validity intervals.
-3. **A hybrid store, not a better extractor.** Raw turns beat extracted facts on aggregate, for
-   free, and the gap *widened* under v2 (0.451 vs 0.387). Extraction still earns its keep only
-   where it derives something the turn does not carry. Keep the episodes *and* the derived facts,
-   and retrieve over both.
-4. **A full v2 re-run of the matrix.** Every row above is a v1 run. The rankings between rows are
-   unlikely to move (v2 does not touch retrieval), but the headline numbers are now stale by
-   construction and the boosters × flash combination is still unrun.
-5. **Extraction recall on implied and causal facts (tk #11)** drops further down. The raw-turn
-   result keeps suggesting the fact representation itself, not its recall, is the weaker part.
-6. **open_domain (0.06–0.17) is the weakest category everywhere,** including the oracle, which
-   manages 0.17 under v2 (up from 0.06). A category the ceiling itself cannot score is a
-   question-format or judge problem, not a memory one — and the judge is still unvalidated
-   against human labels, which is the one piece of the harness nothing else can backstop.
+2. ~~Persist source timestamps on facts.~~ **Done: `Fact.ObservedAt`**, worth +0.085 answerable and
+   temporal 0.24 → 0.53. It also killed the flash-extraction recommendation.
+3. **Build the hybrid store.** No longer a hypothesis: under the same answer prompt, raw turns and
+   dated facts are statistically tied on aggregate and split the categories cleanly — episodes win
+   single-hop (0.555 vs 0.476), facts win temporal (0.533 vs 0.414). Keep both, retrieve over both.
+   This is the next real build, and the measurement says what to expect from it.
+4. **A full v2 + ObservedAt re-run of the matrix.** The lever matrix is all v1, untimestamped runs.
+   Consolidation, the bigger embedder, and the boosters have never been measured on a store whose
+   facts carry dates, and the boosters × extraction combination is still unrun.
+5. **Validate the judge against human labels.** Now the largest un-derisked thing in the harness.
+   Everything above is steered on a binary LLM judge that has never been checked against a person,
+   on a benchmark where a scoring bug already cost a full paid re-run. A stratified hand-labelled
+   sample of ~100 questions would put an error bar on every number in this file.
+6. **open_domain (0.09–0.17) is the weakest category everywhere,** including the oracle, which
+   manages only 0.17. A category the ceiling itself cannot score is a question-format or judge
+   problem, not a memory one — and #5 is how you find out which.
+7. **Extraction recall on implied and causal facts (tk #11)** drops further down again.
 
 ## Scoring history
 

bench/observed_test.go

@@ -0,0 +1,115 @@
+package bench
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/AccursedGalaxy/mneme"
+)
+
+func TestParseSessionDate(t *testing.T) {
+	got, ok := parseSessionDate("1:56 pm on 8 May, 2023")
+	if !ok {
+		t.Fatal("LoCoMo's session date format must parse; it is every fact's ObservedAt")
+	}
+	want := time.Date(2023, 5, 8, 13, 56, 0, 0, time.UTC)
+	if !got.Equal(want) {
+		t.Errorf("got %v, want %v", got, want)
+	}
+
+	// A date we cannot parse must stay zero ("unknown"). Defaulting it would
+	// stamp every fact in the session with a confidently wrong timestamp, which
+	// is worse than carrying none: nothing downstream could detect it.
+	if got, ok := parseSessionDate("sometime last spring"); ok || !got.IsZero() {
+		t.Errorf("an unparseable date must be zero/false, got %v/%v", got, ok)
+	}
+}
+
+func TestIngestMessagesStampsEveryTurn(t *testing.T) {
+	sess := Session{
+		Date: "1:56 pm on 8 May, 2023",
+		Messages: []mneme.Message{
+			{Role: "user", Name: "Caroline", Content: "I went to a support group yesterday."},
+			{Role: "user", Name: "Melanie", Content: "That's great!"},
+		},
+	}
+	msgs := ingestMessages(sess)
+	if len(msgs) != 3 {
+		t.Fatalf("want the dated note + 2 turns, got %d", len(msgs))
+	}
+	want := time.Date(2023, 5, 8, 13, 56, 0, 0, time.UTC)
+	for i, m := range msgs {
+		if !m.Timestamp.Equal(want) {
+			t.Errorf("message %d timestamp = %v, want %v", i, m.Timestamp, want)
+		}
+	}
+}
+
+// The whole point of ObservedAt is that it reaches the answer model. A fact that
+// knows when it was said must say so, or answer prompt v2's relative-date rule
+// has nothing to resolve against and the store is no better off than before.
+func TestBuildAnswerUserRendersObservedAt(t *testing.T) {
+	at := time.Date(2023, 5, 8, 13, 56, 0, 0, time.UTC)
+	got := buildAnswerUser("When?", []mneme.Fact{
+		{Text: "Caroline attended a support group.", ObservedAt: at},
+		{Text: "Melanie paints."}, // no timestamp: renders bare, as before
+	})
+	if !strings.Contains(got, "(said on 8 May, 2023) Caroline attended a support group.") {
+		t.Errorf("a dated fact must render its date, got:\n%s", got)
+	}
+	if !strings.Contains(got, "- Melanie paints.\n") {
+		t.Errorf("an undated fact must render bare, got:\n%s", got)
+	}
+}
+
+// A session date the layouts cannot parse must not stamp the zero time over
+// timestamps the messages already carry. Overwriting a real timestamp with
+// "unknown" is the same silent corruption the mechanism exists to prevent.
+func TestIngestMessagesKeepsExistingTimestampsWhenDateUnparseable(t *testing.T) {
+	own := time.Date(2023, 5, 8, 13, 56, 0, 0, time.UTC)
+	sess := Session{
+		Date: "sometime last spring", // matches no layout
+		Messages: []mneme.Message{
+			{Role: "user", Content: "I went to a support group yesterday.", Timestamp: own},
+		},
+	}
+	msgs := ingestMessages(sess)
+	if len(msgs) != 2 {
+		t.Fatalf("want the prose note + 1 turn, got %d", len(msgs))
+	}
+	if !msgs[1].Timestamp.Equal(own) {
+		t.Errorf("an unparseable session date must leave the message's own timestamp alone, got %v want %v",
+			msgs[1].Timestamp, own)
+	}
+}
+
+// An unparsed session date is invisible in the score — it just reads as "temporal
+// did not improve" — so the harness must count it and be able to say so.
+func TestUnparsedSessionDatesIsCounted(t *testing.T) {
+	before := UnparsedSessionDates()
+	ingestMessages(Session{
+		Date:     "sometime last spring",
+		Messages: []mneme.Message{{Role: "user", Content: "hi"}},
+	})
+	if got := UnparsedSessionDates(); got != before+1 {
+		t.Errorf("an unparseable session date must be counted: %d -> %d", before, got)
+	}
+}
+
+// LongMemEval's published haystack_dates carry a weekday and time; the ISO-only
+// assumption would have silently dropped every one of them.
+func TestParseSessionDateCoversLongMemEvalShapes(t *testing.T) {
+	for _, tc := range []struct {
+		in   string
+		want time.Time
+	}{
+		{"2023/05/20 (Sat) 02:21", time.Date(2023, 5, 20, 2, 21, 0, 0, time.UTC)},
+		{"2023-05-08", time.Date(2023, 5, 8, 0, 0, 0, 0, time.UTC)},
+	} {
+		got, ok := parseSessionDate(tc.in)
+		if !ok || !got.Equal(tc.want) {
+			t.Errorf("parseSessionDate(%q) = %v/%v, want %v/true", tc.in, got, ok, tc.want)
+		}
+	}
+}

bench/qa.go

@@ -93,6 +93,21 @@ func answerSystem(version string) string {
 	return answerPrompts[DefaultAnswerVersion]
 }
 
+// RenderFact is how one memory is presented to the answer model: its text,
+// prefixed with the date it was said when the fact knows one.
+//
+// It is exported and used for the prediction dump as well as the prompt, so the
+// dump records the memory string the model actually saw. That equivalence is what
+// makes cmd/replay exact rather than approximate — a dump that stored bare
+// f.Text would replay without the dates and quietly under-report every run that
+// depends on them.
+func RenderFact(f mneme.Fact) string {
+	if f.ObservedAt.IsZero() {
+		return f.Text
+	}
+	return fmt.Sprintf("(said on %s) %s", f.ObservedAt.Format("2 January, 2006"), f.Text)
+}
+
 // Answer is the retrieve→answer step's second half: it feeds the retrieved
 // facts and the question to the answer LLM under the versioned QA prompt and
 // returns the model's answer. With no facts it still asks (the prompt makes the
@@ -108,14 +123,22 @@ func Answer(ctx context.Context, llm provider.LLM, version, question string, fac
 
 // buildAnswerUser renders the MEMORIES block and the QUESTION. Facts are listed
 // highest-scored first (Search already returns them in that order).
+//
+// A fact that knows when it was observed is rendered with that date in front of
+// it, which is what gives answer prompt v2's relative-date rule something to
+// resolve against. Raw turns have always carried their timestamp inline (that is
+// most of why rawturns beat the fact store on temporal questions); a fact carries
+// it in ObservedAt, and it is only useful if it reaches the answer model.
+//
+// Facts with no ObservedAt render bare, exactly as before.
 func buildAnswerUser(question string, facts []mneme.Fact) string {
 	var b strings.Builder
 	b.WriteString("MEMORIES:\n")
 	if len(facts) == 0 {
 		b.WriteString("(none)\n")
 	} else {
 		for _, f := range facts {
-			fmt.Fprintf(&b, "- %s\n", f.Text)
+			fmt.Fprintf(&b, "- %s\n", RenderFact(f))
 		}
 	}
 	b.WriteString("\nQUESTION: ")