Summary

• add a production-faithful LongMemEval runner for pgvector main, pgvector candidate, and self-hosted Mem0 OSS
• replay sessions chronologically and trigger extraction after every user/assistant pair
• isolate every question and run while preserving the source observation date
• add stratified sampling, targeted replay, fixed-memory retrieval refresh, resumable answering and judging, bad-case analysis, and candidate comparison
• redact answer, evidence, memory, retrieval, and metric content from blind console and per-case logs while retaining sealed raw artifacts and operational usage
• bind formal runs to a versioned preregistration manifest that verifies selection, dataset, protocol, exclusions, case metadata, and benchmark revision before provider setup
• record extraction operations, persistence diffs, retrieval hits, evidence recall, answers, errors, timings, token/cache/embedding usage, sanitized runtime configuration, and exact build provenance
• keep memory lineage pair-scoped so an answer-bearing extraction miss cannot mark a later unrelated memory as answer evidence
• allow saved retrieval results to migrate only the answer/judge contract while preserving and recording the source protocol digest
• allow answer-disabled retrieval-only artifacts to be re-answered without relaxing replay, retrieval, model, or selection provenance
• aggregate independent answer and judge replicates into a machine-verifiable three-arm quality, category, error, usage, and cost gate
• keep LoCoMo and LongMemEval logic in dataset-specific files and maintain unified English/Chinese reports

Fairness and diagnostics

All three arms receive the same dated user/assistant pairs, answer model and prompt, semantic judge protocol, Top-K 30 search limit, and isolated identity scopes. Pgvector uses the production MemoryService.EnqueueAutoMemoryJob path and waits for each job before replay continues. Self-hosted Mem0 uses its public OSS API and configured open-source update behavior. Provider calls whose responses omit usage remain counted through usage_missing_calls instead of being treated as zero cost.

The benchmark pins both the trpc-agent-go root module and independently versioned memory/pgvector module to the current 969fb16a918d candidate. Manifests record question IDs, replay counts, model variants, prompt versions, dependency replacements, table identities, cache hashes, snapshot hashes, source revisions, image digests, and usage coverage.

Selection-only mode writes the full versioned protocol payload and digest. A preregistered run rejects ad hoc filters, resampling, truncation, dataset or protocol drift, exclusion drift, case-metadata drift, dirty builds, and benchmark-revision drift before model initialization. Re-answering may replace only the answer/judge fields and records the original protocol digest; retrieval and selection changes still fail closed.

Traces separate extraction misses, reconciliation or persistence losses, retrieval misses, answer failures, and judge failures. Pair-scoped provenance prevents later unrelated memory from being credited for an answer-bearing pair that produced no memory. Formal answer replicates use empty independent answer and judge ledgers while sharing immutable ingestion and retrieval snapshots.

Current evidence

The frozen protocol-v2 baseline uses one fixed, already-observed 16-question development set spanning all six LongMemEval types. Each arm ingested all 183 dated conversation pairs with no truncation, Top-K 30, three independent answers per question, and three independent judge votes per answer.

The candidate baseline has no category deficit. Main's stable misses include two assistant-result cases, one preference-history case, and one answer-context case where the required memory was already stored and retrieved. Mem0's stable failures expose assistant-result provenance and retrieval-context selection issues; its additional unstable case scores 2/3.

The current private assistant-result prompt/schema compaction reduces dev16 memory tokens from 1,790,001 to 1,686,365 (-5.79%) and ingest time by 1.97%, while embedding tokens rise 5.68% and memories rise from 416 to 421. Its preregistered formal promotion remains rejected on the exact call bound (208 versus 207). A separate non-promotional diagnostic retains 48/48 correct answers and 144/144 valid judge votes.

Independent LoCoMo checks use fresh isolated parent/compact stores and three fixed-memory QA repeats. Extraction prompt tokens fall 5.90%, total extraction tokens fall 4.17%, and mean F1 is 0.6100 for parent versus 0.7320 for compact. The synthetic human-to-assistant role mapping limits this to cross-dataset cost and regression evidence.

Exact digests and the integrated interpretation are in the English report and Chinese report. No unseen blind holdout has been selected or run.

Agent dependency

• root module: github.com/Rememorio/trpc-agent-go@v0.0.0-20260721100707-969fb16a918d
• pgvector module: github.com/Rememorio/trpc-agent-go/memory/pgvector@v0.0.0-20260721100707-969fb16a918d
• upstream draft: memory: improve automatic memory extraction and retrieval trpc-agent-go#2196

Validation

• go test ./... in memory/trpc-agent-go-impl
• prior full validation: go test ./... -count=1, go test -race ./... -count=1, and go vet ./...
• synthetic three-arm gate tests covering retrieval drift, cache-ledger isolation, missing usage, invalid vote counts, malformed manifests, and resource-bound failures
• clean-revision runner smoke covering full-haystack Top-K 30 preregistration and pre-provider rejection of protocol, exclusion, and case-set drift
• self-hosted Mem0 add, search, answer, and judge smoke with complete provider usage
• audited protocol-v2 baseline, assistant-prompt cost/quality diagnostics, and LoCoMo artifacts with immutable snapshots and verified checksums