memory: add production-faithful LongMemEval benchmark

[Rememorio]

Summary

• add a production-faithful LongMemEval runner for pgvector main, pgvector candidate, and self-hosted Mem0 OSS
• replay sessions chronologically and trigger extraction after every user/assistant pair
• isolate every question and run while preserving the source observation date
• add stratified sampling, targeted replay, fixed-memory retrieval refresh, resumable answering and judging, bad-case analysis, and candidate comparison
• bind formal runs to a versioned preregistration manifest that verifies the exact selection, dataset, protocol payload, exclusion set, case metadata, and benchmark revision before provider setup
• record extraction operations, persistence diffs, retrieval hits, evidence recall, answers, errors, timings, token/cache/embedding usage, sanitized runtime configuration, and exact build provenance
• keep memory lineage pair-scoped so an answer-bearing extraction miss cannot mark a later unrelated memory as answer evidence
• aggregate independent answer and judge replicates into a machine-verifiable three-arm quality, category, error, usage, and cost promotion gate
• keep LoCoMo and LongMemEval logic in dataset-specific files and share protocol-level evaluation helpers

Fairness and diagnostics

All three arms receive the same dated user/assistant pairs, answer model and prompt, semantic judge protocol, Top-K 30 search limit, and isolated identity scopes. Pgvector uses the production MemoryService.EnqueueAutoMemoryJob path and waits for each job before replay continues. Self-hosted Mem0 uses its public OSS API and configured open-source update behavior. Provider calls whose responses omit usage remain counted through usage_missing_calls instead of being treated as zero-cost calls.

The benchmark pins both the trpc-agent-go root module and independently versioned memory/pgvector module to the evaluated 0797067f4074 candidate. Manifests record question IDs, replay counts, model variants, prompt versions, dependency replacements, table identities, cache hashes, snapshot hashes, source revisions, image digests, and usage coverage.

Selection-only mode writes the full versioned protocol payload and digest. A preregistered run rejects ad hoc filters, resampling, truncation, dataset or protocol drift, exclusion-set drift, case-metadata drift, dirty builds, and benchmark-revision drift before model or provider initialization. Frozen exclusions are validated against the active dataset and represented by a canonical count and digest without retaining a local path.

Traces separate extraction misses, reconciliation or persistence losses, retrieval misses, answer failures, and judge failures. Pair-scoped provenance prevents a later unrelated memory from being credited for an answer-bearing pair that produced no memory. Recovery traces retain initial and recovered answers, the selected source, and whether recovery was applied. Formal answer replicates use empty, independent answer and judge cache ledgers while sharing immutable ingestion and retrieval snapshots.

Current evidence

The formal protocol-v2 run uses one fixed, already observed 16-question development set spanning all six LongMemEval types. Each arm ingested the complete selected haystack: 183 dated conversation pairs, no session or pair truncation, Top-K 30, three independent answers per question, and three independent judge votes per answer.

Arm	Primary	Majority	Correct answer replicates	Unstable cases	Pipeline errors
pgvector upstream main	12/16	12/16	36/48	0	0
self-hosted Mem0 OSS	14/16	14/16	41/48	1	0
pgvector candidate	16/16	16/16	48/48	0	0

The candidate had no category deficit on this set: knowledge-update 2/2, multi-session 4/4, assistant 2/2, preference 2/2, user 3/3, and temporal 3/3. Main's stable misses include two assistant-result cases, one preference-history case, and one answer-context case where the required memory was already stored and retrieved. Mem0's stable failures expose assistant-result provenance and retrieval-context selection issues; its additional unstable case scored 2/3. These distinctions are retained in the traces so answer behavior is not mislabeled as extraction or retrieval quality.

The candidate passed the preregistered resource bounds relative to main, but it is not a cost improvement: memory-layer LLM tokens were 1.547x, embedding tokens 1.682x, persisted memories 2.715x, and ingestion time increased 33.5%. Relative to Mem0, candidate memory-layer LLM tokens increased 4.2%, embedding tokens decreased 68.4%, persisted memories decreased 15.5%, ingestion time increased 37.1%, and search time decreased 28.8%. Answer and judge usage is reported separately because cache behavior makes it unsuitable for a memory-layer cost comparison.

An immutable 10-question LoCoMo regression sample, stratified as two questions per category with three QA repeats, scored 0.6825 mean F1 for candidate versus 0.7107 for the previous focused-passage control. The -0.0282 delta passes the preregistered -0.05 bound but is reported as bounded regression, not an improvement.

The formal run passed checksum, isolation, provenance, usage-completeness, replication, category, error, and cost audits. Exact digests and integrated interpretation are in the English report and Chinese report. Earlier seed48 and seed137 experiments remain only as historical protocol-v1 diagnostics. No new unseen blind holdout has been run or selected for this candidate.

Agent dependency

• root module: github.com/Rememorio/trpc-agent-go@v0.0.0-20260720212903-0797067f4074
• pgvector module: github.com/Rememorio/trpc-agent-go/memory/pgvector@v0.0.0-20260720212903-0797067f4074
• upstream draft: memory: improve automatic memory extraction and retrieval trpc-agent-go#2196

Validation

• go test ./... -count=1 in memory/trpc-agent-go-impl
• go test -race ./... and go vet ./... in memory/trpc-agent-go-impl
• synthetic three-arm gate tests covering retrieval drift, cache-ledger isolation, missing usage, invalid vote counts, malformed manifests, and resource-bound failures
• clean-revision runner smoke covering full-haystack Top-K 30 preregistration and pre-provider rejection of protocol, exclusion, and case-set drift
• self-hosted Mem0 add, search, answer, and judge smoke with complete provider usage
• audited protocol-v2 dev16 and LoCoMo artifacts with immutable snapshots, exact provenance, atomic finalization, and verified checksums