memory: add production-faithful LongMemEval benchmark

memory/data/README.md

@@ -1,9 +1,13 @@
-# LoCoMo Dataset
+# Memory Benchmark Datasets
 
-This directory contains the LoCoMo benchmark dataset for evaluating long-term
-conversational memory.
+This directory contains or references benchmark datasets for evaluating
+long-term conversational memory.
 
-## Download
+## LoCoMo Dataset
+
+LoCoMo data is stored in this directory.
+
+### Download
 
 Download the LoCoMo dataset from the official repository:
 
@@ -15,7 +19,7 @@ git clone https://github.com/snap-research/locomo.git
 cp locomo/data/locomo10/*.json ./
 ```
 
-## Dataset Format
+### Dataset Format
 
 The LoCoMo dataset contains long-term conversational data with the following
 structure:
@@ -52,14 +56,60 @@ structure:
 }
 ```
 
-## QA Categories
+### QA Categories
 
 - `single-hop`: Single-hop questions answerable from one conversation segment.
 - `multi-hop`: Multi-hop questions requiring multiple conversation segments.
 - `temporal`: Temporal reasoning questions involving time relationships.
 - `open-domain`: Open-domain questions requiring world knowledge.
 - `adversarial`: Adversarial questions designed to test robustness.
 
-## Reference
+### Reference
 
 [LoCoMo: Long-Context Conversational Memory Benchmark](https://arxiv.org/abs/2402.17753)
+
+## LongMemEval Dataset
+
+LongMemEval data is shared with the summary benchmark under
+`summary/data/longmemeval-cleaned/` instead of being duplicated in this
+directory.
+
+### Download
+
+Download LongMemEval from the official repository:
+
+```bash
+# Clone the LongMemEval repository.
+git clone https://github.com/xiaowu0162/LongMemEval.git
+
+# Place cleaned files under the shared benchmark data directory.
+mkdir -p ../../summary/data/longmemeval-cleaned
+cp LongMemEval/data/longmemeval_oracle.json ../../summary/data/longmemeval-cleaned/
+```
+
+### Dataset Format
+
+The LongMemEval dataset contains per-question haystack conversations:
+
+```json
+{
+  "question_id": "gpt4_2655b836",
+  "question_type": "temporal-reasoning",
+  "question": "What was the first issue...",
+  "question_date": "2023/04/10 (Mon) 23:07",
+  "answer": "GPS system not functioning correctly",
+  "answer_session_ids": ["answer_4be1b6b4_2"],
+  "haystack_dates": ["2023/04/10 (Mon) 14:47"],
+  "haystack_session_ids": ["answer_4be1b6b4_3"],
+  "haystack_sessions": [
+    [
+      {"role": "user", "content": "..."},
+      {"role": "assistant", "content": "..."}
+    ]
+  ]
+}
+```
+
+### Reference
+
+[LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://arxiv.org/abs/2410.10813)

memory/results/.gitignore

@@ -2,5 +2,6 @@
 *.json
 *.log
 *.pdf
+lme-*/
 tmp/
-tools/
+tools/

memory/results/README.md

@@ -6,8 +6,22 @@ This directory stores memory benchmark evaluation results.
 
 | File | Description |
 |------|-------------|
-| [REPORT.md](REPORT.md) | Full evaluation report (English) |
-| [REPORT.zh_CN.md](REPORT.zh_CN.md) | Full evaluation report (Chinese) |
+| [REPORT.md](REPORT.md) | Unified LoCoMo and LongMemEval evaluation report (English) |
+| [REPORT.zh_CN.md](REPORT.zh_CN.md) | Unified LoCoMo and LongMemEval evaluation report (Chinese) |
+
+## LongMemEval Memory Evaluation Summary
+
+The current production-style LongMemEval Oracle evaluation uses one fixed,
+already-observed 16-question development regression spanning all six types.
+Every arm replays all 183 selected user/assistant pairs, generates three
+independent answers per question, and judges every answer three times.
+Pgvector main scores 12/16 (36/48 answer replicates), self-hosted Mem0 OSS
+scores 14/16 (41/48), and the final pgvector candidate scores 16/16 (48/48).
+A compact-recovery ablation reduces candidate memory LLM tokens from 1,838,020
+to 1,790,001 while retaining quality, but does not reduce final memory count or
+wall-clock ingestion time. No unseen protocol-v2 holdout has been selected or
+run. See the unified reports for replay design, usage, bad cases, provenance,
+and limitations.
 
 ## LoCoMo Benchmark Evaluation Summary