Merge pull request #61 from vektori-ai/exp/locomo-bench

Exp/locomo bench

Author: laxmanclo

README.md

@@ -53,18 +53,22 @@ Search hits Facts, graph discovers Episodes, traces back to source Sentences. SQ
 
 ## Benchmarks
 
-Tested on long-horizon memory benchmarks — hundreds of turns, real user details buried deep in history.
+> **What is F1?** F1 is the harmonic mean of precision (what fraction of the model's answer was correct) and recall (what fraction of the correct answer the model actually covered). A high F1 means the answer is both accurate and complete. Most memory system vendors report LLM-judge accuracy instead — a separate LLM scores each answer as correct or not, which is easier to run but less granular.
 
-| Benchmark | Vektori | Mem0 | Zep | Supermemory | Letta |
-|-----------|---------|------|-----|-------------|-------|
-| LoCoMo | **66%** | 66% | 58%† | ~70% | ~83% |
-| LongMemEval-S | **73%** | — | 64% | 85% | — |
+Tested on long-horizon memory benchmarks — hundreds of turns, real user details buried deep in history.
 
-†Zep's self-reported score is 75%; independently re-evaluated at 58%. Scores across systems are not always directly comparable — model choice (GPT-4o vs GPT-4.1-mini vs local) significantly affects results. 
+| System | LoCoMo | LongMemEval-S | DMR | F1 avg (LoCoMo)† | Search p95 | Total p95 |
+|--------|--------|---------------|-----|-----------------|-----------|----------|
+| Vektori | 66% | 73% | — | — | — | — |
+| Mem0 | 66.88% | — | — | 41.0 | 0.200s | 1.440s |
+| Zep | 75.14%‡ | 71.2% | 94.8% | — | 0.778s | 2.926s |
+| Supermemory | — | 81.6% | — | — | — | — |
+| Letta | 74.0% | — | — | — | — | — |
 
-We used gemini-2.5-flash-lite because of token cost, better models imporve accuracy a lot. Benchmarks at L1 level
+†F1 = harmonic mean of precision (how much of the answer was correct) and recall (how much of the correct answer was covered). Only Mem0 publishes token-level F1; 41.0 is the average across single-hop 38.72, multi-hop 28.64, open-domain 47.65, temporal 48.93.  
+‡Zep self-reported; Mem0's paper measured Zep at 65.99% on the same run. Latency from Mem0's paper. Model choice significantly shifts all scores — we used gemini-2.5-flash-lite for cost.
 
-On LoCoMo and longmemEval, **the retrieved context contains the answer in 95% of questions** — the gap to 66% is a synthesis problem, not a retrieval one. Actively working on closing it, exploring RL.
+On LoCoMo and LongMemEval, **the retrieved context contains the answer in 95% of questions** — the gap to 66% is a synthesis problem, not a retrieval one. Actively working on closing it, exploring RL.
 
 Still improving — PRs and evals welcome. Run your own: [`/benchmarks`](benchmarks/)
 

benchmarks/locomo/locomo_judge.py

@@ -24,7 +24,7 @@
 logger = logging.getLogger("locomo_judge")
 
 VALID_VERDICTS = {"CORRECT", "PARTIALLY_CORRECT", "WRONG", "ABSTAINED"}
-DEFAULT_JUDGE_MODEL = "gemini:gemini-2.5-flash-lite"
+DEFAULT_JUDGE_MODEL = "gemini:gemini-3-flash-preview"
 
 ABSTENTION_PHRASES = (
     "i don't have that information",
@@ -72,11 +72,14 @@
 - For date answers, accept equivalent formats. These are always CORRECT:
   "2023-10-21" = "21 October 2023" = "October 21, 2023"
   "2022-11-10" = "10 November 2022" = "November 10, 2022"
-- Absolute dates that equal a relative expression are CORRECT. Verify the arithmetic before marking wrong:
-  "July 14, 2023" = "The Friday before 15 July 2023" (July 14 is a Friday)
-  "11 August 2023" = "The Friday before 14 August 2023" (August 11 is a Friday)
+- Absolute dates that equal a relative expression are CORRECT. To check: compute the calendar date the relative expression describes, then compare. If they land on the same day (±0), it is CORRECT. Show your arithmetic before concluding.
+  "July 14, 2023" = "The Friday before 15 July 2023" (July 14 is a Friday ✓)
+  "11 August 2023" = "The Friday before 14 August 2023" (August 11 is a Friday ✓)
+  "2023-07-17" = "The weekend before 17 July 2023" (July 17 is itself a Monday — weekend is July 15/16; mark accordingly)
   A model that gives the computed absolute date when the expected uses relative phrasing is CORRECT, not wrong.
-- Extra correct details do not make an answer wrong.
+- If the MODEL ANSWER is short (under 25 words) — a bare name, date, quantity, or short phrase — do NOT invent reasons it is wrong. Only ask: does this core fact match the expected answer? If yes → CORRECT. Do not fabricate attribution errors, entity confusions, or "context doesn't mention this" claims for a short answer. A short answer that matches the expected answer is CORRECT.
+- If the EXPECTED ANSWER is short (5 words or fewer), check whether the model answer CONTAINS that core fact, regardless of surrounding phrasing. "Caroline is single, having experienced a tough breakup" is CORRECT for expected="Single". "John explored the Pacific Northwest on a road trip" is CORRECT for expected="Pacific Northwest". Extra phrasing around the core fact does NOT make the answer PARTIALLY_CORRECT.
+- Extra correct details do not make an answer wrong or partially correct.
 
 JSON schema:
 {{"verdict": "CORRECT|PARTIALLY_CORRECT|WRONG|ABSTAINED", "context_has_answer": true|false, "failure_mode": null|"QA_FAILURE"|"RETRIEVAL_FAILURE", "explanation": "one sentence"}}"""
@@ -90,6 +93,7 @@ class JudgeConfig:
     seed: int = 42
     output_dir: str = "benchmark_results"
     qids: list[str] = field(default_factory=list)
+    concurrency: int = 5
 
 
 @dataclass
@@ -191,12 +195,11 @@ def load_entries(full_results_path: Path, qids: list[str], n: int, seed: int) ->
     return rng.sample(entries, n)
 
 
-async def evaluate_entry(entry: dict[str, Any], judge_model: str) -> dict[str, Any]:
-    llm = create_llm(judge_model)
+async def evaluate_entry(entry: dict[str, Any], llm: Any) -> dict[str, Any]:
     prompt = build_prompt(entry)
     t0 = time.perf_counter()
     try:
-        raw = await llm.generate(prompt, max_tokens=350)
+        raw = await llm.generate(prompt, max_tokens=400)
         latency_ms = (time.perf_counter() - t0) * 1000
         verdict = parse_verdict(raw, latency_ms)
     except Exception as exc:
@@ -251,15 +254,20 @@ def summarize(results: list[dict[str, Any]]) -> dict[str, Any]:
         if result["context_has_answer"]:
             by_type[qtype]["ctx_ok"] += 1
 
+    answered = total - abstained
     return {
         "total": total,
         "correct": correct,
         "partially_correct": partial,
         "wrong": wrong,
         "abstained": abstained,
+        "answered": answered,
         "correct_rate": round(correct / total, 4) if total else 0.0,
         "combined_rate": round((correct + partial) / total, 4) if total else 0.0,
+        "answer_precision": round(correct / answered, 4) if answered else 0.0,
+        "answer_recall": round(correct / total, 4) if total else 0.0,
         "context_has_answer": context_has_answer,
+        "retrieval_recall": round(context_has_answer / total, 4) if total else 0.0,
         "context_has_answer_rate": round(context_has_answer / total, 4) if total else 0.0,
         "qa_failure": qa_failure,
         "retrieval_failure": retrieval_failure,
@@ -270,17 +278,25 @@ def summarize(results: list[dict[str, Any]]) -> dict[str, Any]:
 
 async def run(config: JudgeConfig) -> dict[str, Any]:
     entries = load_entries(Path(config.full_results), config.qids, config.n, config.seed)
-    print(f"Judging {len(entries)} LoCoMo answers with {config.judge_model}")
+    total = len(entries)
+    print(f"Judging {total} LoCoMo answers with {config.judge_model} (concurrency={config.concurrency})")
 
-    results: list[dict[str, Any]] = []
-    for idx, entry in enumerate(entries, 1):
-        result = await evaluate_entry(entry, config.judge_model)
-        results.append(result)
+    llm = create_llm(config.judge_model)
+    sem = asyncio.Semaphore(config.concurrency)
+    counter = {"n": 0}
+
+    async def judge_one(entry: dict[str, Any]) -> dict[str, Any]:
+        async with sem:
+            result = await evaluate_entry(entry, llm)
+        counter["n"] += 1
         print(
-            f"[{idx}/{len(entries)}] {result.get('question_id')} "
+            f"[{counter['n']}/{total}] {result.get('question_id')} "
             f"{result['verdict']} ctx={result['context_has_answer']} "
             f"failure={result['failure_mode']}"
         )
+        return result
+
+    results: list[dict[str, Any]] = await asyncio.gather(*[judge_one(e) for e in entries])
 
     summary = summarize(results)
     output = {"config": asdict(config), "summary": summary, "results": results}
@@ -317,6 +333,8 @@ def main() -> None:
     parser.add_argument("--output-dir", default="benchmark_results")
     parser.add_argument("--qids", default="", help="Comma-separated question IDs")
     parser.add_argument("--qids-file", default="", help="One question ID per line")
+    parser.add_argument("--concurrency", type=int, default=5,
+                        help="Max concurrent judge requests (default 5; lower if hitting RPM limits)")
     args = parser.parse_args()
 
     try:
@@ -331,6 +349,7 @@ def main() -> None:
         seed=args.seed,
         output_dir=args.output_dir,
         qids=qids,
+        concurrency=args.concurrency,
     )
     asyncio.run(run(config))
 

benchmarks/locomo/locomo_runner.py

@@ -74,7 +74,7 @@ class LoCoMoConfig:
 
     # Retrieval
     retrieval_depth: str = "l1"
-    top_k: int = 10
+    top_k: int = 15
     context_window: int = 3
     enable_retrieval_gate: bool = False
 
@@ -549,10 +549,18 @@ def _avg(vals: list[float]) -> float | None:
         qa_vals = _collect("qa_ms")
         total_vals = _collect("total_question_ms")
 
+        def _p95(vals: list[float]) -> float | None:
+            if not vals:
+                return None
+            return round(sorted(vals)[int(len(vals) * 0.95)], 1)
+
         metrics["latency_ms"] = {
             "retrieval_avg": _avg(retrieval_vals),
+            "retrieval_p95": _p95(retrieval_vals),
             "qa_avg": _avg(qa_vals),
+            "qa_p95": _p95(qa_vals),
             "total_question_avg": _avg(total_vals),
+            "total_question_p95": _p95(total_vals),
         }
 
         return metrics
@@ -897,7 +905,15 @@ def _relative_time_note(text: str, timestamp: Any) -> str:
         if phrase not in lower:
             continue
         if kind == "anchor":
-            notes.append(f'"{phrase}" anchored to session date {reference_date.isoformat()}')
+            if phrase in ("recently", "lately", "earlier this week", "later this week"):
+                prior_start = (reference_dt - timedelta(days=7)).date().isoformat()
+                prior_end   = (reference_dt - timedelta(days=1)).date().isoformat()
+                notes.append(
+                    f'"{phrase}" → event occurred approximately {prior_start} to {prior_end} '
+                    f'(days/weeks BEFORE session date {reference_date.isoformat()}, not on it)'
+                )
+            else:
+                notes.append(f'"{phrase}" anchored to session date {reference_date.isoformat()}')
         elif kind == "year":
             delta_y = -1 if phrase.startswith("last") else 1
             resolved_year = reference_dt.replace(year=reference_dt.year + delta_y).year
@@ -922,13 +938,24 @@ def _format_retrieved_context(search_results: Any) -> str:
 
     lines: list[str] = []
 
+    episodes = search_results.get("episodes") or []
+    if episodes:
+        lines.append("## Episodes")
+        for i, ep in enumerate(episodes, 1):
+            timestamp = _timestamp_for_context(ep)
+            date = _date_prefix(timestamp)
+            date_prefix = f"[{date}] " if date else ""
+            text = str(ep.get("text", str(ep))).strip()
+            text = f"{text}{_relative_time_note(text, _event_time_only(ep))}"
+            lines.append(f"{i}. {date_prefix}{text}")
+
     facts = search_results.get("facts") or []
     if facts:
         facts = sorted(
             facts,
             key=_fact_context_sort_key,
         )
-        lines.append("## Facts (ranked by relevance and specificity)")
+        lines.append("\n## Facts (ranked by relevance and specificity)")
         for i, fact in enumerate(facts, 1):
             timestamp = _timestamp_for_context(fact)
             date = _date_prefix(timestamp)
@@ -937,17 +964,6 @@ def _format_retrieved_context(search_results: Any) -> str:
             text = f"{text}{_relative_time_note(text, _event_time_only(fact))}"
             lines.append(f"{i}. {date_prefix}{text}")
 
-    episodes = search_results.get("episodes") or []
-    if episodes:
-        lines.append("\n## Episodes")
-        for i, ep in enumerate(episodes, 1):
-            timestamp = _timestamp_for_context(ep)
-            date = _date_prefix(timestamp)
-            date_prefix = f"[{date}] " if date else ""
-            text = str(ep.get("text", str(ep))).strip()
-            text = f"{text}{_relative_time_note(text, _event_time_only(ep))}"
-            lines.append(f"{i}. {date_prefix}{text}")
-
     syntheses = search_results.get("syntheses") or []
     if syntheses:
         lines.append("\n## Syntheses")

scripts/run_and_judge.py

@@ -72,9 +72,11 @@ def _parse_args() -> argparse.Namespace:
     p.add_argument("--qa-thinking-level", default=None,
                    help="Thinking level for QA eval model: high/medium/low/minimal")
     # Judge args
-    p.add_argument("--judge-model", default="gemini:gemini-2.5-flash-lite")
+    p.add_argument("--judge-model", default="gemini:gemini-3-flash-preview")
     p.add_argument("--judge-n", type=int, default=0,
                    help="How many answers to judge (0 = all, default)")
+    p.add_argument("--judge-concurrency", type=int, default=5,
+                   help="Max concurrent judge requests (default 5)")
     return p.parse_args()
 
 
@@ -127,21 +129,24 @@ async def _run_judge(
     run_output_dir: str,
     judge_model: str,
     judge_n: int,
+    judge_concurrency: int = 5,
 ) -> dict:
     from benchmarks.locomo.locomo_judge import JudgeConfig, run as judge_run
 
     print(f"\n{'='*60}")
     print(f"STEP 2 — JUDGE")
-    print(f"  input  : {full_results}")
-    print(f"  model  : {judge_model}")
-    print(f"  sample : {'ALL' if judge_n <= 0 else judge_n}")
+    print(f"  input       : {full_results}")
+    print(f"  model       : {judge_model}")
+    print(f"  sample      : {'ALL' if judge_n <= 0 else judge_n}")
+    print(f"  concurrency : {judge_concurrency}")
     print(f"{'='*60}\n")
 
     config = JudgeConfig(
         full_results=str(full_results),
         judge_model=judge_model,
         n=judge_n if judge_n > 0 else 0,
         output_dir=run_output_dir,
+        concurrency=judge_concurrency,
     )
 
     return await judge_run(config)
@@ -160,9 +165,13 @@ def _print_scorecard(judge_output: dict, run_name: str, ppr_on: bool) -> None:
 
     correct_pct = 100 * s.get("correct_rate", 0)
     combined_pct = 100 * s.get("combined_rate", 0)
+    precision_pct = 100 * s.get("answer_precision", 0)
+    recall_pct = 100 * s.get("answer_recall", 0)
+    ret_recall_pct = 100 * s.get("retrieval_recall", 0)
     ctx_pct = 100 * s.get("context_has_answer_rate", 0)
     qa_fail = s.get("qa_failure", 0)
     ret_fail = s.get("retrieval_failure", 0)
+    answered = s.get("answered", total - abstained)
 
     print(f"\n{'='*60}")
     print(f"FINAL SCORECARD — {run_name}")
@@ -174,6 +183,10 @@ def _print_scorecard(judge_output: dict, run_name: str, ppr_on: bool) -> None:
     print(f"  Combined C+PC : {combined:>4}  ({combined_pct:.1f}%)")
     print(f"  Wrong         : {wrong:>4}")
     print(f"  Abstained     : {abstained:>4}")
+    print(f"  Answered      : {answered:>4}")
+    print(f"  Ans Precision : {precision_pct:.1f}%  (correct / answered)")
+    print(f"  Ans Recall    : {recall_pct:.1f}%  (correct / total)")
+    print(f"  Ret Recall    : {ret_recall_pct:.1f}%  (ctx has answer / total)")
     print(f"  Context OK    : {s.get('context_has_answer'):>4}  ({ctx_pct:.1f}%)")
     print(f"  QA failures   : {qa_fail}")
     print(f"  Retrieval fail: {ret_fail}")
@@ -207,6 +220,7 @@ async def main() -> None:
         run_output_dir,
         args.judge_model,
         args.judge_n,
+        args.judge_concurrency,
     )
     _print_scorecard(judge_output, run_name, ppr_on=not args.no_ppr)
 

vektori/ingestion/extractor.py

@@ -329,6 +329,11 @@
 - source_quotes: verbatim text from the ASSISTANT turn
 
 General:
+- Proper nouns, brand names, place names, and specific quantities MUST appear verbatim in the fact text. Never substitute a category or synonym for a specific name:
+  BAD: "Calvin purchased a luxury car."      GOOD: "Calvin purchased a Ferrari 488 GTB."
+  BAD: "John's old area was flooded."        GOOD: "John's area West County was flooded."
+  BAD: "Tim visited a Harry Potter shop."    GOOD: "Tim visited House of MinaLima."
+  The 20-word guide does NOT override this — preserve the specific name even at 22–25 words.
 - One fact per statement. Short and crisp.
 - subject: 'user' when about the person speaking; a named entity when about someone/something they mention; 'assistant' for assistant facts
 - Extract at most {max_facts} facts total — prioritize high-confidence, significant ones