Exp/locomo bench

[laxmanclo]

Summary by CodeRabbit

Release Notes

• New Features

  • Added concurrent evaluation processing with configurable concurrency control.

• Improvements

  • Updated evaluation model to latest version for improved accuracy.
  • Enhanced fact extraction to preserve proper nouns and specific quantities.
  • Refined Q&A answer construction with stricter matching rules.
  • Added new performance metrics including p95 latencies, answer precision/recall, and retrieval recall.
  • Improved temporal phrase resolution and retrieved context formatting.

• Documentation

  • Updated benchmarks section with new metrics and results.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bae3dcad-b045-4afd-8140-74b8e3e84a6d

📥 Commits

Reviewing files that changed from the base of the PR and between f74bab6 and d5e5562.

📒 Files selected for processing (7)

• README.md
• benchmarks/locomo/locomo_judge.py
• benchmarks/locomo/locomo_runner.py
• scripts/run_and_judge.py
• vektori/ingestion/extractor.py
• vektori/qa/generator.py
• vektori/retrieval/search.py

---

📝 Walkthrough

Walkthrough

This PR updates the LoCoMo benchmarking pipeline with concurrent judging, improves evaluation metrics and prompts, refines context retrieval and QA generation rules, and updates benchmark documentation. Changes span the judge, runner, CLI, Vektori QA/retrieval subsystems, and facts extraction.

Changes

Judge Concurrency & Concurrent Evaluation

Layer / File(s)	Summary
Configuration & Model Update benchmarks/locomo/locomo_judge.py	DEFAULT_JUDGE_MODEL updated to gemini:gemini-3-flash-preview; JudgeConfig gains concurrency: int = 5 field.
Evaluation Prompt & Logic benchmarks/locomo/locomo_judge.py	JUDGE_PROMPT_TEMPLATE expanded with stricter date-equivalence rules (requiring calendar-date computation) and improved handling for short answers; evaluate_entry now accepts pre-built llm instance and increases max_tokens from 350 to 400.
Concurrent Judging Pipeline benchmarks/locomo/locomo_judge.py	run() switches from sequential per-entry LLM construction to concurrent execution using a shared LLM instance, asyncio.Semaphore, and asyncio.gather, with per-item progress preserved.
Summary Metrics benchmarks/locomo/locomo_judge.py	summarize() adds answered, answer_precision, answer_recall, and retrieval_recall fields alongside existing rates and failure/parse-error counts.
CLI Wiring & Output scripts/run_and_judge.py	CLI adds --judge-concurrency flag (default 5), updates default --judge-model, wires concurrency into JudgeConfig, and extends "STEP 2 — JUDGE" output; _print_scorecard adds Answered, Ans Precision, Ans Recall, and Ret Recall metrics.

LoCoMo Runner Evaluation Enhancements

Layer / File(s)	Summary
Configuration & Defaults benchmarks/locomo/locomo_runner.py	LoCoMoConfig.top_k increased from 10 to 15; passed to Vektori as default_top_k.
Latency Metrics & P95 Computation benchmarks/locomo/locomo_runner.py	_compute_metrics adds _p95 helper and computes retrieval_p95, qa_p95, and total_question_p95 alongside existing average-latency fields.
Temporal Resolution & Context Formatting benchmarks/locomo/locomo_runner.py	_relative_time_note resolves recently, lately, earlier this week, and later this week to approximate pre-session date ranges; _format_retrieved_context reorders "Episodes" before "Facts" and adds explicit ## Facts (ranked by relevance and specificity) header.

Vektori QA, Retrieval & Facts Quality Improvements

Layer / File(s)	Summary
QA Answer Generation Rules vektori/qa/generator.py	QA_PROMPT rewritten to mandate scanning both Facts and Episodes for counting/list/aggregation queries, apply dedup only for clearly duplicated items, handle "recently/lately/latest" by selecting single most recent match, and require literal person-name matching without inferred attribution; removed length-matching constraint.
Facts Extraction Specificity vektori/ingestion/extractor.py	FACTS_PROMPT adds explicit rule that proper nouns, brand/place names, and specific quantities must appear verbatim in extracted fact text, overriding the 20-word guideline; includes "bad vs good" examples.
Retrieval Diversity vektori/retrieval/search.py	_diverse_top_k increases per_session default from 2 to 3, raising the maximum highest-scoring facts selected per session before remainder phase.

Documentation Updates

Layer / File(s)	Summary
Benchmark Results & Metrics README.md	Benchmarks section rewritten with F1 definition (harmonic mean of precision and recall), new results table including "F1 avg (LoCoMo)", "Search p95", "Total p95", and updated footnotes clarifying Zep self-reporting vs re-evaluation and Mem0 token-level F1 measurement.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• vektori-ai/vektori#60: Modifies overlapping temporal-resolution and retrieved-context code paths (_relative_time_note, context formatting), with coordinated changes to the run_and_judge workflow.
• vektori-ai/vektori#6: Judge changes (Gemini model switching, pre-built LLM instance, per-call max_tokens, concurrent LLM usage) directly relate to this PR's introduction of a Gemini LLM provider and broader LLMProvider API updates.

Poem

🐰 With gemini's swift flash now in charge,
And facts precise, no synomyn barge,
Concurrent judges leap through the night,
P95 metrics shining bright—
The benchmark hops toward perfect sight! ✨

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title 'Exp/locomo bench' is vague and does not clearly convey the main changes; it lacks specificity about what was actually modified or improved.	Revise the title to clearly describe the primary change, such as 'Improve LoCoMo benchmark evaluation with concurrent judging and updated metrics' or similar.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch exp/locomo-bench

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Review rate limit: 0/1 reviews remaining, refill in 60 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

docs/locomo_benchmark_learnings.md (1)

47-56: 💤 Low value

Minor: Add blank line before table for markdown compliance.

Per markdownlint MD058, tables should be surrounded by blank lines. There's a missing blank line before the table at line 48.

 ### Type definitions
+
 | Type | Name | Count | What it tests |

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/locomo_benchmark_learnings.md` around lines 47 - 56, Insert a single
blank line immediately before the markdown table that starts with the header row
"| Type | Name | Count | What it tests |" (under the "Type definitions" heading)
so the table is surrounded by a blank line per markdownlint MD058; ensure the
heading "Type definitions" and the table are separated by exactly one empty
line.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@docs/locomo_benchmark_learnings.md`:
- Around line 47-56: Insert a single blank line immediately before the markdown
table that starts with the header row "| Type | Name | Count | What it tests |"
(under the "Type definitions" heading) so the table is surrounded by a blank
line per markdownlint MD058; ensure the heading "Type definitions" and the table
are separated by exactly one empty line.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a405c19c-376c-4f85-8292-7351ef01d779

📥 Commits

Reviewing files that changed from the base of the PR and between 83901e1 and f74bab6.

📒 Files selected for processing (7)

• benchmarks/locomo/locomo_judge.py
• benchmarks/locomo/locomo_runner.py
• docs/locomo_benchmark_learnings.md
• scripts/run_and_judge.py
• vektori/ingestion/extractor.py
• vektori/qa/generator.py
• vektori/retrieval/search.py