Benchmark: Evaluation parser defaults to WRONG on ambiguous grader output, and README figures don't reproduce from the shipped dataset

[Habinkj]

Benchmark: scoring parser defaults to WRONG on ambiguous LLM output, and README figures don't reproduce from the shipped dataset

Hi — I've been running CORE locally and went through the benchmark harness to understand the LoCoMo methodology. Three findings, roughly in order of what I think matters most. Happy to open PRs for any of these if useful.

Filing here rather than on core-benchmark since this repo is more active. All references below are to RedPlanetHQ/core-benchmark at commit 0a73c18.

---

1. The evaluation parser defaults to WRONG, which may be understating reported accuracy

I noticed the prompt already guards against ambiguous grader output at services/evaluateService.js:34:

Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script.

So this failure mode is clearly known. My suggestion is just to make it detectable as well as discouraged, because a prompt instruction is a request rather than an enforcement, and right now the failure is silent.

evaluateAnswer() initialises the verdict at line 50:

let label = "WRONG";

It's only reassigned if one of two paths succeeds:

• JSON path (55–59): JSON.parse(response) succeeds and jsonResponse.label is exactly "CORRECT" or "WRONG".
• Text path (62–66): the response contains "CORRECT" and not "WRONG", or vice versa.

If neither fires, the answer is scored as incorrect by default — indistinguishable in the results from a genuine WRONG verdict.

The path to that state looks reachable, because the prompt asks for two different output shapes: line 33 requests a one-sentence explanation ending in a verdict, while line 36 requests JSON with a label key. A grader following line 33 emits prose plus a label. If that prose contains the other token — e.g. "the generated answer is not wrong, it gives the same date" — both tokens are present, neither text branch fires, and the WRONG default stands even though the grader ruled it correct. Line 34 discourages this, but doesn't prevent it.

Relatedly, the reasoning extraction at line 68 uses an end-anchored split:

const parts = response.split(/(CORRECT|WRONG)$/);

This won't match if the response ends with a period or newline after the label.

Net effect: parse ambiguity is silently scored as a wrong answer rather than surfaced. If this is firing at any material rate, CORE's true accuracy is higher than what's published.

Suggested fix: initialise label = null, treat unparseable responses as a distinct UNPARSEABLE outcome, and report that count separately so it's visible rather than absorbed into the WRONG bucket.

---

2. LLM verdicts and heuristic-fallback verdicts are aggregated together

evaluateAnswer() catches errors from makeModelCall and falls back to a word-overlap heuristic at evaluateService.js:99:

const isCorrect = matchRatio > 0.3; // If 30% of important words match

returning method: "heuristic_fallback" (line 105). That's a reasonable safety net, but the aggregation in locomo/evaluate_qa.js counts verdicts without reference to method (e.g. line 311, line 351 — both filter only on evaluationResult === "CORRECT").

So a run where every question was graded by the LLM and a run where 200 questions hit rate limits and fell through to 30%-word-overlap scoring produce output that looks identical. There's no field in the results that tells a reader which happened.

Suggested fix: carry method into the results and print an llm vs heuristic_fallback split in the summary. One line in the output, and it makes any published number auditable.

---

3. The README's question counts don't reproduce from locomo10.json

This is the one I can't explain, so I'll just show the measurement.

locomo/evaluate_qa.js:161 excludes adversarial questions:

if (qa.category === 5) return null;

That's correct and matches convention. But counting the shipped dataset:

const d = require("./locomo/locomo10.json");
let n = 0, cat = {};
d.forEach(c => (c.qa || []).forEach(q => { n++; cat[q.category] = (cat[q.category] || 0) + 1; }));
console.log("total:", n, cat);
// total: 1986 { '1': 282, '2': 321, '3': 96, '4': 841, '5': 446 }

Excluding category 5 leaves 1,540 questions. The README's sample output reports 1,247 total. Comparing the category breakdowns:

Category	Dataset (excl. cat 5)	README
1	282	269 (Single Hop)
2	321	184 (Multi Hop)
3	96	405 (Open Domain)
4	841	384 (Temporal)
Total	1,540	1,247

The gap isn't a truncated run — cumulative non-adversarial counts per conversation are 152 / 233 / 385 / 584 / 762 / 885 / 1035 / 1226 / 1382 / 1540, so 1,247 doesn't fall on a conversation boundary. And the distributions differ in shape, not just size: the smallest dataset bucket is 96 while the smallest reported bucket is 184; the largest is 841 versus 405.

Two smaller inconsistencies in the same block: the README's four category counts sum to 1,242 rather than the 1,247 stated as the total, and the headline percentages at the top of the README are exactly the sample-output ratios (245/269 = 91.1%, 156/184 = 84.8%, 287/405 = 70.9%, 337/384 = 87.8%), so this appears to be a real run rather than an illustrative table.

Question: was the published run performed against an earlier revision of locomo10.json? Reproducing against the current repo gives 1,540 non-adversarial questions with a different category distribution, so anyone re-running today won't be able to match the published figures.

Also worth noting: this repo's README cites 88.24% average accuracy and directs readers to core-benchmark for full results, but that repo's README reports 85% overall. It isn't clear to an outside reader which figure is current.

---

4. Two small documentation mismatches

• The README states GPT-4 evaluation is used instead of simple string matching. That's true of the primary path, but the error fallback (lines 92–106) is simple string matching. Worth a sentence so readers know what a degraded run looks like.
• "Match Ratio" is listed under Key Metrics as a semantic similarity score. It's computed at lines 77–79 as the fraction of gold-answer words of length > 2 found via generatedLower.includes(word) — substring containment, not semantic similarity. (It's also unweighted, and unanchored, so "cat" matches inside "catastrophe".) On the LLM path it's returned as a metric only and doesn't affect the label; the metric name is the issue, not the computation.
• The category-5 exclusion isn't mentioned in the README, so the 1,986 → 1,540 step is invisible unless you read the source.

---

Happy to send PRs for the documentation items and the method split in the summary output, if you'd like them separately. Thanks for open-sourcing the benchmark harness — being able to read the scoring path end-to-end is what made this possible.

# Benchmark: scoring parser defaults to WRONG on ambiguous LLM output, and README figures don't reproduce from the shipped dataset

Hi — I've been running CORE locally and went through the benchmark harness to understand the LoCoMo methodology. Three findings, roughly in order of what I think matters most. Happy to open PRs for any of these if useful.

Filing here rather than on core-benchmark since this repo is more active. All references below are to [RedPlanetHQ/core-benchmark](https://github.com/RedPlanetHQ/core-benchmark) at commit 0a73c18.

---

1. The evaluation parser defaults to WRONG, which may be understating reported accuracy

I noticed the prompt already guards against ambiguous grader output at services/evaluateService.js:34:

Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script.

So this failure mode is clearly known. My suggestion is just to make it detectable as well as discouraged, because a prompt instruction is a request rather than an enforcement, and right now the failure is silent.

evaluateAnswer() initialises the verdict at line 50:

let label = "WRONG";

It's only reassigned if one of two paths succeeds:

• JSON path (55–59): JSON.parse(response) succeeds and jsonResponse.label is exactly "CORRECT" or "WRONG".
• Text path (62–66): the response contains "CORRECT" and not "WRONG", or vice versa.

If neither fires, the answer is scored as incorrect by default — indistinguishable in the results from a genuine WRONG verdict.

The path to that state looks reachable, because the prompt asks for two different output shapes: line 33 requests a one-sentence explanation ending in a verdict, while line 36 requests JSON with a label key. A grader following line 33 emits prose plus a label. If that prose contains the other token — e.g. "the generated answer is not wrong, it gives the same date" — both tokens are present, neither text branch fires, and the WRONG default stands even though the grader ruled it correct. Line 34 discourages this, but doesn't prevent it.

Relatedly, the reasoning extraction at line 68 uses an end-anchored split:

const parts = response.split(/(CORRECT|WRONG)$/);

This won't match if the response ends with a period or newline after the label.

Net effect: parse ambiguity is silently scored as a wrong answer rather than surfaced. If this is firing at any material rate, CORE's true accuracy is higher than what's published.

Suggested fix: initialise label = null, treat unparseable responses as a distinct UNPARSEABLE outcome, and report that count separately so it's visible rather than absorbed into the WRONG bucket.

---

2. LLM verdicts and heuristic-fallback verdicts are aggregated together

evaluateAnswer() catches errors from makeModelCall and falls back to a word-overlap heuristic at evaluateService.js:99:

const isCorrect = matchRatio > 0.3; // If 30% of important words match

returning method: "heuristic_fallback" (line 105). That's a reasonable safety net, but the aggregation in locomo/evaluate_qa.js counts verdicts without reference to method (e.g. line 311, line 351 — both filter only on evaluationResult === "CORRECT").

So a run where every question was graded by the LLM and a run where 200 questions hit rate limits and fell through to 30%-word-overlap scoring produce output that looks identical. There's no field in the results that tells a reader which happened.

Suggested fix: carry method into the results and print an llm vs heuristic_fallback split in the summary. One line in the output, and it makes any published number auditable.

---

3. The README's question counts don't reproduce from locomo10.json

This is the one I can't explain, so I'll just show the measurement.

locomo/evaluate_qa.js:161 excludes adversarial questions:

if (qa.category === 5) return null;

That's correct and matches convention. But counting the shipped dataset:

const d = require("./locomo/locomo10.json");
let n = 0, cat = {};
d.forEach(c => (c.qa || []).forEach(q => { n++; cat[q.category] = (cat[q.category] || 0) + 1; }));
console.log("total:", n, cat);
// total: 1986 { '1': 282, '2': 321, '3': 96, '4': 841, '5': 446 }

Excluding category 5 leaves 1,540 questions. The README's sample output reports 1,247 total. Comparing the category breakdowns:

Category	Dataset (excl. cat 5)	README
1	282	269 (Single Hop)
2	321	184 (Multi Hop)
3	96	405 (Open Domain)
4	841	384 (Temporal)
Total	1,540	1,247

The gap isn't a truncated run — cumulative non-adversarial counts per conversation are 152 / 233 / 385 / 584 / 762 / 885 / 1035 / 1226 / 1382 / 1540, so 1,247 doesn't fall on a conversation boundary. And the distributions differ in shape, not just size: the smallest dataset bucket is 96 while the smallest reported bucket is 184; the largest is 841 versus 405.

Two smaller inconsistencies in the same block: the README's four category counts sum to 1,242 rather than the 1,247 stated as the total, and the headline percentages at the top of the README are exactly the sample-output ratios (245/269 = 91.1%, 156/184 = 84.8%, 287/405 = 70.9%, 337/384 = 87.8%), so this appears to be a real run rather than an illustrative table.

Question: was the published run performed against an earlier revision of locomo10.json? Reproducing against the current repo gives 1,540 non-adversarial questions with a different category distribution, so anyone re-running today won't be able to match the published figures.

Also worth noting: this repo's README cites 88.24% average accuracy and directs readers to core-benchmark for full results, but that repo's README reports 85% overall. It isn't clear to an outside reader which figure is current.

---

4. Two small documentation mismatches

• The README states GPT-4 evaluation is used instead of simple string matching. That's true of the primary path, but the error fallback (lines 92–106) is simple string matching. Worth a sentence so readers know what a degraded run looks like.
• "Match Ratio" is listed under Key Metrics as a semantic similarity score. It's computed at lines 77–79 as the fraction of gold-answer words of length > 2 found via generatedLower.includes(word) — substring containment, not semantic similarity. (It's also unweighted, and unanchored, so "cat" matches inside "catastrophe".) On the LLM path it's returned as a metric only and doesn't affect the label; the metric name is the issue, not the computation.
• The category-5 exclusion isn't mentioned in the README, so the 1,986 → 1,540 step is invisible unless you read the source.

---

Happy to send PRs for the documentation items and the method split in the summary output, if you'd like them separately. Thanks for open-sourcing the benchmark harness — being able to read the scoring path end-to-end is what made this possible.