Bug: Case-sensitive parsing in LLMChecker causes 100% failure rate with gpt-5-mini judge

[bhavishya-pohani]

Bug

LLMChecker uses case-sensitive parsing ("[[True]]" in response), but gpt-5-mini outputs [[true]] (lowercase). This causes 100% failure rate on ambiguity benchmarks.

Location

are/simulation/validation/utils/llm_utils.py lines 146-149

Fix

response_lower = response.lower()
if self.success_str.lower() in response_lower:
    votes.append(True)
elif self.failure_str.lower() in response_lower:
    votes.append(False)

Minimal Reproducible Test

# Before fix: "lowercase" test FAILS
# After fix: all tests PASS

success_str = "[[True]]"
test_responses = [
    ("[[True]]", True),   # uppercase - works
    ("[[true]]", True),   # lowercase - FAILS without fix
]

for response, expected in test_responses:
    # Current code (case-sensitive):
    result = success_str in response
    print(f"'{response}': {result} (expected: {expected})")

Output before fix:

'[[True]]': True (expected: True)
'[[true]]': False (expected: True)  ← BUG

Author

Bhavishya Pohani (@bhavishya-pohani)

[bhavishya-pohani]

trace_judge_full.py

Here's the complete file that I used to debug the issue ^ , this specifically changes the results significantly & I'm seeing much closer results to the paper now.

[tshu-w]

#26

[sacrusha]

This is independent of the judge model. gpt-5-mini's formatting isn't the trigger; I'd suggest retitling away from the model-specific framing.

All models are affected because the verdict markers are normalized lowercase before LLMChecker ever sees them.

LiteLLMEngine.chat_completion rewrites every response it returns:

# are/simulation/agents/llm/litellm/litellm_engine.py:125
res = res.replace("False", "false").replace("True", "true")

HuggingFaceEngine carries the identical line at are/simulation/agents/llm/hf/hf_engine.py:152, and those are the only two engines in the repo. LLMEngine.__call__ is defined once and delegates straight to chat_completion, and create_judge_engine (are/simulation/validation/configs.py) unconditionally constructs a LiteLLMEngine for every provider, including the documented default, meta-llama/Meta-Llama-3.3-70B-Instruct.

So every judge that correctly emits [[False]] has it rewritten to [[false]] in transit. Then:

# are/simulation/validation/utils/llm_utils.py:142-147
if self.success_str in response:      # "[[True]]"
    votes.append(True)
elif self.failure_str in response:    # "[[False]]"
    votes.append(False)
...
return None                            # no votes

Blast radius, and why it fails silently

Four of the nine soft checkers use the [[True]]/[[False]] marker pair - signature_checker, sanity_checker, cab_checker, tone_checker - and they gate high-traffic tools (are/simulation/validation/constants.py, PER_TOOL_TO_SOFT_CHECKER_TYPES):

checker	tools gated
sanity_checker	AgentUserInterface__send_message_to_user
signature_checker, tone_checker	EmailClientApp__send_email, EmailClientApp__reply_to_email
cab_checker	CabApp__order_ride

The remaining five (content, email, message, user_message, event) use [[Success]]/[[Failure]], which the rewrite doesn't touch - so they work, and the failure is partial rather than total.

The silence comes from None being falsy at the call site:

# are/simulation/validation/tool_judge.py:583
if not checker_fn(...):
    return False

An unparsable verdict is therefore indistinguishable from a genuine rejection, with nothing logged. The affected checkers also sit ahead of the semantic ones in each chain, so the chain aborts before the checker that would actually compare content ever runs.

Probably the same root cause as #26 / #25

#26 removes exactly these two engine lines, and closes #25, which reports Gemini-2.5-Pro scoring 0.0 ± 0.0 on both Ambiguity and Adaptability - zeroes concentrated where soft checkers carry the most weight. Three reports, one cause.

Why I'd fix it in the checker rather than in the engine

The engine line isn't gratuitous: are/simulation/agents/default_agent/tools/json_action_executor.py:43 runs json.loads on the agent's action blob, where Python-style True/False would raise. The lowercasing is an agent-loop concern that leaks into the judge because both share LLMEngine. Deleting it (as #26 does) risks the agent action parser.

The fix proposed in this issue avoids that, and matches what this repo already does elsewhere:
gaia2-cli/core/gaia2_core/judge/checkers.py:342-345 is case-insensitive by construction:

response_lower = response.lower()
if self.success_str.lower() in response_lower:

and its judge engine returns content verbatim, with no rewrite. So the newer stack is immune on both layers.

Worth considering: log a warning when LLMChecker returns None. The absence of any signal is what makes this expensive to find — from outside, a dead judge is indistinguishable from a strict one.

The paper's numbers seem unaffected, were likely run by a version without this issue present.