Skip to content

fix: warn when vLLM grammar output does not match grammar#1456

Open
octo-patch wants to merge 1 commit into
guidance-ai:mainfrom
octo-patch:fix/issue-1383-warn-on-grammar-mismatch
Open

fix: warn when vLLM grammar output does not match grammar#1456
octo-patch wants to merge 1 commit into
guidance-ai:mainfrom
octo-patch:fix/issue-1383-warn-on-grammar-mismatch

Conversation

@octo-patch

Copy link
Copy Markdown

Fixes #1383

Problem

When using the VLLMModel or LiteLLM model with grammar constraints, vLLM's guided decoding sometimes produces output that doesn't fully match the guidance grammar (e.g. due to backtracking limitations). In this case, node.match() returns None and the code silently fell through with a pass, leaving named captures (variables) unset.

This caused a confusing KeyError when users tried to access captured variables (e.g. lm['response']), with no indication of why the variable was missing or what went wrong during generation.

Solution

Replace the silent pass (with a TODO: should probably raise... comment) with a RuntimeWarning that:

  • Clearly states that the grammar match failed and captures will not be accessible
  • Shows the first 200 characters of the generated output for debugging
  • Explains possible causes (vLLM guided decoding backtracking limitations, version incompatibility)
  • Suggests remediation steps (simplify grammar, check vLLM version)

This fix is applied consistently to both VLLMInterpreter._grammar_vllm and LiteLLMInterpreter._grammar_vllm, which share the same pattern.

Testing

The underlying backtracking issue requires a vLLM server to reproduce. The warning path is triggered when node.match() returns None, which happens when vLLM generates text that does not satisfy the grammar constraints.

@ErenAta16 ErenAta16 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent here is right and matches #1383 well, but as written this breaks the package entirely rather than just improving the warning. {buffer!r[:200]} is not valid f-string syntax — a conversion specifier (!r) has to be immediately followed by : or the closing }, not a slice. Checked it out and confirmed:

$ python3 -c "import guidance"
  File ".../guidance/models/experimental/_litellm.py", line 235
    f"checking vLLM version compatibility.",
                                           ^
SyntaxError: f-string: expecting `}"

That is a module-level SyntaxError, so import guidance itself fails on this branch — not just the VLLMModel/LiteLLM path. Same construct is duplicated in _vllm.py. I think what was intended is slicing before applying !r, i.e. {buffer[:200]!r}, which parses fine and gives the same truncated-repr output the message is going for.

No CI run has landed on this branch yet (checked check-runs, empty), so this would have been the first thing to catch it — worth a quick push once thats swapped in the two files and Ill re-review right away.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

'Can't backtrack' Error with Experimental hosted_vllm LiteLLM

2 participants