fix: warn when vLLM grammar output does not match grammar#1456
fix: warn when vLLM grammar output does not match grammar#1456octo-patch wants to merge 1 commit into
Conversation
ErenAta16
left a comment
There was a problem hiding this comment.
The intent here is right and matches #1383 well, but as written this breaks the package entirely rather than just improving the warning. {buffer!r[:200]} is not valid f-string syntax — a conversion specifier (!r) has to be immediately followed by : or the closing }, not a slice. Checked it out and confirmed:
$ python3 -c "import guidance"
File ".../guidance/models/experimental/_litellm.py", line 235
f"checking vLLM version compatibility.",
^
SyntaxError: f-string: expecting `}"
That is a module-level SyntaxError, so import guidance itself fails on this branch — not just the VLLMModel/LiteLLM path. Same construct is duplicated in _vllm.py. I think what was intended is slicing before applying !r, i.e. {buffer[:200]!r}, which parses fine and gives the same truncated-repr output the message is going for.
No CI run has landed on this branch yet (checked check-runs, empty), so this would have been the first thing to catch it — worth a quick push once thats swapped in the two files and Ill re-review right away.
Fixes #1383
Problem
When using the
VLLMModelorLiteLLMmodel with grammar constraints, vLLM's guided decoding sometimes produces output that doesn't fully match the guidance grammar (e.g. due to backtracking limitations). In this case,node.match()returnsNoneand the code silently fell through with apass, leaving named captures (variables) unset.This caused a confusing
KeyErrorwhen users tried to access captured variables (e.g.lm['response']), with no indication of why the variable was missing or what went wrong during generation.Solution
Replace the silent
pass(with aTODO: should probably raise...comment) with aRuntimeWarningthat:This fix is applied consistently to both
VLLMInterpreter._grammar_vllmandLiteLLMInterpreter._grammar_vllm, which share the same pattern.Testing
The underlying backtracking issue requires a vLLM server to reproduce. The warning path is triggered when
node.match()returnsNone, which happens when vLLM generates text that does not satisfy the grammar constraints.