Skip to content

fix: preserve temperature=0.0 and max_tokens=0 in subgrammar() factory#1459

Open
alvinttang wants to merge 1 commit into
guidance-ai:mainfrom
alvinttang:fix/subgrammar-zero-temperature
Open

fix: preserve temperature=0.0 and max_tokens=0 in subgrammar() factory#1459
alvinttang wants to merge 1 commit into
guidance-ai:mainfrom
alvinttang:fix/subgrammar-zero-temperature

Conversation

@alvinttang

Copy link
Copy Markdown

Summary

guidance._grammar.subgrammar() used if temperature: / if max_tokens: to decide whether to attach those kwargs to the built RuleNode. Because 0.0 and 0 are falsy in Python, callers passing temperature=0.0 (the canonical way to ask for deterministic / greedy sampling) or max_tokens=0 had their values silently dropped — the resulting grammar had no RuleNode.temperature or RuleNode.max_tokens set.

Every other factory in the codebase (guidance.json, guidance._grammar.gen) already uses is not None for the same kwargs, so subgrammar() was the only inconsistent path.

Root cause

# guidance/_grammar.py (before)
if max_tokens:          # wrong: 0 is valid
    body = token_limit(body, max_tokens=max_tokens)
if temperature:         # wrong: 0.0 is valid
    body = with_temperature(body, temperature=temperature)

Python treats 0 and 0.0 as falsy, so the guards silently drop the valid-but-falsy values before they reach the RuleNode.

Fix

Two-line change in guidance/_grammar.py: swap both guards for is not None, matching the rest of the codebase:

if max_tokens is not None:
    body = token_limit(body, max_tokens=max_tokens)
if temperature is not None:
    body = with_temperature(body, temperature=temperature)

Tests

tests/unit/library/test_subgrammar.py adds two parallel test classes:

  • TestSubgrammarFactoryTemperature: verifies temperature=0.0 propagates (RED on main), temperature=0.5 still works, temperature=None sets nothing.
  • TestSubgrammarFactoryMaxTokens: same three cases for max_tokens=0 / max_tokens=16 / max_tokens=None.

A helper walks the resulting grammar tree via GrammarNode.children() and collects every RuleNode.temperature (or max_tokens) that is not None.

Locally on darwin/arm64:

  • pytest tests/unit/library/test_subgrammar.py → 14 passed.
  • Full unit suite → 1582 passed, 1 skipped, 220 xfailed (all pre-existing; no new failures).
  • ruff check clean.

Risk notes

  • Behaviour change: callers who were passing temperature=0.0 or max_tokens=0 expecting them to be ignored will now see them forwarded to llguidance. llguidance accepts max_tokens=0 cleanly (verified locally via LLMatcher smoke test) — max_tokens=0 is also already the honoured value in gen() / json() / lark() per _ast.py serialization, so this just brings subgrammar() in line.
  • None (the documented default) still means "unset" and sets no limit. That invariant is covered by the _none_sets_nothing tests.
  • No API shape change.

Related to the long-standing observation in #600 that temperature can be silently dropped on certain call paths; this PR addresses the subgrammar() path specifically.

The factory function in guidance._grammar.subgrammar used a truthiness
check ("if temperature:") to decide whether to apply a temperature to
the generated RuleNode. This silently dropped the canonical greedy
setting temperature=0.0, meaning that

    subgrammar(body, temperature=0.0)

produced a grammar with no temperature constraint at all — so sampling
was not forced to deterministic on the server side. The same truthiness
check was applied to max_tokens, which would have discarded max_tokens=0
as well (less impactful, but equally surprising).

Switch both checks to "is not None", matching the convention used by
guidance.json() and guidance._grammar.gen() throughout the codebase.

Regression tests walk the resulting grammar tree and confirm that
temperature=0.0 is reachable on a RuleNode.temperature field, and that
non-zero and None cases behave as expected.
@alvinttang alvinttang force-pushed the fix/subgrammar-zero-temperature branch from ae79052 to f49660d Compare April 24, 2026 01:06

@riedgar-ms riedgar-ms left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK with this. @hudson-ai @nking-1 ?

@ErenAta16 ErenAta16 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed this is the same falsy-truthiness class as the metrics trimming bug in #1474. Checked token_limit() and with_temperature() in _grammar.py: both take the value as-is via dataclasses.replace(...max_tokens=max_tokens) / temperature=temperature, no special casing for 0. So subgrammar()'s old if max_tokens: / if temperature: guards silently dropped exactly the two values (max_tokens=0, temperature=0.0) that callers most plausibly pass on purpose, greedy decoding and a zero-token stop condition are both legitimate.

The new tests walk the actual grammar tree rather than asserting on subgrammar()'s return type directly, which is the right level to test at since RuleNode wrapping can nest. test_temperature_none_sets_nothing and test_max_tokens_none_sets_nothing are good inclusions too, they confirm the None branch still short-circuits instead of accidentally wrapping with a sentinel.

@ErenAta16 ErenAta16 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced the drop independently before looking at the fix: walking the RuleNode tree built by subgrammar(lexeme(r"\d+"), temperature=0.0) on main gives an empty temperature set, confirming 0.0 never reaches the node. Same for max_tokens=0. The is not None swap matches how guidance.json/gen already handle these two kwargs, so this also removes the one inconsistent path rather than adding a new convention.

Ran the new test file after checking out the branch — 14 passed, including the nonzero and None cases so the guard cannot silently regress to is not None catching too much either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants