[Bug] Ask (Q&A) answers are silently truncated due to hardcoded max_tokens=2000 in graphs/ask.py — especially visible in non-English languages

[inkainka]

Summary

POST /api/search/ask/simple returns HTTP 200 with an answer that is cut off mid-sentence, with no error indication to the caller. Root cause appears to be that open_notebook/graphs/ask.py hardcodes max_tokens=2000 in all three LLM calls of the ask graph, with no config/env override path. This is the same pattern already fixed for transformations (#565 -> #568) and podcasts (#639 -> #982/#992/#991/#1197), but the ask/Q&A graph appears to have been missed.

Because non-English languages (e.g. Japanese) often consume substantially more tokens for the same amount of content, this cap is hit far more easily in those languages, making the bug much more visible for non-English-speaking users while being harder to notice for English-only maintainers/testers.

Environment

• Docker image: lfnovo/open_notebook:v1-latest (rolling tag; exact commit could not be pinned down — Docker Hub shows the same digest as 1.14.0 for linux/amd64 as of 2026-07-25, but this was not independently verified beyond the tag page)
• Chat model: Anthropic claude-sonnet-5
• Embedding model: Ollama nomic-embed-text
• Confirmed independently that the same hardcoding exists in the current main branch on GitHub (see "Root cause" below), so this is not specific to our image/version.

Steps to Reproduce

1. Set up a notebook with sources in Japanese (or any token-dense non-English language).
2. Call POST /api/search/ask/simple with a Japanese question.
3. Repeat with a few different Japanese questions against the same notebook.

Observed Behavior

• HTTP status is 200 (no error surfaced).
• In our testing, 2 out of 3 Japanese questions returned answers cut off mid-sentence / mid-token. One answer ended abruptly mid-sentence; another ended in the middle of what looks like a source ID reference.
• This is consistent with hitting a hard output-token cap during generation, though we did not inspect finish_reason/stop_reason from the underlying provider response to confirm this directly — flagging this as something maintainers may want to check when reproducing.
• There is no warning, error, or truncation indicator returned to the API caller. The user has no way to know the answer is incomplete.

Root Cause (verified against upstream main)

In open_notebook/graphs/ask.py (main branch, as of 2026-07-25), all three provision_langchain_model(...) calls pass a hardcoded max_tokens=2000, with no config or environment variable to override it:

• call_model_with_messages() — the strategy-generation call (structured/JSON output)
• provide_answer() — per-search-result answer generation
• write_final_answer() — final answer synthesis (the one most likely responsible for the user-visible truncation)

We were not able to obtain a fully reliable exact line number for these three call sites — two different retrieval methods against the same file gave inconsistent line numbers (apparently due to a rendering artifact, not a real code difference), so we're intentionally not quoting a specific line number here to avoid pointing at the wrong line. They are easy to find with:

grep -n "max_tokens=2000" open_notebook/graphs/ask.py

We only confirmed the hardcoding exists — we have not confirmed that all three call sites are equally responsible for the truncation we observed. The first call (call_model_with_messages) requests structured/JSON output, so hitting the cap there would more likely surface as a JSON-parsing error rather than a silently truncated prose answer. The truncation we actually observed looks more consistent with the provide_answer / write_final_answer stages.

Why This Is More Visible in Non-English Languages

For a given amount of semantic content, tokenizers used by most LLM providers typically require more tokens to represent Japanese (and several other non-English/non-Latin-script languages) than to represent equivalent English content. A fixed 2000-token output cap that is rarely hit in English can be hit routinely in Japanese, especially for longer or more detailed answers. This means the same codebase can look "fine" in English-language testing while silently degrading for non-English users.

Precedent

This exact class of bug (hardcoded max_tokens causing silent truncation) has already been reported and fixed twice in this repo for other code paths:

• [Bug] Transformation output truncated due to hardcoded max_tokens limit (5055) #565 — Transformation output truncated due to hardcoded max_tokens (fixed in fix: update esperanto dep and increase transformation max_tokens #568)
• [Bug] Podcast generation fails due to hardcoded max_tokens limits in podcast_creator (3000 outline / 5000 transcript) #639 — Podcast generation hardcoded max_tokens (fixed via fix: pass max_tokens through to podcast_creator for outline/transcript generation #982/feat(frontend): expose episode profile max_tokens in the UI #992, further exposed in UI via [Feature]: expose episode profile max_tokens in the UI #991/fix(podcasts): clarify episode max_tokens field and fix empty-state error #1197)

The ask/Q&A graph (graphs/ask.py) appears to be the one remaining major code path with the same hardcoding.

Separately, #947 (Model & Provider Configuration design, ratified 2026-07-21) lists a universal parameter tier including max_tokens as part of its Phase 1 scope, and #830 notes that provision_langchain_model() already supports per-call max_tokens overrides via Esperanto >=2.21.0. This suggests the underlying plumbing to fix this already exists; it's a matter of wiring graphs/ask.py to use a configurable value rather than a literal 2000.

Suggested Fix (any of the following, not mutually exclusive)

1. Make max_tokens for the ask graph configurable via settings/env var (consistent with how [Feature]: expose episode profile max_tokens in the UI #991/fix(podcasts): clarify episode max_tokens field and fix empty-state error #1197 exposed it for podcasts).
2. Raise the hardcoded default meaningfully (e.g. similar to how fix: update esperanto dep and increase transformation max_tokens #568 raised the transformation default), acknowledging that a single number still won't be right for all languages/models.
3. At minimum, detect when the model output was cut off due to the token cap (e.g. via finish_reason/stop_reason if available from the provider) and surface a warning to the API caller / UI instead of silently returning HTTP 200 with an incomplete answer.

Open Questions for Maintainers

• Can you confirm via finish_reason/stop_reason whether the observed truncation is in fact due to hitting the 2000-token cap, as opposed to some other cause?
• Does this reproduce with providers other than Anthropic? We have only tested with claude-sonnet-5.

Notes on What We Could Not Verify

• We could not pin down the exact commit corresponding to our lfnovo/open_notebook:v1-latest image (rolling tag).
• We did not verify this against a from-source / non-Docker install.
• We did not search GitHub Discussions (only Issues/PRs) for prior reports.

[tiganabryan]

Bump. I'm also experiencing this issue.

[akulmehta]

I am also facing this as I am using a thinking model and most of the tokens are as "thinking" tokens. So this is not just an issue with Non English Languages. It would be good to set an ASK_MAX_TOKENS environment variable for max tokens.

[bferanmi806-sketch]

Hi! I’d like to work on this. I’ll trace the Ask graph’s existing token-budget and configuration conventions, implement a narrowly scoped configurable budget for the affected Ask stages, and add focused regression coverage while preserving unrelated model behavior. I’ll also verify whether the structured strategy step should retain a separate smaller budget rather than applying one value blindly to all three calls.

[julienmasanes]

Confirming this bug, and reporting a second, more severe failure mode of the same root cause: with a reasoning model, the 2000-token cap does not truncate the answer, it produces an empty one, and the app then reports "no documents found" while the corpus actually contains the answer.

This also answers the open question at the end of the report about finish_reason.

Environment

• Image lfnovo/open_notebook@sha256:e53f90d6153fcf4a64604d9a0c12cb0428a32cfb8dbcbb72e81ab12c013ee330 (v1-latest)
• LLM: Ollama qwen3.6:35b-a3b-q8_0, num_ctx 131072. Thinking is on by default on this model, as it is on most recent open-weight releases.
• Embeddings: Ollama bge-m3
• Corpus: one 466k-character EPUB, 396 embedded chunks

Symptom

Ask returns, with HTTP 200:

The searches performed in your workspace returned no specific documents (['', '', '']). I therefore cannot cite sources from your corpus, but I can outline the main lines of academic criticism...

The three search terms are empty strings. The model then answers from parametric memory, which is worse than a truncated answer: the output looks complete and authoritative, and the user has no signal that retrieval never happened. In our case a direct vector search for the same term returned a matching chunk at similarity 0.475, well above the 0.2 minimum_score, so retrieval itself was fine.

Root cause, measured

Reasoning tokens are counted against the same budget, and clean_thinking_content() then strips them, leaving nothing.

Reproducing the strategy step directly against the model, with the same budget and format: json:

eval_count   : 381
done_reason  : "stop"
thinking     : 1661 characters
response     : 0 characters   <-- empty

Note done_reason: "stop", not "length". The model finished cleanly. It simply spent its budget reasoning and emitted no JSON at all. So for call_model_with_messages() this is not truncation in the usual sense, and it does not surface as a JSON parsing error as hypothesized in the report: PydanticOutputParser yields a Strategy whose searches have empty term strings, vector_search dutifully returns nothing, and write_final_answer() concludes that the corpus has no relevant documents.

provide_answer() fails the same way: clean_thinking_content() returns an empty string, so every partial answer is empty.

Worth noting that graphs/chat.py uses max_tokens=8192 and works perfectly with the same model on the same corpus. The contrast is a good confirmation that the budget, not the model, is the problem.

Users cannot work around this by disabling thinking

On Ollama, only the API-level think: false disables reasoning, and Open Notebook does not expose it. Both alternatives fail:

• PARAMETER think false in a Modelfile: Error: unknown parameter 'think'
• The Qwen /no_think marker, in the prompt and in the system prompt: ignored by this model, reasoning still emitted, response still empty

So the only user-side workarounds today are patching ask.py or picking a non-reasoning model for all three stages.

Fix confirmed

Raising the three call sites to max_tokens=8000, same question, same corpus, same model:

	2000	8000
Search terms	['', '', '']	3 relevant terms
Partial answers	0 chars	2165 / 2459 / 2552 chars
Final answer	"no documents found"	correct, with verbatim quotes checked against the source
Wall clock	1m33	4m27

Suggestion, in addition to the three already listed

Since reasoning models are becoming the default rather than the exception, the third suggestion in the report (surface a warning instead of silently returning 200) matters even more here than for truncation: an empty strategy or an empty partial answer is never a legitimate result and could be detected cheaply. Failing loudly on len(strategy.searches) == 0 or on any empty term would have turned a silent wrong answer into an actionable error.

Optionally, passing think=False on the two structured/JSON stages where reasoning brings little value would fix the cost side too, rather than paying for reasoning tokens that are immediately discarded by clean_thinking_content().

[bferanmi806-sketch]

Thanks for the detailed reproduction. This exposed a gap in #1247. The PR originally kept the structured strategy step at 2000 tokens, but your reasoning-model case shows that the strategy stage can exhaust that budget before producing usable structured output.

I've updated #1247 so OPEN_NOTEBOOK_ASK_MAX_TOKENS now applies to all three Ask model calls, including strategy generation, with the existing 8192 default. The separate empty-strategy/empty-answer validation point is also useful, but I've kept that outside this narrowly scoped token-budget fix.