Skip to content

Commit e56aaca

Browse files
committed
fix(textGeneration): finalize empty truncated generations
Gate the completion handling on `generated_text != null` instead of truthiness so a terminal output with empty text is still finalized. A model can hit finish_reason "length" before emitting any visible content (hidden-reasoning models, very small budgets); previously that output was skipped (the special token was ignored) and the truncation warning never fired. Now the truncation is flagged + logged for the empty case too. Note: a genuinely empty body still surfaces as a "no output" error downstream (content equals the initial empty content), which is the appropriate UX; this change is about observability and a consistent interrupted flag, not changing that outcome. Adds an adapter test for the empty length-terminated stream.
1 parent 65c9feb commit e56aaca

2 files changed

Lines changed: 20 additions & 2 deletions

File tree

src/lib/server/endpoints/openai/openAIChatToTextGenerationStream.spec.ts

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,21 @@ describe("openAIChatToTextGenerationStream truncation handling", () => {
3939
expect(final?.generated_text).toContain("<think>");
4040
});
4141

42+
it("emits a terminal truncated output even when no content was produced", async () => {
43+
// Hidden-reasoning models can burn the whole budget without surfacing any tokens,
44+
// then terminate with finish_reason "length". The adapter must still yield a terminal
45+
// output (generated_text "" rather than null) carrying truncated:true so the pipeline
46+
// can finalize/flag it instead of silently dropping the empty completion.
47+
const chunks = [{ choices: [{ index: 0, delta: {}, finish_reason: "length" }] }];
48+
49+
const outputs = await collect(openAIChatToTextGenerationStream(mockStream(chunks)));
50+
const final = outputs.find((o) => o.generated_text !== null);
51+
52+
expect(final).toBeDefined();
53+
expect(final?.generated_text).toBe("");
54+
expect(final?.truncated).toBe(true);
55+
});
56+
4257
it("does not flag truncated on a normal finish_reason 'stop'", async () => {
4358
const chunks = [
4459
{ choices: [{ index: 0, delta: { content: "Hello" } }] },

src/lib/server/textGeneration/generate.ts

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,8 +86,11 @@ export async function* generate(
8686
continue;
8787
}
8888
}
89-
// text generation completed
90-
if (output.generated_text) {
89+
// text generation completed. Use `!= null` (not truthiness) so a terminal output
90+
// with empty text is still finalized: a model can hit finish_reason "length" before
91+
// emitting any visible content (hidden-reasoning models, tiny budgets), and that
92+
// truncation must still be flagged + logged rather than silently dropped.
93+
if (output.generated_text != null) {
9194
// If an abort happened just before final output, stop here and let
9295
// the caller emit an interrupted final answer with partial text.
9396
const abortTime = AbortedGenerations.getInstance().getAbortTime(conv._id.toString());

0 commit comments

Comments
 (0)