Negotiate the output-token parameter instead of assuming it (#60)

OpenAI's newer models (GPT-5 family, o-series) reject `max_tokens` with a
400 and ask for `max_completion_tokens`. BYOK users pointing Nives at them
got that error instead of an answer; every other endpoint we support —
including the one Cloud uses — still wants `max_tokens`, so we cannot
simply switch.

Sniffing model names would be wrong twice over: the same model takes
different parameters depending on who serves it, and the list would need
editing on every OpenAI launch. Instead, send `max_tokens` and, only if
the endpoint objects, send it again the other way and remember the answer
for the process. Endpoints that work today are byte-identical; affected
ones pay one 400 (zero tokens billed) per model per restart.

Retrying is safe for the streaming call because the 400 arrives on the
initial request, before any chunk reaches the caller. Verified over real
HTTP against a fake endpoint that reproduces the reported 400.

Also: on these models the cap covers hidden reasoning tokens, so a
reasoning-heavy turn can exhaust it before writing a visible word. The
truncation hint now says that when it applies, rather than blaming prompt
size.

Author: hoornet

nives/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 2.4.24
+
+- **Bring-your-own-key now works with OpenAI's newest models.** Pointing Nives at GPT-5 or the o-series with your own OpenAI key failed before the model ever got a chance to answer: those models renamed the setting that limits how long a reply may be, and Nives was still sending the previous name. Nives now asks the endpoint which name it expects and remembers the answer, so the setting arrives correctly everywhere — OpenAI's newest models, OpenAI's older ones, and every other OpenAI-compatible endpoint Nives supports, local models included. There is nothing to configure, and setups that already worked are untouched. Thanks to @Kristofer-KNE for the report (#60).
+- **A clearer explanation when a thinking model runs out of room.** Models that reason before they answer spend the same reply-length budget on that reasoning, so a long deliberation can leave nothing over for the answer itself. When that happens, Nives now tells you exactly that and suggests a non-reasoning model or a lower reasoning effort, rather than pointing you at the size of your prompt.
+
 ## 2.4.23
 
 - **There is now exactly one place to set the assistant's personality.** Until now there were two Custom Prompt fields — one in the add-on's Configuration tab and one tucked behind the Nives integration's Configure button — and whichever was set in the second silently won. Fill in one while something sat in the other and your prompt did nothing, with no hint as to why. The integration field is gone; the add-on's Configuration tab — where your keys, logs and updates already live — is now the single home for the personality, and what you write there is always what runs. If your prompt lived in the integration field, move it over once (Settings → Apps → Nives → Configuration, then restart the add-on when asked) — and asking Nives to change its name will point you to the right place too.

nives/config.yaml

@@ -1,5 +1,5 @@
 name: "Nives"
-version: "2.4.23"
+version: "2.4.24"
 slug: "nives"
 description: "AI assistant with cognitive memory for Home Assistant"
 url: "https://github.com/hoornet/nives"

server/src/home-mind-server/src/llm/openai-client.test.ts

@@ -43,6 +43,7 @@ vi.mock("./tool-handler.js", () => ({
 }));
 
 import { OpenAIChatEngine } from "./openai-client.js";
+import { resetTokenCapCache } from "./token-cap.js";
 import { handleToolCall, extractAndStoreFacts } from "./tool-handler.js";
 
 describe("OpenAIChatEngine", () => {
@@ -54,6 +55,7 @@ describe("OpenAIChatEngine", () => {
   let config: Config;
 
   beforeEach(() => {
+    resetTokenCapCache();
     mockCreate.mockReset();
     vi.mocked(handleToolCall).mockReset();
     vi.mocked(extractAndStoreFacts).mockReset();
@@ -404,6 +406,29 @@ describe("OpenAIChatEngine", () => {
     expect(createCall.max_tokens).toBe(2048);
   });
 
+  it("retries with max_completion_tokens when the model rejects max_tokens (issue #60)", async () => {
+    const rejection = Object.assign(
+      new Error(
+        "400 Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead."
+      ),
+      { status: 400, param: "max_tokens", code: "unsupported_parameter" }
+    );
+    mockCreate.mockRejectedValueOnce(rejection).mockResolvedValue(
+      makeStream([
+        { choices: [{ delta: { content: "Hello" }, finish_reason: null }] },
+        { choices: [{ delta: {}, finish_reason: "stop" }] },
+      ])
+    );
+
+    const result = await engine.chat({ message: "Hi", userId: "user-1" });
+
+    expect(result.response).toBe("Hello");
+    expect(mockCreate.mock.calls[0][0].max_tokens).toBe(2048);
+    expect(mockCreate.mock.calls[0][0].max_completion_tokens).toBeUndefined();
+    expect(mockCreate.mock.calls[1][0].max_completion_tokens).toBe(2048);
+    expect(mockCreate.mock.calls[1][0].max_tokens).toBeUndefined();
+  });
+
   it("fires extractAndStoreFacts after response", async () => {
     mockCreate.mockResolvedValue(
       makeStream([

server/src/home-mind-server/src/llm/openai-client.ts

@@ -9,6 +9,7 @@ import { buildSystemPromptText } from "./prompts.js";
 import { TOOL_DEFINITIONS, toOpenAITools } from "./tool-definitions.js";
 import { randomUUID } from "node:crypto";
 import { handleToolCall, extractAndStoreFacts, type ToolContext } from "./tool-handler.js";
+import { withTokenCap, usesMaxCompletionTokens } from "./token-cap.js";
 import type {
   ChatRequest,
   ChatResponse,
@@ -234,13 +235,18 @@ export class OpenAIChatEngine implements IChatEngine {
 
   private classifyEmptyResponse(finishReason: string | null): ChatError {
     if (finishReason === "length") {
-      return {
-        code: "MAX_TOKENS_TRUNCATED",
-        hint:
-          "Response was cut off at max_tokens before the model finished. " +
+      // On OpenAI's newer models the output cap also covers hidden reasoning
+      // tokens, so a reasoning-heavy turn can exhaust it before writing a single
+      // visible word. That is a different problem from a prompt being too large,
+      // and it deserves a different instruction.
+      const hint = usesMaxCompletionTokens(this.config.llmModel)
+        ? "The model used its entire output budget on internal reasoning and had " +
+          "none left for the answer. Try a non-reasoning model, or a lower " +
+          "reasoning effort if your provider exposes one."
+        : "Response was cut off at max_tokens before the model finished. " +
           "If you're seeing this often, the conversation prompt may be too large " +
-          "for the model's output budget — try a model with more output tokens.",
-      };
+          "for the model's output budget — try a model with more output tokens.";
+      return { code: "MAX_TOKENS_TRUNCATED", hint };
     }
     if (finishReason === "content_filter") {
       return {
@@ -271,16 +277,21 @@ export class OpenAIChatEngine implements IChatEngine {
     finishReason: string | null;
     toolCalls: FunctionToolCall[];
   }> {
-    const stream = await this.client.chat.completions.create({
-      model: this.config.llmModel,
-      max_tokens: isVoice ? 500 : 2048,
-      messages,
-      tools: OPENAI_TOOLS,
-      // Keep the tool list in the request (history already references it) but
-      // stop the model from issuing more calls.
-      ...(disableTools ? { tool_choice: "none" as const } : {}),
-      stream: true,
-    });
+    const stream = await withTokenCap(
+      this.config.llmModel,
+      isVoice ? 500 : 2048,
+      (cap) =>
+        this.client.chat.completions.create({
+          model: this.config.llmModel,
+          ...cap,
+          messages,
+          tools: OPENAI_TOOLS,
+          // Keep the tool list in the request (history already references it) but
+          // stop the model from issuing more calls.
+          ...(disableTools ? { tool_choice: "none" as const } : {}),
+          stream: true,
+        })
+    );
 
     let text = "";
     let finishReason: string | null = null;

server/src/home-mind-server/src/llm/token-cap.test.ts

@@ -0,0 +1,127 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+import {
+  withTokenCap,
+  isMaxTokensUnsupported,
+  resetTokenCapCache,
+} from "./token-cap.js";
+
+/** The shape the OpenAI SDK throws for GPT-5 / o-series when sent `max_tokens`. */
+function unsupportedParamError() {
+  return Object.assign(
+    new Error(
+      "400 Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead."
+    ),
+    {
+      status: 400,
+      param: "max_tokens",
+      code: "unsupported_parameter",
+      type: "invalid_request_error",
+    }
+  );
+}
+
+describe("isMaxTokensUnsupported", () => {
+  it("recognises the structured OpenAI rejection", () => {
+    expect(isMaxTokensUnsupported(unsupportedParamError())).toBe(true);
+  });
+
+  it("recognises a proxy that keeps the message but drops the fields", () => {
+    const err = Object.assign(new Error("Use 'max_completion_tokens' instead."), {
+      status: 400,
+    });
+    expect(isMaxTokensUnsupported(err)).toBe(true);
+  });
+
+  it("ignores unrelated 400s so they surface to the user", () => {
+    const err = Object.assign(new Error("model not found"), {
+      status: 400,
+      code: "model_not_found",
+    });
+    expect(isMaxTokensUnsupported(err)).toBe(false);
+  });
+
+  it("ignores non-400 failures", () => {
+    const err = Object.assign(new Error("rate limited"), {
+      status: 429,
+      param: "max_tokens",
+      code: "unsupported_parameter",
+    });
+    expect(isMaxTokensUnsupported(err)).toBe(false);
+  });
+
+  it("ignores non-objects", () => {
+    expect(isMaxTokensUnsupported("boom")).toBe(false);
+    expect(isMaxTokensUnsupported(null)).toBe(false);
+  });
+});
+
+describe("withTokenCap", () => {
+  beforeEach(() => {
+    resetTokenCapCache();
+  });
+
+  it("sends max_tokens and does not retry when the endpoint accepts it", async () => {
+    const send = vi.fn().mockResolvedValue("ok");
+
+    await expect(withTokenCap("gpt-4o-mini", 2048, send)).resolves.toBe("ok");
+
+    expect(send).toHaveBeenCalledTimes(1);
+    expect(send).toHaveBeenCalledWith({ max_tokens: 2048 });
+  });
+
+  it("retries with max_completion_tokens when the model rejects max_tokens", async () => {
+    const send = vi
+      .fn()
+      .mockRejectedValueOnce(unsupportedParamError())
+      .mockResolvedValueOnce("ok");
+
+    await expect(withTokenCap("gpt-5.6", 2048, send)).resolves.toBe("ok");
+
+    expect(send).toHaveBeenCalledTimes(2);
+    expect(send).toHaveBeenNthCalledWith(1, { max_tokens: 2048 });
+    expect(send).toHaveBeenNthCalledWith(2, { max_completion_tokens: 2048 });
+  });
+
+  it("remembers the answer, so only the first call pays the extra round-trip", async () => {
+    const send = vi
+      .fn()
+      .mockRejectedValueOnce(unsupportedParamError())
+      .mockResolvedValue("ok");
+
+    await withTokenCap("gpt-5.6", 500, send);
+    send.mockClear();
+    await withTokenCap("gpt-5.6", 500, send);
+
+    expect(send).toHaveBeenCalledTimes(1);
+    expect(send).toHaveBeenCalledWith({ max_completion_tokens: 500 });
+  });
+
+  it("learns per model, not globally", async () => {
+    const rejecting = vi
+      .fn()
+      .mockRejectedValueOnce(unsupportedParamError())
+      .mockResolvedValue("ok");
+    await withTokenCap("gpt-5.6", 500, rejecting);
+
+    const other = vi.fn().mockResolvedValue("ok");
+    await withTokenCap("openai/gpt-5.6-luna", 500, other);
+
+    expect(other).toHaveBeenCalledTimes(1);
+    expect(other).toHaveBeenCalledWith({ max_tokens: 500 });
+  });
+
+  it("propagates unrelated errors without a retry", async () => {
+    const err = Object.assign(new Error("model not found"), { status: 400 });
+    const send = vi.fn().mockRejectedValue(err);
+
+    await expect(withTokenCap("nope", 500, send)).rejects.toThrow("model not found");
+    expect(send).toHaveBeenCalledTimes(1);
+  });
+
+  it("does not loop when the retry itself fails", async () => {
+    const send = vi.fn().mockRejectedValue(unsupportedParamError());
+
+    await expect(withTokenCap("gpt-5.6", 500, send)).rejects.toThrow();
+    expect(send).toHaveBeenCalledTimes(2);
+  });
+});

server/src/home-mind-server/src/llm/token-cap.ts

@@ -0,0 +1,85 @@
+/**
+ * Output-token cap, negotiated rather than guessed.
+ *
+ * OpenAI's newer models (GPT-5 family, o-series) reject `max_tokens` outright:
+ *
+ *   400 Unsupported parameter: 'max_tokens' is not supported with this model.
+ *       Use 'max_completion_tokens' instead.
+ *
+ * Every other endpoint we support still wants `max_tokens` — OpenRouter, Ollama,
+ * LM Studio, llama.cpp and the various OpenAI-compatible shims. So we cannot
+ * simply switch, and we deliberately do NOT sniff the model name: the same model
+ * takes different parameters depending on who is serving it (`openai/gpt-5.x` via
+ * OpenRouter accepts `max_tokens` happily), and a name list would need editing
+ * every time OpenAI ships a model.
+ *
+ * Instead: send `max_tokens`, and if the endpoint tells us it wants the other
+ * spelling, send it again the other way and remember the answer for the rest of
+ * the process. The probe costs one round-trip per model, only ever on the first
+ * call, and it is a 400 — no tokens are billed for it. Endpoints that work today
+ * see no change at all.
+ *
+ * Retrying is safe for streaming calls because the 400 arrives on the initial
+ * request, before any chunk has been produced or forwarded to the caller.
+ */
+
+/** Models known — from their own error response — to need `max_completion_tokens`. */
+const needsMaxCompletionTokens = new Set<string>();
+
+export type TokenCapParam =
+  | { max_tokens: number }
+  | { max_completion_tokens: number };
+
+/**
+ * True when the error is an endpoint telling us `max_tokens` is the wrong
+ * spelling. Kept narrow on purpose: a generic 400 (bad model name, malformed
+ * tool schema) must fall through and surface to the user unchanged.
+ */
+export function isMaxTokensUnsupported(error: unknown): boolean {
+  if (typeof error !== "object" || error === null) return false;
+  const err = error as { status?: number; param?: string; code?: string; message?: string };
+  if (err.status !== 400) return false;
+  if (err.param === "max_tokens" && err.code === "unsupported_parameter") return true;
+  // Providers that proxy OpenAI often keep the message but drop the structured
+  // fields, so fall back to the one phrase that is unambiguous.
+  return typeof err.message === "string" && err.message.includes("max_completion_tokens");
+}
+
+/**
+ * Run `send` with whichever output-cap parameter this model accepts, learning
+ * the answer from the endpoint on first use.
+ */
+export async function withTokenCap<T>(
+  model: string,
+  maxTokens: number,
+  send: (cap: TokenCapParam) => Promise<T>
+): Promise<T> {
+  if (needsMaxCompletionTokens.has(model)) {
+    return send({ max_completion_tokens: maxTokens });
+  }
+
+  try {
+    return await send({ max_tokens: maxTokens });
+  } catch (error) {
+    if (!isMaxTokensUnsupported(error)) throw error;
+    needsMaxCompletionTokens.add(model);
+    console.info(
+      `[llm] ${model} rejects max_tokens; using max_completion_tokens for this model from now on.`
+    );
+    return send({ max_completion_tokens: maxTokens });
+  }
+}
+
+/**
+ * True once this model has told us it wants `max_completion_tokens` — i.e. it is
+ * one of OpenAI's newer models, where the cap covers reasoning tokens as well as
+ * the visible answer. Used only to explain a truncation accurately.
+ */
+export function usesMaxCompletionTokens(model: string): boolean {
+  return needsMaxCompletionTokens.has(model);
+}
+
+/** Test seam — the learned set is process-global by design. */
+export function resetTokenCapCache(): void {
+  needsMaxCompletionTokens.clear();
+}

server/src/home-mind-server/src/memory/openai-extractor.ts

@@ -2,6 +2,7 @@ import OpenAI from "openai";
 import type { ExtractedFact, Fact } from "./types.js";
 import type { IFactExtractor } from "../llm/interface.js";
 import { EXTRACTION_PROMPT, VALID_CATEGORIES } from "./extraction-prompt.js";
+import { withTokenCap } from "../llm/token-cap.js";
 
 export class OpenAIFactExtractor implements IFactExtractor {
   private client: OpenAI;
@@ -56,14 +57,16 @@ ${JSON.stringify(factsJson, null, 2)}`;
         .replace("{user_message}", userMessage)
         .replace("{assistant_response}", assistantResponse);
 
-      const response = await this.client.chat.completions.create({
-        model: this.model,
-        max_tokens: this.maxTokens,
-        messages: [{ role: "user", content: prompt }],
-        ...(this.responseFormat
-          ? { response_format: { type: this.responseFormat } }
-          : {}),
-      });
+      const response = await withTokenCap(this.model, this.maxTokens, (cap) =>
+        this.client.chat.completions.create({
+          model: this.model,
+          ...cap,
+          messages: [{ role: "user", content: prompt }],
+          ...(this.responseFormat
+            ? { response_format: { type: this.responseFormat } }
+            : {}),
+        })
+      );
 
       const text = response.choices[0]?.message?.content ?? "";