AI Assistant: mid-run observation compaction (#37234) sends two system messages, breaking strict OpenAI-compatible backends (System message must be at the beginning) #37589

Bug description

Since 2.38.1 (contains #37234 "Trigger observational memory mid-run on a token budget"), long AI Assistant runs fail deterministically right after the first mid-run compaction, with:

APICallError: could not encode request: System message must be at the beginning.

UI shows "Something went wrong before I could finish that response" / "No output generated. Check the stream for errors."

Root cause: when the observation log becomes non-empty, buildSystemMessages() in
packages/@n8n/agents/src/runtime/model/message-list.ts returns two system entries
(base instructions + volatile sections incl. observation log). The OpenAI Chat Completions
spec tolerates multiple system messages, and OpenAI/Anthropic/OpenRouter accept them —
but many open-weights chat templates reject any system message that is not the single
first message, including Qwen-family ChatML templates served by vLLM
(the could not encode request: prefix wraps the Jinja raise_exception).

This makes the compaction feature unusable on a large class of self-hosted models,
even though the compaction itself completes successfully.

To Reproduce

1. n8n 2.38.1 (docker), AI Assistant with a custom OpenAI-compatible base URL pointing at vLLM serving a Qwen-family model (e.g. Qwen3 chat template).
2. Start an AI Assistant conversation long enough to trigger mid-run observational-memory compaction (logs: Observational memory task queued/started/completed).
3. The very next main-loop streamText call sends system: [base, observation-log] → HTTP 400 → run dies.

Minimal probe against the model server (no n8n needed):

Expected behavior

Compaction continues the run; the request shape stays compatible with strict OpenAI-compatible servers.

Proposed fix

Concatenate the base instructions and volatile sections into a single system message (they are already joined with \n\n semantics). A single leading system message is the lowest-common-denominator shape every OpenAI-compatible server accepts. Alternatively, put the observation log into the unmasked user-role continuation message, or expose a setting.

Workarounds currently used

1. A small sidecar proxy that merges all system messages into one leading system message before forwarding to vLLM.
2. N8N_INSTANCE_AI_OBSERVER_MESSAGE_TOKENS=10000 (default 30000): because fix(core): Gate observation on the model-facing window budget (no-changelog) #37438 makes estimateVisibleBudget under-count large tool results ([omitted large blob]), lowering the threshold lets the (under-)estimated budget cross the soft threshold early enough that mid-run compaction fires before a ~131K local window overflows.

Environment

• OS: macOS (Docker Desktop)
• n8n version: 2.38.1 (pre-release)
• Database: SQLite
• AI backend: self-hosted vLLM (uvicorn), OpenAI-compatible custom base URL

Related

• feat: Trigger observational memory mid-run on a token budget #37234 (merged) — introduces the two-system shape
• fix(core): Gate observation on the model-facing window budget (no-changelog) #37438 (open) — mid-run budget estimation under-counts large tool results, so on blob-heavy runs the two-system shape is only reached after the real window already overflowed
• Related errors on small-context local models: context_length_exceeded (no compression available before 2.38)