Add synthetic respond tool for chat in tool-equipped sessions (#20)

The model calls respond(message="...") instead of producing bare text,
eliminating the trust_text_intent tradeoff. Three paths:

- WorkflowRunner: respond_tool() factory returns a ready-made ToolDef
- Proxy: auto-injected when tools present, stripped on return (client
  sees finish_reason="stop", never knows respond exists)
- Middleware: include "respond" in tool_names, handle in your loop
  (see examples/foreign_loop.py Part 3)

Also updates docs for KV cache quantization and multi-slot support.

Closes #15

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: antoinezambelli

README.md

@@ -110,7 +110,7 @@ python -m forge.proxy --backend llamaserver --gguf path/to/model.gguf --port 808
 
 Then configure your client to use `http://localhost:8081/v1` as the API base URL.
 
-**Note:** The proxy trusts the model's intent when it responds with text instead of calling tools (`trust_text_intent=True`). This eliminates retry latency on conversational turns but means forge won't nudge the model if it *should* have called a tool. In our eval testing, unconditionally trusting intent dropped an 8B model from 100% to as low as 4% on reasoning-heavy workflows — which is why WorkflowRunner and middleware default to `trust_text_intent=False` (full guardrail protection). The proxy accepts this tradeoff for zero-code integration; the client's own agentic loop handles re-prompting. See [ADR-013](docs/decisions/013-text-response-intent.md).
+**Note:** The proxy automatically injects a synthetic `respond` tool when tools are present in the request. The model calls `respond(message="...")` instead of producing bare text, keeping it in tool-calling mode where forge's full guardrail stack applies. The `respond` call is stripped from the outbound response — the client sees a normal text response (`finish_reason: "stop"`) and never knows the tool exists. This eliminates the `trust_text_intent` tradeoff described in [ADR-013](docs/decisions/013-text-response-intent.md) — no retries wasted on conversational turns, no accuracy loss on tool-calling turns.
 
 ## Backends
 
@@ -182,13 +182,15 @@ src/forge/
   prompts/
     templates.py       # Tool prompt builders (prompt-injected path)
     nudges.py          # Retry and step-enforcement nudge templates
+  tools/
+    respond.py         # Synthetic respond tool (respond_tool(), respond_spec())
   proxy/
     proxy.py           # ProxyServer — programmatic start/stop API
     server.py          # Raw asyncio HTTP server, SSE streaming
     handler.py         # Request handler — bridge between HTTP and run_inference
     convert.py         # OpenAI messages ↔ forge Messages conversion
 tests/
-  unit/                # 562 deterministic tests — no LLM backend required
+  unit/                # 655 deterministic tests — no LLM backend required
   eval/                # Eval harness — model qualification against real backends
 ```
 

docs/ARCHITECTURE.md

@@ -1234,6 +1234,10 @@ forge/
 │   │   ├── handler.py             # Request handler — bridge between HTTP and run_inference
 │   │   └── convert.py             # OpenAI messages ↔ forge Messages conversion
 │   │
+│   ├── tools/
+│   │   ├── __init__.py            # Built-in tool exports
+│   │   └── respond.py             # Synthetic respond tool (respond_tool(), respond_spec())
+│   │
 │   ├── server.py                  # ServerManager, BudgetMode, setup_backend()
 │   └── errors.py                  # ForgeError hierarchy
 │
@@ -1540,9 +1544,13 @@ class ServerManager:
         """
         ...
 
-    async def start(self, model, gguf_path, mode="native", extra_flags=None, ctx_override=None) -> None:
+    async def start(self, model, gguf_path, mode="native", extra_flags=None,
+                    ctx_override=None, cache_type_k=None, cache_type_v=None,
+                    n_slots=None) -> None:
         """Start a llama-server/llamafile process. No-op for Ollama.
-        Reuses existing process if same model+mode+ctx+flags."""
+        Reuses existing process if same model+mode+ctx+flags+cache+slots.
+        cache_type_k/v: KV cache quantization (e.g. "q8_0") — halves cache VRAM.
+        n_slots: number of concurrent slots (--parallel N)."""
         ...
 
     async def stop(self) -> None:
@@ -1554,7 +1562,9 @@ class ServerManager:
         ...
 
     async def start_with_budget(self, model, gguf_path, mode="native",
-                                budget_mode=BudgetMode.BACKEND, ...) -> int:
+                                budget_mode=BudgetMode.BACKEND, ...,
+                                cache_type_k=None, cache_type_v=None,
+                                n_slots=None) -> int:
         """Start server + resolve budget in one call. Returns budget in tokens."""
         ...
 ```
@@ -1587,3 +1597,41 @@ runner = WorkflowRunner(client=client, context_manager=ctx)
 # ... run workflows ...
 await server.stop()
 ```
+
+---
+
+## Synthetic Respond Tool
+
+When tools are present but the user sends a conversational message, small models must choose between calling a tool and responding with text. They frequently choose wrong — producing text when they should call tools, or vice versa. The `trust_text_intent` flag (ADR-013) addressed this for the proxy by trusting the model's finish reason, but this dropped 8B models from 100% to as low as 4% on reasoning-heavy workflows.
+
+The respond tool eliminates this ambiguity. The model calls `respond(message="...")` instead of producing bare text. From forge's perspective, every response is a valid tool call — no retries wasted on conversational turns, no accuracy loss on tool-calling turns.
+
+### Three paths
+
+**WorkflowRunner:** Use `respond_tool()` from `forge.tools` to get a ready-made `ToolDef`. Set it as the terminal tool. The callable returns the message string.
+
+```python
+from forge import respond_tool
+
+tools = {
+    "search": search_tool,
+    "respond": respond_tool(),
+}
+workflow = Workflow(..., tools=tools, terminal_tool="respond")
+```
+
+**Proxy:** The respond tool is injected automatically when tools are present in the request. The client never sees it — respond calls are converted to plain text responses (`finish_reason: "stop"`) before returning. This makes `trust_text_intent` irrelevant for the proxy path.
+
+**Middleware:** Include `"respond"` in `tool_names` when creating a `ResponseValidator` or `Guardrails` instance. The respond call passes validation like any other tool call. See `examples/foreign_loop.py` Part 3 for a complete example.
+
+### Why this works for small models
+
+Small models struggle with open-ended decisions ("should I use tools or chat?") but are good at structured choices ("which tool should I call?"). The respond tool converts an open-ended decision into a structured one. The model stays in tool-calling grammar/template at all times, which is where it performs best.
+
+### KV Cache Quantization
+
+`ServerManager.start()` and `setup_backend()` accept `cache_type_k` and `cache_type_v` parameters for KV cache quantization. Using Q8 (`cache_type_k="q8_0"`, `cache_type_v="q8_0"`) roughly halves KV cache VRAM, effectively doubling usable context for the same GPU memory. Measured: 36,864 → 68,608 tokens (1.86x) on Ministral 8B Q4 with no eval regression.
+
+### Multi-Slot Support
+
+`LlamafileClient` accepts a `slot_id` parameter to route requests to specific llama-server slots. `ServerManager.start()` accepts `n_slots` to start the server with `--parallel N`. This enables multi-agent architectures where each agent targets a dedicated KV cache slot.

docs/WORKFLOW.md

@@ -19,6 +19,7 @@ Visual guide to the forge agentic tool-calling loop.
 - `src/forge/context/strategies.py` - TieredCompact (3-phase compaction)
 - `src/forge/clients/base.py` - LLMClient protocol
 - `src/forge/server.py` - ServerManager, BudgetMode, setup_backend()
+- `src/forge/tools/respond.py` - Synthetic respond tool (respond_tool(), respond_spec())
 - `src/forge/prompts/nudges.py` - Retry, unknown-tool, and step nudges
 - `src/forge/prompts/templates.py` - Prompt-injected tool prompt, extract/rescue
 

examples/foreign_loop.py

@@ -7,9 +7,10 @@
     result = guardrails.check(response)   # before execution
     done   = guardrails.record(["tool"])  # after execution
 
-This file has two parts:
+This file has three parts:
   Part 1 -- The simple API (Guardrails facade)
   Part 2 -- The granular API (individual middleware components)
+  Part 3 -- Using the respond tool for chat in tool-equipped sessions
 
 No LLM backend required -- uses scripted responses to show each guardrail.
 """
@@ -112,7 +113,7 @@ def handle_response_granular(response):
 # Demo
 # =====================================================================
 
-EXAMPLES = [
+EXAMPLES_WORKFLOW = [
     ("Model returns plain text (no tool call)",
      TextResponse(content="I think the answer is 42.")),
     ("Model jumps to terminal tool, skipping required steps",
@@ -125,9 +126,74 @@ def handle_response_granular(response):
      [ToolCall(tool="answer", args={"text": "The answer is 42."})]),
 ]
 
+# =====================================================================
+# Part 3: Respond tool
+#
+# In tool-equipped sessions, the model must choose between calling a
+# tool and responding with text.  Small models frequently get this
+# wrong -- they produce text when they should call tools, or vice versa.
+#
+# The respond tool eliminates this ambiguity: the model calls
+# respond(message="...") instead of producing bare text.  From forge's
+# perspective, every response is a valid tool call -- no retries wasted
+# on conversational turns, no accuracy loss on tool-calling turns.
+#
+# For WorkflowRunner consumers: use respond_tool() from forge.tools.
+# For proxy consumers: respond is injected automatically.
+# For middleware consumers: include "respond" in tool_names and handle
+# the respond tool call in your execution logic.
+# =====================================================================
+
+from forge.tools import RESPOND_TOOL_NAME, respond_spec
+
+# Middleware with respond -- include it in tool_names
+respond_guardrails = Guardrails(
+    tool_names=["search", "lookup", "answer", RESPOND_TOOL_NAME],
+    required_steps=["search", "lookup"],
+    terminal_tool="answer",
+)
+
+
+def handle_response_with_respond(response):
+    """Process a response that may include a respond() call."""
+
+    result = respond_guardrails.check(response)
+
+    if result.action == "fatal":
+        return f"FATAL: {result.reason}"
+
+    if result.action in ("retry", "step_blocked"):
+        return f"{result.action}: {result.nudge.content[:80]}..."
+
+    tool_calls = result.tool_calls
+
+    # Check if the model called respond -- extract the message
+    for tc in tool_calls:
+        if tc.tool == RESPOND_TOOL_NAME:
+            message = tc.args.get("message", "")
+            return f"MODEL SAYS: {message}"
+
+    # Normal tool execution
+    executed = [tc.tool for tc in tool_calls]
+    done = respond_guardrails.record(executed)
+    return f"executed {executed}" + (" -- DONE" if done else "")
+
+
+EXAMPLES_RESPOND = [
+    ("Model produces bare text (retried -- respond not used)",
+     TextResponse(content="Hello, how can I help?")),
+    ("Model correctly uses respond for chat",
+     [ToolCall(tool="respond", args={"message": "Hello! How can I help you?"})]),
+    ("Model calls a real tool",
+     [ToolCall(tool="search", args={"query": "meaning of life"})]),
+    ("Model calls respond after completing work",
+     [ToolCall(tool="respond", args={"message": "I found the answer: 42."})]),
+]
+
+
 if __name__ == "__main__":
     print("=== Part 1: Simple API (Guardrails) ===")
-    for i, (desc, response) in enumerate(EXAMPLES, 1):
+    for i, (desc, response) in enumerate(EXAMPLES_WORKFLOW, 1):
         print(f"\n{i}. {desc}")
         print(f"   {handle_response_simple(response)}")
 
@@ -137,6 +203,11 @@ def handle_response_granular(response):
     errors._consecutive_retries = 0
 
     print("\n\n=== Part 2: Granular API (individual components) ===")
-    for i, (desc, response) in enumerate(EXAMPLES, 1):
+    for i, (desc, response) in enumerate(EXAMPLES_WORKFLOW, 1):
         print(f"\n{i}. {desc}")
         print(f"   {handle_response_granular(response)}")
+
+    print("\n\n=== Part 3: Respond tool ===")
+    for i, (desc, response) in enumerate(EXAMPLES_RESPOND, 1):
+        print(f"\n{i}. {desc}")
+        print(f"   {handle_response_with_respond(response)}")

src/forge/__init__.py

@@ -26,6 +26,7 @@
     detect_hardware,
 )
 from forge.server import BudgetMode, ServerManager, setup_backend
+from forge.tools import RESPOND_TOOL_NAME, respond_spec, respond_tool
 from forge.prompts import build_tool_prompt, extract_tool_call, rescue_tool_call, retry_nudge, step_nudge
 from forge.guardrails import (
     CheckResult,
@@ -100,6 +101,10 @@
     "BudgetMode",
     "ServerManager",
     "setup_backend",
+    # Built-in tools
+    "RESPOND_TOOL_NAME",
+    "respond_spec",
+    "respond_tool",
     # Guardrails
     "CheckResult",
     "Guardrails",

src/forge/proxy/handler.py

@@ -19,6 +19,7 @@
     text_response_to_openai,
     text_to_sse_events,
 )
+from forge.tools.respond import RESPOND_TOOL_NAME, respond_spec
 
 logger = logging.getLogger("forge.proxy")
 
@@ -79,6 +80,14 @@ async def handle_chat_completions(
     # Convert inbound
     messages = openai_to_messages(openai_messages)
     tool_specs = _extract_tool_specs(request_tools)
+
+    # Inject respond tool when tools are present.  The model calls
+    # respond(message="...") instead of producing bare text, keeping it
+    # in tool-calling mode where guardrails apply.  The respond call is
+    # stripped from the outbound response — the client never sees it.
+    if tool_specs and not any(s.name == RESPOND_TOOL_NAME for s in tool_specs):
+        tool_specs.append(respond_spec())
+
     tool_names = _extract_tool_names(tool_specs)
 
     # No tools → plain chat completion, no guardrails needed.
@@ -133,7 +142,28 @@ async def handle_chat_completions(
 
     tool_calls = result.response
 
-    # Convert outbound
+    # Strip respond() calls — convert to plain text for the client.
+    # If the model called respond(message="..."), the client sees a
+    # normal text response (finish_reason="stop"), not a tool call.
+    respond_calls = [tc for tc in tool_calls if tc.tool == RESPOND_TOOL_NAME]
+    other_calls = [tc for tc in tool_calls if tc.tool != RESPOND_TOOL_NAME]
+
+    if respond_calls and not other_calls:
+        # Pure respond — convert to text
+        text = respond_calls[0].args.get("message", "")
+        logger.info("Stripping respond() call, returning as text")
+        if is_stream:
+            return text_to_sse_events(text, model=model_name)
+        return text_response_to_openai(text, model=model_name)
+
+    if other_calls:
+        # Real tool calls (possibly mixed with respond) — return the
+        # real tool calls only, drop respond.
+        if is_stream:
+            return tool_calls_to_sse_events(other_calls, model=model_name)
+        return tool_calls_to_openai(other_calls, model=model_name)
+
+    # Shouldn't happen, but handle empty tool_calls gracefully
     if is_stream:
-        return tool_calls_to_sse_events(tool_calls, model=model_name)
-    return tool_calls_to_openai(tool_calls, model=model_name)
+        return text_to_sse_events("", model=model_name)
+    return text_response_to_openai("", model=model_name)

src/forge/tools/__init__.py

@@ -0,0 +1,13 @@
+"""Built-in tools provided by forge."""
+
+from forge.tools.respond import (
+    RESPOND_TOOL_NAME,
+    respond_spec,
+    respond_tool,
+)
+
+__all__ = [
+    "RESPOND_TOOL_NAME",
+    "respond_spec",
+    "respond_tool",
+]

src/forge/tools/respond.py

@@ -0,0 +1,61 @@
+"""Synthetic respond tool — structured alternative to bare text responses.
+
+The model calls respond(message="...") instead of producing bare text.
+This keeps the model in tool-calling mode where forge's full guardrail
+stack applies, eliminating the trust_text_intent tradeoff.
+
+Usage:
+
+    WorkflowRunner consumers:
+        tools["respond"] = respond_tool()
+        workflow = Workflow(..., tools=tools, terminal_tool="respond")
+
+    Proxy:
+        Injected automatically. respond() calls are converted to plain
+        text responses before returning to the client.
+"""
+
+from __future__ import annotations
+
+from pydantic import BaseModel, Field
+
+from forge.core.workflow import ToolDef, ToolSpec
+
+
+RESPOND_TOOL_NAME = "respond"
+
+RESPOND_DESCRIPTION = (
+    "Respond to the user with a message. Use this when the user is chatting, "
+    "asking a question, when you need to ask a clarifying question before "
+    "proceeding, or when no other tool action is needed. Also use this "
+    "after completing the user's request to report the result."
+)
+
+
+class RespondParams(BaseModel):
+    message: str = Field(description="The message to send to the user.")
+
+
+def respond_spec() -> ToolSpec:
+    """Return the ToolSpec for the respond tool (what the LLM sees)."""
+    return ToolSpec(
+        name=RESPOND_TOOL_NAME,
+        description=RESPOND_DESCRIPTION,
+        parameters=RespondParams,
+    )
+
+
+def respond_tool() -> ToolDef:
+    """Return a complete ToolDef for the respond tool.
+
+    The callable simply returns the message string. Use as a terminal
+    tool in WorkflowRunner workflows.
+    """
+
+    def _respond(message: str) -> str:
+        return message
+
+    return ToolDef(
+        spec=respond_spec(),
+        callable=_respond,
+    )