forge v0.7.4 — an open model reached frontier parity on our agentic-reliability eval (caveats inside)

[antoinezambelli]

On forge's agentic tool-calling eval, Qwen3.6-35B-A3B (Q4, llama.cpp, native tool-calling), reforged, scores 94.8% — a statistical tie with claude-haiku-4-5 reforged at 94.5%. First time I've seen an open, locally-runnable model reach the frontier tier on this suite. The caveats are important.

What this is, and what it isn't

• One narrow eval. It measures agentic tool-calling reliability — well-formed calls, right-tool selection, error recovery, staying on protocol over multi-step tasks. Nothing about coding, reasoning depth, knowledge, or long-horizon capability. Observability milestone, not "Qwen beats Claude."
• "Reforged" = the model plus forge's retry/guardrail layer, nothing more. This is bare model vs. forge's error-correction — not bare vs. a full orchestration harness (Claude Code, Pi, etc.) wrapped around the model, and not forge vs. a harness either (forge isn't that kind of tool). Just the guardrail impact, isolated.
• Parity, not a win. 94.8 vs 94.5 is 0.3 points on N=50 — inside the noise. "Qwen3.6-35B reached the haiku tier," not "beat it." It does not approach sonnet-reforged (98.4) or opus-reforged (99.2).
• Frontier numbers are carried forward from an earlier eval generation (badged as lagging on the dashboard); a re-sweep refreshes them.

The lift itself is the usual forge story you already know: bare, claude-haiku-4-5 sits at 46.5% on this suite and opus at 87.9%; reforged, 94.5% and 99.2%. The same layer carries Qwen3.6-35B into the haiku tier, and puts four open reforged configs above bare opus. The eval is just one lens, of course.

Malformed tool-call args now ride the tool-error channel

When a model emits a structurally valid call whose arguments are unparseable or not an object, forge now returns a tool-error result (role="tool", anchored to the model's own tool_call_id) instead of a trailing user-role retry nudge.

Plainly, on the evidence: this came out of a private, higher-difficulty agentic eval I've been dogfooding, where that failure mode actually shows up — routing the correction onto the tool channel looked better there. The published suite above doesn't exercise this error type much, so it isn't really reflected in those numbers. I'm calling it a conditioning bet — a model in native tool-calling mode may self-correct better on the channel it was pretrained on — not a claim that malformed JSON "is" a runtime tool error. Native mode only; prompt mode keeps the prior behavior. I didn't see regressions in testing, but the honest measurement comes with the next eval generation. Rationale in ADR-016.

Links

• Release: https://github.com/antoinezambelli/forge/releases/tag/v0.7.4
• Dashboard / full tables: docs/results/
• ADR-016: https://github.com/antoinezambelli/forge/blob/main/docs/decisions/016-malformed-args-tool-error-channel.md

---

PS: not all configs were shining bright this run. Ministral 3 has topped the 4-14B bracket since the original CAIS publication through v0.6.0 to today, but Mistral Small 3.2 only pulled a lukewarm ~78%. If anyone has any good llama.cpp settings/model params that pull better numbers, let me know!