Where forge is, and where it's going

[antoinezambelli]

Where forge is, and where it's going

Quick recap of the last couple weeks plus what's on deck. Things have moved fast since launch — three releases in a week — so here's the state of play.

What shipped

v0.7.0 — lineup refresh + tool-error channel. Prerequisite and step-enforcement violations now ride the tool-result channel instead of user-role nudges — models tend to treat a tool error as something to correct, where a user nudge often got merged into the prior turn and ignored. Also refreshed the evaluated-model lineup, added MODEL_REGISTRY.md (which models forge has actually been run against, with the sampling params used), and rewrote the docs.

v0.7.1 — forge works with Claude Code. The proxy now speaks the Anthropic Messages API (/v1/messages), so you can point Claude Code — or anything that speaks Anthropic — at a forge-guarded model. Two paths: translation for local OpenAI-compatible backends (llama.cpp, Ollama), and passthrough for Anthropic-shape downstreams. Plus --mode {native,prompt} and token-usage reporting when downstreams provide it.

v0.7.2 — vLLM backend. VLLMClient, in both managed and external proxy modes. Reads vLLM's server-side tool-call and reasoning fields directly rather than re-parsing.

Thanks to the folks who landed work this stretch — @mhajder (Docker, token usage, model-name passthrough) and @srinathh (vLLM served-name discovery) in particular. The issue tracker's been busy since launch and it's directly shaping what's below.

What's next

Evals on the 24–35B class. Formal coverage so far is the small tier (≤14B) plus the Anthropic API tier. Next eval push extends to the 24–35B class — the models that fit on one mid-range GPU, which is where a lot of self-hosted agentic work actually lives. Mistral Small 3.2 24B and Qwen 35B-A3B-class models are two we're specifically looking at, alongside others. Same scenario suite, new dataset, numbers reported the same way.

OpenAI-compatible hosted providers. forge currently assumes a local, unauthenticated backend, or the Anthropic SDK's own auth path. @lucasgerads has an open PR in review for hosted OpenAI-compatible providers — Cloudflare Workers AI, Fireworks, OpenRouter, that class — including bearer-token auth and provider-style base URLs. Thanks Lucas for the careful writeup in #88. Once this lands, forge should be much easier to use with small models in the cloud instead of only on local hardware.

Proxy hardening. Recent issue traffic and review work have surfaced a few proxy-path edge cases worth tightening before the next bigger feature push: slow-backend timeout behavior, backend disconnect diagnostics, and consistency across OpenAI-shape clients when providers return malformed tool calls or unexpected response shapes. None of this changes forge's core model, but it should make the proxy fail louder, recover more predictably, and be easier to debug when a backend behaves strangely.

Auth — the broader version. Hosted-provider auth is the first slice. A later release brings auth to the proxy layer generally: forge forwards your request headers to the backend as-is, with no opinion about which ones are "auth" or whether your credentials are valid — that part's between you and your backend. Goal is that whatever client you point at the proxy, with whatever auth the backend wants, it just works.

Notes

• Version numbers are sequencing, not promises — whatever's ready ships next and gets the next number. The above is direction, not a schedule.
• Releases are the stable surface; main is clean-ish but unblessed. Want a pinned, reproducible forge? Install from PyPI at a tagged version.

Issues, ideas, and PRs welcome.

[rofl0r]

Great news. Since you ask for ideas, here's a detail that's maybe worth keeping in mind - the number of dependencies. Frankly, given the age of this project, I was quite surprised by the amount of dependencies that were installed, including 2 that required binary wheels (pygments and something else). I always get an uncomfortable feeling when I see binaries getting pulled in by pip since afaik they are produced by the respective library authors so it's really hard to scrutinize them. Unlike packages provided by the distro, which are automatically built on the distro's build servers. Also given the common python community's idiom of using one venv per application, this essentially leads to each program having full separate copies of each library stored, and storage is no longer cheap (last time I checked, the cheapest 2 TB SSD now sells for $400, the price of an entry level laptop).

[antoinezambelli]

Thanks! I took a look at deps. I feel pretty good about how lean they are for a regular install/package. But they can get heavier with a [dev] install (which pulls in a bunch of testing related stuff. So a pip install -e ".[dev]" is going to a bit heavier, but not needed to run the core runtime, only if running tests/evals.

I might fold in a cleanup pass eventually...something like: test = ["pytest", "pytest-asyncio"], coverage = ["pytest-cov"], eval = ["anthropic", "mpmath"], and keep dev as the kitchen sink.

[mhajder]

I think the packages are added correctly; you don't need to install the optional dev or anthropic groups for this to work properly. By default, only pydantic and httpx are installed. You could use uv, which will symlink the packages, so if other projects on your computer have similar requirements, you won't use twice as much space.

Therefore, I would leave the packages in groups as they are added.

[mhajder]

I would suggest:

• Migrate the project to uv with package locking.
• Use ruff (linter) and ty (type checker), preferably called via prek (a new, better alternative to pre-commit). See my repository for an example: https://github.com/mhajder/librenms-mcp/blob/main/.pre-commit-config.yaml https://github.com/mhajder/librenms-mcp/blob/main/pyproject.toml
• Use conventional commits syntax in the project.
• Currently, if I remember correctly, the proxy does not pass any vision data, it should also be sent to the model.
• We should also add workflows for docker build and pypi release, something like: https://github.com/mhajder/librenms-mcp/tree/main/.github/workflows

If you like any of my suggestions, let me know and I will implement it.

[antoinezambelli]

Thanks @mhajder — and thanks again for the work you've already landed this stretch. Let me go through these, since they split into a few different buckets and a couple I'd genuinely welcome as PRs.

Repo tooling (uv + locking, conventional commits, prek): holding for now. These are all reasonable in isolation, but they're project-policy changes that add process surface, and I want to keep that stable while the proxy/backend hardening is mid-flight. Not a "no" forever — more "not while the foundation is still moving."

ruff (+ ty): this one I do want, just sequenced after the in-flight proxy/backend changes land — dropping a linter/type-checker onto a surface that's actively churning means fighting the same diffs twice. Once it settles I'll revisit, and there's one wrinkle I'll flag up front: there's an intentional unused line in server.py that needs to survive linting (it'll get a targeted noqa rather than being deleted), so whoever wires up ruff should expect that.

Vision passthrough: this is a real gap and a legitimate feature — you're right that the proxy doesn't forward vision content today. It's version-bump-scoped work, though, so I want to slot it deliberately after the current hardening flurry rather than inline. It's now on the roadmap.

Docker build CI: yes — a focused PR adding a Docker build workflow (build + validate on PR, like unit tests) is welcome, and that's a clean, self-contained piece if you'd like to take it. The one boundary: Docker publish and PyPI release automation stay owner-controlled — releasing is the one place I want to keep manual/credentialed rather than automated, at least for now.

If you want to pick up the Docker build CI, go for it — and thanks for offering to implement rather than just suggesting. That's appreciated.

[richardchen874-sys]

The part that stands out to me is "Where forge is, and where it's going". That is usually where OpenAI-compatible provider integration starts to matter in practice, because the integration works at small scale but gets harder once real users, retries, and longer sessions show up.

I would add a small provider capability check for streaming, tools, usage fields, model aliases, and auth mode before users run a real workflow. It prevents many compatible-endpoint failures from looking like model problems.

I am testing an OpenAI-compatible multi-model API layer around official Chinese models, so this kind of OpenAI-compatible provider integration is directly relevant. For forge, is the main pain today endpoint compatibility, model quality, or keeping usage/cost visible across providers?

[antoinezambelli]

Good timing — most of that capability list actually exists now; it shipped in the releases since this post was written:

• Auth mode: v0.8.0 did the "broader version" teased above. Forge forwards exactly one credential and relocates it across protocols (x-api-key ↔ Authorization: Bearer) when frontend and backend differ. Zero credentials against a gated backend fails loud with a 401 instead of an opaque downstream error; two at once is a 400.
• Model aliases: served-model-name discovery, plus an explicit --model override that pins the wire identity on multi-model gateways so an arbitrary listing never picks your model. Pair it with --budget-tokens and startup makes zero metadata probes.
• Capability probing: the context-length/served-name probe now runs on the first request, authenticated by that request's credential, so gated backends don't 401 at startup.
• Usage fields: token usage is reported when the downstream provides it (since v0.7.1).
• Tools: models that emit tool calls as free text get rescue-parsed (fenced JSON, Qwen <tool_call> XML, Mistral [TOOL_CALLS], and a couple of other syntaxes) rather than failing the request, and malformed backend responses fail loud instead of leaking into the conversation — v0.8.1 hardened the llama.cpp side of that specifically.

What doesn't exist is your actual suggestion's shape: a user-facing preflight command that runs those checks and prints a verdict before a real workflow. Today it's implicit — you find out on the first request, with (now) clear errors. A forge check-backend diagnostic is a reasonable idea; filed as an enhancement if there's interest.

To your question: endpoint compatibility, clearly. The long tail of "OpenAI-compatible" servers that are almost-but-not-quite — nonstandard tool-call syntax, missing usage fields, surprising error shapes — is where most issue traffic comes from. Model quality is the user's choice (MODEL_REGISTRY.md documents what we've validated), and cost visibility across providers is out of scope for now.