A single self-hosted server that aggregates the free tiers of 16 major LLM providers behind one unified API key — speaking both the OpenAI wire format (/v1/chat/completions, /v1/responses) and the Anthropic Messages format (/v1/messages). Point an OpenAI client, the Anthropic SDK, or Claude Code at the same base URL. A smart router picks the best available model per request, fails over when one is rate-limited, and tracks per-key usage so you never blow a free-tier cap.
Quick start · Providers · API docs · How it works · Contributing
Every serious AI lab now ships a free tier — a few million tokens a month, a few thousand requests a day. On its own, each one is a toy. Stacked together, they add up to roughly 1.7 billion tokens per month of working inference, across 100+ models from small-and-fast to genuinely capable.
The catch is that stacking them by hand is miserable: sixteen SDKs, sixteen rate limits, sixteen places a request can fail. FreeAIGateway collapses all of that into one gateway that speaks two protocols — OpenAI (/v1/chat/completions, /v1/responses) and Anthropic Messages (/v1/messages). Point any compatible client at a single base URL, and the gateway routes transparently across whichever providers you've added keys for — with optional prompt caching and built-in web-search, web-extract, and image-generation tools layered on top.
TL;DR — A drop-in, self-hosted, OpenAI and Anthropic-compatible proxy that turns 16 free tiers into one reliable endpoint with automatic failover, encrypted key storage, and a full admin dashboard.
| Protocols | OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/images/generations) · Anthropic Messages (/v1/messages) |
| Providers | 16 free tiers + any custom OpenAI-compatible endpoint |
| Capacity | ~1.7B tokens/month aggregate · 100+ models |
| Routing | Priority-based with health checks, rate-limit ledger, automatic failover (up to 20 attempts) |
| Auth | One unified freellmapi-… key for apps · email + password for the dashboard |
| Security | AES-256-GCM encrypted key storage in SQLite |
| Footprint | Node 20+ · multi-arch Docker (amd64 + arm64, Raspberry Pi ready) · ~40 MB RSS at idle |
| Extras | Prompt cache · built-in agent tools · streaming · tool calling · vision input · embeddings |
| License | MIT |
➕ Custom provider — point at any OpenAI-compatible endpoint (llama.cpp, LM Studio, vLLM, a local Ollama, or a remote gateway) directly from the Keys page.
- OpenAI-compatible —
POST /v1/chat/completionsandGET /v1/modelswork with the official OpenAI SDKs and any OpenAI-compatible client (LangChain, LlamaIndex, Continue, Hermes, …). Just changebase_url. - Anthropic-compatible —
POST /v1/messages(plusPOST /v1/messages/count_tokens) speaks the native Claude Messages wire format, so Claude Code, the Anthropic SDKs, and any Messages-API client point straight at the gateway and get answered by any free provider behind it. Full SSE streaming (message_start→content_block_delta→message_stop), tool use (tool_use/tool_result), system prompts, and image blocks are translated over the same router. Authenticates viax-api-keyorAuthorization: Bearer. Unknown Claude model ids (e.g.claude-sonnet-4-5) transparently fall through to auto-routing. - Responses API —
POST /v1/responses(the wire format current Codex CLI versions require) is a translating shim over the same router, with full streaming events and tool calls.
- Automatic failover — on a 429, 5xx, or timeout the router skips the provider, puts the key on a short cooldown, and retries the next model in your fallback chain (up to 20 attempts).
- Per-key rate tracking — RPM, RPD, TPM, and TPD counters per
(platform, model, key), so the router always picks a key that's under its caps. - Sticky sessions — multi-turn conversations keep talking to the same model for 30 minutes, avoiding the hallucination spike from mid-conversation model switches.
- Health checks — periodic probes mark keys
healthy,rate_limited,invalid, orerrorso dead keys are skipped automatically.
- Prompt cache — opt-in in-memory TTL cache. An identical request returns the stored completion with
X-Cache: HITand no provider call — saving free-tier quota and answering instantly. Works for streaming (replayed as SSE) and non-streaming; bypass per-request withx-cache: no-store. Live hit-rate/entries stats in the dashboard. - Streaming & non-streaming — Server-Sent Events for
stream: true, JSON otherwise. Every provider adapter implements both.
- Built-in agent tools — the gateway can run tools itself, turning auto-routed chat into a lightweight agent: web_search (live web via r.jina.ai over DuckDuckGo), web_extract (URL → clean markdown), and generate_image (text → PNG saved server-side and appended as an inline markdown image). On by default, individually toggleable in Settings; applied only to auto-routed, tool-free, non-streaming requests, and bypassable with
x-builtin-tools: off. - Tool calling — OpenAI-style
tools/tool_choicerequests pass through, and assistanttool_calls+tool-role follow-ups round-trip across providers. - Vision input — standard OpenAI
image_urlblocks restrict routing to vision-capable models automatically. - Image generation —
POST /v1/images/generations(OpenAI Images shape) backed by a0.dev's keyless text-to-image endpoint. No provider key needed. - Embeddings —
/v1/embeddingswith family-based routing: failover only happens between providers serving the same model, never across models.
- Encrypted key storage — provider keys are AES-256-GCM encrypted before hitting SQLite; decryption happens in-memory just before a request.
- Unified API key — clients authenticate with a single
freellmapi-…bearer token. Upstream provider keys are never exposed to your apps. - Dashboard login — the admin UI and
/api/*routes are gated behind an email + password account (scrypt-hashed, session-token auth), set on first run. - Admin dashboard — React + Vite UI to manage keys, reorder the fallback chain, inspect analytics, and run prompts in a playground. Dark mode included.
- Analytics — per-request logging with latency, token counts, success rate, and per-provider breakdowns.
- Runs anywhere Node 20+ runs — Windows, macOS, Linux, or an ARM SBC (Raspberry Pi included). ~40 MB RSS at idle behind PM2 / systemd / any supervisor.
The scope is deliberately narrow. If a feature isn't listed above and isn't here, assume it isn't there yet — PRs welcome (see Contributing).
- Audio / speech (
/v1/audio/*) - Legacy completions (
/v1/completions) — only the chat endpoint is implemented - Moderation (
/v1/moderations) n > 1for chat — supported on/v1/images/generations, not on chat- Per-user billing / multi-tenant auth — single-user by design
Recommended: Docker Compose. Runs the API and dashboard together on port
3001and persists SQLite in a named volume.Prerequisites: Docker, Docker Compose, OpenSSL.
git clone https://github.com/Hansade2005/FreeAIGateway.git
cd FreeAIGateway
# Generate an encryption key for at-rest key storage
ENCRYPTION_KEY="$(openssl rand -hex 32)"
printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env
docker compose up -dOpen http://localhost:3001, add your provider keys on the Keys page, reorder the Fallback Chain to taste, and grab your unified API key from the Keys page header. That unified key is what you point your OpenAI or Anthropic SDK at.
Reaching it from another machine? By default the container is published only on
127.0.0.1, sohttp://<server-ip>:3001won't load from another device. To expose it on your LAN — e.g. a Raspberry Pi athttp://192.168.1.x:3001— start it withHOST_BIND=0.0.0.0:HOST_BIND=0.0.0.0 docker compose up -dOnly do this on a trusted network: the proxy is single-user and guarded only by the unified API key.
Local development (Node.js)
Prerequisites: Node.js 20+, npm.
git clone https://github.com/Hansade2005/FreeAIGateway.git
cd FreeAIGateway
npm install
cp .env.example .env
ENCRYPTION_KEY="$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')"
printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env
npm run devENCRYPTION_KEY is required for startup. The server only falls back to a database-stored development key when DEV_MODE=true and NODE_ENV is not production; do not use that fallback with real provider keys.
Request analytics are retained for 90 days or 100,000 request rows by default, whichever prunes first. Set REQUEST_ANALYTICS_RETENTION_DAYS=0 or REQUEST_ANALYTICS_MAX_ROWS=0 in .env to disable either limit.
Open http://localhost:5173 (the Vite dev UI) and configure as above.
Production build without Docker
npm run build
node server/dist/index.js # server + dashboard both served on :3001FreeAIGateway publishes a single production image containing the Express server and the built React dashboard:
docker pull ghcr.io/Hansade2005/FreeAIGateway:latest # or pin a release, e.g. :v1.2.3The image is multi-arch (linux/amd64 + linux/arm64, so it runs on a Raspberry Pi). Published tags: latest (default branch), v*.*.* (git release tags), and sha-<commit>.
The included docker-compose.yml is the recommended install path:
docker compose up -d
docker compose logs -f freeaigatewayBy default the container's port binds to 127.0.0.1 (localhost only). To reach the dashboard/API from another machine, publish on all interfaces with HOST_BIND=0.0.0.0 docker compose up -d — only on a trusted LAN, since the proxy is single-user.
SQLite data lives in the freellmapi-data volume at /app/server/data. Keep the same .env ENCRYPTION_KEY and volume when upgrading, because provider keys are encrypted at rest.
More Docker operations and examples live in docker/README.md.
A native menu-bar app lives in desktop/: the entire router + dashboard running locally from your tray, with a glass popover showing live request stats.
No published binaries — it builds from this repo in a few minutes:
npm install
npm run desktop:dist # macOS: desktop/dist-electron/FreeAIGateway-…-arm64.dmg
npm run desktop:dist:win # Windows installerWindows: the build config is in place but not yet tested — a quick report (working or not) in an issue would be much appreciated.
Locally built apps launch without Gatekeeper/SmartScreen warnings — no code signing involved. Full instructions in desktop/README.md.
Any OpenAI-compatible client works — just change the base_url.
OpenAI · Python
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:3001/v1",
api_key="freellmapi-your-unified-key",
)
resp = client.chat.completions.create(
model="auto", # let the router pick; or specify e.g. "gemini-2.5-flash"
messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)
print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))OpenAI · curl
curl http://localhost:3001/v1/chat/completions \
-H "Authorization: Bearer freellmapi-your-unified-key" \
-H "Content-Type: application/json" \
-d '{
"model": "auto",
"messages": [{"role": "user", "content": "hi"}]
}'OpenAI · streaming
stream = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)The same server speaks the native Anthropic Messages wire format at POST /v1/messages, so Claude-native clients work unchanged — just point the base URL at the gateway. The router still picks whichever free provider is available; the model field is accepted but real Claude ids fall through to auto-routing.
Anthropic Python SDK
from anthropic import Anthropic
client = Anthropic(
base_url="http://localhost:3001",
api_key="freellmapi-your-unified-key",
)
msg = client.messages.create(
model="claude-sonnet-4-5", # accepted; routed to a free model
max_tokens=256,
messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)
print(msg.content[0].text)Claude Code
export ANTHROPIC_BASE_URL="http://localhost:3001"
export ANTHROPIC_API_KEY="freellmapi-your-unified-key"
claudeAnthropic · curl
curl http://localhost:3001/v1/messages \
-H "x-api-key: freellmapi-your-unified-key" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-4-5",
"max_tokens": 256,
"messages": [{"role": "user", "content": "hi"}]
}'Streaming ("stream": true) emits the full Anthropic SSE sequence — message_start, content_block_start/delta/stop, message_delta, message_stop — and tool use round-trips via tool_use / tool_result blocks. POST /v1/messages/count_tokens returns an input_tokens estimate for context sizing.
curl http://localhost:3001/v1/images/generations \
-H "Authorization: Bearer freellmapi-your-unified-key" \
-H "Content-Type: application/json" \
-d '{ "prompt": "a neon city skyline at dusk", "size": "1792x1024" }'
# → { "data": [{ "url": "https://api.a0.dev/assets/image?text=...&aspect=16:9" }] }Works with the OpenAI Images SDK (client.images.generate(...)). size maps to an aspect ratio, or pass "aspect": "16:9" | "9:16" | "1:1" directly; "response_format": "b64_json" inlines the bytes.
Pass OpenAI-style tools and tool_choice; the assistant response round-trips back through the proxy exactly like the OpenAI API. Multi-step flows (assistant tool_calls → tool-role follow-up → final answer) work across every provider the router can reach.
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
# 1. Model asks for a tool call
first = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
tools=tools,
tool_choice="required",
)
call = first.choices[0].message.tool_calls[0]
# 2. You execute the tool, feed the result back
final = client.chat.completions.create(
model="auto",
messages=[
{"role": "user", "content": "What's the weather in Karachi?"},
first.choices[0].message,
{"role": "tool", "tool_call_id": call.id, "content": '{"temp_c": 32, "cond": "sunny"}'},
],
tools=tools,
)
print(final.choices[0].message.content)Works with stream=True too — you'll get delta.tool_calls chunks followed by a finish_reason: "tool_calls" close. Under the hood, OpenAI-compatible providers get the request passed through; Gemini requests are translated into Google's functionDeclarations / functionResponse shape and back.
Send images with standard OpenAI image_url content blocks (base64 data: URLs or http(s) URLs). When a request contains an image, the router restricts itself to vision-capable models and ignores text-only ones (tagged with a Vision badge on the Fallback Chain page — currently Gemini 2.5/3.x, Llama 4 Scout/Maverick on Groq/NVIDIA/SambaNova, and GitHub's GPT-4o / GPT-4.1).
resp = client.chat.completions.create(
model="auto", # auto-routes to a vision model
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}},
],
}],
)
print(resp.choices[0].message.content)If no vision-capable model is enabled, an image request returns a clear 422 (code: "no_vision_model") rather than silently dropping the image. (Image input on /v1/responses isn't supported yet — use /v1/chat/completions.)
Every response carries an
X-Routed-Via: <platform>/<model>header so you can see which provider served each call. If a request fell over between providers, you'll also seeX-Fallback-Attempts: N.
/v1/embeddings is OpenAI-compatible with one deliberate difference: failover never crosses models. Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top. So embeddings route by family (one model identity + dimension), and failover only walks the providers serving that same family.
resp = client.embeddings.create(
model="auto", # default family; or a family name like "bge-m3"
input=["the quick brown fox", "pack my box with five dozen liquor jugs"],
)
print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")Family (model) |
Dims | Providers (failover order) |
|---|---|---|
gemini-embedding-001 (default) |
3072 | |
text-embedding-3-large |
3072 | GitHub Models |
text-embedding-3-small |
1536 | GitHub Models |
embed-v4.0 |
1536 | Cohere |
bge-m3 |
1024 | Cloudflare → Hugging Face |
qwen3-embedding-0.6b |
1024 | Cloudflare |
nv-embedqa-e5-v5 |
1024 | NVIDIA |
llama-nemotron-embed-1b-v2 |
2048 | NVIDIA |
llama-nemotron-embed-vl-1b-v2 |
2048 | NVIDIA → OpenRouter |
embeddinggemma-300m |
768 | Cloudflare |
model accepts auto (configured default), a family name, or a provider-specific model id (resolved to its family). Defaults and per-provider priorities live on the dashboard's Models → Embeddings page. Pick a family once and stick with it for a given vector store — that's the whole point.
|
Models & routing — tune the routing strategy or drag a manual fallback order, watch live reliability / speed / intelligence scores, and track the per-provider monthly token budget.
|
Settings — your unified key, the OpenAI and Anthropic endpoints, the prompt cache (toggle / TTL / live stats), and the built-in tools, all in one place.
|
|
Playground — a pro console: switch between OpenAI and Anthropic protocols, stream responses, compare models side by side, generate images, and read per-turn route / cache / latency / token badges.
|
Analytics — request volume, success rate, tokens, estimated savings, latency percentiles (p50/p95/p99 + TTFB), prompt-cache hit-rate, per-model breakdown, and a live request log.
|
┌──────────────────┐ Bearer / x-api-key ┌─────────────────────────┐
│ OpenAI SDK or │ ──────────────────────▶ │ Express proxy (:3001) │
│ Anthropic SDK / │ │ /v1/chat/completions │
│ Claude Code / │ ◀────────────────────── │ /v1/responses │
│ curl / any │ streamed tokens │ /v1/messages │
└──────────────────┘ └────────────┬────────────┘
│
(Anthropic & Responses requests are
translated to the internal chat
format, then share the router below)
▼
┌────────────────────────────────────────────────┐
│ Router │
│ 1. Pick highest-priority model that │
│ (a) has a healthy key and │
│ (b) is under all its rate limits. │
│ 2. Decrypt key, call provider SDK. │
│ 3. On 429/5xx → cooldown + retry next model. │
└────────────────────────────────────────────────┘
│
┌──────────────┬────────────┬──────────┴─────────┬─────────────┬──────────┐
▼ ▼ ▼ ▼ ▼ ▼
Google Groq Cerebras OpenRouter HF …10 more
| Component | Path | Role |
|---|---|---|
| Router | server/src/services/router.ts |
Picks a model per request |
| Rate-limit ledger | server/src/services/ratelimit.ts |
In-memory RPM/RPD/TPM/TPD counters backed by SQLite, with cooldowns on 429s |
| Provider adapters | server/src/providers/*.ts |
One file per provider implementing chatCompletion() and streamChatCompletion() |
| Health service | server/src/services/health.ts |
Periodic probe keeps key status fresh |
| Dashboard | client/ |
React + Vite + shadcn/ui admin surface |
| Storage | SQLite (better-sqlite3) |
AES-256-GCM envelope encryption for keys |
Stacking free tiers has real trade-offs. Be honest with yourself about them:
- No frontier models. The free-tier catalog tops out around Llama 3.3 70B, GLM-4.5, Qwen 3 Coder, and Gemini 2.5 Pro. You will not get GPT-5 or Claude Opus-class reasoning through this. For hard problems, pay for a real API.
- Intelligence degrades through the day. Your top-ranked models have the lowest daily caps; once they hit them, the router falls down the chain to smaller models. Expect effective intelligence to drop in the late hours of each day, then reset at UTC midnight.
- Latency is highly variable. Cerebras and Groq are extremely fast; others are not. You get whichever one is available.
- Free tiers change without notice. Providers tighten, loosen, or remove free tiers regularly. When that happens you'll see 429s or auth errors until you update the catalog. Re-seed scripts live in
server/src/scripts/. - No SLA, by definition. If you need reliability, use a paid provider with a contract.
- Local-first. No multi-tenant auth. Run this for yourself; don't expose it to the internet.
Contributors very welcome! Good first PRs:
- Add a provider — copy
server/src/providers/openai-compat.tsas a template, wire it intoserver/src/providers/index.ts, seed its models inserver/src/db/index.ts, add a test inserver/src/__tests__/providers/. - Add an endpoint — images, moderations, audio. The provider base class can grow new methods; adapters declare which they support.
- Improve the router — cost-aware routing, latency-weighted priority, regional pinning.
- Dashboard polish — Analytics charts, key rotation UX, batch import of keys from
.env. - Docs — more examples, client snippets for Go/Rust, deployment recipes.
Development loop:
npm install
npm run dev # server on :3001, dashboard on :5173, both with HMR
npm test # server vitest; also runs client tests if the workspace adds them
npm run build # compile server and dashboardPRs should include a test, keep the suite green, and match the .editorconfig / tsconfig defaults already in the repo. Issues and discussions are open.
A self-hosted, single-user, personal-use setup was re-reviewed against each provider's ToS (May 2026). Summary:
| Provider | Verdict | Notes |
|---|---|---|
| Google Gemini | March 2026 ToS narrows scope to "professional or business purposes, not for consumer use" — a self-hosted dev proxy is still defensible, but the clause is new. | |
| Groq | ✅ Likely OK | GroqCloud Services Agreement permits Customer Application integration. |
| Cerebras | ✅ Likely OK | Permitted; explicitly forbids selling/transferring API keys. |
| Mistral | ✅ Likely OK | APIs allowed for personal/internal business use. |
| OpenRouter | ✅ Likely OK | April 2026 ToS sharpens the no-resale clause; private single-user proxy still fine. |
| SambaNova | EULA §1.5(c) blocks resale and "service bureau" use; single-user with no third-party access is fine. | |
| Cloudflare Workers AI | No anti-proxy clause; covered by general Self-Serve Subscription Agreement. | |
| NVIDIA NIM | Trial ToS §1.2 / §1.4: "evaluation only, not production." Disabled in default catalog. | |
| GitHub Models | Free tier explicitly scoped to "experimentation" and "prototyping." | |
| Cohere | ❌ Avoid | Terms §14 still forbids "personal, family or household purposes." |
| Zhipu (open.bigmodel.cn) | ✅ Likely OK | Personal/non-commercial research carve-out still in the platform docs. |
| Z.ai (api.z.ai) | Singapore entity (distinct from Zhipu CN). §III.3(l) anti-traffic-redirect clause could be read against a proxy; no explicit personal-use carve-out. | |
| Ollama Cloud | ✅ Likely OK | Free plan permits cloud-model access (1 concurrent, 5-hour session caps). No anti-proxy / anti-resale clauses found. (Integration tracked in #14.) |
Rules of thumb that keep most providers happy: one account per provider · no reselling · no sharing your endpoint with other humans · don't hammer a free tier as a paid production backend. This is informational, not legal advice — read each provider's ToS and make your own call.
Removed since the April 2026 review: Hugging Face, Moonshot, and MiniMax direct integrations were dropped (HF — tool-call format issues; Moonshot — paid only; MiniMax — superseded by the OpenRouter
minimax/minimax-m2.5:freeroute).
This project is for personal experimentation and learning, not production. Free tiers exist so developers can prototype against them; they aren't a stable, supported inference substrate. If you build something real on top of FreeAIGateway, swap in a paid API before you ship. Your relationship with each upstream provider is governed by the terms you accepted when you created your account — those terms still apply when traffic is proxied through this project, and you're responsible for complying with them.
If FreeAIGateway saves you a few API bills, consider leaving a ⭐ — it genuinely helps.