Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ COPY packages/adapters/gemini-local/package.json packages/adapters/gemini-local/
COPY packages/adapters/openclaw-gateway/package.json packages/adapters/openclaw-gateway/
COPY packages/adapters/opencode-local/package.json packages/adapters/opencode-local/
COPY packages/adapters/pi-local/package.json packages/adapters/pi-local/
COPY packages/adapters/litellm-gateway/package.json packages/adapters/litellm-gateway/

RUN pnpm install --frozen-lockfile

Expand Down
145 changes: 145 additions & 0 deletions docs/adapters/litellm-gateway.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
---
title: LiteLLM Gateway
summary: LiteLLM proxy gateway adapter for unified multi-provider access
---

The `litellm_gateway` adapter connects to a LiteLLM proxy server, enabling unified access to multiple LLM providers (OpenAI, Anthropic, Cohere, etc.) through a single gateway endpoint.

## When to Use

- Centralized LLM provider management via LiteLLM proxy
- Load balancing across multiple LLM providers
- Cost tracking and fallback routing via LiteLLM
- Need to switch between providers without changing agent config

## When Not to Use

- Direct provider access is available and preferred (use `claude_local`, `codex_local`, etc.)
- WebSocket streaming is required (LiteLLM uses HTTP/SSE)
- Running agents that need local CLI tools

## Prerequisites

- LiteLLM proxy server running and accessible
- LiteLLM configured with desired model mappings
- API key for LiteLLM proxy (if authentication is enabled)

## Configuration Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `baseUrl` | string | Yes | LiteLLM proxy base URL (e.g., `http://localhost:4000`) |
| `model` | string | Yes | Model identifier as configured in LiteLLM |
| `apiKey` | string | No | API key for LiteLLM proxy (or use `LITELLM_API_KEY` env var) |
| `promptTemplate` | string | No | Prompt template for all runs (default: `You are agent {{agent.id}} ({{agent.name}}). Continue your Paperclip work.`) |
| `temperature` | number | No | Sampling temperature 0.0-2.0 (default: 0.7) |
| `maxTokens` | number | No | Maximum output tokens (default: 4096) |
| `timeoutSec` | number | No | Request timeout in seconds (default: 300) |
| `headers` | object | No | Additional HTTP headers for requests |

## How It Works

1. Paperclip renders the `promptTemplate` with context variables (agent, run, task info)
2. The rendered prompt is sent to LiteLLM via OpenAI-compatible `/v1/chat/completions` API
3. LiteLLM routes the request to the configured provider
4. Responses stream back via SSE with real-time output
5. Token usage is tracked and returned in the result

## Prompt Templates

Templates support `{{variable}}` substitution:

| Variable | Value |
|----------|-------|
| `{{agentId}}` | Agent's ID |
| `{{companyId}}` | Company ID |
| `{{runId}}` | Current run ID |
| `{{agent.name}}` | Agent's name |
| `{{agent.id}}` | Agent's ID |
| `{{context.taskId}}` | Current task/issue ID |
| `{{context.wakeReason}}` | Wake reason (e.g., `issue_assigned`) |

Example:
```json
{
"promptTemplate": "You are {{agent.name}}, working on task {{context.taskId}}. Wake reason: {{context.wakeReason}}."
}
```

## Model Discovery

The adapter automatically fetches available models from `GET {baseUrl}/v1/models` and caches them for 60 seconds. Models appear in the UI dropdown when creating or editing agents.

If discovery fails, the adapter falls back to cached results or an empty list.

## API Key Resolution

The adapter resolves API keys in this order:
1. `LITELLM_API_KEY` environment variable (highest priority)
2. `apiKey` field in agent config
3. No authentication (if LiteLLM proxy doesn't require auth)

## LiteLLM Configuration Example

LiteLLM proxy `config.yaml`:
```yaml
model_list:
- model_name: gpt-4
litellm_params:
model: openai/gpt-4
api_key: os.environ/OPENAI_API_KEY

- model_name: claude-3-opus
litellm_params:
model: anthropic/claude-3-opus-20240229
api_key: os.environ/ANTHROPIC_API_KEY

- model_name: command-r-plus
litellm_params:
model: cohere/command-r-plus
api_key: os.environ/COHERE_API_KEY
```

Start LiteLLM:
```bash
litellm --config config.yaml --port 4000
```

## Paperclip Agent Configuration

```json
{
"type": "litellm_gateway",
"baseUrl": "http://localhost:4000",
"model": "gpt-4",
"temperature": 0.7,
"maxTokens": 4096,
"promptTemplate": "You are {{agent.name}}. Continue your Paperclip work."
}
```

## Environment Test

Use the "Test Environment" button in the UI to validate the adapter config. It checks:

- `baseUrl` is a valid HTTP/HTTPS URL
- API key configuration (env var or config field)
- Model specified in config
- Connectivity probe to `/v1/models` endpoint
- Shows status: ready, degraded, or not ready

## Token Usage Tracking

The adapter tracks token usage from the LiteLLM response:
- Input tokens (prompt)
- Output tokens (completion)
- Cached input tokens (if supported by provider)

Usage appears in run logs and billing reports.

## Limitations

- No session persistence (each run is stateless)
- Single-turn interaction per run (no conversation history)
- Requires LiteLLM proxy deployment
- HTTP/SSE only (no WebSocket streaming)
9 changes: 9 additions & 0 deletions packages/adapters/litellm-gateway/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# @paperclipai/adapter-litellm-gateway

## 0.1.0

Initial release.

- OpenAI-compatible streaming support
- Model discovery and caching
- Environment testing
37 changes: 37 additions & 0 deletions packages/adapters/litellm-gateway/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# @paperclipai/adapter-litellm-gateway

LiteLLM Gateway adapter for Paperclip.

## Features

- OpenAI-compatible streaming via `/v1/chat/completions`
- Automatic model discovery from `/v1/models`
- Environment variable or config-based API key authentication
- 60-second model cache per baseUrl
- Token usage tracking

## Configuration

```json
{
"baseUrl": "http://localhost:4000",
"apiKey": "sk-litellm-...",
"model": "gpt-4",
"temperature": 0.7,
"maxTokens": 4096
}
```

Or use `LITELLM_API_KEY` environment variable.

## Usage

Add to agent `adapterConfig`:

```typescript
{
type: "litellm_gateway",
baseUrl: "http://localhost:4000",
model: "gpt-4"
}
```
50 changes: 50 additions & 0 deletions packages/adapters/litellm-gateway/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
{
"name": "@paperclipai/adapter-litellm-gateway",
"version": "0.1.0",
"type": "module",
"exports": {
".": "./src/index.ts",
"./server": "./src/server/index.ts",
"./ui": "./src/ui/index.ts",
"./cli": "./src/cli/index.ts"
},
"publishConfig": {
"access": "public",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
},
"./server": {
"types": "./dist/server/index.d.ts",
"import": "./dist/server/index.js"
},
"./ui": {
"types": "./dist/ui/index.d.ts",
"import": "./dist/ui/index.js"
},
"./cli": {
"types": "./dist/cli/index.d.ts",
"import": "./dist/cli/index.js"
}
},
"main": "./dist/index.js",
"types": "./dist/index.d.ts"
},
"files": [
"dist"
],
"scripts": {
"build": "tsc",
"clean": "rm -rf dist",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@paperclipai/adapter-utils": "workspace:*",
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^24.6.0",
"typescript": "^5.7.3"
}
}
3 changes: 3 additions & 0 deletions packages/adapters/litellm-gateway/src/cli/format-event.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export function formatEvent(_event: unknown): string | null {
return null;
}
1 change: 1 addition & 0 deletions packages/adapters/litellm-gateway/src/cli/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export { formatEvent } from "./format-event.js";
62 changes: 62 additions & 0 deletions packages/adapters/litellm-gateway/src/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
export const type = "litellm_gateway";
export const label = "LiteLLM Gateway";

export const models: { id: string; label: string }[] = [];

export const agentConfigurationDoc = `# litellm_gateway agent configuration

Adapter: litellm_gateway

Use when:
- You want Paperclip to use a LiteLLM proxy as a unified gateway to multiple LLM providers.
- You need to access models through a centralized LiteLLM deployment.
- You want to leverage LiteLLM's load balancing, fallback, and cost tracking features.

Don't use when:
- You have direct API access to the provider (use provider-specific adapters instead).
- You need WebSocket streaming (LiteLLM uses HTTP/SSE).

Core fields:
- baseUrl (string, required): LiteLLM proxy base URL (e.g., http://localhost:4000)
- apiKey (string, optional): API key for LiteLLM proxy (can also use LITELLM_API_KEY env var)
- model (string, required): Model identifier as configured in LiteLLM (e.g., gpt-4, claude-3-opus)

Request behavior fields:
- timeoutSec (number, optional): Request timeout in seconds (default 300)
- maxTokens (number, optional): Maximum output tokens (default 4096)
- temperature (number, optional): Sampling temperature 0.0-2.0 (default 0.7)
- headers (object, optional): Additional HTTP headers to send with requests

Environment variables:
- LITELLM_API_KEY: API key for LiteLLM proxy (overridden by config.apiKey if set)

Model discovery:
- The adapter automatically fetches available models from \`GET {baseUrl}/v1/models\`
- Models are cached for 60 seconds per baseUrl
- If discovery fails, falls back to cached or empty list

LiteLLM configuration example:
\`\`\`yaml
# config.yaml for LiteLLM proxy
model_list:
- model_name: gpt-4
litellm_params:
model: openai/gpt-4
api_key: os.environ/OPENAI_API_KEY
- model_name: claude-3-opus
litellm_params:
model: anthropic/claude-3-opus-20240229
api_key: os.environ/ANTHROPIC_API_KEY
\`\`\`

Paperclip agent adapterConfig example:
\`\`\`json
{
"baseUrl": "http://localhost:4000",
"apiKey": "sk-litellm-...",
"model": "gpt-4",
"temperature": 0.7,
"maxTokens": 4096
}
\`\`\`
`;
Loading