Skip to content

Commit a8ed6e9

Browse files
missBergclaude
missBerg
and
claude
authored
docs: add prompt caching capability guide (#1949)
**Description** Adds a capability guide for provider-agnostic prompt caching via the unified cache_control API. Documents support across Anthropic Direct, GCP Vertex AI (Claude), and AWS Bedrock (Claude) with automatic provider-specific translation. This feature was previously only documented in `examples/cache/cache_control.md` and not linked from the main documentation site, making it difficult for adopters to discover. Includes practical JSON examples for system prompt caching, multiple cache points, and tool definition caching, plus response format documentation and best practices. **Related Issues/PRs (if applicable)** Related #1679 **Special notes for reviewers (if applicable)** Content derived from `examples/cache/cache_control.md` and restructured as a proper capability guide with links back to provider-specific setup guides. --------- Signed-off-by: Erica Hughberg <erica.sundberg.90@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 174122d commit a8ed6e9

3 files changed

Lines changed: 176 additions & 0 deletions

File tree

‎site/docs/capabilities/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Support for various Large Language Model providers:
1616
- **[Supported Providers](./llm-integrations/supported-providers.md)**: Compatible AI/LLM service providers
1717
- **[Supported Endpoints](./llm-integrations/supported-endpoints.md)**: Available API endpoints and operations
1818
- **[Vendor-Specific Fields](./llm-integrations/vendor-specific-fields.md)**: Use backend-specific parameters and access provider-unique capabilities in your OpenAI-compatible requests
19+
- **[Prompt Caching](./llm-integrations/prompt-caching.md)**: Provider-agnostic prompt caching using unified cache_control API
1920

2021
## Inference Optimization
2122

‎site/docs/capabilities/llm-integrations/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Envoy AI Gateway provides integration to multiple LLM Providers. This section pr
1515
## Advanced Features
1616

1717
- **[Vendor-Specific Fields](./vendor-specific-fields.md)** - Use backend-specific parameters and access provider-unique capabilities in your OpenAI-compatible requests
18+
- **[Prompt Caching](./prompt-caching.md)** - Provider-agnostic prompt caching using unified cache_control API
1819

1920
## Getting Started with Provider Connectivity
2021

‎site/docs/capabilities/llm-integrations/prompt-caching.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
---
2+
id: prompt-caching
3+
title: Prompt Caching
4+
sidebar_position: 6
5+
---
6+
7+
# Prompt Caching
8+
9+
Envoy AI Gateway provides provider-agnostic prompt caching through a unified `cache_control` API. The same cache syntax works across multiple providers: Direct Anthropic, GCP Vertex AI (Claude models), and AWS Bedrock (Claude models). This reduces costs and improves response times by caching frequently-used content like system prompts, tool definitions, and reference documents.
10+
11+
## Supported Providers
12+
13+
| Provider | API Schema | Cache Support |
14+
| ---------------------- | --------------------- | ------------- |
15+
| Anthropic (Direct) | `Anthropic` | Native |
16+
| GCP Vertex AI (Claude) | `GCPAnthropic` | Translated |
17+
| AWS Bedrock (Claude) | `AWSBedrockAnthropic` | Translated |
18+
19+
## How It Works
20+
21+
- Add `cache_control: {"type": "ephemeral"}` to content blocks in your request.
22+
- AI Gateway translates this to the provider-specific format automatically.
23+
- Cache is maintained per-provider; all providers require a minimum of 1,024 tokens for caching.
24+
- A maximum of 4 cache breakpoints are allowed per request across all providers.
25+
- On cache hit, the provider charges reduced input token costs.
26+
27+
## Usage
28+
29+
No gateway-side configuration is needed. Caching is controlled entirely at the request level by adding `cache_control` to the content blocks you want cached.
30+
31+
### Basic Example: System Prompt Caching
32+
33+
Cache a system prompt so that subsequent requests reuse the cached content:
34+
35+
```json
36+
{
37+
"model": "claude-sonnet-4-5",
38+
"messages": [
39+
{
40+
"role": "system",
41+
"content": [
42+
{
43+
"type": "text",
44+
"text": "You are a helpful assistant with extensive knowledge...(long system prompt)...",
45+
"cache_control": { "type": "ephemeral" }
46+
}
47+
]
48+
},
49+
{
50+
"role": "user",
51+
"content": "What is the capital of France?"
52+
}
53+
]
54+
}
55+
```
56+
57+
### Multiple Cache Points
58+
59+
You can place up to 4 cache breakpoints in a single request to cache different parts of the conversation:
60+
61+
```json
62+
{
63+
"model": "claude-sonnet-4-5",
64+
"messages": [
65+
{
66+
"role": "system",
67+
"content": [
68+
{
69+
"type": "text",
70+
"text": "System instructions...",
71+
"cache_control": { "type": "ephemeral" }
72+
}
73+
]
74+
},
75+
{
76+
"role": "user",
77+
"content": [
78+
{
79+
"type": "text",
80+
"text": "Reference document content...",
81+
"cache_control": { "type": "ephemeral" }
82+
},
83+
{
84+
"type": "text",
85+
"text": "Question about the document"
86+
}
87+
]
88+
}
89+
]
90+
}
91+
```
92+
93+
### Tool Definition Caching
94+
95+
Cache complex tool schemas that remain the same across requests:
96+
97+
```json
98+
{
99+
"model": "claude-sonnet-4-5",
100+
"messages": [
101+
{
102+
"role": "user",
103+
"content": "Help me search for information about cloud computing trends."
104+
}
105+
],
106+
"tools": [
107+
{
108+
"type": "function",
109+
"function": {
110+
"name": "search_knowledge_base",
111+
"description": "Search through a comprehensive knowledge base...",
112+
"parameters": {
113+
"type": "object",
114+
"properties": {
115+
"query": {
116+
"type": "string",
117+
"description": "Natural language search query"
118+
}
119+
},
120+
"required": ["query"]
121+
},
122+
"cache_control": { "type": "ephemeral" }
123+
}
124+
}
125+
]
126+
}
127+
```
128+
129+
## Response Format
130+
131+
When caching is active, the response includes cache information in the `usage` field:
132+
133+
```json
134+
{
135+
"usage": {
136+
"prompt_tokens": 2000,
137+
"completion_tokens": 150,
138+
"prompt_tokens_details": {
139+
"cached_tokens": 1800
140+
}
141+
}
142+
}
143+
```
144+
145+
- `cached_tokens` indicates the number of tokens served from cache at a reduced cost.
146+
- Cache write tokens are tracked internally for billing purposes.
147+
148+
## Best Practices
149+
150+
:::tip
151+
152+
- Place `cache_control` on content that exceeds 1,024 tokens. Content below this threshold will not be cached.
153+
- Cache system prompts, tool definitions, and reference documents that do not change between requests.
154+
- Position cache breakpoints strategically -- cached content must appear at the beginning of the message.
155+
- Monitor `cached_tokens` in responses to verify caching effectiveness and measure cost savings.
156+
:::
157+
158+
## Provider-Specific Notes
159+
160+
:::note
161+
162+
- **All providers**: Minimum 1,024 tokens per cached block, maximum 4 cache breakpoints per request.
163+
- **Anthropic Direct**: Uses the native `cache_control` field directly with no translation.
164+
- **GCP Vertex AI**: AI Gateway translates `cache_control` to Vertex AI's caching format automatically.
165+
- **AWS Bedrock**: AI Gateway translates `cache_control` to Bedrock's cachePoint format automatically.
166+
- All providers support the `"ephemeral"` cache type.
167+
- Existing requests without `cache_control` continue to work with no changes.
168+
:::
169+
170+
## Further Reading
171+
172+
- [Prompt Caching Examples](https://github.com/envoyproxy/ai-gateway/tree/main/examples/cache) -- Detailed examples with curl commands for each provider.
173+
- [Connecting to GCP Vertex AI](../../getting-started/connect-providers/gcp-vertexai.md) -- Set up GCP Vertex AI as a provider.
174+
- [Connecting to AWS Bedrock](../../getting-started/connect-providers/aws-bedrock.md) -- Set up AWS Bedrock as a provider.

0 commit comments

Comments
 (0)