|
| 1 | +--- |
| 2 | +id: quota-policy |
| 3 | +title: Quota Policy |
| 4 | +sidebar_position: 6 |
| 5 | +--- |
| 6 | + |
| 7 | +# Quota Policy |
| 8 | + |
| 9 | +`QuotaPolicy` enables token-based quota management for AI inference services in Envoy AI Gateway. |
| 10 | +When all related backend's quota are exceeded, requests are rejected with a `429 Too Many Requests` status code. |
| 11 | + |
| 12 | +:::note QuotaPolicy vs. Rate Limiting |
| 13 | +QuotaPolicy manages **total consumption budgets** (for example, 100,000 tokens per hour). This is |
| 14 | +distinct from [usage-based rate limiting](./usage-based-ratelimiting.md), which controls **request |
| 15 | +velocity** (for example, requests per second). Use QuotaPolicy when you need to cap cumulative token |
| 16 | +spend across a time window. |
| 17 | +::: |
| 18 | + |
| 19 | +## Overview |
| 20 | + |
| 21 | +Key features of QuotaPolicy: |
| 22 | + |
| 23 | +- **Per-model token quotas** — assign token budgets to individual models served by an `AIServiceBackend`. |
| 24 | +- **CEL cost expressions** — weight input, output, cached, and reasoning tokens differently when |
| 25 | + computing how much a request burns down a quota. |
| 26 | +- **Client-selector bucket rules** — carve out per-tenant or per-header quotas using request attributes. |
| 27 | +- **Shadow mode** — evaluate quota rules without enforcing them, for safe rollout. |
| 28 | + |
| 29 | +## How It Works |
| 30 | + |
| 31 | +1. A `QuotaPolicy` is attached to one or more `AIServiceBackend` resources via `targetRefs`. |
| 32 | +2. For each completed request, the token cost is computed using the configured cost expression |
| 33 | + (defaults to `total_tokens`). |
| 34 | +3. The cost is charged against the matching quota bucket (the per-model default bucket, or a matching |
| 35 | + bucket rule). |
| 36 | +4. When all related quota buckets for that model are exceeded, subsequent matching requests receive `429 Too Many Requests`. |
| 37 | + |
| 38 | +:::tip Prerequisites |
| 39 | +Quota enforcement uses the same infrastructure as usage-based rate limiting: |
| 40 | + |
| 41 | +1. **Redis Deployment**: A Redis instance for storing quota counters. See the [redis.yaml example](https://github.com/envoyproxy/ai-gateway/blob/main/examples/token_ratelimit/redis.yaml) for a simple deployment. |
| 42 | +2. **Envoy Gateway Configuration**: Envoy Gateway must be configured at installation time to enable rate limiting and point to your Redis instance. See the [Envoy Gateway Installation Guide](../../getting-started/prerequisites.md#additional-features-rate-limiting-inferencepool-etc). |
| 43 | + |
| 44 | +See [Usage-based Rate Limiting](./usage-based-ratelimiting.md) for more detail on the rate limit infrastructure that QuotaPolicy builds on. |
| 45 | +::: |
| 46 | + |
| 47 | +## Configuration |
| 48 | + |
| 49 | +### Per-Model Quotas |
| 50 | + |
| 51 | +Use `perModelQuotas` to apply a token budget to a specific model served by the targeted backend(s). |
| 52 | + |
| 53 | +:::warning The model name must match the route |
| 54 | +A `perModelQuotas` entry only applies when its `modelName` matches the `modelNameOverride` set on the |
| 55 | +`AIGatewayRoute` rule's `backendRefs` for the targeted backend. If they do not match, the quota is |
| 56 | +silently **not** applied. |
| 57 | +::: |
| 58 | + |
| 59 | +Given an `AIGatewayRoute` that routes a model to the backend: |
| 60 | + |
| 61 | +```yaml |
| 62 | +apiVersion: aigateway.envoyproxy.io/v1alpha1 |
| 63 | +kind: AIGatewayRoute |
| 64 | +metadata: |
| 65 | + name: my-route |
| 66 | +spec: |
| 67 | + rules: |
| 68 | + - backendRefs: |
| 69 | + - name: my-backend |
| 70 | + modelNameOverride: my-model # <-- the QuotaPolicy modelName must match this |
| 71 | +``` |
| 72 | +
|
| 73 | +attach a `QuotaPolicy` to the backend: |
| 74 | + |
| 75 | +```yaml |
| 76 | +apiVersion: aigateway.envoyproxy.io/v1alpha1 |
| 77 | +kind: QuotaPolicy |
| 78 | +metadata: |
| 79 | + name: my-quota-policy |
| 80 | +spec: |
| 81 | + targetRefs: |
| 82 | + - group: aigateway.envoyproxy.io |
| 83 | + kind: AIServiceBackend |
| 84 | + name: my-backend |
| 85 | + perModelQuotas: |
| 86 | + - modelName: "my-model" |
| 87 | + quota: |
| 88 | + mode: Shared |
| 89 | + defaultBucket: |
| 90 | + limit: 10000 # Maximum tokens allowed in the window. |
| 91 | + duration: "1h" # Sliding window. |
| 92 | +``` |
| 93 | + |
| 94 | +You can attach quotas for multiple models, each with its own budget: |
| 95 | + |
| 96 | +```yaml |
| 97 | +perModelQuotas: |
| 98 | + - modelName: gpt-4 |
| 99 | + quota: |
| 100 | + defaultBucket: |
| 101 | + limit: 10000 # Strict limit for the expensive model. |
| 102 | + duration: "1h" |
| 103 | + - modelName: gpt-3.5-turbo |
| 104 | + quota: |
| 105 | + defaultBucket: |
| 106 | + limit: 100000 # Higher limit for the cost-effective model. |
| 107 | + duration: "1h" |
| 108 | +``` |
| 109 | + |
| 110 | +:::note |
| 111 | +When multiple `QuotaPolicy` resources define the same model for the same `AIServiceBackend`, the |
| 112 | +policy whose namespace/name sorts alphabetically first takes precedence. |
| 113 | +::: |
| 114 | + |
| 115 | +### Custom Cost Expression |
| 116 | + |
| 117 | +By default, a request's cost is its `total_tokens`. You can override this with a |
| 118 | +[CEL](https://github.com/google/cel-spec) expression that weights token types differently. The |
| 119 | +following variables are available in a `costExpression`: |
| 120 | + |
| 121 | +| Variable | Type | Description | |
| 122 | +| ----------------------------- | ------ | ------------------------------------------------ | |
| 123 | +| `input_tokens` | uint | Prompt / input tokens. | |
| 124 | +| `output_tokens` | uint | Completion / output tokens. | |
| 125 | +| `total_tokens` | uint | Total tokens (the default cost). | |
| 126 | +| `cached_input_tokens` | uint | Input tokens served from the provider's cache. | |
| 127 | +| `cache_creation_input_tokens` | uint | Input tokens charged for writing to the cache. | |
| 128 | +| `reasoning_tokens` | uint | Reasoning tokens (for reasoning-capable models). | |
| 129 | +| `model` | string | The resolved model name. | |
| 130 | +| `backend` | string | The serving backend name. | |
| 131 | +| `route_name` | string | The route name. | |
| 132 | + |
| 133 | +```yaml |
| 134 | +perModelQuotas: |
| 135 | + - modelName: gpt-4 |
| 136 | + quota: |
| 137 | + # Cached input tokens count as 1/10 of a regular input token; |
| 138 | + # output tokens count 6x. |
| 139 | + costExpression: "input_tokens + cached_input_tokens / 10u + output_tokens * 6u" |
| 140 | + defaultBucket: |
| 141 | + limit: 50000 |
| 142 | + duration: "1h" |
| 143 | +``` |
| 144 | + |
| 145 | +:::tip |
| 146 | +Use a custom cost expression when token types have significantly different costs with your provider — |
| 147 | +for example, output tokens are typically more expensive than input tokens. |
| 148 | + |
| 149 | +The token variables are **unsigned integers**, so numeric literals must carry a `u` suffix (for |
| 150 | +example `output_tokens * 6u`) and the expression must evaluate to a non-negative integer. Integer |
| 151 | +division truncates (`cached_input_tokens / 10u`); for an exact fractional weight, cast through |
| 152 | +floating point — for example `uint(double(cached_input_tokens) * 0.1)`. |
| 153 | +::: |
| 154 | + |
| 155 | +### Bucket Mode |
| 156 | + |
| 157 | +The `mode` field on a per-model `quota` controls how the `defaultBucket` and matching `bucketRules` |
| 158 | +interact when a request matches one or more rules. |
| 159 | + |
| 160 | +Currently only **`Shared`** mode is supported (it is also the default, so the field can be omitted): |
| 161 | + |
| 162 | +- The request is charged to **all** matching `bucketRules` **and** the `defaultBucket`. |
| 163 | +- The request is allowed only if quota is available in **at least one** matching bucket. |
| 164 | + |
| 165 | +```yaml |
| 166 | +perModelQuotas: |
| 167 | + - modelName: gpt-4 |
| 168 | + quota: |
| 169 | + mode: Shared # Default — may be omitted. |
| 170 | + defaultBucket: |
| 171 | + limit: 10000 |
| 172 | + duration: "1h" |
| 173 | +``` |
| 174 | + |
| 175 | +:::note |
| 176 | +An exclusive bucket mode (charging matching rules **or** the default bucket, but not both) is planned |
| 177 | +but not yet available. The only accepted value today is `Shared`. |
| 178 | +::: |
| 179 | + |
| 180 | +### Client Selectors with Bucket Rules |
| 181 | + |
| 182 | +Bucket rules let you carve out dedicated quotas for specific clients identified by request attributes |
| 183 | +such as headers. This is useful for multi-tenant deployments where each tenant needs its own budget. |
| 184 | +Because the mode is `Shared`, a request that matches a bucket rule is charged against **both** that |
| 185 | +rule's bucket **and** the `defaultBucket`. |
| 186 | + |
| 187 | +```yaml |
| 188 | +perModelQuotas: |
| 189 | + - modelName: gpt-4 |
| 190 | + quota: |
| 191 | + defaultBucket: |
| 192 | + limit: 10000 # Shared budget across all tenants. |
| 193 | + duration: "1h" |
| 194 | + bucketRules: |
| 195 | + # Premium tenant gets a dedicated, higher per-tenant budget. |
| 196 | + - clientSelectors: |
| 197 | + - headers: |
| 198 | + - name: x-tenant-id |
| 199 | + type: Exact |
| 200 | + value: premium-tenant |
| 201 | + quota: |
| 202 | + limit: 50000 |
| 203 | + duration: "1h" |
| 204 | +``` |
| 205 | + |
| 206 | +Use `type: Distinct` to create a separate bucket for every unique value of a header — for example, a |
| 207 | +per-tenant budget keyed on the tenant ID: |
| 208 | + |
| 209 | +```yaml |
| 210 | +bucketRules: |
| 211 | + - clientSelectors: |
| 212 | + - headers: |
| 213 | + - name: x-tenant-id |
| 214 | + type: Distinct # One bucket per unique tenant ID. |
| 215 | + quota: |
| 216 | + limit: 5000 |
| 217 | + duration: "1h" |
| 218 | +``` |
| 219 | + |
| 220 | +Supported header match types are `Exact`, `Distinct`, and `RegularExpression`. |
| 221 | + |
| 222 | +`clientSelectors` reuse the Envoy Gateway [`RateLimitSelectCondition`](https://gateway.envoyproxy.io/docs/api/extension_types/#ratelimitselectcondition) type, but QuotaPolicy currently applies only the `headers` matcher. The other fields on that type (`sourceCIDR`, `methods`, `path`, `queryParams`) are accepted by the schema but **not yet honored** for quota buckets. |
| 223 | + |
| 224 | +### Shadow Mode |
| 225 | + |
| 226 | +Shadow mode lets you test a bucket rule without rejecting traffic. When `shadowMode` is enabled on a |
| 227 | +bucket rule, all quota checks are performed (cache lookups, counter updates, telemetry), but the |
| 228 | +outcome is never enforced — the request always succeeds even if the quota is exceeded. |
| 229 | + |
| 230 | +```yaml |
| 231 | +bucketRules: |
| 232 | + - clientSelectors: |
| 233 | + - headers: |
| 234 | + - name: x-tenant-id |
| 235 | + type: Distinct |
| 236 | + quota: |
| 237 | + limit: 5000 |
| 238 | + duration: "1h" |
| 239 | + shadowMode: true # Evaluate but do not enforce. |
| 240 | +``` |
| 241 | + |
| 242 | +:::tip |
| 243 | +Use shadow mode when rolling out a new quota rule. Monitor the telemetry to confirm the limit is set |
| 244 | +correctly, then enable enforcement by removing `shadowMode` (or setting it to `false`). |
| 245 | +::: |
| 246 | + |
| 247 | +Shadow mode is configured per bucket rule. It cannot be set on the `defaultBucket`. |
| 248 | + |
| 249 | +## Duration Format |
| 250 | + |
| 251 | +The `duration` field selects the sliding-window size. It must be exactly one of the following values: |
| 252 | + |
| 253 | +| Value | Window | |
| 254 | +| ------ | ---------- | |
| 255 | +| `"1s"` | One second | |
| 256 | +| `"1m"` | One minute | |
| 257 | +| `"1h"` | One hour | |
| 258 | +| `"1d"` | One day | |
| 259 | + |
| 260 | +The window is fixed-size — arbitrary multiples such as `"30s"` or `"15m"` are **not** valid and will |
| 261 | +be rejected by the CRD schema. Choose the `limit` to express your budget within one of these windows. |
| 262 | + |
| 263 | +## Service-Wide Quota |
| 264 | + |
| 265 | +:::warning Not yet available |
| 266 | +The API exposes a backend-wide `serviceQuota` field — intended to apply a single budget across all |
| 267 | +models on a backend — but it is **not yet enforced** (it is currently a known TODO in the API). |
| 268 | +Configuring it has no effect on traffic, so use [Per-Model Quotas](#per-model-quotas) to enforce |
| 269 | +token quotas today. This section will document `serviceQuota` once enforcement lands. |
| 270 | +::: |
| 271 | + |
| 272 | +## Full Example |
| 273 | + |
| 274 | +The following `QuotaPolicy` combines per-model quotas, custom cost expressions, bucket rules with |
| 275 | +client selectors, and shadow mode. |
| 276 | + |
| 277 | +```yaml |
| 278 | +apiVersion: aigateway.envoyproxy.io/v1alpha1 |
| 279 | +kind: QuotaPolicy |
| 280 | +metadata: |
| 281 | + name: full-quota-policy |
| 282 | +spec: |
| 283 | + targetRefs: |
| 284 | + - group: aigateway.envoyproxy.io |
| 285 | + kind: AIServiceBackend |
| 286 | + name: my-backend |
| 287 | +
|
| 288 | + perModelQuotas: |
| 289 | + - modelName: gpt-4 |
| 290 | + quota: |
| 291 | + costExpression: "input_tokens + cached_input_tokens / 10u + output_tokens * 6u" |
| 292 | + mode: Shared |
| 293 | + defaultBucket: |
| 294 | + limit: 10000 |
| 295 | + duration: "1h" |
| 296 | + bucketRules: |
| 297 | + # Premium tenant gets a dedicated, higher quota. |
| 298 | + - clientSelectors: |
| 299 | + - headers: |
| 300 | + - name: x-tenant-id |
| 301 | + type: Exact |
| 302 | + value: premium-tenant |
| 303 | + quota: |
| 304 | + limit: 50000 |
| 305 | + duration: "1h" |
| 306 | + # Track per-tenant usage in shadow mode (no enforcement). |
| 307 | + - clientSelectors: |
| 308 | + - headers: |
| 309 | + - name: x-tenant-id |
| 310 | + type: Distinct |
| 311 | + quota: |
| 312 | + limit: 5000 |
| 313 | + duration: "1h" |
| 314 | + shadowMode: true |
| 315 | +
|
| 316 | + - modelName: gpt-3.5-turbo |
| 317 | + quota: |
| 318 | + defaultBucket: |
| 319 | + limit: 100000 |
| 320 | + duration: "1h" |
| 321 | +``` |
| 322 | + |
| 323 | +## References |
| 324 | + |
| 325 | +- [API Reference](../../api/api.mdx) |
| 326 | +- [Usage-Based Rate Limiting](./usage-based-ratelimiting.md) |
| 327 | +- [Provider Fallback](./provider-fallback.md) |
0 commit comments