Skip to content

README.md

Latest commit

 

History

History
329 lines (254 loc) · 10.6 KB

README.md

File metadata and controls

329 lines (254 loc) · 10.6 KB
Raw

Usage-Based Rate Limiting with Envoy AI Gateway

This demo showcases usage-based rate limiting for AI workloads, allowing you to control token consumption per user and model. It builds on top of the 01-getting-started demo environment.

Overview

This demo demonstrates:

  • Token-based rate limiting for LLM requests
  • Per-user and per-model limits with different token quotas
  • Automatic token tracking from LLM responses
  • Rate limit enforcement with 429 status codes when limits are exceeded

Prerequisites

  • kubectl configured to access your cluster
  • task (Taskfile) installed
  • jq for JSON pretty-printing

Demo Execution Steps

Step 1: Setup Rate Limiting

cd demos/02-usage-based-rate-limiting
task setup

This will:

  • Ensure the basic environment from 01-getting-started is running
  • Deploy the BackendTrafficPolicy for rate limiting
  • Replace the AIGatewayRoute with enhanced token tracking
  • Enable support for additional models (gpt-4, gpt-3.5-turbo)

Step 2: Start Port Forwarding

task port-forward

Step 3: Test Rate Limit Enforcement

task test-limits-exceeded

This sends rapid requests to test rate limiting. Expected behavior:

  • First few requests succeed (200 OK)
  • Eventually triggers rate limit (429 status expected, but currently showing all 200s)
  • Each successful request shows 6 input tokens + 6 output tokens = 12 total tokens

Step 4: Monitor Metrics

task metrics

This captures real-time metrics showing:

  • Token Usage: qwen3 model consumed 120 tokens (60 input + 60 output) across 10 requests
  • Rate Limit Status: Shows requests allowed vs blocked per model
  • Extracted Values: Current consumption vs configured limits

Step 5: View Logs

task logs

Check gateway logs for rate limiting events and debugging.

Configuration Details

Token Tracking

The configuration tracks three types of tokens:

  • Input tokens: Tokens from the user's prompt
  • Output tokens: Tokens generated by the model
  • Total tokens: Combined input and output tokens
llmRequestCosts:
  - metadataKey: llm_input_token
    type: InputToken
  - metadataKey: llm_output_token
    type: OutputToken
  - metadataKey: llm_total_token
    type: TotalToken

Rate Limits by Model

Different models have different token limits per hour:

  • gpt-4: 1,000 tokens/hour per user
  • gpt-3.5-turbo: 100 tokens/hour per user
  • qwen3: 50 tokens/hour per user

User Identification

Rate limits are applied per user using the x-user-id header:

curl -H "x-user-id: alice" -H "x-ai-eg-model: gpt-4" ...

Testing Examples

Manual Testing Examples

Alice using qwen3 (50 tokens/hour limit):

curl -H "Content-Type: application/json" \
     -H "x-user-id: alice" \
     -d '{
       "model": "qwen3",
       "messages": [{"role": "user", "content": "Hello from Alice!"}]
     }' \
     http://localhost:8080/v1/chat/completions

Bob using gpt-3.5-turbo (100 tokens/hour limit):

curl -H "Content-Type: application/json" \
     -H "x-user-id: bob" \
     -d '{
       "model": "gpt-3.5-turbo",
       "messages": [{"role": "user", "content": "Hello from Bob!"}]
     }' \
     http://localhost:8080/v1/chat/completions

Note: The x-ai-eg-model header is not needed when the model is specified in the request body.

Test Rate Limit Exceeded

# Send multiple requests to test rate limiting
task test-limits-exceeded

This task sends 10 rapid requests to test rate limiting. Based on the configuration:

  • qwen3: 50 tokens/hour limit
  • Each request: ~12 tokens (6 input + 6 output)
  • 10 requests: 120 tokens total (240% of limit)

To actually trigger 429 errors, you would need to either:

  1. Send more requests (>5 requests to exceed 50 tokens)
  2. Use a different user that has already consumed tokens
  3. Adjust the rate limit to a lower value for testing

Available Tasks

Core Demo Tasks:

  • task setup - Deploy rate limiting configuration
  • task port-forward - Start port forwarding to access gateway
  • task test-limits-exceeded - Send rapid requests to test rate limiting
  • task metrics - Capture raw token usage and rate limiting metrics
  • task logs - Show rate limit related logs

Additional Testing:

  • task test-rate-limits - Run all rate limiting tests
  • task test-user-alice - Test limits for user Alice (qwen3 and gpt-4)
  • task test-user-bob - Test limits for user Bob (gpt-3.5-turbo)
  • task test-streaming - Test streaming with rate limits

Configuration & Cleanup:

  • task status - View rate limiting configuration
  • task cleanup - Remove rate limiting configuration

GenAI Metrics Collection

To collect GenAI metrics (token usage, latency, etc.) and rate limiting stats, use the automated metrics task:

Raw Metrics Analysis:

task metrics

This provides:

  • Token usage metrics from port 1064 (GenAI metrics endpoint)
  • Rate limiting stats from port 19000 (Envoy admin endpoint)
  • Extracted values with current token consumption vs limits
  • Raw Prometheus metrics for integration with monitoring systems

Sample Output:

=== GenAI Metrics (port 1064) ===
--- Token Usage ---
gen_ai_client_token_usage_token_sum{gen_ai_request_model="qwen3",gen_ai_token_type="total"} 95
gen_ai_client_token_usage_token_count{gen_ai_request_model="qwen3",gen_ai_token_type="total"} 8

=== Rate Limiting Metrics (port 19000) ===
--- Rate Limit Decisions ---
cluster.httproute/default/llm-d-inference-sim-route/rule/0.ratelimit.ok: 8
cluster.httproute/default/llm-d-inference-sim-route/rule/0.ratelimit.over_limit: 1

=== Extracted Values ===
qwen3: input=47 output=48 total=95 count=8 (limit=50/hour)
rule/0 (qwen3): ok=8 over_limit=1

Manual Metrics Collection (advanced):

# Get the Envoy pod name
ENVOY_POD=$(kubectl get pods -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway -o jsonpath='{.items[0].metadata.name}')

# Port-forward to metrics endpoints
kubectl port-forward -n envoy-gateway-system $ENVOY_POD 1064:1064 &  # GenAI metrics
kubectl port-forward -n envoy-gateway-system $ENVOY_POD 19000:19000 &  # Envoy admin

# Collect specific metrics
curl -s http://localhost:1064/metrics | grep gen_ai_client_token_usage
curl -s http://localhost:19000/stats | grep ratelimit

Demo Results

Actual Output from Demo Run

Token Consumption (from task metrics):

=== GenAI Metrics (port 1064) ===
--- Token Usage ---
gen_ai_client_token_usage_token_sum{gen_ai_request_model="qwen3",gen_ai_token_type="input"} 60
gen_ai_client_token_usage_token_sum{gen_ai_request_model="qwen3",gen_ai_token_type="output"} 60
gen_ai_client_token_usage_token_sum{gen_ai_request_model="qwen3",gen_ai_token_type="total"} 120
gen_ai_client_token_usage_token_count{gen_ai_request_model="qwen3",gen_ai_token_type="total"} 10

=== Extracted Values ===
qwen3: input=60 output=60 total=120 count=10 (limit=50/hour)

Rate Limiting Behavior:

  • 10 requests sent to qwen3 model
  • Each request consumed 12 tokens (6 input + 6 output)
  • Total consumption: 120 tokens out of 50/hour limit (60% utilized)
  • All requests returned 200 OK (rate limit not yet triggered)

Note: To trigger 429 rate limit errors, you would need to:

  • Send more requests to exceed the 50 token/hour limit
  • Or adjust the rate limit configuration to a lower threshold

How It Works

  1. Route Enhancement: Demo 02 replaces the AIGatewayRoute from demo 01:

    • Adds llmRequestCosts configuration for token tracking
    • Extends support to additional models while preserving the original qwen3 route

  2. Request Processing: When a request arrives, the gateway extracts:

    • User ID from x-user-id header
    • Model from x-ai-eg-model header

  3. Token Tracking: The AI Gateway automatically:

    • Extracts token usage from LLM responses
    • Updates the user's token consumption

  4. Rate Limit Check: Before processing:

    • Checks if the user has remaining tokens
    • Rejects requests that would exceed limits

  5. Response:

    • Successful requests are processed normally
    • Rate-limited requests receive 429 status

Architecture

┌─────────────┐     ┌─────────────────┐     ┌──────────────────┐
│   Client    │────▶│  Envoy AI       │────▶│  LLM Backend    │
│ (w/ user-id)│     │  Gateway        │     │                 │
└─────────────┘     └─────────────────┘     └──────────────────┘
                           │
                           ▼
                    ┌─────────────────┐
                    │  Rate Limiter   │
                    │ (Token-based)   │
                    └─────────────────┘

Troubleshooting

Rate Limits Not Working

  1. Ensure you're including the x-user-id header
  2. Check that the model name in x-ai-eg-model matches configuration
  3. Verify the BackendTrafficPolicy is applied: task status

429 Errors on First Request

  • The rate limiter may have residual state from previous tests
  • Wait a few minutes or use a different user ID

Token Counts Not Accurate

  • The LLM-D simulator returns mock token counts
  • With real LLM backends, token counts will be accurate

Advanced Configuration

Custom Token Costs

You can implement custom token cost calculations using CEL expressions:

llmRequestCosts:
  - metadataKey: custom_cost
    celExpression: "has(llm_input_token) ? llm_input_token * 2 : 0"

Different Time Windows

Modify the rate limit time window:

limit:
  requests: 1000
  unit: Minute  # or Hour, Day

Cleanup

To remove the rate limiting configuration:

task cleanup

This will:

  • Remove the BackendTrafficPolicy
  • Revert the AIGatewayRoute to its original demo 01 configuration
  • Preserve the base environment from 01-getting-started

Next Steps

  • Implement different rate limits for different user tiers
  • Add token cost calculations based on model pricing
  • Configure rate limiting with Redis for distributed deployments
  • Combine with authentication for production use

References