RFC: Simulator and Development Instance

[jwalsh]

RFC: Metrics Dashboard Simulator and Development Instance

Summary

Establish a standardized simulator and development environment for testing Prometheus metrics and Grafana dashboards before production deployment. This enables safe validation of dashboard performance, alert thresholds, and capacity planning under realistic load conditions.

Motivation

Currently, new metrics and dashboards are tested directly in production, leading to:

• Dashboard performance issues discovered after deployment
• Incorrect alert thresholds causing false positives/negatives
• Cardinality explosions impacting Prometheus performance
• No way to validate behavior under extreme load conditions
• Difficulty reproducing production issues in development

Proposal

1. Metrics Simulator Framework

Create a Python-based simulator that generates realistic metric patterns for any Prometheus-instrumented service.

# Core simulator interface
class MetricsSimulator:
    def __init__(self, config):
        self.users = config['users']
        self.activity_model = config['activity_model']
        self.metrics_registry = config['metrics']
    
    def generate_activity(self):
        # Brownian motion with mean reversion
        # Configurable burst patterns
        # Seasonal variations (business hours, weekends)

Key Features:

• Configurable user populations: Power users, regular users, idle users
• Activity models: Brownian motion, Poisson processes, replay from production
• Fault injection: Simulate outages, thundering herds, cardinality bombs
• Time acceleration: Simulate 30 days of metrics in 1 hour

2. Development Instance Architecture

# docker-compose.dev.yml
version: '3.8'

services:
  # Isolated Prometheus for testing
  prometheus-dev:
    image: prom/prometheus:latest
    ports:
      - "9091:9090"
    command:
      - '--storage.tsdb.retention.time=7d'
      - '--storage.tsdb.retention.size=10GB'
      - '--web.enable-lifecycle'  # Allow config reloads
    volumes:
      - ./prometheus-dev.yml:/etc/prometheus/prometheus.yml
      - prometheus-dev-data:/prometheus

  # Isolated Grafana for dashboard development  
  grafana-dev:
    image: grafana/grafana:latest
    ports:
      - "3001:3000"
    environment:
      - GF_DEFAULT_INSTANCE_NAME=dev
      - GF_USERS_DEFAULT_THEME=light  # Distinguish from prod
      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/home.json
    volumes:
      - ./grafana-provisioning-dev:/etc/grafana/provisioning
      - grafana-dev-data:/var/lib/grafana

  # Metrics simulator
  simulator:
    build: ./simulator
    environment:
      - SCENARIO=${SCENARIO:-baseline}
      - USERS=${USERS:-100}
      - DURATION=${DURATION:-3600}
    volumes:
      - ./scenarios:/app/scenarios

3. Testing Scenarios

Baseline Scenario

• 100 users with typical activity patterns
• Normal distribution of requests
• Standard error rates (0.1%)

Load Test Scenarios

scenarios:
  black_friday:
    users: 1000
    activity_multiplier: 10
    duration: 4h
    pattern: "spike"
    
  gradual_rollout:
    users: [10, 50, 100, 500, 1000]
    ramp_time: 30m
    pattern: "linear"
    
  disaster_recovery:
    users: 100
    failures:
      - time: 10m
        type: "total_outage"
        duration: 5m
      - time: 30m  
        type: "cascade_failure"
        affected_percentage: 50

4. Validation Framework

Performance Benchmarks

benchmarks:
  dashboard_load:
    target: < 2s
    panels: all
    time_range: 24h
    
  query_performance:
    simple_counter: < 100ms
    rate_calculation: < 500ms
    histogram_quantile: < 1s
    
  cardinality_limits:
    total_series: < 1M
    per_metric: < 10K
    label_combinations: < 100K

Automated Testing

def test_dashboard_performance():
    # Start simulator with high load
    simulator.start_scenario("black_friday")
    
    # Wait for metrics to accumulate
    time.sleep(300)
    
    # Test dashboard load time
    load_time = grafana.measure_dashboard_load("claude-metrics-v2")
    assert load_time < 2.0, f"Dashboard load too slow: {load_time}s"
    
    # Test query performance
    for panel in dashboard.panels:
        query_time = prometheus.measure_query(panel.query)
        assert query_time < panel.sla, f"Query too slow: {panel.id}"

5. Development Workflow

graph LR
    A[New Metric/Dashboard] --> B[Update Simulator]
    B --> C[Run Test Scenarios]
    C --> D{Performance OK?}
    D -->|No| E[Optimize Queries]
    E --> C
    D -->|Yes| F[Test Alerts]
    F --> G[Document Thresholds]
    G --> H[PR with Tests]
    H --> I[Deploy to Staging]
    I --> J[Shadow Prod Traffic]
    J --> K[Production Release]

Loading

6. Simulator Configuration

# simulator-config.yml
metrics:
  - name: otel_claude_code_session_count_total
    type: counter
    labels:
      - user_id
    generation:
      rate: "10 * activity_level"
      
  - name: otel_claude_code_token_usage_tokens_total  
    type: counter
    labels:
      - user_id
      - model
      - type
    generation:
      rate: "500 * activity_level"
      distribution:
        model:
          claude-3.5-sonnet: 0.4
          claude-3.7-opus: 0.35
          claude-4.0-opus: 0.25
        type:
          input: 0.6
          output: 0.3
          cache: 0.1

activity_models:
  brownian:
    volatility: 0.3
    mean_reversion: 0.1
    dt: 0.1
    
  seasonal:
    business_hours_multiplier: 2.0
    weekend_multiplier: 0.3
    timezone: "America/New_York"

7. Benefits

For Development

• Test dashboards with realistic data before production
• Validate alert thresholds reduce false positives
• Identify performance issues early
• Reproduce production issues locally

For Operations

• Capacity planning with load projections
• Disaster recovery testing
• Cardinality impact assessment
• Query optimization opportunities

For Product

• A/B test dashboard designs
• Validate metric usefulness before instrumentation
• Understand user behavior patterns
• Cost estimation for different usage patterns

8. Implementation Plan

Phase 1: Core Simulator (Week 1-2)

• Basic metric generation
• Brownian motion activity model
• Docker compose setup
• Simple test scenarios

Phase 2: Advanced Features (Week 3-4)

• Fault injection
• Production trace replay
• Seasonal patterns
• Burst/spike generation

Phase 3: Automation (Week 5-6)

• Performance test suite
• CI/CD integration
• Automated reports
• Alert validation

Phase 4: Documentation (Week 7)

• Developer guide
• Scenario library
• Best practices
• Troubleshooting

9. Success Metrics

• 90% of dashboard issues caught before production
• 50% reduction in false positive alerts
• 80% of developers using simulator for testing
• 0 cardinality-related outages

10. Alternatives Considered

Production Sampling

• Pros: Real data
• Cons: Can't test edge cases, privacy concerns

Synthetic Monitoring

• Pros: Continuous validation
• Cons: Limited scenarios, can't test pre-production

Chaos Engineering

• Pros: Real failure testing
• Cons: Risk to production, limited to failures

11. Open Questions

1. Should simulator be a separate service or library?
2. How to handle multi-tenant scenarios?
3. Integration with existing load testing tools?
4. Standardize across all teams or per-service?
5. How to share scenarios between teams?

12. Security Considerations

• No production data in test instances
• Separate networks for dev/prod
• Access controls for test environments
• Audit logging for configuration changes

13. References

• Prometheus Load Testing
• Grafana Performance Optimization
• Time Series Database Benchmarking
• Brownian Motion in Python

Appendix A: Example Usage

# Start development environment
docker-compose -f docker-compose.dev.yml up -d

# Run baseline scenario
./simulator.py --scenario baseline --users 100

# Test black friday load
./simulator.py --scenario black_friday --duration 4h

# Validate dashboard performance
./test_dashboard.py --dashboard claude-metrics-v2 --scenario high_load

# Generate capacity planning report
./simulator.py --scenario growth_projection --report capacity.html

Appendix B: Scenario Template

# scenarios/template.yml
name: "Scenario Name"
description: "What this scenario tests"

users:
  total: 100
  distribution:
    power_users: 
      percentage: 10
      activity_base: 2.5
    regular_users:
      percentage: 70
      activity_base: 1.0
    idle_users:
      percentage: 20
      activity_base: 0.2

timeline:
  - time: 0
    event: "start"
  - time: 10m
    event: "increase_load"
    multiplier: 2.0
  - time: 30m
    event: "inject_failure"
    type: "model_outage"
    affected: ["claude-4.0-opus"]
  - time: 45m
    event: "recovery"

assertions:
  - metric: "error_rate"
    condition: "< 0.05"
  - metric: "p99_latency"
    condition: "< 1000ms"
  - metric: "dashboard_load_time"
    condition: "< 2s"

Appendix C: Complete Implementation Example

#!/usr/bin/env python3
"""
Complete simulator implementation for Claude Code metrics
"""

import time
import random
import numpy as np
from prometheus_client import Counter, Histogram, Gauge, start_http_server
from threading import Thread
from dataclasses import dataclass
from typing import List, Dict, Any
import yaml
import logging

@dataclass
class User:
    id: str
    activity_level: float
    base_activity: float
    volatility: float
    user_type: str  # "power", "regular", "idle"

class ActivityModel:
    """Brownian motion with mean reversion for realistic activity patterns"""
    
    def update(self, user: User, dt: float = 0.1) -> float:
        # Mean reversion strength
        theta = 0.1
        
        # Random shock
        shock = np.random.normal(0, user.volatility * np.sqrt(dt))
        
        # Update with mean reversion
        drift = theta * (user.base_activity - user.activity_level) * dt
        user.activity_level += drift + shock
        
        # Bounds
        user.activity_level = max(0.1, min(5.0, user.activity_level))
        
        return user.activity_level

class MetricsGenerator:
    """Generate Prometheus metrics based on user activity"""
    
    def __init__(self, config: Dict[str, Any]):
        self.config = config
        self._setup_metrics()
        
    def _setup_metrics(self):
        self.session_counter = Counter(
            'otel_claude_code_session_count_total',
            'Total sessions',
            ['user_id']
        )
        
        self.token_counter = Counter(
            'otel_claude_code_token_usage_tokens_total',
            'Total tokens used',
            ['user_id', 'model', 'type']
        )
        
        self.cost_counter = Counter(
            'otel_claude_code_cost_usage_USD_total',
            'Total cost in USD',
            ['user_id', 'model']
        )
        
        self.commit_counter = Counter(
            'otel_claude_code_commit_count_total',
            'Total commits',
            ['user_id']
        )
        
    def generate_session(self, user: User):
        """Generate metrics for a user session"""
        if random.random() > user.activity_level / 5.0:
            return
            
        # Start session
        self.session_counter.labels(user_id=user.id).inc()
        
        # Choose model based on configuration
        model_weights = self.config['metrics']['model_distribution']
        model = random.choices(
            list(model_weights.keys()),
            weights=list(model_weights.values())
        )[0]
        
        # Generate tokens
        base_tokens = int(500 * user.activity_level * random.uniform(0.5, 2.0))
        
        # Input tokens
        input_tokens = int(base_tokens * random.uniform(0.8, 1.2))
        self.token_counter.labels(
            user_id=user.id,
            model=model,
            type='input'
        ).inc(input_tokens)
        
        # Output tokens
        output_tokens = int(base_tokens * 0.6 * random.uniform(0.5, 1.0))
        self.token_counter.labels(
            user_id=user.id,
            model=model,
            type='output'
        ).inc(output_tokens)
        
        # Calculate cost
        costs = self.config['metrics']['costs_per_1k_tokens'][model]
        total_cost = (input_tokens / 1000) * costs['input'] + \
                    (output_tokens / 1000) * costs['output']
        
        self.cost_counter.labels(
            user_id=user.id,
            model=model
        ).inc(total_cost)
        
        # Random commits
        if random.random() < 0.2 * user.activity_level:
            commits = random.randint(1, 3)
            self.commit_counter.labels(user_id=user.id).inc(commits)

class Simulator:
    """Main simulator orchestrating user activities"""
    
    def __init__(self, config_file: str):
        with open(config_file, 'r') as f:
            self.config = yaml.safe_load(f)
            
        self.users: List[User] = []
        self.activity_model = ActivityModel()
        self.metrics_generator = MetricsGenerator(self.config)
        self.running = True
        
        self._setup_logging()
        self._create_users()
        
    def _setup_logging(self):
        logging.basicConfig(
            level=logging.INFO,
            format='%(asctime)s - %(levelname)s - %(message)s'
        )
        self.logger = logging.getLogger(__name__)
        
    def _create_users(self):
        """Create user population based on configuration"""
        user_config = self.config['users']
        total_users = user_config['total']
        
        for i in range(total_users):
            # Determine user type
            rand = random.random()
            if rand < user_config['distribution']['power_users']['percentage']:
                user_type = 'power'
                base = random.uniform(2.0, 3.0)
                volatility = 0.2
            elif rand < (user_config['distribution']['power_users']['percentage'] + 
                         user_config['distribution']['regular_users']['percentage']):
                user_type = 'regular'
                base = random.uniform(0.8, 1.5)
                volatility = 0.3
            else:
                user_type = 'idle'
                base = random.uniform(0.2, 0.5)
                volatility = 0.4
                
            user = User(
                id=f"user_{i:03d}",
                activity_level=base,
                base_activity=base,
                volatility=volatility,
                user_type=user_type
            )
            self.users.append(user)
            
        self.logger.info(f"Created {len(self.users)} users")
        
    def run_scenario(self, scenario_name: str):
        """Run a specific test scenario"""
        scenario = self.config['scenarios'][scenario_name]
        self.logger.info(f"Running scenario: {scenario_name}")
        
        start_time = time.time()
        tick = 0
        
        while self.running:
            current_time = time.time() - start_time
            
            # Check timeline events
            for event in scenario.get('timeline', []):
                if abs(current_time - self._parse_time(event['time'])) < 0.1:
                    self._handle_event(event)
                    
            # Update activities every 10 ticks
            if tick % 10 == 0:
                for user in self.users:
                    self.activity_model.update(user)
                    
            # Generate metrics
            active_count = int(len(self.users) * 0.3)
            active_users = random.sample(self.users, active_count)
            
            for user in active_users:
                self.metrics_generator.generate_session(user)
                
            time.sleep(0.1)
            tick += 1
            
            # Status update
            if tick % 100 == 0:
                avg_activity = np.mean([u.activity_level for u in self.users])
                self.logger.info(f"Time: {current_time:.1f}s, Avg activity: {avg_activity:.2f}")
                
    def _parse_time(self, time_str: str) -> float:
        """Parse time string like '10m' or '2h' to seconds"""
        if time_str.endswith('m'):
            return float(time_str[:-1]) * 60
        elif time_str.endswith('h'):
            return float(time_str[:-1]) * 3600
        else:
            return float(time_str)
            
    def _handle_event(self, event: Dict[str, Any]):
        """Handle scenario events"""
        self.logger.info(f"Event: {event['event']}")
        
        if event['event'] == 'increase_load':
            multiplier = event.get('multiplier', 2.0)
            for user in self.users:
                user.activity_level *= multiplier
                
        elif event['event'] == 'inject_failure':
            failure_type = event.get('type', 'random')
            if failure_type == 'model_outage':
                # Simulate model outage by setting activity to 0 for some users
                affected_count = int(len(self.users) * 0.3)
                for user in random.sample(self.users, affected_count):
                    user.activity_level = 0
                    
    def start(self):
        """Start the simulator and metrics server"""
        # Start Prometheus metrics server
        port = self.config.get('metrics_port', 8000)
        start_http_server(port)
        self.logger.info(f"Metrics server started on port {port}")
        
        # Run default scenario
        scenario = self.config.get('default_scenario', 'baseline')
        self.run_scenario(scenario)

if __name__ == "__main__":
    import sys
    
    config_file = sys.argv[1] if len(sys.argv) > 1 else "simulator-config.yml"
    simulator = Simulator(config_file)
    
    try:
        simulator.start()
    except KeyboardInterrupt:
        simulator.running = False
        logging.info("Simulator stopped")

[aygp-dr]

@claude implement this feature based on the issue description

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[claude]

Claude finished @aygp-dr's task —— View job • claude/issue-3-20250525_000632 • [Create PR ➔](https://github.com/aygp-dr/claude-code-metrics-lab/compare/main...claude/issue-3-20250525_000632?quick_pull=1&title=Issue%20%233%3A%20Changes%20from%20Claude&body=This%20PR%20addresses%20issue%20%233%0A%0AGenerated%20with%20%5BClaude%20Code%5D(https%3A%2F%2Fclaude.ai%2Fcode)

---

I'll analyze this and get back to you.