Skip to content

PERFORMANCE.md

Latest commit

 

History

History
175 lines (134 loc) · 5.69 KB

PERFORMANCE.md

File metadata and controls

175 lines (134 loc) · 5.69 KB
Raw

Performance & Resource Usage

GPU Power Consumption

Shimmy v2.0 uses the Airframe WebGPU engine (wgpu), replacing the llama.cpp backend from v1.x. GPU memory and power consumption patterns are broadly similar since both approaches run the transformer on GPU.

GPU Memory Usage

  • Similar to Ollama: Both use identical GGUF model loading, so GPU memory consumption per model is nearly identical
  • Model-dependent: GPU memory usage depends on model size and quantization level, not the inference server
  • Typical usage:
    • 7B models (Q4_0): ~4-5GB VRAM
    • 13B models (Q4_0): ~8-10GB VRAM
    • 70B models (Q4_0): ~40-45GB VRAM

Power Consumption

  • CPU overhead: Shimmy uses ~50MB RAM vs Ollama's 200MB+ - less CPU power needed
  • GPU utilization: Identical inference workload = same GPU power draw
  • Idle efficiency: Shimmy's smaller binary means faster startup/shutdown cycles

Performance Comparison

Metric Shimmy Ollama Notes
GPU Memory per Model Same Same Both use GGUF model format
GPU Power Draw Same Same Identical inference workload
CPU RAM Overhead ~50MB* 200MB+ Shimmy more efficient
Startup Power Spike Low High Lightweight vs 680MB binary
Idle Power Lower Higher Smaller memory footprint

*Use included benchmarking tools to measure actual overhead on your system

Model-Specific Resource Usage

Small Models (1B-3B parameters)

Model: Phi-3-mini-4k (Q4_0)
- GPU Memory: ~2-3GB
- GPU Power: ~50-80W during inference
- CPU RAM: ~1GB + 50MB Shimmy overhead

Medium Models (7B-13B parameters)

Model: Llama-3.2-7B (Q4_0)
- GPU Memory: ~4-5GB
- GPU Power: ~100-150W during inference
- CPU RAM: ~2GB + 50MB Shimmy overhead

Large Models (30B+ parameters)

Model: Llama-3.1-70B (Q4_0)
- GPU Memory: ~40-45GB (requires multiple GPUs)
- GPU Power: ~300-400W during inference
- CPU RAM: ~8GB + 50MB Shimmy overhead

Optimization Features

Automatic GPU Detection

  • NVIDIA: Vulkan / D3D12 backend selected via wgpu adapter enumeration
  • AMD: Vulkan backend (Linux / Windows); Metal on macOS via wgpu
  • Apple Silicon: Metal backend selected automatically
  • Intel: Vulkan / D3D12 adapter selected by wgpu

Use shimmy gpu-info to see which adapter wgpu selected on your system.

Memory Management

  • Smart batching: Processes multiple requests efficiently
  • Model caching: Keeps frequently used models in VRAM
  • Graceful fallback: CPU inference when GPU memory full

Power Efficiency

  • Fast startup: No warmup period, immediate inference
  • Efficient shutdown: Clean VRAM release on exit
  • Minimal overhead: Single binary, no Python runtime, no daemon dependencies

Benchmarking Tools

Automated Benchmarking

Use the included benchmarking tools to measure actual performance on your system:

# Linux/macOS
./scripts/benchmark.sh --requests 20 --output my_results.json

# Windows
scripts\benchmark.bat --requests 20 --output my_results.json

# Python (if available)
python scripts/benchmark.py --requests 20 --output my_results.json

What the benchmark measures:

  • Real response times and throughput
  • GPU memory usage and power draw (NVIDIA GPUs)
  • CPU and system memory utilization
  • Success rates and error handling
  • Shimmy process overhead

Manual Monitoring

# GPU usage (NVIDIA)
nvidia-smi -l 1

# System resources
top -p $(pgrep shimmy)

# Detailed GPU memory
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1

Built-in Metrics

# Check model memory usage
curl http://localhost:11434/v1/models

# Health check with resource info
curl http://localhost:11434/health

Comparing with Ollama

Identical Aspects

  • Model loading: Same GGUF format, same memory requirements
  • GPU utilization: Both run the transformer on GPU; Shimmy uses Airframe (wgpu), Ollama uses llama.cpp
  • Model compatibility: Same GGUF models work in both systems

Shimmy Advantages

  • Lower system overhead: 50MB vs 200MB+ base memory usage
  • Faster startup: <100ms vs 5-10s startup time
  • Smaller footprint: Single binary vs 680MB Ollama installation

Power Efficiency Summary

Scenario: Running Llama-3.2-7B for 1 hour
- GPU power draw: ~120W (same for both)
- CPU overhead: Shimmy ~2W less due to efficiency
- Total savings: ~2W continuous + startup/shutdown efficiency

Best Practices

For Maximum Efficiency

  1. Use appropriate quantization: Q4_0 for most use cases
  2. Monitor GPU memory: Don't exceed 90% VRAM capacity
  3. Batch requests: Let Shimmy handle request queuing
  4. Close unused models: Free VRAM when switching between large models

Power Management

  1. Development: Use smaller models (1B-3B) for testing
  2. Production: Scale up to larger models as needed
  3. Multi-GPU: Shimmy supports automatic multi-GPU distribution

Troubleshooting High Usage

# Check which models are loaded
curl http://localhost:11434/v1/models

# Force model unload (restart server)
pkill shimmy && shimmy serve

# Monitor memory leaks
watch -n 1 'ps aux | grep shimmy'

Conclusion

GPU consumption with Shimmy is essentially identical to Ollama since both use the same inference engine. The key advantages are:

  • Lower system overhead (50MB vs 200MB+)
  • Faster startup/shutdown cycles
  • Same inference performance with less bloat

For GPU power consumption specifically, expect the same usage patterns as Ollama with the same models.