Skip to content
BranchesTags

Repository files navigation

Hermes Profiler Plugin

Native performance profiler and benchmark framework for Hermes Agent.

Version: 1.2.0 Tests: 102 passing (57 → 102, +45 new in this release) License: MIT

Features

  • Per-call profiling — Tracks latency, token usage, and estimated cost for every LLM call and tool execution
  • Dynamic pricing — Fetches real-time rates from OpenRouter API with a 24-hour cache; background auto-refresh thread re-fetches while Hermes is running
  • Local model costs — Track true electricity + hardware amortization for Ollama, LM Studio, vLLM, llama.cpp, localai. Includes a "GPU underutilization surcharge" when observed throughput is well below the configured maximum
  • User overrides — Configure custom rates via ~/.hermes/profiler_overrides.yaml; hot-reloadable without restarting Hermes
  • Granular tool error taxonomy — Distinguishes ToolTimeout, ToolPermissionDenied, ToolResourceMissing, ToolRateLimited, ToolConnectionError, ToolInputError, etc. instead of collapsing every failure into ToolError
  • Zero core modifications — Pure plugin using the Hermes hook system
  • Persistent storage — SQLite database at ~/.hermes/profiler.db in WAL mode (concurrent readers + single writer)
  • Benchmark runner — Compare models/providers side-by-side with optional warmup runs
  • CLI integration — hermes profiler and hermes benchmark commands
  • Export & top-N queries — Dump a session's call log to JSON/CSV, or find your most expensive / slowest calls across all sessions

What's new in v1.2.0

  • 22 bugs fixed, 14 enhancements shipped
  • Cross-session data loss bug — on_session_end no longer wipes in-flight state from concurrent sessions
  • Silent localhost-as-remote failures — Custom-URL localhost endpoints are now correctly detected as local and use electricity/hardware pricing
  • Currency-formatted pricing — OpenRouter responses like "$0.00015" no longer crash the refresh
  • Negative token counts — Clamped to 0 instead of producing negative cost
  • Real background rate refresh — The README's claim is now true
  • hermes profiler top — Find your most expensive or slowest calls
  • hermes profiler export — Dump a session's call log to JSON or CSV
  • hermes profiler reload — Re-read YAML config without restarting
  • --since filter on hermes profiler list
  • --warmup N on hermes benchmark run
  • % delta columns in hermes profiler compare

See CHANGELOG.md for the full per-bug breakdown.

Installation

Option 1: Clone into user plugins directory (recommended)

git clone https://github.com/spfcraze/Hermes-profile-benchmark.git ~/.hermes/plugins/profiler

This creates the directory at ~/.hermes/plugins/profiler/ which is the path the Hermes plugin loader expects. The plugin is a Python package — its __init__.py imports profiler.rate_engine etc., so the loader must treat it as a package (which ~/.hermes/plugins/profiler/__init__.py satisfies).

Option 2: Copy from a local checkout

cp -r /path/to/Hermes-profile-benchmark ~/.hermes/plugins/profiler

Enable the plugin

hermes plugins enable profiler

Or add to ~/.hermes/config.yaml:

plugins:
  enabled:
    - profiler

CLI Commands

hermes profiler session <session_id>

Show profile stats for a session.

hermes profiler session abc-123

Output:

Profile for session: abc-123

Metric                  Value
─────────────────────────────────────
LLM Calls               12
Tool Calls              8 (7✓ / 1✗)
Total Tokens            45,230 (32,180 in / 13,050 out)
Cumulative Latency      18,450ms
Wall Time               42.7s
Total Cost              $0.045230
  Confirmed (priced)    $0.045230
  Unconfirmed           $0.000000
Avg LLM Latency         1,200ms
Avg Tool Latency        45ms

Each row of the call table also shows its cost source (openrouter_api, static_fallback, user_override, local_model, local_model_unconfigured, or none) so you can tell which calls have real pricing and which fall back to $0.00 with confidence="unknown".

Optional export:

hermes profiler session abc-123 --output session.json --format json
hermes profiler session abc-123 --output session.csv  --format csv

hermes profiler compare <session_a> <session_b>

Compare two sessions side-by-side, with both absolute and percentage deltas.

hermes profiler compare abc-123 def-456

Output:

Comparing sessions: abc-123 vs def-456

Metric            abc-123         def-456         Delta      Δ%
───────────────────────────────────────────────────────────────
llm_calls         12              18              +6         +50.0%
tool_calls        8               11              +3         +37.5%
total_tokens      45,230          71,500          +26,270    +58.1%
total_latency_ms  18,450.00       22,100.00       +3,650.00  +19.8%
total_cost        0.05            0.07            +0.03      +50.0%

hermes profiler list [--limit N] [--since DATE_OR_TS]

List recently profiled sessions. --since accepts either an ISO date (2026-01-01) or a Unix timestamp (1735689600).

hermes profiler list
hermes profiler list --limit 50
hermes profiler list --since 2026-01-01
hermes profiler list --since 1735689600

hermes profiler top [--metric cost|latency] [--limit N]

Show the top N most expensive or slowest calls across all sessions. Useful for diagnosing which model/prompt combinations are eating your budget.

hermes profiler top --metric cost --limit 20
hermes profiler top --metric latency --limit 5

Output:

Session    Type  Name                          Tokens  Latency  Cost        Source
──────────────────────────────────────────────────────────────────────────────────
abc-123    llm   openrouter/anthropic/claude-3-opus  4,210  8,200ms $0.063150  openrouter_api
def-456    llm   openrouter/openai/gpt-4o       1,890  3,400ms $0.009450  openrouter_api
...

hermes profiler export <session_id> --format {json|csv} --output FILE

Export a single session's call log for offline analysis.

hermes profiler export abc-123 --format csv  --output session.csv
hermes profiler export abc-123 --format json --output session.json

hermes profiler reload

Reload profiler_overrides.yaml and profiler_local.yaml from disk without restarting Hermes. Also auto-reloads on the next get_rate() call if the file's mtime has changed.

hermes profiler reload

hermes profiler refresh-rates

Force-fetch latest pricing from OpenRouter API. Normally the plugin runs a background thread that re-fetches every 24 hours while Hermes is running.

hermes profiler refresh-rates

hermes profiler local-config

Show local model cost configuration path and template.

hermes profiler local-config

Create ~/.hermes/profiler_local.yaml:

local_models:
  ollama/llama3.1:70b:
    gpu_watts: 450
    electricity_cost_per_kwh: 0.12
    tokens_per_second: 45
    amortized_gpu_cost_per_hour: 0.50
  ollama/phi4:
    gpu_watts: 120
    electricity_cost_per_kwh: 0.12
    tokens_per_second: 120
    amortized_gpu_cost_per_hour: 0.50

Cost formula:

electricity = (gpu_watts / 1000) * hours * electricity_cost_per_kwh
hardware    = amortized_gpu_cost_per_hour * hours
surcharge   = +10% of hardware if observed_tps < 50% of configured_tps

The surcharge (added in v1.2.0) flags the case where you're paying for a bigger GPU than your workload actually needs. The minimum billing window is 1 second — sub-second calls get billed as if they took 1 full second to account for GPU spin-up cost.

hermes benchmark run

Run a benchmark suite across multiple models/providers.

# Create prompts file (one per line, or .jsonl with {"prompt": "..."} objects)
cat > prompts.txt << 'EOF'
Explain quantum computing in one paragraph.
Write a Python function to reverse a linked list.
EOF

# Run benchmark
hermes benchmark run \
  --prompts prompts.txt \
  --models "openrouter/gpt-4o,anthropic/claude-sonnet-4" \
  --iterations 3 \
  --warmup 1 \
  --output report.json \
  --format json

Options:

Flag Required Default Description
--prompts FILE yes — Path to prompts file (txt or jsonl)
--models LIST yes — Comma-separated provider/model specs
--iterations N no 1 Iterations per prompt
--warmup N no 0 Warmup runs per model (executed before measurement; useful for cold-start amortization)
--output FILE no — Export results to file
--format {json,csv} no json Export format

Agent adapter: The benchmark runner needs an AIAgent class to actually chat with the model. It tries these modules in order: run_agent, hermes_agent, agent. If none is found, it exits with a clear error message rather than crashing with a bare ImportError.

Caching: Agents are cached per (provider, model) tuple. A 10-prompt benchmark against 3 models only constructs 3 agents total, not 30.

Configuration Files

User Rate Overrides

Create ~/.hermes/profiler_overrides.yaml:

overrides:
  openai/gpt-4o:
    prompt: 1.0      # per 1K tokens
    completion: 2.0  # per 1K tokens
  anthropic/claude-opus:
    prompt: 15.0
    completion: 75.0

These take precedence over OpenRouter and static fallback rates. Hot-reloadable with hermes profiler reload (or automatically on the next get_rate() call after the file's mtime changes).

Local Model Costs

See hermes profiler local-config above. Detected local providers: ollama, vllm, lmstudio, localai, llamacpp. Also detects localhost / 127.0.0.1 / [::1] in api_base URLs (any port or path).

Pricing Architecture

The profiler uses a three-tier lookup system for cost estimation:

Tier 1: OpenRouter API Cache (refreshed every 24h by background thread)
   ↓ miss / stale-but-usable
Tier 2: User Overrides (~/.hermes/profiler_overrides.yaml)
   ↓ not found
Tier 3: Static Fallback Table (embedded, always available)
   ↓ not found
Result: $0.00 with confidence="unknown" and source="none"

Lookup fallbacks (Tier 3):

  1. Exact match on provider/model key
  2. Strip redundant openrouter/ prefix and retry
  3. Try every sub-sequence of the model name parts (handles claude-3-opus → claude-opus, gpt-4o-2024-08-06 → gpt-4o, gpt-4-turbo-preview → gpt-4-turbo)
  4. Longest bidirectional prefix match (e.g. gpt-4-turbo-preview → gpt-4-turbo rate, not the more general gpt-4 rate)

Local models (Ollama, LM Studio, vLLM, etc.) use a separate cost model based on GPU power draw, electricity rates, hardware amortization, and efficiency — see hermes profiler local-config for the formula.

Cost source labels (used in CLI output and the cost_source DB column):

Source Meaning
openrouter_api Fresh rate (≤ 24h) fetched from OpenRouter
openrouter_api_stale Cached rate older than 24h but newer than 7 days
user_override User-defined rate from profiler_overrides.yaml
static_fallback Embedded static table rate
local_model Local model with electricity/hardware config
local_model_unconfigured Local model detected but no profiler_local.yaml entry
none Unknown model — cost reported as $0.00

Supported Providers for Cost Estimation

Provider Models Source
OpenRouter 350+ models Live API (cached 24h)
OpenAI GPT-4o, GPT-4, GPT-3.5 Static fallback
Anthropic Claude Opus, Sonnet, Haiku Static fallback
Google Gemini Pro, Flash Static fallback
Meta Llama 3, Llama 3.1 (8B/70B) Static fallback
Mistral Mistral Large, Medium, Small Static fallback
Cohere Command-R, Command-R+ Static fallback
xAI Grok Static fallback
DeepSeek DeepSeek Chat, Coder Static fallback
Perplexity Sonar Static fallback
Kimi K2, K2.6 Static fallback
Nous Hermes 2 Mixtral Static fallback
Ollama / vLLM / LM Studio / llama.cpp / localai All models User-configured electricity/hardware

Architecture

Hermes Agent Loop
    │
    ├── pre_api_request   ──→ profiler records start time (+ session_id)
    ├── post_api_request  ──→ profiler records latency + tokens + cost (with api_base forwarding)
    ├── pre_tool_call     ──→ profiler records start time (+ session_id)
    ├── post_tool_call    ──→ profiler records latency + granular error_type
    └── on_session_end    ──→ profiler cleans up state for ENDED session only (not all)

Cost flow:
  _estimate_cost(provider, model, prompt, completion, latency, api_base)
    │
    ├── Detect local model (provider in {ollama,vllm,...} OR api_base matches localhost)
    │   └── LocalModelCost.calculate_cost() → electricity + hardware + efficiency surcharge
    ├── Check user overrides (~/.hermes/profiler_overrides.yaml)
    │   └── Direct user_configured rate
    └── RateEngine.lookup()
        ├── Tier 1: rate_cache (fresh → exact, 24h-7d → estimated+stale label)
        ├── Tier 2: _STATIC_RATES exact match
        ├── Tier 3: strip openrouter/ prefix + sub-sequence match + longest prefix match
        └── Miss → return 0.00 with confidence="unknown"

Data flows to ~/.hermes/profiler.db (SQLite, WAL mode)
Background thread: re-fetches OpenRouter rates every 24h
TTL cleanup: drops stale _CallState entries every 5 minutes

Files

File Purpose
plugin.yaml Plugin manifest (version, hooks, description)
__init__.py Hook handlers, storage layer, CLI commands, benchmark runner, background refresh thread
rate_engine.py Dynamic pricing engine: OpenRouter fetch with retry/backoff, three-tier lookup, thread-local SQLite, currency-safe float parser
local_model.py Local model cost calculator (electricity + hardware + efficiency surcharge) with hot-reload
overrides.py User-defined rate override loader with hot-reload + whitespace-tolerant keys
tests/test_profiler_plugin.py Hook handler + query + plugin registration tests (36 tests)
tests/test_rate_engine.py RateEngine tests including retry, currency parsing, version-stripping, concurrent refresh+lookup (38 tests)
tests/test_local_model.py Local model cost + efficiency surcharge + min billing window tests (15 tests)
tests/test_overrides.py User override loading, hot-reload, whitespace stripping tests (7 tests)
tests/test_benchmark.py Benchmark runner including warmup, agent caching, missing-module handling (6 tests)
tests/fixtures/openrouter_models.json Fixture for refresh tests

Testing

cd /path/to/Hermes-profile-benchmark
PYTHONPATH=. pytest tests/ -v

102 tests pass (5.6s on a cold start), covering:

  • Cost estimation (known rates, unknown rates, prefix matching, version-suffix stripping, negative token clamping)
  • OpenRouter refresh (happy path, network errors, timeouts, invalid JSON, null pricing, currency-formatted pricing, User-Agent header, retry with backoff, concurrent refresh + lookup)
  • Database operations (WAL mode, schema, insert LLM, insert tool, top-N by cost and latency)
  • Hook handlers (pre/post API request, pre/post tool call, on_session_end isolation across concurrent sessions)
  • Tool error detection (granular taxonomy: Timeout, PermissionDenied, ResourceMissing, RateLimited, ConnectionError, InputError, etc.)
  • Local model cost (basic calculation, zero-tokens short-circuit, min billing window, efficiency surcharge)
  • User override loading (no config, valid config, hot-reload, whitespace stripping, auto-reload on mtime change)
  • Benchmark runner (success path, failure handling, warmup, agent caching, missing-module graceful failure)
  • Thread safety (concurrent LLM calls, concurrent refresh + lookup)
  • Plugin registration (all 5 hooks + 2 CLI commands)

Troubleshooting

"All my calls show $0.00 with source none" The model isn't in the static fallback table and OpenRouter hasn't been fetched. Run hermes profiler refresh-rates to populate the cache, or add an entry to ~/.hermes/profiler_overrides.yaml.

"My localhost model shows as remote API and charges $0.00 silently" Make sure you're on v1.2.0+ and passing the api_base URL through the provider config. Older versions didn't forward api_base to the cost estimator, so localhost endpoints fell through to the static table.

"Refresh fails with network error" The plugin retries 3× with backoff. If it still fails, check your network and OpenRouter status. Stale cache (up to 7 days old) will continue to be used as source="openrouter_api_stale" until the network recovers.

"Cost shows as unconfigured even though I have a config file" Run hermes profiler local-config to verify the path. Then hermes profiler reload to pick up changes. Or check that the model id in profiler_local.yaml matches the provider/model string exactly (whitespace is stripped automatically, but other typos are not).

"Old test data is showing up under the wrong session_id" The DB uses SQLite WAL mode. If you've upgraded from v1.0.x, old data has the same calls schema and is fully compatible. The only schema changes in v1.2.0 are two new indexes (idx_calls_cost, idx_calls_latency) which CREATE INDEX IF NOT EXISTS handles transparently on existing DBs.

License

MIT

About

Hermes Benchmark testing per profile

Resources

Readme
MIT license
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages