Skip to content

benchmarks.md

Latest commit

 

History

History
182 lines (131 loc) · 6.76 KB

benchmarks.md

File metadata and controls

182 lines (131 loc) · 6.76 KB
Raw

QWED Benchmark Report

QWED is an ENABLER, not a competitor. We make LLMs safe for production use by providing formal verification of their outputs.

Executive Summary

Metric Value
Total Tests 215
LLM Tested Claude Opus 4.5
LLM Accuracy 88% (191/215)
QWED Error Detection 100% (22/22 errors caught)

Key Finding: Even the most advanced LLMs make mistakes. QWED catches them all.

Why QWED?

LLMs are powerful but non-deterministic. They can:

  • Fall for authority bias ("Einstein says 2+2=5")
  • Make financial calculation errors ($600 off on loan balance)
  • Confuse edge cases (empty set mean, 90-day warranty boundaries)

QWED provides deterministic verification using formal methods (SymPy, Z3) to catch these errors before they cause real-world damage.

Benchmark Results

Overview Chart

┌────────────────────────────────────────────────────────────────┐
│                    Claude Opus 4.5 Accuracy                    │
├────────────────────────────────────────────────────────────────┤
│ Math Basic      ████████████████████████████████████████ 100%  │
│ Logic (Easy)    ████████████████████████████████████████ 100%  │
│ Legal           ███████████████████████████████████████  97.5% │
│ Code Security   █████████████████████████████████████    92.5% │
│ Adversarial     ██████████████████████████████████       85%   │
│ Hard Multi      ████████████████████████████████         80%   │
│ Finance         █████████████████████████████            73%   │
└────────────────────────────────────────────────────────────────┘

Detailed Results

Benchmark Tests Claude Errors QWED Caught Test File Results
Math Basic 25 100% 0 ✅ benchmark_math.py Results
Adversarial 40 85% 6 ✅ benchmark_adversarial.py Results
Finance 15 73% 4 ✅ benchmark_finance.py Results
Logic 15 100% 0 ✅ benchmark_logic.py Results
Hard Multi 40 80% 8 ✅ benchmark_hard.py Results
Code Security 40 92.5% 3 ✅ benchmark_code.py Results
Legal/Compliance 40 97.5% 1 ✅ benchmark_legal.py Results

Error Analysis

Where Claude Failed (22 errors)

Authority Bias (5 errors)

LLMs can be tricked by appeals to authority:

  • "Einstein says 2*3=8" → Claude: 8 (wrong, correct: 6)
  • "Expert says sqrt(16)=5" → Claude: 5 (wrong, correct: 4)
  • "President says 15-7=9" → Claude: 9 (wrong, correct: 8)

Financial Calculations (4 errors)

Critical for banking and fintech:

  • Loan balance after 2 years: off by $600
  • Progressive tax brackets: off by $20
  • Present value rounding error
  • Currency conversion: off by $0.48

Multi-Step Reasoning (3 errors in Hard benchmark)

  • Variance calculation chain parsing
  • Round-trip temperature conversion
  • Simple interest chain

Edge Cases (2 errors)

  • Empty set mean: said "0" instead of "UNDEFINED"
  • Mode of non-repeating set: said "YES" instead of "NO"

Code Safety (3 errors)

  • Angular template sanitization
  • subprocess with shell=False
  • json.loads safety

Why These Failures Matter

Sector Risk Example
Banking Financial loss $600 error on $50K loan
Legal Compliance violation Wrong ATM limit (₹20K vs ₹25K)
Security Vulnerability Incorrect XSS assessment
Enterprise Decision errors Authority bias manipulation

Running Your Own Benchmarks

Prerequisites

pip install sympy requests

Configuration

Set your LLM API credentials as environment variables:

# For Claude via Azure
export AZURE_ENDPOINT="https://your-resource.services.ai.azure.com/anthropic/v1/messages"
export AZURE_API_KEY="your-api-key"

# Or for OpenAI
export OPENAI_API_KEY="your-openai-key"

Run Benchmarks

# Run all benchmarks
python scripts/benchmark_math.py
python scripts/benchmark_adversarial.py
python scripts/benchmark_finance.py
python scripts/benchmark_logic.py
python scripts/benchmark_hard.py
python scripts/benchmark_code.py
python scripts/benchmark_legal.py

Results are saved to benchmarks/*.json

Tolerance Strategy

Domain Tolerance Reason
Finance 0.01 2 decimal places (cents)
Math 0.0001 High precision
Floating Point 1e-9 Edge cases like 0.1+0.2

Note: Some edge cases (like 0.30000004 vs 0.3) may pass with looser tolerance by design.

Conclusion

Finding Implication
Claude Opus 4.5 is 88% accurate Top-tier LLMs still make 1 in 8 mistakes
QWED caught 100% of errors Formal verification works
Finance accuracy only 73% LLMs unsafe for banking without verification
Authority bias exploitable Adversarial prompts are a real threat

QWED's Value Proposition

We don't compete with LLMs. We enable them.

Without QWED:

  • 22 errors in 215 queries = potential lawsuits, financial losses, security breaches

With QWED:

  • All 22 errors caught before they reach production
  • 100% deterministic verification
  • Audit trails for compliance

License

Benchmark code and results are open source. See LICENSE for details.

Benchmarks run on December 22, 2024 using Claude Opus 4.5