Commit 970920c
committed
feat: add model benchmark system for multi-LLM stock prediction validation
New module: src/services/model_benchmark.py (1062 lines)
Provides automated comparison of LLM models for US stock prediction:
- DISCOVER: reads all configured models from llm_model_list (LLM_CHANNELS
+ legacy env vars), no manual enumeration needed
- ANALYZE: runs identical stock analysis through each model, stores
results tagged with model_id in analysis_history.context_snapshot
- EVALUATE: reuses existing BacktestEngine to backtest predictions
against actual forward prices after configurable eval window
- REPORT: ranks models by composite score (direction accuracy 35%,
win rate 30%, excess return 20%, consistency 10%, conviction 5%)
CLI: python -m src.services.model_benchmark [--analyze|--evaluate|--report|--full]
Design:
- Zero schema changes — model_id stored in existing context_snapshot JSON
- _ModelOverride context manager temporarily sets LITELLM_MODEL per run
- Deduplication via get_configured_llm_models() from authoritative config
- Per-stock breakdown in report for granular accuracy analysis
Docs:
- docs/design-model-benchmark.html — full architecture, scoring formula,
data model, CLI reference, edge cases, limitations & future work
- docs/LINUX_DEPLOYMENT.html — added benchmark CLI section to quick reference
Governance:
- AGENTS.md: new hard rule — all new features require "三件套" (design doc,
deploy doc update, commit comments) to be considered complete
Verified: import chain, CLI --help, --list-models, text + JSON report
formatting with synthetic data.1 parent d0c96e2 commit 970920c
5 files changed
Lines changed: 1568 additions & 0 deletions
File tree
- docs
- src/services
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
27 | 31 | | |
28 | 32 | | |
29 | 33 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
632 | 632 | | |
633 | 633 | | |
634 | 634 | | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
635 | 653 | | |
636 | 654 | | |
637 | 655 | | |
638 | 656 | | |
639 | 657 | | |
| 658 | + | |
640 | 659 | | |
641 | 660 | | |
642 | 661 | | |
| |||
0 commit comments