Skip to content

Commit 970920c

Browse files
committed
feat: add model benchmark system for multi-LLM stock prediction validation
New module: src/services/model_benchmark.py (1062 lines) Provides automated comparison of LLM models for US stock prediction: - DISCOVER: reads all configured models from llm_model_list (LLM_CHANNELS + legacy env vars), no manual enumeration needed - ANALYZE: runs identical stock analysis through each model, stores results tagged with model_id in analysis_history.context_snapshot - EVALUATE: reuses existing BacktestEngine to backtest predictions against actual forward prices after configurable eval window - REPORT: ranks models by composite score (direction accuracy 35%, win rate 30%, excess return 20%, consistency 10%, conviction 5%) CLI: python -m src.services.model_benchmark [--analyze|--evaluate|--report|--full] Design: - Zero schema changes — model_id stored in existing context_snapshot JSON - _ModelOverride context manager temporarily sets LITELLM_MODEL per run - Deduplication via get_configured_llm_models() from authoritative config - Per-stock breakdown in report for granular accuracy analysis Docs: - docs/design-model-benchmark.html — full architecture, scoring formula, data model, CLI reference, edge cases, limitations & future work - docs/LINUX_DEPLOYMENT.html — added benchmark CLI section to quick reference Governance: - AGENTS.md: new hard rule — all new features require "三件套" (design doc, deploy doc update, commit comments) to be considered complete Verified: import chain, CLI --help, --list-models, text + JSON report formatting with synthetic data.
1 parent d0c96e2 commit 970920c

5 files changed

Lines changed: 1568 additions & 0 deletions

File tree

AGENTS.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@
2424
- **新增用户自维护的技术文档**(部署指南、配置指南、专题说明书等)优先使用 HTML 格式:语义化 HTML5 + 内嵌 `<style>`,不使用外部 CSS/JS 依赖。HTML 提供一致的跨平台渲染、原生 `<details>` 折叠、CSS 提示框组件、更明确的 DOM 结构便于 AI 工具解析与 diff 审核。上游 fork 同步的 `.md` 文档保持原格式不动,不转换。
2525
- 变更中英双语文档之一时,需评估另一份是否需要同步;若未同步,交付说明里要写明原因。
2626
- 注释、docstring、日志文案以清晰准确为准,不强制要求英文,但应与文件语境保持一致。
27+
- **新增功能/模块/子系统时,必须同步产出"三件套"方可视为交付完成**
28+
1. **设计文档**`docs/design-<name>.html`,说明架构、数据流、算法/打分公式、模块交互、边界条件、配置项语义;
29+
2. **部署/运维文档更新** — 如涉及新 CLI 入口、新配置项、新环境变量、新数据表或定时任务,必须同步更新对应的部署指南和配置指南;
30+
3. **英文 commit comments** — 供 maintainer 参考,包含 what/why/验证情况/风险点,遵循 `AGENTS.md` 交付结构。
2731

2832
## 2. AI 协作资产治理
2933

docs/LINUX_DEPLOYMENT.html

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -632,11 +632,30 @@ <h3>常用命令快速参考</h3>
632632
systemctl restart stock-analyzer # Systemd</code></pre>
633633

634634
<!-- ═══════════════════════════════════════════════════════════════ -->
635+
<h2>模型基准测试(Model Benchmark)</h2>
636+
637+
<p>系统支持多模型对比验证,用于评估不同 LLM 对美股预测的准确度:</p>
638+
639+
<pre><code># 发现所有已配置的模型
640+
python -m src.services.model_benchmark --list-models
641+
642+
# 对指定美股运行多模型分析
643+
python -m src.services.model_benchmark --analyze --stocks AAPL,NVDA,MSFT
644+
645+
# 等待评估窗口后运行回测
646+
python -m src.services.model_benchmark --evaluate --days 5
647+
648+
# 生成对比排行榜
649+
python -m src.services.model_benchmark --report</code></pre>
650+
651+
<p>详见 <a href="design-model-benchmark.html">Model Benchmark 设计文档</a></p>
652+
635653
<h2>相关文档</h2>
636654
<ul>
637655
<li><a href="DEPLOY.md">部署指南</a> - 详细的部署说明</li>
638656
<li><a href="full-guide.md">完整配置指南</a> - 高级配置选项</li>
639657
<li><a href="desktop-package.md">桌面端打包说明</a> - 桌面应用部署</li>
658+
<li><a href="design-model-benchmark.html">Model Benchmark 设计文档</a> - 多模型对比验证</li>
640659
<li><a href="FAQ.md">FAQ</a> - 常见问题解答</li>
641660
</ul>
642661

0 commit comments

Comments
 (0)