You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat: add parallel execution, debug comparison, and performance tracking to model
benchmark
What:
- Parallel analysis mode (--parallel/-p): ThreadPoolExecutor grouped by model_id
(models sequential for env-var safety, stocks within each model concurrent)
- Live debug comparison (--debug/-d): side-by-side table per stock showing
Model | Signal | Conf | Latency | Tokens | Cost | SL/TP, with
consensus/divergence detection and fastest/cheapest per stock highlighted
- Performance tracking: latency via time.perf_counter(), token usage from
agent result stats, cost estimation from per-model pricing table
(_MODEL_PRICING, 18 models), stored in context_snapshot.benchmark_meta
- Cost-efficiency score: composite_score / log10(1 + total_cost_usd * 100)
(higher = more accuracy per dollar)
- Configurable --max-parallel (default 3) to throttle API concurrency
- Performance leaderboard in reports: avg_latency_ms, total_tokens, cost_usd,
cost_efficiency — with "Value Pick" recommendation
Why:
- Sequential model-by-model analysis was too slow for practical benchmarking
(8 models × 5 stocks = 40 sequential analysis runs)
- Users needed visibility into latency/cost tradeoffs — a cheap model may be
slightly less accurate but 10x cheaper, making it the practical best choice
- Debug mode enables real-time observation of model divergence on the same
stock, helping users understand when models agree/disagree
Verification:
- Syntax check: python -c "import ast; ast.parse(...)" passed
- Import chain: all 7 classes + 14 top-level functions import correctly
- CLI help: --parallel, --max-parallel, --debug flags all present
- _ModelOverride context manager: correctly sets LITELLM_MODEL + AGENT_LITELLM_MODEL
- Cost estimation: estimate_cost("claude-sonnet-4-6", 5000, 1000) = $0.03 ✓
- Pricing table: 18 models with per-M-token pricing (Anthropic, OpenAI, Google, DeepSeek,
etc.)
Not verified:
- End-to-end analysis with real API calls (needs valid .env and API credits)
- Backtest + report pipeline with benchmark-tagged data (needs forward price data)
- Thread safety under heavy parallel load (tested with Lock pattern, not stress-tested)
Risk:
- ThreadPoolExecutor + _ModelOverride env var mutation: race condition possible
if two threads call setup_env() simultaneously despite Lock. Mitigation:
models are processed sequentially, only stocks within the same model are
parallelized.
- Rate limits: parallel mode may trigger API rate limiting. Mitigation:
--max-parallel controls concurrency; users should start low (2-3) and increase.
- Cost estimation: hardcoded prices may drift from actual API billing.
Mitigation: self-documenting pricing table in source, easy to update.
Rollback:
- All new code is additive; removing the commit restores sequential-only
behavior and removes debug/performance fields from reports.
- No schema changes; benchmark_meta in context_snapshot is ignored by
non-benchmark code paths.
Commit 2 (doc): Design doc update
docs: update design-model-benchmark.html for parallel/debug/performance features
What:
- Updated architecture diagram to include ThreadPoolExecutor, debug comparison,
and performance/cost-efficiency ranking layers
- Added new design decisions: parallel execution, debug comparison, performance tracking
- Updated Phase 1 workflow with --parallel and --debug examples
- Updated CLI reference table with --parallel, --max-parallel, --debug flags
- Revised thread safety callout to explain model-grouping strategy
- Added edge cases: rate limit throttling, cost estimation accuracy
- Marked completed items: cost tracking ✅, parallel execution ✅
- Updated planned enhancements table
Why: AGENTS.md rule 1 — new features require design doc updates.
All three new capabilities (parallel, debug, performance) needed to be
documented in the architecture, workflow, CLI, edge cases, and limitations
sections.
Verification:
- HTML structure intact (xmllint --html passes, HTML5 warnings only)
- Key terms present: parallel (14x), debug (6x), cost-efficiency (3x),
ThreadPoolExecutor (7x)
- File size: 29KB (up from 23KB)
Risk: None (docs only)
Rollback: Revert to previous design doc version
<tr><td>Benchmark tagging via <code>context_snapshot</code></td><td><code>{"model_id": "...", "benchmark": true}</code> — report phase JOINs <code>backtest_results</code> ↔ <code>analysis_history</code> to filter by model</td></tr>
232
251
<tr><td>5-day default eval window</td><td>US equities move faster than A-shares; 5 trading days is one calendar week and sufficient for short-term signal validation</td></tr>
233
252
<tr><td>Separate analyze/evaluate phases</td><td>Analysis runs immediately; evaluation must wait for forward price data. Decoupling allows scheduling each phase independently</td></tr>
253
+
<tr><td>Parallel execution (--parallel)</td><td><code>ThreadPoolExecutor</code> grouped by model: models run sequentially (env-var safety), stocks within each model run concurrently (speed). Configurable <code>--max-parallel</code> workers (default 3)</td></tr>
254
+
<tr><td>Debug comparison (--debug)</td><td>Live side-by-side table per stock: Model | Signal | Conf | Latency | Tokens | Cost | SL/TP — with consensus/divergence analysis, fastest & cheapest per stock highlighted</td></tr>
255
+
<tr><td>Performance tracking</td><td>Latency via <code>time.perf_counter()</code>, token usage from agent result stats, cost estimation via per-model pricing table (<code>_MODEL_PRICING</code>). Stored in <code>benchmark_meta</code> within <code>context_snapshot</code>. Report includes performance leaderboard + cost-efficiency ranking</td></tr>
<li><code>discover_models()</code> reads <code>config.llm_model_list</code> (aggregated from LLM_CHANNELS + legacy env vars)</li>
307
-
<li>For each model, <code>_ModelOverride</code> temporarily sets <code>LITELLM_MODEL</code></li>
339
+
<li>Work items are grouped by model_id; models are processed sequentially (for env-var safety via <code>_ModelOverride</code>)</li>
340
+
<li>Within each model group, stocks are dispatched concurrently via <code>ThreadPoolExecutor</code> (if <code>--parallel</code> is set)</li>
308
341
<li><code>build_agent_executor()</code> creates a fresh agent with the overridden model</li>
309
342
<li>Full pipeline runs (Technical → Intel → Decision) for each stock</li>
310
-
<li>Dashboard output is stored in <code>analysis_history</code> with <code>context_snapshot={"model_id":"...","benchmark":true}</code></li>
343
+
<li>If <code>--debug</code>: latency (<code>time.perf_counter()</code>), token usage, cost are captured and displayed in real-time side-by-side comparison table</li>
344
+
<li>Dashboard output is stored in <code>analysis_history</code> with <code>context_snapshot={"model_id":"...","benchmark":true,"benchmark_meta":{...}}</code></li>
311
345
</ol>
312
346
313
347
<h3>Phase 2: Evaluate (Day 5+)</h3>
@@ -390,7 +424,7 @@ <h2 id="model-override">6. Model Override Mechanism</h2>
390
424
391
425
<divclass="callout callout-warning">
392
426
<divclass="callout-title">Thread Safety</div>
393
-
<p>This mutates <em>process-level</em> environment variables. Concurrent benchmark runs on the same process are <strong>not supported</strong>. Each phase (analyze/evaluate/report) should run sequentially in its own process invocation.</p>
427
+
<p>This mutates <em>process-level</em> environment variables. The parallel execution strategy (<code>--parallel</code>) mitigates this by grouping work items by model and processing model groups <strong>sequentially</strong> — only stocks within the same model group run concurrently via <code>ThreadPoolExecutor</code>. A <code>threading.Lock</code> guards env var mutations. Subprocess isolation is recommended for fully independent multi-model parallelism.</p>
<tr><td><code>--models</code></td><td>Limit to specific models (default: all discovered)</td><td><code>--models gemini/gemini-3.1-pro-preview,openai/gpt-5.5</code></td></tr>
409
443
<tr><td><code>--days</code></td><td>Evaluation window in trading days (default: 5)</td><td><code>--days 10</code></td></tr>
410
444
<tr><td><code>--json</code></td><td>Output report as JSON instead of formatted text</td><td><code>--report --json</code></td></tr>
445
+
<tr><td><code>--parallel, -p</code></td><td>Run model analyses concurrently via ThreadPoolExecutor (stocks within each model parallelized)</td><td><code>--parallel</code></td></tr>
446
+
<tr><td><code>--max-parallel</code></td><td>Max concurrent stock invocations per model (default: 3)</td><td><code>--max-parallel 5</code></td></tr>
447
+
<tr><td><code>--debug, -d</code></td><td>Capture performance metadata (latency, tokens, cost) and show live side-by-side comparison</td><td><code>--debug</code></td></tr>
<summary><strong>Config reload side effects</strong></summary>
444
-
<p><code>_ModelOverride</code> calls <code>setup_env(override=True)</code> which reloads <code>dotenv</code>. This may affect other loaded modules. Mitigation: benchmark CLI always runs as a fresh process invocation.</p>
481
+
<p><code>_ModelOverride</code> calls <code>setup_env(override=True)</code> which reloads <code>dotenv</code>. This may affect other loaded modules. Mitigation: benchmark CLI always runs as a fresh process invocation. Parallel mode groups work by model to minimize env-var thrash.</p>
<p>Running multiple model invocations concurrently may trigger API rate limits (especially for OpenAI and Anthropic). Mitigations: (a) use <code>--max-parallel</code> to limit concurrency (default 3), (b) models are still processed sequentially — only stocks within one model are parallelized, (c) fall back to sequential mode if rate-limit errors persist.</p>
<p>Cost estimates use a hardcoded pricing table (<code>_MODEL_PRICING</code>) and token counts from the agent result metadata. Actual API billing may differ due to cached tokens, batch pricing, or provider-specific discounts. Estimates should be treated as <em>comparative</em> (Model A vs Model B) rather than <em>absolute</em> dollar amounts.</p>
<h2id="limitations">9. Limitations & Future Work</h2>
449
496
450
497
<h3>Current Limitations</h3>
451
498
<ul>
452
-
<li><strong>Process-level env mutation</strong> — <code>_ModelOverride</code> is not thread-safe. Parallel model analysis would require subprocess isolation.</li>
453
-
<li><strong>No cost tracking</strong> — Does not factor in API cost per model. A slightly less accurate but 10x cheaper model may be the practical best choice.</li>
499
+
<li><strong>Process-level env mutation</strong> — <code>_ModelOverride</code> is not thread-safe. Parallel mode (<code>--parallel</code>) mitigates by grouping work items by model and processing model groups sequentially with a <code>threading.Lock</code>. Full multi-model parallelism would require subprocess isolation.</li>
500
+
<li><strong><del>No cost tracking</del> ✅ Implemented</strong> — Cost estimation via <code>_MODEL_PRICING</code> table, token capture from agent stats, cost-efficiency ranking in performance leaderboard. Limitations: hardcoded prices, no real-time billing API integration.</li>
501
+
<li><strong><del>Sequential-only execution</del> ✅ Implemented</strong> — <code>--parallel</code> flag now supports concurrent stock analysis via <code>ThreadPoolExecutor</code> with per-model grouping.</li>
454
502
<li><strong>Fixed eval window</strong> — All stocks share the same 5-day window. In practice, volatile vs stable stocks may need different horizons.</li>
455
503
<li><strong>No statistical significance test</strong> — Current ranking is purely score-based. Small sample sizes may produce misleading rankings.</li>
456
504
<li><strong>Single run per (stock, model)</strong> — LLM outputs have inherent randomness. Multiple runs per combination would give confidence intervals.</li>
0 commit comments