Skip to content

Commit 05239a5

Browse files
klau1011claude
andauthored
Model routing + adaptive risk debate; memory relevance + wider lessons injection (PRs 4-5) (#3)
* feat(graph): per-stage model routing + adaptive risk debate Two opt-in config knobs, both defaulting to today's exact behavior: - agent_llm_map: assign quick/deep per stage (analysts, researchers, research_manager, trader, risk_analysts, portfolio_manager, investor_briefing); validated eagerly in GraphSetup. Empty dict = current defaults. Reachable via config / RunnerConfig.extra_config. - adaptive_extra_rounds: risk debate may run up to N extra rounds past the base cap, but only while the Research Manager's and Trader's rating directions disagree (deterministic parse_rating/direction check, no LLM call). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(memory): |alpha|-ranked cross-ticker retrieval + wider lessons injection - get_past_context: cross-ticker selection changes from last-3 to top-3 by |alpha| (tie-break recency, unparseable alpha last) via a small _parse_pct helper; same-ticker selection stays recency. - The Portfolio Manager's guarded past-lessons block now also reaches the bull/bear researchers, the trader, and the three risk debators — rendered only when past_context is non-empty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: mark PRs 4-5 done in improvements plan Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: label-aware rating parse for Recommendation/Action; ruff import fix Addresses Codex review: parse_rating's label pass now recognizes Recommendation:/Action: (the Research Manager's and Trader's rendered shapes), so the adaptive-debate disagreement check no longer depends on the first-rating-word fallback, which prose like "despite the Buy recommendation, Sell" would mislead. Benefits all parse_rating callers. Also removes a double blank line after imports in graph/setup.py (ruff I001). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 852402a commit 05239a5

17 files changed

Lines changed: 450 additions & 117 deletions

docs/improvements-plan.md

Lines changed: 14 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,17 @@ shortest working diff, reuse existing utilities, no new dependencies.
1717
`<results_dir>/web_runs/`, API-key preflight (400), sanitized `ErrorEvent`,
1818
opaque `report_folder` references + "View report" link, History staleness
1919
fixes, rehydrated-run handling in `useRunStream`.
20-
21-
**Remaining: PRs 3–6 below.** Each is a self-contained change; suggested
20+
- **PR 4 — Model routing + adaptive risk debate** (`0517829`):
21+
`agent_llm_map` per-stage quick/deep assignment (validated in
22+
`GraphSetup`); `adaptive_extra_rounds` extends the risk debate past the
23+
base cap only while `direction(parse_rating(...))` disagrees between the
24+
Research Manager's and Trader's plans. Both default to today's behavior.
25+
- **PR 5 — Memory & reflection** (`4b0ef75`): cross-ticker
26+
`get_past_context` selection now top-3 by |alpha| (`_parse_pct`,
27+
tie-break recency, unparseable last); guarded past-lessons block injected
28+
into bull/bear researchers, trader, and the three risk debators.
29+
30+
**Remaining: PRs 3 and 6 below.** Each is a self-contained change; suggested
2231
order preserved. Per-PR verification: `pytest -m "unit or smoke"` (conftest
2332
stubs API keys), plus `npm run build` in `web/frontend` when frontend files
2433
change.
@@ -56,78 +65,6 @@ Verify: `event_to_dict(StatsEvent)` round-trip + stub-graph stats-emission
5665
test in `tests/test_runner_events.py` (`_StubGraph` pattern);
5766
`python -c "import cli.main"` smoke.
5867

59-
## PR 4 — Model routing + adaptive risk debate
60-
61-
Two opt-in config knobs, both defaulting to today's exact behavior.
62-
63-
**Per-stage model assignment**
64-
65-
1. `tradingagents/default_config.py`: add `"agent_llm_map": {}`. Stages:
66-
`analysts`, `researchers`, `research_manager`, `trader`, `risk_analysts`,
67-
`portfolio_manager`, `investor_briefing`; values `"quick"` / `"deep"`.
68-
Empty dict = current behavior (analysts + trader on quick, managers on
69-
deep). No env override (the `_coerce` machinery doesn't do dicts).
70-
2. `tradingagents/graph/setup.py`: `GraphSetup.__init__` gains
71-
`llm_map: dict | None = None` plus a `_llm(stage, default)` helper
72-
(raise `ValueError` on unknown stage/value); replace the hardcoded
73-
quick/deep picks (~lines 61–78).
74-
3. `tradingagents/graph/trading_graph.py` (~line 116): pass
75-
`self.config.get("agent_llm_map")`.
76-
77-
No UI exposure; reachable via config / `RunnerConfig.extra_config`.
78-
79-
**Adaptive risk debate**
80-
81-
Cheap deterministic disagreement signal, verified present in state: by the
82-
time `should_continue_risk_analysis` runs, `state["investment_plan"]`
83-
(Research Manager) and `state["trader_investment_plan"]` (Trader) both carry
84-
parseable ratings (`parse_rating` in `agents/utils/rating.py`; `direction()`
85-
landed in PR 1).
86-
87-
1. `tradingagents/graph/conditional_logic.py`: `__init__` gains
88-
`adaptive_extra_rounds=0`. In `should_continue_risk_analysis`, past the
89-
base cap but under `3 * (max_risk_discuss_rounds + adaptive_extra_rounds)`,
90-
continue only if
91-
`direction(parse_rating(investment_plan)) != direction(parse_rating(trader_investment_plan))`.
92-
2. `default_config.py`: `"adaptive_extra_rounds": 0`;
93-
`trading_graph.py` (~lines 112–115) passes it through.
94-
95-
Not doing: adaptive bull/bear debate (they disagree by construction),
96-
LLM-judged disagreement scoring.
97-
98-
Verify: pure-state `ConditionalLogic` unit tests (no LLM): agreement → stop
99-
at base cap; disagreement → exactly N extra cycles then stop; default 0 →
100-
identical to today.
101-
102-
## PR 5 — Memory & reflection
103-
104-
**Relevance-based retrieval.** `tradingagents/agents/utils/memory.py`
105-
`get_past_context` (~line 70): same-ticker selection stays recency (recent
106-
context on the same name is genuinely most relevant); cross-ticker selection
107-
changes from "last 3" to "top 3 by |alpha|" (biggest realized wins/losses
108-
carry the most instructive reflections), tie-break recency. Needs a small
109-
`_parse_pct("+3.2%") -> float | None` helper since entries store alpha as a
110-
formatted string (written in `update_with_outcome`, ~line 122). This is a
111-
magnitude heuristic; the upgrade path is tag/embedding matching. Not doing:
112-
embeddings, LLM relevance scoring, regime tags (on-disk format change).
113-
114-
**Wider injection.** Copy the guarded block pattern from
115-
`agents/managers/portfolio_manager.py:35-40` ("Past lessons" section,
116-
rendered only when `past_context` is non-empty) into:
117-
`researchers/bull_researcher.py`, `researchers/bear_researcher.py`,
118-
`trader/trader.py`, `risk_mgmt/aggressive_debator.py`,
119-
`risk_mgmt/conservative_debator.py`, `risk_mgmt/neutral_debator.py`.
120-
`past_context` is already in graph state (`graph/propagation.py:40`); all six
121-
nodes already receive `state`. Accepted token cost: the string is bounded by
122-
`n_same`/`n_cross` and the 300-char cross-ticker truncation.
123-
124-
Verify: `tests/test_memory_log.py` — |alpha| cross-ticker ordering and
125-
`_parse_pct`; per-agent prompt-capture tests with a fake `llm.invoke`
126-
(researchers/debators call `llm.invoke(prompt)` directly; trader via the
127-
fake-structured-LLM pattern in `tests/test_structured_agents.py`). Assert
128-
the block is present iff `past_context` is set. Measure the behavior change
129-
with the PR 1 harness before/after.
130-
13168
## PR 6 — Sizing outputs
13269

13370
Make the Portfolio Manager's sizing real instead of dead prose fields.
@@ -168,3 +105,6 @@ pass-through test in `tests/test_reports_module.py`.
168105
real use case.
169106
- Web run persistence is JSON-per-run with no locking (single-process
170107
server); move to sqlite if multi-worker ever happens.
108+
- PR 5's behavior change (|alpha| retrieval + wider injection) has not been
109+
measured with the PR 1 eval harness yet — run a before/after backtest when
110+
spending eval budget next.

tests/test_memory_log.py

Lines changed: 133 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,16 @@
55
import pandas as pd
66
import pytest
77

8+
from tradingagents.agents import (
9+
create_aggressive_debator,
10+
create_bear_researcher,
11+
create_bull_researcher,
12+
create_conservative_debator,
13+
create_neutral_debator,
14+
)
815
from tradingagents.agents.managers.portfolio_manager import create_portfolio_manager
916
from tradingagents.agents.schemas import PortfolioDecision, PortfolioRating
10-
from tradingagents.agents.utils.memory import TradingMemoryLog
17+
from tradingagents.agents.utils.memory import TradingMemoryLog, _parse_pct
1118
from tradingagents.graph.propagation import Propagator
1219
from tradingagents.graph.reflection import Reflector
1320
from tradingagents.graph.trading_graph import TradingAgentsGraph
@@ -36,10 +43,10 @@ def make_log(tmp_path, filename="trading_memory.md"):
3643
return TradingMemoryLog(config)
3744

3845

39-
def _seed_completed(tmp_path, ticker, date, decision_text, reflection_text, filename="trading_memory.md"):
46+
def _seed_completed(tmp_path, ticker, date, decision_text, reflection_text, filename="trading_memory.md", alpha="+0.5%"):
4047
"""Write a completed entry directly to file, bypassing the API."""
4148
entry = (
42-
f"[{date} | {ticker} | Buy | +1.0% | +0.5% | 5d]\n\n"
49+
f"[{date} | {ticker} | Buy | +1.0% | {alpha} | 5d]\n\n"
4350
f"DECISION:\n{decision_text}\n\n"
4451
f"REFLECTION:\n{reflection_text}"
4552
+ _SEP
@@ -263,7 +270,7 @@ def test_get_past_context_cross_ticker(self, tmp_path):
263270
log = make_log(tmp_path)
264271
_seed_completed(tmp_path, "AAPL", "2026-01-05", "Buy AAPL — Services growth.", "Correct.")
265272
ctx = log.get_past_context("NVDA")
266-
assert "Recent cross-ticker lessons" in ctx
273+
assert "Cross-ticker lessons" in ctx
267274
assert "Past analyses of NVDA" not in ctx
268275

269276
def test_n_same_limit_respected(self, tmp_path):
@@ -748,8 +755,8 @@ def test_same_ticker_prioritised(self, tmp_path):
748755
_resolve_entry(log, "AAPL", "2026-01-06", DECISION_SELL, "Overvalued.")
749756
result = log.get_past_context("NVDA")
750757
assert "Past analyses of NVDA" in result
751-
assert "Recent cross-ticker lessons" in result
752-
same_block, cross_block = result.split("Recent cross-ticker lessons")
758+
assert "Cross-ticker lessons" in result
759+
same_block, cross_block = result.split("Cross-ticker lessons")
753760
assert "NVDA" in same_block
754761
assert "AAPL" in cross_block
755762

@@ -869,3 +876,123 @@ def test_full_pipeline_no_regression(self, tmp_path):
869876
assert len(entries) == 1
870877
assert entries[0]["ticker"] == "NVDA"
871878
assert entries[0]["pending"] is True
879+
880+
881+
# ---------------------------------------------------------------------------
882+
# Cross-ticker relevance: |alpha| ordering (PR 5)
883+
# ---------------------------------------------------------------------------
884+
885+
@pytest.mark.unit
886+
class TestCrossTickerRelevance:
887+
888+
def test_parse_pct(self):
889+
assert _parse_pct("+3.2%") == 3.2
890+
assert _parse_pct("-1.5%") == -1.5
891+
assert _parse_pct("n/a") is None
892+
assert _parse_pct(None) is None
893+
assert _parse_pct("") is None
894+
895+
def test_cross_ticker_ordered_by_abs_alpha(self, tmp_path):
896+
log = make_log(tmp_path)
897+
_seed_completed(tmp_path, "AAPL", "2026-01-05", "d", "AAPL lesson.", alpha="+1.0%")
898+
_seed_completed(tmp_path, "MSFT", "2026-01-06", "d", "MSFT lesson.", alpha="-8.0%")
899+
_seed_completed(tmp_path, "TSLA", "2026-01-07", "d", "TSLA lesson.", alpha="+3.0%")
900+
_seed_completed(tmp_path, "AMZN", "2026-01-08", "d", "AMZN lesson.", alpha="+0.2%")
901+
ctx = log.get_past_context("NVDA")
902+
assert "AMZN lesson." not in ctx # smallest |alpha| drops out of top 3
903+
assert (
904+
ctx.index("MSFT lesson.")
905+
< ctx.index("TSLA lesson.")
906+
< ctx.index("AAPL lesson.")
907+
)
908+
909+
def test_tie_break_most_recent_first(self, tmp_path):
910+
log = make_log(tmp_path)
911+
_seed_completed(tmp_path, "AAPL", "2026-01-05", "d", "Older lesson.", alpha="+2.0%")
912+
_seed_completed(tmp_path, "MSFT", "2026-01-06", "d", "Newer lesson.", alpha="-2.0%")
913+
ctx = log.get_past_context("NVDA")
914+
assert ctx.index("Newer lesson.") < ctx.index("Older lesson.")
915+
916+
def test_unparseable_alpha_sorts_last(self, tmp_path):
917+
log = make_log(tmp_path)
918+
_seed_completed(tmp_path, "AAPL", "2026-01-06", "d", "No-alpha lesson.", alpha="n/a")
919+
_seed_completed(tmp_path, "MSFT", "2026-01-05", "d", "Small-alpha lesson.", alpha="+0.1%")
920+
ctx = log.get_past_context("NVDA")
921+
assert ctx.index("Small-alpha lesson.") < ctx.index("No-alpha lesson.")
922+
923+
def test_same_ticker_stays_recency_ordered(self, tmp_path):
924+
log = make_log(tmp_path)
925+
_seed_completed(tmp_path, "NVDA", "2026-01-05", "Old decision.", "r", alpha="+9.0%")
926+
_seed_completed(tmp_path, "NVDA", "2026-01-06", "New decision.", "r", alpha="+0.1%")
927+
ctx = log.get_past_context("NVDA")
928+
assert ctx.index("New decision.") < ctx.index("Old decision.")
929+
930+
931+
# ---------------------------------------------------------------------------
932+
# Past-lessons injection into researchers and risk debators (PR 5)
933+
# ---------------------------------------------------------------------------
934+
935+
def _researcher_state(past_context=""):
936+
return {
937+
"company_of_interest": "NVDA",
938+
"investment_debate_state": {
939+
"history": "", "bull_history": "", "bear_history": "",
940+
"current_response": "", "count": 0,
941+
},
942+
"market_report": "Market report.",
943+
"sentiment_report": "Sentiment report.",
944+
"news_report": "News report.",
945+
"fundamentals_report": "Fundamentals report.",
946+
"past_context": past_context,
947+
}
948+
949+
950+
def _risk_debator_state(past_context=""):
951+
return {
952+
"company_of_interest": "NVDA",
953+
"risk_debate_state": {
954+
"history": "", "aggressive_history": "", "conservative_history": "",
955+
"neutral_history": "", "current_aggressive_response": "",
956+
"current_conservative_response": "", "current_neutral_response": "",
957+
"count": 0,
958+
},
959+
"market_report": "Market report.",
960+
"sentiment_report": "Sentiment report.",
961+
"news_report": "News report.",
962+
"fundamentals_report": "Fundamentals report.",
963+
"trader_investment_plan": "Trader plan.",
964+
"past_context": past_context,
965+
}
966+
967+
968+
_LESSONS_HEADER = "Past lessons from prior decisions and outcomes"
969+
_INJECTION_AGENTS = [
970+
(create_bull_researcher, _researcher_state),
971+
(create_bear_researcher, _researcher_state),
972+
(create_aggressive_debator, _risk_debator_state),
973+
(create_conservative_debator, _risk_debator_state),
974+
(create_neutral_debator, _risk_debator_state),
975+
]
976+
977+
978+
@pytest.mark.unit
979+
class TestPastContextInjection:
980+
981+
@pytest.mark.parametrize(
982+
"factory,make_state", _INJECTION_AGENTS,
983+
ids=lambda x: getattr(x, "__name__", ""),
984+
)
985+
def test_lessons_block_present_iff_past_context(self, factory, make_state):
986+
llm = MagicMock()
987+
llm.invoke.return_value = MagicMock(content="ok")
988+
node = factory(llm)
989+
990+
node(make_state(past_context="[2026-01-05 | AAPL | Buy | +5.0% | +2.0% | 5d]\nGreat call."))
991+
prompt = llm.invoke.call_args[0][0]
992+
assert _LESSONS_HEADER in prompt
993+
assert "Great call." in prompt
994+
995+
llm.invoke.reset_mock()
996+
node(make_state(past_context=""))
997+
prompt = llm.invoke.call_args[0][0]
998+
assert _LESSONS_HEADER not in prompt

tests/test_model_routing.py

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
"""Tests for PR 4: per-stage model routing and the adaptive risk debate."""
2+
3+
from unittest.mock import MagicMock
4+
5+
import pytest
6+
7+
from tradingagents.graph.conditional_logic import ConditionalLogic
8+
from tradingagents.graph.setup import LLM_MAP_STAGES, GraphSetup
9+
10+
# ---------------------------------------------------------------------------
11+
# Per-stage LLM map
12+
# ---------------------------------------------------------------------------
13+
14+
15+
def _make_setup(llm_map=None):
16+
quick, deep = MagicMock(name="quick"), MagicMock(name="deep")
17+
setup = GraphSetup(quick, deep, {}, ConditionalLogic(), llm_map=llm_map)
18+
return setup, quick, deep
19+
20+
21+
@pytest.mark.unit
22+
class TestAgentLlmMap:
23+
def test_empty_map_returns_defaults(self):
24+
setup, quick, deep = _make_setup()
25+
for stage in LLM_MAP_STAGES:
26+
assert setup._llm(stage, quick) is quick
27+
assert setup._llm(stage, deep) is deep
28+
29+
def test_map_overrides_single_stage(self):
30+
setup, quick, deep = _make_setup({"research_manager": "quick"})
31+
assert setup._llm("research_manager", deep) is quick
32+
assert setup._llm("portfolio_manager", deep) is deep
33+
assert setup._llm("analysts", quick) is quick
34+
35+
def test_deep_override(self):
36+
setup, quick, deep = _make_setup({"risk_analysts": "deep"})
37+
assert setup._llm("risk_analysts", quick) is deep
38+
39+
def test_unknown_stage_raises(self):
40+
with pytest.raises(ValueError, match="unknown stage"):
41+
_make_setup({"trader_agent": "quick"})
42+
43+
def test_unknown_value_raises(self):
44+
with pytest.raises(ValueError, match="'quick' or 'deep'"):
45+
_make_setup({"trader": "gpt-5.5"})
46+
47+
48+
# ---------------------------------------------------------------------------
49+
# Adaptive risk debate
50+
# ---------------------------------------------------------------------------
51+
52+
AGREE_PLAN = "**Recommendation**: Buy\n\n**Rationale**: Strong setup."
53+
AGREE_TRADER = "**Action**: Buy\n\nFINAL TRANSACTION PROPOSAL: **BUY**"
54+
DISAGREE_TRADER = "**Action**: Sell\n\nFINAL TRANSACTION PROPOSAL: **SELL**"
55+
56+
57+
def _risk_state(count, investment_plan=AGREE_PLAN, trader_plan=AGREE_TRADER,
58+
latest_speaker="Neutral"):
59+
return {
60+
"risk_debate_state": {"count": count, "latest_speaker": latest_speaker},
61+
"investment_plan": investment_plan,
62+
"trader_investment_plan": trader_plan,
63+
}
64+
65+
66+
@pytest.mark.unit
67+
class TestAdaptiveRiskDebate:
68+
def test_default_zero_stops_at_base_cap_even_on_disagreement(self):
69+
logic = ConditionalLogic(max_risk_discuss_rounds=1)
70+
state = _risk_state(3, trader_plan=DISAGREE_TRADER)
71+
assert logic.should_continue_risk_analysis(state) == "Portfolio Manager"
72+
73+
def test_agreement_stops_at_base_cap_despite_extra_rounds(self):
74+
logic = ConditionalLogic(max_risk_discuss_rounds=1, adaptive_extra_rounds=2)
75+
state = _risk_state(3)
76+
assert logic.should_continue_risk_analysis(state) == "Portfolio Manager"
77+
78+
def test_disagreement_runs_exactly_n_extra_cycles(self):
79+
logic = ConditionalLogic(max_risk_discuss_rounds=1, adaptive_extra_rounds=2)
80+
# Past the base cap (3) but under the hard cap (9): keeps debating.
81+
for count in range(3, 9):
82+
state = _risk_state(count, trader_plan=DISAGREE_TRADER)
83+
assert logic.should_continue_risk_analysis(state) != "Portfolio Manager"
84+
# At the hard cap: stops regardless of disagreement.
85+
state = _risk_state(9, trader_plan=DISAGREE_TRADER)
86+
assert logic.should_continue_risk_analysis(state) == "Portfolio Manager"
87+
88+
def test_speaker_rotation_below_cap_unchanged(self):
89+
logic = ConditionalLogic(max_risk_discuss_rounds=1, adaptive_extra_rounds=2)
90+
cases = {
91+
"Aggressive": "Conservative Analyst",
92+
"Conservative": "Neutral Analyst",
93+
"Neutral": "Aggressive Analyst",
94+
}
95+
for speaker, expected in cases.items():
96+
state = _risk_state(1, latest_speaker=speaker)
97+
assert logic.should_continue_risk_analysis(state) == expected
98+
# Rotation also holds in the adaptive window while disagreeing.
99+
state = _risk_state(4, trader_plan=DISAGREE_TRADER, latest_speaker="Aggressive")
100+
assert logic.should_continue_risk_analysis(state) == "Conservative Analyst"
101+
102+
def test_hold_vs_hold_counts_as_agreement(self):
103+
logic = ConditionalLogic(max_risk_discuss_rounds=1, adaptive_extra_rounds=2)
104+
state = _risk_state(
105+
3,
106+
investment_plan="**Recommendation**: Hold",
107+
trader_plan="**Action**: Hold",
108+
)
109+
assert logic.should_continue_risk_analysis(state) == "Portfolio Manager"

0 commit comments

Comments
 (0)