Commit 4193a00
fix: repair the outcome-grading loop and ground decisions in evidence (#5)
* fix: repair the outcome-grading loop and ground decisions in evidence
Six issues found reviewing the analysis methodology against the 17-entry
decision log in ~/.tradingagents/memory/.
Outcome grading was training the system on noise. `_fetch_returns` graded a
past call with whatever bars happened to exist rather than the full holding
window, so re-running a ticker the next day scored the prior call on one day
of price action, wrote a confident reflection from it, and injected that
reflection into every later analysis of the ticker. The log had entries
tagged `| +5.0% | 1d]` and two more at `4d`. Grading is now all-or-nothing on
the full window, and pending entries resolve across every ticker instead of
only the one being analyzed — a ticker analyzed once previously stayed
pending forever and was never scored. The reflector is told its horizon so it
stops turning a short-window wobble into a lesson.
The judges never saw the primary evidence. The Research Manager received only
the bull/bear transcript, and the Trader only the investment plan — while its
system prompt claimed it had the analyst reports. Any fact neither advocate
chose to cite was lost before the first decision. Both now receive the four
reports via a shared `get_analyst_reports_from_state` helper, with the
Research Manager instructed to treat reports as primary evidence over
advocacy and to surface material facts neither side raised.
The rating scale had collapsed. 17 runs produced 11 Overweight, 4
Underweight, 2 Hold and zero Buy or Sell: the anti-Hold nudge appeared twice
with nothing defining the tier above it. Buy and Sell now carry an explicit
bar, shared through one `RATING_BAR_GUIDANCE` constant so the two stages
cannot drift. `what_would_change_it` makes a rating checkable on re-analysis
instead of re-derived from scratch.
FRED was serving revised data. The observations query passed no realtime
window, so a 2026-02-05 analysis saw January and February CPI — published
weeks later — and November's revised value rather than the one known at the
time. Pinned to the analysis date.
Also adds an optional per-run position note (threaded through the single
`resolve_instrument_context` chokepoint, with anti-anchoring wording and
bounds at the API boundary), batch ticker runs, form defaults that survive a
reload, and a track-record page over the decision log.
Fixes a latent test-isolation bug while here: test_web_backend patched one
`DEFAULT_CONFIG` object while web.backend.runs held another (importlib.reload
in test_ollama_base_url splits them), so those tests globbed the developer's
real ~/.tradingagents/logs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfW2XdaNF8rtqW3Nyp5enk
* fix: address review findings on the outcome-grading PR
Six issues raised by automated review, all reproduced before fixing.
**avg_alpha was rendering 100x too large.** `_parse_pct` returns percent units
("+2.0%" -> 2.0) while `hit_rate` is a fraction, and the frontend rescales
both by 100 — so an average alpha of 0.98% displayed as 98%. Normalized to a
fraction so the two fields share units.
**Concurrent runs could lose log updates or abort.** The dashboard runs three
analyses as threads in one process and each now resolves the full pending set
at start-up, so all three raced on the same read-modify-write through a shared
`.tmp` path: one worker's `replace` moved another's file into place and the
loser raised FileNotFoundError. Adds a reentrant module lock around the
mutating methods and gives each write a pid+thread-unique temp name.
Resolution holds the lock across the whole read/reflect/write cycle, so the
first worker in does the work and the others find nothing pending instead of
paying for the same reflections three times. Verified: without the lock, 10 of
12 concurrent writers fail.
**A failing reflection aborted the whole analysis.** Resolution runs before
the pipeline, so one transient provider error — or one deterministically
un-reflectable old entry — killed every subsequent run and retried the same
entry forever. Now isolated per entry: log it, leave it pending, keep going.
The blast radius grew with cross-ticker resolution, so this matters more than
it did.
**Track-record entries sorted by log position, not analysis date.** Any
non-future date is accepted, so analysing a historical date after a current
one put the older call first while the UI claimed "most recent first".
**Oversized batches became unmonitorable.** Past `MAX_RECENT_RUNS` (50) the
earliest runs drop out of History with no way to cancel them, and the page
had already discarded their IDs. Capped at 25 with an explanatory message.
**Sticky settings could submit a removed model.** After a deploy trims the
model catalog — which this repo does — a persisted value was restored
unvalidated; the `<select>` showed its first option while state held the stale
one, so submission sent a model the runner cannot construct. Stale provider,
model, language and analyst values now snap to current options on load.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfW2XdaNF8rtqW3Nyp5enk
* fix(ci): guard the fastapi import and sort imports
Two CI failures, both mine.
`tests/test_position_context.py` imported `web.backend.api` at module level,
but the test job installs only the core dev extras — no fastapi — so
collection aborted on every Python version. The repo's convention is
`pytest.importorskip("fastapi")` at the top of the file, but that would skip
the graph and runner coverage in this module too, which needs no fastapi. The
import is now lazy inside the API-boundary class, so those four tests skip
while the other seven still run.
Also applies `ruff check --fix` to the combined import in the test_web_backend
fixture.
Verified on a real Python 3.10 with fastapi absent: 668 passed, 8 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfW2XdaNF8rtqW3Nyp5enk
* fix: address second review pass
**A long market closure could strand an entry as pending forever.** Making the
holding-window guard strict left a fixed `holding_days + 10` calendar cutoff
behind it, and that span can hold fewer than the required sessions. Verified
against live data: a 2025-01-27 Shanghai call spans Lunar New Year and returns
exactly 5 bars where 6 are needed, so the window never grows and every retry
reads the same incomplete slice — the outcome was available since 2025-02-11.
Now queries through today and indexes the Nth bar, which is correct at any
closure length. Reading past the trade date is fine here: this scores an
already-made decision and feeds no analyst. A genuinely too-recent entry still
defers.
**Buy and Sell had no valid answer for the change triggers.** Asking for the
development that would move the rating "one tier more bullish" is unanswerable
at the top of the scale, inviting an invented tier like Strong Buy. Endpoints
now get explicit handling: the adjacent move that exists, plus what would
confirm holding the endpoint rating.
**Track-record rows could share a React key.** Date+rating is not unique — once
an outcome resolves, re-analysing that date appends a second entry that can
match both — so a refetch flipping one row pending -> scored could reconcile
the wrong row. Added the occurrence index.
**The track-record page rendered nothing on a failed request.** It read neither
`error` nor `isError`, so once the skeleton cleared, a failure looked like
missing UI. Now shows the same EmptyState treatment the report page uses.
Verified: ruff clean, 682 passed on 3.14 and 669 on a real 3.10 with fastapi
absent (CI's condition).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UfW2XdaNF8rtqW3Nyp5enk
* fix: address third review pass
**Average alpha ignored the call's direction.** A correct bearish call has
negative alpha, so the unsigned average subtracted it while the hit rate counted
it as a win — the page could report "100% hit rate, -2% avg alpha" for a single
correct Underweight. Verified on the real log: the 2026-06-23 PLTR Underweight
(-1.8% alpha, a hit) was dragging the average down, showing +0.98% where
captured alpha is +1.70%. Now averages direction * alpha, which is the same
definition backtest.summarize already uses for mean_alpha; a test pins the two
together so they cannot drift. Relabelled in the UI as "Avg captured alpha".
**Cancellation could not stop the backlog.** Outcome resolution runs before
_stream, which is where cancel_event is checked, and it now walks every pending
ticker at a price fetch plus an LLM call each while holding LOG_LOCK — so
cancelling a run kept spending. Takes an optional should_stop predicate, checked
between entries; the runner passes cancel_event.is_set. Entries already resolved
are still written rather than discarded.
**Persisted research depth was not reconciled.** The catalog check added last
pass covered provider, models, language and analysts but not depth, so a stale
or hand-edited value (e.g. 2) stayed in state and 422'd at
StartRunRequest._v_depth. Now validated against opts.research_depths.
Verified: ruff clean, 686 passed on 3.14, 673 on py3.10 with fastapi absent.
* fix: skip outcome fetches that cannot have matured
A same-day batch made outcome resolution quadratic. Every finished run appends a
pending entry, and resolution now walks all tickers, so run N probed all N-1
earlier entries even though five trading sessions plainly could not have elapsed.
At the 25-ticker batch cap that is ~600 yfinance requests guaranteed to return
None — and because the entries stay pending, the whole sweep repeated on every
run for the following week, all serialized under LOG_LOCK and delaying every
queued analysis. The batch feature added earlier in this PR is what made it bite.
Adds `_window_could_have_closed`: pure calendar arithmetic, no network. Trading
sessions can never outnumber calendar days, so a False means the outcome
definitely is not available yet, while True only means it might be — the bar
count in `_fetch_returns` remains the authority. Conservative in the safe
direction, so it never skips a resolvable entry, and an unparseable date fails
open rather than dropping a malformed tag forever.
The holding period moves to a `HOLDING_DAYS` class constant shared by the
pre-check and the fetch default, so the two cannot drift.
Confirmed against the real log: all 12 currently pending entries remain eligible;
only genuinely immature ones are suppressed.
Verified: ruff clean, 688 passed on 3.14, 675 on py3.10 with fastapi absent.
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>1 parent 0ed1d4e commit 4193a00
29 files changed
Lines changed: 1458 additions & 143 deletions
File tree
- tests
- tradingagents
- agents
- managers
- trader
- utils
- dataflows
- graph
- web
- backend
- frontend
- src
- routes
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
153 | 169 | | |
154 | 170 | | |
155 | 171 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
432 | 432 | | |
433 | 433 | | |
434 | 434 | | |
435 | | - | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
436 | 440 | | |
437 | 441 | | |
438 | 442 | | |
439 | | - | |
| 443 | + | |
440 | 444 | | |
441 | | - | |
442 | 445 | | |
443 | 446 | | |
444 | 447 | | |
445 | 448 | | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
446 | 486 | | |
447 | 487 | | |
448 | 488 | | |
| |||
529 | 569 | | |
530 | 570 | | |
531 | 571 | | |
532 | | - | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
533 | 577 | | |
534 | 578 | | |
535 | 579 | | |
| |||
540 | 584 | | |
541 | 585 | | |
542 | 586 | | |
543 | | - | |
544 | | - | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
545 | 638 | | |
546 | 639 | | |
547 | 640 | | |
| |||
642 | 735 | | |
643 | 736 | | |
644 | 737 | | |
645 | | - | |
646 | | - | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
647 | 744 | | |
648 | 745 | | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
649 | 749 | | |
650 | 750 | | |
| 751 | + | |
| 752 | + | |
651 | 753 | | |
652 | | - | |
653 | | - | |
654 | | - | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
655 | 898 | | |
656 | 899 | | |
657 | 900 | | |
| |||
662 | 905 | | |
663 | 906 | | |
664 | 907 | | |
| 908 | + | |
665 | 909 | | |
666 | | - | |
| 910 | + | |
667 | 911 | | |
668 | 912 | | |
669 | 913 | | |
| |||
0 commit comments