Document the tally, steering, budgets, and snapshot semantics

- README: tally line in the build stages, statistical-rules-through-
  the-tally framing, add_budget/fail-fast note, atlas_message in the
  unattended section
- getting started: res$tally in the results anatomy; snapshot refresh
- sessions: results-are-snapshots section; complete run-directory table
  (test.rds, tally.csv, turns archives, test_leaderboard.csv,
  message.txt)
- pkgdown: atlas_message in the reference index; site rebuilt
- NEWS: add_budget and fail-fast tell entries

Author: mattyoreilly

NEWS.md

@@ -1,5 +1,10 @@
 # atlas (development version)
 
+* New `$add_budget(steps, seconds)`: explicitly extend an exhausted
+  session's mechanical budget; the extension persists for resumes.
+* `$tell()` on an out-of-budget session now fails fast in R - nothing is
+  sent to the LLM - with the remedy in the error message.
+
 * Fixed: a warning inside an agent code chunk (e.g. `glm.fit: fitted
   probabilities numerically 0 or 1 occurred`) aborted the rest of the
   chunk, so later statements - including the `atlas_models` assignment -

README.Rmd

@@ -83,6 +83,14 @@ Every run walks the same six stages, and you hold the pen at stage 2:
    constraints. Approval is a conversation: whatever you type is folded
    into the plan.
 3. **Build** - up to `n_models` candidates, compared on held-out data.
+   Every model and tweak is scored the moment it is evaluated: atlas keeps
+   a live tally, compares against the best so far, and answers with a
+   mechanical verdict the agent must obey - if it doesn't improve, it goes.
+
+   ```
+   [tally #7 | gbm_depth3: rmse = 2.412 | best: glm_gamma = 2.380 | flat: 2/8 -> DISCARD]
+   ```
+
 4. **Refine** - the winner's features are iterated one change at a time
    until the stopping rules call convergence.
 5. **Verify** - machine-checked constraints run against every final model;
@@ -91,12 +99,16 @@ Every run walks the same six stages, and you hold the pen at stage 2:
    plots, all in the run directory.
 
 Two kinds of dial control how long this takes. The *statistical* stopping
-rules (H2O-style names) are applied by the agent: `stopping_rounds`
-consecutive attempts without improvement end an iteration, and gains below
-`stopping_tolerance` don't count. The *mechanical* budgets are enforced in
-code and cannot be talked past: after `max_steps` code executions or
-`max_runtime` seconds, the execution tool refuses to run anything more and
-the agent must finalise with what it has.
+rules (H2O-style names) run through the tally: `stopping_rounds`
+consecutive attempts without improvement end an iteration, gains below
+`stopping_tolerance` don't count, and atlas does the counting - the tally
+verdict tells the agent, in so many words, when the rule has triggered.
+The *mechanical* budgets are enforced in code and cannot be talked past:
+after `max_steps` code executions or `max_runtime` seconds, the execution
+tool refuses to run anything more and the agent must finalise with what it
+has. A finished session that is out of budget refuses further `$tell()`
+calls before any tokens are spent - grant more explicitly with
+`res$session$add_budget(steps = 25)`.
 
 ## Constraints
 
@@ -136,6 +148,16 @@ res <- atlas(claims, "severity",
              test_prop = 0.2)                     # the ungameable judge
 
 res$test_leaderboard   # held-out performance, best first
+res$tally              # every attempt: KEEP / DISCARD, best-so-far
+```
+
+Autonomous doesn't mean unreachable: steer a running build from any other
+R session or terminal, and the message reaches the agent at its next step
+as its highest-priority instruction:
+
+```{r}
+atlas_message("~/atlas-runs/severity",
+              "focus on the gamma GLM family; stop trying trees")
 ```
 
 Long runs stay affordable: past a token budget (`compact_at`) the

README.md

@@ -73,26 +73,40 @@ Every run walks the same six stages, and you hold the pen at stage 2:
 
 1.  **Explore** - dimensions, types, missingness, the outcome’s
     distribution; automatic leakage screening.
+
 2.  **Plan, then stop for you** - candidate families matched to the
     outcome’s distribution, a validation scheme, suggested monotone
     constraints. Approval is a conversation: whatever you type is folded
     into the plan.
+
 3.  **Build** - up to `n_models` candidates, compared on held-out data.
+    Every model and tweak is scored the moment it is evaluated: atlas
+    keeps a live tally, compares against the best so far, and answers
+    with a mechanical verdict the agent must obey - if it doesn’t
+    improve, it goes.
+
+        [tally #7 | gbm_depth3: rmse = 2.412 | best: glm_gamma = 2.380 | flat: 2/8 -> DISCARD]
+
 4.  **Refine** - the winner’s features are iterated one change at a time
     until the stopping rules call convergence.
+
 5.  **Verify** - machine-checked constraints run against every final
     model; violations trigger repair rounds.
+
 6.  **Document** - report, leaderboard, reproducible script, validation
     plots, all in the run directory.
 
 Two kinds of dial control how long this takes. The *statistical*
-stopping rules (H2O-style names) are applied by the agent:
+stopping rules (H2O-style names) run through the tally:
 `stopping_rounds` consecutive attempts without improvement end an
-iteration, and gains below `stopping_tolerance` don’t count. The
-*mechanical* budgets are enforced in code and cannot be talked past:
-after `max_steps` code executions or `max_runtime` seconds, the
-execution tool refuses to run anything more and the agent must finalise
-with what it has.
+iteration, gains below `stopping_tolerance` don’t count, and atlas does
+the counting - the tally verdict tells the agent, in so many words, when
+the rule has triggered. The *mechanical* budgets are enforced in code
+and cannot be talked past: after `max_steps` code executions or
+`max_runtime` seconds, the execution tool refuses to run anything more
+and the agent must finalise with what it has. A finished session that is
+out of budget refuses further `$tell()` calls before any tokens are
+spent - grant more explicitly with `res$session$add_budget(steps = 25)`.
 
 ## Constraints
 
@@ -133,6 +147,16 @@ res <- atlas(claims, "severity",
              test_prop = 0.2)                     # the ungameable judge
 
 res$test_leaderboard   # held-out performance, best first
+res$tally              # every attempt: KEEP / DISCARD, best-so-far
+```
+
+Autonomous doesn’t mean unreachable: steer a running build from any
+other R session or terminal, and the message reaches the agent at its
+next step as its highest-priority instruction:
+
+``` r
+atlas_message("~/atlas-runs/severity",
+              "focus on the gamma GLM family; stop trying trees")
 ```
 
 Long runs stay affordable: past a token budget (`compact_at`) the

_pkgdown.yml

@@ -35,6 +35,7 @@ reference:
   - atlas
   - atlas_session
   - atlas_resume
+  - atlas_message
 
 - title: Domain knowledge
   desc: >

docs/articles/atlas.html

@@ -118,16 +118,22 @@ agent must finalise from what it has.
 
 ``` r
 
-res                    # print: leaderboard, constraint status, report
+res                    # print: leaderboards, tally, constraints, report, cost
 res$models             # named list of fitted models
 predict(res$models[[1]], head(mtcars))
 res$leaderboard        # validation metric per model, best first
+res$tally              # every attempt the agent made: KEEP / DISCARD
 res$report             # markdown: how each model was built, and why
 res$constraints        # compliance table (if constraints were set)
 cat(res$code, sep = "\n\n")   # the full script the agent ran
 res$dir                # the run directory holding all of the above
 ```
 
+`res$tally` is the run’s ledger: one row per model or tweak the agent
+evaluated, scored live by atlas against the best so far and answered
+with a mechanical KEEP or DISCARD verdict - changes that didn’t improve
+on current performance were reverted on the spot.
+
 One element deserves special mention: `res$session` is the live session
 object, with the full conversation still in context. Anything you would
 ask a colleague who just built these models, you can ask it:
@@ -136,6 +142,8 @@ ask a colleague who just built these models, you can ask it:
 
 res$session$tell("why did the refined model beat the original?")
 res$session$tell("build one more candidate that uses at most 3 predictors")
+
+res <- res$session$results()   # results are snapshots: refresh after changes
 ```
 
 Sessions also survive R itself - see