Session and push cursor are global, so logging into a second stage silently and permanently skips records in the first

[khaliqgant]

Filed at devhistory's request, from a real incident on 2026-08-06 that cost prod data. They'll add their timeline and repair procedure so this becomes a runbook rather than a warning.

Summary

ai-hist login stores one session and one cursor per machine, with no per-stage separation. The background push/sync services take no --base-url and follow whatever session was stored last. So a single ai-hist login --base-url <other-stage> on a machine running the push service:

1. silently redirects that machine's pushes to the other stage, and
2. permanently skips those records in the original stage, because the cursor is shared.

(2) is the load-bearing half and was not obvious to anyone during the incident — including me. It does not self-heal.

Mechanism

All three pieces of state live in one directory with no stage in the key (crates/ai-hist/src/cloud.rs:42-78):

pub fn config_dir() -> PathBuf {
    if let Some(dir) = std::env::var_os("RELAYHISTORY_HOME") { return PathBuf::from(dir); }
    ...home.join(".agentworkforce/relayhistory")
}
fn auth_path()    -> PathBuf { config_dir().join("auth.json") }   // session — which stage
fn cursor_path()  -> PathBuf { config_dir().join("cursor.json") } // watermark — what's been shipped
fn machine_path() -> PathBuf { config_dir().join("machine-id") }

push advances cursor.json only after the server accepts the batch — correct as a durable outbox, but the cursor is not scoped to the stage that accepted it. Records shipped to stage B advance the watermark that stage A reads, so stage A never sees them again.

Real incident

A dev login at 07:53:47Z redirected a machine's 300s push service. 525 events went to dev instead of prod over ~3.5 minutes. Sampling 12 of the diverted event_ids against prod found 7 genuinely missing; the 5 present were from an unrelated later push.

Every push returned accepted: N and looked healthy throughout. Nothing surfaced the prod gap — it was found only by going to look at the storage layer for an unrelated reason.

Repair required manually rewinding cursor.json and re-pushing (ingest is idempotent on event_id, so re-sending is a no-op). Verified afterward across all 525: 525 present_in_prod, 0 still_missing.

Why the obvious mitigation misses

Operators reach for ~/.config/ai-hist/auth.json — it is what #17 documented. The Rust CLI never writes that path. On the incident machine, ~/.agentworkforce/relayhistory/auth.json was modified the same day; ~/.config/ai-hist/auth.json had been untouched since Jun 30. Backing up the latter is a no-op that reads as a successful precaution.

What a fix needs

Both halves, or it doesn't hold:

• Session — key auth.json by stage (base_url), so logging into dev doesn't evict the prod session.
• Cursor — key cursor.json by stage too. A stage-scoped session alone still corrupts the watermark, which is the half that loses data.

Alternatively, have the installed service pin the stage it was installed against and pass it explicitly, so a foreground login cannot redirect a background service at all. That is arguably the better shape: it fixes the surprise at the source rather than making the surprise survivable.

machine-id should probably stay shared — the same physical machine across stages is correct.

Note RELAYHISTORY_HOME already relocates all three together, so per-stage state is achievable today by an external caller that sets it consistently. That is a workaround for sandbox/CI callers, not a fix — it must be applied to every invocation (login, sync, push), because config_dir() is resolved independently at each call site. Partial application strands the session where later pushes can't find it, which fails in exactly the same quiet way.

Guard worth adding regardless

push could refuse to run — or at minimum warn loudly — when the stored session's base_url differs from the one the cursor was last advanced against. That turns a silent data-skip into a visible error, independent of how the storage layout is ultimately fixed.

Related

• AgentWorkforce/relayhistory-cloud#17 — GET /v1/machines / ai-hist coverage makes an unexpected machine visible in a stage, but cannot see the resulting gap in the other stage. See the note below.
• Cloud calls send the rth_at_ bearer to the stored base_url without a scheme check #56 — separate: cloud calls send the bearer to the stored base_url with no scheme check.

On detecting this from coverage data

It was suggested that coverage could flag "a cursor that has run ahead of the data". That does not work as stated, and the reason is worth recording so nobody builds it and trusts it:

build_outbox_batch advances the cursor to the max id scanned, not emitted (crates/ai-hist-core/src/outbox.rs:44-47) — incognito-suppressed rows and skipped/empty rows advance it by design, precisely so they are not re-scanned. A cursor ahead of delivered records is therefore the normal state, not a bug signature.

Worse, the cursor is in client-local coordinates (history.id, trajectories.rowid) and the server stores no column holding them per event, so the server cannot join a cursor position to ingested rows at all.

The signal that is computable server-side is a cursor jump between consecutive sync_batches rows that is much larger than the records delivered in that batch — which is what a diversion looks like from the abandoned stage. But incognito produces the same shape legitimately, so it is a heuristic needing a threshold and a way to distinguish intent, not a detector. Worth doing deliberately if at all; it is not a small addition to the coverage endpoint.