Skip to content

Add OrcaReplay to Monitoring & Analytics - #263

Open
xizhuomengcontin wants to merge 1 commit into
RoggeOhta:mainfrom
xizhuomengcontin:orcareplay-monitoring
Open

xizhuomengcontin wants to merge 1 commit into
RoggeOhta:mainfrom
xizhuomengcontin:orcareplay-monitoring

Conversation

@xizhuomengcontin

Copy link
Copy Markdown

Adds OrcaReplay to Monitoring & Analytics, next to watchtower — which is the closest
neighbour in that section, and the reason I want to be precise about what this adds.

watchtower inspects the API traffic between an agent and its provider. OrcaReplay records that
traffic and then serves it back: run the same Codex command again and it reaches the same
result with no model called and no key needed.

$ orca record codex -- codex exec "fix the failing test"
info recorded run=run_8f21c3 events=41 exit=0

$ orca replay last
info replaying exchanges=6 egress=blocked
info replay.done reused=6/6 exact=6 divergences=0 exit=0

Why this is Codex-specific and not a generic LLM tool

Your rule says general AI/LLM tools don't belong without specific Codex integration, so here is
the Codex-specific part rather than a claim that it "supports Codex":

  • It reads your config.toml. The adapter resolves the top-level model_provider key and
    redirects model_providers.<name>.base_url for the launched process only — so a Codex pointed
    at a non-OpenAI provider is captured at the origin it actually uses, not the default one. It
    honours CODEX_HOME.
  • It reads Codex's own rollout. Sessions under $CODEX_HOME/sessions are parsed as JSONL, and
    the harness preamble (skills, environment context) is separated from what you actually typed, so
    the timeline shows your turns rather than Codex's scaffolding.
  • Codex is one of the adapters in the project's integration test suite: it records against a stub
    origin, kills the origin, replays, and asserts the exchange counts.

One limit, stated up front

egress=blocked means model-provider egress, not network isolation. Replay serves the model's
answers from the trace, but the recorded tool calls still execute for real — a shell command that
ran curl runs it again. Replay is not a sandbox; the entries in your Docker & Sandboxing section
are the tools for that.

Apache-2.0. Happy to move it to Cross-Agent Tools instead if you think it sits better there — the
same recorder handles Claude Code, goose, opencode and others, so either section is defensible;
Monitoring & Analytics seemed closer to what a reader would be looking for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant