Add OrcaReplay to Monitoring & Analytics - #263
Open
xizhuomengcontin wants to merge 1 commit into
Open
xizhuomengcontin wants to merge 1 commit into
xizhuomengcontin wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds OrcaReplay to Monitoring & Analytics, next to
watchtower— which is the closestneighbour in that section, and the reason I want to be precise about what this adds.
watchtowerinspects the API traffic between an agent and its provider. OrcaReplay records thattraffic and then serves it back: run the same Codex command again and it reaches the same
result with no model called and no key needed.
Why this is Codex-specific and not a generic LLM tool
Your rule says general AI/LLM tools don't belong without specific Codex integration, so here is
the Codex-specific part rather than a claim that it "supports Codex":
config.toml. The adapter resolves the top-levelmodel_providerkey andredirects
model_providers.<name>.base_urlfor the launched process only — so a Codex pointedat a non-OpenAI provider is captured at the origin it actually uses, not the default one. It
honours
CODEX_HOME.$CODEX_HOME/sessionsare parsed as JSONL, andthe harness preamble (skills, environment context) is separated from what you actually typed, so
the timeline shows your turns rather than Codex's scaffolding.
origin, kills the origin, replays, and asserts the exchange counts.
One limit, stated up front
egress=blockedmeans model-provider egress, not network isolation. Replay serves the model'sanswers from the trace, but the recorded tool calls still execute for real — a shell command that
ran
curlruns it again. Replay is not a sandbox; the entries in your Docker & Sandboxing sectionare the tools for that.
Apache-2.0. Happy to move it to Cross-Agent Tools instead if you think it sits better there — the
same recorder handles Claude Code, goose, opencode and others, so either section is defensible;
Monitoring & Analytics seemed closer to what a reader would be looking for.