Investigate: the Advisor tool (advisor_20260301)

[cryptiklemur]

Summary

Anthropic shipped a beta Advisor tool that lets Claude consult a secondary
model mid-conversation. The advisor runs its own inference with a model you
specify in the tool definition, and returns its analysis to the primary model.

I'd like to understand how this behaves through better-ccflare before relying on
it, since it doesn't fit the normal "one request in, one response out" shape the
strategies assume.

Tool definition / beta header for reference:

{ "type": "advisor_20260301", "name": "advisor", "model": "claude-sonnet-4-6" }

Beta header: advisor-tool-2026-03-01 (SDK sets it automatically on
client.beta.messages.create()).

Why it matters for a proxy

The advisor appears to be a server-side tool — its secondary call likely
runs inside Anthropic's request loop rather than coming back out as a separate
HTTP request through the proxy. If that's true, the proxy sees a single inbound
request, but the response carries additional token usage and potentially a
different model than the one requested. That has implications for account
selection, usage accounting, and analytics.

Questions to investigate

• Does the advisor's secondary inference ever surface as separate traffic
  through the proxy, or is it entirely server-side (invisible to ccflare)?
• Does the advisor-tool-2026-03-01 beta header pass through cleanly, or is
  anything stripping/normalizing beta headers?
• Usage accounting: does ccflare's response/usage parsing capture the
  advisor's token usage, or does it undercount? Where does advisor usage
  show up in the response usage block?
• Account selection: does an advisor request interact oddly with
  session / leastUsed / session-affinity (e.g. does the advisor's
  model differing from the parent request confuse any model-aware logic)?
• Does anything in the proxy need to change to support it, or does it "just
  work" because it's one request? (Confirm, don't assume.)

Out of scope

Not proposing a new routing strategy — this is about correctly handling requests
that use the Advisor tool under the existing strategies.

[github-actions]

🤖 Issue Triage

Severity: medium

Analysis:
This is a high-quality investigation task regarding Anthropic's new 'Advisor' tool (advisor_20260301). The author has correctly identified that this server-side tool breaks the standard 'one request in, one response out' assumption typical of proxy load balancing. The core concern is whether the secondary inference occurs transparently within Anthropic's infrastructure (invisible to the proxy) or generates distinct traffic. This has direct implications for token accounting accuracy, model-aware routing strategies (session/leastUsed), and header passthrough. The issue is well-scoped, excluding new routing strategies to focus on correctness.

---

Thanks for this detailed analysis, @cryptiklemur. You've raised an excellent point about the architectural implications of server-side tools like Advisor on proxy accounting and routing logic.

This is a valuable investigation task. To help us verify the behavior, could you please provide:

1. better-ccflare version (e.g., v2.1.0, or commit hash if built from source)
2. Installation method (npm, bun, pre-compiled binary, Docker, etc.)
3. Operating system and architecture

Once we have this, we can prioritize testing the specific questions you've outlined—particularly around token usage parsing and beta header passthrough. If you're able to perform any of these tests yourself (e.g., checking logs for the advisor-tool-2026-03-01 header or comparing usage blocks), please share your findings here!

---

This automated triage was performed by the better-ccflare Issue Triage Agent using glm-4.7.

[tombii]

Investigation results

Traced the actual proxy code for each of your questions. Summary: the proxy already handles the advisor tool transparently for headers/routing/streaming; there's one open, unconfirmed question about usage accounting.

Beta header passthrough — works, no changes needed.
prepareHeaders() (packages/providers/src/providers/anthropic/provider.ts:266-305) copies all incoming client headers via new Headers(headers), and only specifically touches authorization/x-api-key and anthropic-beta (additive merge — it appends/dedupes values, never drops unrecognized ones) plus deletes host. There's no allowlist/denylist of beta header values anywhere. advisor-tool-2026-03-01 passes straight through untouched. (Note: non-Anthropic providers like vertex-ai and codex do strip anthropic-beta entirely, but that's expected — those providers don't get the Anthropic beta surface at all — and irrelevant to the real claude account path.)

Server-side visibility — confirmed server-side, single request/response.
Nothing in the codebase treats the advisor's secondary inference as separate outbound traffic, which matches your assumption: from the proxy's perspective this is one inbound request and one outbound response, same as any other tool call. There's no code path where the proxy would need to (or does) fan this out into two upstream calls.

Account selection / model-aware routing — unaffected.
least-used.ts and session-affinity.ts (packages/load-balancer/src/strategies/) don't reference model at all — grepped, zero hits. Selection is priority/session based only, keyed on the account, not on the request or response model field. Every .model read in the proxy is either from the request body (used for things like cache_control stripping, agent interception) or from the response (used only for analytics display in usage-collector.ts, never routing). So if the advisor's internal model swap surfaces in usage.model/message.model, at most it could mislabel that request's logged model in analytics — it can't confuse routing, since routing never looks at response model.

Usage accounting — mostly works, one open gap.
extractUsageFromJson and the message_start/message_delta handlers in packages/proxy/src/usage-collector.ts (~lines 212-320) read exactly 4 named sub-fields off the usage object: input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens. There's no generic pass-through of the whole usage object.

• If Anthropic folds the advisor's secondary-inference cost into those same 4 fields (likely, since it's one response) → captured correctly, no changes needed.
• If Anthropic introduces a new, distinctly-named usage sub-field for advisor cost (e.g. something advisor-specific, analogous to how some APIs break out server_tool_use cost separately) → it would be silently dropped from stats/cost calculation today.

This is the one item I can't confirm without seeing an actual advisor response body — none of us have beta access to test against right now (and per repo policy we don't script traffic against real Anthropic accounts to find out). If you or anyone has a real usage block from an advisor-tool response, please paste it here (redact anything sensitive) — that would let us confirm whether a code change is actually needed.

Server-side tool precedent — none exists.
Checked for existing special-casing of other Anthropic server-side tools (web_search, code_execution, computer_use, server_tool_use) — there isn't any in the proxy/provider/usage code. So advisor doesn't need to "match" a pattern; the current design already treats all server-side tool usage as opaque and folded into the top-level usage block by default.

SSE stream to the client — untouched.
The usage-collector only extracts stats from the SSE stream for internal accounting; it doesn't rewrite or filter the bytes sent to the client. content_block_start/content_block_delta handling only does timestamp bookkeeping and doesn't branch on block subtype, so unknown block types (e.g. an advisor-related block) pass through unaffected. The client always sees Anthropic's full original event stream.

Bottom line

No code changes needed for headers, routing, or streaming passthrough — advisor should "just work" today. The only unresolved question is whether Anthropic bills advisor inference through a new usage sub-field we don't currently parse. Happy to add generic/forward-compatible usage field handling if someone can share a sample response confirming that's needed.

[cryptiklemur]

Interesting. Looks like they may have made it more compatible. THe last time i tried it (a month or so ago), it just "didn't work".

I'll test more with it today.