Support benchmarking ADK-built agents

[jessie1111101]

Goal

Make it possible to benchmark agents built with the Agent Development Kit, so ADK teams can run their agents against devops-bench tasks and get a scored trajectory back. Pathfinder is the first consumer.

This issue tracks the work end to end.

In flight

• feat(agents): add an ADK agent harness #137 — feat(agents): add an ADK agent harness. Registers --agent-type adk. Imports the agent named by AGENT_TARGET, deep-copies it, applies MCP/skills/rules, drives it through ADK's Runner, and folds the event stream into a trajectory.
• fix(deps): cap mcp below 2.0 #136 — fix(deps): cap mcp below 2.0. One-line dependency guard. Split out because it touches the api harness, not just ADK.

Validated so far

On a real cluster, using throwaway probe agents so any failure was unambiguously the harness's fault:

• Single agent — tasks/common/opa-remediation on kind with a live gke-mcp toolset. 30 tool calls, VerificationCoverage: 1.0, OutcomeScore: 0.8165.

• Multi-agent tree — a three-node delegating tree (toolless coordinator → cluster operator → reporter). This found a real bug: the harness was giving the MCP toolset and the rules brief to the root agent only, so the sub-agent doing the cluster work had no tools at all. ADK resolves tools from whichever agent is active, with no inheritance from a parent.

  The broken run reported status: success with errors: [] and scored 0.0 — a harness fault looked exactly like an agent failing the task. Fixed in feat(agents): add an ADK agent harness #137; capabilities now go to every agent in the tree. Confirmed by re-running: the sub-agent went from 0 tools to holding the toolset and making 6 real MCP calls.

  Also confirmed working: delegation, callbacks, and the workspace directory holding across a two-hop delegation.

• Scoring parity — the same tree then ran opa-remediation start to finish and scored exactly what the single agent scored: OutcomeScore 0.8165, VerificationCoverage 1.0, status: success. So a tree is graded like a single agent, with no harness penalty. Run used a quarantined HOME (prior runs of this task family had left repos and a report in the real home, where this task provisions its own repo), and all 26 shell commands were audited afterwards — all legitimate task work, no rubric hunting.

Still to validate

• A real ADK agent, not a probe. Everything above used purpose-built probe agents, so any failure was unambiguously the harness's fault. This is the gate that decides which follow-ups below are real and which are theoretical.

Blocking question for the ADK team: is your agent importable, or does it only exist as a deployed A2A service? If it's deployment-only, the A2A item jumps the queue and everything else waits.

Follow-ups

Each is small and independent; none blocks #137.

• Sub-agent attribution in the trajectory. Capabilities now reach every agent, but the trajectory still records tool calls without which agent made each one. Invisible for a single agent; matters for a tree.
• Support non-Gemini models. ADK's LiteLlm wrapper needs google-adk[extensions], which the adk extra does not install, so Anthropic/OpenAI-backed ADK agents can't run today. Verified that LiteLlm imports and constructs fine with that extra, so this is a packaging decision, not a design problem — the open question is whether the extra lockfile churn is worth it, which is a question for whoever needs it.
• Wire AGENT_MAX_TURNS to ADK's RunConfig.max_llm_calls. Today it's api-harness-only, so the effective cap is ADK's default of 500 rather than the knob the rest of the bench uses.
• Map AGENT_PROVIDER / AGENT_API_KEY onto ADK's GOOGLE_* env. ADK resolves credentials itself, so these are not consumed and Vertex setup is a documented manual step.
• Token accounting for non-Gemini models. Assumes genai usage field names.
• Persist harness metadata into results.json. The harness records what capabilities landed (mcp_agents, instruction_agents, model_override_count) on AgentResult.metadata, but the orchestrator's result record drops it, so a finished run can't be audited from its artifact. Pre-existing and affects every harness, not just ADK.

Out of scope (for now)

• Evaluating a deployed A2A agent. A2A can issue a prompt but exposes no trajectory, which is most of what the benchmark scores. That's a separate harness with different scoring semantics, not an extension of feat(agents): add an ADK agent harness #137. Worth its own issue if the need is real.

Docs

docs/components/agents.md covers harness selection, AGENT_TARGET spellings, deep-copy semantics, multi-agent tree behaviour, the workspace/cwd behaviour, and the Vertex env recipe.

[jessie1111101]

/assign