Skip to content

Add GitHub Copilot SDK benchmark agent - #47

Draft
Lurkrazy wants to merge 6 commits into
NVIDIA-NeMo:mainfrom
Lurkrazy:lurkrazy-pareto-agent-copilot-sdk
Draft

Add GitHub Copilot SDK benchmark agent#47
Lurkrazy wants to merge 6 commits into
NVIDIA-NeMo:mainfrom
Lurkrazy:lurkrazy-pareto-agent-copilot-sdk

Conversation

@Lurkrazy

Copy link
Copy Markdown

What does this PR do?

Adds an additive GitHub Copilot SDK execution path to nooa-bench so evaluations can use a signed-in Copilot account and models such as gpt-5.6-sol without replacing the existing NOOA/LiteLLM benchmark agent.

The new CopilotBenchAgent owns the SDK session directly, enforces structured task results, captures deduplicated usage events, and bounds startup, execution, and teardown. Runner and Harbor dispatch select this path with agent_type=copilot, scope credentials to Copilot runs, keep COPILOT_HOME private and ephemeral, and install the SDK runtime with read-only group/other permissions. Documentation, a minimal Harbor configuration, and lifecycle/contract regression coverage are included.

CopilotBenchAgent intentionally does not subclass nooa.Agent: both NOOA CodeAct and the Copilot SDK are agent runtimes that own their tool loop and session lifecycle. The existing BenchAgent remains unchanged from main.

Validation includes 45 nooa-bench tests on Python 3.12 and 3.13, five-repeat lifecycle and runner/Harbor stress runs, Ruff, Pyright with zero errors, wheel installation/import, permission semantics, and a live signed-in gpt-5.6-sol structured-completion smoke using xhigh and long_context.

Related issues

N/A

Checklist

  • Code follows the project style (Ruff lint passes repo-wide; changed Python files pass ruff format --check)
  • Tests added/updated and passing (uv run pytest)
  • Docs updated if behavior or public APIs changed
  • New source files carry an SPDX license header

Lurkrazy and others added 6 commits July 28, 2026 15:13
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@furgalep furgalep self-assigned this Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants