Skip to content

Workflow agent bypasses polyresearch CLI, calling gh directly to create PRs and skip attempt recording #173

Description

@alanzabihi

Context

Branch: hybrid-agent-cli
Wave: 1 (critical, blocks protocol integrity)
Found by: polytest run 20260423-133619, hit on 5 repos

Bug

The contribute workflow agent bypasses the polyresearch CLI commands and uses gh directly to create PRs, push branches, and interact with GitHub. It skips polyresearch attempt entirely, meaning no experiment result is recorded in the protocol trail. It skips polyresearch submit, creating PRs manually with gh pr create.

This breaks the entire protocol chain: no attempt comment on the issue, no metric in the ledger, the lead cannot decide the PR (no attempt to evaluate), and results.tsv is never updated.

Repos affected

  • DOMPurify — agent created PR without polyresearch submit, bypassed attempt
  • litellm — agent bypassed both polyresearch attempt and polyresearch submit, created PR with gh pr create
  • remarkable — agent skipped polyresearch attempt, no experiment data recorded
  • markdown-it — no attempt recorded, results.tsv empty
  • tokenizers — released thesis without recording attempt

Evidence (litellm)

The lead could not decide PR #6 because no attempt was recorded:

Lead cannot decide PR #6: no improved attempt recorded for the thesis branch.

The PR was created manually by the agent with gh pr create instead of polyresearch submit.

Evidence (DOMPurify)

PR created and merged, but results.tsv shows "improved" for an attempt that was later policy-rejected. The attempt was recorded inconsistently because the agent mixed CLI and direct gh calls.

Evidence (remarkable)

Contribute ran for full cycle, released thesis, but no polyresearch attempt comment exists on the issue. The experiment ran successfully but the result was never recorded in the protocol.

Root cause

The contribute workflow prompt (contribute-workflow.md) instructs the agent to use CLI commands, but the agent has full shell access and sometimes decides to use gh directly — either because it's more familiar with gh, or because it encounters an error with the CLI command and falls back to gh.

Possible fixes

  • The experiment prompt should be more forceful about requiring CLI commands
  • The polyresearch submit command could be a hard gate: refuse to create a PR unless a valid attempt exists
  • Consider restricting the agent's tool access to only allow polyresearch commands (not raw gh)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingwave-1Parallel batch 1: no dependencies

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions