Document --message + --auto-test for headless CI/benchmark usage #4923

Summary

When using aider in headless/CI mode (--message + --yes-always), code changes are applied in a single pass without verification. This means compilation errors and test failures in the generated code go uncorrected.

Adding --auto-test --test-cmd "go build ./... && go test ./..." should enable aider to iterate on failures, but this combination isn't documented or showcased for headless workflows.

Context

We benchmarked aider against 10+ other AI coding CLIs (codex, claude, copilot, goose, etc.) on the same 10-bug Go project. Aider consistently produced reasonable first-pass fixes but shipped with compilation or test errors — grading C vs A for agents with built-in verify loops.

Our aider command:

aider --file internal/task/task.go --file internal/store/store.go --file internal/cli/cli.go \
  --file internal/task/task_test.go \
  --message "Fix these 10 bugs and write tests..." \
  --yes-always --no-auto-commits --no-gitignore \
  --model openrouter/anthropic/claude-sonnet-4

Results across 8 model+provider combinations:

• All runs applied SEARCH/REPLACE edits correctly (after adding --file flags)
• All runs had at least 1 compilation error or test failure
• No run produced passing go build && go test output
• Same models (Claude Sonnet 4, Grok 4.1, DeepSeek V3) scored A-grade through other agents that have verify-fix loops

What we think would help:

aider --file ... --message "..." \
  --yes-always --no-auto-commits --no-gitignore \
  --auto-test --test-cmd "go build ./... && go test -race -count=1 ./..." \
  --model openrouter/anthropic/claude-sonnet-4

Request

1. Document that --auto-test --test-cmd works with --message mode for headless/CI usage
2. Consider adding --auto-test to the scripting/headless examples in the docs
3. If --auto-test doesn't work with --message mode, consider making it work — it's the key gap vs competitors in headless benchmarks

Environment

• aider: v0.86.2
• Python: 3.12.13
• macOS arm64
• Models tested: Claude Sonnet 4, DeepSeek Chat, Grok 4.1 Fast, Mistral-Large-3, Llama-4-Maverick, GPT-4o (via OpenRouter, Azure AI, xAI direct)