ci: test guides by actually following them, auto-fix failures on main#1680
ci: test guides by actually following them, auto-fix failures on main#1680catmcgee wants to merge 6 commits into
Conversation
|
@catmcgee is attempting to deploy a commit to the Solana Foundation Team on Vercel. A member of the Team first needs to authorize it. |
Greptile SummaryIntroduces a Guides CI workflow that catches guide rot by running a Claude agent through each changed guide exactly as a reader would, failing CI on any blocker or warning. The design — a
Confidence Score: 5/5Safe to merge — the workflow is additive, touches only CI infrastructure, and the gate + detect design correctly prevents false greens. All four new files are CI tooling additions with no changes to application code. The gate job's dual check on detect and test results is sound, the report evaluator handles malformed agent output gracefully, and the autofix job's add-paths constraint prevents the bot from committing outside the guides directory. .github/workflows/guides-ci.yml — the tag maintainer step and the autofix condition are worth a second look before the scheduled sweep produces its first autofix PR. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Trigger: pull_request / schedule / workflow_dispatch] --> B[detect job\nCollect changed .mdx guides]
B -- any=false --> C[test SKIPPED]
B -- any=true --> D[test matrix\nWalk through each guide via Claude agent]
B -- detect fails --> E[gate FAILS]
C --> E
D -- all pass --> F[gate PASSES]
D -- any fail --> G[gate FAILS]
D -- any fail + schedule or dispatch+open_fix_prs --> H[autofix job\nClaude agent reads reports\nFixes guide MDX files]
H --> I[peter-evans/create-pull-request\nadd-paths: apps/docs/content/guides/**]
I --> J[Tag maintainer on new PR]
I --> K[Auto-fix PR triggers guides-ci again\nvalidates fix]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[Trigger: pull_request / schedule / workflow_dispatch] --> B[detect job\nCollect changed .mdx guides]
B -- any=false --> C[test SKIPPED]
B -- any=true --> D[test matrix\nWalk through each guide via Claude agent]
B -- detect fails --> E[gate FAILS]
C --> E
D -- all pass --> F[gate PASSES]
D -- any fail --> G[gate FAILS]
D -- any fail + schedule or dispatch+open_fix_prs --> H[autofix job\nClaude agent reads reports\nFixes guide MDX files]
H --> I[peter-evans/create-pull-request\nadd-paths: apps/docs/content/guides/**]
I --> J[Tag maintainer on new PR]
I --> K[Auto-fix PR triggers guides-ci again\nvalidates fix]
Reviews (4): Last reviewed commit: "ci: tag maintainer on auto-fix PRs" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| - name: Check results | ||
| env: | ||
| TEST_RESULT: ${{ needs.test.result }} | ||
| run: | | ||
| if [ "$TEST_RESULT" = "failure" ] || [ "$TEST_RESULT" = "cancelled" ]; then | ||
| echo "One or more guide walkthroughs failed." | ||
| exit 1 | ||
| fi | ||
| echo "All guide walkthroughs passed (or no guides changed)." | ||
|
|
||
| autofix: | ||
| name: Open fix PR | ||
| needs: [detect, test] | ||
| if: | ||
| always() && github.event_name == 'push' && needs.test.result == 'failure' |
There was a problem hiding this comment.
detect fails
The gate job only inspects needs.test.result. When the detect job itself errors (e.g., a jq or git diff failure under set -euo pipefail), the test job's if: needs.detect.outputs.any == 'true' condition evaluates to false and test is skipped. The gate then receives TEST_RESULT="skipped", which doesn't match "failure" or "cancelled", so it exits 0 — a false green on the required check. Adding a needs.detect.result check prevents this silent pass-through.
| /* Make header black for privacy page */ | ||
| header, | ||
| header nav { | ||
| background: #000 !important; | ||
| } |
There was a problem hiding this comment.
header selector here is not nested inside .privacy-page-design, contradicting the comment just below that says "All rules nested under .privacy-page-design so they don't leak into the rest of the site." Since privacy.css is a global stylesheet (not a CSS Module), this rule is truly global — it affects the site-wide header whenever this layout is active. The !important flag makes it especially hard to override from other layouts.
| /* Make header black for privacy page */ | |
| header, | |
| header nav { | |
| background: #000 !important; | |
| } | |
| /* Make header black for privacy page. | |
| NOTE: This rule is intentionally global (not under .privacy-page-design) | |
| because the header lives outside that wrapper. It applies only while the | |
| /privacy route segment is active. */ | |
| :root:has(.privacy-page-design) header, | |
| :root:has(.privacy-page-design) header nav { | |
| background: #000 !important; | |
| } |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
|
|
||
| Guide to test: ${{ matrix.guide }} | ||
| Write the JSON report to: ${{ runner.temp }}/guide-report.json | ||
| claude_args: | | ||
| --model claude-opus-4-8 | ||
| --max-turns 250 | ||
| --allowedTools "Bash,Read,Write,Edit,Glob,Grep,WebFetch,WebSearch" | ||
|
|
||
| - name: Evaluate report | ||
| run: | ||
| node scripts/guides-ci/check-report.mjs | ||
| "$RUNNER_TEMP/guide-report.json" | ||
|
|
There was a problem hiding this comment.
Every action reference here — anthropics/claude-code-action@v1, actions/create-github-app-token@v2, peter-evans/create-pull-request@v7, etc. — is pinned to a mutable semver tag rather than an immutable commit SHA. If any of these tags are moved (intentionally or after a supply-chain compromise), the next run picks up the new code. The risk is heightened in the autofix job, which combines contents: write / pull-requests: write permissions, the GitHub App private key, and the ANTHROPIC_API_KEY. Best practice is to pin each third-party action to a full commit SHA and keep it up to date with Dependabot or Renovate.
Adds a Guides CI workflow that walks through any changed guide under apps/docs/content/guides the way a reader would - installing the tools the guide says to install, running its commands, building its code - and fails the run if the guide is broken. Runs on PRs (blocking via the gate job) and on pushes to main, where failures also open a fix PR through the guides bot app. Prompts and the report checker live in scripts/guides-ci.
A full walkthrough takes 15-90 minutes and a few dollars per guide, and changed guides are already gated on the PR. A Monday sweep of all guides catches the ones that rot on their own; failures still open fix PRs.
Warnings like dead links or unrunnable snippets are still real problems for readers, so they now fail the walkthrough too. Pure environment noise (devnet rate limits, steps needing third-party credentials) gets its own severity and stays non-failing so unrelated PRs don't go red.
Fail the gate when guide detection itself errors instead of silently going green, and tell the fix agent the walkthrough reports are nested one per artifact directory rather than flat files.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
1 participant
Problem
Guides under
apps/docs/content/guidesrot silently: commands, package versions, and APIs drift, and we only find out when a reader gets stuck. Nothing in CI actually exercises what the guides tell people to do.Summary of changes
Adds a Guides CI workflow that walks through any changed guide the way a reader would and fails if the guide is broken.
.mdxfiles underapps/docs/content/guides, then one job per guide runs an agent (Opus, viaanthropics/claude-code-action) that follows the guide literally in a scratch directory — installs the tools the guide says to install, writes the exact code shown, runs every command. It reports findings as JSON;scripts/guides-ci/check-report.mjsfails the job on any blocker. The "Guides CI gate" job is the single check to mark as required in branch protection — it passes instantly on PRs that don't touch guides.solana-docs-checkerGitHub App so the PR runs CI normally.workflow_dispatchaccepts a comma-separated list of guide paths orallfor a full sweep.Credential-gated steps (third-party accounts, paid keys) and devnet flakiness are reported as warnings rather than failures to keep the check from being flaky. Prompts and the report checker live in
scripts/guides-ci/.Secrets used:
ANTHROPIC_API_KEY,DOCS_CHECKER_APP_ID,DOCS_CHECKER_APP_PRIVATE_KEY(all already configured on this repo).