[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-05-27

[github-actions[bot]]

Summary

• Date: 2026-05-27
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The docs have a clear value proposition and a solid quick-start flow, but a few gaps could trip up a brand-new user before they even install the tool.

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/602c2e0ab5ebb7f886a1d49cd607864cb214b9457b48701d86726502a888efff.png?raw=true

---

🔴 Critical Issues (Block getting started)

1. add-wizard command syntax not explained before use

In Step 2 of the Quick Start the user is asked to run:

gh aw add-wizard githubnext/agentics/daily-repo-status

The <owner>/<repo>/<workflow-name> format is explained after the command. A beginner will paste it and wonder "do I replace githubnext?". Add a short pre-amble sentence before the code block.

2. Secret setup is buried mid-step

The COPILOT_GITHUB_TOKEN creation instructions are inside a [!NOTE] callout after the command. The interactive wizard asks for the token, but the user won't know they need a fine-grained PAT beforehand. If they cancel the wizard to create the token, they lose context. Suggest surfacing this as a "Before you begin" sub-step.

3. No mental model for the dual-file pattern

Step 2 mentions two files are added (.md + .lock.yml) but the explanation of why is buried in a dense paragraph. A one-sentence upfront summary would help: "This adds a human-readable .md file you edit, and a compiled .lock.yml GitHub Actions reads."

---

🟡 Confusing Areas (Slow down learning)

1. "Frontmatter" jargon used without definition on first use

Step 4 says "If you changed the frontmatter (the configuration block between the --- markers)" — helpful inline, but the term appears in navigation and other pages without explanation. A glossary link or tooltip on first use would help.

2. When to run gh aw compile is unclear

The wizard auto-compiles, but Step 4 says "if you changed the frontmatter, regenerate". Beginners won't know the rule. Simplify to: "Run gh aw compile any time you edit the .md file."

3. CLI page lacks a "first-use path" callout

The CLI command reference lists ~20+ commands. There is no "if you're new, start here" callout showing the typical sequence: init → add-wizard → run → status.

4. Early-access warning is easy to miss

The important safety caveat ("in early development... use with caution") is a plain blockquote in the hero section. It should be a more visible [!WARNING] callout or banner.

5. Four AI engines listed without a recommended default

Prerequisites list Copilot, Claude, Codex, and Gemini simultaneously. A beginner faces decision paralysis. Recommend: mark one as "Recommended for beginners" and collapse the others.

---

🟢 What Worked Well

• "⏱️ Estimated time: 10 minutes" sets expectations immediately — great UX.
• Prerequisites checklist is comprehensive with version requirements.
• Inline token setup callouts are thorough once found.
• Video component on the Quick Start page is a great visual aid.
• Home page hero clearly communicates value in plain English.
• CLI commands table at the top of the CLI page is an excellent quick reference.
• Step 3 outcome description is motivating — tells users what the result looks like.
• add-wizard interactive flow is well-designed end-to-end.

---

Recommendations

Quick wins (low effort, high impact)

1. Add a recommended default engine in prerequisites to reduce decision fatigue.
2. Move token creation before the add-wizard command, not after.
3. Add a one-sentence mental model for .md + .lock.yml at the top of Step 2.
4. Add a glossary link on first use of "frontmatter".
5. Add a "Getting Started path" callout at the top of the CLI page.

Longer-term improvements

6. Add a "What if my first run fails?" troubleshooting section to Quick Start.
7. Make the early-access warning more prominent (banner or [!WARNING]).
8. Add a glossary page for terms like: frontmatter, lock file, engine, activation job, safe outputs.
9. Consider an inline engine selector on Quick Start to customize the steps shown.

---

References:

• §26492002884

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · sonnet46 3.8M · ◷

• expires on May 28, 2026, 5:19 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-05-28T05:19:15.374Z.

Closed by Workflow