docs: replace starlight-llms-txt with custom llms.txt/agents.txt pointing to .github/aw/*.md #38630
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| # GitHub Agentic Workflows - AI crawler permissions | ||
| Allow: / | ||
| Llms-txt: https://github.github.com/gh-aw/llms.txt | ||
| Agents-txt: https://github.github.com/gh-aw/agents.txt | ||
| Summary: https://github.github.com/gh-aw/ai/summary.json |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| /** | ||
| * Shared helper: reads all agent-optimised prompts from .github/aw/*.md and | ||
| * returns metadata needed to build llms.txt / agents.txt. | ||
| */ | ||
| import { readdirSync, readFileSync } from 'node:fs'; | ||
| import { join } from 'node:path'; | ||
|
|
||
| export const RAW_BASE = | ||
| 'https://raw.githubusercontent.com/github/gh-aw/main/.github/aw'; | ||
|
|
||
| export interface AwPrompt { | ||
| file: string; | ||
| description: string; | ||
| rawUrl: string; | ||
| } | ||
|
|
||
| function parseFrontmatterDescription(content: string): string { | ||
| const match = content.match(/^---[\r\n]+([\s\S]*?)[\r\n]+---/); | ||
| if (!match) return ''; | ||
| // Simple key extraction – avoids pulling in a YAML parser at this layer | ||
| const descMatch = match[1].match(/^description:\s*(.+)$/m); | ||
|
Contributor
There was a problem hiding this comment. YAML-quoted description values will leak surrounding quotes into output: the regex captures the raw YAML value including any wrapping quote characters, so a description like 💡 Suggested fixStrip wrapping single/double quotes after trimming: const raw = descMatch[1].trim();
return raw.replace(/^(['"])(.*)\1$/, '$2');All current |
||
| return descMatch ? descMatch[1].trim() : ''; | ||
| } | ||
|
|
||
| export function getAwPrompts(): AwPrompt[] { | ||
| // process.cwd() is the docs/ directory during `astro build` | ||
| const awDir = join(process.cwd(), '../.github/aw'); | ||
|
Contributor
There was a problem hiding this comment. Fragile CWD assumption: path resolution via 💡 Suggested fixUse import { fileURLToPath } from "node:url";
// ...
const awDir = fileURLToPath(new URL("../../../.github/aw", import.meta.url));
The comment documenting the assumption is a smell — if the code were robust, the comment would be unnecessary. |
||
| return readdirSync(awDir) | ||
|
Contributor
There was a problem hiding this comment.
💡 DetailsThe intent seems to be indexing agent prompt files, and it is genuinely unclear whether Two options: Option A – explicit shallow-only intention (add a comment): // Intentionally shallow: only root-level prompt files, not subdirectories (runbooks/, imports/)
const awDir = ...;
return readdirSync(awDir)
...Option B – recursive inclusion: import { readdirSync, statSync, readFileSync } from 'node:fs';
function findMdFiles(dir: string): string[] {
return readdirSync(dir).flatMap((f) => {
const full = join(dir, f);
return statSync(full).isDirectory()
? findMdFiles(full).map((sub) => join(f, sub))
: f.endsWith('.md') ? [f] : [];
});
}The current code is a silent correctness hole: new files added to subdirectories will never appear in any published index without a code change. |
||
| .filter((f) => f.endsWith('.md')) | ||
| .sort() | ||
| .map((file) => { | ||
| const content = readFileSync(join(awDir, file), 'utf-8'); | ||
| return { | ||
| file, | ||
| description: parseFrontmatterDescription(content), | ||
| rawUrl: `${RAW_BASE}/${file}`, | ||
| }; | ||
| }); | ||
| } | ||
|
Comment on lines
+25
to
+39
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| import type { APIRoute } from 'astro'; | ||
| import { getAwPrompts } from './_aw-prompts.js'; | ||
|
|
||
| export const GET: APIRoute = () => { | ||
| const prompts = getAwPrompts(); | ||
|
|
||
| const lines = [ | ||
| '# GitHub Agentic Workflows – Agent Prompts', | ||
| '', | ||
| '> Agent-optimised prompt files for GitHub Agentic Workflows (gh-aw).', | ||
| '> Use these files to ground AI agents working with gh-aw workflows.', | ||
| '', | ||
| '## Prompts', | ||
| '', | ||
| ...prompts.map(({ file, description, rawUrl }) => { | ||
| const label = file.replace(/\.md$/, ''); | ||
| return description | ||
| ? `- [${label}](${rawUrl}): ${description}` | ||
| : `- [${label}](${rawUrl})`; | ||
| }), | ||
| ]; | ||
|
Comment on lines
+4
to
+21
Comment on lines
+7
to
+21
|
||
|
|
||
| return new Response(lines.join('\n'), { | ||
| headers: { 'Content-Type': 'text/plain; charset=utf-8' }, | ||
| }); | ||
| }; | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RAW_BASEis hardcoded to themainbranch: raw URLs will be stale/broken during branch builds and PR previews where new.github/aw/*.mdfiles are added but not yet merged tomain.💡 Details
The constant
https://raw.githubusercontent.com/github/gh-aw/main/.github/awis baked in. A PR that adds a new prompt file will generate allms.txtentry with araw.githubusercontent.com/.../main/...URL that 404s until the PR merges. Any agent or tool consuming a preview deploy of the docs will follow a dead link.Consider injecting the commit SHA at build time:
This ensures the generated index always points to the exact commit that was used to build it.