Skip to content

Skills reference references/*.md by relative path, which silently fails when the skill lives outside the workspace #1008

Description

@AbdelHamdi

Summary

Skills instruct the agent to load references/*.md using workspace-relative paths. When the
skill is installed outside the workspace (the normal case for VS Code agent plugins), workspace-scoped
search tools cannot resolve those paths. The agent concludes the references "aren't available",
silently falls back to the abbreviated inline recipes, and reports a degraded result as if it were complete.

Environment

  • VS Code + GitHub Copilot agent mode
  • Skills installed at ~/.vscode/agent-plugins/github.com/dotnet/skills/...
  • Workspace open at an unrelated path (~/Desktop/project/orders-api-review)
  • macOS

Repro

  1. Install the plugins; open a workspace anywhere other than the plugin directory.
  2. Invoke analyzing-dotnet-performance.
  3. Agent follows Step 1 and tries to resolve references/critical-patterns.md.
  4. Workspace-scoped file search returns 0 matches — the glob is rooted at the workspace folder, and the
    skill directory is outside it.
  5. Agent proceeds in fallback mode.

The files exist the whole time:

$ ls plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/
async-patterns.md  collections-and-linq.md  critical-patterns.md
io-and-serialization.md  memory-and-strings.md  regex-patterns.md
structural-patterns.md

Why it happens

references/… is relative, and the skill never says relative to what. The agent has no basis to
resolve it against the SKILL.md file's own directory, so it falls back to workspace-relative — the only
root its search tools accept.

analyzing-dotnet-performance makes this materially worse

Step 1 instructs the agent to stop trying after the first failed lookup:

If reference files are not found (e.g., in a sandboxed environment or when the skill is embedded as
instructions only), skip file loading and proceed directly to Step 3 using the scan recipes listed
inline below. Do not spend time searching the filesystem for reference files — if they aren't at the
expected relative path, they aren't available.

This converts a recoverable path-resolution failure into a permanent one. One well-intentioned
efficiency guardrail guarantees the degraded path.

Impact

The fallback is silent. Nothing in the output tells the user the scan was partial.

In a real review, loading the references after the fact surfaced findings the inline recipes have no
detection for at all — System.Text.Json source-generation, anonymous types blocking source-gen, and
System.Threading.Channels for producer/consumer. The inline recipe list contains no equivalent checks,
so these were not missed through agent error; they were unreachable.

Scope — not limited to this skill

microbenchmarking/SKILL.md uses the same relative-link convention with a much stronger mandate:

You MUST read the reference files relevant to the task before writing any code — your training data
likely contains outdated or incorrect BDN patterns.

...and links as [references/comparison-strategies.md](references/comparison-strategies.md). Same
resolution failure, higher stakes: the skill's own premise is that unaided model knowledge of BDN is
wrong. Silently skipping the references yields exactly the outdated output the skill exists to prevent.

Other skills shipping references/: dotnet-trace-collect, coverage-analysis, and others.

Suggested fix

  1. State the resolution rule explicitly. Reference paths resolve against the SKILL.md file's own
    directory. Skills should say so, and instruct agents to read them with a direct file-read tool using
    an absolute path, not a workspace-scoped glob/search tool.
  2. Require one directory listing before declaring absence. Replace "do not spend time searching" with
    a bounded check: list <skill_dir>/references/ once. Cheap, deterministic, eliminates the false negative.
  3. Make degradation visible. If references genuinely can't be loaded, require the agent to state so in
    its output. A partial scan presented as complete is worse than a failed one.
  4. Consider templating the skill directory (e.g. ${SKILL_DIR}/references/…) so the intended root is
    unambiguous.

Item 3 matters most: the current failure mode is invisible to the user.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions