Add CI test to validate Azure MCP tool names in skill markdown files by Copilot · Pull Request #1709 · microsoft/GitHub-Copilot-for-Azure

Copilot · 2026-04-03T21:33:34Z

Skill markdown files reference Azure MCP tools by name, but there was no automated check to catch stale or mistyped tool names that don't match what the server actually exposes.

Changes

New test tests/skills-validation/mcp-tool-names.test.ts — unit test that runs as part of the existing test:unit suite (picked up by the test-all-skills.yml CI workflow). It:
- Spawns @azure/mcp@latest and queries the live tool list via JSON-RPC tools/list
- Recursively scans all .md files under plugin/skills/ for two naming conventions:
  - mcp_azure_mcp_<tool> (GitHub Copilot MCP tool reference format)
  - azure__<tool> (alternative format)
- Fails with file path + line number diagnostics for every invalid tool name found
Bug fixes in 9 MD files — discovered by the new test: mcp_azure_mcp_get_bestpractices is not a real tool; the correct name is mcp_azure_mcp_get_azure_bestpractices. Fixed across azure-upgrade, azure-cloud-migrate, and azure-prepare skill reference files.

Failure output example

When a file references a non-existent tool, the test produces:

● all Azure MCP tool names referenced in skill markdown files are valid

  expect(received).toHaveLength(0)

  Received:
    ["azure-upgrade/SKILL.md:28 — "get_bestpractices" is not a valid Azure MCP tool name",
     ...]

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

centralus-2.in.applicationinsights.azure.com
- Triggering command: /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp server start (dns block)
- Triggering command: /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp server start grep -l t_bestpractices ure/plugin/skills/azure-compliance/references/azure-quick-review.md ure/tests/node_modules/.bin/sh t_bestpractices otnet.md rgo/bin/grep grep -l overage b-Copilot-for-Azure/plugin/skills/azure-deploy/references/recipes/cicd/verify.md ep t_bestpractices nces/ptu-guide.m-c k/_temp/ghcca-no"jest" '--testPathIgnorePatterns=node_modules|_template|integration' grep (dns block)
- Triggering command: /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp server start grep ure/�� t_bestpractices b-Copilot-for-Azure/plugin/skills/azure-deploy/references/recipes/terraform/errors.md de/node/bin/grep t_bestpractices md ndor/bin/grep grep ure/�� d .md modules/@npmcli/run-script/lib/node-gyp-bin/npx t_bestpractices d grep grep (dns block)
westus-0.in.applicationinsights.azure.com
- Triggering command: /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp server start (dns block)
- Triggering command: /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp server start grep -l t_bestpractices ure/plugin/skills/azure-compliance/references/azure-quick-review.md ure/tests/node_modules/.bin/sh t_bestpractices otnet.md rgo/bin/grep grep -l overage b-Copilot-for-Azure/plugin/skills/azure-deploy/references/recipes/cicd/verify.md ep t_bestpractices nces/ptu-guide.m-c k/_temp/ghcca-no"jest" '--testPathIgnorePatterns=node_modules|_template|integration' grep (dns block)
- Triggering command: /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp /home/REDACTED/.npm/_npx/541454632b79112e/node_modules/@azure/mcp-linux-x64/dist/azmcp server start grep ure/�� t_bestpractices b-Copilot-for-Azure/plugin/skills/azure-deploy/references/recipes/terraform/errors.md de/node/bin/grep t_bestpractices md ndor/bin/grep grep ure/�� d .md modules/@npmcli/run-script/lib/node-gyp-bin/npx t_bestpractices d grep grep (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

…eferences Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/362a2400-266c-4070-9a8b-625601205ea3 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

kvenkatrajan · 2026-04-03T22:04:21Z

@copilot - bump the version on 'azure-cloud-migrate' , 'azure-prepare', 'azure-upgrade' skills

…de skills Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/5a67a30f-b6d1-43a9-8c38-a2ad05209e1b Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot · 2026-04-03T22:08:36Z

@copilot - bump the version on 'azure-cloud-migrate' , 'azure-prepare', 'azure-upgrade' skills

Done in commit 243a564. Bumped versions for the three modified skills:

azure-cloud-migrate: 1.0.1 → 1.0.2
azure-prepare: 1.1.6 → 1.1.7
azure-upgrade: 1.0.0 → 1.0.1

Copilot

Pull request overview

Adds automated validation to ensure Azure MCP tool names referenced in skill markdown stay in sync with the tools actually exposed by @azure/mcp, and fixes existing stale tool references found by that validation.

Changes:

Add a new unit test that boots @azure/mcp and validates referenced Azure MCP tool names across plugin/skills/**/*.md.
Fix stale tool references (get_bestpractices → get_azure_bestpractices) across affected skill docs.
Update azure-upgrade and azure-cloud-migrate SKILL tool lists/usages to the corrected tool name.

Show a summary per file

File	Description
tests/skills-validation/mcp-tool-names.test.ts	New unit test that queries Azure MCP `tools/list` and validates tool references in skill markdown.
plugin/skills/azure-upgrade/SKILL.md	Fixes Azure MCP best-practices tool name in rules/tool table.
plugin/skills/azure-upgrade/references/services/functions/assessment.md	Fixes best-practices tool reference in tip.
plugin/skills/azure-upgrade/references/global-rules.md	Fixes best-practices tool reference in global rules.
plugin/skills/azure-prepare/references/recipes/terraform/README.md	Fixes best-practices tool reference in Terraform recipe table.
plugin/skills/azure-prepare/references/recipes/azcli/README.md	Fixes best-practices tool reference in Azure CLI recipe table.
plugin/skills/azure-cloud-migrate/SKILL.md	Fixes Azure MCP best-practices tool name in rules.
plugin/skills/azure-cloud-migrate/references/services/functions/global-rules.md	Fixes best-practices tool reference in rules.
plugin/skills/azure-cloud-migrate/references/services/functions/code-migration.md	Fixes best-practices tool reference in prerequisites/steps.
plugin/skills/azure-cloud-migrate/references/services/functions/assessment.md	Fixes best-practices tool reference in tip.

Copilot's findings

Files reviewed: 11/11 changed files
Comments generated: 3

Copilot · 2026-04-03T22:13:23Z

tests/skills-validation/mcp-tool-names.test.ts

+    serverProcess.on("close", () => clearTimeout(timeoutHandle));
+
+    // Unref the child process so it doesn't prevent the Jest worker from exiting
+    serverProcess.unref();


If the npx @azure/mcp@latest server start process exits early (e.g., download failure, missing runtime), this promise currently waits until the 110s timeout before failing, which slows CI and hides the underlying exit reason. Consider capturing stderr/stdout and rejecting immediately in the close handler when the process exits before a tools/list response is received (include exit code/signal and last stderr lines for diagnostics).

@copilot apply changes based on this feedback

Done in commit 26d67d8. The close handler now rejects immediately when the process exits before a tools/list response is received, including the exit code, signal, and last 20 lines of stderr. A settled flag guards all paths (success, error event, timeout, and early close) against double-resolution.

plugin/skills/azure-upgrade/SKILL.md

plugin/skills/azure-cloud-migrate/SKILL.md

kvenkatrajan · 2026-04-03T22:26:07Z

Fixes #778

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/bfd49079-1961-4c4e-b89f-d5e1a195b0ac Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Initial plan

9106121

Copilot AI assigned Copilot and kvenkatrajan Apr 3, 2026

Copilot started work on behalf of kvenkatrajan April 3, 2026 21:33 View session

Copilot AI linked an issue Apr 3, 2026 that may be closed by this pull request

PR CI tests for ensuring that the tool names in md file reflect actual Azure MCP server tools #778

Open

Copilot AI changed the title ~~[WIP] Add CI tests for validating Azure MCP tool names in markdown files~~ Add CI test to validate Azure MCP tool names in skill markdown files Apr 3, 2026

Copilot finished work on behalf of kvenkatrajan April 3, 2026 21:55

Copilot AI requested a review from kvenkatrajan April 3, 2026 21:55

kvenkatrajan marked this pull request as ready for review April 3, 2026 21:59

kvenkatrajan requested review from saikoumudi, tmeschter and wbreza as code owners April 3, 2026 21:59

Copilot AI review requested due to automatic review settings April 3, 2026 21:59

Copilot started reviewing on behalf of kvenkatrajan April 3, 2026 22:03 View session

Copilot started work on behalf of kvenkatrajan April 3, 2026 22:04 View session

Copilot finished work on behalf of kvenkatrajan April 3, 2026 22:09

Copilot AI reviewed Apr 3, 2026

View reviewed changes

Copilot started work on behalf of kvenkatrajan April 3, 2026 22:19 View session

Reject early on MCP server process exit with stderr diagnostics

26d67d8

Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/bfd49079-1961-4c4e-b89f-d5e1a195b0ac Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com>

Copilot finished work on behalf of kvenkatrajan April 3, 2026 22:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CI test to validate Azure MCP tool names in skill markdown files#1709

Add CI test to validate Azure MCP tool names in skill markdown files#1709
Copilot wants to merge 4 commits intomainfrom
copilot/add-ci-test-for-azure-mcp-tools

Copilot AI commented Apr 3, 2026 •

edited

Loading

Uh oh!

kvenkatrajan commented Apr 3, 2026

Uh oh!

Copilot AI commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

kvenkatrajan Apr 3, 2026

Uh oh!

Copilot AI Apr 3, 2026

Uh oh!

Uh oh!

Uh oh!

kvenkatrajan commented Apr 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Failure output example

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

kvenkatrajan commented Apr 3, 2026

Uh oh!

Copilot AI commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

kvenkatrajan Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kvenkatrajan commented Apr 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Apr 3, 2026 •

edited

Loading