Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions evals/behavioral/cases/audit.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"slug": "audit",
"origin_skill": "auth0-checkmate",
"skill_name": "auth0-checkmate",
"evals": [
{
"id": 1,
"prompt": "Run an Auth0 CheckMate tenant audit for the company acmecorp.io. The Auth0 tenant is acmecorp.us.auth0.com. Generate a personalized security/configuration report as a PDF.",
"expected_output": "Phased CheckMate audit producing a personalized Auth0 Platform Development Review with triaged findings, immediate vs. post-upgrade action lists, and a plan recommendation. Report saved as markdown, HTML, and PDF.",
"files": [],
"assertions": [
"Bootstraps the Auth0 CLI (auth0 login or verifies existing session)",
"Creates or reuses a dedicated CheckMate M2M application in the tenant",
"Writes credentials to ~/.auth0-checkmate.env with mode 600 (chmod 600)",
"Substitutes <tenant_domain>, <client_id>, and <client_secret> before writing the env file",
"Runs the CheckMate audit tool against the tenant",
"Gathers lightweight company context for acmecorp.io (business model, products, auth surfaces)",
"Produces a findings list with severity, title, and personalized remediation notes",
"Triages findings into Immediate Actions (current plan) vs After Upgrading (requires upgrade)",
"Emits a plan recommendation with exact cost from the pricing reference",
"Saves report as auth0checkmate_<tenant>_<timestamp>.{md,html,pdf}",
"State written to ~/.auth0-checkmate/state/ (not ~/.claude/skills/)",
"Does not include sales fit scores or firmographic data in the report"
]
},
Comment thread
coderabbitai[bot] marked this conversation as resolved.
{
"id": 2,
"prompt": "I already have my Auth0 tenant set up and the M2M app created. Just run CheckMate and give me the findings — skip the bootstrap steps.",
"expected_output": "CheckMate audit skipping Phase 1/2 bootstrap if credentials exist, producing findings and a report.",
"files": [],
"assertions": [
"Reads existing credentials from ~/.auth0-checkmate.env or ~/.auth0-checkmate/state/setup.json",
"Skips M2M app creation if client_id is already cached",
"Runs the CheckMate audit and parses data.report.summary[]",
"Produces a report with triaged findings"
]
}
],
"graders": null
}
7 changes: 7 additions & 0 deletions evals/routing-cases.json
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,13 @@
"framework": "nextjs",
"tooling": "terraform",
"expect_refs": ["feature-migration.md", "tooling-terraform.md", "framework-nextjs.md"]
},
{
"id": "audit-tenant",
"intent": "audit",
"framework": null,
"tooling": "cli",
"expect_refs": ["feature-audit.md", "feature-audit-pricing.md", "feature-audit-remediation.md", "tooling-cli.md"]
}
]
}
2 changes: 1 addition & 1 deletion plugins/auth0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ npx skills add auth0/agent-skills/plugins/auth0

| Skill | Description | Documentation |
|-------|-------------|---------------|
| [auth0](skills/auth0) | Adds Auth0 authentication to any app. Covers 35+ frameworks (React, Next.js, Vue, Angular, Express, Flask, FastAPI, Spring Boot, Swift, Android, Flutter, Laravel, Go, PHP, .NET MAUI, ASP.NET Core, React Native, Expo, Ionic, and more), MFA, Organizations, custom domains, ACUL screen generation, branding, debugging auth errors, security best practices, and migration from other providers. | [SKILL.md](skills/auth0/SKILL.md) |
| [auth0](skills/auth0) | Adds Auth0 authentication to any app. Covers 35+ frameworks (React, Next.js, Vue, Angular, Express, Flask, FastAPI, Spring Boot, Swift, Android, Flutter, Laravel, Go, PHP, .NET MAUI, ASP.NET Core, React Native, Expo, Ionic, and more), MFA, Organizations, custom domains, ACUL screen generation, branding, debugging auth errors, security best practices, running a tenant security & configuration audit (CheckMate), and migration from other providers. | [SKILL.md](skills/auth0/SKILL.md) |

## Forcing the skill with `/auth0`

Expand Down
12 changes: 11 additions & 1 deletion plugins/auth0/skills/auth0/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: auth0
description: Use when adding, fixing, or improving authentication in any app — login, logout, signup, route protection, JWT/access token validation, refresh token rotation, MFA, passkeys, step-up auth, SSO, RBAC, Organizations for B2B multi-tenant SaaS, custom login domains, ACUL, or Universal Login branding. Use even if Auth0 isn't mentioned — any time a developer asks how to authenticate users, secure an API, debug a 401, CORS error, callback URL mismatch, redirect loop, or 429 rate limit, or migrate from Clerk, NextAuth.js, Firebase Auth, Supabase, Cognito, or Passport.js. Covers React, Next.js, Vue, Nuxt, Angular, Express, Flask, FastAPI, Spring Boot, Go, Swift, Android, Flutter, PHP, Laravel, ASP.NET Core, React Native, Expo, Ionic, and all Auth0 SDKs.
description: Use when adding, fixing, or improving authentication in any app — login, logout, signup, route protection, JWT/access token validation, refresh token rotation, MFA, passkeys, step-up auth, SSO, RBAC, Organizations for B2B multi-tenant SaaS, custom login domains, ACUL, or Universal Login branding. Use to audit a tenant (CheckMate) or fix findings. Use even if Auth0 isn't mentioned — any time a developer asks how to authenticate users, secure an API, debug a 401, CORS error, callback URL mismatch, redirect loop, or 429 rate limit, or migrate from Clerk, NextAuth.js, Firebase, Supabase, Cognito, or Passport.js. Covers React, Next.js, Vue, Nuxt, Angular, Express, Flask, FastAPI, Spring Boot, Go, Swift, Android, Flutter, PHP, Laravel, ASP.NET Core, React Native, Expo, Ionic, and all Auth0 SDKs.
license: Apache-2.0
metadata:
author: Auth0 <support@auth0.com>
Expand Down Expand Up @@ -45,6 +45,7 @@ section heading (`### feature:mfa`) listing which reference files to load.
| Build fully custom login/signup screens with your own code or framework, beyond what theme settings allow. *Auth0: Advanced Customization for Universal Login (ACUL).* | **feature:acul** |
| Change how the login page looks — logo, colors, fonts, background, overall theme. *Auth0: branding, Universal Login customization.* | **feature:branding** |
| Bind tokens to the client so a stolen or leaked token can't be reused/replayed from another machine. *Auth0: DPoP (Demonstrating Proof-of-Possession), sender-constrained tokens.* | **feature:dpop** |
| Audit an existing Auth0 tenant for security & configuration issues and produce a report — run CheckMate, get a tenant configuration / platform development review, then optionally fix findings via the CLI. *Auth0: tenant audit, CheckMate.* | **audit** |
| Ask for best practices, "is this secure?", how to handle tokens safely, "how should I do X". *Auth0: guidance / security.* | **guidance** |
| Hit an error: 401 Unauthorized, 403 Forbidden, CORS, callback URL mismatch, redirect loop. *Auth0: debugging.* | **debug** |
| Hit rate limiting: 429 Too Many Requests, quota exceeded. *Auth0: rate limits.* | **debug:rate-limit** |
Expand Down Expand Up @@ -339,6 +340,15 @@ Read: references/tooling-{tooling}.md
If framework detected: Read references/framework-{framework}.md
```

### audit
```
Read: references/feature-audit.md
Read: references/feature-audit-pricing.md
Read: references/feature-audit-remediation.md
Read: references/tooling-{tooling}.md
Apply findings only with per-command confirmation; verify each change by re-fetch.
```

### upgrade-sdk
```
Read: references/framework-{framework}.md
Expand Down
Loading
Loading