bench: add skill benchmarking framework with promptfoo

[iskysun96]

Summary

• Adds a structured eval framework for all 15 AI skills (7 Move + 8 TS SDK) using promptfoo
• Measures compilation success, security pattern compliance, V2 syntax adherence, TypeScript type-checking, SDK pattern correctness, token usage, cost, and latency
• 29 test cases across 15 skill configs with custom assertion scripts

What's included

Category	Count	Details
Assertion scripts	10	3 shared utilities + 5 Move assertions + 2 TS assertions
Fixture contracts	5	NFT, FA, insecure, unoptimized, V1 legacy
Skill configs	15	promptfooconfig.yaml + tests.yaml per skill
Prompt builder	1	Composes SKILL.md + pattern files into prompt.json
Runner scripts	2	setup.sh, run-all.sh
CI workflow	1	Auto-evals on PRs modifying skills/patterns

Usage

cd bench
npm install
npm run setup                          # one-time: cache deps + build prompts
npm run eval                           # run all 15 skill evals (29 test cases)
npm run eval:move                      # Move skills only
npm run eval:sdk                       # TS SDK skills only
npm run eval:skill -- write-contracts  # single skill
npx promptfoo view                     # browse results in web UI

Test plan

• cd bench && npm install succeeds
• npm run build-prompts generates prompt.json for all 15 skills
• npm run setup creates cached workspaces
• npm run eval completes all 29 test cases (requires ANTHROPIC_API_KEY)
• npx promptfoo view shows web UI with results

🤖 Generated with Claude Code

[github-actions]

🔬 Skill Eval Results

Skill	Status	Pass	Fail	Tokens	Cost
move_analyze-gas-optimization_20260311	❌	1	1	33798	N/A
move_deploy-contracts_20260311	❌	0	1	6411	N/A
move_generate-tests_20260311	❌	0	2	38866	N/A
move_modernize-move_20260311	❌	0	2	33194	N/A
move_search-aptos-examples_20260311	✅	2	0	14624	N/A
move_security-audit_20260311	❌	0	2	54242	N/A
move_write-contracts_20260311	❌	0	4	244648	N/A
sdk_ts-sdk-account_20260311	❌	0	2	12791	N/A
sdk_ts-sdk-address_20260311	❌	0	1	7340	N/A
sdk_ts-sdk-client_20260311	❌	0	2	12037	N/A
sdk_ts-sdk-transactions_20260311	❌	0	2	13883	N/A
sdk_ts-sdk-types_20260311	❌	0	2	12804	N/A
sdk_ts-sdk-view-and-query_20260311	❌	0	2	9892	N/A
sdk_ts-sdk-wallet-adapter_20260311	✅	0	0	N/A	N/A
sdk_use-ts-sdk_20260311	✅	1	0	11162	N/A

Failure Details

move_analyze-gas-optimization_20260311

✓ [PASS] Analyze this Move smart contract for gas optimization opportunities. Identify ex
    ✗ [FAIL] Analyze this Move smart contract for gas optimization opportunities. Identify ex
      ✗ llm-rubric: The response should acknowledge that the contrac — The output fails to acknowledge that the contract is already reasonably well-optimized. Instead, it claims "30-40% reduction" potential and identifies "3 High, 2 Medium" priority issues. Several of th
    Results: 1 passed, 1 failed, 2 total

move_deploy-contracts_20260311

✗ [FAIL] Guide me through deploying a compiled Move contract to testnet. The contract mod
      ✗ icontains: aptos move publish — Expected output to contain "aptos move publish"
    Results: 0 passed, 1 failed, 1 total

move_generate-tests_20260311

✗ [FAIL] Write comprehensive Move unit tests for this NFT collection contract. Include te
      ✗ assertion — No Move code blocks found in the output. Expected at least one fenced ```move block containing test code.
    ✗ [FAIL] Write comprehensive Move unit tests for this fungible token contract. Include te
      ✗ assertion — No Move code blocks found in the output. Expected at least one fenced ```move block containing test code.
    Results: 0 passed, 2 failed, 2 total

move_modernize-move_20260311

✗ [FAIL] Modernize this Move V1 contract to V2 syntax and patterns. Replace resource acco
      ✗ assertion — No Move code blocks found in the output. Expected at least one fenced ```move block.
      ✗ assertion — No Move code blocks found in the output. Cannot perform V2 syntax checks.
    ✗ [FAIL] Modernize only the V1 patterns in this contract while preserving the V2 patterns
      ✗ assertion — No Move code blocks found in the output. Cannot perform V2 syntax checks.
    Results: 0 passed, 2 failed, 2 total

move_security-audit_20260311

✗ [FAIL] Perform a comprehensive security audit of this Move smart contract. Identify all
      ✗ assertion — No output provided to check for audit report categories.
    ✗ [FAIL] Perform a comprehensive security audit of this Move smart contract. Identify all
      ✗ assertion — No output provided to check for audit report categories.
    Results: 0 passed, 2 failed, 2 total

move_write-contracts_20260311

✗ [FAIL] Write a Move V2 smart contract for a fungible asset token called 'MyToken' with 
      ✗ assertion — No Move code blocks found in the output. Expected at least one fenced ```move block.
      ✗ assertion — No Move code blocks found in the output. Cannot perform security pattern checks.
      ✗ assertion — No Move code blocks found in the output. Cannot perform V2 syntax checks.
    ✗ [FAIL] Write a Move V2 smart contract for an NFT collection called 'ArtCollection' usin
      ✗ assertion — No Move code blocks found in the output. Expected at least one fenced ```move block.
      ✗ assertion — No Move code blocks found in the output. Cannot perform security pattern checks.
      ✗ assertion — No Move code blocks found in the output. Cannot perform V2 syntax checks.
    ✗ [FAIL] Write a Move V2 smart contract for a staking vault where users can stake APT tok
      ✗ assertion — No Move code blocks found in the output. Expected at least one fenced ```move block.
      ✗ assertion — No Move code blocks found in the output. Cannot perform security pattern checks.
      ✗ assertion — No Move code blocks found in the output. Cannot perform V2 syntax checks.
    ✗ [FAIL] Write a Move V2 smart contract for a DAO governance system. Include proposal cre
      ✗ assertion — No Move code blocks found in the output. Expected at least one fenced ```move block.
      ✗ assertion — No Move code blocks found in the output. Cannot perform security pattern checks.
      ✗ assertion — No Move code blocks found in the output. Cannot perform V2 syntax checks.
    Results: 0 passed, 4 failed, 4 total

sdk_ts-sdk-account_20260311

✗ [FAIL] Create a new Aptos account and fund it on devnet using the TypeScript SDK. Show 
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    ✗ [FAIL] Create an Aptos account from an existing private key using the TypeScript SDK. S
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    Results: 0 passed, 2 failed, 2 total

sdk_ts-sdk-address_20260311

✗ [FAIL] Show how to parse and validate an Aptos address in TypeScript. Include handling 
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    Results: 0 passed, 1 failed, 1 total

sdk_ts-sdk-client_20260311

✗ [FAIL] Set up an Aptos SDK client configured for testnet in a TypeScript project. Show 
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    ✗ [FAIL] Set up an Aptos SDK client with a custom API key for higher rate limits. Show ho
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    Results: 0 passed, 2 failed, 2 total

sdk_ts-sdk-transactions_20260311

✗ [FAIL] Show me how to build and submit a transaction that calls an entry function on an
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    ✗ [FAIL] Show how to simulate a transaction before submitting it on Aptos using the TypeS
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    Results: 0 passed, 2 failed, 2 total

sdk_ts-sdk-types_20260311

✗ [FAIL] Show me how to map Move types to TypeScript types for an Aptos module. The modul
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    ✗ [FAIL] Show me how to parse a view function response from Aptos in TypeScript. The view
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    Results: 0 passed, 2 failed, 2 total

sdk_ts-sdk-view-and-query_20260311

✗ [FAIL] Show me how to call a view function on Aptos using the TypeScript SDK. The funct
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
    ✗ [FAIL] Show me how to get an account's APT balance using the Aptos TypeScript SDK. Use 
      ✗ assertion — No TypeScript code blocks found in the output. Expected at least one fenced ```typescript or ```ts block.
      ✗ assertion — No TypeScript code blocks found in the output. Cannot perform SDK pattern checks.
      ✗ not-icontains: getAccountAPTAmount — Expected output to not contain "getAccountAPTAmount"
    Results: 0 passed, 2 failed, 2 total

---

Generated by skill-eval CI