|
| 1 | +--- |
| 2 | +name: pr-quiz |
| 3 | +description: >- |
| 4 | + Quiz the PR author on their own branch before they request review, to prove they |
| 5 | + actually understand the change - especially code an AI wrote for them. Scopes the |
| 6 | + branch diff vs its base, reads the changed code, then asks graded questions about |
| 7 | + what changed, why, how it works, what it could break, and which edge cases it must |
| 8 | + handle. Presents all questions first, waits for the author's answers, then grades |
| 9 | + each honestly against the real code (Correct / Partial / Incorrect with the true |
| 10 | + answer and file:line), scores it, and gives a readiness verdict that names the |
| 11 | + areas to re-study before asking humans to review. Use when asked to quiz me on my |
| 12 | + PR/branch, "test my understanding before review", a self-check gate before opening |
| 13 | + a PR, or before requesting reviewers. Administered as an interactive |
| 14 | + multiple-choice quiz (clickable options) by default; pass --free-text for |
| 15 | + written answers, --questions N to set count, --save to write a scorecard. |
| 16 | +argument-hint: "[branch-or-base-ref] [--questions N] [--free-text] [--save]" |
| 17 | +allowed-tools: Bash, Read, Grep, Glob, Write, AskUserQuestion |
| 18 | +--- |
| 19 | + |
| 20 | +# PR Quiz |
| 21 | + |
| 22 | +Test whether the **author** genuinely understands their own branch before they ask |
| 23 | +other people to spend time reviewing it. This is a self-check gate: the point is to |
| 24 | +catch changes - often AI-written - that the author would not be able to explain or |
| 25 | +defend in review. Be a fair but honest examiner, not a pushover. |
| 26 | + |
| 27 | +`$ARGUMENTS` may name a base ref or branch to diff against; default is this branch |
| 28 | +vs where it forked from the main line. Flags: |
| 29 | +- `--questions N` - target N questions (else scale to diff size, see below). |
| 30 | +- `--free-text` - administer as a written numbered list instead of the default |
| 31 | + interactive multiple-choice. |
| 32 | +- `--save` - also write a scorecard file after grading. |
| 33 | + |
| 34 | +## Integrity rules (read first - the whole skill depends on these) |
| 35 | + |
| 36 | +1. **Present every question before revealing any answer.** Ask, then wait. Never |
| 37 | + show the answer key alongside the questions. |
| 38 | +2. **Do not give hints or the answer while the quiz is open.** If the author asks |
| 39 | + "what's the answer?" or "is it X?" before committing, decline warmly and tell |
| 40 | + them to give their best answer first - guessing is part of the signal. |
| 41 | +3. **Grade truthfully.** Vague, hand-wavy, or "the AI did it" non-answers are |
| 42 | + Partial or Incorrect, not Correct. Do not inflate the score to be nice; a false |
| 43 | + pass defeats the entire purpose. |
| 44 | +4. **Ground everything in code you actually read.** Every question and every model |
| 45 | + answer must trace to a real line in the diff. Cite `path:line`. No trivia |
| 46 | + ("how many lines?"), no invented behavior. |
| 47 | +5. **Credit real understanding.** If the author explains it correctly in their own |
| 48 | + words, mark it Correct even if worded differently than your key. |
| 49 | + |
| 50 | +## Process |
| 51 | + |
| 52 | +### 1. Scope the change (silently) |
| 53 | +- Find the base. Prefer the fork point off the main line so the quiz covers only |
| 54 | + this branch's work: |
| 55 | + ```bash |
| 56 | + git fetch -q origin 2>/dev/null; \ |
| 57 | + BASE=$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main); \ |
| 58 | + git diff --stat "$BASE"...HEAD |
| 59 | + ``` |
| 60 | + If `$ARGUMENTS` names a ref, diff against that instead. |
| 61 | +- If the diff is empty, stop and say there's nothing to quiz on. |
| 62 | +- Read commit messages / PR description for the *stated* intent, but verify it |
| 63 | + against the actual diff - a mismatch is itself a good question. |
| 64 | + |
| 65 | +### 2. Understand the code well enough to examine on it |
| 66 | +Read the full diff plus enough surrounding context and related files to answer |
| 67 | +every question you plan to ask. You cannot grade understanding you don't have. |
| 68 | +Note the non-obvious parts: the design decisions, the risky lines, the edge cases, |
| 69 | +the cross-file ripples, and anything that violates or upholds repo conventions |
| 70 | +(for this repo e.g. `@app/*` import layering, all file ops via FileContext, |
| 71 | +Jackson 3 / Spring Boot 4 APIs, engine typed-contract boundaries). |
| 72 | + |
| 73 | +### 3. Build the question set |
| 74 | +Scale count to the change unless `--questions N` is given: |
| 75 | +small (< ~50 changed lines) 3-4, medium 5-8, large 9-12. Cap at 12. |
| 76 | +Draw from these categories - weight toward the ones the diff actually exercises: |
| 77 | +- **Intent** - what problem this solves; why it was needed now. |
| 78 | +- **Mechanism** - how a specific non-trivial piece actually works ("walk me |
| 79 | + through what `foo()` does when called with X"). |
| 80 | +- **Decisions & alternatives** - why this approach over an obvious alternative; |
| 81 | + what a reviewer would reasonably push back on. |
| 82 | +- **Blast radius** - what else this touches or could break; what you'd retest. |
| 83 | +- **Edge cases** - inputs/states the change must handle (null, empty, large, |
| 84 | + concurrent, error paths). |
| 85 | +- **Conventions & correctness** - does it follow the repo's rules; is there a |
| 86 | + latent bug the author should be able to spot. |
| 87 | +Prefer questions the author can only answer if they read and understood the code. |
| 88 | +Keep a private answer key with `path:line` for each - do **not** show it yet. |
| 89 | + |
| 90 | +### 4. Administer the quiz |
| 91 | +- **Default (multiple choice):** use the `AskUserQuestion` tool. Per question write |
| 92 | + 3-4 options where **every** option is independently plausible - each distractor a |
| 93 | + real-but-wrong reading of the code, not filler. Two hard rules so the answer |
| 94 | + can't be spotted by shape rather than knowledge: |
| 95 | + - **Randomise the correct option's position** across questions - never default |
| 96 | + it to first. Spread it roughly evenly over the slots. |
| 97 | + - **Keep all options the same depth and length.** Do not describe the correct |
| 98 | + one more fully than the distractors - a longer or more-detailed option is a |
| 99 | + dead giveaway. Trim the right answer or flesh out the wrong ones until a |
| 100 | + reader can't tell them apart by size. |
| 101 | + The tool caps a call at 4 questions, so ask in batches of 4 - but run them as |
| 102 | + one continuous flow: fire the next batch immediately after the previous |
| 103 | + returns, with no narration ("Round 2 of 3") and no grading between batches. |
| 104 | + The author always has an "Other" free-text escape, which is fine. |
| 105 | +- **`--free-text`:** present all questions in one numbered list, then say |
| 106 | + "Answer in one reply; number your answers. I won't grade until you're done." |
| 107 | + Wait for the author's answers. |
| 108 | +- Do not proceed to grading until every answer is in. |
| 109 | + |
| 110 | +### 5. Grade |
| 111 | +For each question, in order: |
| 112 | +- Verdict: **Correct** / **Partial** / **Incorrect**. |
| 113 | +- The model answer in one or two sentences, citing the real `path:line`. |
| 114 | +- One line on the gap when Partial/Incorrect - what they missed and where to look. |
| 115 | +Then a **Score** (e.g. 6/8, counting Partial as half) and a one-line summary of |
| 116 | +the pattern (e.g. "solid on intent, shaky on the error paths"). |
| 117 | + |
| 118 | +### 6. Readiness verdict |
| 119 | +End with a clear call: |
| 120 | +- **Ready for review** - understanding is sound; note anything to mention to |
| 121 | + reviewers proactively. |
| 122 | +- **Study first** - list the specific files/concepts to re-read before requesting |
| 123 | + review, each as a clickable `path:line`. Be concrete: "re-read the null handling |
| 124 | + in X before you send this out." |
| 125 | +Keep it honest - if they'd get grilled in review on something, say so now. |
| 126 | + |
| 127 | +### 7. If `--save` |
| 128 | +Write `pr-quiz/<branch>-scorecard.md`: the questions, their answers, your grades |
| 129 | +and model answers, the score, and the verdict. Don't commit it unless asked. |
| 130 | + |
| 131 | +## Principles |
| 132 | +- **The author is the examinee, not the collaborator.** During the quiz you withhold |
| 133 | + answers; you're measuring them, not helping them pass. |
| 134 | +- **A failed quiz is a successful outcome** - it caught a gap before a human's time |
| 135 | + was spent. Frame it that way, not as a scolding. |
| 136 | +- **True to the code.** Every question, answer, and grade traces to a line you read. |
| 137 | +- **Terse and direct** in chat - the questions and the verdict, minimal preamble. |
0 commit comments