Skip to content

CLAUDE.md

Latest commit

 

History

History
146 lines (133 loc) · 12.2 KB

CLAUDE.md

File metadata and controls

146 lines (133 loc) · 12.2 KB
Raw

cve-reverser — agent guide

Reverses publicly disclosed WordPress plugin CVEs from their patches into a local-lab PoC + a Nuclei detection template, via an rlm-kit RLM harness. A downstream consumer of rlm-kit (git source, commit-pinned in uv.lock) — it follows a set of hard-won conventions on purpose (don't re-derive them worse). See README.md for the roles/pipeline and rlm-kit's "Building a consumer" for the extension contract (base/wrap, the promotion rule, the rollout-vs-reward stage boundary) this project lives within.

One companion rule ships under .claude/rules/:

  • @.claude/rules/handoff.md — what must survive context compaction. Read it before auto-compacting.

Verify

  • uv run python -m pytest — pipeline + wiring tests. No live model, no network, no Deno. dspy-bearing tests use a DummyLM / skip if dspy is absent. (uv sync pulls rlm-kit from git first; add uv run pytest studio/tests for the studio console.)
  • A live run needs planner/lifeline creds (CR_*), a generator endpoint (self backend), and a Deno sandbox. The nuclei binary is optional (host-side authoritative validation only).

Running — always through the CLI

  • Run via cli (one / today), never an ad-hoc script. cli.run(lead, …) is THE programmatic entry. The DELIVERABLE for one CVE is a per-CVE folder (the video author's layout) <outdir>/<YEAR>/<CVE_ID>/: README.md (analysis), poc.md, report.md, nuclei.yaml, changes.diff, and our run-report.md (the trajectory). Written by render.cve_folder_files. The traceability core traces/{id}.jsonl (reset per run — TraceRecorder appends, so a re-run drops the stale file first) stays FLAT so export "traces/*.jsonl" globs it, alongside sources/{id}.txt + responses/{id}.json. Don't drive reverse_from_lead / render_* from a private script — extend cli.py. Re-render/re-serialize a past run offline: python -m cve_reverser render|response traces/{id}.jsonl {id}.
  • today dedups by existing folder + stops on empty. It lists Patchstack, skips any CVE whose <YEAR>/<CVE_ID>/ folder exists (already reversed), and prints "nothing new qualified" rather than inventing work — the author's behavior, so cron re-runs are safe.

Hard invariants — do not break

  • The generator TOOL is the only producer of template YAML — and it is a SEAM. The planner and the lifeline never write or fix YAML. All YAML comes from generate_nuclei_template. This is STRUCTURAL: the planner's SUBMIT type (PlannerResult / TemplateJudgement) has NO yaml/valid/errors field, so a planner cannot re-type (and mangle) YAML into it. It references the template by template_id; render.assemble_result attaches the generator's VERBATIM yaml (last ok per id) + DERIVES valid/errors from postprocess. Assembly is the read-time step BOTH the CLI (live) and result_from_events (re-render) and rl_export._assembled_template run, so labels are FACTS not the policy's self-report. The seam: generator_tool.make_template_tool picks a backend by config.template_backend — "self" (a general model self-authors, via rlm-kit make_model_tool) NOW, "harness" (delegate to a mature downstream harness via a future make_harness_tool) LATER. Keep template authoring routed through this ONE chat_fn so the swap stays localized; never add a second YAML path. info.author is the SOLE field code owns. The "self" backend is a GENERAL, untrained model: it emits author: nuclei from its training, so relying on it to "default" the author (an assumption that holds only for a fine-tuned generator, i.e. the future harness backend) is wrong here. The generator PROMPT asks for the canonical author AND postprocess._stamp_author stamps it deterministically — changing only that one line, leaving every other field the model's verbatim output. This is the one documented exception to "the generator is the sole author"; do NOT extend code-side rewriting to any other field (that would desync the trace from the saved template).
  • Models are ROLES, configured by env (CR_ROOT_LM / CR_SUB_LM / CR_GENERATOR_LM). Refer to them by role in code, docs, and the prompt. No hardcoded model name.
  • The iteration budget is HARD — max_retries=1, no whole-RLM retry (set in reverse.setup). One task = one trajectory, so the trace stays valid training data. An RLMTaskError is almost always infra (a planner-endpoint hiccup / a JSONAdapter parse failure), NOT a PlannerResult coercion problem — grep the server logs for the real cause; don't reach for a schema fix.
  • No premature give-up. A generator "NEEDS MORE INFO"/decline/INVALID is ALWAYS recoverable. The planner may not finalize an invalid/incomplete template while budget remains unless, for the blocking gap, it has re-called the generator OR escalated to llm_query at least once. A spec is NOT a template — you may not finalize until the generator returned a format-VALID template for your template_id once. This is a reverse.py INSTRUCTIONS hard rule (a prompt/policy rule that makes rollouts better, NOT a reward); the same fact is surfaced as the objective rl_export.gave_up_early metric.
  • Format validation is pure-Python and in-loop; nuclei -validate is host-side and post-run only. postprocess (extract → parse → structural + gates) runs inside the sandboxed tool. NEVER shell out to nuclei -validate from inside the live dspy.RLM/asyncio process — it hangs (a hard-won lesson); nuclei_binary.py runs it only host-side from the CLI, as an authoritative addendum. Never execute or scan a target.
  • The feed source is FIXED: Patchstack, parsed from its page — no provider knobs. feed.py fetches the Patchstack database page and resolves its Nuxt (devalue) payload into leads; selection is the author's fixed house-rules (plugins only, CVSS ≥ 7, last ~24h, non-admin, highest-CVSS-first, cap 10). Do NOT re-introduce a pluggable-feed abstraction or a Wordfence path — that flexibility was cut on purpose. The parser is validated against a REAL captured payload (tests/fixtures/patchstack_payload.json); if Patchstack changes its serialization, fix the resolver against a fresh capture — never hand-write a parser from assumption (the Wordfence-410 lesson). A lead with no CVE gets a <slug>-<version> id.
  • A CVE with no official patch is still in scope. When the lead has no fixed version (plugin closed/abandoned), the planner reverses from the vulnerable source directly, sets analysis.patch_available=False, and makes patch_summary a clearly-labelled SYNTHESIZED fix (rendered into changes.diff with a "not an upstream diff" header). Never invent a fixed version that doesn't exist.
  • Publishing is a HOST-SIDE deterministic step, NOT an agentic tool. Committing the assembled per-CVE folder to GitHub (publish.py, opt-in via CR_GITHUB_REPO+CR_GITHUB_TOKEN and --publish) runs AFTER the run, straight against the GitHub REST API — the planner never gets write credentials and the commit stays OUT of the trajectory. This keeps the sub-LM/tool split honest: only choices the policy MAKES are tools; writing a finished deliverable is plumbing. Do NOT wire publishing back as a planner tool (that was the old Zapier-MCP design; it was removed). A publish failure never sinks a completed run (it's reported; the local folder is the source of truth). For a CVSS ≥ 9.0 unauth bug it also opens a [CRITICAL] issue.
  • The lifeline never touches bulk context. It receives only a short distilled question — never the raw lead, never fetched source. Bulk is chewed by the cheap planner (REPL) and the generator. fetch_url (fetch_tool.py) is SSRF-guarded (is_safe_url + a resolved-IP re-check on the direct provider, since URLs come from untrusted feed/advisory content); a successful fetch feeds the EvidencePack. A host behind a fake-IP proxy / split-DNS VPN (Clash/Mihomo/Surge map every host into the reserved 198.18.0.0/16) would otherwise have EVERY fetch refused as "internal"; CR_FETCH_ALLOW_CIDRS (fetch_allow_cidrs) opt-in carves out ONLY the operator's proxy range at the resolved-IP layer — the syntactic is_safe_url still refuses localhost/metadata. Empty by default (full strictness).
  • Grounded analysis, judgement-only truth. The source→sink the planner reports MUST correspond to the diffed lines (patch) or the located vulnerable lines (no patch) it holds in a REPL variable. matches_patch / detection_tier are the planner's judgement; valid is postprocess of the generator's bytes. The CVE-relevance gate (postprocess.lead_*_cve_mismatch) catches a template/id that "corrected" a post-cutoff CVE — the lead's identifier is ground truth. The run_start meta self-describes the run's CONFIG (roles, budgets, author, lead_cve, the lead + instructions as initial state) so offline readers (assemble_result, rl_export) read real per-run values and per-turn SFT has its first input.
  • This is a ROLLOUT source — trajectories, NOT reward. rl_export is reward-free (reward=None to rlm-kit's exporters); it emits the generator/orchestrator splits + sft_turns + per-run LABELS (the SFT keep-filter complete and valid) + objective METRICS. Reward/credit-assignment/GRPO are a separate fine-tuning project. A prompt/policy rule that improves rollout quality is in scope; a reward is not. The generator-vs-everything split is load-bearing (the generator never sees skills/fetch) — any new planner tool defaults to orchestrator data.
  • A planner REFUSAL on an authorized source is MODEL ALIGNMENT — fix it with the MISSION FRAME, never a safety guard. reverse.py INSTRUCTIONS carry the MISSION frame at the top (authorized DEFENSIVE research, never-execute, reframe an offensively-worded lead into the research objective). If a safety-tuned planner still balks, the lever is a less safety-tuned planner — never remove the SSRF / never-execute guards.
  • Reversing knowledge lives in cve_reverser/skills/, not the prompt. Progressive disclosure (load_skills_as_tools(discovery="inject")): the catalog is injected at startup, read_skill(name) pulls a body JIT. Fix a wrong convention by editing/adding a skill, NOT by bloating reverse.py INSTRUCTIONS (which holds identity + the MISSION frame + the tool cost-model + the loop skeleton + the output-gating hard rules). Skills are force-included in the wheel (pyproject.toml).
  • Keep the dspy-free modules dspy-free. config.py, schema.py, postprocess.py, feed.py, render.py, response.py, rl_export.py, evidence.py, fetch_tool.py, nuclei_binary.py must not import dspy/rlm-kit runtime at module top. import cve_reverser must not import dspy (ReverseCVE / setup / run are lazy in __init__).
  • The studio is an in-repo WORKSPACE MEMBER, never in the harness wheel. studio/ (the visual console, package cve-reverser-studio) is a uv workspace member with its OWN pyproject.toml; it reads this package's trace / ReversalResponse contract, so co-locating keeps reader + producer in sync. Its web stack (fastapi/uvicorn/frontend) MUST stay behind that package boundary + the live OPTIONAL extra — never fold it into cve_reverser's core dependencies, and keep the root wheel packages = ["cve_reverser"] so uv build never sweeps studio/ in. A harness-only install (pip install cve-reverser, for a training/rollout pipeline) must stay web-free. The console reaches the harness ONLY via cve_reverser's public surface (cli.run, feed, response), never by forking it.

Layout / promotion

  • The generic "model-tool + validate + circuit-breaker" core is rlm-kit's make_model_tool; generator_tool is the project wrapper (backend + nuclei postprocess + messages + tracing).
  • When the harness backend ships, promote the generic "wrap another rlm-kit harness as a tool" shape into rlm-kit as make_harness_tool (the generic half in the kit; the downstream-harness endpoint/wiring here) — the same base/wrap split as make_model_tool. Until then _harness_chat is a documented stub.

Versioning

  • Keep pyproject.toml [project].version and cve_reverser.__version__ in sync.