|
| 1 | +# Adjudicator |
| 2 | + |
| 3 | +The Adjudicator is an optional pass that runs **between the deterministic analyzers and the LLM analyzer**. For each deterministic HIGH/CRITICAL finding it asks an LLM whether the file around the matched line actually contains the threat the rule was designed to catch, or whether the regex fired on benign content. Findings the LLM identifies as literal-regex false positives are demoted to `INFO` for downstream verdict computation. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +When enabled via the `--adjudicate` CLI flag (or `ScanPolicy.adjudicator.enabled = True` on the API), the Adjudicator performs: |
| 8 | + |
| 9 | +- **False positive demotion**: Uses an LLM to reason about whether a deterministic HIGH/CRITICAL match represents a real instance of the threat, or a coincidental regex hit on benign prose. |
| 10 | +- **Cascade prevention**: Because it runs before the LLM analyzer, demoted findings never enter the LLM analyzer's static-finding enrichment context — so a wrong deterministic HIGH cannot be amplified into LLM findings citing the same pattern hit. |
| 11 | +- **Audit trail**: Every finding it considers is recorded in `scan_metadata.adjudicator.audit` with the LLM's verdict, confidence, and reason. |
| 12 | + |
| 13 | +The adjudicator only touches deterministic findings (static, pipeline, behavioral, bytecode, yara analyzers) at HIGH or CRITICAL severity. LLM and other advisory findings are outside its scope. |
| 14 | + |
| 15 | +## Safety property |
| 16 | + |
| 17 | +**The Adjudicator is demote-only.** It can lower a finding's severity to `INFO`, and can never raise a finding's severity. |
| 18 | + |
| 19 | +The demote-only constraint bounds the failure surface. On **error paths** — LLM unavailable, timeout, malformed JSON, unexpected verdict, out-of-range confidence, path-escape attempt — the finding stays at its original severity and no metadata is written. Those specific paths cannot introduce false negatives. |
| 20 | + |
| 21 | +**A wrong `false_positive` verdict from the LLM itself can still demote a real threat.** That is a genuine failure mode and is why the pass is off by default, why the confidence threshold is configurable, and why every demotion is preserved in `finding.metadata["adjudication"]` for review and override. Adversarial content in the scanned skill can also attempt to bias the adjudicator (see the system-prompt hardening in the implementation) — again, the worst case is a real finding demoted to `INFO`, but this is a real (not zero-probability) risk that operators should weigh when enabling the pass. |
| 22 | + |
| 23 | +## How It Works |
| 24 | + |
| 25 | +1. **Deterministic analyzers run first** (static, pipeline, behavioral, bytecode, yara). This is unchanged. |
| 26 | +2. **Adjudicator inspects HIGH/CRITICAL findings from those analyzers.** For each finding it: |
| 27 | + - Extracts the matched line and a wide surrounding context (whole file for files ≤ 600 lines, ± 25 lines otherwise). The wide window makes the adjudicator resilient to the scanner's occasional off-by-N line-number reports on markdown content. |
| 28 | + - Pulls the rule's `description`, `category`, and `default_severity` from the rule registry. |
| 29 | + - Sends both to the LLM with a fixed prompt asking `real` vs `false_positive`, a 1–5 confidence, and a one-sentence reason. |
| 30 | +3. **Demotes on high-confidence FP verdicts.** If the LLM returns `verdict = "false_positive"` and `confidence >= min_fp_confidence` (default 3), the finding's severity is lowered to `INFO`. The original severity is preserved in `finding.metadata["adjudication"]["original_severity"]` for audit. |
| 31 | +4. **LLM analyzer runs next** (if enabled). Its `static_findings_summary` enrichment now excludes demoted findings, so the LLM analyzer cannot cross-confirm a false-positive deterministic HIGH. |
| 32 | +5. **The rest of the pipeline is unchanged**: severity overrides, disabled rules, analyzability, deduplication, policy fingerprinting. |
| 33 | + |
| 34 | +## CLI Usage |
| 35 | + |
| 36 | +```bash |
| 37 | +# Basic — deterministic analyzers only, with adjudicator active |
| 38 | +skill-scanner scan /path/to/skill --adjudicate |
| 39 | + |
| 40 | +# Full stack — adjudicator gates the LLM analyzer's static enrichment |
| 41 | +skill-scanner scan /path/to/skill --use-llm --adjudicate |
| 42 | + |
| 43 | +# Full stack with meta-analysis |
| 44 | +skill-scanner scan /path/to/skill --use-llm --enable-meta --adjudicate |
| 45 | +``` |
| 46 | + |
| 47 | +**Requirements:** |
| 48 | + |
| 49 | +- An LLM model must be configured via `SKILL_SCANNER_LLM_MODEL` (or `SKILL_SCANNER_ADJUDICATOR_LLM_MODEL` to override for the adjudicator specifically). |
| 50 | +- API key via `SKILL_SCANNER_LLM_API_KEY` for providers that need one; AWS credentials for `bedrock/...` models. |
| 51 | +- LiteLLM must be installed. |
| 52 | + |
| 53 | +If any of the above are missing, the adjudicator logs a debug message and skips every finding — the scan behaves identically to a run with `--adjudicate` off. This is intentional: unavailability is not an error, it's a no-op. |
| 54 | + |
| 55 | +## Configuration |
| 56 | + |
| 57 | +The Adjudicator is configured via `AdjudicatorPolicy` on the `ScanPolicy`: |
| 58 | + |
| 59 | +```yaml |
| 60 | +adjudicator: |
| 61 | + enabled: false # master toggle; --adjudicate CLI flag also sets this |
| 62 | + min_fp_confidence: 3 # 1-5; LLM confidence required to demote (default 3) |
| 63 | +``` |
| 64 | +
|
| 65 | +Environment variables (in order of precedence): |
| 66 | +
|
| 67 | +- `SKILL_SCANNER_ADJUDICATOR_LLM_MODEL` — model override specific to the adjudicator |
| 68 | +- `SKILL_SCANNER_ADJUDICATOR_LLM_TEMPERATURE` — temperature override, or `"none"` to omit |
| 69 | +- `SKILL_SCANNER_LLM_MODEL` — fallback if the adjudicator-specific var is unset |
| 70 | +- `SKILL_SCANNER_LLM_TEMPERATURE` — fallback if the adjudicator-specific var is unset |
| 71 | + |
| 72 | +## Output |
| 73 | + |
| 74 | +Demoted findings appear in the final report with: |
| 75 | + |
| 76 | +- `severity: "INFO"` (the effective severity used for verdict computation) |
| 77 | +- `metadata.adjudication`: |
| 78 | + - `original_severity`: what it was before demotion (e.g. `"HIGH"`) |
| 79 | + - `verdict`: `"false_positive"` |
| 80 | + - `confidence`: `1-5` |
| 81 | + - `reason`: one-sentence rationale from the LLM |
| 82 | + - `demoted_to`: `"INFO"` |
| 83 | + - `model_id`: which model made the decision |
| 84 | + |
| 85 | +The scan's `scan_metadata.adjudicator` section summarizes the pass: |
| 86 | + |
| 87 | +```json |
| 88 | +{ |
| 89 | + "adjudicator": { |
| 90 | + "considered": 4, |
| 91 | + "demoted": 1, |
| 92 | + "audit": [ |
| 93 | + { |
| 94 | + "rule_id": "PROMPT_INJECTION_CONCEALMENT", |
| 95 | + "verdict": "false_positive", |
| 96 | + "confidence": 5, |
| 97 | + "reason": "The phrase 'do not notify the user' refers to a routine pre-flight column-update succeeding silently; the write itself is visible.", |
| 98 | + "demoted_to": "INFO", |
| 99 | + "model_id": "bedrock/converse/anthropic.claude-opus-4-8-20240229-v1:0" |
| 100 | + }, |
| 101 | + ... |
| 102 | + ] |
| 103 | + } |
| 104 | +} |
| 105 | +``` |
| 106 | + |
| 107 | +## Cost |
| 108 | + |
| 109 | +Per skill: 0–3 LLM calls × ~200 input tokens + ~50 output tokens ≈ $0.005 on Opus 4.x. Negligible relative to the existing LLM analyzer + meta-analyzer cost. |
| 110 | + |
| 111 | +## Concurrency |
| 112 | + |
| 113 | +The adjudicator uses a module-level lock to serialize its LLM calls across parallel workers. This prevents adjudicator calls from competing with the main LLM analyzer's calls for backend rate limits. Because adjudicator calls are short (~250 output tokens) and rare (0–3 per skill), the serialization overhead is at most a few seconds per scan and eliminates a class of transient 5xx-induced regressions. |
| 114 | + |
| 115 | +## When to enable |
| 116 | + |
| 117 | +- **Yes**: when your policy treats deterministic HIGH+ as auto-reject and false positives on that class of finding are causing real friction for reviewers. |
| 118 | +- **Yes**: when you're running the LLM analyzer and observing findings that cite deterministic pattern hits (the "confirmation cascade" failure mode). |
| 119 | +- **Maybe**: for CI gating where you accept a small per-scan LLM cost in exchange for fewer benign auto-rejects. |
| 120 | +- **No**: for pure deterministic gates that never enable LLM analysis anyway — the adjudicator has nothing to demote that would matter. |
| 121 | + |
| 122 | +## Relation to other analyzers |
| 123 | + |
| 124 | +- **Adjudicator vs. Meta-analyzer**: they solve different problems and can be enabled together. The adjudicator runs *before* the LLM analyzer and demotes deterministic false positives at their source. The meta-analyzer runs *after* all analyzers and re-scores or correlates the full finding set. Both can be on simultaneously; they don't conflict. |
| 125 | +- **Adjudicator vs. `--llm-consensus-runs`**: consensus reduces run-to-run flap on LLM findings by voting across N runs of the LLM analyzer. Consensus does not affect deterministic findings and does not address the cross-analyzer confirmation cascade. The adjudicator addresses a different failure mode and is complementary. |
0 commit comments