fix(scanner): include adjudicator token usage

[vineethsai7]

What changed

• track token usage for every successful adjudicator LiteLLM completion
• include adjudicator input, output, and total tokens in ScanResult.llm_usage
• preserve billed usage when an adjudicator response cannot be parsed or later processing fails
• document that aggregate usage includes analyzer, meta-analysis, and adjudication calls

Why

PR #146 aggregated the regular LLM and meta-analyzer token usage, but PR #139's adjudicator calls LiteLLM directly and its usage was not included. Scans with --adjudicate therefore under-reported total LLM consumption.

Validation

• 79 focused tests passed across adjudicator, CLI usage, request-handler, and model serialization suites
• post-rebase adjudicator and CLI integration tests passed
• Ruff and repository hooks passed
• CodeRabbit CLI raised 0 issues

Summary by CodeRabbit

• New Features
  • Added token-usage tracking for adjudication calls.
  • Scan-level LLM usage totals now include adjudication input, output, and total tokens, including calls that return malformed results or encounter errors.
• Documentation
  • Updated architecture and JSON output documentation to describe adjudication usage aggregation and reporting conditions.
• Tests
  • Added coverage for token accumulation and combined analyzer and adjudicator usage totals.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fb3334b2-dd5d-432d-8cfa-2e39358c8160

📥 Commits

Reviewing files that changed from the base of the PR and between 6155f06 and cdb5614.

📒 Files selected for processing (1)

• docs/reference/output-formats.md

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/reference/output-formats.md

---

📝 Walkthrough

Walkthrough

The Adjudicator now tracks LLM token usage for each run. SkillScanner adds this usage to aggregate scan totals, including failed adjudication calls. Documentation and tests cover the updated accounting.

Changes

Adjudicator LLM usage aggregation

Layer / File(s)	Summary
Adjudicator usage tracking and validation skill_scanner/core/analyzers/adjudicator.py, tests/test_adjudicator.py	The Adjudicator resets and accumulates token usage for each run. Usage is recorded before response parsing, including malformed responses. Tests cover accumulation and scanner aggregation.
Scan integration and output documentation skill_scanner/core/scanner.py, docs/architecture/analyzers/adjudicator.md, docs/reference/output-formats.md	SkillScanner records Adjudicator usage in a finally block and adds it to aggregate scan totals. Documentation describes adjudication usage in llm_usage output.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SkillScanner
  participant Adjudicator
  participant LLMProvider
  participant ScanResult
  SkillScanner->>Adjudicator: run adjudication
  Adjudicator->>LLMProvider: request adjudication
  LLMProvider-->>Adjudicator: response with token usage
  Adjudicator-->>SkillScanner: return or raise
  SkillScanner->>SkillScanner: add usage in finally
  SkillScanner->>ScanResult: emit aggregate llm_usage

Loading

Possibly related issues

• feat: expose LLM token usage in ScanResult JSON output #136 — Tracks and aggregates LLM token usage for adjudicator and analyzer calls.

Possibly related PRs

• cisco-ai-defense/skill-scanner#146 — Adds related LLM token-usage aggregation that this PR extends to adjudicator calls.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: including adjudicator token usage in scanner totals.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/include-adjudicator-token-usage

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/output-formats.md`:
- Line 114: Correct the llm_usage documentation to state that the field is
included only when non-zero provider usage is reported, rather than whenever an
LLM call is made; alternatively, update the scanner’s usage aggregation to track
executed calls and preserve the documented call-based presence condition.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e2b51637-1a0b-4c64-a222-4afcc7b1ca24

📥 Commits

Reviewing files that changed from the base of the PR and between 5dca0c5 and 6155f06.

📒 Files selected for processing (5)

• docs/architecture/analyzers/adjudicator.md
• docs/reference/output-formats.md
• skill_scanner/core/analyzers/adjudicator.py
• skill_scanner/core/scanner.py
• tests/test_adjudicator.py