Skip to content

Commit 05e5e37

Browse files
amber-beasley-liatriorisaweojacquesgyrospectrevineethsai7
authored
feat(scanner): expose LLM token usage in ScanResult JSON output (#146)
* feat(adjudicator): demote literal-regex false positives on deterministic HIGH+ findings Introduces an optional per-finding adjudicator that runs between the deterministic analyzers (static / pipeline / behavioral / bytecode / yara) and the LLM analyzer. For each deterministic HIGH or CRITICAL finding it asks the configured LLM whether the file around the matched line actually contains the threat the rule was designed to catch, or whether the regex fired on benign content. Findings identified as literal-regex false positives (verdict=false_positive, confidence>=threshold) are demoted to INFO for downstream verdict computation. The original severity and the LLM's reasoning are preserved in the finding's metadata for audit. Because the adjudicator runs before the LLM analyzer, demoted findings do not enter the LLM analyzer's static-finding enrichment context — so a wrong deterministic HIGH cannot be amplified into further LLM findings citing the same pattern hit. This addresses the confirmation-cascade failure mode described in the linked issue without any changes to llm_analyzer.py. Safety property (load-bearing): the adjudicator can only demote findings, never promote them. LLM errors, timeouts, malformed output, or unexpected verdicts all leave the finding at its original severity. Enabling this pass cannot introduce false negatives. Off by default (--adjudicate CLI flag, ScanPolicy.adjudicator.enabled). Reuses the same env-var conventions as the LLM analyzer for model and temperature configuration, so Claude 4.x on Bedrock and OpenAI o1-series work out of the box. Includes: - New skill_scanner/core/analyzers/adjudicator.py (self-contained module) - AdjudicatorPolicy in ScanPolicy - --adjudicate CLI flag on both scan entry points - Integration in scanner.py between Phase 1 and Phase 2 analyzers - Audit records surfaced in ScanResult.scan_metadata.adjudicator - 11 pytest tests covering the load-bearing safety cases - docs/architecture/analyzers/adjudicator.md * style(adjudicator): ruff --fix + ruff format for pre-commit compliance * fix(adjudicator): address CodeRabbit review — path containment, YAML round-trip, override precedence, prompt hardening Six review-driven fixes plus a soften-the-language docs pass. 1. **Path containment** (adjudicator.py:_adjudicate_one) — resolve ``skill_dir / file_path`` and reject anything that lands outside the skill directory. Absolute paths and ``..`` traversal now short-circuit to a "skipped" AdjudicationResult before any file is read. 2. **YAML round-trip for AdjudicatorPolicy** (scan_policy.py:_from_dict and _to_dict) — custom policy files with an ``adjudicator:`` section were silently discarded on load and never emitted on ``to_yaml``. Wired through following the AnalyzersPolicy / LLMAnalysisPolicy pattern already in the file. 3. **--adjudicate on scan-repo** (cli.py:scan_repo_command) — the flag was defined in _add_common_scan_flags but scan_repo_command didn't read it, so ``skill-scanner scan-repo <url> --adjudicate`` silently did nothing. Mirrors the toggle already present on scan_command and scan_all_command. 4. **Confidence range enforcement** (adjudicator.py:_adjudicate_one) — the response contract is confidence 1-5, but the code accepted any integer. A malformed ``{"verdict":"false_positive","confidence":999}`` would demote past any min_fp_confidence threshold. Now validated and fails closed on out-of-range values (including 0). 5. **Preserve adjudicator demotion through severity_overrides** (scanner.py:_apply_severity_overrides) — a policy severity-override entry could raise a demoted finding back to HIGH/CRITICAL, defeating the demote-to-INFO contract. _apply_severity_overrides now skips findings that carry ``metadata['adjudication']['demoted_to']``. 6. **Prompt-injection hardening** (adjudicator.py:_call_llm and new _SYSTEM_PROMPT constant) — scanned file content is untrusted evidence. Split the request into a role=system rubric (trusted) and a role=user payload (rubric + untrusted evidence), with the system prompt explicitly instructing the model to ignore any instructions embedded in the file content. Defense in depth: the demote-only invariant already bounds the blast radius, but keeping trusted rubric separate from untrusted evidence is standard practice for LLM-as-judge flows. 7. **Docs / docstring softening** — replaced "cannot introduce false negatives" with a more accurate statement: error paths (LLM unavailable, malformed output, out-of-range confidence, path escape) preserve severity, but a wrong ``false_positive`` verdict from the LLM itself can still demote a real threat. That's why the pass is off by default and every demotion is preserved in metadata for review. New tests (all 16 pass): - test_out_of_range_confidence_keeps_original_severity - test_zero_confidence_keeps_original_severity - test_absolute_path_outside_skill_dir_is_skipped - test_parent_traversal_is_skipped - test_llm_receives_system_prompt Ruff-clean, format-clean. * feat: add trusted_reference_domains to LLM analysis policy Add a policy-driven mechanism to demote LLM findings (transitive trust, supply chain) to LOW when all referenced domains are declared as trusted in the scan policy. This addresses systematic false positives for organizations hosting skills and documentation on self-managed GitLab/GitHub instances, where the LLM analyzer flags references to internal repos as external threats. Changes: - LLMAnalysisPolicy: new trusted_reference_domains field (set[str]) - llm_analyzer._convert_to_findings: demotion logic after existing is_internal_file_reading filter - Two helpers: _references_only_trusted_domains (URL extraction) and _mentions_only_trusted_domains (plain-text domain detection) - default_policy.yaml: empty default list with documentation The mechanism mirrors known_installer_domains: findings are demoted to LOW (maintaining visibility) rather than suppressed entirely. * fix(llm): allow Vertex AI to use ambient Application Default Credentials ProviderConfig.validate() required a truthy credential for every provider except Bedrock and Ollama, and the only credential source it checked for Vertex was GOOGLE_APPLICATION_CREDENTIALS. This blocked ambient auth via a GCE/Cloud Run attached service account or Workload Identity, even though LiteLLM/google-auth already fall back to it automatically when no explicit credential is passed -- the same pattern already supported for Bedrock's IAM role. Excludes is_vertex from the check, mirroring the Bedrock/Ollama precedent, and documents the fallback. * Bumps to resolve security findings * feat(scanner): expose LLM token usage in ScanResult JSON output (#136) LiteLLM and the Google GenAI SDK both return prompt/completion token counts on every response, but the scanner discarded them after pulling out the text content, leaving downstream pipelines with no way to attribute LLM call cost to a scan without monkey-patching provider internals. Extract token usage at the provider-response boundary for both LiteLLM and Google SDK request paths, accumulate it per analyzer (covering multi-call cases like consensus-judging runs and meta-analysis follow-up passes), aggregate across all LLM analyzers in SkillScanner, and fold in MetaAnalyzer's spend separately since it runs as a post-processing step outside the scanner's own aggregation. Surfaces as an additive `llm_usage` field on ScanResult.to_dict(), omitted entirely on static-only scans. * fix(llm): don't leak Vertex ADC credential path into GEMINI_API_KEY - _resolve_api_key() now returns None for Vertex instead of the GOOGLE_APPLICATION_CREDENTIALS path, since vertex_ai/gemini-* models set both is_vertex and is_gemini, which was causing the file path to be written into GEMINI_API_KEY. - Regenerated configuration-reference.md via generate_reference_docs.py instead of hand-editing, and updated the underlying descriptions so the doc doesn't drift on next regeneration. Addresses CodeRabbit review feedbak on #144. * revert unrelated cli-command-reference.md regeneration * fix: address CodeRabbit review comments on PR #146 - Fix inconsistent JSON example: analyzers_used now includes llm_analyzer/meta_analyzer to match the shown non-empty llm_usage block. - Expose ScanResult.llm_usage through the /scan API response: ScanResponse was missing the field, so single-scan (and /scan-upload, which delegates to it) API clients never received token usage even though the scanner computed it. Batch scan was unaffected since it serializes via to_dict(). - Add regression tests covering both the combined analyzer+meta-analyzer usage case and the omitted-when-disabled case. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(llm): scope trusted-domain severity demotion --------- Co-authored-by: Rohan Isawe <rohan.isawe@smartsheet.com> Co-authored-by: Olivier Jacques <ojacques2@gmail.com> Co-authored-by: gyrospectre <7224858+gyrospectre@users.noreply.github.com> Co-authored-by: Vineeth Sai Narajala <vnarajal@cisco.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 parent d76b6c1 commit 05e5e37

14 files changed

Lines changed: 779 additions & 8 deletions

docs/reference/output-formats.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,17 +95,24 @@ skill-scanner scan evals/skills/data-exfiltration/environment-secrets --format j
9595
],
9696
"scan_duration_seconds": 0.13,
9797
"duration_ms": 127,
98-
"analyzers_used": ["static_analyzer", "bytecode", "pipeline"],
98+
"analyzers_used": ["static_analyzer", "bytecode", "pipeline", "llm_analyzer", "meta_analyzer"],
9999
"timestamp": "2026-02-19T21:58:33.032573",
100100
"scan_metadata": {
101101
"policy_name": "default",
102102
"policy_version": "1.0",
103103
"policy_preset_base": "balanced",
104104
"policy_fingerprint_sha256": "45b486..."
105+
},
106+
"llm_usage": {
107+
"input_tokens": 5312,
108+
"output_tokens": 842,
109+
"total_tokens": 6154
105110
}
106111
}
107112
```
108113

114+
`llm_usage` is only present when at least one LLM call was made (`--use-llm` and/or `--enable-meta`), aggregated across every LLM analyzer and meta-analysis call for the scan. It's omitted entirely on static-only scans.
115+
109116
Use `--compact` to remove pretty-printing for machine pipelines.
110117

111118
### Table

skill_scanner/api/router.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,14 +97,20 @@
9797

9898
MetaAnalyzer: type | None
9999
apply_meta_analysis_to_results: Callable[..., list] | None
100+
merge_meta_analyzer_usage: Callable[..., None] | None
100101
try:
101-
from ..core.analyzers.meta_analyzer import MetaAnalyzer, apply_meta_analysis_to_results
102+
from ..core.analyzers.meta_analyzer import (
103+
MetaAnalyzer,
104+
apply_meta_analysis_to_results,
105+
merge_meta_analyzer_usage,
106+
)
102107

103108
META_AVAILABLE = True
104109
except (ImportError, ModuleNotFoundError):
105110
META_AVAILABLE = False
106111
MetaAnalyzer = None
107112
apply_meta_analysis_to_results = None
113+
merge_meta_analyzer_usage = None
108114

109115
router = APIRouter()
110116

@@ -215,6 +221,7 @@ class ScanResponse(BaseModel):
215221
scan_duration_seconds: float
216222
timestamp: str
217223
findings: list[dict]
224+
llm_usage: dict[str, int] | None = None
218225

219226

220227
class HealthResponse(BaseModel):
@@ -456,6 +463,8 @@ def run_scan():
456463
)
457464
result.findings = filtered_findings
458465
result.analyzers_used.append("meta_analyzer")
466+
if merge_meta_analyzer_usage is not None:
467+
merge_meta_analyzer_usage(result, meta_analyzer)
459468
except Exception as meta_error:
460469
logger.warning("Meta-analysis failed: %s", meta_error)
461470

@@ -469,6 +478,7 @@ def run_scan():
469478
scan_duration_seconds=result.scan_duration_seconds,
470479
timestamp=result.timestamp.isoformat(),
471480
findings=[f.to_dict() for f in result.findings],
481+
llm_usage=result.llm_usage,
472482
)
473483

474484
except SkillLoadError as e:
@@ -712,6 +722,8 @@ async def _run_batch_meta(scanner_ref, report_ref, policy_ref):
712722
)
713723
result.findings = filtered_findings
714724
result.analyzers_used.append("meta_analyzer")
725+
if merge_meta_analyzer_usage is not None:
726+
merge_meta_analyzer_usage(result, meta_analyzer)
715727
except Exception:
716728
pass
717729

skill_scanner/cli/cli.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,14 +54,20 @@
5454
# Optional Meta analyzer
5555
MetaAnalyzer: type | None
5656
apply_meta_analysis_to_results: Callable[..., list] | None
57+
merge_meta_analyzer_usage: Callable[..., None] | None
5758
try:
58-
from ..core.analyzers.meta_analyzer import MetaAnalyzer, apply_meta_analysis_to_results
59+
from ..core.analyzers.meta_analyzer import (
60+
MetaAnalyzer,
61+
apply_meta_analysis_to_results,
62+
merge_meta_analyzer_usage,
63+
)
5964

6065
META_AVAILABLE = True
6166
except (ImportError, ModuleNotFoundError):
6267
META_AVAILABLE = False
6368
MetaAnalyzer = None
6469
apply_meta_analysis_to_results = None
70+
merge_meta_analyzer_usage = None
6571

6672
logger = logging.getLogger("skill_scanner.cli")
6773

@@ -411,6 +417,8 @@ def scan_command(args: argparse.Namespace) -> int:
411417
)
412418
result.findings = filtered
413419
result.analyzers_used.append("meta_analyzer")
420+
if merge_meta_analyzer_usage is not None:
421+
merge_meta_analyzer_usage(result, meta_analyzer)
414422

415423
# Surface meta-analysis insights into scan_metadata
416424
if result.scan_metadata is None:
@@ -524,6 +532,8 @@ def scan_all_command(args: argparse.Namespace) -> int:
524532
total_new += len(meta_result.missed_threats)
525533
result.findings = filtered
526534
result.analyzers_used.append("meta_analyzer")
535+
if merge_meta_analyzer_usage is not None:
536+
merge_meta_analyzer_usage(result, meta_analyzer)
527537

528538
# Surface meta-analysis insights
529539
if result.scan_metadata is None:
@@ -649,6 +659,8 @@ def scan_repo_command(args: argparse.Namespace) -> int:
649659
total_new += len(meta_result.missed_threats)
650660
result.findings = filtered
651661
result.analyzers_used.append("meta_analyzer")
662+
if merge_meta_analyzer_usage is not None:
663+
merge_meta_analyzer_usage(result, meta_analyzer)
652664

653665
# Surface meta-analysis insights
654666
if result.scan_metadata is None:

skill_scanner/core/analyzers/llm_analyzer.py

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,13 @@
4141
from .base import BaseAnalyzer
4242
from .llm_prompt_builder import PromptBuilder
4343
from .llm_provider_config import ProviderConfig
44-
from .llm_request_handler import _TEMPERATURE_UNSET, LLMRequestHandler
44+
from .llm_request_handler import (
45+
_TEMPERATURE_UNSET,
46+
LLMRequestHandler,
47+
LLMTokenUsage,
48+
_add_token_usage,
49+
_empty_token_usage,
50+
)
4551
from .llm_response_parser import ResponseParser
4652

4753
if TYPE_CHECKING:
@@ -254,6 +260,9 @@ def __init__(
254260
self.rate_limit_delay = rate_limit_delay
255261
self.timeout = timeout
256262

263+
# Cumulative token usage across all LLM calls in the most recent analyze() run.
264+
self._llm_usage: LLMTokenUsage = _empty_token_usage()
265+
257266
# Enriched context from other analyzers (set externally before analyze())
258267
self.enrichment_context: str | None = None
259268

@@ -263,6 +272,11 @@ def __init__(
263272
# Tracks the last analysis error (read by the scanner for analyzers_failed)
264273
self.last_error: str | None = None
265274

275+
@property
276+
def llm_usage(self) -> LLMTokenUsage:
277+
"""Cumulative token usage from the most recent analyze() run."""
278+
return dict(self._llm_usage) # type: ignore[return-value]
279+
266280
def set_enrichment_context(
267281
self,
268282
*,
@@ -327,6 +341,7 @@ async def analyze_async(self, skill: Skill) -> list[Finding]:
327341
Returns:
328342
List of security findings
329343
"""
344+
self._llm_usage = _empty_token_usage()
330345
findings = []
331346
budget_skipped: list[dict] = []
332347

@@ -450,6 +465,7 @@ async def analyze_async(self, skill: Skill) -> list[Finding]:
450465
response_content = await self.request_handler.make_request(
451466
messages, context=f"threat analysis for {skill.name}"
452467
)
468+
_add_token_usage(self._llm_usage, self.request_handler.last_usage)
453469
analysis_result = self.response_parser.parse(response_content)
454470
findings.extend(self._convert_to_findings(analysis_result, skill))
455471
else:
@@ -503,6 +519,7 @@ async def _consensus_analyze(self, messages: list[dict], skill: Skill) -> list[F
503519
response_content = await self.request_handler.make_request(
504520
messages, context=f"consensus run {run_idx + 1}/{self.consensus_runs} for {skill.name}"
505521
)
522+
_add_token_usage(self._llm_usage, self.request_handler.last_usage)
506523
analysis_result = self.response_parser.parse(response_content)
507524
run_findings = self._convert_to_findings(analysis_result, skill)
508525
all_run_findings.append(run_findings)

skill_scanner/core/analyzers/llm_request_handler.py

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,63 @@
2828
import os
2929
import warnings
3030
from pathlib import Path
31-
from typing import Any
31+
from typing import Any, TypedDict
3232

3333
from .llm_provider_config import ProviderConfig
3434

35+
36+
class LLMTokenUsage(TypedDict):
37+
"""Provider-normalized token counts for one or more LLM calls."""
38+
39+
input_tokens: int
40+
output_tokens: int
41+
total_tokens: int
42+
43+
44+
def _empty_token_usage() -> LLMTokenUsage:
45+
return {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}
46+
47+
48+
def _extract_token_usage(response: Any) -> LLMTokenUsage:
49+
"""Read token counts from a LiteLLM (or compatible) response object.
50+
51+
LiteLLM exposes usage as ``response.usage.prompt_tokens`` /
52+
``response.usage.completion_tokens``. Both fields are normalised to the
53+
``input_tokens`` / ``output_tokens`` names used in our output schema so
54+
callers never need to know which provider returned which key.
55+
"""
56+
usage = getattr(response, "usage", None)
57+
if usage is None:
58+
return _empty_token_usage()
59+
input_tokens = int(getattr(usage, "prompt_tokens", 0) or 0)
60+
output_tokens = int(getattr(usage, "completion_tokens", 0) or 0)
61+
total_tokens = int(getattr(usage, "total_tokens", 0) or input_tokens + output_tokens)
62+
return {"input_tokens": input_tokens, "output_tokens": output_tokens, "total_tokens": total_tokens}
63+
64+
65+
def _add_token_usage(total: LLMTokenUsage, delta: LLMTokenUsage) -> None:
66+
"""Accumulate *delta* into *total* in-place."""
67+
total["input_tokens"] += delta["input_tokens"]
68+
total["output_tokens"] += delta["output_tokens"]
69+
total["total_tokens"] += delta["total_tokens"]
70+
71+
72+
def _extract_google_sdk_token_usage(response: Any) -> LLMTokenUsage:
73+
"""Read token counts from a Google GenAI SDK ``GenerateContentResponse``.
74+
75+
The SDK exposes usage as ``response.usage_metadata.prompt_token_count`` /
76+
``candidates_token_count``, normalised here to the same ``input_tokens`` /
77+
``output_tokens`` names ``_extract_token_usage`` produces for LiteLLM.
78+
"""
79+
usage = getattr(response, "usage_metadata", None)
80+
if usage is None:
81+
return _empty_token_usage()
82+
input_tokens = int(getattr(usage, "prompt_token_count", 0) or 0)
83+
output_tokens = int(getattr(usage, "candidates_token_count", 0) or 0)
84+
total_tokens = int(getattr(usage, "total_token_count", 0) or input_tokens + output_tokens)
85+
return {"input_tokens": input_tokens, "output_tokens": output_tokens, "total_tokens": total_tokens}
86+
87+
3588
logger = logging.getLogger(__name__)
3689

3790
acompletion: Any
@@ -151,6 +204,14 @@ def __init__(
151204
self.response_schema = self._load_response_schema()
152205
self._use_plain_json_output = self._env_flag_enabled("SKILL_SCANNER_LLM_FORCE_JSON_OBJECT")
153206

207+
# Token usage for the most recent make_request() call (reset each call).
208+
self._last_usage: LLMTokenUsage = _empty_token_usage()
209+
210+
@property
211+
def last_usage(self) -> LLMTokenUsage:
212+
"""Token counts from the most recent make_request() call."""
213+
return dict(self._last_usage) # type: ignore[return-value]
214+
154215
def _env_flag_enabled(self, env_name: str) -> bool:
155216
"""Treat common truthy env values as enabled."""
156217
raw_value = os.getenv(env_name, "")
@@ -283,6 +344,7 @@ async def make_request(self, messages: list[dict[str, str]], context: str = "")
283344
Raises:
284345
Exception: If all retries exhausted
285346
"""
347+
self._last_usage = _empty_token_usage()
286348
if self.provider_config.use_google_sdk:
287349
# For Google SDK, combine system and user messages into a single prompt
288350
# Google SDK doesn't have separate system/user roles like OpenAI/Anthropic
@@ -322,6 +384,7 @@ async def _make_litellm_request(self, messages: list[dict[str, str]], context: s
322384

323385
response = await acompletion(**request_params, drop_params=True)
324386
content: str = response.choices[0].message.content or ""
387+
self._last_usage = _extract_token_usage(response)
325388
return content
326389

327390
except Exception as e:
@@ -336,6 +399,7 @@ async def _make_litellm_request(self, messages: list[dict[str, str]], context: s
336399
retry_params["response_format"] = {"type": "json_object"}
337400
response = await acompletion(**retry_params, drop_params=True)
338401
content: str = response.choices[0].message.content or ""
402+
self._last_usage = _extract_token_usage(response)
339403
return content
340404

341405
last_exception = e
@@ -407,6 +471,7 @@ def generate():
407471
return response
408472

409473
response = await loop.run_in_executor(None, generate)
474+
self._last_usage = _extract_google_sdk_token_usage(response)
410475

411476
# Extract text from response (new SDK format)
412477
# Response has .text attribute directly

skill_scanner/core/analyzers/meta_analyzer.py

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,10 +45,18 @@
4545
from typing import TYPE_CHECKING, Any
4646

4747
from ...threats.threats import ThreatMapping
48-
from ..models import Finding, Severity, Skill, ThreatCategory
48+
from ..models import Finding, ScanResult, Severity, Skill, ThreatCategory
4949
from .base import BaseAnalyzer
5050
from .llm_provider_config import ProviderConfig
51-
from .llm_request_handler import _TEMPERATURE_UNSET, LLMRequestHandler, _resolve_temperature
51+
from .llm_request_handler import (
52+
_TEMPERATURE_UNSET,
53+
LLMRequestHandler,
54+
LLMTokenUsage,
55+
_add_token_usage,
56+
_empty_token_usage,
57+
_extract_token_usage,
58+
_resolve_temperature,
59+
)
5260
from .llm_request_options import resolve_llm_user, supports_openai_user_param
5361

5462
if TYPE_CHECKING:
@@ -365,9 +373,17 @@ def __init__(
365373
self.max_retries = max_retries
366374
self.timeout = timeout
367375

376+
# Cumulative token usage across all LLM calls in the most recent analyze_with_findings() run.
377+
self._llm_usage: LLMTokenUsage = _empty_token_usage()
378+
368379
# Load prompts
369380
self._load_prompts()
370381

382+
@property
383+
def llm_usage(self) -> LLMTokenUsage:
384+
"""Cumulative token usage from the most recent analyze_with_findings() run."""
385+
return dict(self._llm_usage) # type: ignore[return-value]
386+
371387
def _load_prompts(self):
372388
"""Load meta-analysis prompt templates from files."""
373389
prompts_dir = Path(__file__).parent.parent.parent / "data" / "prompts"
@@ -427,6 +443,8 @@ async def analyze_with_findings(
427443
Returns:
428444
MetaAnalysisResult with validated findings, false positives, and recommendations
429445
"""
446+
self._llm_usage = _empty_token_usage()
447+
430448
if not findings:
431449
return MetaAnalysisResult(
432450
overall_risk_assessment={
@@ -877,6 +895,7 @@ async def _make_llm_request(self, system_prompt: str, user_prompt: str) -> str:
877895
try:
878896
response = await acompletion(**api_params, drop_params=True)
879897
content: str = response.choices[0].message.content or ""
898+
_add_token_usage(self._llm_usage, _extract_token_usage(response))
880899
return content
881900

882901
except Exception as e:
@@ -1128,3 +1147,20 @@ def apply_meta_analysis_to_results(
11281147
result_findings.extend(missed_findings)
11291148

11301149
return result_findings
1150+
1151+
1152+
def merge_meta_analyzer_usage(result: ScanResult, meta_analyzer: MetaAnalyzer) -> None:
1153+
"""Fold a MetaAnalyzer's token usage into a scan result's aggregated ``llm_usage``.
1154+
1155+
MetaAnalyzer always runs as a separate post-processing step after
1156+
``SkillScanner`` has already produced a ``ScanResult`` (see
1157+
``analyze_with_findings``), so its token spend isn't captured by
1158+
``SkillScanner``'s own per-scan aggregation. Call this immediately after
1159+
``analyze_with_findings()`` to fold the meta-analysis call(s) in.
1160+
"""
1161+
usage = meta_analyzer.llm_usage
1162+
if not any(usage.values()):
1163+
return
1164+
aggregated: LLMTokenUsage = dict(result.llm_usage) if result.llm_usage else _empty_token_usage() # type: ignore[assignment]
1165+
_add_token_usage(aggregated, usage)
1166+
result.llm_usage = dict(aggregated) # type: ignore[arg-type]

skill_scanner/core/models.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,7 @@ class ScanResult:
218218
analyzability_score: float | None = None
219219
analyzability_details: dict[str, Any] | None = None
220220
scan_metadata: dict[str, Any] | None = None
221+
llm_usage: dict[str, int] | None = None
221222

222223
@property
223224
def is_safe(self) -> bool:
@@ -265,6 +266,8 @@ def to_dict(self) -> dict[str, Any]:
265266
}
266267
if self.analyzers_failed:
267268
result["analyzers_failed"] = self.analyzers_failed
269+
if self.llm_usage:
270+
result["llm_usage"] = self.llm_usage
268271
return result
269272

270273

0 commit comments

Comments
 (0)