fix: raise default max_tokens from 4000 to 8192, add --llm-max-tokens CLI flag (#51)

* fix: raise default max_tokens from 4000 to 8192

The default max_tokens of 4000 for LLMAnalyzer (and 8000 for
MetaAnalyzer) frequently causes truncated JSON responses. In a batch
of 200 skills, 30.5% of scans produced truncated output at the
4000-token limit; raising to 32768 at the call site eliminated all
truncation.

Set the default to 8192 — the Gemini API default for maxOutputTokens,
and safely within the output ceiling of all major providers.

Fixes #50

Made-with: Cursor

* feat: add --llm-max-tokens CLI flag

Expose max_tokens as a CLI argument so users can tune the output
token budget without dropping to the Python SDK. Threaded through
to both LLMAnalyzer and MetaAnalyzer via the centralized
analyzer_factory.

Made-with: Cursor

* feat: add max_output_tokens to scan policy

Add llm_analysis.max_output_tokens as a policy-level knob alongside
the existing input budget fields. The CLI --llm-max-tokens flag
overrides the policy value. Both the analyzer factory and
_build_meta_analyzer now read from the policy when no CLI override is
given.

Update docs: custom-policy-configuration, policy-quick-reference,
cli-command-reference, and llm-analyzer architecture.

Made-with: Cursor

* fix: add max_output_tokens to TUI, use is-not-None checks, add round-trip tests

- Add max_output_tokens Input widget to policy TUI (configure-policy)
  so users can set the output token budget interactively
- Replace `or` with `is not None` in analyzer_factory and cli for
  correct handling of explicit zero values
- Add TestMaxOutputTokensPolicy with YAML round-trip, _to_dict, and
  preset coverage tests

* fix: add max_output_tokens to all presets, expand test coverage

- Explicitly set max_output_tokens: 8192 in strict_policy.yaml and
  permissive_policy.yaml (previously only inherited via deep merge)
- Add tests: custom YAML override, CLI-beats-policy in build_analyzers,
  policy-used-when-CLI-is-None, MetaAnalyzer forwarding from CLI,
  MetaAnalyzer fallback to policy

---------

Co-authored-by: vnarajal <vnarajal@cisco.com>

Author: maresb

README.md

@@ -231,6 +231,7 @@ if not result.is_safe:
 | `--use-llm` | Enable LLM analyzer (requires API key) |
 | `--llm-provider` | LLM provider for CLI routing: `anthropic` or `openai` |
 | `--llm-consensus-runs N` | Run LLM analysis `N` times and keep majority-agreed findings |
+| `--llm-max-tokens N` | Maximum output tokens for LLM responses (default: 8192) |
 | `--use-virustotal` | Enable VirusTotal binary scanner |
 | `--vt-api-key KEY` | Provide VirusTotal API key directly (optional) |
 | `--vt-upload-files` | Upload unknown binaries to VirusTotal (optional) |

docs/architecture/analyzers/llm-analyzer.md

@@ -281,6 +281,7 @@ export AWS_REGION=us-east-1
 - Retries transient failures (`429`/timeouts/network issues) with exponential backoff
 - Supports async execution (`analyze_async`) and optional consensus passes
 - Applies prompt budget gates from policy (`llm_analysis.*`) and emits `LLM_CONTEXT_BUDGET_EXCEEDED` when content is skipped
+- Output token limit is controlled by `llm_analysis.max_output_tokens` in scan policy (default 8192), overridable via `--llm-max-tokens` CLI flag
 
 ## Error Handling
 

docs/reference/cli-command-reference.md

@@ -104,7 +104,8 @@ usage: cli.py scan [-h] [--format {summary,json,markdown,table,sarif,html}]
                    [--aidefense-api-key AIDEFENSE_API_KEY]
                    [--aidefense-api-url AIDEFENSE_API_URL]
                    [--llm-provider {anthropic,openai}]
-                   [--llm-consensus-runs N] [--use-trigger] [--enable-meta]
+                   [--llm-consensus-runs N] [--llm-max-tokens N]
+                   [--use-trigger] [--enable-meta]
                    [--policy PRESET_OR_PATH] [--lenient] [--custom-rules PATH]
                    [--taxonomy PATH] [--threat-mapping PATH]
                    skill_directory
@@ -162,6 +163,8 @@ options:
                         Run LLM analysis N times and keep only findings with
                         majority agreement (reduces false positives, increases
                         cost)
+  --llm-max-tokens N    Maximum output tokens for LLM responses (default:
+                        8192). Raise if scans produce truncated JSON.
   --use-trigger         Enable trigger specificity analysis
   --enable-meta         Enable meta-analysis FP filtering (2+ analyzers)
   --policy PRESET_OR_PATH
@@ -203,7 +206,8 @@ usage: cli.py scan-all [-h] [--recursive] [--check-overlap]
                        [--aidefense-api-key AIDEFENSE_API_KEY]
                        [--aidefense-api-url AIDEFENSE_API_URL]
                        [--llm-provider {anthropic,openai}]
-                       [--llm-consensus-runs N] [--use-trigger]
+                       [--llm-consensus-runs N] [--llm-max-tokens N]
+                       [--use-trigger]
                        [--enable-meta] [--policy PRESET_OR_PATH] [--lenient]
                        [--custom-rules PATH] [--taxonomy PATH]
                        [--threat-mapping PATH]
@@ -264,6 +268,8 @@ options:
                         Run LLM analysis N times and keep only findings with
                         majority agreement (reduces false positives, increases
                         cost)
+  --llm-max-tokens N    Maximum output tokens for LLM responses (default:
+                        8192). Raise if scans produce truncated JSON.
   --use-trigger         Enable trigger specificity analysis
   --enable-meta         Enable meta-analysis FP filtering (2+ analyzers)
   --policy PRESET_OR_PATH

docs/reference/policy-quick-reference.md

@@ -253,7 +253,8 @@ Controls LLM context budget thresholds for LLM and meta analyzers. Content withi
 | max_code_file_chars | int | 15000 | Maximum character length for a single code file sent to the LLM |
 | max_referenced_file_chars | int | 10000 | Maximum character length for a single referenced file sent to the LLM |
 | max_total_prompt_chars | int | 100000 | Maximum total characters across the entire LLM prompt |
-| meta_budget_multiplier | float | 3.0 | Multiplier applied to all limits above for the meta analyzer (e.g. 3x = 60K instruction, 45K/file, 300K total) |
+| max_output_tokens | int | 8192 | Maximum output tokens for LLM responses (both LLM analyzer and meta-analyzer) |
+| meta_budget_multiplier | float | 3.0 | Multiplier applied to all input limits above for the meta analyzer (e.g. 3x = 60K instruction, 45K/file, 300K total) |
 
 </details>
 

docs/user-guide/custom-policy-configuration.md

@@ -518,12 +518,14 @@ llm_analysis:
   max_code_file_chars: 15000           # Max chars per individual code file
   max_referenced_file_chars: 10000     # Max chars per referenced file
   max_total_prompt_chars: 100000       # Total prompt budget across all files
+  max_output_tokens: 8192              # Max tokens for LLM responses
   meta_budget_multiplier: 3.0          # Meta-analyzer multiplies above limits by this factor
 ```
 
 **Impact:**
 - Files or instruction bodies exceeding these limits are skipped entirely (no truncation) and a budget-skip metadata entry is attached to the scan result.
-- The meta-analyzer applies `meta_budget_multiplier` on top of the base limits. With the defaults, the meta-analyzer gets 60K instruction, 45K per file, and 300K total.
+- `max_output_tokens` controls the output token budget for both the LLM analyzer and meta-analyzer. Raise this if scans produce truncated JSON (`LLM_ANALYSIS_FAILED` findings). The CLI flag `--llm-max-tokens` overrides this value.
+- The meta-analyzer applies `meta_budget_multiplier` on top of the base input limits. With the defaults, the meta-analyzer gets 60K instruction, 45K per file, and 300K total.
 - Increase these values for skills with large codebases or extensive instructions. Decrease them to reduce LLM API costs.
 
 </details>

skill_scanner/cli/cli.py

@@ -118,6 +118,7 @@ def _build_analyzers(policy: ScanPolicy, args: argparse.Namespace, status: Calla
         use_trigger=getattr(args, "use_trigger", False),
         llm_provider=getattr(args, "llm_provider", None),
         llm_consensus_runs=getattr(args, "llm_consensus_runs", 1),
+        llm_max_tokens=getattr(args, "llm_max_tokens", None),
     )
 
     # Emit status messages for the optional analyzers that were activated.
@@ -138,7 +139,13 @@ def _build_analyzers(policy: ScanPolicy, args: argparse.Namespace, status: Calla
     return analyzers
 
 
-def _build_meta_analyzer(args: argparse.Namespace, analyzer_count: int, status: Callable[[str], None], policy=None):
+def _build_meta_analyzer(
+    args: argparse.Namespace,
+    analyzer_count: int,
+    status: Callable[[str], None],
+    policy=None,
+    max_tokens: int | None = None,
+):
     """Optionally build a MetaAnalyzer if ``--enable-meta`` is set."""
     if not getattr(args, "enable_meta", False):
         return None
@@ -157,13 +164,19 @@ def _build_meta_analyzer(args: argparse.Namespace, analyzer_count: int, status:
         meta_model = os.getenv("SKILL_SCANNER_META_LLM_MODEL") or os.getenv("SKILL_SCANNER_LLM_MODEL")
         meta_base_url = os.getenv("SKILL_SCANNER_META_LLM_BASE_URL") or os.getenv("SKILL_SCANNER_LLM_BASE_URL")
         meta_api_version = os.getenv("SKILL_SCANNER_META_LLM_API_VERSION") or os.getenv("SKILL_SCANNER_LLM_API_VERSION")
-        meta = MetaAnalyzer(
+        kwargs: dict = dict(
             model=meta_model,
             api_key=meta_api_key,
             base_url=meta_base_url,
             api_version=meta_api_version,
             policy=policy,
         )
+        effective_max_tokens = (
+            max_tokens if max_tokens is not None else (policy.llm_analysis.max_output_tokens if policy else None)
+        )
+        if effective_max_tokens is not None:
+            kwargs["max_tokens"] = effective_max_tokens
+        meta = MetaAnalyzer(**kwargs)
         status("Using Meta-Analyzer for false positive filtering and finding prioritization")
         return meta
     except Exception as e:
@@ -320,7 +333,8 @@ def scan_command(args: argparse.Namespace) -> int:
 
     policy = _load_policy(args)
     analyzers = _build_analyzers(policy, args, status)
-    meta_analyzer = _build_meta_analyzer(args, len(analyzers), status, policy=policy)
+    llm_max_tokens = getattr(args, "llm_max_tokens", None)
+    meta_analyzer = _build_meta_analyzer(args, len(analyzers), status, policy=policy, max_tokens=llm_max_tokens)
 
     scanner = SkillScanner(analyzers=analyzers, policy=policy)
     lenient = getattr(args, "lenient", False)
@@ -405,7 +419,8 @@ def scan_all_command(args: argparse.Namespace) -> int:
 
     policy = _load_policy(args)
     analyzers = _build_analyzers(policy, args, status)
-    meta_analyzer = _build_meta_analyzer(args, len(analyzers), status, policy=policy)
+    llm_max_tokens = getattr(args, "llm_max_tokens", None)
+    meta_analyzer = _build_meta_analyzer(args, len(analyzers), status, policy=policy, max_tokens=llm_max_tokens)
 
     scanner = SkillScanner(analyzers=analyzers, policy=policy)
 
@@ -718,6 +733,13 @@ def _add_common_scan_flags(parser: argparse.ArgumentParser) -> None:
         metavar="N",
         help="Run LLM analysis N times and keep only findings with majority agreement (reduces false positives, increases cost)",
     )
+    parser.add_argument(
+        "--llm-max-tokens",
+        type=int,
+        default=None,
+        metavar="N",
+        help="Maximum output tokens for LLM responses (default: 8192). Raise if scans produce truncated JSON.",
+    )
     parser.add_argument("--use-trigger", action="store_true", help="Enable trigger specificity analysis")
     parser.add_argument("--enable-meta", action="store_true", help="Enable meta-analysis FP filtering (2+ analyzers)")
     parser.add_argument(

skill_scanner/cli/policy_tui.py

@@ -481,6 +481,9 @@ def compose(self) -> ComposeResult:
             with Horizontal(classes="field-row"):
                 yield Label("Max total prompt (chars)")
                 yield Input(value="100000", id="llm-max-total-prompt", type="integer")
+            with Horizontal(classes="field-row"):
+                yield Label("Max output tokens")
+                yield Input(value="8192", id="llm-max-output-tokens", type="integer")
             with Horizontal(classes="field-row"):
                 yield Label("Meta budget multiplier")
                 yield Input(value="3.0", id="llm-meta-multiplier")
@@ -578,6 +581,7 @@ def _sync_form_from_policy(self) -> None:
         self.query_one("#llm-max-code-file", Input).value = str(p.llm_analysis.max_code_file_chars)
         self.query_one("#llm-max-ref-file", Input).value = str(p.llm_analysis.max_referenced_file_chars)
         self.query_one("#llm-max-total-prompt", Input).value = str(p.llm_analysis.max_total_prompt_chars)
+        self.query_one("#llm-max-output-tokens", Input).value = str(p.llm_analysis.max_output_tokens)
         self.query_one("#llm-meta-multiplier", Input).value = str(p.llm_analysis.meta_budget_multiplier)
 
     def _sync_policy_from_form(self) -> None:
@@ -697,6 +701,10 @@ def _sync_policy_from_form(self) -> None:
             p.llm_analysis.max_total_prompt_chars = int(self.query_one("#llm-max-total-prompt", Input).value)
         except ValueError:
             pass
+        try:
+            p.llm_analysis.max_output_tokens = int(self.query_one("#llm-max-output-tokens", Input).value)
+        except ValueError:
+            pass
         try:
             p.llm_analysis.meta_budget_multiplier = float(self.query_one("#llm-meta-multiplier", Input).value)
         except ValueError:

skill_scanner/config/config.py

@@ -38,7 +38,7 @@ class Config:
     llm_model: str = "claude-3-5-sonnet-20241022"
     llm_base_url: str | None = None
     llm_api_version: str | None = None
-    llm_max_tokens: int = 4000
+    llm_max_tokens: int = 8192
     llm_temperature: float = 0.0
     llm_rate_limit_delay: float = 2.0
     llm_max_retries: int = 3

skill_scanner/core/analyzer_factory.py

@@ -90,6 +90,7 @@ def build_analyzers(
     aidefense_api_url: str | None = None,
     use_trigger: bool = False,
     llm_consensus_runs: int = 1,
+    llm_max_tokens: int | None = None,
 ) -> list[BaseAnalyzer]:
     """Build the full analyzer list (core + optional).
 
@@ -100,6 +101,11 @@ def build_analyzers(
     hook, and eval scripts so that every entry point uses the exact same
     construction logic.
 
+    Args:
+        llm_max_tokens: Override the default ``max_tokens`` for the
+            :class:`LLMAnalyzer`.  When *None* the analyzer's own
+            default (8192) is used.
+
     Returns:
         A list of analyzer instances ready to be passed to
         :class:`SkillScanner`.
@@ -124,10 +130,18 @@ def build_analyzers(
             key = llm_api_key or os.getenv("SKILL_SCANNER_LLM_API_KEY")
             base_url = llm_base_url or os.getenv("SKILL_SCANNER_LLM_BASE_URL")
             api_version = llm_api_version or os.getenv("SKILL_SCANNER_LLM_API_VERSION")
+            extra_kwargs: dict = {}
+            effective_max_tokens = (
+                llm_max_tokens if llm_max_tokens is not None else policy.llm_analysis.max_output_tokens
+            )
+            if effective_max_tokens is not None:
+                extra_kwargs["max_tokens"] = effective_max_tokens
             if llm_provider and not llm_model and not os.getenv("SKILL_SCANNER_LLM_MODEL"):
-                llm = LLMAnalyzer(provider=llm_provider, policy=policy)
+                llm = LLMAnalyzer(provider=llm_provider, policy=policy, **extra_kwargs)
             else:
-                llm = LLMAnalyzer(model=model, api_key=key, base_url=base_url, api_version=api_version, policy=policy)
+                llm = LLMAnalyzer(
+                    model=model, api_key=key, base_url=base_url, api_version=api_version, policy=policy, **extra_kwargs
+                )
             if llm_consensus_runs > 1:
                 llm.consensus_runs = llm_consensus_runs
             analyzers.append(llm)

skill_scanner/core/analyzers/llm_analyzer.py

@@ -116,7 +116,7 @@ def __init__(
         self,
         model: str | None = None,
         api_key: str | None = None,
-        max_tokens: int = 4000,
+        max_tokens: int = 8192,
         temperature: float = 0.0,
         max_retries: int = 3,
         rate_limit_delay: float = 2.0,