release: v1.108.182 — a stall has a name and a ceiling (#375 reopened)

Darius Kiaulakis' re-run at 1.108.176 measured no improvement on the subtree
that stalled at 1.108.169: 268s vs 240s, both unfinished. The 1.108.171 regex
fix removed one known cost centre. It could not remove the shape of the
failure, which is that provider discovery and per-file parsing were both
unbounded and both reported nothing while they ran.

Three bounds:

- discover_providers runs each provider detect+load under a wall-clock ceiling
  (JCODEMUNCH_PROVIDER_BUDGET_SECONDS, default 30s, 0 disables). An overrun is
  skipped and named in providers_skipped plus a warning stating the
  consequence: symbols indexed, no context from that provider, and its import
  edges missing from the graph.
- iter_source_files prunes dependency dirs at the walk. ExpressProvider.load
  ran five recursive globs and dropped node_modules by substring after the walk
  had already enumerated it; _scan_package_json_forced_paths had the same
  defect. Measured 0.639s -> 0.001s for the identical 41 files over a
  9,600-file node_modules, warm and alternating.
- parse_file_budgeted bounds one file's parse
  (JCODEMUNCH_PARSE_BUDGET_SECONDS, default 20s), armed only at or above
  128 KiB so the common path stays inline.

Stated limit: a watchdog stops the caller waiting, it cannot stop the work.
Python cannot preempt a thread and tree-sitter is C, so an abandoned parse or
provider keeps burning CPU until it finishes or polls budget_expired().

tests/test_v1_108_182.py (23). 5947 passed, 7 skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: jgravelle

CHANGELOG.md

@@ -2,6 +2,51 @@
 
 All notable changes to jcodemunch-mcp are documented here.
 
+## [1.108.182] - 2026-07-26 - a stall has a name and a ceiling
+
+### Fixed
+
+- **Provider discovery was unbounded, and it runs before a single file is
+  indexed.** [#375](https://github.com/jgravelle/jcodemunch-mcp/issues/375),
+  reopened by @dkiaulakis after a re-run at 1.108.176 measured no improvement on
+  the subtree that stalled at 1.108.169 — 268s versus 240s, both unfinished. The
+  1.108.171 regex fix removed one known cost centre. It could not remove the
+  SHAPE of the failure: `discover_providers` ran all fourteen providers'
+  `detect()` + `load()` inline with no ceiling and no attribution, so any
+  provider that ground took the whole index down with it and the caller saw
+  silence. Each provider now runs under a wall-clock budget
+  (`JCODEMUNCH_PROVIDER_BUDGET_SECONDS`, default 30s, `0` disables). An overrun
+  is skipped and NAMED — `providers_skipped` on the index result plus a warning
+  that states the consequence, that symbols are indexed but carry no context
+  from that provider and its import edges (route mounts, template renders) are
+  missing from the graph. A budget cannot make a slow provider fast; it makes
+  the gap survivable and attributable instead of an unbounded wait.
+- **Source walks paid for dependency trees they then discarded.**
+  `ExpressProvider.load` ran five recursive `Path.glob("**/*.js")`-shaped passes
+  and dropped `node_modules` by substring AFTER the walk had already enumerated
+  it — five full descents through a dependency tree to index none of it. Same
+  defect in `_scan_package_json_forced_paths`, which `rglob`'d for manifests and
+  filtered `node_modules` out afterwards. New `iter_source_files` prunes at the
+  walk. Measured on a 42-file project over a 9,600-file `node_modules`, warm and
+  alternating: **0.639s → 0.001s for the identical 41 files.**
+- **One pathological file could stall a whole index.** `parse_file` now runs
+  under a per-file wall-clock ceiling
+  (`JCODEMUNCH_PARSE_BUDGET_SECONDS`, default 20s, `0` disables) via
+  `parse_file_budgeted`. On overrun the file is skipped and named in the index
+  result's `warnings` instead of the run hanging with nothing to point at. The
+  watchdog is armed only for files at or above 128 KiB, so the common path
+  parses inline exactly as before.
+
+### Known limits, stated
+
+- A watchdog stops the CALLER waiting; it cannot stop the work. Python cannot
+  preempt a thread, and tree-sitter is C code, so an abandoned parse or provider
+  keeps consuming CPU until it finishes or notices its cooperative deadline.
+  These bounds make the index finish and the gap visible — they do not cap CPU.
+- `ContextProvider.budget_expired()` is the cooperative half, and it only helps
+  where a provider polls it. Wired into the Express walk; the other thirteen
+  providers still rely on the thread watchdog alone.
+
 ## [1.108.181] - 2026-07-26 - uncommitted work is represented, per scope
 
 ### Fixed

CLAUDE.md

@@ -1,12 +1,12 @@
 # jcodemunch-mcp — Project Brief
 
 ## Current State
-- **Version:** 1.108.181 — **Uncommitted work is represented, PER SCOPE (#377 hardening item 5 — closes the last open 1.x item).** ⚠ **Git HEAD sits still while the tree holds a modified file, a brand-new UNTRACKED implementation, a deletion or a rename into/out of scope — so every freshness gate we had could report `fresh` over a corpus that never read the file the target lives in. A scan that RETURNS rows carries per-file freshness ON them; a scan that returns NOTHING has nowhere to put it.** New `subject_state.working_tree_state` gives the SCOPE its own state (`clean`/`dirty_in_scope`/`dirty_outside_scope`/`unknown`/`not_applicable`) as `verdict.working_tree`, wired into `search_symbols` / `search_text` / BOTH `get_ranked_context` exits. ⚠ **TWO restrictions are what keep it from becoming noise, and both are load-bearing: (1) work OUTSIDE the scanned scope NEVER blocks (his own distinction — a dirty file elsewhere cannot invalidate a narrow proof); (2) an edit the index has ALREADY re-read is NOT a gap — `files_not_in_index` compares each dirty in-scope path against `index.file_mtimes`, so a watcher-fresh corpus keeps proving absence. Without (2) the gate fires on EVERY developer with unsaved work in an up-to-date repo, and a signal that fires constantly is one people learn to ignore.** ⚠ **Measured for a ZERO-RESULT scan only** (a `git status` per search would tax the common path); reading is TTL-cached and SHARED with the 1.108.178 cached-negative revalidation. ⚠ **Rename handling: `R old -> new` records the NEW path — the one that exists in the tree and the one a search would have had to see;** quoted/backslashed paths normalised to compare against index paths. `working_tree` published in the verdict schema. New `tests/test_v1_108_181.py` (27). No tool-count/INDEX_VERSION change.
+- **Version:** 1.108.182 — **A stall has a name and a ceiling (#375 REOPENED).** ⚠ **@dkiaulakis' re-run at 1.108.176 measured NO improvement on the subtree that stalled at 1.108.169 — 268s vs 240s, both unfinished. The 1.108.171 regex fix removed one known cost centre; it could not remove the SHAPE of the failure, which is that discovery and parsing were both unbounded and both reported nothing.** Three bounds: (1) `discover_providers(folder, budget_seconds=, skipped=)` runs each provider's detect+load under a wall-clock ceiling (`JCODEMUNCH_PROVIDER_BUDGET_SECONDS`, 30s, 0 disables); an overrun is skipped and NAMED via `providers_skipped` + a warning that states the CONSEQUENCE (symbols indexed, but no context from that provider and its import edges — route mounts, template renders — absent from the graph). (2) New `_route_utils.iter_source_files` prunes dependency dirs AT THE WALK: `ExpressProvider.load` ran FIVE recursive globs and dropped `node_modules` by substring AFTER enumerating it, same defect in `_scan_package_json_forced_paths`'s `rglob` — **measured 0.639s → 0.001s for the identical 41 files over a 9,600-file node_modules, warm and alternating.** (3) `parse_file_budgeted` bounds one file's parse (`JCODEMUNCH_PARSE_BUDGET_SECONDS`, 20s); raises `ParseBudgetExceeded`, which the three existing call sites' `except Exception` already turns into a by-name warning. ⚠ **Armed only at/above 128 KiB — the common path parses inline, unchanged.** ⚠ **THE LIMIT, STATED: a watchdog stops the CALLER waiting; it cannot stop the work.** Python cannot preempt a thread and tree-sitter is C, so an abandoned parse/provider keeps burning CPU. `ContextProvider.budget_expired()` is the cooperative half and only helps where a provider POLLS it — wired into the Express walk, not the other thirteen. ⚠ **Not yet reproduced on Darius' box; the walk + regex evidence says his 268s should be gone, and the strongest untested hypothesis is a stale resolved install (25+ leaked stdio instances, a :3049 server on the same db, hand-patched files an upgrade overwrote).** New `tests/test_v1_108_182.py` (23). No tool-count/INDEX_VERSION change.
+- **Prior (1.108.181):** — **Uncommitted work is represented, PER SCOPE (#377 hardening item 5 — closes the last open 1.x item).** ⚠ **Git HEAD sits still while the tree holds a modified file, a brand-new UNTRACKED implementation, a deletion or a rename into/out of scope — so every freshness gate we had could report `fresh` over a corpus that never read the file the target lives in. A scan that RETURNS rows carries per-file freshness ON them; a scan that returns NOTHING has nowhere to put it.** New `subject_state.working_tree_state` gives the SCOPE its own state (`clean`/`dirty_in_scope`/`dirty_outside_scope`/`unknown`/`not_applicable`) as `verdict.working_tree`, wired into `search_symbols` / `search_text` / BOTH `get_ranked_context` exits. ⚠ **TWO restrictions are what keep it from becoming noise, and both are load-bearing: (1) work OUTSIDE the scanned scope NEVER blocks (his own distinction — a dirty file elsewhere cannot invalidate a narrow proof); (2) an edit the index has ALREADY re-read is NOT a gap — `files_not_in_index` compares each dirty in-scope path against `index.file_mtimes`, so a watcher-fresh corpus keeps proving absence. Without (2) the gate fires on EVERY developer with unsaved work in an up-to-date repo, and a signal that fires constantly is one people learn to ignore.** ⚠ **Measured for a ZERO-RESULT scan only** (a `git status` per search would tax the common path); reading is TTL-cached and SHARED with the 1.108.178 cached-negative revalidation. ⚠ **Rename handling: `R old -> new` records the NEW path — the one that exists in the tree and the one a search would have had to see;** quoted/backslashed paths normalised to compare against index paths. `working_tree` published in the verdict schema. New `tests/test_v1_108_181.py` (27). No tool-count/INDEX_VERSION change.
 - **Prior (1.108.180):** — **Unknown freshness stops collapsing into fresh (#377 hardening item 4).** ⚠ **`FreshnessProbe.repo_is_stale` is a BOOLEAN and a boolean has nowhere to put "I could not find out": it returned False BOTH when the SHAs matched AND when either was missing, and the verdict rendered False as `channels.index: "fresh"` — so an index whose freshness was NEVER ESTABLISHED claimed current-snapshot equivalence, and every absence gate downstream trusted a comparison that was never made.** New `FreshnessProbe.repo_freshness` → `fresh`/`stale`/`unknown`/`not_tracked`; `build_verdict(freshness=)` renders it into `channels.index`. ⚠ **`unknown` and `not_tracked` get OPPOSITE treatment on purpose: `unknown` = a capability we HAVE, failing (git absent, source root moved, index stored no SHA) ⇒ zero-result scan is `degraded`, absence REFUSED. `not_tracked` = a capability the SUBJECT does not support (a plain indexed folder has no revision, ever) ⇒ DISCLOSED, absence still citable — the jdata v1.26.0 call. Refusing there would strip absence evidence from EVERY folder index over a permanent, already-stated limitation unrelated to whether the target exists.** ⚠ **A SUBDIRECTORY of a checkout is tracked by the checkout ABOVE it — trackability walks up the way git does (`_is_git_backed`), else every monorepo subdir index silently understates to `not_tracked`.** Precedence `rebuilding` > `partial` > `stale` > `unknown`/`not_tracked` > `fresh`. ⚠ **The `channels.index` enum is CLOSED in the published schema — both new values had to be added or every affected response fails our own contract (the 1.108.168 trap, third occurrence).** Zero blast radius without `freshness=` (pinned); an unrecognized value is ignored, not propagated. New `tests/test_v1_108_180.py` (26) incl. a REAL one-commit checkout, because `fresh` is the one state a fixture cannot fake. No tool-count/INDEX_VERSION change.
-- **Prior (1.108.179):** — **The subject has to hold still for the scan (#377 hardening item 6).** `search_symbols` / `search_text` / `get_ranked_context` now capture subject identity BEFORE retrieval and compare AFTER: a concurrent edit, watcher reindex, incremental save, published generation or a long scan crossing a rebuild all mean the scan describes NEITHER the state it started against NOR the one it finished in. Zero-result + movement ⇒ `degraded` + `verdict.moved_during_scan` naming what moved. ⚠ **Scoped to a ZERO-RESULT scan, and that scoping is what makes it affordable: the AFTER reading bypasses the TTL-cached HEAD lookup and pays for a real `git rev-parse`, because a before/after comparison INSIDE the 2s cache window compares one reading with itself.** A scan that returned rows read them from a generation that really held them, and the freshness channels already disclose the tree moved. ⚠ **Second find, worse than the one being fixed: `get_ranked_context`'s no-candidate EARLY RETURN carried NO verdict at all** — it asserted "No implementation found ... Do not claim this feature exists" in PROSE with no freshness, no coverage, no scan counts and none of the absence gates, because it returned before `build_verdict` was ever called. Now builds the same verdict every other retrieval answer gets (so a genuine no-candidate result is CITABLE, and a stale/partial/rebuilding/moving one is REFUSED). `subject_state.changed(when=)` now distinguishes the two callers' sentences: "after this scan was cached" (1.108.178 cache replay) vs "while the scan was running" (this). ⚠ **Working-tree movement DURING a scan is still NOT checked** — the BEFORE reading would have to pay `git status` on every search; whole-scope tree state is item 5. `moved_during_scan` published in the verdict schema. New `tests/test_v1_108_179.py` (19), all three tools in one parametrized e2e. No tool-count/INDEX_VERSION change.
-- **Older releases (1.108.178 and earlier):** see `CHANGELOG.md` — the authoritative version history (416 entries, 1.0.0 onward).
+- **Older releases (1.108.179 and earlier):** see `CHANGELOG.md`. The 1.108.179 entry ("the subject has to hold still for the scan", #377 item 6) is there in full.
 - **INDEX_VERSION:** 17
-- **Tests:** 5924 passed, 7 skipped (1.108.181) + the KNOWN 12 local-ONNX `test_semantic_search` env failures (green in CI on all 4 ubuntu jobs — never read them as a regression)
+- **Tests:** 5947 passed, 7 skipped (1.108.182) + the KNOWN 12 local-ONNX `test_semantic_search` env failures (green in CI on all 4 ubuntu jobs — never read them as a regression)
 - **Python:** >=3.10
 - **Tool count:** 90 in `full` (front door hidden; +1 v1.108.111 `get_parity_map`, +1 v1.108.112 `get_decorator_census`, +1 v1.108.113 `get_architecture_metrics`); `tool_surface=counter` exposes a 3-tool front door (`order`/`menu`/`route`) instead
 
@@ -237,6 +237,8 @@ Tree-sitter grammar lacks clean named fields for these — custom regex extracto
 | `JCODEMUNCH_ORG_INGEST_ENABLED` | 0 | Set 1 on the org host to accept `POST /org/report` (two-key turn with `JCODEMUNCH_HTTP_TOKEN`) |
 | `JCODEMUNCH_LICENSE_KEY` | — | (v1.108.42) jCodeMunch license key (config key `license_key`). Gates the `org-rollup` team feature ONLY; everything else is free. Validated online vs `validate.php` (sticky-offline cache; 14-day grace for new orgs). **Requires a multi-seat tier — Studio or Platform** (v1.108.43); Builder doesn't unlock org-rollup. Check with the `license` CLI. |
 | `JCODEMUNCH_INDEX_CACHE_TTL` | 0 (off) | (v1.108.172) Seconds an unused hydrated index may sit in the in-memory cache before being released. **OPT-IN: 0/unset/garbage = disabled = today's behavior exactly.** ⚠ **Do NOT default this on** — cold hydration of a 665k-symbol index was measured at 7.5-11.4 min (#370), so evicting during a quiet spell hands the next query that bill. For hosts whose MCP client leaks stdio servers (#375: 25+ instances, ~17 GB), where each idle process otherwise sits on its own cache. Swept on access, no timer thread. |
+| `JCODEMUNCH_PROVIDER_BUDGET_SECONDS` | 30.0 | (v1.108.182) Wall-clock ceiling on ONE context provider's `detect()`+`load()`. Discovery runs before a single file is indexed, so an unbounded provider takes the whole index down with it (#375). On overrun the provider is skipped and NAMED in `providers_skipped` + `warnings`. `0`/negative = no ceiling (pre-.182 inline behaviour). ⚠ **A watchdog stops the CALLER waiting; it cannot stop the work** — Python cannot preempt a thread, so the abandoned provider keeps burning CPU until it finishes or polls `budget_expired()`. Only the Express walk polls it so far. |
+| `JCODEMUNCH_PARSE_BUDGET_SECONDS` | 20.0 | (v1.108.182) Per-file wall-clock ceiling on `parse_file`, via `parse_file_budgeted`. On overrun the file is skipped and named in the index result's `warnings` instead of the run hanging. ⚠ **Armed only at or above 128 KiB** (`_PARSE_WATCHDOG_MIN_BYTES`) so the common path stays inline — a 2 KB file that takes 20s is a bug to see, not to paper over. `0`/negative disables. Same no-preemption caveat: tree-sitter is C code. |
 | `JCODEMUNCH_SCIP_MAX_ROWS` | 200000 | (v1.108.96) Row cap for `scip_edges` / `scip_unmapped` (compile-time evidence from `import-scip`); FIFO-evicted oldest-first in 1k batches. Negative disables the cap; env-only, deliberately not a config key. |
 | `JCODEMUNCH_LAUNCH_ID` | — | (v1.108.152) Opaque host-supplied launch token echoed back as `launch_id` in the `munch://runtime/identity` resource (#371). Fallback: suite-generic `MUNCH_LAUNCH_ID`. Omitted from the payload when unset. Env-only, not a config key. |
 

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "jcodemunch-mcp"
-version = "1.108.181"
+version = "1.108.182"
 description = "Token-efficient MCP server for source code exploration via tree-sitter AST parsing"
 readme = "README.md"
 requires-python = ">=3.10"

src/jcodemunch_mcp/parser/context/_route_utils.py

@@ -7,12 +7,52 @@
 from __future__ import annotations
 
 import json
+import os
 import re
 import sys
+from collections.abc import Iterator
 from pathlib import Path
 
 from .base import FileContext
 
+# Directories a source walk must never descend into. Dependency trees dwarf
+# first-party source — a mid-sized node_modules is six figures of files — so the
+# cost of touching them at all swamps the scan they precede.
+DEFAULT_SKIP_DIRS = frozenset({
+    "node_modules", "dist", "build", "vendor", "venv", ".venv", "target",
+    ".next", ".nuxt", ".git", ".svn", ".hg", "__pycache__", ".tox",
+    ".mypy_cache", ".pytest_cache", ".gradle", "bower_components",
+})
+
+
+def iter_source_files(
+    root: Path,
+    suffixes: frozenset[str] | set[str] | tuple[str, ...],
+    skip_dirs: frozenset[str] | set[str] = DEFAULT_SKIP_DIRS,
+) -> Iterator[tuple[Path, str]]:
+    """Yield ``(absolute_path, repo_relative_posix_path)`` for matching files.
+
+    Prunes ``skip_dirs`` AT THE WALK, which is the whole point: ``Path.glob``
+    and ``rglob`` descend into every directory and can only be filtered after
+    the fact, so a post-hoc ``if "node_modules" in path: continue`` still pays
+    to enumerate the entire dependency tree — measured at ~3.3s per 9,600
+    skipped files, repeated once per glob pattern. os.walk lets us drop the
+    subtree before it is entered.
+
+    Suffix matching is case-insensitive and each file is yielded once, however
+    many patterns would have matched it.
+    """
+    wanted = {s.lower() for s in suffixes}
+    root_str = str(root)
+    for dirpath, dirnames, filenames in os.walk(root_str):
+        dirnames[:] = [d for d in dirnames if d not in skip_dirs]
+        for filename in filenames:
+            if os.path.splitext(filename)[1].lower() not in wanted:
+                continue
+            abs_path = Path(dirpath) / filename
+            rel = os.path.relpath(str(abs_path), root_str).replace(os.sep, "/")
+            yield abs_path, rel
+
 # Try to use tomllib for proper TOML parsing (Python 3.11+)
 if sys.version_info >= (3, 11):
     import tomllib