fix: reliable analytical-query timeout signal + injectable runner - #284
Conversation
There was a problem hiding this comment.
Code Review
This pull request hardens query timeout handling by introducing a typed QueryTimeoutError and a structured _is_timeout_exc helper to detect both client-side asyncio timeouts and server-side PostgreSQL statement timeouts (SQLSTATE 57014). It also refactors server creation to support injecting an AnalyticalRunner and ensures run_analytical_query is only registered when a runner is present. Additionally, documentation has been updated, and comprehensive unit tests have been added. The review feedback suggests extending the timeout detection helper to also check exc.__context__ for implicit exception chaining and adding a corresponding unit test to ensure robust error handling.
| if isinstance(exc, TimeoutError) or isinstance(exc.__cause__, TimeoutError): | ||
| return True | ||
| return bool(getattr(exc, "sqlstate", None) == _PG_QUERY_CANCELED_SQLSTATE) |
There was a problem hiding this comment.
The _is_timeout_exc helper currently checks exc and exc.__cause__ to detect client-side TimeoutError exceptions. However, in Python, exceptions can also be chained implicitly via the __context__ attribute (for example, if a ValueError or QueryError is raised while a TimeoutError is active). To ensure robust timeout detection and prevent the self-correcting fallback from failing to fire under implicit chaining, we should also check exc.__context__.
| if isinstance(exc, TimeoutError) or isinstance(exc.__cause__, TimeoutError): | |
| return True | |
| return bool(getattr(exc, "sqlstate", None) == _PG_QUERY_CANCELED_SQLSTATE) | |
| if isinstance(exc, TimeoutError) or isinstance(exc.__cause__, TimeoutError) or isinstance(exc.__context__, TimeoutError): | |
| return True | |
| return bool(getattr(exc, "sqlstate", None) == _PG_QUERY_CANCELED_SQLSTATE) |
There was a problem hiding this comment.
Addressed in c826185, with a more general fix than the single __context__ check: _is_timeout_exc now walks the full __cause__/__context__ chain (bounded by a seen-set against cyclic chains), so a timeout buried under any number of wrapping layers is detected — not just one level deep. Added the __context__ test you suggested plus deep-cause and cyclic-chain cases.
Generated by Claude Code
| def test_is_timeout_exc_asyncio_cap_wrapped() -> None: | ||
| # SafeSqlDriver re-wraps the asyncio TimeoutError in a ValueError but | ||
| # chains the original as __cause__ — we detect it via the cause, not text. | ||
| wrapped = ValueError("Query execution timed out after 30 seconds") | ||
| wrapped.__cause__ = TimeoutError() | ||
| assert _is_timeout_exc(wrapped) is True |
There was a problem hiding this comment.
Add a unit test to verify that _is_timeout_exc correctly detects TimeoutError when it is implicitly chained via the __context__ attribute.
def test_is_timeout_exc_asyncio_cap_wrapped() -> None:
# SafeSqlDriver re-wraps the asyncio TimeoutError in a ValueError but
# chains the original as __cause__ — we detect it via the cause, not text.
wrapped = ValueError("Query execution timed out after 30 seconds")
wrapped.__cause__ = TimeoutError()
assert _is_timeout_exc(wrapped) is True
def test_is_timeout_exc_asyncio_cap_implicitly_chained() -> None:
# We also check __context__ to support implicit exception chaining.
wrapped = ValueError("Query execution timed out after 30 seconds")
wrapped.__context__ = TimeoutError()
assert _is_timeout_exc(wrapped) is TrueThere was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The new
analytical_availableplumbing meanscreate_server(..., analytical_runner=...)will registerrun_analytical_queryeven whensettings.enable_analytical_queriesisFalse; consider AND‑ing the flag with the runner presence so operators can reliably disable the tool via config even if a runner is injected. - Timeout detection in
_is_timeout_exconly looks at the immediate exception and its direct__cause__; if the driver or caller adds extra wrapping layers, these timeouts may be missed, so it may be worth walking the full__cause__/__context__chain (and possibly checkingsqlstatethere as well).
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `analytical_available` plumbing means `create_server(..., analytical_runner=...)` will register `run_analytical_query` even when `settings.enable_analytical_queries` is `False`; consider AND‑ing the flag with the runner presence so operators can reliably disable the tool via config even if a runner is injected.
- Timeout detection in `_is_timeout_exc` only looks at the immediate exception and its direct `__cause__`; if the driver or caller adds extra wrapping layers, these timeouts may be missed, so it may be worth walking the full `__cause__`/`__context__` chain (and possibly checking `sqlstate` there as well).
## Individual Comments
### Comment 1
<location path="src/mcpg/tools.py" line_range="2992-2999" />
<code_context>
- # it so it needn't predict duration up front.
- settings = ctx.request_context.lifespan_context.settings
- if settings.enable_analytical_queries and "timed out" in str(exc).lower():
+ except query.QueryTimeoutError as exc:
+ # Self-correcting fallback: a timeout — whether the client-side
+ # wall-clock cap or a server-side statement_timeout — is surfaced
+ # as a typed QueryTimeoutError, so we branch on the type instead of
+ # matching message text. Point the agent at the analytical path,
+ # but only when it's actually wired up (a runner is present); the
+ # runner is the source of truth, not the settings flag.
+ if ctx.request_context.lifespan_context.analytical_runner is not None:
raise query.QueryError(
f"{exc} If this is a legitimate long-running analytical query, retry with "
</code_context>
<issue_to_address>
**issue (bug_risk):** QueryTimeoutError is swallowed when no analytical runner is present, causing silent None returns.
As written, when `analytical_runner` is `None` the `except` block completes without re-raising, so the function returns `None` instead of propagating the timeout. Previously all timeouts produced a `QueryError`, so this changes the contract and can lead to confusing `None` results. Please re-raise the timeout when no runner is present, e.g. by adding:
```python
except query.QueryTimeoutError as exc:
if ctx.request_context.lifespan_context.analytical_runner is not None:
raise query.QueryError(
f"{exc} If this is a legitimate long-running analytical query, retry with "
"run_analytical_query — a higher, bounded timeout on an isolated connection."
) from exc
raise # or `raise exc`
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Follow-up hardening for run_analytical_query, addressing two review points on #283 (both bots flagged them independently): 1. Timeouts now raise a typed QueryTimeoutError (a QueryError subclass) detected structurally — the client-side asyncio wall-clock cap (via __cause__) and the server-side statement_timeout (SQLSTATE 57014). The run_select "retry with run_analytical_query" hint branches on the type instead of matching "timed out" in the message, so it now fires reliably and locale-independently for the Postgres-side timeout it previously missed. The hint is shown only when a runner is present. 2. AnalyticalRunner is injectable via create_server(analytical_runner=), and run_analytical_query is registered only when a runner will back it (register_tools gains analytical_available). An injected-database setup no longer advertises a non-functional tool. Docs: fold the analytical env vars into the README env-var table, the T3 (DoS) threat model in security.md, and the pool-overhead budget in scaling.md — spots the original feature PR missed. No tool-surface change (still 254). Adds structured-detection + registration-alignment unit tests; full unit + contract suite green (2866). Also syncs uv.lock to pyproject's pglast==8.4 (dependabot #281 bumped the pin but the lock stayed at 8.2 on main). SQL-kernel adversarial + fuzz + safety suites pass unchanged under 8.4. Roadmap linkage: N/A — follow-up hardening on #283's reactive capability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122yLZLJ8t4W43sdN6BmTZc
8e909e8 to
64b2b02
Compare
Two review points from the #284 bot reviews (Gemini + Sourcery), both small hardening tweaks: - _is_timeout_exc now walks the full __cause__/__context__ exception chain (bounded by a seen-set against cycles) instead of only the immediate exception and its direct __cause__ — extra wrapping layers or an implicit re-raise (which chains via __context__) can no longer hide a timeout signal. - MCPG_ENABLE_ANALYTICAL_QUERIES is now the authoritative off-switch: when false, create_server neither builds nor honours an injected runner, so run_analytical_query is never registered regardless of injection. An operator's config reliably wins over a caller-supplied runner. Note: Sourcery's "QueryTimeoutError swallowed -> returns None" comment is a false positive — the except block already ends in a bare `raise` that re-raises the timeout when no runner is present. New unit tests: deep-cause chain, implicit __context__ chain, cyclic chain tolerance, and flag-off-wins-over-injected-runner. Full unit + contract suite green (2870). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122yLZLJ8t4W43sdN6BmTZc
CI installed deps with `uv sync --frozen`, which installs from uv.lock as-is and never checks it against pyproject.toml. That masked a real drift: dependabot #281 bumped the pglast pin (8.2 -> 8.4) but left uv.lock at 8.2, and --frozen let it merge green — worse, CI was silently testing pglast 8.2, not the 8.4 the pin advertised. Switch all four dep-install steps (ci.yml x3, publish.yml x1) to `uv sync --locked`, which asserts the lock is up-to-date with the manifest and fails otherwise. A future manifest bump that forgets to regenerate the lock now fails CI instead of drifting invisibly. Verified end-to-end: this PR was red at the install step until #284 (the lock sync) merged, then green after rebase.
Summary
Follow-up hardening for
run_analytical_query, addressing the two review points raised on #283 (both Gemini and Sourcery flagged them independently).1. Reliable, structured timeout detection. The old
run_selectfallback that hints callers towardrun_analytical_querymatched"timed out" in str(exc).lower()— brittle, locale-dependent, and it missed the server-side path entirely: a Postgresstatement_timeoutsurfaces ascanceling statement due to statement timeout(SQLSTATE 57014), which never contains "timed out". So the hint fired for the client-side asyncio cap but silently not for the GUC timeout — the more common production trigger.Now both timeout mechanisms raise a typed
QueryTimeoutError(aQueryErrorsubclass, so existingexcept QueryErrorhandlers keep working). Detection is structural, not textual:SafeSqlDriverre-wraps theasyncio.TimeoutErrorin aValueErrorbut chains the original as__cause__, so we check the cause;statement_timeout— psycopg's SQLSTATE57014.The
run_selecthint branches on the type, and is shown only when a runner is actually wired up (the runner is the source of truth, not the settings flag).2. Injectable runner; registration aligned with runner presence. With an injected
databaseandenable_analytical_queries=True, #283 still registeredrun_analytical_querywhile the runner wasNone— advertising a tool that errors on call. Now:create_server(..., analytical_runner=...)accepts an injectable runner (mirroringlisten_manager/cursor_manager);register_toolsgainsanalytical_available, so the tool is registered only when a runner will back it. Contract tests that callregister_toolsdirectly fall back to the settings flag, so the surface snapshot is unchanged.Docs (spots the original feature PR missed). Folded the analytical env vars into the README env-var table, added the bounded-analytical-path mitigation to the T3 (DoS) threat model in
security.md, and added an analytical pool-overhead entry toscaling.md's peak-connection budget.Lockfile drift (folded in).
origin/mainhadpyproject.tomlpinningpglast==8.4(dependabot #281) butuv.lockstill at8.2. Synced the lock to8.4; the SQL-kernel adversarial + fuzz + safety suites pass unchanged under the parser bump.No tool-surface change (still 254). Full unit + contract suite green (2866 passed, +7 new);
ruff/ruff format/mypy src/mcpgclean.Roadmap linkage
Advances roadmap row: N/A — follow-up hardening on #283's reactive capability (itself N/A).
Checklist
ruff,ruff format, andmypy src/mcpgpassCHANGELOG.mdupdated under[Unreleased]N/A); row marked ✅ when this PR completes itsrc/mcpg/_vendor/🤖 Generated with Claude Code
https://claude.ai/code/session_0122yLZLJ8t4W43sdN6BmTZc