[Bug] Consensus orchestration runs synchronously on the event loop: cross-tenant DoS + permanent worker-pool exhaustion

[rahuldass19]

Audit finding: BUG-R2-S2-A2-H3 | CWE-400 | CVSS 6.5 (A:H) | PoC: reproduced, EXP confirmed

Summary

verify_with_consensus is async def (api/main.py:1459) but calls the fully synchronous orchestrator directly (api/main.py:1486) — no await, no executor, no try/except. A correctly designed verify_async (consensus_verifier.py:446, run_in_executor + asyncio.wait_for) exists in the same class and has zero callers. Three defects:

1. Default 'single' mode -> exactly one engine -> _execute_sequential runs method(query) inline on the event-loop thread: a synchronous LLM HTTP round trip (_parse_math_query) + sympy evaluation with no timeout anywhere. Every single-mode consensus request blocks the entire service for the full duration; a compute-heavy translated expression makes it unbounded. EXP-measured: 12 benign requests/min (12% of one key's budget) held the service unavailable 83.6% of the window; a second tenant's /verify/math inflated 25-66x; ~6 req/min crash-loops a k8s replica.
2. Parallel modes: for future in as_completed(futures, timeout=30) — the TimeoutError is raised by the for-statement itself and is not caught by the inner try -> uncaught HTTP 500 after blocking the loop 30 s; futures are never cancelled, so slow engines keep their worker threads. Four hung engines permanently occupy all 4 workers of the process-global pool — every subsequent consensus request from any tenant degrades until restart (probe-blind: /health stays at 2 ms).
3. The circuit breaker only records completed futures (hung engines never open it); the rate limiter bounds count, not duration.

Suggested fix

• Call await consensus_verifier.verify_async(query, mode, timeout_seconds=30) from the endpoint.
• Catch concurrent.futures.TimeoutError around the as_completed loop; cancel pending futures, record breaker failures, degrade to BLOCKED instead of HTTP 500.
• Never run an engine inline on the caller's thread in _execute_sequential — submit to the executor with a hard future.result(timeout=30).

Note: a thread already started cannot be force-killed in Python; the paired compute-cost issue owns the wall-clock bound inside the SymPy engine. Distinct from the CircuitBreaker deadlock issue (must fix both).

See audit patch sketch.

---

Source: OpenVuln external audit of QWED-AI/qwed-verification (snapshot v7.0.0, re-verified against current main v7.1.0). All code anchors below were spot-checked against current main before filing.