Keep blocking work off the event loop

[fangge518]

Every async def route handler that does blocking SQLAlchemy or JWKS work runs that work on the event loop. On AWS, each ECS task is a single uvicorn process with one event loop — one slow request stalls all concurrent requests on the same task, including ALB health checks. The anyio threadpool (80 threads, set in the lifespan) and connection pool (60/task, core/db.py) were sized in PR #623 for exactly this conversion; this is the code-level half of that fix.

Fix: Convert 38 async def route handlers to plain def across main.py, cms/main.py, comments/main.py, exports/main.py, save_share/main.py, thumbnails/main.py — FastAPI runs plain-def handlers in the anyio threadpool. Handlers with genuine awaits (Turnstile verification, request.body()) stay async; their sync DB portions now run through run_in_threadpool. update_assignments extracts a _sync_update_assignments helper so request.body() keeps its await while the 300-line DB body runs off the loop. The three graph handlers (get_unassigned_geoids, check_document_contiguity, get_connected_component_bboxes) are converted to def and call get_graph(...) directly — get_graph is a plain sync function (S3 fetch + unpickle); the prior await run_in_threadpool(get_graph, ...) was scaffolding around blocking code, not a genuine async operation. VerifyToken.verify converts to sync def (core/security.py) — JWKS fetch and jwt.decode now run in the threadpool as a FastAPI dependency. A convention comment at the top of the handler block documents the rule for future endpoints, and the dispatch decision (with the full async def endpoint inventory) is recorded in docs/decisions.md. Review follow-ups: the invariant comments dropped by the _sync_update_assignments extraction are restored, and the two zero-work handlers the review flagged (root, /_debug/cache) are converted to def for consistency with the convention.

Capacity: threadpool 80 threads vs. pool 60 connections — the 20-thread headroom queues at pool-acquire, a natural backpressure mechanism; the db.py comment arithmetic stays accurate.

Supersedes PR #556 ("API Blocking work async cleanup," June 10). That branch was 120 commits behind and had 15 conflict hunks; of its six change groups, three survived (this handler conversion, VerifyToken.verify, and the comments Turnstile threading). The rest was already fixed (share-endpoint password prints, PR #619), absorbed (graph call sites), or obsolete (fly.toml/WEB_CONCURRENCY — AWS runs single-process uvicorn per ECS task, not Fly's multi-process model).

Tested

• Full backend pytest: 531 passed on the feat/agent-skills-rebuild base (was 337 passed / 117 skipped on the dev base; the graph PR's tests joined the suite). The run also caught three rebase semantic conflicts — await run_in_threadpool sites the base added inside handlers this branch had converted to def — fixed as direct get_graph calls.
• Smoke: GET /, GET /db_is_alive, GET /api/gerrydb/views, GET /api/documents/list, GET /api/cms/content/page/list, GET /api/cms/site_settings — all respond correctly after conversion
• Event-loop blocking repro: under a 30s PostGIS geometry dissolve, a fast probe (GET /db_is_alive) stalled 63s on dev and held 3–11 ms on this branch
• Stress test on the dev stack (paired before/after, plan below)

Dev stress-test plan

Paired runs of the existing Locust harness (backend/stress_test/) against the dev stack, comparing dev as deployed today vs. dev with this PR (and PR #740's base) merged:

1. Baseline: seed via ECS Exec on the dev cluster, then run at SCALE=0.25 (~3,200 users, sized to dev's 1–2 task / db.t4g.small capacity), BASE_URL=https://api.dev.districtr.org.
2. Merge this PR, let the dev deploy land.
3. Re-run with identical SCALE, window, and RNG seed; compare per-route p50/p95/failure-rate CSVs, expecting the largest deltas on mixed traffic (slow geometry work concurrent with fast reads).
4. Cleanup both runs' documents via stress-test-cleanup.

The harness already parameterizes everything needed (STACK_PREFIX, CLUSTER, BASE_URL, RESULTS_BUCKET); this PR adds a "Dev stack run" section to stress_test/README.md documenting the overrides, the capacity sizing, and the seed-slug precheck against the dev DB.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.