fix(opal-server): git resilience — never stuck on an offline repo (PR3)

[dshoen619]

PR3 — Git resilience + the deferred items from the leak series

Closes PER-15157.

Ships PR3's original charter (no git operation can hang the server) plus the six items deferred out of PR2 (#923), and — because the resilience fix required it — the parallel scope loading originally scoped as PR4 (see §6; PR4 is closed as redundant). Design: docs/superpowers/specs/2026-07-19-pr3-deferred-items-design.md.

1. Git resilience — hung remotes can't starve healthy scopes

Scope clone/fetch used pygit2 with no timeout, on the shared default executor. POLICY_REPO_CLONE_TIMEOUT is wired only to the legacy non-scopes path, so scopes mode had no bound at all: boot (preload_scopes()) blocked indefinitely on one unreachable repo, and in steady state a hung op held the per-source lock and a shared-pool thread.

• Hard per-operation timeout (SCOPES_GIT_FETCH_TIMEOUT) on clone and fetch.
• Zombie-aware executor. Each op runs on its own single-use daemon-thread executor; SCOPES_GIT_MAX_WORKERS bounds only live ops via a semaphore. A timed-out op releases its capacity slot immediately — its pygit2 call lingers on a private thread until the OS gives up, but no longer consumes capacity. (A fixed pool starves once lingering threads exceed its size: 40 blackholed repos vs. 10 slots meant healthy scopes never got a thread. That's the test_offline_repo_does_not_block_healthy_scopes gate.)
• git_op_in_flight(source_id) guards every path that would free a pygit2 handle or delete a clone dir while a thread may still be using it.
• Bounded-concurrency scope sweep (per-scope failures isolated) — this is also the parallel-boot fix; see §6.

2. Fleet-wide cache purge — two-phase (deferred items 1 + 3)

PR2's purge was process-local: only the worker serving the DELETE dropped its caches, so the leader — which populates most of repos/repo_locks/repos_last_fetched — leaked until restart. A non-leader DELETE also rmtree'd the shared clone tree, breaking the "only the leader mutates the clone tree" invariant.

New every-worker channel SCOPES_PURGE_CHANNEL (__opal_scope_purge__, following the __opal_stats_* convention), carrying a two-phase purge:

1. Routes publish a purge request (confirmed=False).
2. Only the leader acts on requests: sibling-check under lock_source, then disk work — in a background task, never on the publish path (publish() awaits subscriber callbacks inline, so an inline handler would put a lock wait on DELETE/PUT latency).
3. The leader broadcasts the confirmation (confirmed=True), and that is what every worker's memory handler acts on.

Phase 2 exists because purging on the raw request over-purges: a source shared by a sibling scope had its cache dropped while the sibling still used it. Only the leader can sibling-check, so only the leader may authorize a memory purge.

• Repoint (item 3): PUT /scopes with a changed source_id publishes a purge for the old source. Same channel, same handlers.
• Deferred-purge split: when a git op is still in flight, the clone dir and pygit2 handle wait for the sweep, but the lock/timestamp entries — which the thread never touches — drain immediately and the purge still confirms.
• Mixed-fleet safe by construction: old workers aren't subscribed; confirmed is additive with a False default.

3. Scope-liveness check before clone (item 2)

A sync that loaded a scope before its DELETE and reached it mid-sync re-cloned the dead scope and re-populated the caches. GitPolicyFetcher now takes a liveness probe, checked under lock_source immediately before the clone; ScopesService supplies one that re-reads from Redis and also confirms the scope still points at this source (so a repointed scope's stale sync doesn't resurrect the old clone either). Fails open on store errors.

4. Orphan clone-dir sweep (item 6)

Leader-only reconciliation after boot, periodic, and refresh-triggered syncs: clone dirs under git_sources/ referencing no live scope are reclaimed. One set-difference covers crash orphans, redis-wiped boots, and old-shard dirs after a SCOPES_REPO_CLONES_SHARDS change. A store-scan error aborts the sweep (a transient Redis error must never read as "no scopes exist"); each orphan is re-checked under its own lock and skipped while a git op is in flight.

5. ⚠️ Behavior change — retryable 503 for a live scope's broken clone (item 5)

GET /scopes/{scope_id}/policy:

Case	Before	After
Scope record missing	default scope's bundle	unchanged
Record present, clone invalid/vanished	default scope's bundle	503 + Retry-After: 5
Record present, make_bundle raises raw OSError	unhandled 500	503 + Retry-After: 5

A live tenant was briefly served another tenant's policy. The condition is transient by construction (recovery or the next sync re-creates the clone), so a retryable error is the honest answer. opal-client tolerates this: its PolicyFetcher retries with tenacity backoff and a failed cycle is skipped, not fatal — it does not read Retry-After (uses its own backoff). Direct consumers of this endpoint should expect 503 and retry.

6. Parallel scope loading — the ~20-minute boot fix (folds in PR4)

sync_scopes (boot preload and every periodic/refresh sweep) was serial: one await sync_scope(...) per scope. With a fleet of repos that meant a ~20-minute boot, and — once §1 added a per-fetch timeout — a serial loop would still stall healthy scopes behind each unreachable repo's timeout. So the resilience fix in §1 requires concurrency to actually deliver "one offline repo can't block the others"; the two ship together and PR4 (which planned this same work as a standalone change) is closed as redundant.

The sweep runs in two phases, each asyncio.gather under its own semaphore:

• Phase 1 — distinct repos (network clone/fetch). Bounded by SCOPES_GIT_MAX_WORKERS. Combined with the zombie-aware executor (§1), a hung remote consumes only its own slot until its timeout, then releases it.
• Phase 2 — scopes reusing an already-handled repo (local change-check only). In the common case _should_fetch returns False, so there's no network fetch — just a disk open + change-check + notify. It therefore gets a separate, wider bound (max(SCOPES_GIT_MAX_WORKERS, 32)) instead of inheriting phase 1's network cap; 32 matches asyncio's default thread-pool ceiling, which is what actually limits those disk opens. Any rare re-fetch phase 2 does trigger (a branch missing after phase 1) is still throttled by the inner live-ops semaphore, so the network is never over-driven.

Per-scope failures stay isolated (ScopeNotFoundError from a mid-sweep delete is skipped; other exceptions are logged), and sync_scope re-reads each scope by id right before use, so a delete that lands after the snapshot never re-clones a dead scope.

No new config key — the split is internal. SCOPES_GIT_MAX_WORKERS remains the single operator lever for git concurrency; a dedicated sync-concurrency knob was considered (PR4's SCOPES_SYNC_CONCURRENCY) and deliberately dropped to avoid a redundant, easily-misconfigured second bound.

⚠️ Caveat — the timeout is soft, not a hard kill

The timeout unblocks the event loop and the awaiting coroutine, but the underlying pygit2 call keeps running on its private daemon thread until the OS network timeout. Daemon threads never block shutdown, and git_op_in_flight keeps anything from freeing that repo meanwhile — but worst-case thread count during an outage is SCOPES_GIT_MAX_WORKERS plus the number of lingering timed-out ops. Hard-kill via subprocess is explicitly out of scope.

New config keys (opal-server, server-only, additive)

Env var	Type	Default	Purpose
OPAL_SCOPES_GIT_FETCH_TIMEOUT	float (s)	120.0	Hard timeout for a single scope git clone/fetch. 0 = no limit.
OPAL_SCOPES_GIT_MAX_WORKERS	int	10	Bounds live git ops. Timed-out ops release their slot, so hung remotes never starve healthy scopes.
OPAL_SCOPES_PURGE_CHANNEL	str	__opal_scope_purge__	Worker-to-worker channel for the fleet-wide cache purge.

Invariants (enforced and tested)

1. repo_locks entries are popped only while holding that source's lock.
2. forget_repo / rmtree never run while a git op is in flight for that source.
3. Only the leader mutates the clone tree.

The purge confirmation is published while holding lock_source — it frees this process's cached handle via the inline local subscriber, so releasing first opened a use-after-free against a re-created scope's _notify_on_changes (which holds the handle across an await, then calls set_target() on it).

Verification

• opal-server unit suite: 151 passed. The §6 per-pass split adds sync_scopes_perpass_test.py (2 tests: phase 2 runs wider than the git cap; phase 1 still respects it) — both green, and the related sync/delete/preload suites pass unchanged.
• app-tests/git-leak docker bed: all 10 acceptance gates green — 19/19 in the main phase plus test_offline_repo_does_not_block_healthy_scopes. The bed (not the unit suite) is the real gate: it caught three product bugs unit tests could not — the purge blocking DELETE/PUT on the publish path, the shared-source over-purge, and the thread-pool starvation.
• Bed changes: OPAL_SCOPES_GIT_FETCH_TIMEOUT=10 for realistic serve windows; the delete-vs-inflight-sync churn exclusion lifted (closed by the liveness probe); the repoint gate rewritten as a green guard (it was racing a purge that now completes in ~4ms); allow_worker_restart on the force-recreating boot test; chown after the restoring compose cp.

Known limitation (pre-existing, not introduced here)

test_server_recovers_after_postgres_bounce can fail in a full-file bed run. A worker receives backbone messages only if its broadcaster reader is running, which happens via STATISTICS_ENABLED (default False) or a connected websocket client — subscribe() alone does not start it. The bed has no opal-client service and statistics off, so a non-leader worker there is deaf to the backbone and a publish it buffers during an outage never replays. With clients connected (or statistics on) the reader runs and the replay works — verified in the logs of a passing run. Unrelated to this PR's changes; tracked separately.

Consumer surface

packages/opal-client and packages/opal-common are untouched. No OPAL_* key renamed or removed; all three new keys are additive with behavior-preserving defaults. The 503 above is the one intentional contract change.

🤖 Generated with Claude Code

[linear-code]

PER-15157

[netlify]

✅ Deploy Preview for opal-docs ready!

Name	Link
🔨 Latest commit	fb1b823
🔍 Latest deploy log	https://app.netlify.com/projects/opal-docs/deploys/6a6908d03ec8380008355259
😎 Deploy Preview	https://deploy-preview-924--opal-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

This PR improves opal-server resilience when syncing scope policy repos by ensuring git clone/fetch operations can’t block indefinitely or starve the server’s shared executor.

Changes:

• Add a dedicated, bounded ThreadPoolExecutor and run_in_git_executor(...) helper to run blocking pygit2 operations with an asyncio.wait_for timeout.
• Apply the helper + new timeout config to scope repo clone and fetch paths.
• Add server config keys for timeout and executor sizing, plus focused unit tests for timeout behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/opal-server/opal_server/git_fetcher.py	Introduces dedicated git executor + timeout helper; routes scope clone/fetch through it.
packages/opal-server/opal_server/config.py	Adds SCOPES_GIT_FETCH_TIMEOUT and SCOPES_GIT_MAX_WORKERS configuration.
packages/opal-server/opal_server/tests/git_executor_test.py	Tests config defaults and run_in_git_executor basic behavior.
packages/opal-server/opal_server/tests/fetch_timeout_test.py	Tests that a hanging git op times out quickly (doesn’t block).
.claude/plans/docs/05-config-reference.md	Internal config reference entry for the new env vars and caveat.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zeevmoney]

Review notes — overlap + gate location (planned with the opal-development skill: references/add-config-key.md, references/debug-pubsub.md)

Faithful to the PR3 plan, with two improvements over it: normalizing asyncio.TimeoutError → builtin TimeoutError (so callers catch the builtin on 3.9/3.10 where they're distinct), and moving repos_last_fetched to after a successful fetch — a timed-out fetch no longer falsely marks the source "fresh," and it's lock-safe because _should_fetch runs inside the per-source repo_lock. The two config keys follow references/add-config-key.md (server-only OpalServerConfig, bare names, mandatory descriptions, no double-prefix).

Three things to resolve:

1. Direct overlap with Fix git clone/fetch hanging indefinitely on unreachable repos #875 (PER-13817). Same root cause — pygit2 clone_repository / remotes.fetch going through run_sync with no timeout — and the same edited blocks in git_fetcher.py (_clone and fetch_and_notify_on_changes). They cannot both merge; whichever lands second will conflict. This PR is the stronger of the two:

   • Default-on protection: OPAL_SCOPES_GIT_FETCH_TIMEOUT=120 vs Fix git clone/fetch hanging indefinitely on unreachable repos #875 reusing POLICY_REPO_CLONE_TIMEOUT (default 0 = no timeout, so Fix git clone/fetch hanging indefinitely on unreachable repos #875 protects nothing unless an operator opts in).
   • Thread isolation: a dedicated bounded ThreadPoolExecutor for scope git work, so a hang can't starve bundle-serving — Fix git clone/fetch hanging indefinitely on unreachable repos #875 stays on the shared default executor. (This is the starvation concern from the offline-repo test in test(opal-server): git leak/resilience test environment (PR1) #922.)
   • Best-effort boot: _clone catches (GitError, TimeoutError), logs, and skips the scope rather than letting the caller crash.

   Recommend closing Fix git clone/fetch hanging indefinitely on unreachable repos #875 in favor of this PR.

2. Its regression gate lives in PR1 (test(opal-server): git leak/resilience test environment (PR1) #922). test_offline_repo_does_not_block_healthy_scopes is the fail-now/pass-after gate for this fix and only exists on test(opal-server): git leak/resilience test environment (PR1) #922. So this can't be validated end-to-end until test(opal-server): git leak/resilience test environment (PR1) #922 merges (or is merged into this branch). Suggested order: test(opal-server): git leak/resilience test environment (PR1) #922 → this.

3. This PR is currently check-blocked by branch protection (required checks / review), not by a merge conflict — needs CI green + an approval.

Minor:

• The dedicated pool reads SCOPES_GIT_MAX_WORKERS once, lazily, and caches the executor for the process lifetime — it isn't runtime-reconfigurable and is never shut down. Matches the plan's design; worth a one-line note in the docstring.
• The line numbers cited in .claude/plans/docs/05-config-reference.md for the two keys are stale vs where they actually land on this branch (cosmetic).

[dshoen619]

Thanks @zeevmoney — addressed in bad21c1.

Minor items

• Executor lifetime docstring — added a note to _get_git_executor: SCOPES_GIT_MAX_WORKERS is read once on first use, the executor is cached for the process lifetime (not runtime-reconfigurable) and is never explicitly shut down, matching the PR3 design.
• Stale line numbers — good catch. They'd drifted after the master merge shifted config.py. Fixed the 05-config-reference.md refs: SCOPES_GIT_FETCH_TIMEOUT 150-156 → 196-202, SCOPES_GIT_MAX_WORKERS 157-163 → 203-209.

Things to resolve

1. Overlap with Fix git clone/fetch hanging indefinitely on unreachable repos #875 (PER-13817) — agreed this is the stronger fix (default-on OPAL_SCOPES_GIT_FETCH_TIMEOUT=120, dedicated bounded pool, best-effort boot). Plan is to close Fix git clone/fetch hanging indefinitely on unreachable repos #875 in favor of this.
2. Regression gate in test(opal-server): git leak/resilience test environment (PR1) #922 — agreed; the fail-now/pass-after gate (test_offline_repo_does_not_block_healthy_scopes) lives on PR1, so the plan is to land test(opal-server): git leak/resilience test environment (PR1) #922 → this and validate end-to-end there.
3. Check-blocked — required checks (E2E, builds 3.9–3.12, pre-commit) were green on the prior head and are re-running on bad21c1e; the remaining blocker is the required approving review. A review once it's green would unblock it.

Also confirmed the two improvements you flagged are in place: asyncio.TimeoutError → builtin normalization, and repos_last_fetched moved to after a successful fetch.

[zeevmoney]

Review (PER-15157 / PR3) — git resilience: never stuck on an offline repo

What this PR does. Makes scope git clone/fetch resilient to unreachable repos. It moves scope git work off the shared default executor onto a dedicated daemon-thread ThreadPoolExecutor (SCOPES_GIT_MAX_WORKERS, default 10), wraps each clone/fetch in a soft per-op timeout (SCOPES_GIT_FETCH_TIMEOUT, default 120s) via run_in_git_executor using asyncio.wait (so a timeout unblocks the event loop without cancelling the still-running pygit2 call), adds a per-repo in-flight guard so a lingering timed-out op is not touched concurrently (pygit2 Repository is not thread-safe), records repos_last_fetched only on fetch success, makes scope sync concurrent (bounded by the pool via a semaphore), and adds fork-safety (os.register_at_fork reset + shutdown_git_executor() after pre-fork preload). Plus two unit-test files and a private config-reference doc.

Verdict: REQUEST_CHANGES. Per the severity rule, there is one Postable HIGH finding (credential-redaction bypass in the new log lines) → REQUEST_CHANGES. Independently, the PR is not mergeable: GitHub reports CONFLICTING and git_fetcher.py has a content conflict with master (see Blockers).

The core design is sound and a real improvement over master: isolating git work onto a dedicated pool means a hung clone/fetch can no longer starve bundle serving or the event loop (on master these share the default executor via run_sync, so one offline repo hangs the whole server). The soft-timeout + single-flight + fork-safety mechanics are correct and well-tested (test_busy_key_stays_in_flight_until_call_returns, test_hanging_git_op_raises_timeout, the config-default tests). The two prior open Copilot threads are already addressed by the current code (asyncio.get_running_loop() replaces get_event_loop; repos_last_fetched is now written only after a successful fetch) — not re-raised.

Findings

Postable:

#	Severity	File:Line	Category	Description
1	HIGH	packages/opal-server/opal_server/git_fetcher.py:412 (also 376, 465)	Security	New skip/timeout/clone-error log lines log the raw self._source.url; master redacts every repo URL in this file via redact_url(). Once merged these are the only un-redacted URL logs → credential exposure. redact_url isn't imported.
2	MEDIUM	.claude/plans/docs/05-config-reference.md:27	Doc accuracy	The ceil(offline / workers) × timeout boot/poll bound is optimistic: timed-out ops keep their pool thread until the OS network timeout, so with offline >= workers healthy repos queue behind lingering threads longer than the stated bound.

Informational (not posted inline):

#	Severity	File:Line	Category	Description
3	LOW	packages/opal-server/opal_server/git_fetcher.py:52-91	Maintainability	_DaemonThreadPoolExecutor._adjust_thread_count reimplements CPython concurrent.futures.thread internals. It falls back to super() if _worker/_threads_queues disappear, but a signature change to _worker (name kept, args changed) would pass the hasattr check yet break. Acceptable given the fallback + # pragma: no cover, but a fragility to track across Python upgrades (repo targets 3.9–3.12).
4	LOW	.claude/plans/docs/05-config-reference.md (new tracked file)	Cross-PR coordination	PR #922 adds .claude/ to .gitignore while this PR tracks a file under .claude/plans/docs/. Not a conflict on master today (.claude/ isn't ignored), but once both land the tracked doc sits under a gitignored path — coordinate.

Design note (not a blocker). The "never stuck" guarantee delivered is: the event loop / HTTP surface / bundle serving never block, and each sync slot stalls at most SCOPES_GIT_FETCH_TIMEOUT. It is not that a fixed pool immediately reclaims capacity on timeout — a timed-out op lingers on its thread until the OS network timeout (by design; pygit2 can't be cancelled). For the realistic case (a few offline repos among many healthy) this is fine — the offline ops each hold one lingering thread and the rest of the pool serves healthy repos. The pathological case (offline repos >= pool size) can saturate the git pool; the daemon threads still let the process exit promptly and other server work is unaffected. Finding #2 asks the doc to reflect this precisely.

Blast radius: Production opal-server git-fetcher + scopes sync path — affects every scoped deployment's boot and poll behavior. No client-facing symbol from references/pdp-impact.md §3 is renamed/removed (new module-level helpers + two config keys only; GitPolicyFetcher public shape unchanged), so no PDP import-surface break. Two new OPAL_* keys are additive with sane defaults, no env-name collision, no OPAL_ double-prefix. Pub/sub topology unchanged — this only changes how the leader fetches git; the scope publish path is untouched, so PDP policy-update propagation is unaffected except that offline repos now fail fast (skip + retry) instead of hanging.

Isolation / scope: Well-isolated. All changes serve the stated purpose (git resilience); no unrelated refactors, no half-done work.

Blockers:

• CONFLICTING / not mergeable. git_fetcher.py has a content conflict with master (master added redact_url() to the log lines this PR also edits). Rebase/merge master and resolve — and when doing so, apply redact_url() to the new log lines too (finding #1). Reviewed here against the merge-base three-dot diff (d30e462...d31a0b63), which is unaffected by the conflict.

[Zivxx]

Round-2 review addressed — 16 of 21, bed-validated

Thanks for the thorough pass @zeevmoney. Pushed the fixes (d2ad0455..37021775, 23 commits, grouped by finding with per-comment trailers) and replied inline on each thread.

Fixed + resolved (16): fork/teardown safety (fork-lock triple, reset_caches in-flight guard, keep-markers, bounded drain); zombie cap + gauge; CPython _worker shape-check + python_requires<3.13; honest 409-vs-503; reason-aware fail-open; always-on orphan-sweep timer; hybrid sweep (one snapshot/pass, off-loop is_dir, logged re-check); freeze-exempt purge channel; wrapped config desc; ported TOCTOU/shard/wiring/route tests + restored the defensive WARNING assertion (and fixed a real CancelledError-swallow the tests surfaced); public docs for the new keys + dropped the private .claude doc.

Validation: opal-server unit suite green, and the git-leak docker bed is 21/21 with zero invariant violations. The bed earned its keep — it caught that an attempted local-memory-purge fix for service.py:213 minted stray repo_locks (I4) and over-purged shared sources, so I reverted it (37021775).

Left open, deferred to a follow-up (lock-lifecycle / "Plan B"): the two HIGH use-after-free items (worker-side lock_source + pop-last ordering + epoch) and the purge-channel auth (fail-closed + server-minted marker) — these want a small delivery/verification design pass; the dead clone_path field rides along with that schema change. And the memory half of service.py:213 (draining a client-less non-leader) lands there too; its disk half is already fixed here via the always-on sweep timer.

Happy to do a quick sync on the purge-delivery split for the UAF fix before I implement it.

[Zivxx]

PR #924 — post-review hardening pass (for @zeevmoney)

Ran a 3-round adversarial + constructive review over the whole branch (correctness,
test coverage, best-practices, observability, security), then verified every finding
adversarially and re-triaged severity by hand. Headline: no critical / data-loss /
crash / fleet-break bugs in the concurrency-heavy diff. Fixes below landed in
ff5904ba; 199 opal-server unit tests + 21 git-leak bed tests green.

Fixed in ff5904ba

Security

• Purge channel is now server-to-server only — addresses your [HIGH] on
  purge.py:73. New _verify_permitted_topics-style channel restriction rejects
  SCOPES_PURGE_CHANNEL publish/subscribe from any external RPC peer. Confirmed the
  full chain: RpcEventServerMethods.publish is client-callable and the
  permitted_topics guard default-allows tokens with no such claim, so a forged
  fleet-wide purge was reachable from any PDP. Legitimate publishers (delete/repoint/
  sweep + the cross-server broadcaster relay) all notify() with channel=None and
  bypass the gate; clients only ever publish STATISTICS_ADD_CLIENT_CHANNEL, so
  no client is affected and no redeploy is required — pinned by a test.

Correctness

• _get_current_branch_head no longer conflates permanent vs transient. A missing
  ref (KeyError) is permanent → BranchHeadNotFoundError → 409; a transiently gutted
  object store (pygit2.GitError) now propagates → retryable 503, instead of telling a
  self-healing scope "not retryable". Serving-path only; get_commit_hash and the
  notify path are untouched. The real raise path is now covered (was mock-only).

Observability (Datadog)

• Log the preload drain timeout instead of discarding drain_git_ops()'s bool —
  that is the exact condition that carries lingering git ops across the fork, and it
  was silent.
• Continuous git_ops_in_flight gauge (not just the one-shot log at the zombie cap).
• Orphan-sweep heartbeat summary so a healthy no-op sweep is visible.
• metrics.event on the 503/409 policy-unavailable responses.

Hygiene

• Clean-log GitConcurrencyLimitExceeded (no per-scope traceback during a zombie-cap
  outage); \Z (not $) in the source_id regex (trailing-newline bypass); skip the
  always-on orphan-sweep timer when polling already sweeps (no duplicate scans/
  broadcasts); fix the misleading ScopePurgeCommand.reason / clone_path comments;
  drop an unused import.

Still deferred to Plan B (unchanged — flagging for clarity)

These remain open with our earlier "deferred" replies; this pass did not implement
them, so they stay open:

• purge.py:62 [HIGH] — worker frees the shared pygit2 handle outside lock_source
  (Plan B lock-discipline).
• purge.py:249 [HIGH] — repo_locks popped before the confirmation publish (same
  Plan B change: pop last).
• service.py:213 [MEDIUM] — DELETE's memory-half still depends on the leader's
  reader; only the disk half (always-on sweep) is done.
• purge.py:44 [LOW] — clone_path's comment is now accurate, but dropping the unread
  wire field is still bundled into the Plan B schema change.

Also from the review, not blocking

One latent medium we chose to measure before fixing: a preload git op that outlasts
the fetch timeout + the drain can persist in the gunicorn master across the fork and
race a worker on the shared clone dir. Trigger is narrow (op must be actively writing,
not just hung) and self-healing; the new drain-timeout log now tells us how often the
window even opens in prod before we invest in the temp-clone+rename fix.

[zeevmoney]

Changes requested — 2 HIGH, 4 MEDIUM.

Blocking:

• HIGH packages/opal-server/opal_server/pubsub.py:388 — the ALL_TOPICS exemption reopens the purge channel on the subscribe path; verified against the installed fastapi_websocket_pubsub that an external peer subscribed as ALL_TOPICS (bare or in a list) receives every ScopePurgeCommand
• HIGH packages/opal-server/opal_server/scopes/purge.py:305 — the sweep's snapshot filter is O(dirs × scopes) with two sha256 per pair and no await; measured 1.25s of blocked event loop at 2000 scopes, growing quadratically

Non-blocking:

• MEDIUM packages/opal-server/opal_server/tests/purge_channel_test.py:112 — the only test guarding the confirmed gate passes with the gate deleted (verified by mutation; whole suite stays green)
• MEDIUM packages/opal-common/setup.py:76 — opal-client was not capped alongside opal-common/opal-server, so pip install opal-client on 3.13 fails on a transitive dependency
• MEDIUM packages/opal-server/opal_server/scopes/task.py:43 — the leader purge subscription has no stop() teardown and _pending_purges is not cancelled or awaited, so a shutdown can abandon an rmtree and skip the confirmation publish
• MEDIUM packages/opal-server/opal_server/tests/purge_channel_test.py:657 — the third test dropped in the delete_scope_cache_purge_test.py rewrite (recreate-after-delete) still has no equivalent

Details are in the inline comments on each line.

---

PR description accuracy (not inline findings — the description does not match the diff):

• The "New config keys" table lists 3 keys; config.py adds 6. SCOPES_GIT_PRELOAD_DRAIN_TIMEOUT (10.0), SCOPES_GIT_MAX_ZOMBIES (40) and SCOPES_ORPHAN_SWEEP_INTERVAL (300) are missing. All six are documented correctly in configuration.mdx — verified programmatically that every documented default matches config.py.
• "The 503 above is the one intentional contract change" — api.py also adds a 409 CONFLICT for BranchHeadNotFoundError, a case that previously fell through to _generate_default_scope_bundle and returned 200. That is a second contract change and it is not in the §5 table. Worth noting for direct consumers that opal-client cannot act on the "non-retryable" distinction: throw_if_bad_status_code turns any non-200 into a ValueError, so 409 and 503 are both retried identically by its tenacity policy.
• "packages/opal-client and packages/opal-common are untouched" — packages/opal-common/setup.py is touched (python_requires narrowed to <3.13), which is a packaging change for downstream consumers.
• The "⚠️ Caveat" says worst-case thread count is SCOPES_GIT_MAX_WORKERS plus lingering timed-out ops, with no bound. SCOPES_GIT_MAX_ZOMBIES now bounds exactly that, so the caveat understates the code.
• "opal-server unit suite: 151 passed" — the suite is 199 on this head (green, re-run locally).
• The linked design doc path docs/superpowers/specs/2026-07-19-pr3-deferred-items-design.md does not exist in the repo.

Verified clean: all 199 unit tests pass on this head; pre-commit (black / isort / docformatter / codespell) passes on the diff; the six new config defaults match their configuration.mdx entries exactly; no new log line carries an unredacted URL or credential (only server-local filesystem paths); _reject_external_purge_channel correctly does not fire for server-internal publishes (PubSubEndpoint.publish passes channel=None, and restrictions run only if channel:); the endpoint's publisher id and subscriber id are distinct, so the "confirmation runs the local subscriber inline" rationale holds; the executor object-lifetime chain keeps a timed-out op's executor alive to completion without leaking, and the live-ops semaphore is released exactly once on both the success and timeout paths.

[zeevmoney]

Second pass — changes still requested. 2 further HIGH, 5 further MEDIUM.

Blocking:

• HIGH packages/opal-server/opal_server/scopes/purge.py:96 — the every-worker handler calls forget_repo → Repository.free() with no lock, so on every process except the publisher it is the exact use-after-free that purge.py:257-264 locks the confirmation publish to prevent; git_op_in_flight does not cover _notify_on_changes or the run_sync(_get_valid_repo) executor thread
• HIGH packages/opal-server/opal_server/git_fetcher.py:276 — "until the OS network timeout" is enforced by nothing (no GIT_OPT_SET_SERVER_TIMEOUT, no socket read timeout, libgit2 1.7 defaults to none), so a black-holed remote pins its _git_busy marker for process life; at SCOPES_GIT_MAX_ZOMBIES such entries every git op is refused fleet-wide, logged only as warning-level backpressure

Non-blocking:

• MEDIUM packages/opal-server/opal_server/scopes/purge.py:340 — the sweep rmtrees any subdirectory of git_sources/ without the _SOURCE_ID_RE check every other deletion path uses
• MEDIUM packages/opal-server/opal_server/scopes/task.py:57 — the orphan-sweep timer is skipped when polling is on, contradicting config.py:513-516, task.py:95 and the .mdx
• MEDIUM packages/opal-server/opal_server/scopes/service.py:292 — max(..., 32) is a floor, so SCOPES_GIT_MAX_WORKERS cannot lower phase-2 concurrency; config.py:230-231 claims it can, and phase 2 shares the executor that serves policy bundles
• MEDIUM packages/opal-server/opal_server/scopes/task.py:64 — the boot sync is outside the try that protects the sweep; a store error kills the task with no log and, at the default POLICY_REFRESH_INTERVAL=0, no retry
• MEDIUM packages/opal-server/opal_server/git_fetcher.py:502 — neither the in-flight sync guard nor "a timed-out fetch must not stamp repos_last_fetched" has a test; both mutations leave the suite green

Details are in the inline comments on each line.

---

Correction to the previous pass. The suggested fix on scopes/task.py:43 was wrong and that comment has been updated in place. PubSubEndpoint.subscribe registers all server-side subscriptions under one shared _subscriber_id, and EventNotifier.unsubscribe deletes by subscriber id per topic, so the unsubscribe([SCOPES_PURGE_CHANNEL]) I proposed would also have removed the every-worker handle_purge_message registered at server.py:395. The corrected comment proposes a dedicated subscriber id (or an _is_leader flag) instead. The finding itself stands; only the remedy changed. Also narrowed there: there is no in-process restart path, so the handler cannot double-subscribe — the real exposure is the window between the flock being released and the old worker dying.

Pre-existing, not introduced here, but worth a decision. api.py:308-315 still returns _generate_default_scope_bundle (HTTP 200, carrying the default scope's policy modules) when a scope record is missing, while the new comment at api.py:352-356 states the principle that "serving the default scope's bundle here would hand a live tenant another tenant's policy". _allowed_scoped_authenticator (api.py:89-98) authorizes the caller for the requested scope_id only, never for default. If tenant B's record is briefly absent — a Redis failover, a partial restore — B's PDP receives 200 plus the default scope's policy and loads it into OPA. The PR applies its own stated rule to the broken-clone case and not to the missing-record case, and scope_policy_fallback_test.py:117-132 pins the latter. Out of scope for this PR, but the two halves should not stay inconsistent indefinitely.

Also flagged, not filed inline (lower value, or better handled as a batch): metrics.event on the 503/409 paths is a DogStatsD event rather than a counter, emitted per failed request with a caller-controlled scope_id tag — during a clone outage retrying PDPs bury the signal; the confirmation publish at purge.py:266 and :349 does an unbounded backbone round trip inside lock_source, so a hung broadcaster pins that source's lock (there is mitigating precedent at git_fetcher.py:778); sweep_orphans and purge_source_if_unshared measure at cyclomatic complexity 17 and 11, and fetch_and_notify_on_changes grew from 82 to 133 lines with early returns carrying load-bearing invariants six levels deep; asyncio.gather in sync_scopes materializes one task per scope with no bound on scope count, including in the gunicorn master during preload; handle_purge_message reads the module-global BASE_DIR while LeaderScopePurger uses its injected base_dir; shutdown_git_executor() shuts down no executor and _release_once's guard is currently unreachable; git_executor_test.py mutates the process-global _git_busy without an autouse fixture and asserts on wall-clock elapsed time, which is flaky by construction under -p xdist or a randomizer.

Verified clean on this pass: the two-phase design holds — publish() runs local subscribers inline, and the publisher's and subscriber's ids are distinct so the leader really does receive its own confirmation; the replay case is sound with no epoch token needed (a scope re-created before the sibling check is seen under lock_source, so no confirmation is published at all — I ran this: the recreated scope's clone dir and cached handle both survived, zero confirmations published, while the truly-deleted control purged and confirmed); disk purges lost to a leaderless window are genuinely backstopped by the sweep; _confined_clone_path rejects traversal and the wire-supplied clone_path never reaches rmtree or free(); the publish-direction channel guard works for single, mixed and list topic forms; mixed-fleet is safe in both directions; route authz is unchanged and correct; Python 3.9 and pydantic v1 compatibility hold across all new constructs; the live-ops semaphore is released exactly once on every exit path and no executor leaks.

[Zivxx]

Round-2 review addressed — all 13 threads fixed in fb1b823d

Thanks @zeevmoney — both submissions from this pass (2 HIGH + 4 MEDIUM, then 2 further HIGH + 5 further MEDIUM) are resolved. CI is green on fb1b823d (pre-commit, build 3.9–3.12, E2E + Alpine), and the fixes were additionally verified across the local git-leak bed (invariant I4 clean).

HIGH

• pubsub ALL_TOPICS reopens purge channel — _reject_external_purge_channel now rejects ALL_TOPICS and SCOPES_PURGE_CHANNEL on the subscribe path.
• sweep O(dirs×scopes) blocks the leader loop — precomputed live_source_ids set (O(scopes)) + periodic await asyncio.sleep(0).
• every-worker UAF freeing the pygit2 handle unlocked — handle_purge_message now frees under lock_source on every process, and pops its minted repo_locks entry (I4).
• "until the OS network timeout" enforced by nothing — the claim was the bug: corrected to soft timeout across git_fetcher.py / config.py / configuration.mdx, with the real hung-remote bound (SCOPES_GIT_MAX_ZOMBIES + daemon threads) documented.

MEDIUM

• confirmed-gate test now uses a valid source_id (fails with the gate deleted)
• opal-client capped <3.13 to match its hard dep
• stop() unsubscribes + drains the purger (no abandoned rmtree)
• recreate-after-delete now has a unit test
• sweep rmtree validates via _confined_clone_path
• orphan sweep runs unconditionally on its own interval (docs now match)
• phase-2 concurrency is max(1, SCOPES_GIT_MAX_WORKERS) — a true ceiling
• boot sync_scopes() wrapped so a raise doesn't silently kill the task
• both _get_current_branch_head safety behaviours now have tests

Ready for re-review against fb1b823d.