feat(render): networkidle2 in-flight wait, replacing the completion-reset idle probe#120
Merged
Merged
Conversation
…eset probe The wait_network_idle probe ran in-page on a PerformanceObserver: every resource *completion* reset its 500ms quiet timer. Analytics beacons and polling complete more often than every 500ms on much of the open web, so those pages always rode the 12s hard cap — measured live on a page with 200ms beacons: 15.6s with the old probe, 4.0s with this change. Count in-flight requests instead, Python-side over CDP Network events (Puppeteer's networkidle2 semantics, the industry default for capturing arbitrary pages): Network.requestWillBeSent opens a request, loadingFinished/loadingFailed closes it, and readiness means "page loaded AND <=2 requests in flight for 500ms", capped at LOAD_TIMEOUT_MS as before. Persistent connections (long-poll, websockets — different CDP methods, never counted) no longer block; SPA hydration bursts (>2 concurrent) still hold capture open. The quiet clock runs from navigation, so a page that is already quiet when `load` fires proceeds immediately — static pages now pay ~nothing where the old probe always paid >=500ms. Mechanics: the watcher owns the websocket between Page.navigate and the in-page height probe (the per-page CDP flow is sequential), because _cdp_send discards event frames while awaiting a response. An initial raw document.readyState probe catches pages whose load event fired before the watcher attached. _readiness_expr drops its in-page idle step and parameter; Network.enable in _setup_page is now load-bearing for the watcher and commented as such. The turbo path still bypasses the idle wait (it has no watcher yet) — now with an explicit log line instead of a silent fallback. _NetIdleState is pure and clock-injected; tests cover the tolerance, load gating, beacon timing, the hard cap, and the pre-loaded-page probe with a fake websocket.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
wait_network_idleprobe ran in-page on aPerformanceObserver: every resource completion reset its 500ms quiet timer. Analytics beacons / polling complete more often than every 500ms on much of the open web, so those pages always rode the 12s hard cap.Measured live on a page firing a beacon every 200ms:
Change
Count in-flight requests instead, Python-side over CDP
Network.*events — Puppeteer's networkidle2 semantics, the industry default for capturing arbitrary pages (tolerate ≤2 pending requests for 500ms, hard-capped at 12s as before):requestWillBeSentopens,loadingFinished/loadingFailedcloses; readiness = loaded AND ≤2 in flight for 500ms.loadfires proceeds immediately — static pages now pay ~0 where the old probe always paid ≥500ms.Page.navigateand the in-page height probe (per-page CDP flow is sequential;_cdp_senddiscards event frames, so the watcher must be the reader while the page settles). An initial rawdocument.readyStateprobe catches pages that loaded before it attached._NetIdleStateis pure and clock-injected; tests cover tolerance, load gating, beacon timing, the hard cap, and the pre-loaded-page path via a fake websocket (8 new tests, no Chrome needed).Verified end-to-end (real Chrome)
example.comwith/without the flag: complete tiles, ~0.5s idle overhead on a single-request page.Not in this PR
The turbo (fast_cdp) path still bypasses the idle wait — it needs the same watcher wired into its
_Connevent loop, and I can't benchmark turbo on this machine (/dev/shm+ patched headless_shell). The silent fallback is now an explicit log line; turbo idle support is a scoped follow-up.