Need observability on the prefetching/prerendering page

[stoyan]

The page that initiates the prefetch or prrender should know about any attempted and (even more importantly) failed attempts.

Ideally the information available in devtools under Application -> Speculative loads -> Speculations should be available to the page to report back, like:

URL: example.org
Action: Prefetch
Status: Speculative load failed.
Failure reason: The prefetch was discarded because ...

Could be Reporting API and/or JavaScript API

[domenic]

Can you state more about your use case? We generally do not want to expose this as this goes against the nature of the feature as a progressive enhancement.

See also previous discussion in #306.

[stoyan]

Thank you Domenic! The thing is to roll prefetching on a big site with millions of pages is inherently a risky endeavor. The goal is to have as much visibility into what's happening as possible. If some prefetches fail in edge cases/pages/situations that were not tested before going live, the developers would like to know.

I know maybe a combination of Sec-Purpose and usual server-side logging (e.g. for 404s) can help, but there might be cases where the request doesn't even reach the server, like maybe http vs https and so on, client-side issues. For example one case is a prefetch failing because the URL was rewritten on hover. Something like this I don't think can be detected server-side

[domenic]

Thanks! That is very helpful. So it sounds like some sort of aggregate reporting API, which doesn't necessarily give insight into individual preloads, but instead gives you statistics (and maybe failure reasons, if possible) would work. Am I understanding correctly? That would be possible to do while still preserving the nature as a progressive enhancement.

[stoyan]

Well, not in aggregate, but a concrete failures, so we can debug what happened. For example we'd like to know:

• https://example/page.html tried to preload example/page2.html but failed because "The URL was not eligible to be prerendered because its scheme was not http: or https:" Oh, the protocol is missing in this link, I'll go fix this
• https://example/page.html tried to preload https://external/page2.html but failed because "The prerendering navigation was blocked by a Content Security Policy." Wait why are we preloading external sites, something's off with our regex that matches page2.html

[domenic]

Hmm. That level of granularity is tricky, because it makes it easy for people to start depending on specific preloads succeeding or failing, which is against our goal. (It's also just a ton of engineering work to specify and maintain all those failure reasons.)

Is there a reason this kind of testing needs to be done in production? Would better automated testing APIs help with these use cases instead?