Upstreaming speculation rules and prefetch to HTML

[domenic]

What is the issue with the HTML Standard?

Chromium has been developing and specifying various speculative loading features over in https://github.com/WICG/nav-speculation.

Recently, Mozilla expressed interest in seeing a subset of these upstreamed to HTML: see mozilla/standards-positions#620 (comment) and preceding comments. We're very excited to work on this from the Chromium side, although it might take a quarter to shift around engineering resources before we start really making active progress.

This issue is for tracking and planning that work. A tentative plan, very open to feedback:

• The initial target is speculation rules and prefetch
  • We would omit prerender-specific pieces for now, e.g. the prefetch + prerender integration parts of the prefetch spec, and the prerender type in the speculation rules spec.
    • (Or maybe we should include the prerender type but allow it to trigger prefetch behavior?)
• We should fix all relevant issues tagged needs spec edits and bug before the upstreaming.
  • Most of these are around service worker integration, and Chromium is working through them now as we implement service worker + prefetch support.
• Similar to the discussion in URL-bar triggered prerendering #7533, we can add a specific carveout to allow browser UI to trigger prefetching, separate from speculation rules.

[domenic]

I did not manage to finish this work before retiring. Here are my outoing thoughts for those who will be pursuing this after me (primarily @domfarolino):

Base speculation rules spec

#11426 was merged successfully and is a good foundation. The minor follow-up PRs w3c/webappsec-csp#776 and whatwg/fetch#1841 seem to be a bit stuck, which is unfortunate since now the spec ecosystem is in an inconsistent state. Someone will need to work on getting them over the finish line.

#11697 is a currently-in-draft addition that I discussed with @vickiez and her team offline, and I am excited for. There is some complexity there around CSP, which may combine with the outstanding PRs mentioned above. (The plan was that, unlike the Speculation-Rules header, we want external <script type=speculationrules> to be subject to CSP.) But I am hopeful it can land quickly regardless. We know developers have asked for this and it will make them happy.

Upstreaming the prefetch spec to HTML

The remaining issue

My plan was to finish the issues in https://github.com/WICG/nav-speculation/issues?q=is%3Aissue%20state%3Aopen%20milestone%3A%22Upstream%20prefetch%22 and then work on upstreaming to HTML. We got very close, with almost all issues fixed or with a candidate fix out.

However, WICG/nav-speculation#367 remains unresolved, and it is pretty significant. Basically, if A prefetches C, and then A navigates to B which redirects to C, the Chromium implementation will use the prefetch of C during the navigation. The existing prefetch spec does not do that. This is not something we can sweep under the rug: we know of large sites using this pattern, e.g. with B being an analytics redirector. So I cannot in good conscience urge other browsers to implement the prefetch spec as-is, because they would get much worse prefetch utilization than Chromium.

Note that @hiroshige-g wrote WPTs which expect the current spec behavior, and which Chromium therefore fails, at https://wpt.fyi/results/speculation-rules/prefetch?label=master&label=experimental&aligned&q=redirect. Fixing this issue will therefore involve flipping those test expectations.

I analyzed the problem space in WICG/nav-speculation#367 (comment) and unfortunately it looks complex. Essentially, the prefetch spec as-is patches HTML on the level of navigations, whereas if we need to handle individual redirect legs, we probably need to get deeper into the Fetch spec's territory. (This is related to some implementation complexities that @hiroshige-g and the rest of the Chromium engineering team have been struggling with over the last year or so, especially as we worked on service worker integration. This note in the spec also seems like a symptom of the problem.)

Regardless, although it will be hard, I am hopeful that sitting down and spending a concentrated working day or two can produce a patch to fix this in WICG/nav-speculation. Once that is fixed, the upstreaming can begin.

Concrete upstreaming plan

In this particular case there are a number of refactors to HTML that could be performed ahead of time, in separate PRs. I would suggest delaying such work until after WICG/nav-speculation#367 is resolved, since that may change the contents dramatically. But if you look at the spec as it is now, stuff like adding delivery type, factoring out create a reserved client, factoring out create a cross-origin opener policy enforcement result for navigation, and factoring out create a navigation request could all be done ahead of time.

Otherwise, as with all upstreaming, I find the best path is to just manually copy over the spec contents, fixing issues, adapting to HTML's style, and restructuring the spec flow with the benefit of years of hindsight as one goes.

Specific section logistics I had envisioned:

• https://wicg.github.io/nav-speculation/prefetch.html#concepts, https://wicg.github.io/nav-speculation/prefetch.html#prefetch-algorithms, https://wicg.github.io/nav-speculation/prefetch.html#cookies were to go in https://html.spec.whatwg.org/#navigational-prefetching . Maybe with subsections, maybe not; maybe with subsections corresponding to those three sections, maybe not.
• https://wicg.github.io/nav-speculation/prefetch.html#html-patches gets distributed throughout other parts of https://html.spec.whatwg.org/#browsers, maybe https://html.spec.whatwg.org/#loading-web-pages-supporting-concepts and https://html.spec.whatwg.org/#navigation-supporting-concepts being worth keeping an eye on as good places to slot miscellaneous things.
• https://wicg.github.io/nav-speculation/prefetch.html#sec-purpose-header probably moves to Fetch, instead of HTML.
• https://wicg.github.io/nav-speculation/prefetch.html#automated-testing gets its own subsection of https://html.spec.whatwg.org/#speculative-loading , maybe right before Security/Privacy considerations. I'm not sure it needs as many subsections as it has right now.

Most of the issues listed in https://wicg.github.io/nav-speculation/prefetch.html#issues-index are not really serious; they are just noting potential improvements which, after several years building this technology, Chromium has always found too low-priority to care about. (https://wicg.github.io/nav-speculation/prefetch.html#issue-9a2d41cc about trying to create a better spec for IP anonymization is somewhat serious but realistically I think it can continue to be delayed as a kind of monkeypatch.) I suggest just omitting most of these during the upstreaming, or converting them into notes.

One area for improvement: something I've found difficult when working with the current spec is clearly understanding what parts are related to prefetch, vs. activation, vs. no-prefetch-involved navigation---since all three use the navigation spec infrastructure. The crucial algorithms are:

• "attempt to populate the history entry's document" patch: intercedes on normal navigations to activate any existing prefetch
• "create navigation params from a prefetch record": called by "attempt to populate the history entry's document" in that activation case
• "create navigation params by fetching" with prefetchRecord argument omitted: the normal no-prefetch-involved navigation fetch case
• "start a referrer-initiated navigational prefetch": actually starts the prefetch, called from the speculation rules machinery
• "create navigation params by fetching" with prefetchRecord argument passed: called by "start a referrer-initiated navigational prefetch" in order to do the prefetch

I think better naming, wrapper algorithms, etc., could help this. Or maybe just more non-normative notes.

Another minor note: much of the prefetch spec was written before we settled on the "speculative loading" name, or the distinction between "prefetching" and "navigational prefetching". Whenever it might be easy to confuse with <link rel=preload>, <link rel=prefetch>, etc., try to update to use those terms instead. (Like I did when upstreaming speculation rules.)

Beyond prefetch upstreaming

We should at some point specify how user agent-initiated prefetches are performed. The prerender spec has both "start user-agent initiated prerendering" and "start a referrer-initiated navigational prefetch", but the prefetch spec does not. It shouldn't be difficult to add something similar. HTML can then mention it, maybe in https://html.spec.whatwg.org/#nav-traversal-ui . This is somewhat important because the behavior is observable to websites, e.g. in the Navigation Timing deliveryType or in the Sec-Purpose header. And Chromium does tons of these today, and I suspect any browser which implements speculation rules prefetch will want to get the benefits for browser-initiated navigations as well like Chromium does.

The prerender spec has a few open bugs still, and is not suggested for the Interop 2026 timeframe. But it would be great to eventually upstream it as well. Especially since it has so many patches onto other specifications, which are annoying to float. With regards to two-implementer interest, we had some tentative expressions in #7533, but that was a long time ago and these days I would probably suggest the stages process.

There's a nascent feature called prerender-until-script which we have heard a good deal of enthusiasm about from web developers, and aligns well with what Gecko expressed interest in at mozilla/standards-positions#620 (comment) . It's possible that this feature might meet the two-implementer-interest bar faster than prerender itself.

[zcorpan]

Thanks for writing that up, @domenic!

Another minor note: much of the prefetch spec was written before we settled on the "speculative loading" name, or the distinction between "prefetching" and "navigational prefetching". Whenever it might be easy to confuse with <link rel=preload>, <link rel=prefetch>, etc., try to update to use those terms instead. (Like I did when upstreaming speculation rules.)

We also have speculative fetch as part of speculative HTML parsing, which is separate. But maybe the HTML parser terms can be tweaked.