large scenes can trigger HTTP2 GOAWAY:REFUSED_STREAM error frames

[sdumetz]

This only happens when loading a very large scene, sometimes once or twice on first load, cascading to 100s of errors when reloading the scene. Furthermore, it only manifests itself when the service is behind nginx.

During the initial (thumb quality) loading phase, voyager can notify of failed to fetch (internally, net:ERR_HTTP2_SERVER_REFUSED_STREAM) in chrome and a very unhelpfulTypeError: Content-Length header of network response exceeds response Bodyin firefox (misleading, because it may happen on requests that are gzip-encoded on the fly and have noContent-Length` header...).

After some debugging I've found we were hitting nginx's keepalive_requests cap, which cause a GOAWAY:REFUSED_STREAM HTTP/2 frame from which Voyager can't cleanly recover.

The problem is mitigated at the proxy level by raising keepalive_requests (defaults to 1000, raised to 100000 in my case) - which explains why you will no longer see the mentioned errors on the linked scene.

Though the issue is server-dependant and the mitigation is reasonable-enough, RFC9113 for REFUSED_STREAM states:

Requests that have not been processed have not failed; clients MAY automatically retry them, even those with non-idempotent methods.

So maybe we should retry on such an error. Additionally, digging into this got me thinking we might want to dial down a bit how aggressively we initially fetch the thumbnails, or we might someday get a (admittedly very funny) GOAWAY ENHANCE_YOUR_CALM frame.

I've made a branch with possible fixes for those two issues. The first one should be safe enough. The second one will probably only affect LOD scenes?

[gjcope]

Thanks @sdumetz ! Interesting info and useful for anyone who might be in the same situation.

The retry on server error will likely be rare for us but seems like a reasonable implementation for that case.

I'm a little more hesitant about capping the concurrent requests. And if we do it, that 6 is the right number... For your large model examples that are hitting the 1k request limit (just curious) how many discrete objects are there? I assume quite a few to accommodate the LOD but 1k still seems like a healthy number.

[sdumetz]

The largest scenes holds 3 to 5 hundred individual models. To be honest, I'm not even sure how we managed to hit that limit, especially since it sometimes happened on first load. Raising the ceiling was what fixed the error but I can't 100% explain what was happening.

6 concurrent requests is what we do for the LOD swap algorithm. That was chosen because it was the default max number of pre-HTTP/2 concurrent request in all browsers (I think only ever enforced on plain HTTP, not HTTPS, though). In any other case (probably 100% of real-world usage by now...) that's just an arbitrary number.

500 concurrent requests is certainly not ideal. There is no speed benefit to expect by fetching more than a handful of models at once. I'm not sure we'd loose any time if we capped parallel downloads at 1.

With all that said, I'm not sure that many models should be considered At this kind of size everything starts breaking down anyways (though it still kind-of-works on mobile devices...). So it's probably harmless to just handle the retry case and leave the rest for now. If hundreds of models ever becomes considered "normal" for voyager to handle, then it might be time to revisit this.