Infrequent Headless Aborted

[nathan-ahn]

Description

Hey William! I have another fun headless edge case to report. We're seeing intermittent headless crashes for the past few weeks. If I were to ballpark, these are on the scale of 1 in 10,000 renders. We've just recently upgraded to v2.5.3 (from v2.2.14) and now have a somewhat more useful error message to report.

Before this upgrade, we've been seeing vague error messages of [RuntimeError: memory access out of bounds] { componentStack: '\nu8t' } and table index is out of bounds (which may or may not be related to this new issue). The most recent errors that we've been seeing are Aborted(). Build with -sASSERTIONS for more info. and Cannot read properties of null (reading 'getCanvas').

These are running on AWS Lambda infrastructure, which is likely where the safeguards are coming from to prevent out of bounds memory access.

I apologize bringing this issue up with such little reproducibility. I'm open to any hints as to where we can experiment, or steps we can take to improve future debugging. Totally understandable if this issue is out of scope considering the lack of reproducibility here.

React Native Skia Version

2.5.3

React Native Version

N/A

Using New Architecture

• Enabled

Steps to Reproduce

Large-scale rendering on AWS Lambda. Unfortunately, this issue is so intermittent and infrequent that I don't think it'd be feasible to reproduce. We'll likely need a few days of production deployment to validate any potential solutions here.

Snack, Code Example, Screenshot, or Link to Repository

N/A

[wcandillon]

hey @nathan-ahn 👋

The Skia team just pushed a new version of CanvasKit, so it's always good to use a version of CanvasKit that is closer to the Skia version we run on native. Now that new version has a breaking change with Path (paths are now immutable). I have it ready as a PR but I think I will do an easier migration path where first we introduce the new API and throw a warning for mutable path methods and then remove it completely.

Now to the error itself. It looks like at some point the surface is null (null.getCanvas()). And probably the surface create fails with an assertion related to the memory available? What kind of safeguards/fallbacks do you have in place there? Potentially when the surface creation fails, you could report a lot of useful information.

Just to triple check, you only use Surface.MakeOffscreen right? and not Surface.Make?

[wcandillon]

We can also look at the surface creation process, which assertions may fail. and another possibility eventually is to use the debug build of canvas (much bigger) that will give you much more information

[nathan-ahn]

Makes sense! We'll upgrade once the new version is ready.

From what I can tell, the safeguards are AWS Lambda system safeguards. This means that our work is getting killed immediately upon breaching the safeguards without opportunity to log an error with context. The null.getCanvas() wasn't killed by the system, so we do have slightly better error reporting there.

I believe so: I didn't find any usages of Surface.Make across our codebase, and only one usage of Surface.MakeOffscreen. Are there any helper functions that I should look out for that use either of these in the implementation?

I just gave the debug build a shot, but it end up crashing our service 100% of the time on our dev environment with Cannot read properties of undefined (reading 'activeTexture'). Definitely not going to risk pushing that up to production. I'm really surprised though since the switch should be stable. I just switched the file from canvaskit-wasm/bin/full/canvaskit.wasm to canvaskit-wasm/bin/profiling/canvaskit.wasm without any version changes.

Here's the code snippet form our codebase that may be relevant to both the Surface.MakeOffscreen and surface.getCanvas() question:

const surface = Skia.Surface.MakeOffscreen(exportSize, exportSize);
if (!surface) {
    throw new Error("Failed to create offscreen surface");
}
const canvas = surface.getCanvas();

On the bright side, this section of server rendering is deprecated. The only invocations that we get here are from older clients. It wouldn't be an issue for us to delete this section, which would hopefully mean we're no longer impacted by the underlying bug here.

[wcandillon]

Does Lambda has GPU support? If it does, what kind of drivers are available? If doesn't maybe it is worth trying to use the non GPU backend? Which might be already the case? I may have lost context here.

[nathan-ahn]

We're running on pure CPU with no weird workarounds or performance optimizations on the rendering side.

We do have a separate service which uses workers for Skia rendering which is also failing intermittently, but in this case, this is on the main NodeJS thread.