The plugin does not work in asynchronous mode

[lifelonglearner127]

On Android, this plugin get my app crashed when I am trying to resize the frame asynchronosuly. If I am resizing synchronously, everything works as expected.

const frameProcessor = useFrameProcessor((frame: Frame) => {
    'worklet'

    runAsync(frame, () => {
      'worklet'
      const resized = resize(frame, {
        scale: {
          width: 192,
          height: 192
        },
        pixelFormat: 'rgb',
        dataType: 'uint8'
      })

      const firstPixel = {
        r: resized[0],
        g: resized[1],
        b: resized[2]
      }
      console.log(firstPixel)
    })
  }, [])


`<Camera`
  ref={camera}
  style={StyleSheet.absoluteFill}
  device={device}
  isActive={isCameraActive}
  video={true}
  frameProcessor={frameProcessor}
/>

Error logs

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x8a2c82b004dfe

[francesco-clementi-92]

Did you manage to find a workaround?

[IkeStudios]

Ran into the same issue on iOS. The root of this async resize problem is actually explained here here. runOnJS (returned by Worklets.createRunOnJs) operates on the default ReactJS thread while the array buffer returned by the resize function is stored in a custom worklet 'VisionCamera.async'. From my basic testing, you can't call resize within a runOnJS function nor can you take the TypedArray returned by the resize function and pass it into the runOnJS function because the intrinsic buffer just becomes undefined.

You might try to drop runOnJS and run your frame processing function within the 'VisionCamera.async' worklet using runAsync. But the problem with runAsync is that you get this annoying bug. Apparently, it was fixed for an earlier version of Reanimated but processNestedWorklets seems to have been removed from the Babel plugin as it was just an experimental feature. Without runOnJS or runAsync, useFrameProcessor alone can't process an asynchronous function: it just ends up crashing at 'await model.run'.

For the latest versions, we just have to settle with a synchronous frame processing function (i.e. using model.runSync, etc.). runAtTargetFps also works as well if you want to implement a cap (the Camera component also has an fps prop too). Just had to make this comment for anyone else new to react-native-fast-tflite because Cursor's not gonna tell you this stuff.

[FieldMarshallVague]

Hi @GrandChieftain Did you manage to get this running well enough? I'm struggling to see how it's useable on-device as I also get these SigSeg errors/segfaults when passing the resized frame buffer to model.run (async or not).

Looking at the other threads covering this issue (inc. the one you linked above), mrousavy recommends proxying/copying the buffer natively to avoid the problem (outside my skillset atm), which I've raised as a feature request. But if there is a 'good enough' approach that you've found, I'd love to hear it! Did you get it working asynchronously at all?

FWIW, the processNestedWorklets config option does appear to work still (in that it doesn't error).

EDIT: I made the changes to my local RNVisionCamera node module (as described in Ghost's diff output) and that got me over the line. I don't know if this is an advisable change (mrousavy seemed to warn about it, but I can't say I'm sure what he was referring to). It's not crashing now, at least and I'm getting native fps in the camera view, with about 1fps running on inference frames.

[jslok]

The changes from TheGhostFish here seem to work so far.
mrousavy/react-native-vision-camera#2520 (comment)

[mrousavy]

VisionCamera V5 will have an all new resizer/converter directly in core.
It is GPU-accelerated via Metal Kernel/OpenGL compute shader, and is about 10-20x faster than vision-camera-resize-plugin.

This issue will likely also be fixed with that