Inconsistencies in the return values (android)

[c-goettert]

To verify / understand the content of the frames, I created a simple demo app that stores the content returned from the resize plugin and just draws it right below the original image (via react-native-skia).

The resize call within my frame processor looks like follows:

const data = resize(frame, {
  scale: {
    width: 320,
    height: 320
  },
  pixelFormat: 'rgb',
  dataType: 'uint8'
})
  
// Note: As I somehow failed to use the arraybuffer-values directly within my component.
// I found that storing the values in a separate array worked for me. So I reshape the frame data like this
// before I use it to draw my skia image:

const arrayData = new Array(width * height * 4).fill(255)
for (let i = 0, j = 0; i < data.length; i += 3,  j += 4) {
  arrayData[j] = data[i] // R
  arrayData[j+1] = data[i + 1] // G
  arrayData[j+2] = data[i + 2] // B
  arrayData[j+3] = 255 // A
}

My camera settings look like this:

  const format = useCameraFormat(device, [
    { videoResolution: { width: 640, height: 480 } },
    { fps: 30 },
    { videoStabilizationMode: 'auto' }
  ])

I can could verify my approach is generally working on iOS, where my test-frame is rendered correctly:

iPhone X:

However, on my Android phone (Pixel 7A, Android 14), the frame seems to be rotated by -90°. Also, as you can see in the image below, the format is returned in bgr instead rgb..

Pixel 7A:

Furthermore, I don't actually want to use the default centre-crop, but the upper square of the image. I have therefore adapted the resize call as follows:

const data = resize(frame, {
  scale: {
    width,
    height
  },
  crop: {
    y: 0,
    x: 0,
    width: frame.width,
    height: frame.width
  },
  pixelFormat: 'rgb',
  dataType: 'uint8'
})

This setting strangely cuts the result into 3 image-areas:

To summarize the issues:

• Everything runs as expected on iOS
• On Android (Pixel 7A, Android 14) there are the following problems
  • The image is rotated
  • The image is bgr despite rgb was requested
  • The crop mechanism seems to have a glitch (possibly related to the rotation?)

I use the following configuration:

• react-native: 0.73.4
• react-native-vision-camera: 3.9.1
• react-native-worklets-core: 0.3.0
• vision-camera-resize-plugin: 2.0.1

If it helps to trace the issue I can gladly add my complete test component.

[mrousavy]

Hey - impressive work that you can draw it to a skia canvas! Is that running smooth? What FPS?

The VisionCamera Skia integration will use GPU Hardware Buffers, so it's gonna run really smooth, but I was wondering how smooth your approach is.

Regarding your issue; hm, well to be honest I don't have a lot of time to investigate this right now, but maybe it's due to the buffers just always being in device native orientation on Android?

I'd gladly accept PRs!

[c-goettert]

Hi, thanks for you answer!
Regarding performance: My current approach is only suitable for debugging purposes. The app only runs with approx. 2-3 frames and crashes after a few seconds (probably a memory leak somewhere). However, it is sufficient for checking the frame content.

screen.record.mp4

Regarding the issue:

maybe it's due to the buffers just always being in device native orientation on Android?

That would also be my guess, is there an easy way to check the native orientation of the device?

[mrousavy]

device.sensorOrientation - but this is not guaranteed to be correct by Android. Some vendors choose to not implement it properly, specifically on Samsungs it's 180 deg rotated.

CameraX has workarounds for all such quirks, in VisionCamera V4 that should be fixed.

[mrousavy]

We now have rotation support in vision-camera-resize-plugin - thanks @rodgomesc! ❤️

So you can now use rotation: '90deg' to fix the rotation issue.

[mrousavy]

I'm not sure why it's blueish though. This might not be a problem of the resize plugin, but rather of the way you display it maybe? Because when I inspect the UIImage of the ARGB 8888 array on the native side it looks correct

[rodgomesc]

I'm not sure why it's blueish though. This might not be a problem of the resize plugin, but rather of the way you display it maybe? Because when I inspect the UIImage of the ARGB 8888 array on the native side it looks correct

it's blueish for me as well if i use rgba on ios, since i don't have time to investigate now my workaround is

pixelFormat: IS_ANDROID ? 'rgba' : 'argb'

[c-goettert]

Strange for me it seems the other way around..
My request format for the resize plugin is pixelFormat: 'rgb'. I interpret it as described in my first post:

const data = resize(frame, {
  scale: {
    width: 320,
    height: 320
  },
  pixelFormat: 'rgb',
  dataType: 'uint8'
})

const arrayData = new Array(width * height * 4).fill(255)
for (let i = 0, j = 0; i < data.length; i += 3,  j += 4) {
  arrayData[j] = data[i] // R
  arrayData[j+1] = data[i + 1] // G
  arrayData[j+2] = data[i + 2] // B
  arrayData[j+3] = 255 // A
}

If I use the same code on iOS and Android (same display method), it looks fine on iOS (tested on iPhone X) but blueish on Android (tested on Pixel 7A). I can fix it for android by switching the blue and red channel (= interpret values as BGR):

const arrayData = new Array(width * height * 4).fill(255)
for (let i = 0, j = 0; i < data.length; i += 3,  j += 4) {
  arrayData[j] = data[i + 2] // R
  arrayData[j+1] = data[i + 1] // G
  arrayData[j+2] = data[i] // B
  arrayData[j+3] = 255 // A
}

[mrousavy]

Maybe it is a bug then in this method:

vision-camera-resize-plugin/ios/ResizePlugin.mm

Lines 183 to 250 in 8a190a3

	- (FrameBuffer*)convertARGB:(FrameBuffer*)buffer to:(ConvertPixelFormat)destinationFormat {
	vImage_Error error = kvImageNoError;
	Pixel_8888 backgroundColor{0, 0, 0, 255};
	// If possible, do all conversions in-memory.
	FrameBuffer* destinationBuffer = buffer;
	size_t targetBytesPerPixel = [FrameBuffer getBytesPerPixel:destinationFormat withType:UINT8];
	if (buffer.bytesPerPixel != targetBytesPerPixel) {
	// The bytes per pixel are not the same, so we need an intermediate array allocation.
	if (_convertBuffer == nil || _convertBuffer.width != buffer.width || _convertBuffer.height != buffer.height ||
	_convertBuffer.pixelFormat != destinationFormat) {
	_convertBuffer = [[FrameBuffer alloc] initWithWidth:buffer.width
	height:buffer.height
	pixelFormat:destinationFormat
	dataType:UINT8
	proxy:_proxy];
	}
	destinationBuffer = _convertBuffer;
	}
	// Source and Destination _might_ be the same buffer.
	const vImage_Buffer* source = buffer.imageBuffer;
	const vImage_Buffer* destination = destinationBuffer.imageBuffer;
	switch (destinationFormat) {
	case RGB: {
	NSLog(@"Converting ARGB_8 Frame to RGB_8...");
	error = vImageFlatten_ARGB8888ToRGB888(source, destination, backgroundColor, false, kvImageNoFlags);
	break;
	}
	case BGR: {
	NSLog(@"Converting ARGB_8 Frame to BGR_8...");
	error = vImageFlatten_ARGB8888ToRGB888(source, destination, backgroundColor, false, kvImageNoFlags);
	uint8_t permuteMap[4] = {2, 1, 0};
	error = vImagePermuteChannels_RGB888(destination, destination, permuteMap, kvImageNoFlags);
	break;
	}
	case ARGB: {
	// We are already in ARGB_8. No need to do anything.
	break;
	}
	case RGBA: {
	NSLog(@"Converting ARGB_8 Frame to RGBA_8...");
	uint8_t permuteMap[4] = {3, 1, 2, 0};
	error = vImagePermuteChannels_ARGB8888(source, destination, permuteMap, kvImageNoFlags);
	break;
	}
	case BGRA: {
	NSLog(@"Converting ARGB_8 Frame to BGRA_8...");
	uint8_t permuteMap[4] = {3, 2, 1, 0};
	error = vImagePermuteChannels_ARGB8888(source, destination, permuteMap, kvImageNoFlags);
	break;
	}
	case ABGR: {
	NSLog(@"Converting ARGB_8 Frame to ABGR_8...");
	uint8_t permuteMap[4] = {0, 3, 2, 1};
	error = vImagePermuteChannels_ARGB8888(source, destination, permuteMap, kvImageNoFlags);
	break;
	}
	}
	if (error != kvImageNoError) {
	[NSException raise:@"RGB Conversion Error" format:@"Failed to convert RGB layout! Error: %zu", error];
	}
	return destinationBuffer;
	}

You could try playing around with those values or hitting breakpoints to see if all values are correct, it seems like some channels are swapped wrong?

[rodgomesc]

If I use the same code on iOS and Android (same display method), it looks fine on iOS (tested on iPhone X) but blueish on Android (tested on Pixel 7A). I can fix it for android by switching the blue and red channel (= interpret values as BGR):

you are right! the rgb buffer is incorrect, right now it's returning bgr we can validate that creating the bitmap right before the return of the SharedValue in ResizePlugin.kt

diff --git a/node_modules/vision-camera-resize-plugin/android/src/main/java/com/visioncameraresizeplugin/ResizePlugin.kt b/node_modules/vision-camera-resize-plugin/android/src/main/java/com/visioncameraresizeplugin/ResizePlugin.kt
index 8c8c0c5..b018bf4 100644
--- a/node_modules/vision-camera-resize-plugin/android/src/main/java/com/visioncameraresizeplugin/ResizePlugin.kt
+++ b/node_modules/vision-camera-resize-plugin/android/src/main/java/com/visioncameraresizeplugin/ResizePlugin.kt
@@ -1,5 +1,6 @@
 package com.visioncameraresizeplugin
 
+import android.graphics.Bitmap
 import android.graphics.ImageFormat
 import android.media.Image
 import android.util.Log
@@ -156,7 +157,25 @@ class ResizePlugin(private val proxy: VisionCameraProxy) : FrameProcessorPlugin(
       targetType.ordinal
     )
 
-    return SharedArray(proxy, resized)
+    val result = SharedArray(proxy, resized)
+
+
+    val bitmap = Bitmap.createBitmap(scaleWidth, scaleHeight, Bitmap.Config.ARGB_8888)
+    val intArray = IntArray(scaleWidth * scaleHeight)
+
+    result.byteBuffer.rewind()
+
+    for (i in 0 until intArray.size) {
+      // note: use bgr instead of rgb  
+      val b = result.byteBuffer.get().toInt() and 0xFF
+      val g = result.byteBuffer.get().toInt() and 0xFF
+      val r = result.byteBuffer.get().toInt() and 0xFF
+      intArray[i] = (0xFF shl 24) or (r shl 16) or (g shl 8) or b
+    }
+
+   
+    bitmap.setPixels(intArray, 0, scaleWidth, 0, 0, scaleWidth, scaleHeight)
+    // breakpoint the line above to visualize the correct preview
+    return result
   }
 
   private enum class PixelFormat {

[pweglik]

Hi @rodgomesc @mrousavy I think I was able to track the bug down to libyuv library and fix it - check out PR: #53

[tarasovladislav]

device.sensorOrientation - but this is not guaranteed to be correct by Android. Some vendors choose to not implement it properly, specifically on Samsungs it's 180 deg rotated.

CameraX has workarounds for all such quirks, in VisionCamera V4 that should be fixed.

Hi, any ideas when VisionCamera V4 will be published ? :)