WE2 Ethos-U output diverges from CPU TFLite on a real MobileFaceNet subgraph
Hi Himax team,
We found a reproducible CPU-TFLite vs Grove Vision AI Module V2 / WE2 Ethos-U numeric mismatch while validating an INT8 face embedding model. The attached MRE keeps the real upstream activation distribution from the original model; it is not a random standalone operator test.
Hardware / software
- Board: Seeed Grove Vision AI Module V2
- SoC / accelerator: Himax WE2, Ethos-U55-64
- Vela:
5.0.0
- TFLite schema compiler used for graph surgery:
flatc 25.12.19
- Repository baseline during local testing:
sscma-example-we2 commit 5f3d985707d32c55ec5e1b59a2bb4c2fa18ab34f
- Firmware build target:
GROVE_VISION_AI_V2
- Test input: deterministic fixed INT8 input generated with seed
7
What is in the MRE
The package himax_w600k_ch30_mre.zip contains:
- Prefix probe models:
w600k_stage96_pairft.op28_proj16_ch30only.tflitew600k_stage96_pairft.op29_proj16_ch30only.tflite- their Vela-compiled versions
- WE2 FACEDBG captures for the same fixed input seed
- CPU reference intermediate tensors for original tensors 166, 167, 168
run_compare.shscripts/analyze_ch30.pyscripts/compare_facedbg_tensor.py
The probe models keep the original model prefix from input through:
op27 Conv2D -> op28 PReLU -> op29 DepthwiseConv2D
The final debug projection selects only global channel 30 into output slot 0. Other slots are zero projections and match exactly, so the debug projection itself is not the source of the mismatch.
Expected behavior
For the same fixed INT8 input and the same Vela model, WE2 Ethos-U output should match local CPU TFLite output within normal INT8 tolerance. This is especially important for face recognition: enrollment may be done on a host CPU TFLite pipeline while recognition runs on WE2.
Actual behavior
op29 channel 30 diverges reproducibly:
op29_ch30only:
diff_abs_max=3
diff_abs_mean=0.0359 # over 12544 output bytes; only slot 0 is non-zero projection
diff_nonzero=394/12544
normalized_cosine=0.997164
status=FAIL
Looking only at the selected channel slot:
op28_ch30only slot0:
max_abs=5
mean_abs=0.6633
nonzero=305/784
op29_ch30only slot0:
max_abs=3
mean_abs=0.5740
nonzero=394/784
The full 128D embedding for the original model is much worse:
full embedding:
diff_abs_max=12
diff_abs_mean=3.3906
diff_nonzero=116/128
normalized_cosine=0.970066
Why this MRE is meaningful
The issue is not visible from a random standalone DepthwiseConv2D test. The mismatch needs the real upstream activation distribution around:
op27 Conv2D -> op28 PReLU -> op29 DepthwiseConv2D
CPU op27 channel-30 distribution around zero_point=-12:
negative: 404 / 784
zero: 5 / 784
positive: 375 / 784
The mismatch is not isolated to the PReLU negative branch:
op28 grouped by CPU op27 channel-30 sign:
negative mean_abs=0.6906
positive mean_abs=0.6213
op29 grouped by CPU op27 channel-30 sign:
negative mean_abs=0.5718
positive mean_abs=0.5733
Channel 30 is also not extreme by op29 per-channel weight scale. It ranks 44/80, so the evidence points more toward runtime rounding/requantization behavior under the real activation distribution than toward an obviously abnormal channel scale.
Experiments already tried
We tried two simple graph-surgery fixes:
-
Insert explicit QUANTIZE after op28 using the original tensor-167 quant params.
- Vela compiled it fully to NPU (
CPU operators = 0).
- Result was unchanged:
op29_ch30only cosine=0.997164.
-
Recalibrate tensor-167 scale only.
0.002983 -> 0.004712: local ch30 probe improved to cosine=0.999130, but full embedding only reached 0.973331.0.002983 -> 0.005966: local ch30 probe improved to cosine=0.999455, but full embedding dropped to 0.965706.
So a simple identity quant boundary or single-tensor scale edit is not sufficient.
How to reproduce offline comparison
After extracting the MRE package inside the repository root:
cd sscma-example-we2/artifacts/himax_w600k_ch30_mre
chmod +x run_compare.sh
./run_compare.sh
This compares the provided WE2 FACEDBG captures against CPU TFLite and prints the per-channel distribution analysis.
Request
Could you help check whether this is an Ethos-U runtime / Vela scheduling / quantized operator rounding issue on WE2, especially around the real PReLU -> DepthwiseConv2D boundary? If possible, please also advise whether a specific Vela/TFLM/Himax SDK version is expected to avoid this divergence.
WE2 Ethos-U output diverges from CPU TFLite on a real MobileFaceNet subgraph
Hi Himax team,
We found a reproducible CPU-TFLite vs Grove Vision AI Module V2 / WE2 Ethos-U numeric mismatch while validating an INT8 face embedding model. The attached MRE keeps the real upstream activation distribution from the original model; it is not a random standalone operator test.
Hardware / software
5.0.0flatc 25.12.19sscma-example-we2commit5f3d985707d32c55ec5e1b59a2bb4c2fa18ab34fGROVE_VISION_AI_V27What is in the MRE
The package
himax_w600k_ch30_mre.zipcontains:w600k_stage96_pairft.op28_proj16_ch30only.tflitew600k_stage96_pairft.op29_proj16_ch30only.tfliterun_compare.shscripts/analyze_ch30.pyscripts/compare_facedbg_tensor.pyThe probe models keep the original model prefix from input through:
The final debug projection selects only global channel 30 into output slot 0. Other slots are zero projections and match exactly, so the debug projection itself is not the source of the mismatch.
Expected behavior
For the same fixed INT8 input and the same Vela model, WE2 Ethos-U output should match local CPU TFLite output within normal INT8 tolerance. This is especially important for face recognition: enrollment may be done on a host CPU TFLite pipeline while recognition runs on WE2.
Actual behavior
op29channel 30 diverges reproducibly:Looking only at the selected channel slot:
The full 128D embedding for the original model is much worse:
Why this MRE is meaningful
The issue is not visible from a random standalone
DepthwiseConv2Dtest. The mismatch needs the real upstream activation distribution around:CPU op27 channel-30 distribution around
zero_point=-12:The mismatch is not isolated to the PReLU negative branch:
Channel 30 is also not extreme by
op29per-channel weight scale. It ranks 44/80, so the evidence points more toward runtime rounding/requantization behavior under the real activation distribution than toward an obviously abnormal channel scale.Experiments already tried
We tried two simple graph-surgery fixes:
Insert explicit
QUANTIZEafterop28using the original tensor-167 quant params.CPU operators = 0).op29_ch30only cosine=0.997164.Recalibrate tensor-167 scale only.
0.002983 -> 0.004712: local ch30 probe improved tocosine=0.999130, but full embedding only reached0.973331.0.002983 -> 0.005966: local ch30 probe improved tocosine=0.999455, but full embedding dropped to0.965706.So a simple identity quant boundary or single-tensor scale edit is not sufficient.
How to reproduce offline comparison
After extracting the MRE package inside the repository root:
cd sscma-example-we2/artifacts/himax_w600k_ch30_mre chmod +x run_compare.sh ./run_compare.shThis compares the provided WE2 FACEDBG captures against CPU TFLite and prints the per-channel distribution analysis.
Request
Could you help check whether this is an Ethos-U runtime / Vela scheduling / quantized operator rounding issue on WE2, especially around the real
PReLU -> DepthwiseConv2Dboundary? If possible, please also advise whether a specific Vela/TFLM/Himax SDK version is expected to avoid this divergence.