[BUG]Silent encoder/buffer stall after ~11–13 h continuous capture (IMX477 / CM4, kernel 6.12.75, libcamera 0.7.1) — matches #1090/#733 on a post-#6429 stack

[ballen4705]

Describe the bug

After ~11–13 hours of continuous capture at ~35 fps, the camera pipeline stops delivering new frames. There is no exception, no libcamera error, and no kernel/CSI error logged — the pre_callback simply stops being invoked and the last frame is repeated forever. One thread of the process is left stuck in uninterruptible kernel sleep (D state), the signature of a blocked DMA/buffer wait (a dma-buf / VIDIOC_DQBUF / poll() completion) that never returns. A full process restart recovers the camera immediately; nothing short of re-opening the device does. The problem is intermittent and so far only seen on long runs, so it is hard to reproduce on demand.

This signature is a very close match to the (closed) reports #1090 and rpicam-apps #733 — "locks up after hours, receiving no more buffers", thread stuck in task:python3 state:D blocked in v4l2_m2m_fop_poll. Those were attributed to a VideoCore dma-buf bug and fixed by kernel PR raspberrypi/linux#6429, but residual long-run lockups with the identical trace were reported afterward. Our stack (kernel 6.12.75 / libcamera 0.7.1) already contains that fix, so this looks like a post-fix recurrence or a related buffer-starvation / encoder-poll stall on a newer stack.

Likely sub-system. This is a Compute Module 4 (BCM2711), so the active path is the on-SoC VC4/unicam receiver (bcm2835-unicam) + bcm2835-isp + the hardware bcm2835-codec (V4L2 mem2mem) encoder + the vc_sm_cma dma-buf bridge — i.e. libcamera's rpi/vc4 pipeline handler. It is not the RP1/CFE (rp1-cfe) path (that is Pi 5 / CM5). Because MJPEGEncoder runs on the bcm2835-codec mem2mem device, the most likely location of the stuck D-state thread is the encoder's poll()/DQBUF, not the camera capture node — which would explain why libcamera's camera-side frame-timeout watchdog never fired and why dmesg shows no unicam/CSI error.

Terminology — "stall" / "block". Both words denote the same condition: the pipeline enters a permanently blocked state in which the process is still alive and "running" (it has not crashed) but makes no further progress, because a buffer it is waiting on is never returned. No error is raised and no timeout fires, so it cannot recover by itself — only re-opening the camera device clears it.

Hardware & software

Component	Version
Hardware	Raspberry Pi Compute Module 4 Rev 1.1
Sensor	Sony IMX477 (HQ Camera), dtoverlay=imx477,cam0
OS	Debian GNU/Linux 13 (trixie), Raspberry Pi reference 2026-04-21 (pi-gen 4ad56cc)
Kernel	6.12.75+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.75-1+rpt1 (2026-03-11) aarch64
Firmware (vcgencmd version)	Feb 11 2026 18:28:41, ce768004a1c9657e60b33b0cc413d8e07320cb0d (clean) (release)
libcamera	0.7.1+rpt20260429-1 (libcamera0.7, libcamera-ipa, python3-libcamera)
picamera2	python3-picamera2 0.3.36-1
Camera path	VC4/unicam (bcm2835-unicam) + bcm2835-isp + bcm2835-codec (HW encoder, V4L2 mem2mem) + vc_sm_cma dma-buf bridge; libcamera rpi/vc4 pipeline handler (CM4/BCM2711 — not rp1-cfe)

Display pipeline: dtoverlay=vc4-kms-v3d. The camera is opened headless (Preview.NULL).

Camera configuration / workload

Picamera2 configuration (built via picam2.configure(...)):

• main stream: 2028x1520, YUV420
• lores stream: 640x480, YUV420
• display="lores", encode="lores"
• Raw/sensor mode: picam2.sensor_modes[2] (2028x1520, SBGGR12), passed as config["sensor"] = {"output_size": ..., "bit_depth": ...}
• buffer_count = 6
• queue = True
• colour_space = Rec709, identity Transform
• Controls: fully manual — AeEnable=False, ExposureTimeMode=Manual, AnalogueGainMode=Manual, AwbEnable=False, NoiseReductionMode=0, FrameDurationLimits=(25000, 25000) µs (40 fps cap; sustained ~35 fps), ColourGains=(0.01, 0.01), AeMeteringMode=Matrix.
• Encoding: MJPEGEncoder on the lores stream via picam2.start_recording(encoder, FileOutput(output)).
• A picam2.pre_callback is installed (~6 s after start_recording). It does non-trivial CPU work per frame: OpenCV overlay drawing into the MappedArray, axis-sum reductions on a ThreadPoolExecutor (4 workers), numba-JIT model evaluation, and SciPy curve_fit. Per-frame callback time is normally ~22 ms p50 / ~30 ms p95 (occasionally ~34 ms).
• AnalogueGain is adjusted from the lores mean-Y once per frame inside the callback (manual AEC).

The application runs as a systemd service capturing continuously, indefinitely.

Steps to reproduce

1. CM4 + IMX477 on the VC4/unicam path, configured as above (main 2028×1520 YUV420 + lores 640×480 YUV420, buffer_count=6, queue=True, fully manual exposure/gain/AWB).
2. start_recording() with MJPEGEncoder on the lores stream, and install a pre_callback that does ~20–34 ms of CPU work per frame.
3. Run continuously at ~35 fps.
4. After ~11–13 h, frame delivery stops: the MJPEG preview goes black/frozen, with no exception or log message.

This is intermittent and not reproducible on demand.

Expected behaviour

Capture continues indefinitely; or, if the encoder/camera genuinely loses a buffer, the stack surfaces an error or a timeout (so the application can detect and recover) rather than blocking silently forever in D.

Actual behaviour / evidence

1. The live MJPEG preview (served from the encoder output) goes black/frozen — clients receive no new frames.
2. The application's HTTP API, which returns values computed from the most recent frame, keeps responding 200 OK but with byte-for-byte identical (frozen) data indefinitely.
3. No errors, warnings, or tracebacks are emitted by the application, picamera2, or libcamera from the moment frames stop.

Evidence that the stall is server-side (not a client/network artifact):

• A fresh, local request to the MJPEG stream endpoint received 0 bytes / 0 frame boundaries in 3 seconds while the service reported active (running).
• Two consecutive samples of the frame-derived API were identical down to the byte — the pipeline was frozen on a single frame.
• Process thread states (ps -T) showed one worker thread in Dsl (uninterruptible sleep):

 TID STAT  %CPU     TIME COMMAND
156321 Ssl  0.0 00:00:12 python3        # main
156371 Ssl  1.5 00:12:08 CameraManager
156380 Dsl  1.1 00:08:47 python3        # <-- stuck in uninterruptible kernel sleep
...

All other threads were in normal interruptible sleep (Ssl). Given the workload (HW MJPEG encoder on the bcm2835-codec mem2mem device), the most likely call site is the encoder's v4l2_m2m_fop_poll / VIDIOC_DQBUF — exactly the trace seen in #1090 / rpicam-apps #733. We will confirm the precise kernel call site on the next occurrence (see Diagnostics, below).

Ruled out:

• Not a Python crash: no traceback anywhere in the session log.
• Not OOM: free -h showed ~753 MiB free, ~1.3 GiB available, no swap configured/used.
• Not thermal throttling at fault: vcgencmd measure_temp = 58.4 °C at inspection; vcgencmd get_throttled = 0x80000 indicates a past "soft temperature limit occurred" event (bit 19) but no active throttling. Noted in case sustained thermal load is a contributing factor.
• No kernel CSI/sensor error: dmesg had no unicam/bcm2835/csi/imx/i2c/dma/vc_sm errors or timeouts. (We did not have the hung-task detector output captured; on a long enough hang, [BUG] picamera2 locks up after hours, receiving no more buffers #1090 reporters eventually saw INFO: task python3:<tid> blocked for more than 120 seconds.)
• No libcamera frame-timeout / request-completion warning was logged — consistent with the stall being on the encoder mem2mem node, since libcamera's V4L2 Dequeue timer … expired / Camera frontend has timed out watchdog (see [HOW-TO] debug dequeue timer #952) only monitors the camera side.

Recovery

A full service restart recovered the camera immediately and cleanly (restart returned in ~0.13 s; a fresh stream then produced 111 frame boundaries in 3 s, i.e. ~37 fps). The D-state thread did not block process teardown — the blocked wait was released when the device was torn down and re-opened.

Frequency

• Intermittent; observed after long continuous runs (many hours / overnight). In the closest prior report ([BUG] picamera2 locks up after hours, receiving no more buffers #1090) time-to-failure was typically 10–20 h and correlated with higher die temperature.
• At least the second "black/frozen preview after a long run" on this unit. (An earlier, separate incident was an application bug — a Python IndexError in our callback, since fixed; that one produced a traceback. This one is different: no traceback, thread stuck in D.)
• Novelty: we could not find a public report for this exact combination (IMX477 + kernel 6.12.75 + libcamera 0.7.1 on CM4, i.e. after PR#6429). The close matches are on older stacks (kernel 6.6.x / libcamera 0.3–0.5) nominally fixed by #6429, so this reads as a fresh post-fix recurrence matching the historical [BUG] picamera2 locks up after hours, receiving no more buffers #1090 / [BUG] expanding user #733 signature.

Impact / current mitigation

For an unattended instrument this is a hard failure: capture silently stops and never recovers on its own, and because start_recording keeps the service "running" with no error, the application cannot distinguish "healthy" from "stalled" without independently timing frame arrivals. We have since added an in-process frame-arrival watchdog that dumps diagnostics and restarts the service after ~0.5 s without a new frame.

Related public reports / prior art

Most-relevant first. Several were closed by kernel PR raspberrypi/linux#6429 (lens-shading dma-buf re-import per frame, which eventually blocked VideoCore buffer messaging), but residual long-run lockups with the same D-state ppoll/v4l2_m2m_fop_poll trace were reported afterward.

• [BUG] picamera2 locks up after hours, receiving no more buffers #1090 — "picamera2 locks up after hours, receiving no more buffers" (closed, 2024-08 → 2024-11). Closest match: continuous streaming "stops receiving buffers after multiple hours" (typically 10–20 h), thread in task:python3 state:D blocked in v4l2_m2m_fop_poll → v4l2_poll → ppoll, "blocked for more than 120 seconds", no program error, recovers on camera close/restart. Root cause = lens-shading dma-buf re-imported every frame; fixed by linux#6429. Some users still saw residual lockups afterward. [BUG] picamera2 locks up after hours, receiving no more buffers #1090
• rpicam-apps [BUG] expanding user #733 — "rpicam-vid Encoding Halts After Several Hours Without Exceptions" (closed). Same signature in the C++ stack on a CM4 (rules out Python/picamera2), confirmed encoder-side; merged into the [BUG] picamera2 locks up after hours, receiving no more buffers #1090 investigation/fix. [BUG] rpicam-vid Encoding Halts After Several Hours Without Exceptions rpicam-apps#733 · forum: https://forums.raspberrypi.com/viewtopic.php?t=376766
• rpicam-apps previews: Overlays won't work in DRM/QTGL previews if buffer_count is 1 #664 — "rpicam-vid hangs when h264 codec is used" (closed). Best description of the mechanism relevant to our buffer_count=6: once all 6 HW-codec output buffers are lost, "there are no buffers that can be recycled back into the HW codec, hence the stall", and poll() "constantly returns 0, so we never attempt a DQBUF and things just grind to a halt." [BUG] rpicam-vid hangs when h264 codec is used rpicam-apps#664
• [BUG] RPI camera stuck #1275 — "RPI camera stuck" (open, 2025-06). After a couple of days the stack blocks in uninterruptible sleep; processes survive kill -9 (STAT … DL+); even libcamera-hello hangs. VC4/unicam path; cross-references [BUG] picamera2 locks up after hours, receiving no more buffers #1090. [BUG] RPI camera stuck #1275
• [BUG] picamera2 randomly stops feeding frames while recording (imx519) #1004 — "picamera2 randomly stops feeding frames while recording (imx519)" (open). "Process continues to work, just no frames are produced … no trace." Minimal repro uses MJPEGEncoder on a lores YUV420 stream (close to our setup); reporter tried buffer_count 1/3/7/9 and queue True/False — changed time-to-failure but did not fix it. [BUG] picamera2 randomly stops feeding frames while recording (imx519) #1004
• [HOW-TO] debug dequeue timer #952 — "debug dequeue timer" (open). Documents the libcamera camera-side watchdog (V4L2 Dequeue timer of 1000000.00us has expired! → RPI Camera frontend has timed out!) and the camera_timeout_value_ms setting in rpi_apps.yaml. In our event this watchdog did not fire — consistent with an encoder-node (not camera-node) stall. [HOW-TO] debug dequeue timer #952
• Kernel fix referenced above: raspberrypi/linux PR #6429 (lens-shading dma-buf caching).
• Picamera2 manual (buffer/event-loop semantics): a consumer that fails to recycle buffers will "start dropping ever more camera frames, and eventually the camera system will stall completely"; heavy work in pre_callback blocks the event loop and "is likely to drop some frames".

Root-cause analysis (source review of the exact versions in our stack)

We reviewed the relevant open-source code at the exact versions in our stack. Two independent reviews — one of the kernel ARM-side path, one of the userspace buffer managers — converged on the same conclusion: the defect is below userspace, in the kernel bcm2835-codec mem2mem / vc_sm_cma path plus the closed VideoCore firmware. picamera2 and libcamera appear correct; they only propagate the stall silently. All ARM-side code is public; the VideoCore firmware is closed and not reviewable.

Kernel side (raspberrypi/linux, branch rpi-6.12.y):

• PR#6429 is present on rpi-6.12.y (commit 615f539982dde45108372a08ab3474380724fb33, the lens-shading-table dma-buf cache). The original [BUG] picamera2 locks up after hours, receiving no more buffers #1090 re-import mechanism is therefore closed on our stack — this is a post-fix recurrence, not the original bug.
• The "poll returns 0, no error" state is confirmed in source. v4l2_m2m_poll_for_data() (drivers/media/v4l2-core/v4l2-mem2mem.c) only returns EPOLLERR when a queue stops streaming, has q->error set, or has an empty queued_list. When buffers are stranded "with the VPU" they stay on queued_list, done_list stays empty, and poll() returns 0 indefinitely. The waiting thread blocks forever inside v4l2_m2m_fop_poll — exactly the observed D state.
• Nothing can break that poll. vb2_queue_error() (which would set q->error and wake pollers with EPOLLERR) is never called anywhere in the entire drivers/staging/vc04_services tree. There is no dequeue timeout / watchdog on the encoder path.
• No recovery on the data path. In bcm2835-codec (drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c), the per-buffer return accounting (the buffers_with_vpu decrement and vb2_buffer_done()) happens only on a BULK_RECEIVE_DONE completion; buffer_to_host_work_cb() merely pr_errs on failure and does not recover or error the queue. So if a VC→ARM bulk-receive completion for an encoded buffer is lost, that buffer is never returned or re-queued. With buffer_count = 6, losing six such returns drains the pool permanently, and only a full device re-open resets it — matching our restart-recovers behavior.

Most likely trigger: a dropped/stale VC→ARM bulk-receive (or VCHIQ slot) message under sustained load, which lives in the closed VideoCore firmware and is unreviewable. The reviewable, fixable ARM-side defect is the absence of any timeout/watchdog on the encoder mem2mem queue.

Userspace side (picamera2 v0.3.36, libcamera v0.7.1+rpt20260429):

• picamera2 is not the root cause, and proves the stall is kernel-level. Its encoder loop uses a bounded poll(400 ms) (picamera2/encoders/v4l2_encoder.py, ~line 213). A thread stuck in uninterruptible D inside v4l2_m2m_fop_poll for >120 s therefore cannot be a userspace wait — it can only be an uninterruptible wait in the kernel codec / vc_sm_cma / firmware. (rpicam-apps [BUG] expanding user #733 reproduced the same stall in pure C++, independently ruling out Python/picamera2.)
• The userspace symptom matches our ps exactly. Once the codec stops returning buffers, an untimed self.buf_available.get() (v4l2_encoder.py, ~line 319) blocks the camera event-loop thread in interruptible sleep (S). That is the propagation of the stall, not its origin, and it explains one thread in D, the rest in Ssl, no traceback.
• A heavy/late pre_callback or queue=True does not leak buffers here. A callback only fails to release a request if it throws (which would emit a traceback — absent in our logs). queue=True only holds one extra request. These affect time-to-failure at most, not correctness.
• libcamera's dequeue watchdog could never fire for this. It is wired solely to the unicam frontend node (frontendDevice() → unicam_[Unicam::Image]), and the MJPEG encoder is not a libcamera device at all — picamera2 drives bcm2835-codec directly via V4L2. This is exactly why no dequeue-timer / frontend-timeout warning appeared.

Conclusion: the root cause is a kernel/firmware buffer-return failure on the bcm2835-codec encoder path — a lost VC→ARM buffer-completion (closed firmware) combined with the complete absence of a dequeue timeout or vb2_queue_error() recovery on the ARM side. picamera2 and libcamera are not at fault; picamera2 merely surfaces the stall silently because its _encode buffer wait (buf_available.get()) is untimed.

This suggests two fixes: (1, primary) add a dequeue timeout / vb2_queue_error() recovery to the bcm2835-codec encoder queue (and/or fix the lost-completion trigger in firmware); (2, secondary) give picamera2's encoder an optional timeout on its buffer wait, so a stalled codec surfaces as an error instead of a silent hang.

Questions / requests for the maintainers

1. Is there a supported way to detect this stall from the API? A frame-arrival timeout, a "camera errored / disconnected" signal, or a callback hook so the application knows frames have stopped (rather than polling our own last-frame timestamp)?
2. Is there a recommended request/dequeue timeout that should fire when a buffer never completes, instead of blocking forever in D? camera_timeout_value_ms in rpi_apps.yaml (per [HOW-TO] debug dequeue timer #952) covers the camera node — is there an equivalent for the bcm2835-codec encoder mem2mem node, which appears to be where we stall?
3. Could the in-callback CPU work contribute? Our pre_callback is moderately heavy (~22 ms p50, occasionally ~34 ms). With buffer_count=6 and queue=True, could sustained callback latency / GC pauses starve the buffer pool such that it blocks permanently rather than just dropping frames (cf. the "lose all 6 buffers → poll() returns 0 → never DQBUF" mechanism in rpicam-apps previews: Overlays won't work in DRM/QTGL previews if buffer_count is 1 #664)? We intend to test whether reducing buffer_count (e.g. 3 or 2) or queue=False changes time-to-failure, as in [BUG] picamera2 randomly stops feeding frames while recording (imx519) #1004.
4. Is this the post-#6429 residual lockup, or a new stall? Our signature matches [BUG] picamera2 locks up after hours, receiving no more buffers #1090 / rpicam-apps [BUG] expanding user #733 exactly, but on a newer stack (kernel 6.12.75 / libcamera 0.7.1, CM4 VC4/unicam) that should already contain PR#6429. Are there known residual long-run lockups on the rpi/vc4 + bcm2835-codec path after that fix?
5. Lens-shading (ALSC) involvement? [BUG] picamera2 locks up after hours, receiving no more buffers #1090's root cause was the lens-shading dma-buf re-imported every frame. We run fully manual exposure/gain and disable AWB, but lens shading is a separate algorithm and may still be active in our tuning file. Would disabling x.rpi.alsc (or removing the LSC table) be a useful diagnostic to change time-to-failure?

Diagnostics we will capture on the next occurrence

Mostly automated now in our frame-stall watchdog (dumps to persistent storage the instant a stall is detected, before restarting). On the next stall, before restart, we will collect:

• py-spy dump --pid <PID> (native + Python stacks) to pinpoint where the D-state thread is blocked. If the trace shows v4l2_m2m_fop_poll / __vb2_queue_cancel it confirms the [BUG] picamera2 locks up after hours, receiving no more buffers #1090 / [BUG] expanding user #733 (encoder mem2mem) lineage; if it shows a unicam/capture-node DQBUF it is a different, camera-side stall.
• cat /proc/<PID>/task/<TID>/stack and /proc/<PID>/task/<TID>/wchan / syscall for the blocked TID (kernel call site).
• Full dmesg (watching for the task python3:<tid> blocked for more than 120 seconds hung-task warning) and journalctl with LIBCAMERA_LOG_LEVELS=*:DEBUG (or targeted RPI/V4L2 categories).
• vcgencmd get_throttled / measure_temp history.

Please let us know which traces are most helpful and we'll attach them from the next event.

[davidplowman]

@6by9 Hi, the TLDR here is that there's a suspicion, though unproven, that V4L2 codec buffers (specifically the JPEG m2m encoder) are not being returned on the VC4 platform. Are there debug or kernel files we can monitor that might shed any light on this?

@ballen4705 I'll run a script that tries to emulate this kind of behaviour - but of course don't know if it will show the same symptom. If you had a simpler script that we could run, and which is known to provoke the problem, that would be helpful. Thanks!

[6by9]

Add start_debug=1 to /boot/firmware/config.txt. On issue sudo vclog -a will list out any asserts within the firmware that have occurred.
Even without that, sudo vclog -m would log any errors that have occurred within the firmware.

Is this actually using the JPEG encoder (/dev/video31) or the MJPEG encoder (/dev/video11 with format V4L2_PIX_FMT_MJPEG)?
I'm guessing the latter based on the use of MJPEGEncoder.

[ballen4705]

@davidplowman, @6by9, thank you both, this is very helpful.

@6by9 — yes, it's the MJPEG encoder, /dev/video11 with V4L2_PIX_FMT_MJPEG, not the /dev/video31 JPEG block: this is picamera2's MJPEGEncoder driving the bcm2835-codec mem2mem encode node. I'll confirm it empirically from the running process (reading the open /dev/video* fds) and paste the result here. [EDIT: CONFIRMED /dev/video11].

@davidplowman — I'm putting together a small, self-contained reproducer so you have something concrete to run. It's evolved from the stock mjpeg_server_2.py example: hardware MJPEGEncoder on a lores 640×480 YUV420 stream alongside a 2028×1520 main, buffer_count=6, queue=True, ~40 fps, plus a pre_callback that does ~25 ms of CPU "busy work" per frame (tunable to ~150% load) to mimic our real workload. I'll leave it running overnight to see whether it provokes the same silent stall, and share the script (and result) regardless.

I'm also folding your debug suggestions into an in-process frame-arrival watchdog so the next stall is captured automatically before restart. On a detected stall (2 seconds with no frames) it will dump:

• sudo vclog -m and sudo vclog -a (with start_debug=1 set in /boot/firmware/config.txt) for firmware errors/asserts;
• py-spy dump plus /proc/<pid>/task/<tid>/{stack,syscall,wchan} for every thread, to pinpoint the D-state call site (to confirm v4l2_m2m_fop_poll / VIDIOC_DQBUF);
• the open /dev/video* nodes, dmesg tail (hung-task warnings), and vcgencmd temp/throttle state.
• use os.execv() to restart the process to see if another silent stall can be produced.

Will report back with the reproducer and whatever the watchdog captures on the next occurrence.

[ballen4705]

@davidplowman, @6by9, my "reproduce the bug" code has been running overnight, now up to about 7 hours and a million frames, and has not yet stalled. I'll let it continue to run during the day today, and will check up on it (and here) in about 12 to 18 hours. Once it triggers the bug I'll put a copy here.

In setting up the code, I overworked and overheated the CPU/GPU and this throttled the frame rate. The code currently has a feedback loop that maintains the CPU temp between 78 and 80C. If the bug was triggered by thermal throttling I'll have to modify that to instead maintain a frame rate between 38 and 40FPS to tickle the throttling.

[davidplowman]

Thanks for the update. I'm running my own attempted simulation as well, with start_debug=1, though it's not showing any problems as yet. I'm actually monitoring the number of camera buffers queued to and dequeued from the encoder as one ought to be able to spot a single buffer going missing (whereas a total lock up would probably require all six to vanish). Anyway, I'll post back if anything happens...

[ballen4705]

My code has been running for 19 hours and 2.74 million frames, and still no stall. So rather than keeping this running longer, I'm going to modify it to provide enough work to drop the frame rate to 38-39FPS and restart it for another overnight run. I'll check in tomorrow.

[ballen4705]

Still no stall overnight. Operating on the theory that this might be related to CPU throttling, I have modified my testing code to monitor the hard throttle state and adjust the workload to repeatedly return to that limit.

[ballen4705]

Update

Run 1 (~24 h): The stall reproducer code ran about a day at the hard thermal-throttle edge (~85 °C, ~300% CPU, ~25–40 fps). The CM4 then went down hard (unreachable on the network, very warm) and needed a power cycle. The unit uses overlayroot; logs were on /home/pi/ and did not survive reboot, so there is no watchdog dump from that failure. No stall had been detected before it went offline.

Run 2 (in progress): Restarted with logs on a persistent ext4 /data partition and /data (and root) remounted RW for this testing. After ~8.5 h: still no stall, stream healthy on /dev/video11, heartbeats continuing.
Two incidental messages at camera/encoder open (not seen again over 8+ h of streaming, not correlated with any stall):

1. dmesg: bcm2835-codec bcm2835-codec: Failed setting ctrl 00990a67, ret -3 (twice, at bring-up). ret -3 is ENOENT; likely a missing V4L2 control. Seems harmless here.
2. vclog -m: mmal: mmal_vll_load: could not load VLL 'videnc.vll': (twice, boot and later). MJPEG on /dev/video11 still ran for hours. Noted for completeness; no stall correlation so far.

I'll leave this running during the day and have another look this evening.

[6by9]

00990a67 is V4L2_CID_MPEG_VIDEO_H264_LEVEL, and that is only applicable to H264 encoding rather than MJPEG. As the encoder handles multiple codecs the control exists in all cases, but the firmware throws an error when trying to set it on an inappropriate codec.

videnc.vll is a left over from many years back. The firmware at one point had a common shared library for all codecs called videnc that was then replaced by libraries per codec. The code still tries to load the common one and logs a failure due to not finding it, before correctly loading the codec specific one. That really ought to get ditched as it does no good.

If it is related to temperature triggered throttling, then it's going to be a pain to track down.
It's also not going to be great for a useful system as when it throttles you'll end up dropping even more frames due to the reduced performance whilst it tries to cool down, and then bounce between fast and slow response.

Do you see the same lockup if you use H264 instead of MJPEG? I suspect it's more related to the clock rate change, but it'd be a useful data point.

[davidplowman]

I left my own Pi 4 running over the weekend, but no signs of anything untoward at all. But I'm quite certain, really, that I'm not "tickling" the system in the right way. If you come up with anything that it would be worth me trying out, do let me know.

[ballen4705]

@davidplowman
David, I ran my code for close to a week but have not been able to reproduce the stall, so my report was premature. I added a watchdog to my application which will dump lots of info. So if this stalls again, I'll be able to provide more details.

@6by9
I started work on my project 3 years ago when the IMX477 max frame rate was 40FPS at 2028x1520. A few days ago, just playing around, I tried a faster frame rate on a CM4, and much to my surprise, it worked! Which led me to your improvements in raspberrypi/linux#6798 . I've now shifted my app from using YUV420@40 FPS to SRGGB12@45 FPS. Thank you for that!

[6by9]

I'll never claim that the firmware encoder and decoder drivers are perfect as there are so many different combinations of things that can be thrown at them. Throttling should generally be transparent, but you never can quite tell.

If on a CM4 then I'd expect you can use dtoverlay=imx477,link-frequency=600000000 in config.txt to increase from the current link frequency of 450MHz to 600MHz, and that should allow for 2028x1520@60fps. That is slightly over the spec for CM4 (500MHz / 1Gbit/s), but it's generally reliable up to around that level. The Global Shutter camera (IMX296) runs at 594MHz although with only one data lane vs the 2 for HQ (IMX477).
Of course that may then exacerbate your thermal issues!
(Pi5/CM5 are specified for up to 750MHz, so would give you 2028x1520 @ 75fps)

I thought I had patches to support 4 data lanes from IMX477 aimed at Pi5 and higher frame rates. However then that would also require a 3rd party IMX477 module as the Pi one only exposes 2 data lanes, so perhaps I'm misremembering my sensors. It probably doesn't help in this situation anyway.

[ballen4705]

@6by9
I've had fun with this. The cable from the HQ camera to the CM4 is 5cm long, so I first tried 600MHz and then stepped up by 3MHz to 750MHz, which got 75.2 FPS. The processing loop couldn't keep up so I also boosted the CPU frequency.

arm_boost=1
over_voltage=6
arm_freq=2000

This ran all night at 72FPS with 2028×1520 12-bit and no signs of thermal or other problems (system has good passive cooling). Now I'm going to see if neon SIMD intrinsics can speed up the processing loop a bit.

[EDIT 12.6.2026]
This has run with CSI @ 750MHz and CPU @ 2GHz for > 48 hours with no issues. YMMV