Asset processing: transcode non-H.264/H.265 video, normalize exotic image formats, render PDFs via pdf.js

[vpetersson]

Summary

Add a server-side processing step (Celery) that normalizes uploads into formats the existing viewer plays directly, plus client-side pdf.js so PDFs become a viewer-renderable asset. This broadens the set of files an end user can drop in and have play correctly, without expanding the viewer's runtime media stack.

Three workstreams in one PR (or three small PRs landed together):

1. Video — probe on upload, transcode anything that isn't H.264/H.265 to H.264 + AAC in MP4.
2. Image — convert HEIC, HEIF, TIFF to lossless WebP on upload.
3. PDF — vendor pdf.js so PDFs upload as a webpage-style asset that paginates client-side at runtime.

Why now

Today Anthias accepts a narrow set of extensions (anthias_app/helpers.py:43, static/src/components/add-asset-modal/file-upload-utils.ts:10-50) and routes them straight to mpv/vlc or the Qt webview. Anything outside that list either errors at upload or fails silently at playback. The fix is small and bounded: a Celery normalisation pass at upload time, mirroring the existing download_video_from_youtube async pattern (api/serializers/mixins.py:67).

Workstreams

1. Video normalization (Celery)

On every video upload, run a fast ffprobe to detect codec and container. The output of the probe decides whether the file passes through or is transcoded.

Directly playable (no transcode, just rename .original → final URI):

• Container ∈ {mp4, mkv, mov, webm, ts, mpg/mpeg, flv, avi}
• Video codec ∈ {h264, hevc}
• Audio codec ∈ {aac, mp3, opus, vorbis, ac3, none}

Otherwise, transcode:

ffmpeg -threads 2 -i <input> -c:v libx264 -preset medium -crf 23 \
       -c:a aac -b:a 192k -movflags +faststart <out>.mp4

ffmpeg is already a runtime dependency for lib.utils.get_video_duration and the YouTube ingest path. No new system packages.

Implementation:

• New Celery task normalize_video_asset(asset_id) in celery_tasks.py.
• Upload path sets Asset.is_processing=True and enqueues the task.
• On success: update Asset.uri, set Asset.duration via get_video_duration, flip is_processing=False, write metadata['original_ext'] + metadata['transcoded'].
• On failure: write metadata['error_message'], leave is_processing=False (no stuck rows).
• time_limit=1800 (30 min ceiling).

2. Image normalization (Celery)

Convert HEIC, HEIF, and TIFF uploads to lossless WebP so the Qt webview only ever sees a format it can render. WebP keeps source fidelity (lossless), preserves alpha (which HEIC/TIFF can carry and JPEG would drop), is well below PNG in file size for photographic content, and is already accepted by Anthias's upload list (static/src/components/add-asset-modal/file-upload-utils.ts:13) so the viewer/Qt webview path is exercised.

Toolchain:

• Pillow (already used elsewhere in the stack); WebP support is built into Pillow's libwebp linkage on Debian/Pi.
• pillow-heif (Python, ARM- and x86-friendly libheif wrapper).
• libheif1 (apt, ~1 MB) — runtime dep for pillow-heif.

Implementation:

• New Celery task normalize_image_asset(asset_id):

  Image.open(<input>).convert('RGBA').save(<out>.webp, 'WEBP', lossless=True)

  (Use RGBA rather than RGB so transparency is preserved when present.)
• Upload path sets is_processing=True for {heic, heif, tif, tiff} and enqueues the task.
• On success: Asset.uri → .webp path, Asset.mimetype='image', metadata['original_ext']=ext, is_processing=False.
• Failure path mirrors video.

3. PDF rendering (pdf.js)

PDFs become a webpage-type asset that loads a self-hosted pdf.js page. The viewer's existing view_webpage branch handles runtime display; no viewer-side code changes.

Toolchain:

• pdfjs-dist (Mozilla PDF.js), pinned to current LTS major.
• Vendored at build time under static/vendor/pdfjs/, served by WhiteNoise per anthias_django/settings.py.

Implementation:

• New Django route GET /docs/<asset_id>/pdf/?page_duration=<s> in a new anthias_app/document_views.py that streams a small HTML shell. The shell loads pdf.js, opens /anthias_assets/<asset_id>.pdf, renders page-by-page on a <canvas>, advances every page_duration seconds, and loops.
• Page count read at upload time via a small Celery task that uses pdfjs-dist under Node. Asset.duration = pages * page_duration_s.
• Encrypted PDFs are rejected at upload with a clear error.
• Per-asset page_duration is editable; default comes from a new device setting default_document_page_duration (10s).

Schema groundwork

Add metadata = JSONField(default=dict, blank=True) to the Asset model (anthias_app/models.py:11). New migration. Used by all three workstreams to carry original_ext, transcoded, error_message, and document_pages without inflating the schema.

Expose metadata on AssetSerializerV2 (read+write) in api/serializers/v2.py:22. v1.x serializers expose it read-only for back-compat.

Resource constraints

The Celery worker runs on the same device as the viewer (per CLAUDE.md, the worker container even shares the rootfs with the server). A naive ffmpeg transcode will pin all four cores on a Pi 4/5 and starve the viewer mid-playback. Two layers of throttling:

1. Lower the worker's process priority at launch — set CPU and I/O priority on the Celery worker itself so every subprocess inherits it. Update the anthias-celery service command in docker-compose.yml.tmpl:

   nice -n 19 ionice -c 3 celery -A celery_tasks worker ...
   

   nice -n 19 is the lowest CPU priority (highest niceness); ionice -c 3 is idle-class I/O priority. Both are no-ops when the system is idle and only kick in under contention — so background sweeps still finish fast on an idle device while never disrupting active playback.

2. Cap ffmpeg thread count — pass -threads 2 to the transcode invocation so two cores stay free for the viewer on Pi 4/5 (and on Pi 3, ffmpeg will already be slow enough that single-threaded vs multi-threaded matters less than the priority cap).

Pillow's HEIC/TIFF → WebP conversion is light (typically sub-second per image) and inherits the worker's nice level, so no per-call thread cap is needed.

Verification: Pi 5 smoke test — start a video playback, then upload a ~100 MB ProRes MOV. Watch for any playback stutter while the transcode runs in the background. Expectation: zero visible stutter.

Acceptance criteria

• Uploading a ProRes MOV (or any non-H.264/H.265 source) yields a playable H.264 MP4 asset; an already-H.264 MP4 is a no-op rename with no re-encode.
• Uploading a HEIC yields a playable lossless WebP asset; same for TIFF.
• Uploading a multi-page PDF yields a webpage asset whose viewer-side display paginates at the configured page duration and loops.
• Failure paths (corrupt video, encrypted PDF, broken HEIC) leave the row in a clear error state with metadata.error_message, never stuck in is_processing=True.
• Unit tests under api/tests/test_assets.py cover each Celery task with deterministic fixtures (small ProRes clip, HEIC, 2-page PDF). Tests assert on output codec/extension and on the failure-state contract.
• uv run pytest -m "not integration" passes; integration suite passes per CLAUDE.md.
• New metadata field appears in the v2 OpenAPI schema (drf-spectacular).
• Pi 5 smoke test: upload one video, one HEIC, one PDF — confirm playback in the live dev environment with no observable stutter while a transcode runs in the background.

Critical files

• anthias_app/models.py — add metadata field, new migration
• celery_tasks.py — normalize_video_asset, normalize_image_asset, process_pdf_asset
• api/views/v2.py, api/views/mixins.py — upload path branches by extension; sets is_processing=True and enqueues the right task
• api/serializers/v2.py — expose metadata; add default_document_page_duration to device settings
• static/src/components/add-asset-modal/file-upload-utils.ts — accept heic, heif, tif, tiff, pdf
• static/vendor/pdfjs/ — vendored pdf.js
• anthias_app/document_views.py (new) — /docs/<id>/pdf/ route
• pyproject.toml — pillow-heif
• package.json — pdfjs-dist
• docker/anthias-server.Dockerfile.j2 — libheif1
• docker-compose.yml.tmpl — wrap anthias-celery command with nice + ionice

Patterns to reuse

• download_video_from_youtube (api/serializers/mixins.py:67) — model the async upload+convert flow after this: synchronous URI placeholder + is_processing=True + Celery task that flips is_processing=False.
• get_video_duration (lib/utils.py) — ffprobe duration extraction; reuse for the transcoded MP4.
• The <asset_id>.tmp rename pattern in celery_tasks.py:cleanup — .original suffix during conversion, atomic rename on success. The 1-hour mtime guard already handles in-flight files correctly.

[vpetersson]

Update: After a lot of experimentation, doing asset re-encoding on the device was scrapped. While it might work fine on x86 (but even there it suffers). The updated approach is to reject all assets that are unsupported for videos and provide guidance for how to re-encode for the board.

[the-other-martin]

@vpetersson
I did a fresh install withe latest Anthias version on raspberrypi 5 and tried to overcome the following issue with handbrake transcoded h265 material: video has been uploaded as expected and played, but slowed down: because of slower playing the video stopped after the correct time, but didn‘t play complete content- started again from beginning.
I tested with powerpoint exported video slideshow with 20sec per slide, interestingly Anthias showed slides for around 30 seconds because it played too slow.
So currently there is no solution by transcoding to h.265 (e.g. with handbrake).
Could you please investigate, thanks Martin

[vpetersson]

Hi @the-other-martin — thanks for the report. I tried to reproduce this on our Pi 5 testbed and couldn't, but the negative result is informative.

Environment under test: Pi 5 8 GB, Debian 13, kernel 6.12.75+rpt-rpi-2712, ffmpeg 7.1.3-0+deb13u1+rpt1, qt6-multimedia 6.8.2-8 — i.e. a current Anthias master image.

What I tried (all played at 1.0 ×, ≤0.2 % frame drops):

Clip	Codec / container	Viewer result (playback-stats.log)
HandBrake-equivalent 1080p30 8-bit (x265 medium, CRF 22, hvc1 + AAC, faststart)	hevc Main, MP4	elapsed 20031 ms / position 19945 ms
1080p30 Main10 (10-bit)	hevc Main10, MP4	elapsed 20091 ms / position 20000 ms
1080p30 HEVC in MKV	hevc Main, MKV	elapsed 19984 ms / position 19876 ms
''PowerPoint-style'' 1080p30 slideshow with sparse keyframes	hevc Main, MP4	elapsed 20027 ms / position 19933 ms
BBB 4K30 HEVC (60 s)	hevc Main, MP4	elapsed 59962 ms / position 60000 ms (4 drops of 1797)
BBB 4K60 HEVC (60 s)	hevc Main, MP4	elapsed 60043 ms / position 60000 ms (8 drops of 3603)

HW decoder is confirmed engaged. While 4K60 HEVC was playing I walked /proc/<AnthiasViewer-pid>/fd and saw:

• /dev/video19 (rpi-hevc-dec Hantro G2)
• /dev/media2 (its media controller)
• /dev/dma_heap/linux,cma (CMA-backed DMA pool)
• four anon_inode:request v4l2-request hwaccel allocations
• dozens of /dmabuf: decoder-output frames

The av:hevc:df0..df4 libavcodec frame-decoder threads sat at ~1 % CPU each. Standalone ffmpeg -f null - (no hwaccel hint) on the same Pi 5 software-decodes the same 4K30 clip at 0.736 × real-time — which is suspiciously close to your reported 30 s / 20 s = 0.67 ×. So your symptom is consistent with HW decode not being engaged on your device.

Likely causes on your side, in order:

1. An older Anthias image that predates the v4l2 device mappings or the +rpt1 libav pin.
2. The Hantro v4l2 nodes aren't reaching the viewer container.
3. A clip property the Hantro decoder refuses (forcing SW fallback).

How to check on your Pi 5

If you can SSH into the Pi (or open a terminal session on it directly), the following commands narrow it down. Each one produces only a few lines — please paste them in a reply.

1. What image you're running:

docker ps --format '{{.Names}}\t{{.Image}}'
docker images | grep -i anthias

2. What your HandBrake output actually is (resolution / profile / level matter):

docker exec anthias-anthias-viewer-1 ffprobe -hide_banner \
    /data/anthias_assets/<your-asset-id>.mp4 2>&1 | head -30

(Or run ffprobe directly on the original .mp4 if you still have it on a workstation.)

3. Are the v4l2 HEVC decoder nodes visible inside the viewer container?

docker exec anthias-anthias-viewer-1 \
    ls -la /dev/video19 /dev/media2 /dev/dma_heap/linux,cma 2>&1

We expect crw-rw---- 1 root video ... for all three. If any are missing, the container can't reach the Hantro decoder and everything falls back to software.

4. While the slow clip is playing, check whether the viewer has /dev/video19 open:

PID=\$(docker exec anthias-anthias-viewer-1 pgrep AnthiasViewer)
docker exec anthias-anthias-viewer-1 sh -c \
    \"ls -la /proc/\$PID/fd 2>/dev/null | grep -E 'video|media|dma_heap|dmabuf|request'\"

A working Pi 5 shows /dev/video19, /dev/dma_heap/linux,cma, /dev/media2, several anon_inode:request, and dozens of /dmabuf: symlinks while playing. If you only see the source .mp4 and none of those, that confirms it's software-decoding.

5. What the viewer itself recorded for your clip:

docker exec anthias-anthias-viewer-1 tail -40 /data/.anthias/playback-stats.log

Look at elapsed_ms vs position-ms on the SAMPLE / STOP lines — elapsed_ms / position_ms is the wallclock-to-video ratio. On a working Pi 5 those numbers are within ~50 ms of each other; in your case I'd expect a ~1.5 × gap.

Once we have those five outputs we can pinpoint whether the fix is an image upgrade, a device-passthrough fix, or a re-encode recipe for the specific HEVC features your clip uses.

[the-other-martin]

Hi @vpetersson, thanks for your investigation! Please find my answers to your questions below:

Environment under test:
Linux tv01 6.18.29+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.18.29-1+rpt1 (2026-05-12) aarch64
Raspberry Pi 5 Model B Rev 1.1 4GB
Anthias Version: [v2026.5.1](https://github.com/Screenly/Anthias/commit/57b4f25)

1. What image you're running:
   docker images | grep -i anthias anthias-anthias-viewer-1 ghcr.io/screenly/anthias-viewer:latest-pi5 anthias-anthias-celery-1 ghcr.io/screenly/anthias-server:latest-pi5 anthias-anthias-server-1 ghcr.io/screenly/anthias-server:latest-pi5 anthias-redis-1 ghcr.io/screenly/anthias-redis:latest-pi5 ghcr.io/screenly/anthias-redis:latest-pi5 03592539f662 257MB 70.1MB U ghcr.io/screenly/anthias-server:latest-pi5 a68e3c02e132 1.43GB 371MB U ghcr.io/screenly/anthias-viewer:latest-pi5 eff31c53099b 2.61GB 689MB U

2. What your HandBrake output actually is (resolution / profile / level matter):
   PS: I'Ve created H.265 coded file on WIN11 with handbrake 1.11.1 (2026032200); Fast 1080p 30; changed Videoencoder to: H.265 (x265)
   docker exec anthias-anthias-viewer-1 ffprobe -hide_banner /data/anthias_assets/b121a75c475d4558a4851bbba4851b1a.mp4 2>&1 | head -30 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/data/anthias_assets/b121a75c475d4558a4851bbba4851b1a.mp4': Metadata: major_brand : mp42 minor_version : 512 compatible_brands: mp42iso2mp41 creation_time : 2026-05-22T17:11:43.000000Z encoder : HandBrake 1.11.1 2026032200 Duration: 00:02:27.85, start: 0.000000, bitrate: 708 kb/s Stream #0:0[0x1](und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 705 kb/s, 29.98 fps, 30 tbr, 90k tbn (default) Metadata: creation_time : 2026-05-22T17:11:43.000000Z handler_name : VideoHandler vendor_id : [0][0][0][0]

3. Are the v4l2 HEVC decoder nodes visible inside the viewer container?
   docker exec anthias-anthias-viewer-1 \ ls -la /dev/video19 /dev/media2 /dev/dma_heap/linux,cma 2>&1 crw-rw---- 1 root video 252, 2 May 22 15:58 /dev/dma_heap/linux,cma crw-rw---- 1 root video 511, 2 May 22 15:58 /dev/media2 crw-rw---- 1 root video 81, 12 May 22 15:58 /dev/video19

4. While the slow clip is playing, check whether the viewer has /dev/video19 open:
   not sure if the commands really work as planned (shouldn't the pgrep go for AnthiasWebview? Is the "PID=$...." correct?

PID=\$(docker exec anthias-anthias-viewer-1 pgrep AnthiasViewer) docker exec anthias-anthias-viewer-1 sh -c \ \"ls -la /proc/\$PID/fd 2>/dev/null | grep -E 'video|media|dma_heap|dmabuf|request'\" -bash: syntax error near unexpected token ('`

5. What the viewer itself recorded for your clip:
   docker exec anthias-anthias-viewer-1 tail -40 /data/.anthias/playback-stats.log tail: cannot open '/data/.anthias/playback-stats.log' for reading: No such file or directory

Hope this helps to analyze!
Martin

[IT-Walbrzych]

I’d also like to add that I have exactly the same problem as @the-other-martin. I get the same results as he does.
Also, I can't open videos created with PowerPaint. I’m getting this message: “The ‘h264’ video codec is not hardware-decoded on this device. Supported: hevc.”

ffmpeg -i ‘video.mp4’ -c:v libx265 -preset medium -crf 28 -tag:v hvc1 -c:a aac -b:a 192k -movflags +faststart ‘video.hevc.mp4’