fix(iso/initramfs): apply build-pipeline audit findings (#220)

From the multi-agent ISO/initramfs audit vs upstream titanoboa/Bluefin/
Bazzite. Boot-relevant + hardening; none change the production kernel.

- live-env/src/iso.yaml: bound the live writable overlay
  (rd.live.overlay.size=4096) on both entries [#1]; add a third
  'verbose, no splash, no serial' GRUB entry for field-debugging — drops
  quiet/rhgb AND console=ttyS0 (the latter can stall every printk on a
  mini-PC with no real UART, observed on a Beelink SER8) [#4].
- custom-kernel/install.sh: refresh linux-firmware + amd-ucode after the
  CachyOS kernel swap so a newer kernel cannot reference amdgpu blobs the
  base lacks (black-screen before GDM) [#3]; zstd-guard + explicit --zstd
  on the initramfs regen so dracut cannot silently gzip.
- 50-branding/install.sh: same zstd-guard + --zstd on its initramfs regen.
- build-disk.yml: assert the ISO squashfs is zstd (guards the mksquashfs
  -e patch from a silent gzip fallback); warn-only 2 GiB constrained boot
  pass (surfaces live-overlay overflow on low usable RAM); correct the
  titanoboa pin comment (two -> three carried patches).
- qemu-boot-wait.sh: --mem flag (enables the constrained pass).
- check-upstream-pins.yml: note the third, fork-permanent titanoboa patch.
- Stale-comment fixes: GRUB font (Liberation Mono -> Noto Sans Mono);
  README titanoboa pin (upstream SHA -> fork SHA + 3 patches).

Author: daniel-g-carrasco

.github/scripts/qemu-boot-wait.sh

@@ -16,7 +16,7 @@
 # --gui-watch, gui=pass|fail|timeout|none (Layer C verdict, warn-only).
 set -euo pipefail
 
-MODE="" IMAGE="" LOG="serial.log" TIMEOUT=1800 GUI_WATCH=0
+MODE="" IMAGE="" LOG="serial.log" TIMEOUT=1800 GUI_WATCH=0 MEM=4096
 OK_REGEX='Started.*gdm\.service|Reached target graphical\.target|margine login:'
 FAIL_REGEX=""
 while [[ $# -gt 0 ]]; do
@@ -27,6 +27,7 @@ while [[ $# -gt 0 ]]; do
     --timeout) TIMEOUT="$2"; shift 2 ;;
     --ok-regex) OK_REGEX="$2"; shift 2 ;;
     --fail-regex) FAIL_REGEX="$2"; shift 2 ;;
+    --mem) MEM="$2"; shift 2 ;;
     --gui-watch) GUI_WATCH=1; shift ;;
     *) echo "unknown arg: $1" >&2; exit 2 ;;
   esac
@@ -63,7 +64,7 @@ fi
 rm -f qemu.pid "$LOG"
 qemu-system-x86_64 \
   -enable-kvm \
-  -m 4096 -smp 4 \
+  -m "$MEM" -smp 4 \
   -machine q35 \
   -drive "if=pflash,format=raw,readonly=on,file=$OVMF_CODE" \
   -drive if=pflash,format=raw,file=ovmf_vars.fd \

.github/workflows/build-disk.yml

@@ -62,12 +62,18 @@ on:
 env:
   # Titanoboa pin — the official ISO builder since 2026-06-11
   # (ADR-0008 Phase 5). Pinned to OUR fork branch margine-pins =
-  # upstream post-#138 HEAD (5c457c3d) + exactly two patches:
+  # upstream post-#138 HEAD (5c457c3d) + THREE carried patches:
   #   1. mksquashfs -e ordering (upstream PR ublue-os/titanoboa#147)
   #      — without it the squashfs silently falls back to gzip
-  #   2. grub2.extra_cfg raw fragment (upstream PR pending) — used
-  #      for the Secure Boot "Enroll key (MokManager)" menu entry
-  # Revert to the upstream pin once both PRs land. Renovate disabled.
+  #   2. grub2.extra_cfg raw fragment (upstream PR ublue-os/titanoboa#148)
+  #      — the Secure Boot "Enroll key (MokManager)" menu entry
+  #   3. grub.cfg trailing-slash: write grub.cfg only into EFI
+  #      SUBdirectories (cce73fc, NO upstream PR — fork-permanent).
+  #      Load-bearing for our plain-file EFI/MOK.der placement; without it
+  #      upstream's `for dir in /work/EFI/*` glob hits the MOK.der FILE and
+  #      hard-fails the ISO build with "Not a directory".
+  # Reverting to an upstream pin is impossible until all three land.
+  # Renovate disabled.
   TITANOBOA_REF: "daniel-g-carrasco/titanoboa@cce73fc476e97fed626283afb6c518e0882a12d7"
   IMAGE_REGISTRY: "ghcr.io/${{ github.repository_owner }}"
   DEFAULT_TAG: "stable"
@@ -363,6 +369,26 @@ jobs:
           echo "size_mb=$((SIZE_BYTES / 1024 / 1024))" >> "$GITHUB_OUTPUT"
           echo "Live ISO: $((SIZE_BYTES / 1024 / 1024)) MB"
 
+      - name: Assert squashfs is zstd (guards the mksquashfs -e patch)
+        # The mksquashfs -e ordering patch on our titanoboa fork keeps the
+        # squashfs at zstd-19; an accidental revert to an upstream pin
+        # would silently produce a gzip squashfs (slow boot, bigger ISO)
+        # that the boot-test below would NOT catch. Assert it explicitly.
+        run: |
+          set -euo pipefail
+          sudo apt-get update -qq
+          sudo apt-get install -y -qq squashfs-tools >/dev/null
+          ISO="$(ls "${{ steps.iso.outputs.outdir }}"/*.iso | head -1)"
+          MNT="$(mktemp -d)"
+          sudo mount -o loop,ro "$ISO" "$MNT"
+          SQ="$(sudo find "$MNT" -name 'squashfs.img' | head -1)"
+          if [[ -z "$SQ" ]]; then sudo umount "$MNT"; echo "::error::no squashfs.img in ISO"; exit 1; fi
+          COMP="$(sudo unsquashfs -s "$SQ" | grep -i 'Compression' || true)"
+          echo "squashfs: $COMP"
+          sudo umount "$MNT"
+          echo "$COMP" | grep -qi 'zstd' \
+            || { echo "::error::squashfs is NOT zstd (mksquashfs -e patch regressed?)"; exit 1; }
+
       # Gate (2026-06-12, review item #9): boot the freshly built ISO
       # in UEFI QEMU and require the live system to reach a usable
       # state. Catches Titanoboa/dracut/livesys regressions that static
@@ -389,6 +415,18 @@ jobs:
             --timeout 900 \
             --ok-regex 'Started.*gdm\.service|Reached target [Gg]raphical|livesys.*Finished|[Mm]argine login:' \
             --fail-regex 'dracut-emergency|Failed to start.*dmsquash|emergency shell'
+          # Constrained-RAM pass (2 GiB), WARN-ONLY: surfaces a live-overlay
+          # Overflow / read-only remount on low usable RAM (large iGPU UMA
+          # carveouts) without blocking the ship. Flip to gating once proven.
+          echo "=== constrained-RAM (2 GiB) boot pass (warn-only) ==="
+          sudo -E .github/scripts/qemu-boot-wait.sh \
+            --cdrom "$ISO" \
+            --mem 2048 \
+            --log iso-serial-2g.log \
+            --timeout 900 \
+            --ok-regex 'Started.*gdm\.service|Reached target [Gg]raphical|livesys.*Finished|[Mm]argine login:' \
+            --fail-regex 'Overflow|Remounting.*read-only|dracut-emergency|emergency shell' \
+            || echo "::warning::constrained-RAM (2 GiB) boot did not reach a usable state — see iso-serial-2g.log (live-overlay overflow on low usable RAM?)"
 
       - name: Upload ISO serial log
         # Always — a failed boot-test is debugged from this log (the
@@ -397,7 +435,7 @@ jobs:
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: iso-boot-serial-log
-          path: iso-serial.log
+          path: iso-serial*.log
           retention-days: 7
           if-no-files-found: ignore
 

.github/workflows/check-upstream-pins.yml

@@ -42,7 +42,11 @@ jobs:
       - name: Check carried patches against upstream
         run: |
           set -euo pipefail
-          # PR -> short description of the carried patch
+          # PR -> short description of the carried patch. NB: the fork
+          # carries a THIRD patch (cce73fc, grub.cfg trailing-slash) that
+          # has NO upstream PR — it is fork-permanent (load-bearing for our
+          # plain-file EFI/MOK.der placement), so it is intentionally NOT
+          # tracked here; only the upstreamable PRs are watched.
           declare -A PRS=(
             ["147"]="mksquashfs -e ordering (zstd fallback fix)"
             ["148"]="grub2.extra_cfg (Secure Boot enroll menu entry)"

README.md

@@ -315,9 +315,9 @@ full ~25-min image build. See the inline history comment at the
 top of [`.github/workflows/build.yml`](.github/workflows/build.yml)
 for the full rationale.
 
-**Titanoboa migration:** Phase 0 scaffolding is in progress; see
+**Titanoboa migration:** complete since 2026-06-11 (ADR-0008 Phase 5); see
 [`margine-fedora-atomic` ADR-0008](https://github.com/daniel-g-carrasco/margine-fedora-atomic/blob/main/docs/adr/0008-titanoboa-migration-plan.md).
-Pin: `ublue-os/titanoboa@5c457c3d`.
+Pin: `daniel-g-carrasco/titanoboa@cce73fc` (our fork of upstream `ublue-os/titanoboa@5c457c3d` + 3 carried patches; see `.github/workflows/build-disk.yml`).
 
 **Enabled GNOME extensions**: AppIndicator Support, Bazaar Integration,
 Dash to Dock, Gradia Integration, GSConnect (from Bluefin); o-tiling,

build_files/50-branding/install.sh

@@ -101,9 +101,14 @@ fi
 # ostree dracut module (without which switch-root fails — see comment
 # in custom-kernel/install.sh).
 if command -v dracut >/dev/null 2>&1; then
+  # zstd-compress explicitly (don't rely on inheriting Bluefin's
+  # 10-compression.conf); fail loudly if zstd is missing rather than let
+  # dracut silently gzip the initramfs.
+  command -v zstd >/dev/null || { log "ERROR: zstd missing — dracut would silently gzip the initramfs"; exit 1; }
+  export DRACUT_NO_XATTR=1
   for kver_dir in /usr/lib/modules/*/; do
     kver=$(basename "$kver_dir")
-    dracut --force --no-hostonly --no-hostonly-cmdline \
+    dracut --force --zstd --no-hostonly --no-hostonly-cmdline \
         --add "ostree" \
         --kver "$kver" \
         "${kver_dir}initramfs.img"

build_files/custom-kernel/install.sh

@@ -229,6 +229,22 @@ restore_kernel_install_hooks
 log "Removing COPR repo file (kept only at build time)"
 rm -f /etc/yum.repos.d/*copr*
 
+# Refresh GPU firmware + microcode to match the (newer) CachyOS kernel.
+# The CachyOS kernel can be newer than the base image's linux-firmware
+# snapshot (the CachyOS spec only Recommends: it), so a kernel that
+# references amdgpu blobs absent from the base would fail the GPU probe /
+# black-screen before GDM on bare metal — invisible to the virtio-gpu CI
+# gate. Upgrade ONLY the firmware packages actually installed (so an
+# unknown-package name can't abort the transaction), from the always-on
+# Fedora repos; never block the build on it.
+FW_PKGS="$(rpm -qa --qf '%{NAME}\n' 'linux-firmware*' 'amd-gpu-firmware*' 'amd-ucode*' 2>/dev/null | sort -u | tr '\n' ' ')"
+if [[ -n "${FW_PKGS// /}" ]]; then
+  log "Refreshing firmware/microcode for $KERNEL_VERSION: $FW_PKGS"
+  # shellcheck disable=SC2086 # intentional word-split of the package list
+  dnf -y upgrade --refresh $FW_PKGS \
+    || log "WARN: firmware/microcode refresh non-fatal failure (continuing)"
+fi
+
 # ---------------------------------------------------------------------------
 # v4l2loopback against the CachyOS kernel (OPTIONAL — skip on build error)
 # ---------------------------------------------------------------------------
@@ -562,9 +578,15 @@ log "Regenerating initramfs for all installed kernels (generic, bootc-path, ostr
 # Verified missing 2026-05-30: `lsinitrd <initramfs> | grep ostree`
 # returned ZERO lines on our published image. `--no-hostonly` alone
 # is insufficient; ostree dracut module must be EXPLICITLY requested.
+# The initramfs MUST be zstd-compressed (today this only works by
+# inheriting Bluefin's 10-compression.conf; nothing pins it). Pass --zstd
+# explicitly AND fail loudly if the tool is missing, so dracut can never
+# silently fall back to gzip (slower boot, bigger initramfs).
+command -v zstd >/dev/null || { err "zstd binary missing — dracut would silently gzip the initramfs"; exit 1; }
+export DRACUT_NO_XATTR=1
 for kver_dir in /usr/lib/modules/*/; do
   kver=$(basename "$kver_dir")
-  dracut --force --no-hostonly --no-hostonly-cmdline \
+  dracut --force --zstd --no-hostonly --no-hostonly-cmdline \
       --add "ostree" \
       --kver "$kver" \
       "${kver_dir}initramfs.img"

build_files/system_files/usr/lib/bootupd/grub2-static/configs.d/05_margine-gfxmode.cfg

@@ -8,7 +8,7 @@
 # no-op. The resolution-INDEPENDENT lever is a bigger glyph bitmap = a large
 # .pf2 font.
 #
-# margine.pf2 is a 36px Liberation Mono built by grub2-mkfont at image build
+# margine.pf2 is a 36px Noto Sans Mono built by grub2-mkfont at image build
 # (build_files/50-branding/install.sh) with family name "Margine", so its
 # embedded font name is exactly "Margine Regular 36" — that's what
 # gfxterm_font must select (gfxterm uses its tiny built-in font otherwise).

live-env/src/iso.yaml

@@ -34,10 +34,18 @@ grub2:
   timeout: 5
   entries:
     - name: "Install Margine"
-      linux: "/images/pxeboot/vmlinuz quiet rhgb console=tty0 console=ttyS0,115200n8 systemd.show_status=1 root=live:CDLABEL=Margine-Live enforcing=0 rd.live.image"
+      linux: "/images/pxeboot/vmlinuz quiet rhgb console=tty0 console=ttyS0,115200n8 systemd.show_status=1 root=live:CDLABEL=Margine-Live enforcing=0 rd.live.image rd.live.overlay.size=4096"
       initrd: "/images/pxeboot/initrd.img"
     - name: "Install Margine (Basic Graphics Mode)"
-      linux: "/images/pxeboot/vmlinuz quiet rhgb console=tty0 console=ttyS0,115200n8 systemd.show_status=1 root=live:CDLABEL=Margine-Live enforcing=0 rd.live.image nomodeset"
+      linux: "/images/pxeboot/vmlinuz quiet rhgb console=tty0 console=ttyS0,115200n8 systemd.show_status=1 root=live:CDLABEL=Margine-Live enforcing=0 rd.live.image rd.live.overlay.size=4096 nomodeset"
+      initrd: "/images/pxeboot/initrd.img"
+    - name: "Install Margine (verbose, no splash, no serial)"
+      # Field-debugging entry (added after the 2026-06-24 Beelink SER8 /
+      # Ryzen 8745H hang report). Drops `quiet rhgb` so the kernel + systemd
+      # print to the SCREEN, and drops console=ttyS0 so a machine with no
+      # real UART cannot stall every printk on serial TX. The bounded
+      # overlay keeps writable-layer sizing predictable.
+      linux: "/images/pxeboot/vmlinuz console=tty0 root=live:CDLABEL=Margine-Live enforcing=0 rd.live.image rd.live.overlay.size=4096"
       initrd: "/images/pxeboot/initrd.img"
   # Raw fragment appended after the generated entries (margine-pins /
   # upstream PR: grub2.extra_cfg). With Secure Boot ON the live CachyOS