Skip to content

Commit 731dec6

Browse files
ci(gate): real fresh-install Flatpak gate (Anaconda autoinstall + qemu-nbd verify) [skip ci]
* ci(gate): real fresh-install Flatpak gate (Anaconda autoinstall + qemu-nbd verify) Closes the test gap that let the SER8 broken-flatpak P0 ship (forum 12247): no CI ever did a real ISO install + first boot. New install_gate job: - live-env ships a DORMANT margine-autoinstall.service (ConditionKernelCommandLine=margine.autoinstall) + ci-autoinstall.ks. inst.ks/OEMDRV can't trigger Anaconda on this live ISO (its initrd has no anaconda-dracut modules), so a karg-gated 'anaconda --kickstart --cmdline' service is the trigger. No-op on every normal user boot. - qemu-install-wait.sh: boots the ISO's kernel+initrd via -kernel with margine.autoinstall injected, runs the headless install, detects the post-install reboot via QEMU -no-reboot exit. - verify-install-disk.sh: qemu-nbd offline-inspects the installed disk and asserts the bake (repo refs/remotes/flathub, flathub in repo/config, app count, var_lib_t SELinux label) — reusing inject-gui-probe's nbd dance. - ci-autoinstall.ks %includes the SAME post-scripts as the interactive install, so it exercises the real install-flatpaks.ks under test. Non-blocking vs publish_ia for now (new headless-install gate; surface first, tighten to gating once proven). * ci(gate): harden verify-install-disk (drop set -e, trace, btrfs subvol fallback) Phase 1 (headless install) passes; Phase 2 verify died silently with exit 2 before any check ran. A verify script must run ALL checks and report, not die on the first non-zero, so: set -uo pipefail (no -e), explicit error handling on the nbd/mount setup, set -x to trace any setup failure in the CI log, and an ostree-btrfs subvol=root / subvolid=5 mount fallback. * ci(gate): verify the DEDICATED /var subvol, not the empty stateroot stub Round-2 trace proved the install works and the btrfs root mounts; an adversarial review (one agent confirmed it on this very bootc host) then caught a blocker that would have false-FAILED round 3: margine.conf's default_partitioning gives /var its OWN btrfs subvol, and install-flatpaks.ks bakes into that (/mnt/sysimage/var/lib). The per-deployment stateroot var (/ostree/deploy/$sr/var) the previous verify read is just an empty .ostree-selabeled stub, so every assertion would fail on a perfect bake. - Mount the btrfs top (subvolid 5) and resolve VARLIB to the lib/ dir that actually contains flatpak/ ($MNT/var/lib), with find + subvol=var fallbacks. - SELinux: require var_lib_t exactly; drop the var_t arm (var_t is /var's own label — accepting it was a false-positive on the exact mislabel the gate exists to catch).
1 parent 78e88b3 commit 731dec6

6 files changed

Lines changed: 330 additions & 1 deletion

File tree

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
#!/usr/bin/env bash
2+
# Drive a headless automated Margine install in UEFI QEMU and wait for it to
3+
# finish (forum 12247 install gate). Boots the ISO's extracted kernel+initrd
4+
# with `margine.autoinstall` injected (fires the dormant autoinstall service)
5+
# while -cdrom serves the live squashfs and a blank target disk receives the
6+
# install. With -no-reboot the post-install `reboot` halts the VM, which is
7+
# our success signal; verify-install-disk.sh then inspects the disk offline.
8+
#
9+
# usage (root): qemu-install-wait.sh --cdrom ISO --disk QCOW \
10+
# --kernel vmlinuz --initrd initrd.img --log LOG --timeout SEC
11+
set -euo pipefail
12+
13+
CDROM="" DISK="" KERNEL="" INITRD="" LOG="install-serial.log" TIMEOUT=1800
14+
while [[ $# -gt 0 ]]; do
15+
case "$1" in
16+
--cdrom) CDROM="$2"; shift 2 ;;
17+
--disk) DISK="$2"; shift 2 ;;
18+
--kernel) KERNEL="$2"; shift 2 ;;
19+
--initrd) INITRD="$2"; shift 2 ;;
20+
--log) LOG="$2"; shift 2 ;;
21+
--timeout) TIMEOUT="$2"; shift 2 ;;
22+
*) echo "unknown arg: $1" >&2; exit 2 ;;
23+
esac
24+
done
25+
[[ -f "$CDROM" && -f "$DISK" && -f "$KERNEL" && -f "$INITRD" ]] \
26+
|| { echo "usage: --cdrom --disk --kernel --initrd required (files must exist)" >&2; exit 2; }
27+
28+
OVMF_CODE=""
29+
for c in /usr/share/OVMF/OVMF_CODE_4M.fd /usr/share/OVMF/OVMF_CODE.fd; do
30+
[[ -f "$c" ]] && OVMF_CODE="$c" && break
31+
done
32+
OVMF_VARS_SRC=""
33+
for v in /usr/share/OVMF/OVMF_VARS_4M.fd /usr/share/OVMF/OVMF_VARS.fd; do
34+
[[ -f "$v" ]] && OVMF_VARS_SRC="$v" && break
35+
done
36+
[[ -n "$OVMF_CODE" && -n "$OVMF_VARS_SRC" ]] || { echo "✗ OVMF not found"; ls -la /usr/share/OVMF/; exit 1; }
37+
cp "$OVMF_VARS_SRC" ovmf_vars_install.fd; chmod 0644 ovmf_vars_install.fd
38+
39+
# margine.autoinstall fires the dormant service; systemd.unit=multi-user.target
40+
# skips GNOME so anaconda --cmdline runs unobstructed; console=ttyS0 for the
41+
# log; the rd.live args + -cdrom provide the live root.
42+
APPEND="root=live:CDLABEL=Margine-Live rd.live.image rd.live.overlay.size=4096 enforcing=0 margine.autoinstall systemd.unit=multi-user.target console=tty0 console=ttyS0,115200n8 systemd.show_status=1"
43+
44+
rm -f qemu-install.pid "$LOG"
45+
qemu-system-x86_64 \
46+
-enable-kvm -m 6144 -smp 4 -machine q35 \
47+
-drive "if=pflash,format=raw,readonly=on,file=$OVMF_CODE" \
48+
-drive if=pflash,format=raw,file=ovmf_vars_install.fd \
49+
-cdrom "$CDROM" \
50+
-drive "file=$DISK,format=qcow2,if=virtio" \
51+
-kernel "$KERNEL" -initrd "$INITRD" -append "$APPEND" \
52+
-netdev user,id=n0 -device virtio-net-pci,netdev=n0 \
53+
-serial "file:$LOG" -display none -no-reboot \
54+
-daemonize -pidfile qemu-install.pid
55+
QPID="$(cat qemu-install.pid)"
56+
echo "install QEMU PID: $QPID (timeout ${TIMEOUT}s)"
57+
cleanup() { kill "$QPID" 2>/dev/null || true; sleep 2; kill -9 "$QPID" 2>/dev/null || true; chmod a+r "$LOG" 2>/dev/null || true; }
58+
trap cleanup EXIT
59+
60+
FAIL_RE='MARGINE-BAKE-FAIL|anaconda: traceback|Pane is dead|Aborting|installation failed|Kickstart error|Could not find a kickstart'
61+
for (( i = 1; i <= TIMEOUT; i++ )); do
62+
if [[ -f "$LOG" ]] && grep -qE "$FAIL_RE" "$LOG"; then
63+
echo "✗ install failure marker on serial:"; grep -E "$FAIL_RE" "$LOG" | head -5
64+
tail -100 "$LOG"; exit 1
65+
fi
66+
if ! kill -0 "$QPID" 2>/dev/null; then
67+
echo "✓ QEMU exited at ~${i}s (post-install reboot under -no-reboot) — install finished"
68+
grep -E 'MARGINE-BAKE-OK|MARGINE-BAKE-FAIL|Performing post|reboot' "$LOG" | tail -10 || true
69+
exit 0
70+
fi
71+
sleep 1
72+
done
73+
echo "✗ install did NOT finish within ${TIMEOUT}s"; tail -150 "$LOG"; exit 1
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
#!/usr/bin/env bash
2+
# Offline-verify a freshly installed Margine target disk for the Flatpak
3+
# bake (forum 12247 install gate). Exposes the qcow2 via qemu-nbd (reusing
4+
# the hardened nbd dance from inject-gui-probe.sh), mounts the btrfs root
5+
# read-only, and asserts the baked /var/lib/flatpak that broke on the SER8
6+
# is present, configured, populated, and correctly SELinux-labeled.
7+
# usage (root): verify-install-disk.sh <target.qcow2>
8+
#
9+
# NB: deliberately NOT `set -e` — a verify script must run ALL checks and
10+
# report, not die on the first non-zero. Critical setup steps check
11+
# explicitly; assertions accumulate into $fail.
12+
set -uo pipefail
13+
shopt -s nullglob
14+
15+
QCOW="${1:?usage: verify-install-disk.sh <target.qcow2>}"
16+
NBD=/dev/nbd0
17+
MNT=/mnt/verifyroot
18+
set -x # trace: pinpoint any setup failure in CI logs
19+
20+
cleanup() {
21+
set +x
22+
for _ in 1 2 3; do mountpoint -q "$MNT" || break; umount -R "$MNT" 2>/dev/null && break; sleep 1; done
23+
sync
24+
qemu-nbd --disconnect "$NBD" >/dev/null 2>&1 || true
25+
}
26+
trap cleanup EXIT
27+
28+
qemu-nbd --disconnect "$NBD" >/dev/null 2>&1 || true
29+
modprobe -r nbd 2>/dev/null || true
30+
modprobe nbd max_part=16 2>/dev/null || modprobe nbd 2>/dev/null || true
31+
[[ -e /dev/nbd0 ]] || { set +x; echo "::error::nbd module unavailable (/dev/nbd0 missing)"; exit 1; }
32+
33+
if ! qemu-nbd --connect="$NBD" "$QCOW"; then
34+
set +x; echo "::error::qemu-nbd --connect failed for $QCOW"; exit 1
35+
fi
36+
partprobe "$NBD" 2>/dev/null || partx -u "$NBD" 2>/dev/null || true
37+
parts=()
38+
for _ in $(seq 1 20); do
39+
parts=("$NBD"p*)
40+
if [[ ${#parts[@]} -gt 0 ]]; then break; fi
41+
sleep 1
42+
partprobe "$NBD" 2>/dev/null || true
43+
done
44+
45+
ROOT=""
46+
for p in "$NBD"p*; do
47+
if [[ "$(blkid -o value -s TYPE "$p" 2>/dev/null)" == "btrfs" ]]; then ROOT="$p"; break; fi
48+
done
49+
if [[ -z "$ROOT" ]]; then
50+
set +x; echo "::error::no btrfs root partition on $QCOW"; lsblk "$NBD" 2>/dev/null || true; blkid "$NBD"p* 2>/dev/null || true; exit 1
51+
fi
52+
mkdir -p "$MNT"
53+
# Mount the btrfs TOP (subvolid 5) so EVERY subvol is traversable in one mount.
54+
# CRITICAL (review wzskaqvwo, confirmed on this bootc host): Margine's
55+
# margine.conf default_partitioning gives /var its OWN btrfs subvol, and
56+
# install-flatpaks.ks bakes into THAT dedicated /var (/mnt/sysimage/var/lib) —
57+
# NOT the per-deployment stateroot var (/ostree/deploy/$sr/var), which is just
58+
# an empty .ostree-selabeled stub. At the btrfs top the bake therefore lives at
59+
# $MNT/var/lib/flatpak. Reading the stateroot stub would FALSE-FAIL every check.
60+
mount -o ro,subvolid=5 "$ROOT" "$MNT" 2>/dev/null \
61+
|| mount -o ro "$ROOT" "$MNT" 2>/dev/null \
62+
|| { set +x; echo "::error::could not mount btrfs root $ROOT"; exit 1; }
63+
64+
# VARLIB = the lib/ dir that actually contains the baked flatpak/ (the dedicated
65+
# /var subvol), located robustly across subvol layouts.
66+
VARLIB=""
67+
for cand in "$MNT/var/lib" "$MNT/root/var/lib"; do
68+
if [[ -d "$cand/flatpak" ]]; then VARLIB="$cand"; break; fi
69+
done
70+
if [[ -z "$VARLIB" ]]; then
71+
fp="$(find "$MNT" -maxdepth 6 -type d -path '*/var/lib/flatpak' 2>/dev/null | head -1)"
72+
[[ -n "$fp" ]] && VARLIB="$(dirname "$fp")"
73+
fi
74+
if [[ -z "$VARLIB" ]]; then
75+
# last resort: mount the dedicated var subvol explicitly
76+
umount -R "$MNT" 2>/dev/null
77+
if mount -o ro,subvol=var "$ROOT" "$MNT" 2>/dev/null && [[ -d "$MNT/lib/flatpak" ]]; then VARLIB="$MNT/lib"; fi
78+
fi
79+
set +x
80+
if [[ -z "$VARLIB" ]]; then
81+
echo "::error::no var/lib/flatpak on installed disk — bake missing or unexpected subvol layout"
82+
ls -laR "$MNT" 2>/dev/null | head -80 || true
83+
exit 1
84+
fi
85+
echo "varlib=$VARLIB (dedicated /var btrfs subvol)"
86+
ls -la "$VARLIB/flatpak" 2>/dev/null | head -20 || true
87+
88+
fail=0
89+
ok() { printf ' OK %s\n' "$1"; }
90+
bad() { printf '::error::%s\n' "$1"; fail=1; }
91+
92+
[[ -d "$VARLIB/flatpak/repo/refs/remotes/flathub" ]] \
93+
&& ok "flatpak repo has refs/remotes/flathub" \
94+
|| bad "flatpak repo MISSING refs/remotes/flathub (the exact SER8 break)"
95+
96+
grep -q 'remote "flathub"' "$VARLIB/flatpak/repo/config" 2>/dev/null \
97+
&& ok "flathub present in repo/config" \
98+
|| bad "flathub not in flatpak/repo/config"
99+
100+
N=$(find "$VARLIB/flatpak/app" -maxdepth 1 -mindepth 1 -type d 2>/dev/null | wc -l)
101+
echo " baked apps under flatpak/app: $N"
102+
[[ "$N" -ge 10 ]] && ok "app count $N >= 10" || bad "only $N baked apps (expected dozens)"
103+
104+
ctx="$(getfattr -n security.selinux --only-values "$VARLIB/flatpak" 2>/dev/null | tr -d '\0' || true)"
105+
echo " /var/lib/flatpak SELinux context: ${ctx:-<unreadable>}"
106+
case "$ctx" in
107+
*var_lib_t*) ok "SELinux context is var_lib_t" ;;
108+
"") echo " ::warning::could not read security.selinux xattr (non-fatal)" ;;
109+
*) bad "SELinux context '$ctx' wrong — /var/lib/flatpak must be var_lib_t (var_t is NOT enough)" ;;
110+
esac
111+
112+
echo
113+
if [[ "$fail" -eq 0 ]]; then echo "MARGINE-INSTALL-FLATPAK: PASS"; else echo "MARGINE-INSTALL-FLATPAK: FAIL"; fi
114+
exit "$fail"

.github/workflows/build-disk.yml

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -493,6 +493,74 @@ jobs:
493493
done
494494
echo "Pruned $PRUNED stale version(s)."
495495
496+
install_gate:
497+
name: Automated install gate (Flatpak bake)
498+
# Reproduces a real fresh ISO install + verifies /var/lib/flatpak — the
499+
# path that broke on the Beelink SER8 (forum 12247) and that no other CI
500+
# job exercises (smoke-boot uses a qcow2 from the OCI image; the boot-test
501+
# only boots the live session). Runs whenever the live ISO is built for
502+
# stable. Does NOT block publish_ia (a new headless-install gate; surface
503+
# first, tighten to blocking once proven).
504+
needs: build_iso_titanoboa
505+
if: ${{ (github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'iso-test'))) && (inputs.image_tag == '' || inputs.image_tag == 'stable') }}
506+
runs-on: ubuntu-24.04
507+
timeout-minutes: 60
508+
permissions:
509+
contents: read
510+
steps:
511+
- name: Maximize build space
512+
uses: ublue-os/remove-unwanted-software@cc0becac701cf642c8f0a6613bbdaf5dc36b259e # v9
513+
with:
514+
remove-codeql: true
515+
516+
- name: Checkout
517+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
518+
519+
- name: Download Live ISO artifact
520+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
521+
with:
522+
name: margine-live-iso
523+
path: ./iso
524+
525+
- name: Install QEMU + tools
526+
run: |
527+
sudo apt-get update -qq
528+
# qemu-utils=qemu-nbd, parted+util-linux=nbd partition exposure,
529+
# xorriso=ISO extract (no sudo mount), attr=getfattr (SELinux check).
530+
sudo apt-get install -y -qq qemu-system-x86 qemu-utils ovmf xorriso parted util-linux attr
531+
532+
- name: Extract kernel + initrd from the ISO
533+
run: |
534+
set -euo pipefail
535+
ISO="$(ls ./iso/*.iso | head -1)"
536+
echo "ISO=$ISO"
537+
xorriso -osirrox on -indev "$ISO" \
538+
-extract /images/pxeboot/vmlinuz ./ci-vmlinuz \
539+
-extract /images/pxeboot/initrd.img ./ci-initrd.img
540+
chmod a+r ./ci-vmlinuz ./ci-initrd.img
541+
542+
- name: Phase 1 — headless automated install
543+
run: |
544+
set -euo pipefail
545+
ISO="$(ls ./iso/*.iso | head -1)"
546+
qemu-img create -f qcow2 target.qcow2 20G
547+
sudo -E .github/scripts/qemu-install-wait.sh \
548+
--cdrom "$ISO" --disk target.qcow2 \
549+
--kernel ./ci-vmlinuz --initrd ./ci-initrd.img \
550+
--log install-serial.log --timeout 2400
551+
552+
- name: Phase 2 — offline-verify the installed disk (qemu-nbd)
553+
run: sudo -E .github/scripts/verify-install-disk.sh target.qcow2
554+
555+
- name: Upload install-gate logs
556+
if: always()
557+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
558+
with:
559+
name: install-gate-logs
560+
path: install-serial.log
561+
retention-days: 7
562+
if-no-files-found: ignore
563+
496564
publish_ia:
497565
name: Publish ISO (${{ matrix.image }}) to Internet Archive
498566
# 2026-06-11: publishes the Titanoboa live ISO (ADR-0008 Phase 5).
@@ -807,7 +875,7 @@ jobs:
807875
808876
notify:
809877
name: ntfy notification (aggregated)
810-
needs: [build_disk, build_iso_titanoboa, publish_ia, bump_site]
878+
needs: [build_disk, build_iso_titanoboa, install_gate, publish_ia, bump_site]
811879
if: always()
812880
runs-on: ubuntu-24.04
813881
timeout-minutes: 5
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Margine CI install-gate kickstart (forum 12247). NOT shipped behaviour —
2+
# only runs when the live boot carries `margine.autoinstall` (CI only, via
3+
# margine-autoinstall.service). It reproduces the REAL bake: ostreecontainer
4+
# pulls margine:stable and the SAME %include post-scripts the interactive
5+
# install uses run — including install-flatpaks.ks, the thing under test.
6+
cmdline
7+
firstboot --disable
8+
lang en_US.UTF-8
9+
keyboard us
10+
timezone UTC --utc
11+
network --bootproto=dhcp --activate
12+
rootpw --plaintext margineci
13+
14+
# Storage: BTRFS autopart, mirroring margine.conf's AUTOMATIC scheme. ostree
15+
# always carves /var into its own subvol (/ostree/deploy/$sr/var) — exactly
16+
# what install-flatpaks.ks targets via /mnt/sysimage/var. (No anaconda-webui
17+
# here on the --cmdline path, so the webui-68 `part`-crash does not apply.)
18+
zerombr
19+
clearpart --all --initlabel
20+
autopart --type=btrfs --noswap
21+
22+
# Same install source as interactive-defaults.ks (needs network).
23+
ostreecontainer --url=ghcr.io/daniel-g-carrasco/margine:stable --transport=registry --no-signature-verification
24+
25+
# Auto-reboot when done so QEMU -no-reboot halts the VM for offline verify.
26+
reboot
27+
28+
# The REAL post-install path under test — identical %includes to
29+
# interactive-defaults.ks (minus secureboot-enroll-key.ks: a no-op MOK stage
30+
# with Secure Boot off in the headless CI VM).
31+
%include /usr/share/anaconda/post-scripts/bootc-switch.ks
32+
%include /usr/share/anaconda/post-scripts/zstd-compress.ks
33+
%include /usr/share/anaconda/post-scripts/disable-fedora-flatpak.ks
34+
%include /usr/share/anaconda/post-scripts/install-flatpaks.ks
35+
%include /usr/share/anaconda/post-scripts/flatpak-restore-selinux-labels.ks
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# CI-ONLY automated install trigger (forum 12247 install-gate).
2+
#
3+
# The Margine live ISO cannot use inst.ks= / OEMDRV: its initramfs has the
4+
# dmsquash-live hooks but ZERO anaconda-dracut modules, so those installer
5+
# triggers never fire. Instead this dormant unit (shipped in the LIVE env
6+
# only, never the installed system) runs Anaconda non-interactively when —
7+
# and only when — the live boot carries the `margine.autoinstall` kernel
8+
# arg, which CI injects via -append. A normal user boot never has that arg,
9+
# so this is a no-op for everyone except the CI install gate.
10+
[Unit]
11+
Description=Margine CI automated install (margine.autoinstall karg only)
12+
ConditionKernelCommandLine=margine.autoinstall
13+
After=network-online.target livesys.service
14+
Wants=network-online.target
15+
16+
[Service]
17+
Type=oneshot
18+
StandardOutput=journal+console
19+
StandardError=journal+console
20+
# --cmdline = fully non-interactive; aborts (not prompts) on any gap.
21+
ExecStart=/usr/sbin/anaconda --kickstart=/usr/share/anaconda/ci-autoinstall.ks --cmdline
22+
# Anaconda reboots via the `reboot` in the ks; with QEMU -no-reboot that
23+
# halts the VM so the gate can open the installed disk cleanly.
24+
25+
[Install]
26+
WantedBy=multi-user.target

live-env/src/build.sh

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,19 @@ install -Dm0644 "$SRC_DIR/anaconda/profile.d/margine.conf" \
274274
# (same content as the Phase 2 line above) plus bootc-switch + zstd.
275275
cp "$SRC_DIR"/anaconda/post-scripts/*.ks /usr/share/anaconda/post-scripts/
276276

277+
# CI install-gate (forum 12247): a dormant automated-install trigger. Its
278+
# ConditionKernelCommandLine=margine.autoinstall makes it a no-op on every
279+
# normal user boot; CI injects that karg to run a headless Anaconda install
280+
# and verify the Flatpak bake end to end. Live-env only — never reaches the
281+
# installed system.
282+
if [[ -d "$SRC_DIR/autoinstall" ]]; then
283+
install -Dm0644 "$SRC_DIR/autoinstall/ci-autoinstall.ks" \
284+
/usr/share/anaconda/ci-autoinstall.ks
285+
install -Dm0644 "$SRC_DIR/autoinstall/margine-autoinstall.service" \
286+
/usr/lib/systemd/system/margine-autoinstall.service
287+
systemctl enable margine-autoinstall.service
288+
fi
289+
277290
# Append Margine partitioning + ostreecontainer + %includes to the base
278291
# interactive-defaults.ks anaconda-live ships (preserve the base).
279292
cat "$SRC_DIR/anaconda/interactive-defaults.ks" \

0 commit comments

Comments
 (0)