docs/composefs: Update with latest

In particular, this commits us to supporting in-place upgrades
from the current state!

We will leave things like the EROFS v1 switch as a followon,
ensuring that we continue to support that as an upgrade path.

Signed-off-by: Colin Walters <walters@verbum.org>

Author: cgwalters

docs/src/experimental-composefs.md

@@ -5,14 +5,34 @@ do provide feedback on them.
 
 ## Overview
 
-The composefs backend is an experimental alternative storage backend that uses [composefs-rs](https://github.com/containers/composefs-rs) instead of ostree for storing and managing bootc system deployments.
+The composefs backend is an experimental alternative storage backend that uses [composefs-rs](https://github.com/composefs/composefs-rs) instead of ostree for storing and managing bootc system deployments.
 
-**Status**: Experimental. The composefs backend is under active development and not yet suitable for production use. The feature is always compiled in as of bootc v1.10.1.
+**Status**: Experimental, but close to stabilization! We are committed to in-place upgrades from all systems deployed since bootc 1.16.0.
 
-A key goal is custom "sealed" images, signed with your own Secure Boot keys.
-This is based on [Unified Kernel Images](https://uapi-group.org/specifications/specs/unified_kernel_image/)
-that embed a digest of the target container root filesystem, typically alongside a bootloader (such
-as systemd-boot) also signed with your key.
+The composefs backend supports two distinct levels of integrity guarantee, controlled by whether fsverity is strictly enforced on the root filesystem (i.e. whether the image was built with `--allow-missing-verity`):
+
+- **Sealed**: The composefs digest is baked into the kernel command line of a UKI and *required* to match at boot.
+- **Unsealed**: fsverity enforcement is optional, so composefs still provides content-addressed, deduplicated storage and garbage collection, but without a guarantee that the root filesystem matches what was signed. Unsealed composefs most commonly boots via a traditional `vmlinuz`/`initramfs.img` and a BLS boot entry, but a UKI built with `--allow-missing-verity` is *also* unsealed in this sense — packaging as a UKI is a boot convenience here, not by itself a security boundary. See [Bootloader Support](#bootloader-support) below.
+
+## Storage and repository structure
+
+Unlike the ostree backend, which keeps its repository at `/ostree/repo`, the composefs backend splits its on-disk state across two top-level directories in the physical sysroot:
+
+- `/composefs`: The [composefs-rs repository](https://github.com/composefs/composefs-rs/blob/main/crates/composefs/src/repository_format.rs) (mode `0700`), containing:
+  - `objects/`: content-addressed file storage, keyed by SHA-512 fsverity digest and shared via reflink (`FICLONE`) where the filesystem supports it
+  - `images/`: EROFS images describing each deployment's root filesystem metadata
+  - `streams/`: OCI manifest, config, and layer splitstreams captured during image pulls
+  - `bootc/storage/`: the `containers-storage:` instance backing logically bound images, reflink-shared with the composefs object store
+- `/state/deploy/<deployment-id>/`: Persistent per-deployment state, one directory per deployment (named after its composefs digest):
+  - `etc/`: a writable copy of the deployment's `/etc`, bind-mounted onto the booted root's `/etc`
+  - `var`: a symlink to the shared `/state/os/default/var`, bind-mounted onto the booted root's `/var`
+  - `<deployment-id>.origin`: an INI file recording the image reference, boot type (BLS or UKI) and digest, and the OCI manifest digest (the latter is what keeps a deployment's objects alive across garbage collection)
+
+Although composefs-rs supports other fsverity hash algorithms, bootc currently hardcodes `SHA-512` for the repository (see `Algorithm::SHA512` at every repository init/open call site, and the `ComposefsRepository` type alias in `crates/lib/src/store/mod.rs`). This is why deployment and object identifiers throughout this document (and in `bootc status`) are 128-character hex strings.
+
+There is no `/ostree/repo`; the composefs backend doesn't use the ostree repository at all. A minimal `/ostree` directory is still created, but only to hold a compatibility symlink (`ostree/bootc -> ../composefs/bootc`) so that existing tooling expecting `/usr/lib/bootc/storage` to resolve through `ostree/bootc` keeps working.
+
+Transient, not-yet-finalized deployment state (used while staging an update before reboot) lives under `/run/composefs/staged-deployment` and is never persisted to disk.
 
 ## How Sealed Images Work
 
@@ -61,59 +81,86 @@ valid use case is to temporarily disable it in order to test a change locally
 on e.g. one machine, then re-enable it later. However at the current time it
 is not yet streamlined to regenerate the UKI locally.
 
-### Build Pattern: Compute Digest and Generate UKI in One Stage
+Note this is a different, independent weakening from `--allow-missing-verity`
+(see [Overview](#overview) above): disabling Secure Boot only removes firmware
+verification of the UKI's own signature, while the fsverity digest of the root
+filesystem is still enforced. Building with `--allow-missing-verity` instead
+disables that digest enforcement itself, which is what actually makes an image
+unsealed regardless of whether Secure Boot is enabled.
+
+### Build Pattern: Split the Kernel, Then Generate the UKI in a Separate Stage
 
-The key to building sealed images is using a multi-stage Dockerfile where a separate stage mounts the target rootfs, computes its composefs digest, and generates the signed UKI in one step:
+Building a sealed image involves three stages: build the rootfs, split the kernel and initramfs out of it, and generate the signed UKI from the split rootfs in a tools stage:
 
 ```dockerfile
 # Build your rootfs with all packages and configuration
 FROM <base-image> as rootfs
 RUN apt|dnf|zypper install ... && bootc container lint --fatal-warnings
 
+# Split the kernel and initramfs out of the rootfs. This moves
+# /usr/lib/modules/<kver>/{vmlinuz,initramfs.img} into /kernel/<kver>/,
+# since for a sealed image they end up embedded in the UKI instead.
+FROM rootfs as split
+RUN mkdir /kernel && bootc container split-kernel-and-rootfs --rootfs / --output /kernel
+
 # Generate the sealed UKI in a tools stage
 FROM <tools-image> as sealed-uki
-RUN --mount=type=bind,from=rootfs,target=/target \
+RUN --mount=type=bind,from=split,target=/target \
+    --mount=type=bind,from=split,source=/kernel,target=/kernel \
     --mount=type=secret,id=secureboot_key \
     --mount=type=secret,id=secureboot_cert <<EORUN
 set -euo pipefail
 
-# Compute the composefs digest from the mounted rootfs
-digest=$(bootc container compute-composefs-digest /target)
-
-# Find the kernel version
-kver=$(ls /target/usr/lib/modules)
-
-# Generate and sign the UKI with the digest embedded
-ukify build \
-  --linux "/target/usr/lib/modules/${kver}/vmlinuz" \
-  --initrd "/target/usr/lib/modules/${kver}/initramfs.img" \
-  --cmdline "composefs=${digest} rw" \
-  --os-release "@/target/usr/lib/os-release" \
+mkdir -p /out
+kver=$(ls /kernel)
+
+# `bootc container ukify` computes the composefs digest of /target, reads
+# extra kernel arguments from /target/usr/lib/bootc/kargs.d, and invokes the
+# real `ukify` binary with the digest embedded in the cmdline. Everything
+# after `--` is passed straight through to ukify.
+bootc container ukify \
+  --rootfs /target \
+  --kernel-dir "/kernel/${kver}" \
+  -- \
+  --output "/out/${kver}.efi" \
   --signtool sbsign \
   --secureboot-private-key /run/secrets/secureboot_key \
-  --secureboot-certificate /run/secrets/secureboot_cert \
-  --output "/out/${kver}.efi"
+  --secureboot-certificate /run/secrets/secureboot_cert
 EORUN
 
-# Final image: copy the sealed UKI into place
-FROM rootfs
+# Final image: the split rootfs (kernel/initramfs already removed) plus the signed UKI
+FROM split
 COPY --from=sealed-uki /out/*.efi /boot/EFI/Linux/
 ```
 
 This pattern works because:
 
-1. The `--mount=type=bind,from=rootfs` provides read-only access to the target filesystem
-2. `bootc container compute-composefs-digest` computes the SHA-512 hash of the rootfs
-3. `ukify` creates the UKI with that digest in the kernel command line (`composefs=<digest>`)
-4. The final stage copies the signed UKI into the rootfs without modifying any files used in the digest calculation
+1. `bootc container split-kernel-and-rootfs` removes the raw kernel and initramfs from the rootfs ahead of time, so the final image never carries a duplicate copy of them (they end up embedded in the UKI instead)
+2. `bootc container ukify` handles computing the composefs digest and assembling the kernel command line, so you only need to pass `ukify`-specific options (like signing) after `--`
+3. The final stage copies the signed UKI into the already-split rootfs
+
+### The `bootc container ukify` Command
+
+```bash
+bootc container ukify --rootfs <PATH> [OPTIONS] -- [UKIFY_ARGS...]
+```
+
+This is the recommended way to build a UKI for a bootc image. It computes the composefs digest of `--rootfs` (using the lower-level `compute-composefs-digest` primitive described below), reads extra kernel arguments from `/usr/lib/bootc/kargs.d`, and invokes the system `ukify` binary with the resulting cmdline. Anything after `--` is forwarded to `ukify` unchanged (e.g. `--output`, `--signtool`, signing key/cert options).
+
+**Options:**
+
+- `--rootfs <PATH>`: Root filesystem to operate on (default: `/`)
+- `--kernel-dir <PATH>`: Directory containing `vmlinuz`/`initramfs.img`, named `/parent/<kernel-version>`. Needed when the kernel has already been split out of `--rootfs`, e.g. via `split-kernel-and-rootfs`
+- `--allow-missing-verity`: Make fsverity validation optional, for filesystems that don't support it (e.g. XFS)
+- `--write-dumpfile-to <PATH>`: Write a composefs dumpfile for debugging
 
 ### The `bootc container compute-composefs-digest` Command
 
 ```bash
 bootc container compute-composefs-digest [PATH]
 ```
 
-Computes the composefs digest for a filesystem. The digest is a 128-character SHA-512 hex string that uniquely identifies the filesystem contents.
+A lower-level primitive, used internally by `ukify` above, that computes just the composefs digest for a filesystem without building a UKI. The digest is a 128-character SHA-512 hex string that uniquely identifies the filesystem contents. Useful for scripting or debugging outside of the UKI build flow.
 
 **Options:**
 
@@ -146,7 +193,9 @@ See [CONTRIBUTING.md](https://github.com/bootc-dev/bootc/blob/main/CONTRIBUTING.
 
 ## Bootloader Support
 
-To use sealed images, the container image must have a UKI and systemd-boot installed (and not have `bootupd`). If these conditions are met, bootc will automatically detect and use the composefs backend during installation.
+Whenever the container image has a UKI, bootc automatically selects the composefs backend during installation (see [Prerequisites](#prerequisites) above for the currently-supported UKI + systemd-boot configuration for building sealed images). Note that having a UKI does not by itself make an install sealed — that also depends on whether fsverity enforcement is on, per [Overview](#overview) above.
+
+Composefs installs using a traditional `vmlinuz`/`initramfs.img` layout instead of a UKI are always unsealed, and can use either `bootupd` (GRUB) or systemd-boot, the same as the ostree backend. See [bootloaders.md](bootloaders.md) for the general bootloader selection rules. Under the hood, bootc writes standard BLS boot entries for both UKI and traditional kernels; see the [composefs boot module documentation](https://github.com/bootc-dev/bootc/blob/main/crates/lib/src/bootc_composefs/boot.rs) for details on how entry filenames and sort-keys are chosen to sort correctly on both GRUB and systemd-boot.
 
 ## Installation
 
@@ -156,33 +205,24 @@ There is a `--composefs-backend` option for `bootc install` to explicitly select
 
 The composefs backend is experimental; on-disk formats are subject to change.
 
-### Deployment blockers
+### Important
 
-- [Garbage collection](https://github.com/bootc-dev/bootc/pull/2040): In progress
 - Extended install APIs: Ability to cleanly implement anaconda %post and osbuild post mutations and general post-install pre-reboot; right now some tools just mount the deployment directory (note this one also relates to [APIs in general](https://github.com/bootc-dev/bootc/issues/522))
-- [OCI registry install](https://github.com/bootc-dev/bootc/issues/1703): Installing from registry can fail due to config mismatch (suggestion: just clean reject v2s2)
-- [composefs-rs repository finalization](https://github.com/bootc-dev/bootc/issues/1320)
+- [Dual EROFS v1/v2 generation](https://github.com/bootc-dev/bootc/pull/2248): `bootc install` currently generates both EROFS v1 and v2 images for every deployment, since v1 is required on older kernels (e.g. RHEL 9/CentOS Stream 9) while v2 is the modern native format. The overhead is small since EROFS images only hold metadata, but this dual generation is a stopgap until we can drop v1 support.
 
-### Important
+## Related issues
 
-- Extended test suite: Right now we're not covering upgrades well, need to build upgrade image in sealed cases
-- Full workflow test - add composefs into https://gitlab.com/fedora/bootc/tests/bootc-workflow-test for example
-  - Workflow upgrades especially "from old systems"
 - [Unified storage](https://github.com/bootc-dev/bootc/issues/20): Not strictly a blocker but a really nice to have
 - [Sealed image build UX](https://github.com/bootc-dev/bootc/issues/1498): Streamlined tooling for building sealed images
 - In place transitions: 
   - First: support [factory reset](https://github.com/bootc-dev/bootc/issues/404) from ostree to composefs
   - Next: Support copying /etc and /var
-- A lot more practical level docs for using this 
-
-### Minor
-
-- Remove `/usr/lib/bootc/kargs.d` as part of UKI creation (also `bootc container inspect` should show UKI kargs)
 
 ## Additional Resources
 
 - See [filesystem.md](filesystem.md) for information about composefs in the standard ostree backend
 - See [bootloaders.md](bootloaders.md) for bootloader configuration details
-- [composefs-rs](https://github.com/containers/composefs-rs) - The underlying composefs implementation
+- [composefs-rs](https://github.com/composefs/composefs-rs) - The underlying composefs implementation
+- [composefs-rs repository format](https://github.com/composefs/composefs-rs/blob/main/crates/composefs/src/repository_format.rs) - Detailed on-disk layout of the `/composefs` repository
 - [Unified Kernel Images specification](https://uapi-group.org/specifications/specs/unified_kernel_image/)
 - [ukify documentation](https://www.freedesktop.org/software/systemd/man/latest/ukify.html) - Tool for building UKIs