Software RAID: adopting discovered MD arrays without a RAIDArrayConfig

[mcanevet]

Context

I provision bare metal with Tinkerbell and deploy Talos by writing a disk
image, not by running the installer: the workflow creates a RAID1 mirror with
mdadm --create --name boot --homehost=talos --level=1 --metadata=1.0, then
streams an Image Factory metal-amd64-secureboot.raw.zst directly onto
/dev/mdN. The array therefore exists before Talos ever boots, and the
installer never runs on these nodes.

On v1.14.0 this works well, with no RAID-related machine config at all:

• SystemDisks reports /dev/md127, and EFI, META, STATE and
  EPHEMERAL all live on partitions of the array.
• Disks.block.talos.dev for the array already reports
  secondary_disks: [nvme2n1, nvme3n1] and the
  /dev/disk/by-id/md-name-talos:boot symlink.
• MDLastResortController force-runs the array if udev leaves it inactive.

Since firmware has no MD driver and the ESP is a partition inside the array,
the fact that it boots at all confirms the metadata: "1.0" rationale in the
v1.14 release notes: the partition table written through the array stays
readable at the start of each raw member.

The gaps

Without a RAIDArrayConfig document there is no MDArraySpec, so
MDArrayReconcileController never sees the array. Consequently:

1. No health state in the API. talosctl get mdarraystatus is empty. A
   degraded mirror does get logged by MDMonitorController (I confirmed
   mdadm --monitor --scan is running), but there is no resource to query or
   alert on — I have to read /proc/mdstat.
2. No self-healing. If a member fails and is replaced, nothing re-adds the
   new disk; the array stays degraded until someone runs mdadm --add.

Why not just add a RAIDArrayConfig?

I can, but for a boot array in an image-based flow it seems like the wrong
shape, for four reasons:

1. It can never be load-bearing for assembly. The machine config lives on
   STATE, which is on the array, so the array must be assembled before any
   document can be read. A document can only ever describe, after the fact,
   something that already had to work without it — which is the asymmetry I am
   hitting: the array is trusted enough to boot from, but not enough to be
   monitored or repaired.
2. Membership is discovered hardware, not authored config. My workflow
   elects members at provisioning time by ranking disks (size, then
   by-path); their identity isn't known when the config is authored. That
   leaves either a deliberately broad selector, or writing per-node config back
   into my GitOps repo from the provisioning workflow.
3. The selector means "grow", which is not what a boot mirror wants.
   Reconciliation is additive and MDArraySpec has no member cap: unmatched
   disks are --added and the array --grown (as TestRAIDArrayGrow_RAID1
   does, 2 -> 3). These nodes have six further NVMe disks, so a
   slightly-too-broad selector grows the boot mirror rather than replacing a
   failed member. I found no way to express "exactly two members, replace but
   never grow".
4. It restates on-disk state that is already authoritative. The superblock
   records level, metadata version, UUID, name, homehost and raid-devices, and
   Talos already surfaces the members in secondary_disks. A CEL predicate is
   a second, weaker description that can drift from it.

What I'd like to explore

• Status for discovered arrays, spec-backed or not. This needs no policy
  and would close gap 1. Members are already known via secondary_disks.
• Opt-in hot-replace with no member declaration, using slot identity.

mdadm already defines exactly these semantics: POLICY action=spare-same-slot
("if given slot was used by an array that went degraded recently and the device
plugged in has no metadata then it will be automatically added to that array"),
with path= globbing /dev/disk/by-path. Talos also already ships the
recording half of it — 90-md-raid-assembly.rules runs
mdadm -If $name --path $env{ID_PATH} on removal, and --path is documented as
"the 'path' given will be recorded so that if a new device appears at the same
location it can be automatically added to the same array".

What appears to be missing is (a) any mdadm.conf — there is none on the node,
so no POLICY is ever consulted — and (b) a rule offering bare devices to
mdadm --incremental (Talos's rule only reaches --incremental for
*_raid_member); mdadm can generate that rule itself via mdadm --udev-rules=.

Questions, in case I have misread this:

1. Is enabling this through an mdadm.conf POLICY something you would
   consider, or would you rather see the same semantics implemented in a
   controller? A controller could be stricter than POLICY, since Talos knows
   from DiscoveredVolumes/Disks whether a candidate is genuinely unclaimed,
   which path=/type= cannot express.
2. Where does mdadm persist the --path record used by spare-same-slot? If
   it is only the map file under /run, the "recently degraded" knowledge would
   not survive a reboot, so a disk replaced while the node is down would not be
   picked up. Would persisting last-known-healthy member paths in META (as
   done for StateEncryptionConfig) be an acceptable pattern here?
3. Is treating a discovered, unmanaged array as first-class at all in scope, or
   is RAIDArrayConfig intended to be mandatory for anything beyond assembly?

I'm happy to work on a PR if there's a shape you prefer.

[mcanevet]

My proposal for 1st gap: #14339