fix(microvm): the admission budget was never checked against the host's RAM

SH_MAX_COMMITTED_MB=24576 less SH_MEMORY_RESERVE_MB=4096 is a 20 GiB admission
budget, and nothing in the worker reads MemTotal, so on a 16 GiB host the gate
cannot refuse before the OOM killer arrives. Every other invariant in this tier
fails at start rather than trusting the operator (AssertPathExists=/dev/kvm,
SH_MAX_COMMITTED_MB being mandatory, Config.Normalize's cross-field checks); the
one that was missing is the check against the host the unit is running on. It
has a measurement consequence too: a gate that never refuses reads as
confirmation of spec §7.4's prediction 1 rather than as its falsification.

AssertMemory=>=23G in [Unit]. systemd compares it against physical memory
(memory assigned to the container, in a container) and fails the unit at start,
the same posture as AssertPathExists -- and, like it, an assertion does not
trigger Restart=, so it cannot crash-loop. 23G against a 24 GiB budget rather
than 24G because physical memory here is MemTotal, always somewhat below nominal
RAM: a threshold equal to the budget would refuse to start on a genuine 24 GiB
host.

The test extracts the threshold, requires the >= comparison, normalises it to
MiB and bounds it on both sides of SH_MAX_COMMITTED_MB: at least 90% of it (so
raising the budget alone is a test failure, not a discovery made by the OOM
killer) and no more than it (so an assertion above nominal RAM, which would
refuse to start on the very host it was written for, is also a test failure).
Verified red on three mutations: the assertion deleted (3 checks),
SH_MAX_COMMITTED_MB raised to 65536 alone, and the threshold moved to 25G.

Parse-checked against real systemd, since an unknown key would be ignored with
only a warning: systemd 255 (ubuntu:24.04) and 249 (ubuntu:22.04) both accept
AssertMemory and the StartLimit keys, reporting nothing but the absent ExecStart
binary.

This is the unit-level form of the reviewer's suggestion. A MemTotal comparison
inside Config.Normalize would be the other half; internal/vmpool/ is outside
this change's scope, so it is not done here.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>

Author: pdettori

deploy/microvm/microvm-worker.service

@@ -68,6 +68,31 @@ AssertPathExists=/dev/kvm
 # reintroducing.
 StartLimitIntervalSec=300
 StartLimitBurst=5
+# The admission budget below (SH_MAX_COMMITTED_MB) is a promise about how much memory
+# this host has, and nothing in the worker reads MemTotal. On a host with less RAM than
+# the budget the gate therefore cannot refuse before the OOM killer arrives -- it admits
+# VMs against memory that does not exist, and spec §7.4's prediction 1 ("replenishment
+# binds on process/memory count before CPU") gets "confirmed" by a gate that never
+# refuses at all. Every other invariant in this tier fails at start rather than trusting
+# the operator (AssertPathExists=/dev/kvm above, SH_MAX_COMMITTED_MB being mandatory,
+# Config.Normalize's cross-field checks); the one that was missing is the check against
+# the host this unit is actually running on. A comment is not a control.
+#
+# AssertMemory compares the amount of PHYSICAL memory in the system (in a container, the
+# memory assigned to it) -- systemd's ConditionMemory family, available since v244 -- so
+# it is the same fact the budget is asserting, read from the host. Assertion rather than
+# Condition deliberately: a Condition silently skips the start, an Assert fails it
+# loudly. And like AssertPathExists it does NOT trigger Restart=, so it cannot produce
+# the crash-loop the StartLimit keys above exist to bound.
+#
+# 23G against a 24 GiB SH_MAX_COMMITTED_MB, not 24G: physical memory here is MemTotal,
+# which is always somewhat BELOW nominal RAM (firmware and kernel reservations), so a
+# threshold equal to the budget would refuse to start on a genuine 24 GiB host. The band
+# is deliberate -- this catches "the budget is bigger than the machine" (a 16 GiB host
+# refuses), not a 1% discrepancy. deploy/microvm/tests/systemd-units.test.sh does that
+# arithmetic against SH_MAX_COMMITTED_MB below and against that tolerance, so raising
+# either value alone is a test failure rather than a discovery made by the OOM killer.
+AssertMemory=>=23G
 
 [Service]
 Type=simple

deploy/microvm/tests/systemd-units.test.sh

@@ -154,6 +154,50 @@ sh_max_committed="$(grep -oE 'SH_MAX_COMMITTED_MB=[0-9]+' "$SERVICE" | head -n1
 check "SH_MAX_COMMITTED_MB is set" "$([ -n "$sh_max_committed" ] && echo yes || echo no)" "yes"
 check "SH_MAX_COMMITTED_MB is non-zero" "$([ -n "$sh_max_committed" ] && [ "$sh_max_committed" -gt 0 ] && echo yes || echo no)" "yes"
 
+echo "== the admission budget is checked against the host's physical memory, not just against itself"
+# SH_MAX_COMMITTED_MB less SH_MEMORY_RESERVE_MB is what the pool will admit, and nothing
+# in the worker reads MemTotal (grep -rE 'MemTotal|MemAvailable|meminfo|Sysinfo' over
+# remote-worker returns nothing). On a host with less RAM than the budget the gate
+# therefore cannot refuse before the OOM killer arrives, and spec §7.4's prediction 1
+# reads as confirmed by a gate that never refuses at all. AssertMemory is the control
+# that makes the budget a statement about THIS host: systemd compares it against
+# physical memory and fails the unit at start, the same posture as
+# AssertPathExists=/dev/kvm.
+assert_memory_raw="$(sed -n '/^\[Unit\]/,/^\[Service\]/p' "$SERVICE" | grep -oE '^AssertMemory=[<>=]*[0-9]+[KMGT]?' | head -n1 | cut -d= -f2-)"
+check "AssertMemory is set in [Unit]" \
+  "$([ -n "$assert_memory_raw" ] && echo yes || echo no)" "yes"
+# The operator must be >= : "at least this much RAM". An = or <= would assert something
+# else entirely (and a bare size means =, which would refuse on any larger host).
+check "AssertMemory uses the >= comparison (at LEAST this much physical memory)" \
+  "$(printf '%s' "$assert_memory_raw" | grep -cE '^>=[0-9]+[KMGT]?$')" "1"
+# Normalised to MiB so it can be compared with SH_MAX_COMMITTED_MB, which is in MiB.
+# systemd's size suffixes are 1024-based.
+assert_memory_mb=""
+case "$assert_memory_raw" in
+  *T) assert_memory_mb=$(( $(printf '%s' "$assert_memory_raw" | tr -dc '0-9') * 1024 * 1024 )) ;;
+  *G) assert_memory_mb=$(( $(printf '%s' "$assert_memory_raw" | tr -dc '0-9') * 1024 )) ;;
+  *M) assert_memory_mb=$(printf '%s' "$assert_memory_raw" | tr -dc '0-9') ;;
+  *K) assert_memory_mb=$(( $(printf '%s' "$assert_memory_raw" | tr -dc '0-9') / 1024 )) ;;
+  *[0-9]) assert_memory_mb=$(( $(printf '%s' "$assert_memory_raw" | tr -dc '0-9') / 1024 / 1024 )) ;;
+esac
+# Two bounds, and both are load-bearing:
+#
+#   - not far BELOW the budget: raising SH_MAX_COMMITTED_MB without raising the
+#     assertion is exactly the drift that leaves a 16 GiB host admitting 20 GiB of VMs.
+#     The 90% floor is the tolerance the assertion needs, because physical memory here is
+#     MemTotal, which is always somewhat below nominal RAM.
+#   - not ABOVE the budget: an assertion at or above nominal RAM refuses to start on a
+#     host of exactly the size the budget was written for, because of that same MemTotal
+#     shortfall. That would be a self-inflicted outage, so it is a test failure too.
+budget_floor=$(( sh_max_committed * 90 / 100 ))
+check "AssertMemory is within the tolerance band of SH_MAX_COMMITTED_MB (>= 90% of it, and not above it)" \
+  "$([ -n "$assert_memory_mb" ] && [ "$assert_memory_mb" -ge "$budget_floor" ] && [ "$assert_memory_mb" -le "$sh_max_committed" ] && echo yes || echo no)" "yes"
+# The reserve is headroom INSIDE the ceiling (budget.go: MaxCommittedBytes less
+# MemoryReserveBytes), so the ceiling itself is what the host has to be able to hold.
+sh_reserve="$(grep -oE 'SH_MEMORY_RESERVE_MB=[0-9]+' "$SERVICE" | head -n1 | cut -d= -f2)"
+check "SH_MEMORY_RESERVE_MB is set and smaller than SH_MAX_COMMITTED_MB (Config.Normalize refuses otherwise)" \
+  "$([ -n "$sh_reserve" ] && [ "$sh_reserve" -lt "$sh_max_committed" ] && echo yes || echo no)" "yes"
+
 echo "== the slice's swap posture (spec §6 mitigation #3: fast, attributable in-cgroup OOM kill)"
 check "MemorySwapMax=0 is set on the slice" \
   "$([ "$(grep -cF 'MemorySwapMax=0' "$SLICE")" -ge 1 ] && echo yes || echo no)" "yes"