#214 was closed via edd3d94 ("earlyoom.service: set Restart=always").
I hit the same fatal on 1.8.2 and can supply both the dumped buffer
contents you asked for back then, and evidence that Restart=always does
not survive the actual failure mode.
Root cause of the fatal, finally caught (thanks to the dump-buffer
code): lxcfs transiently served a u64-wrapped negative SwapFree during
host memory churn -- see lxc/lxcfs#695, same value class:
MemTotal: 14834464 kB
MemFree: 10668780 kB
MemAvailable: 12749012 kB
SwapCached: 0 kB
SwapTotal: 0 kB
SwapFree: 18446744073709551360 kB
fatal: could not find entry 'SwapFree:' in /proc/meminfo: Numerical result out of range
18446744073709551360 = 2^64 - 256. strtoll correctly returns ERANGE and
earlyoom exits 104 -- fine so far.
The problem: the glitch lasts a few seconds, and systemd's defaults
(RestartSec=100ms, StartLimitBurst=5 per 10s) burn out inside it:
09:57:46.332 earlyoom.service: Main process exited, code=exited, status=104/n/a
...(6 more identical crashes, ~250ms apart)...
09:57:47.821 earlyoom.service: Scheduled restart job, restart counter is at 8.
09:57:47.821 earlyoom.service: Start request repeated too quickly.
09:57:47.821 earlyoom.service: Failed with result 'exit-code'.
Total elapsed: 1.5 seconds, then the unit is failed forever -- the OOM
guard silently down on precisely the kind of memory-stressed machine
that runs earlyoom. It stayed down through a memory-pressure episode
until a manual health check noticed, an hour later.
Suggestion for the shipped unit (this is what I deployed):
[Unit]
StartLimitIntervalSec=0
[Service]
RestartSec=10
A watchdog crash-looping at 6/min is strictly better than one that gave
up. Alternatively (or additionally), treat ERANGE on a swap field as
"swap unknown, assume 0" instead of fatal -- garbage from lxcfs is
evidently a thing that happens.
Minor: the fatal message says "could not find entry 'SwapFree:'" when
the entry was present but strtoll failed; the preceding "get_entry:
strtol() failed" line tells the real story. Worth splitting the
not-found and parse-failure messages.
Environment: ChromeOS Crostini (Debian trixie LXC container under LXD in
the termina VM), earlyoom 1.8.2-1 (Debian package), systemd 257.
#214 was closed via edd3d94 ("earlyoom.service: set Restart=always").
I hit the same fatal on 1.8.2 and can supply both the dumped buffer
contents you asked for back then, and evidence that Restart=always does
not survive the actual failure mode.
Root cause of the fatal, finally caught (thanks to the dump-buffer
code): lxcfs transiently served a u64-wrapped negative SwapFree during
host memory churn -- see lxc/lxcfs#695, same value class:
18446744073709551360 = 2^64 - 256. strtoll correctly returns ERANGE and
earlyoom exits 104 -- fine so far.
The problem: the glitch lasts a few seconds, and systemd's defaults
(RestartSec=100ms, StartLimitBurst=5 per 10s) burn out inside it:
Total elapsed: 1.5 seconds, then the unit is
failedforever -- the OOMguard silently down on precisely the kind of memory-stressed machine
that runs earlyoom. It stayed down through a memory-pressure episode
until a manual health check noticed, an hour later.
Suggestion for the shipped unit (this is what I deployed):
A watchdog crash-looping at 6/min is strictly better than one that gave
up. Alternatively (or additionally), treat ERANGE on a swap field as
"swap unknown, assume 0" instead of fatal -- garbage from lxcfs is
evidently a thing that happens.
Minor: the fatal message says "could not find entry 'SwapFree:'" when
the entry was present but strtoll failed; the preceding "get_entry:
strtol() failed" line tells the real story. Worth splitting the
not-found and parse-failure messages.
Environment: ChromeOS Crostini (Debian trixie LXC container under LXD in
the termina VM), earlyoom 1.8.2-1 (Debian package), systemd 257.