`--avoid` + `--sort-by-rss`: avoided process can be selected as victim (negative VmRSS routes comparison into the zombie fallback)

[kayess00]

Summary

When --avoid REGEX is combined with --sort-by-rss, an avoided process can be selected as the victim — including over processes with far larger RSS. In the worst case the avoid list selects exactly the process it was supposed to protect.

Verified on v1.9.0 and current master (5c3e3ea): the relevant code is unchanged.

Minimal repro

# a large hog (should be the victim in both runs)
$ python3 -c 'b=bytearray(2**30)
for i in range(0,len(b),4096): b[i]=1
import time; time.sleep(120)' &

# a tiny process with a distinctive comm and elevated oom_score_adj
# (any process the operator would protect: container runtimes, supervisors, etc.
#  commonly run at oom_score_adj +200 via systemd OOMScoreAdjust=)
$ cp /bin/sleep /tmp/avoidme; /tmp/avoidme 120 &
$ echo 300 > /proc/$!/oom_score_adj

$ sudo earlyoom --dryrun -m 99 -s 100 --sort-by-rss 2>&1 | grep -m1 'sending SIGTERM'
sending SIGTERM to process 163527 uid 1000 "python3": oom_score 673, oom_score_adj 0, VmRSS 1033 MiB, ...

$ sudo earlyoom --dryrun -m 99 -s 100 --sort-by-rss --avoid '^avoidme$' 2>&1 | grep -m1 'sending SIGTERM'
sending SIGTERM to process 163524 uid 1000 "avoidme": oom_score 866, oom_score_adj 300, VmRSS -3070 MiB, ...

Run 1 correctly selects the 1 GiB hog. Run 2 — with the tiny process avoided — selects the avoided 6 KiB process itself (note the negative VmRSS -3070 MiB printed in the kill line).

A milder variant needs no elevated adj on the avoided process itself: once any avoided process becomes the interim victim during the walk, all subsequent comparisons degenerate to oom_score, so the final victim is the highest-scoring process regardless of --sort-by-rss (observed live: a 203 MiB process with adj +200 beating a 649 MiB one).

Root cause

In is_larger() (kill.c#L466-L470 on master):

if (args->avoid_regex && regexec(...) == 0) {
    if (args->sort_by_rss) {
        cur->VmRSSkiB += VMRSS_AVOID;   // -3145728 KiB
    ...

For any avoided process smaller than 3 GiB this makes VmRSSkiB negative. The RSS comparison (kill.c#L482) guards the normal path with:

if (cur->VmRSSkiB > 0 && victim->VmRSSkiB > 0) {

so a negative-RSS process fails the guard and falls into the else branch — the zombie-main-thread fallback, which compares by oom_score. Consequences:

1. An avoided process keeps its full oom_score in this mode (the OOM_SCORE_AVOID bias is only applied in the non-RSS branch), so if its score exceeds the interim victim's, it becomes the victim — with negative RSS.
2. Once the victim has negative RSS, every later candidate also fails the > 0 && > 0 guard, so the rest of the walk is score-based: --sort-by-rss is silently ignored.
3. The zombie warning doesn't fire (it triggers on VmRSSkiB == 0, not < 0), so there is no diagnostic.

VMRSS_PREFER has the mirror-image hazard in principle, but the avoid direction is the dangerous one.

Suggested fix (either seems sufficient)

• Change the fast-path guard to treat negative as "smaller", e.g. if (cur->VmRSSkiB != 0 && victim->VmRSSkiB != 0) so the zombie branch only handles true rss==0, and a negative (avoided) value simply loses every RSS comparison; or
• clamp after adjustment: if (cur->VmRSSkiB < 1) cur->VmRSSkiB = 1; (with a comment that avoided procs must still lose to any real RSS).

Happy to test a patch. Found while validating an earlyoom deployment on a 62 GiB WSL2 box (where --sort-by-rss is otherwise exactly the right selection mode — thanks for adding it).