Fall back to the recording when the snapshot has no face

Frigate picks its snapshot by highest person score, which is not the same as
"a face is visible" — often it is the moment someone turns away. Measured over
seven days of real events, only 19% of snapshots held a usable face: 55% had
none at all, 14% one below min_face_px, 12% one too uncertain.

Re-checking failed events against the clip found a good face in 9 of 12
(det 0.68-0.87). Those events were previously lost entirely — not recognised,
not even queued for review.

Runs only when the snapshot found nothing, once per event, on its own thread
with a short queue, so live recognition and presence updates are never delayed
and an event burst skips scans instead of building a backlog.

The frame is picked by detection quality, deliberately not by gallery
similarity: choosing whichever of twelve frames looks most like someone known
would flatter the numbers and invite misassignments. Identity is decided
afterwards, exactly as on the snapshot path.

Gains depend on the camera angle, so the docs say so instead of quoting one
number: at a front door at head height the recording rescued 4 of 4, on a
high-mounted indoor camera and a zoomed garden view none. Raising det_size does
not help either (1280 gave results identical to 640 at twice the cost) — the
limit is the viewing angle, not the resolution.

Author: skycryer

CHANGELOG.md

@@ -3,6 +3,40 @@
 All notable changes to FaceID. The Home Assistant app shows this file in the
 update dialog; standalone users can watch GitHub releases.
 
+## 0.9.0 — 2026-08-08
+
+- **Recognition now falls back to the recording when the snapshot has no face.** This is
+  the single biggest limit on recognition — bigger than the gallery, the threshold or
+  `match_top_k`, all of which took far more effort to tune.
+
+  Frigate picks its snapshot by highest *person* score, which is not the same as "a face
+  is visible" — frequently it is the moment someone turns away. Over seven days of real
+  events only **19%** of snapshots held a usable face: 55% had no detectable face at all,
+  14% one below `min_face_px`, 12% one too uncertain to trust.
+
+  Re-checking twelve of the failed events against the clip found a good face in **nine**
+  (det 0.68–0.87).
+
+  How much this gains depends on where a camera points, so measure yours: at the front
+  door, at head height, the recording rescued 4 of 4 events; on a high-mounted indoor
+  camera and a zoomed garden view it rescued none — there the clip holds no face either.
+
+  Three plausible explanations were measured and ruled out: night (IR failed 11 times,
+  colour 12), distance (not one crop was narrower than 120px), and detection resolution
+  (`det_size` 1280 gave results identical to 640 at twice the cost). What remains is the
+  viewing angle.
+
+- Runs only when the snapshot found nothing, once per event, on its own thread with a
+  short queue — live recognition and presence updates are never delayed, and a burst of
+  events skips scans rather than building a backlog. New option **Search the recording**
+  (`clip_fallback`, default on); turn it off on tight hardware.
+- The frame is selected by detection quality, deliberately **not** by gallery similarity:
+  picking whichever of twelve frames looks most like someone known would flatter the
+  numbers and invite misassignments. Identity is decided afterwards, as on the snapshot
+  path.
+- Faces found this way skip the `hires_enroll` pass — they already come from the
+  recording, so re-scanning the same clip would only cost time.
+
 ## 0.8.2 — 2026-08-08
 
 - **Fixed: the app refused to start with "No MQTT broker configured" even with Mosquitto

README.md

@@ -204,9 +204,61 @@ is compared against the snapshot face and the best identity match wins — so wi
 people in frame, the wrong face can't be enrolled. When no frame yields a usable face
 (roughly one event in three), the original snapshot is kept.
 
-Live recognition still uses the fast snapshot path; only enrollment takes the slower,
-sharper route (a few seconds and one clip download per event). Needs recordings enabled
-for the camera.
+Live recognition tries the fast snapshot path first; only when that yields nothing does it
+fall back to the recording (see below). Needs recordings enabled for the camera.
+
+## When the snapshot has no face
+
+**Frigate picks its snapshot by highest *person* score — which is not the same as "a face
+is visible".** Often the clearest view *of a person* is the moment they turn away. This is
+not a rare edge case. Measured over seven days of real events on a four-camera setup:
+
+| what the snapshot gave us | events | |
+|---|---|---|
+| no face detected at all | 23 | 55% |
+| face too small (median 34px) | 6 | 14% |
+| detection too uncertain | 5 | 12% |
+| **usable face** | **8** | **19%** |
+
+Two obvious suspects turned out to be innocent. **Night is not the problem** — IR and
+colour failed at almost the same rate (11 vs 12). **Distance is not the problem either** —
+not one crop was narrower than 120px; the people were plenty large in frame.
+
+The recording tells a different story. Of twelve failed events re-checked frame by frame,
+**nine had a perfectly good face** (det 0.68–0.87) that the snapshot simply missed.
+
+So **Settings → Search the recording** (default on) does exactly that: when an event ends
+and the snapshot never produced a face, FaceID samples frames across the clip and takes
+the best one.
+
+**How much it helps depends entirely on where the camera points**, so measure yours rather
+than expecting a number. On the setup above, checked per camera:
+
+| camera | recording rescued |
+|---|---|
+| front door, at head height | 4/4 (and 9/12 in a wider sample) |
+| child's room, mounted high | 0/4 |
+| garden, zoomed | 0/2 |
+
+Where people walk towards the lens, almost every failed snapshot is recoverable. Where the
+camera looks down at people, or catches them side-on at distance, the clip holds no face
+either — no amount of scanning invents one. Raising `det_size` does not change this: at
+1280 instead of 640 the results were identical, at twice the cost (11.1s vs 5.4s per
+event). The limit is the viewing angle, not the resolution.
+
+Three deliberate details:
+
+* **Only when the snapshot found nothing.** Events that already worked cost nothing extra.
+* **Selection is by detection quality, never by gallery similarity.** Picking whichever of
+  twelve frames happens to look most like someone you know would inflate the numbers and
+  invite misassignments. The frame is chosen on image quality; identity is decided
+  afterwards, exactly as on the snapshot path.
+* **Its own thread, short queue.** A clip scan takes seconds. It must not delay live
+  recognition or presence updates, and during a burst of events it skips rather than
+  building a backlog that runs minutes behind reality.
+
+Turn it off if your hardware is tight — the cost is a few seconds of CPU per *otherwise
+failed* event, plus one clip download.
 
 ### Recovering missed events
 
@@ -308,6 +360,18 @@ backup from `data/backups`) and adds a practical probe against recent Frigate ev
 including how much headroom each recognition has above the threshold. Events whose face
 is already in the gallery are excluded — they score ~1.0 and measure nothing.
 
+**Running FaceID as a Home Assistant app?** You have no shell, so the part that decides
+your threshold was moved into the UI: Settings → *Does it actually work?* runs the
+leave-one-out test and the practical probe as a background job and reports the one number
+that matters, how high a stranger got. That covers threshold and `match_top_k`.
+
+Two things still need a shell, and neither is required to run FaceID well:
+
+* the **coverage report** (`coverage.py`) — which angles, cameras and IR shots each
+  person is missing
+* **comparing two galleries** (`--baseline`) — useful after a round of enrolling, but the
+  UI analysis already tells you where you stand today
+
 
 ## How training stays healthy
 

app/hires.py

@@ -4,8 +4,12 @@
 voller Kameraauflösung auf (z. B. 2560x1920). Für die Galerie lohnt sich deshalb der
 Umweg über die Aufnahme: dort sind Gesichter typischerweise doppelt so groß.
 
-Bewusst nur fürs Enrollment (Review-Queue, Verlaufs-Scan) — die Live-Erkennung bleibt
-schnell auf dem Snapshot.
+Die Live-Erkennung arbeitet zuerst auf dem Snapshot — der ist sofort da und kostet
+nichts. Erst wenn der gar kein Gesicht hergibt, lohnt der Blick in die Aufnahme
+(``find_face_in_clip``): gemessen an sieben Tagen echter Ereignisse hatten nur 19 %
+der Snapshots ein verwertbares Gesicht, der Clip lieferte in 9 von 12 Fällen doch
+noch eines. Frigate wählt seinen Snapshot nach dem höchsten Personen-Score aus, und
+das ist ein anderes Kriterium als "Gesicht sichtbar".
 """
 import logging
 import os
@@ -39,33 +43,29 @@ def _pick(engine, frame, ref_embedding, min_px, identity_min, min_det=0.55):
     return best
 
 
-def _scan_clip(engine, frigate, event_id, ref_embedding, max_frames, min_px, identity_min,
-               min_det):
+def _clip_frames(frigate, event_id, max_frames):
+    """Gleichmäßig über den Clip verteilte Frames — der Generator raeumt selbst auf.
+
+    Wird bewusst immer vollstaendig durchlaufen (kein ``break`` beim Aufrufer), damit
+    das ``finally`` die heruntergeladene Datei sicher wieder loescht.
+    """
     fd, path = tempfile.mkstemp(suffix=".mp4", prefix="faceid-clip-")
     os.close(fd)
     try:
         if not frigate.download_clip(event_id, path):
-            return None
+            return
         cap = cv2.VideoCapture(path)
         try:
             total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
             if total <= 0:
-                return None
-            # Gleichmäßig über den Clip abtasten — Frigate wählt für den Snapshot den
-            # besten Moment, den kennen wir nicht.
-            idxs = np.linspace(0, total - 1, min(max_frames, total)).astype(int)
-            best = None
-            for i in idxs:
+                return
+            # Frigate waehlt fuer den Snapshot den besten Moment nach seinem Kriterium,
+            # den kennen wir nicht — also gleichmaessig abtasten.
+            for i in np.linspace(0, total - 1, min(max_frames, total)).astype(int):
                 cap.set(cv2.CAP_PROP_POS_FRAMES, int(i))
                 ok, frame = cap.read()
-                if not ok or frame is None:
-                    continue
-                cand = _pick(engine, frame, ref_embedding, min_px, identity_min, min_det)
-                if cand is None:
-                    continue
-                if best is None or cand[1] > best[0][1]:
-                    best = (cand, frame)
-            return (best[0][2], best[1]) if best else None
+                if ok and frame is not None:
+                    yield frame
         finally:
             cap.release()
     finally:
@@ -75,6 +75,42 @@ def _scan_clip(engine, frigate, event_id, ref_embedding, max_frames, min_px, ide
             pass
 
 
+def _scan_clip(engine, frigate, event_id, ref_embedding, max_frames, min_px, identity_min,
+               min_det):
+    best = None
+    for frame in _clip_frames(frigate, event_id, max_frames):
+        cand = _pick(engine, frame, ref_embedding, min_px, identity_min, min_det)
+        if cand is None:
+            continue
+        if best is None or cand[1] > best[0][1]:
+            best = (cand, frame)
+    return (best[0][2], best[1]) if best else None
+
+
+def find_face_in_clip(engine, frigate, event_id: str, max_frames: int = 12,
+                      min_px: int = 48, min_det: float = 0.65):
+    """Bestes Gesicht im Clip — fuer Ereignisse, deren Snapshot gar keines hergab.
+
+    Ausgewaehlt wird nach det_score, ausdruecklich NICHT nach Galerie-Aehnlichkeit:
+    sonst sucht man sich aus zwoelf Frames denjenigen heraus, der zufaellig am ehesten
+    wie jemand Bekanntes aussieht, und rechnet sich die Erkennung schoen. Kriterium
+    bleibt die Bildqualitaet, die Zuordnung kommt danach — genau wie beim Snapshot.
+
+    -> (face, frame) oder None
+    """
+    best = None
+    for frame in _clip_frames(frigate, event_id, max_frames):
+        for f in engine.faces(frame):
+            w = float(f.bbox[2] - f.bbox[0])
+            h = float(f.bbox[3] - f.bbox[1])
+            if w < min_px or h < min_px or float(f.det_score) < min_det:
+                continue
+            key = (float(f.det_score), w)
+            if best is None or key > best[0]:
+                best = (key, f, frame)
+    return (best[1], best[2]) if best else None
+
+
 def _scan_recordings(engine, frigate, camera, start_time, end_time, ref_embedding,
                      attempts, min_px, identity_min, min_det):
     """Fallback ohne Clip: einzelne Aufnahme-Frames per HTTP abklopfen."""

app/mqtt_listener.py

@@ -4,6 +4,9 @@
   - Match  >= match_threshold   -> Person publizieren + Frigate sub_label
   - Match  <  unknown_threshold -> als Unbekannter in die Review-Queue
   - dazwischen                  -> unsicher; nur Review-Queue, keine Meldung
+
+Gibt der Snapshot ueberhaupt kein Gesicht her, wird am Ereignisende die Aufnahme
+abgetastet (clip_fallback) — das ist der haeufigste Fall, nicht die Ausnahme.
 """
 import json
 import logging
@@ -16,7 +19,7 @@
 import requests
 
 from .engine import FaceEngine, crop_face
-from .hires import upgrade_face
+from .hires import find_face_in_clip, upgrade_face
 
 log = logging.getLogger("faceid.mqtt")
 
@@ -43,6 +46,18 @@ def __init__(self, cfg: dict, engine, gallery, frigate):
         self.ignore_thr = float(f.get("ignore_threshold", f.get("match_threshold", 0.5)))
         self.ignore_learning = bool(f.get("ignore_learning", True))
         self.hires_enroll = bool(f.get("hires_enroll", True))
+        # Frigate waehlt seinen Snapshot nach dem hoechsten Personen-Score, nicht danach,
+        # ob ein Gesicht zu sehen ist — das ist oft der Moment, in dem jemand weggeht.
+        # Ueber sieben Tage echter Ereignisse hatten nur 19 % der Snapshots ein
+        # verwertbares Gesicht; im Clip fanden sich 9 von 12 doch noch. Deshalb: erst
+        # Snapshot (sofort da, kostet nichts), und nur wenn der leer bleibt, die Aufnahme.
+        self.clip_fallback = bool(f.get("clip_fallback", True))
+        self.clip_frames = int(f.get("clip_fallback_frames", 12))
+        # strenger als beim Snapshot (0.55): aus zwoelf Frames darf man waehlerisch sein
+        self.clip_min_det = float(f.get("clip_fallback_min_det", 0.65))
+        # klein gehalten: bei einem Ereignisschwall lieber welche auslassen als eine
+        # Warteschlange aufbauen, die Minuten hinter der Gegenwart herlaeuft
+        self.clip_queue: "queue.Queue[str]" = queue.Queue(maxsize=20)
         # Ereignisse, die Frigate nicht per MQTT meldet (z. B. per API angelegte
         # Kamera-Meldungen als Zuverlaessigkeits-Bruecke), per Abfrage nachziehen.
         self.poll_interval = float(f.get("poll_interval", 0))
@@ -73,6 +88,11 @@ def start(self):
         self._check_frigate()
         threading.Thread(target=self._worker, daemon=True, name="faceid-worker").start()
         threading.Thread(target=self._finalizer, daemon=True, name="faceid-finalizer").start()
+        # Eigener Thread: ein Clip-Scan dauert Sekunden, im Finalizer wuerde er die
+        # Anwesenheits-Aktualisierung blockieren, im Worker die naechsten Snapshots.
+        # Laeuft unabhaengig von clip_fallback, damit die Option in den Einstellungen
+        # sofort greift statt erst nach einem Neustart — er wartet dann nur an der Queue.
+        threading.Thread(target=self._clip_worker, daemon=True, name="faceid-clip").start()
         if self.poll_interval > 0:
             threading.Thread(target=self._poller, daemon=True, name="faceid-poller").start()
 
@@ -177,6 +197,16 @@ def _process(self, eid: str):
                 log.info("event %s (%s): attempt %d, no face detected in snapshot %dx%d",
                          eid, st["camera"], st["attempts"], w, h)
             return
+        self._handle_face(eid, st, img, face)
+
+    def _handle_face(self, eid: str, st: dict, img, face, source: str = "snapshot"):
+        """Gefundenes Gesicht zuordnen, melden, ablegen.
+
+        Gemeinsam fuer Snapshot und Aufnahme — beide Wege muessen dieselben Schwellen,
+        dieselbe Ignore-Logik und dieselbe Meldung verwenden, sonst haengt das Ergebnis
+        davon ab, welcher Weg zufaellig gegriffen hat.
+        """
+        via = "" if source == "snapshot" else f" (from the {source})"
         emb = face.normed_embedding
         slug, name, score = self.gallery.match(emb)
         ig = self.gallery.match_ignored(emb)
@@ -194,7 +224,8 @@ def _process(self, eid: str):
             log.info("event %s (%s): ignored face (sim %.3f)", eid, st["camera"], ig)
             return
         crop = crop_face(img, face.bbox)
-        log.info("event %s (%s): attempt %d, match %s (%.3f)", eid, st["camera"], st["attempts"], name, score)
+        log.info("event %s (%s): attempt %d, match %s (%.3f)%s", eid, st["camera"],
+                 st["attempts"], name, score, via)
 
         if slug and score >= self.match_thr:
             if score > st["best_score"]:
@@ -209,7 +240,10 @@ def _process(self, eid: str):
             prev = st.get("best_unknown")
             if prev is None or face.det_score > prev["det_score"]:
                 st["best_unknown"] = {"crop": crop, "emb": emb, "det_score": float(face.det_score),
-                                      "guess": name, "guess_score": float(score), "full": img}
+                                      "guess": name, "guess_score": float(score), "full": img,
+                                      # aus der Aufnahme ist bereits das schaerfste Bild —
+                                      # ein zweiter Durchgang durch hires waere derselbe Clip
+                                      "from_clip": source != "snapshot"}
 
     def _poller(self):
         """Frigate-Ereignisse abfragen, die per MQTT nie ankommen.
@@ -259,6 +293,39 @@ def _poller(self):
                 except queue.Full:
                     log.warning("queue full — dropped polled event %s", eid)
 
+    def _clip_worker(self):
+        """Ereignisse nachbearbeiten, deren Snapshot kein Gesicht hergab."""
+        while True:
+            eid = self.clip_queue.get()
+            try:
+                self._process_clip(eid)
+            except Exception:
+                log.exception("error while scanning the recording of event %s", eid)
+            finally:
+                st = self.events.get(eid)
+                if st is not None:
+                    st["clip_pending"] = False
+
+    def _process_clip(self, eid: str):
+        st = self.events.get(eid)
+        if st is None or st["done"]:
+            return
+        t0 = time.time()
+        hit = find_face_in_clip(self.engine, self.frigate, eid,
+                                max_frames=self.clip_frames,
+                                min_px=self.min_face_px,
+                                min_det=self.clip_min_det)
+        took = time.time() - t0
+        if hit is None:
+            log.info("event %s (%s): no face in the recording either (%d frames, %.1fs)",
+                     eid, st["camera"], self.clip_frames, took)
+            return
+        face, frame = hit
+        log.info("event %s (%s): the recording has a face the snapshot missed "
+                 "(%dpx, det %.2f, %.1fs)", eid, st["camera"],
+                 int(face.bbox[2] - face.bbox[0]), float(face.det_score), took)
+        self._handle_face(eid, st, frame, face, source="recording")
+
     def _finalizer(self):
         """Beendete Events abschließen: Unknown ablegen, 'unbekannt' melden, aufräumen."""
         while True:
@@ -273,10 +340,27 @@ def _finalizer(self):
                     continue
                 if now - st["last_try"] < self.retry_secs + 1 and not expired:
                     continue  # letzter Versuch evtl. noch in der Queue
+                # Snapshot hat nichts gefunden -> in der Aufnahme nachsehen, bevor das
+                # Ereignis verworfen wird. Erst hier, weil der Clip erst am Ende steht.
+                if (self.clip_fallback and st["best_person"] is None
+                        and st["best_unknown"] is None and not st.get("clip_tried")):
+                    st["clip_tried"] = True
+                    try:
+                        self.clip_queue.put_nowait(eid)
+                        st["clip_pending"] = True
+                        st["clip_since"] = now
+                        continue
+                    except queue.Full:
+                        log.info("event %s: clip queue full, skipping the recording scan", eid)
+                if st.get("clip_pending"):
+                    if now - st.get("clip_since", now) < 300:
+                        continue      # laeuft noch
+                    log.warning("event %s: recording scan did not finish, closing anyway", eid)
+                    st["clip_pending"] = False
                 if st["best_person"] is None and st["best_unknown"] is not None:
                     u = st["best_unknown"]
                     crop, emb, full = u["crop"], u["emb"], u.get("full")
-                    if self.hires_enroll:
+                    if self.hires_enroll and not u.get("from_clip"):
                         # schärferes Gesicht aus der Aufnahme holen (bessere Referenz)
                         try:
                             hi = upgrade_face(self.engine, self.frigate, st["camera"],