Zombie-detection automation false-positives on healthy completed flows

[matthewc97]

Bug summary

Description

Some prefect.flow-run.* events appear in the event log but are not delivered to the automation trigger evaluation pipeline. Most notably, prefect.flow-run.Completed events are not seen by automation triggers, causing Proactive triggers that expect: prefect.flow-run.* to fire on healthy flows after they complete.

This breaks the canonical zombie-detection automation pattern documented at https://docs.prefect.io/v3/advanced/detect-zombie-flows.

Reproduction

Step 1: Create the zombie-detection automation from the docs

from datetime import timedelta
from prefect.automations import Automation
from prefect.client.schemas.objects import StateType
from prefect.events.actions import ChangeFlowRunState
from prefect.events.schemas.automations import EventTrigger, Posture
from prefect.events.schemas.events import ResourceSpecification

automation = Automation(
    name="Crash zombie flows",
    trigger=EventTrigger(
        after={"prefect.flow-run.heartbeat"},
        expect={"prefect.flow-run.*"},
        match=ResourceSpecification(
            {"[prefect.resource.id](http://prefect.resource.id/)": ["prefect.flow-run.*"]}
        ),
        for_each={"[prefect.resource.id](http://prefect.resource.id/)"},
        posture=Posture.Proactive,
        threshold=1,
        within=timedelta(seconds=600),
    ),
    actions=[
        ChangeFlowRunState(
            state=StateType.CRASHED,
            message="Flow run marked as crashed due to missing heartbeats.",
        )
    ],
)
automation.create()

Step 2: Run a normal flow that emits at least one heartbeat then completes

import time
from prefect import flow

@flow
def heartbeat_test(duration_seconds: int = 900):
    """Long enough to emit multiple heartbeats with default
    PREFECT_RUNNER_HEARTBEAT_FREQUENCY=180s, then complete normally."""
    time.sleep(duration_seconds)

Deploy and run via a Kubernetes worker. Flow runs to completion normally, emitting heartbeats every 180 seconds and then a Completed event.

Step 3: Observe automation firing 600s after the last heartbeat

The automation fires (incorrectly) and attempts to set the flow run to Crashed. The action fails because the run is already in terminal state Completed:

{
  "event": "prefect.automation.action.failed",
  "payload": {
    "reason": "Failed to set state: Run is already in terminal state COMPLETED.",
    "action_type": "change-flow-run-state"
  }
}

Event timeline (real example from our environment)

For flow run fortunate-vicugna:

Time (UTC-equivalent)	Event
13:12:17	prefect.flow-run.heartbeat (1st)
13:15:17	prefect.flow-run.heartbeat (2nd)
13:18:17	prefect.flow-run.heartbeat (3rd)
13:21:17	prefect.flow-run.heartbeat (4th)
13:24:17	prefect.flow-run.heartbeat (5th)
13:27:17	prefect.flow-run.heartbeat (6th, last)
13:27:17	prefect.flow-run.Completed
13:37:22	prefect.automation.triggered (Zombie -> Crashed)
13:37:22	prefect.automation.action.failed (Run already in terminal state COMPLETED)

The prefect.flow-run.Completed event was emitted at the same instant as the last heartbeat (13:27:17) — well within the 600-second within window. The trigger should have considered the expect: prefect.flow-run.* clause satisfied and not fired. Instead, the automation fired exactly 600 seconds after the last heartbeat as if no follow-up event was ever received.

The Completed event is visible both in the UI Event Feed and via the API:

curl -X POST "$PREFECT_API_URL/events/filter" \
  -H "Authorization: Bearer $PREFECT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "resource": {"id": ["prefect.flow-run.<run-id>"]},
      "event": {"name": ["prefect.flow-run.Completed"]}
    },
    "limit": 5
  }'

So the event is in the event log; it just isn't reaching the automation trigger evaluation pipeline.

Expected behavior

The trigger's expect: prefect.flow-run.* clause should be satisfied by any prefect.flow-run.* event arriving within the within window after the after event. prefect.flow-run.Completed should qualify, preventing the Proactive trigger from firing on healthy flows.

Actual behavior

prefect.flow-run.Completed (and possibly other state-change events) reach the event log but not the automation trigger evaluation pipeline. The trigger fires after the within window expires, even though qualifying events were emitted.

Impact

The zombie-detection pattern documented at https://docs.prefect.io/v3/advanced/detect-zombie-flows is unreliable. Every healthy flow run that heartbeats then completes will trigger the automation. The action gracefully fails for terminal states, but there is a race window where the automation could incorrectly mark a still-completing flow as Crashed, and the failed-action events generate significant noise in the automation feed.

Environment notes

• Self-hosted Prefect server running on Kubernetes
• Kubernetes worker submitting flows as Jobs
• Default PREFECT_RUNNER_HEARTBEAT_FREQUENCY=180 (heartbeats every 3 minutes)

Version info

Version:              3.7.0
API version:          0.8.4
Python version:       3.11.15
Git commit:           85c59e0e
Built:                Wed, May 06, 2026 03:04 PM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          ephemeral
Pydantic version:     2.13.4
Server:
  Database:           postgresql
  PostgreSQL version: 15.2
Integrations:
  prefect-redis:      0.2.11

Additional context

No response

[desertaxle]

Thanks for the great investigation @matthewc97! Could you share the full granularity of the event timestamps? I want to makes sure the prefect.flow-run.Completed did in fact come after the last prefect.flow-run.heartbeat event. If it didn't, that suggests the need for a client-side change.

Long term, I'd like to move zombie flow detection into the server-side orchestration rules so that this automation isn't necessary. That's the best solution for this issue, but maybe there's a small enhancement or fix that can help in the short term.

[matthewc97]

Hi @desertaxle, here's the full timeline for the flow run:

Resource events (events from this flow run):

2026-05-13T05:12:10.456140Z  prefect.flow-run.Scheduled
2026-05-13T05:12:11.075506Z  prefect.flow-run.Pending
2026-05-13T05:12:11.163644Z  prefect.flow-run.Submitting
2026-05-13T05:12:11.349026Z  prefect.flow-run.InfrastructurePending
2026-05-13T05:12:14.377273Z  prefect.flow-run.pull-step.executed
2026-05-13T05:12:17.601241Z  prefect.flow-run.Running
2026-05-13T05:12:17.666320Z  prefect.flow-run.heartbeat
2026-05-13T05:15:17.681665Z  prefect.flow-run.heartbeat
2026-05-13T05:18:17.696198Z  prefect.flow-run.heartbeat
2026-05-13T05:21:17.711027Z  prefect.flow-run.heartbeat
2026-05-13T05:24:17.726463Z  prefect.flow-run.heartbeat
2026-05-13T05:27:17.716889Z  prefect.flow-run.Completed       ← Flow completes
2026-05-13T05:27:17.741008Z  prefect.flow-run.heartbeat       ← 24ms AFTER Completed

Related events (worker, K8s pod, automation):

2026-05-13T05:12:11.135650Z  prefect.worker.submitted-flow-run
2026-05-13T05:12:11.237297Z  prefect.worker.executed-flow-run
2026-05-13T05:12:11.367449Z  prefect.kubernetes.pod.pending
2026-05-13T05:12:12.623602Z  prefect.kubernetes.pod.running
2026-05-13T05:27:26.836589Z  prefect.kubernetes.pod.succeeded
2026-05-13T05:37:22.079977Z  prefect.automation.action.triggered
2026-05-13T05:37:22.197935Z  prefect.automation.action.failed

Looks like a heartbeat was emitted ~24ms after the flow entered the terminal Completed state. Could that post-completion heartbeat be re-arming the Proactive trigger?