replay-node ignores the Stop event, so dora replay --loop can't be stopped gracefully #3234

What's happening

dora-replay-node never reads its event stream, so it never sees the Stop event the daemon sends on shutdown. The only thing that ends its replay loop is running out of records, and with --loop that never happens. The daemon ends up force-killing the node 15 seconds later.

Steps to reproduce

dora record dataflow.yml -o capture.drec   # Ctrl-C after a few seconds
dora replay capture.drec --loop
# Ctrl-C

I'd expect the replay node to pick up Stop, finish the entry it's on, and exit. Instead it keeps replaying for about 10 more seconds while downstream nodes keep receiving messages, and then the daemon prints:

<node> was killed due to not stopping within the 15s grace period

With --speed 0, those 10 seconds are the whole recording replayed over and over at full speed.

Why it happens

In binaries/replay-node/src/main.rs:

let (mut node, _events) = DoraNode::init_from_env()?;

_events is never used again. There's no Event::Stop match and no recv or try_recv anywhere in the file. init_from_env does subscribe, so the daemon has a live channel and delivers Stop into it (running_dataflow.rs:630); the node just never reads it. The only way out of the loop is:

if !do_loop {
    break;
}

On the daemon side, stop_all sends Stop, waits DEFAULT_STOP_GRACE (10s), sends SIGTERM, waits another 5s, then SIGKILL. Outputs keep being routed that whole time, since there's no gate on the send path during teardown, which is deliberate so nodes can flush.

Worth noting that the sibling binary gets this right: binaries/record-node/src/main.rs has Event::Stop(_) => break in its loop. Nothing in the replay node mentions shutdown at all, so this reads more like an oversight than a decision.

Why it matters

The main one is that dora replay --loop has no graceful way to stop. Beyond that, replay exists to feed recorded data into live nodes, so those extra 10 seconds mean replayed commands keep reaching a live control node after the operator already asked for a stop. And every dora replay teardown prints a force-kill warning, which makes that warning easy to start ignoring.

The kill is at least classified as planned (grace_duration_kills), so it doesn't surface as a failed dataflow.

Possible fix

Swapping the pacing thread::sleep for events.recv_timeout(...) would cover both the pacing and the Stop check in one call, with no added latency. --speed 0 never sleeps, so that path needs a cheap try_recv poll per iteration too, plus a check between loop passes. One thing to watch: a stop-interrupted pass should return Ok(()) rather than hitting the replay_emitted_nothing_usable bail, otherwise a normal stop turns into an error.