Race condition in cancel function management allows entity evaluations to outlive shutdown

[MayankSharmaCSE]

Describe the issue

During shutdown, ExecutorEventHandler is supposed to cancel all in-flight entity
evaluations so the process can exit cleanly. There's a timing gap in
internal/engine/handler.go that lets some evaluations slip through uncancelled.

The shutdown goroutine (lines 60-68) runs once — it waits for the parent context to
cancel, iterates the cancels slice, calls each cancel function, and exits permanently.
But HandleEntityEvent (line 100) can still append new cancel functions to the slice
during or after that pass has completed.

This means any entity evaluation that kicks off during or after shutdown never gets
cancelled. It runs until it finishes naturally or hits the 5-minute
DefaultExecutionTimeout, delaying clean process termination.

Affected code:

• handler.go:60-68 — Shutdown goroutine iterates e.cancels once and exits
• handler.go:97-101 — New cancel functions appended per incoming event
• handler.go:122-124 — Cleanup removes cancel from slice after evaluation

To Reproduce

1. Start the server with active entity event processing
2. Trigger a shutdown (cancel the parent context passed to NewExecutorEventHandler)
3. Have new entity events arrive via the message queue while shutdown is in progress
4. Observe that newly started evaluations are not cancelled — they continue running
   until they complete on their own or hit the DefaultExecutionTimeout (5 minutes)

Expected: All in-flight evaluations, including ones starting during shutdown, should
be cancelled promptly.

What version are you using?

latest main (commit 63a6a8e)

[sachin9058]

Thanks for reporting this great catch!

It looks like the issue is due to the shutdown goroutine only iterating over the existing cancel functions once, while new evaluations can still be started and appended afterward.

One possible approach could be to prevent new evaluations from starting once shutdown begins, or ensure they are immediately cancelled if the parent context is already done.

I’d be interested in working on a fix. Please let me know if you’re already planning to take it, otherwise I’m happy to pick it up.