Expected Behavior
When a silence is applied for an entity and/or a check, all events for that entity/check should be treated as silenced—including events generated when the check does not run within its TTL (e.g. agent down, check not executed). Handlers using the not_silenced filter should not receive these TTL-expired events when the entity or check is silenced.
Current Behavior
- Normal check events (agent sends a result): Silences are applied correctly via
getSilenced() in eventd.handleMessage(), and the not_silenced filter behaves as expected.
- Check TTL–expired events (check did not run within the expected cadence): The backend creates a failure event in
eventd.handleFailure() and publishes it to the event bus without evaluating silences. The event is never passed through getSilenced(), so event.Check.Silenced and event.Check.IsSilenced remain unset. As a result, TTL-expired events always trigger handlers (e.g. PagerDuty, email) even when the user has silenced that entity or check for maintenance.
So: silences work for “check ran and reported bad”; they do not work for “check never reported in time.”
Possible Solution
In backend/eventd/eventd.go, in handleFailure(), apply silences to the failed-check event before updating the store and publishing, using the same logic as in handleMessage():
- After building the failed-check event (e.g. after
createFailedCheckEvent() returns), call getSilenced(ctx, failedCheckEvent, e.silencedCache).
- If
len(failedCheckEvent.Check.Silenced) > 0, set failedCheckEvent.Check.IsSilenced = true.
- Optionally: if the event is silenced, skip
UpdateEvent and Publish (or still update the store for UI state but do not publish to the bus), so that handlers do not receive silenced TTL events. At minimum, populating Silenced and IsSilenced will allow the pipeline’s not_silenced filter to drop the event.
Relevant code (v6.13.1):
- Silence application for normal events:
eventd.go ~558–562 (handleMessage).
- TTL expiry path:
eventd.go dead() → handleFailure() → createFailedCheckEvent(); no call to getSilenced() before UpdateEvent / Publish (~764–779).
Steps to Reproduce (for bugs)
- Create a check with a TTL (e.g.
ttl: 60) that runs on an entity.
- Apply a silence for that entity (subscription match) or for that check (e.g.
entity:myentity:mycheck or *:mycheck).
- Stop the agent or otherwise prevent the check from executing within the TTL (e.g. 60 seconds).
- Observe: a “check TTL expired” / absent-check event is still generated and delivered to handlers (e.g. passes the
not_silenced filter and triggers alerts).
- Compare: when the agent is running and the check executes and returns non-OK, the same silence correctly suppresses the event.
Context
We use silences for maintenance windows and known issues. We expect that silencing an entity or check suppresses all alerting for it, including “check didn’t run in time” (TTL) events. Currently we receive TTL-based alerts (e.g. “Last check execution was X seconds ago”) even when the entity/check is silenced, which defeats the purpose of the silence during maintenance or when we have already acknowledged the issue.
Your Environment
- Sensu version used: 6.13.1 (backend and agents; confirmed against git tag
v6.13.1).
- Installation method: apt packages
- Operating System and version: Ubuntu, all LTS from 16.04 through 24.04
Expected Behavior
When a silence is applied for an entity and/or a check, all events for that entity/check should be treated as silenced—including events generated when the check does not run within its TTL (e.g. agent down, check not executed). Handlers using the
not_silencedfilter should not receive these TTL-expired events when the entity or check is silenced.Current Behavior
getSilenced()ineventd.handleMessage(), and thenot_silencedfilter behaves as expected.eventd.handleFailure()and publishes it to the event bus without evaluating silences. The event is never passed throughgetSilenced(), soevent.Check.Silencedandevent.Check.IsSilencedremain unset. As a result, TTL-expired events always trigger handlers (e.g. PagerDuty, email) even when the user has silenced that entity or check for maintenance.So: silences work for “check ran and reported bad”; they do not work for “check never reported in time.”
Possible Solution
In
backend/eventd/eventd.go, inhandleFailure(), apply silences to the failed-check event before updating the store and publishing, using the same logic as inhandleMessage():createFailedCheckEvent()returns), callgetSilenced(ctx, failedCheckEvent, e.silencedCache).len(failedCheckEvent.Check.Silenced) > 0, setfailedCheckEvent.Check.IsSilenced = true.UpdateEventandPublish(or still update the store for UI state but do not publish to the bus), so that handlers do not receive silenced TTL events. At minimum, populatingSilencedandIsSilencedwill allow the pipeline’snot_silencedfilter to drop the event.Relevant code (v6.13.1):
eventd.go~558–562 (handleMessage).eventd.godead()→handleFailure()→createFailedCheckEvent(); no call togetSilenced()beforeUpdateEvent/Publish(~764–779).Steps to Reproduce (for bugs)
ttl: 60) that runs on an entity.entity:myentity:mycheckor*:mycheck).not_silencedfilter and triggers alerts).Context
We use silences for maintenance windows and known issues. We expect that silencing an entity or check suppresses all alerting for it, including “check didn’t run in time” (TTL) events. Currently we receive TTL-based alerts (e.g. “Last check execution was X seconds ago”) even when the entity/check is silenced, which defeats the purpose of the silence during maintenance or when we have already acknowledged the issue.
Your Environment
v6.13.1).