-
-
Notifications
You must be signed in to change notification settings - Fork 209
Alerts subsystem 3/4: delta surface, REST API and server wiring #3011
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mairas
wants to merge
8
commits into
SignalK:master
Choose a base branch
from
hatlabs:feat/alerts-server-integration
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
f509df1
feat(alerts): add the alert lifecycle engine
mairas 77f3487
feat(alerts): persist alerts and their audit trail
mairas 2ec3445
feat(alerts): mirror and ingest alert deltas
mairas 3a17ba0
feat(alerts): serve the alerts REST API
mairas 662c8d8
feat(alerts): treat alerts as events
mairas dc65f2b
feat(alerts): wire the subsystem into the server
mairas 1d8c7cf
docs(alerts): document the alerts API
mairas 6debe5b
fix(staleness): skip auto sampler allocation for event paths
mairas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,8 @@ dist/ | |
| .DS_Store | ||
| .vscode/ | ||
| *.db | ||
| *.db-wal | ||
| *.db-shm | ||
| logs/* | ||
| bower_components | ||
| settings/ssl-key.pem | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| --- | ||
| title: Alerts API | ||
| --- | ||
|
|
||
| # Alerts API | ||
|
|
||
| The Alerts API manages the lifecycle of alerts: conditions that need an operator to notice, decide and act. | ||
|
|
||
| The exact request and response schemas are served by the running server at `/doc/openapi` and are generated from the same TypeBox definitions the API validates with. This page describes the model and the behaviour behind them. | ||
|
|
||
| ## Overview | ||
|
|
||
| An alert names a condition and carries the state of the operator's response to it. The path identifies the alert and describes what is wrong — `propulsion.port.oilPressureLow`, not the sensor path that measured it — and there is one active alert per path and context. Data paths the condition concerns go in the optional `references` array, which is informational and never identity. | ||
|
|
||
| The server owns the lifecycle. A source describes a condition; the server decides what state its alert is in, what it takes to leave that state, and when it is over. That ownership is why the subsystem is in server core rather than a plugin: alerts arrive from deltas, from REST clients and from plugins, and a single owner is what keeps them from disagreeing. | ||
|
|
||
| Alerts are published as deltas at `alerts.<alert path>` carrying the whole alert as the value, so any Signal K client can mirror the active set without polling. The active set is republished when the server starts, so a client that connects after a restart sees restored alerts without waiting for something to happen. | ||
|
|
||
| ### Lifecycle | ||
|
|
||
| States follow IEC 62682: | ||
|
|
||
| | State | Meaning | | ||
| | -------------------- | ------------------------------------------------------------ | | ||
| | `unacknowledged` | The condition is present and nobody has acknowledged it | | ||
| | `acknowledged` | The condition is present and an operator has acknowledged it | | ||
| | `rtn-unacknowledged` | The condition ended before anyone acknowledged it | | ||
| | `normal` | Terminal. The alert is resolved and has left the active set | | ||
|
|
||
| A `caution` whose condition ends resolves by itself. Anything more urgent waits for acknowledgment, because an alarm that goes away on its own is an alarm nobody saw. An alert marked `latching` waits for acknowledgment whatever its priority. | ||
|
|
||
| An unacknowledged `warning` escalates to an `alarm` after a configurable window. Escalation resets acknowledgment; there is no de-escalation. | ||
|
|
||
| Silencing is orthogonal to all of it: it quiets the annunciator for a bounded time and never changes state, never reorders the list, and never resolves anything. The bound depends on priority; the escalation window and both silence maxima are in `src/api/alerts/index.ts`. | ||
|
|
||
| ### Staleness | ||
|
|
||
| An alert whose source stops re-emitting it is marked `stale` after the source timeout in `src/api/alerts/alertManager.ts`. It stays visible and stays actionable: a source going quiet is not evidence that the condition resolved. Staleness applies to alerts raised by delta, whose re-emission is the heartbeat. An alert raised through REST or the plugin API is raised once and never goes stale. | ||
|
|
||
| ## REST endpoints | ||
|
|
||
| Under `/signalk/v2/api/alerts`: | ||
|
|
||
| | Method | Path | Purpose | | ||
| | ------ | ------------------- | ------------------------------------------------------ | | ||
| | GET | `/` | The active set, ordered as an operator reads it | | ||
| | POST | `/` | Raise an alert, or update the one already on that path | | ||
| | GET | `/{id}` | One alert | | ||
| | POST | `/{id}/acknowledge` | Acknowledge | | ||
| | POST | `/{id}/silence` | Silence, optionally for a given number of seconds | | ||
| | POST | `/{id}/escalate` | Raise to a higher priority | | ||
| | PUT | `/{id}/condition` | Report whether the condition is still present | | ||
| | POST | `/silence-all` | Silence every active alert | | ||
| | GET | `/history` | The audit trail, filterable and paged | | ||
| | GET | `/status` | Whether alert state is being persisted | | ||
|
|
||
| The list is ordered per IMO MSC.302(87) 9.16: emergencies first, then unacknowledged above returned-to-normal above acknowledged, most urgent and most recent first within each group. | ||
|
|
||
| Reading requires read access and every mutating call requires write access, inherited from the v2 API path prefix. | ||
|
|
||
| ## Raising alerts from a delta | ||
|
|
||
| A device raises an alert by sending a value at its alert path: | ||
|
|
||
| ```json | ||
| { | ||
| "context": "vessels.self", | ||
| "updates": [ | ||
| { | ||
| "$source": "n2k-1", | ||
| "values": [ | ||
| { | ||
| "path": "alerts.propulsion.port.oilPressureLow", | ||
| "value": { "priority": "alarm", "message": "Oil pressure low" } | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Only descriptive fields are read — `priority`, `message`, `group`, `latching`, `references`, `data`. Lifecycle fields in an incoming value are ignored: a device cannot declare its own alert acknowledged. | ||
|
|
||
| Sending `null` at the path, or a value with `state: normal`, reports that the condition ended. Any source may do this, not only the one that raised it; the audit trail records who did. | ||
|
|
||
| ## The plugin API | ||
|
|
||
| `app.alerts` gives a plugin the same operations: | ||
|
|
||
| ```javascript | ||
| const alert = await app.alerts.raise({ | ||
| path: 'propulsion.port.oilPressureLow', | ||
| priority: 'alarm', | ||
| message: 'Oil pressure low' | ||
| }) | ||
|
|
||
| await app.alerts.acknowledge(alert.id) | ||
| await app.alerts.silence(alert.id, 30) // seconds, as over REST | ||
| await app.alerts.clearCondition(alert.id) | ||
|
|
||
| app.alerts.list({ state: 'unacknowledged' }) | ||
| ``` | ||
|
|
||
| Alerts raised this way are attributed to the plugin that raised them. `list`, `get` and `getByPath` read the same active set the REST API and the deltas carry. | ||
|
|
||
| ## Limits | ||
|
|
||
| An alert is held in memory, written to the database and republished to every subscriber, so the path, message, group, `references` and `data` fields are each bounded at every ingress surface. The bounds themselves are in `src/api/alerts/description.ts` and `alertPath.ts`. REST rejects an oversized field with 400; a delta carrying one is dropped with a log line. | ||
|
|
||
| The active set is capped. When it is full, a more urgent alert displaces the least urgent one — the lowest priority, and among equals the one whose state changed longest ago — which is announced and recorded in the audit trail as a displacement. An alert no more urgent than everything active is refused. | ||
|
|
||
| ## Persistence | ||
|
|
||
| Active alerts and the audit trail are stored in SQLite under `serverState/alerts/`, so alert state survives a restart. The audit trail is pruned against a retention window and the freed pages are returned to the filesystem over subsequent prunes. | ||
|
|
||
| A failed write does not stop an alert being raised or announced: annunciation never waits on the disk. `GET /status` reports `degraded` while the store and the active set disagree, which clears by itself once a write succeeds again. A database that cannot be opened at all stops the server with an error naming the file, rather than starting an alarm system that silently persists nothing. | ||
|
|
||
| A backup of the database has to include the `-wal` and `-shm` files beside it, or be taken while the server is stopped. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| /** | ||
| * TypeBox Schema Definitions for the Signal K Alerts API | ||
| * | ||
| * These are the source of truth for the alert shapes: the plugin surface | ||
| * types are derived from them, and the OpenAPI document renders them. | ||
| */ | ||
|
|
||
| import { Type, type Static } from '@sinclair/typebox' | ||
| import { IsoTimeSchema } from './shared-schemas' | ||
|
|
||
| export const AlertPrioritySchema = Type.Union( | ||
| [ | ||
| Type.Literal('emergency'), | ||
| Type.Literal('alarm'), | ||
| Type.Literal('warning'), | ||
| Type.Literal('caution') | ||
| ], | ||
| { | ||
| $id: 'AlertPriority', | ||
| description: | ||
| 'Urgency of the condition, from an emergency down to a caution.' | ||
| } | ||
| ) | ||
|
|
||
| export const AlertStateSchema = Type.Union( | ||
| [ | ||
| Type.Literal('normal'), | ||
| Type.Literal('unacknowledged'), | ||
| Type.Literal('acknowledged'), | ||
| Type.Literal('rtn-unacknowledged') | ||
| ], | ||
| { | ||
| $id: 'AlertState', | ||
| description: | ||
| 'Lifecycle state. `rtn-unacknowledged` is a condition that ended ' + | ||
| 'before anyone acknowledged it.' | ||
| } | ||
| ) | ||
|
|
||
| export const HistoryEventTypeSchema = Type.Union( | ||
| [ | ||
| Type.Literal('raise'), | ||
| Type.Literal('acknowledge'), | ||
| Type.Literal('silence'), | ||
| Type.Literal('unsilence'), | ||
| Type.Literal('clear'), | ||
| Type.Literal('escalate') | ||
| ], | ||
| { $id: 'HistoryEventType', description: 'What happened to an alert.' } | ||
| ) | ||
|
|
||
| const AlertPathSchema = Type.String({ | ||
| description: | ||
| 'The condition this alert names, and its identity. One active alert ' + | ||
| 'per path and context.', | ||
| examples: ['propulsion.port.oilPressureLow'] | ||
| }) | ||
|
|
||
| export const AlertSchema = Type.Object( | ||
| { | ||
| id: Type.String(), | ||
| path: AlertPathSchema, | ||
| references: Type.Optional( | ||
| Type.Array(Type.String(), { | ||
| description: | ||
| 'Data paths the condition concerns. Informational, never identity.' | ||
| }) | ||
| ), | ||
| $source: Type.String(), | ||
| source: Type.Optional(Type.Record(Type.String(), Type.Unknown())), | ||
| priority: AlertPrioritySchema, | ||
| state: AlertStateSchema, | ||
| condition: Type.Boolean({ | ||
| description: 'Whether the underlying condition is still present.' | ||
| }), | ||
| latching: Type.Boolean({ | ||
| description: | ||
| 'A latched alert is held until acknowledged, even once the ' + | ||
| 'condition ends.' | ||
| }), | ||
| silenced: Type.Boolean(), | ||
| silencedUntil: Type.Optional(IsoTimeSchema), | ||
| message: Type.String(), | ||
| group: Type.Optional(Type.String()), | ||
| data: Type.Optional(Type.Record(Type.String(), Type.Unknown())), | ||
| raisedAt: IsoTimeSchema, | ||
| stateChangedAt: IsoTimeSchema, | ||
| acknowledgedAt: Type.Optional(IsoTimeSchema), | ||
| acknowledgedBy: Type.Optional(Type.String()), | ||
| clearedAt: Type.Optional(IsoTimeSchema), | ||
| sourceOnline: Type.Boolean(), | ||
| lastSourceUpdate: IsoTimeSchema, | ||
| stale: Type.Boolean({ | ||
| description: | ||
| 'The source stopped reporting. A stale alert stays visible and ' + | ||
| 'actionable.' | ||
| }), | ||
| context: Type.Optional(Type.String()) | ||
| }, | ||
| { $id: 'Alert', description: 'An alert and its lifecycle state.' } | ||
| ) | ||
|
|
||
| export const RaiseAlertRequestSchema = Type.Object( | ||
| { | ||
| path: AlertPathSchema, | ||
| priority: AlertPrioritySchema, | ||
| message: Type.String(), | ||
| references: Type.Optional(Type.Array(Type.String())), | ||
| context: Type.Optional(Type.String()), | ||
| group: Type.Optional(Type.String()), | ||
| latching: Type.Optional(Type.Boolean()), | ||
| data: Type.Optional(Type.Record(Type.String(), Type.Unknown())) | ||
| }, | ||
| { $id: 'RaiseAlertRequest', description: 'What it takes to raise an alert.' } | ||
| ) | ||
|
|
||
| export const TransitionResultSchema = Type.Object( | ||
| { | ||
| alert: Type.Union([AlertSchema, Type.Null()], { | ||
| description: 'The alert after the transition, or null once it resolved.' | ||
| }), | ||
| cleared: Type.Boolean({ | ||
| description: 'Whether the alert left the active set.' | ||
| }), | ||
| previousState: AlertStateSchema | ||
| }, | ||
| { | ||
| $id: 'TransitionResult', | ||
| description: 'The outcome of a transition.', | ||
| required: ['alert', 'cleared', 'previousState'] | ||
| } | ||
| ) | ||
|
|
||
| export const HistoryEntrySchema = Type.Object( | ||
| { | ||
| id: Type.String(), | ||
| alertId: Type.String(), | ||
| path: AlertPathSchema, | ||
| context: Type.Optional(Type.String()), | ||
| priority: AlertPrioritySchema, | ||
| message: Type.String(), | ||
| $source: Type.String(), | ||
| eventType: HistoryEventTypeSchema, | ||
| timestamp: IsoTimeSchema, | ||
| userId: Type.Optional(Type.String()), | ||
| previousState: Type.Optional(AlertStateSchema), | ||
| newState: Type.Optional(AlertStateSchema), | ||
| previousPriority: Type.Optional(AlertPrioritySchema), | ||
| newPriority: Type.Optional(AlertPrioritySchema), | ||
| details: Type.Optional(Type.Record(Type.String(), Type.Unknown())) | ||
| }, | ||
| { $id: 'HistoryEntry', description: 'One entry in the audit trail.' } | ||
| ) | ||
|
|
||
| export const HistoryQueryResultSchema = Type.Object( | ||
| { | ||
| entries: Type.Array(HistoryEntrySchema), | ||
| total: Type.Integer({ | ||
| description: 'Entries matching the query, before paging.' | ||
| }) | ||
| }, | ||
| { $id: 'HistoryQueryResult' } | ||
| ) | ||
|
|
||
| export const StoreStatusSchema = Type.Object( | ||
| { | ||
| store: Type.Object({ | ||
| degraded: Type.Boolean({ | ||
| description: | ||
| 'True when a write failed. Alerts are still raised and announced, ' + | ||
| 'but state may not survive a restart.' | ||
| }) | ||
| }) | ||
| }, | ||
| { $id: 'StoreStatus' } | ||
| ) | ||
|
|
||
| export type Alert = Static<typeof AlertSchema> | ||
| export type AlertPriority = Static<typeof AlertPrioritySchema> | ||
| export type AlertState = Static<typeof AlertStateSchema> | ||
| export type RaiseAlertRequest = Static<typeof RaiseAlertRequestSchema> | ||
| export type TransitionResult = Static<typeof TransitionResultSchema> | ||
| export type HistoryEntry = Static<typeof HistoryEntrySchema> | ||
| export type HistoryEventType = Static<typeof HistoryEventTypeSchema> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.