Alert Lifecycle

An alert is an episode, not a message. One degradation produces one alert that stays open, updates itself, and closes when the line recovers — however long that takes.

Episodes and deduplication

Each episode is identified by what it is about: the organization, the alert type, and the specific thing affected, such as a procedure, a measurement series, or a unit. While an episode is open, new evidence about the same thing updates it instead of creating a second alert.

Severity is a high-water mark: it only ratchets up. A critical alert that eases to warning stays critical until it resolves, because the critical email already went out and the peak belongs in the record.

Resolution

For trends, an episode resolves only when every grader that fired it has recovered. If a custom rule and automatic detection both fired the alert, both must clear. A grader with no fresh data holds its verdict rather than assuming recovery.

Manual resolve and snoozing

Active alerts offer a Resolve button with an optional comment. Resolving by hand snoozes that alert for its evaluation window: your configured window on a custom rule, one hour on automatic detection. When both grade the same episode, the window of whichever grader currently phrases the alert applies. The same condition will not re-alert during that period. Event alerts (run failed, retest, reference samples) never snooze — each new occurrence is a genuinely new fact.

Automatic resolution never snoozes. Only a human closing an alert implies "I know, give me time".

Resolving a unit retest threshold alert also resets that unit's counter on that procedure.

The alert detail page

The page has five sections.

Details

The summary in plain numbers, then Procedure, Station, Unit, Run, Severity, Status, and the triggered, updated, and resolved timestamps. Entity rows open that entity's side panel.

Triggered by

Every rule whose criteria this data breached, with the severity that rule computed on its own. The alert's severity is the highest of them.

• Rules that merely deliver the alert are not listed here.
• When no custom rule fired, one row reads Automatic monitoring.
• Deleted rules render dashed and marked Deleted.

This list is trimmed live as rules stop firing the alert, so it always reflects what is currently firing. The history lives in the activity feed.

Resolve

Shown on active alerts, to anyone who is not a viewer.

Timeline

A control chart of the metric behind the alert, on a sample-order axis rather than a time axis, so every sample gets equal width regardless of production rate. The chart queries live data on every view: new runs appear as they land, whether or not they changed the alert's severity.

• Lime points are inside the band, or passing.
• Pink points are beyond the threshold, or failing.
• A dashed indigo curve is the detector's own smoothed view, labelled Smoothed in the tooltip against the raw Measured value.
• A grey baseline line marks the reference the detector compared against.
• A red Triggered line marks where the alert opened.

Every type charts except unit retest threshold, which has no continuous metric. The trend types render the full control-chart language above; the event types (run failed, reference samples) render a run outcome strip — each run of the alert's stream as a PASS or FAIL dot around the trigger — so a consecutive-failure streak or a flaky reference unit is readable at a glance.

Activity

Newest first: the alert triggering, each escalation, delivery attempts with their recipients and any errors, rule changes affecting this alert, and the resolution with its comment. Automatic resolutions show no actor.

Delivery

Emails are queued and sent by a worker, so the activity feed can show queued, sent, or failed with the error returned by the mail provider.

Escalation emails thread under the original message, so an episode occupies one conversation in your inbox rather than three.