Skip to content
TofuPilotTofuPilot
Get started
Quality & Process Control

Setting Alert Thresholds That Actually Work

Why a fixed yield threshold fails on real production lines, and how a self-calibrating EWMA detector finds the right band for each one.

JJuliette Lansoy
advanced12 min readJuly 27, 2026

Your line tests units all day and you track its first-pass yield (FPY). At what point should someone be alerted? The intuitive answer — pick a threshold — runs into two walls immediately.

The first is level. A procedure running at 99% FPY should alert at 98%, and loudly at 97%: every point lost there is enormous. A procedure running at 50% — a deliberately harsh burn-in, say — should only alert around 45%, and loudly at 40%. No absolute threshold fits both.

The second is intrinsic variability. Two lines can share the same average FPY of 55%: one is very stable and stays within ±1 point; the other naturally swings between 50% and 60% depending on lots and shifts. A drop to 48% is a major event on the first, an ordinary Tuesday on the second. The relevant threshold therefore depends not only on the level, but on each line's habitual behaviour.

A trustworthy alerting system has to calibrate itself on both axes, for every procedure, without asking the user to guess constants. Here is how statistics goes about it, and the choices we drew from it.

The state of the art: detecting a change in a stream of tests

The problem has had a name for a century: statistical process control (SPC), founded by Walter Shewhart in the 1920s. Its central idea fits in one sentence: every process has routine variation (its "common causes"), and you should only alert when an observation leaves that routine (a "special cause"). Shewhart charts judge each point in isolation; two more recent methods, still the current references, add the decisive ingredient for small drifts: memory.

CUSUM: the evidence accumulator

The CUSUM chart (cumulative sum, Page, 1954) processes each tested unit, one by one, and maintains a counter of evidence. An electronics analogy: an integrator with a constant leak and a floor at zero. Every failure injects charge, every pass lets a little leak away; on a healthy process the leak wins and the counter sits at zero; after a genuine drift the injections take over and the counter climbs inexorably to the alarm threshold. The version adapted to pass/fail streams under 100% inspection (the Bernoulli CUSUM, Reynolds & Stoumbos, 1999) is explicitly presented by its authors as superior to the practice of grouping units into samples or time windows. And CUSUM holds a rare title in statistics: for a given false-alarm rate, it is provably the fastest detector for the drift it was specified for (Moustakides, 1986).

EWMA: smoothed yield

The EWMA chart (exponentially weighted moving average, Roberts, 1959) is a low-pass filter applied to the pass/fail pulse train: its output is a smoothed estimate of the current failure rate, where each unit weighs slightly less than the next one — old data fades gently instead of falling off a cliff as it does in a sliding window. A single parameter, λ, sets the memory depth, expressed in units produced rather than clock time: a fast line covers its memory in a few hours, a slow line in a few days, and both are monitored with the same statistical rigour. You alert when the smoothed yield departs too far from its reference value, the gap being measured in standard deviations.

The difference in nature matters for what follows: EWMA is a gauge (its output reads in the unit of the metric — "smoothed FPY: 91.2%"), CUSUM is an accumulator (its output, "2.9 units of evidence toward a limit of 4", has no direct interpretation for a human eye).

What about the other methods?

Others exist — self-sizing windows from data stream mining (ADWIN), Bayesian changepoint detection, adaptive sampling charts — which we will not explore here: they are either designed for other data regimes, or considerably harder to explain and audit for a quality engineer, which matters when an alert is supposed to trigger action on a production floor. References are at the end of the article for the curious.

The textbook trap: one noise model for every line

Open an SPC textbook and the formulas for CUSUM as well as EWMA on pass/fail data rest on the same assumption: the noise is binomial. In other words, every unit would have exactly the same probability of failure, and a line's variability would follow entirely from its average level through a formula, without ever being measured.

Real production does not honour that contract. The true failure probability wanders from hour to hour: shift changes, component lots, temperature, tooling wear. As a result the observed variability exceeds — sometimes by a lot — what the formula predicts. The phenomenon has a name, overdispersion, and it is the best-documented defect of attribute charts applied as-is in a factory (it is the problem Laney's p′ chart, 2002, was invented to correct). During our training phase on real production datasets (more on that below), we measured variability ranging from 1× to more than 5× the binomial prediction depending on the line, on procedures that were all perfectly healthy.

The consequences of a textbook setting are then inescapable. Calibrate the thresholds for the fluctuating line and they go blind on the stable one: real drops slip under the radar. Calibrate them for the stable line and the fluctuating one fires false alerts in bursts — and everyone knows what follows, documented in every factory in the world: the alerts end up ignored, including the true ones. This is precisely why "picking a threshold" is so hard: there is no good global threshold, only a good threshold per line, and it changes when the line changes.

Why EWMA answers this better

Because the EWMA filter's output is a gauge in the unit of the metric, one correction suggests itself naturally: instead of computing its dispersion theoretically from the binomial formula, measure it. Replay the filter over the line's own history, observe the real distribution of its values, and the alert band becomes: "the smoothed yield departs by N observed standard deviations from its own historical behaviour". Both axes of the problem are then covered at once:

  • Level. The reference is estimated per procedure, and the band's natural width follows the level. The noise of a pass/fail stream is √(p(1−p)): it is mechanically finer at 99% FPY than at 50%. The excellent line is monitored to the point, the difficult line to the multiple of points.
  • Intrinsic variability. The line that sits steady at 55% has a tight smoothed history, so a narrow band; the line swinging between 50% and 60% has a wandering smoothed history, so a wide one. Each is judged against itself.

Can the same correction be applied to CUSUM? Only with difficulty, and this is the technical point that separates the two methods: CUSUM's entire calibration — the alarm threshold, the optimality guarantee, the advertised false-alarm rate — is welded to the binomial model by the very construction of its evidence counter. Under overdispersion those guarantees evaporate, and the variants that restore them require fitting an additional statistical model per line and recalibrating by simulation. Workable in a study, not in a product monitoring hundreds of heterogeneous procedures without human intervention.

Then there is the argument we care about most: what the user sees. An EWMA alert shows itself. The smoothed yield curve is plotted in the unit the quality engineer knows, with its band of normality around it, and the alert reads off it: "smoothed FPY at 91.2% against a normal of 95.1% ± 1.2". Every number is recomputable by hand from the runs. A CUSUM alert has to be translated: the evidence counter has no meaningful axis, and the counts that explain it must be reconstructed after the fact. For a system whose goal is trust — that a quality engineer can audit why they were woken up — the gauge beats the accumulator.

What is left to fix, and why we trained it on real data

Self-calibration does not remove every constant; it changes their nature. Everything that depends on the line (reference level, variability) is measured continuously. Two universal settings remain, the same for everyone:

  • λ, the memory depth. In reality a choice of boundary between what counts as weather (fast fluctuations, absorbed into the band) and what counts as change (sustained drifts, which alert). It is chosen through a promise about reaction time: "confirm a sustained drift within one to two hundred units".
  • The severity scale. The standard-deviation multipliers separating info, warning and critical. This is a false-alarm budget: at what historical rarity does something deserve to disturb someone?

How do you set these values without falling back into arbitrariness? By training them: replay the candidate detector, setting by setting, over real production datasets — tens of thousands of tests covering widely varied profiles (lines at 99% as well as 50% FPY, production in bursts of several hundred units a day as well as steady lines running a few units a day). Then count: how many alerts each setting would have raised, at what severity, and whether the confirmed incidents would have been detected, and within how many units.

That training settled three things. First λ = 0.02 (a useful memory of roughly 50 to 150 units): shorter settings widen the band faster than they deepen the excursions and lose the critical signal on real incidents; longer ones double the reaction time with no detection gain. Next the 2σ/3σ/5σ scale: over two months of real data, 2σ corresponds to a handful of informative episodes per line, 3σ almost exclusively to genuine events, and 5σ exclusively to one authentic incident (an FPY that fell from 97% to 80% over about a hundred units, classified critical by the detector). Finally a guard rail: the measured band is floored by the binomial minimum, so that a flawless line (100% FPY over thousands of units, zero observed dispersion) cannot turn its first failure into an "infinite" alarm.

One last check, from outside the data: we gave sample datasets to an experienced quality engineer and asked at which levels they would have wanted to be alerted. With λ = 0.02, the computed band lands on their answers: at 99% FPY, alert at 98% and strong alert at 97%; at 50%, alert at 45% and strong at 40%. A calibration derived from replay and a field intuition converging without having consulted each other is the kind of agreement you look for.

And the training is not a one-off event: the replay harness is kept, and every evolution of the detector goes through it again before being deployed.

Sources and further reading

Main methods:

The other methods mentioned:

More Guides

Quality & Process Control
What Is a Histogram in Manufacturing Test
A histogram shows the distribution shape of your measurement data. Learn how to read histogram patterns, spot problems, and use them for SPC in TofuPilot.
Quality & Process Control
Cpk vs Ppk: What's the Difference
Cpk and Ppk both measure process capability, but use different standard deviations. Learn when to use each, what the gap reveals, and compare them in TofuPilot.

Put this guide into practice

Start freeTalk to us