# TofuPilot

> Manufacturing test platform for hardware teams. Build and deploy hardware tests faster.

## Guides

### PCBA Functional Testing: A Complete Guide
URL: https://www.tofupilot.com/guides/pcba-functional-testing-a-complete-guide-with-python-and-tofupilot
PCBA functional test with Python: what to measure, how to structure the procedure, fixture and instrument plugs, and tracking yield per measurement.

Functional circuit test (FCT) is the last electrical gate before a board ships or goes into a higher assembly. In-circuit test proves the components are present and correct; functional test proves the assembled board does its job. This guide covers what to measure, how to structure the test, and how to track the results so the line tells you when something changes.

## What FCT Covers

| Area | Typical checks |
|------|---------------|
| Power | Rail voltages under load, inrush, quiescent and active current |
| Programming | Flash firmware, verify the image, read back the version |
| Digital I/O | GPIO loopback, interrupt response, bus enumeration |
| Analog | ADC accuracy, DAC output, reference voltage, sensor readback |
| Communication | UART, SPI, I2C, CAN, Ethernet link and data integrity |
| RF | Transmit power, receive sensitivity, frequency error |
| Calibration | Measure offsets, compute coefficients, write them to the DUT |
| Safety | Hipot, leakage, ground bond where applicable |

FCT is usually 100 % inspection: every board, every time. That makes it the largest source of test data you have, and the place where [first pass yield](/guides/what-is-first-pass-yield-and-how-to-track-it-with-tofupilot) is normally measured.

## FCT vs the Other PCBA Tests

| Test | Catches | Misses |
|------|---------|--------|
| [AOI/SPI/AXI](/guides/what-is-aoi-spi-and-axi-with-tofupilot) | Solder and placement defects, visually | Anything electrical |
| [ICT / flying probe](/guides/ict-vs-fct-vs-flying-probe-when-to-use-each-pcba-test-method) | Wrong, missing, reversed components; shorts and opens | Whether the system works |
| **FCT** | System-level failures, firmware, calibration | Component-level root cause |
| [Burn-in](/guides/burn-in-testing-for-electronics-a-complete-guide) | Infant mortality | Anything a short test would find |

A board that passes ICT and fails FCT usually has a design-marginal or firmware problem, not an assembly defect. That distinction is only visible if both stages record measurements rather than verdicts.

## Structuring the Test

The fixture, the instruments and the DUT connection each become a plug. The test steps become phases. The limits live in the procedure as data.

### The plugs

```python filename="plugs/fixture.py"
class Fixture:
    """Pogo-pin fixture: lid detection and DUT power."""

    def lid_closed(self) -> bool:
        return True  # Replace with a real interlock read

    def power_on(self):
        pass  # Replace with a real relay or PSU call

    def power_off(self):
        pass

    def close(self):
        self.power_off()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def measure_dc_voltage(self) -> float:
        return 3.31  # Replace with a PyVISA query

    def measure_dc_current_ma(self) -> float:
        return 41.3
```

```python filename="plugs/dut.py"
class Dut:
    """Serial console or debug probe on the board under test."""

    def query_firmware(self) -> str:
        return "2.1.0"

    def read_adc_counts(self) -> int:
        return 2048

    def gpio_loopback(self) -> bool:
        return True
```

### The phases

```python filename="phases/safety_check.py"
def safety_check(measurements, fixture):
    """Refuse to energise the board if the lid is open."""
    measurements.lid_closed = fixture.lid_closed()
```

```python filename="phases/power_rails.py"
def power_rails(measurements, fixture, dmm):
    fixture.power_on()
    measurements.rail_3v3 = dmm.measure_dc_voltage()
    measurements.quiescent_current = dmm.measure_dc_current_ma()
```

```python filename="phases/firmware_check.py"
def firmware_check(measurements, dut):
    measurements.firmware_version = dut.query_firmware()
```

```python filename="phases/analog_check.py"
def analog_check(measurements, dut):
    measurements.adc_counts = dut.read_adc_counts()
```

```python filename="phases/digital_check.py"
def digital_check(measurements, dut):
    measurements.gpio_loopback = dut.gpio_loopback()
```

### The procedure

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0
description: FCT for PCBA-100, per TP-100 rev B

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: fixture
    python: plugs.fixture:Fixture
  - name: dmm
    python: plugs.dmm:Multimeter
  - name: dut
    python: plugs.dut:Dut

main:
  - name: Safety Check
    python: phases.safety_check
    measurements:
      - name: Lid Closed
        description: Fixture interlock before power is applied
        validators:
          - operator: "=="
            expected_value: true

  - name: Power Rails
    python: phases.power_rails
    depends_on: [safety_check]
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Quiescent Current
        unit: mA
        validators:
          - operator: "<="
            expected_value: 45.0

  - name: Firmware Check
    python: phases.firmware_check
    depends_on: [power_rails]
    measurements:
      - name: Firmware Version
        validators:
          - operator: "=="
            expected_value: "2.1.0"

  - name: Analog Check
    python: phases.analog_check
    depends_on: [power_rails]
    measurements:
      - name: ADC Counts
        validators:
          - operator: ">="
            expected_value: 2000
          - operator: "<="
            expected_value: 2096

  - name: Digital Check
    python: phases.digital_check
    depends_on: [power_rails]
    measurements:
      - name: GPIO Loopback
        validators:
          - operator: "=="
            expected_value: true
```

The dependency graph carries real meaning here. Safety runs first and everything waits on it. Power runs next. Firmware, analog and digital all depend on power but not on each other, so the engine runs those three concurrently and the cycle time is the longest of them rather than their sum.

Run it with `tofupilot run ./procedure.yaml`.

## Three Decisions That Determine Whether the Data Is Useful

**Record the value, not the verdict.** `rail_3v3 = 3.31` with limits attached supports Cpk, drift detection and a limit review later. A boolean `power_ok` supports none of those, and cannot tell you the rail has been creeping toward the limit for three weeks.

**Keep measurement names stable.** The name is the identity of the measurement across every run ever taken. Renaming `Rail 3V3` to `3V3 Rail` splits the history in two. Pick names at the start and leave them alone.

**Continue past failures where it is safe.** A board that fails the 3.3 V rail will usually fail several downstream checks too, and knowing which ones is diagnostic. A board that fails the lid interlock must stop immediately. That is exactly the distinction `depends_on` encodes.

## Testing Several Boards at Once

When the fixture holds more than one board, declare slots. Each slot gets its own unit, its own measurements and its own outcome, and they run concurrently:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: Slot 1
    - name: Slot 2
    - name: Slot 3
    - name: Slot 4
```

A failure in one slot stops that slot only; the rest keep testing. See [how to run parallel tests on multiple DUTs](/guides/how-to-run-parallel-tests-on-multiple-duts-with-tofupilot).

## Calibration Inside FCT

If the board needs calibration coefficients written into it, that is an FCT phase, and both the raw measurement and the computed coefficient belong in the record:

```python filename="phases/calibrate.py"
def calibrate(measurements, dut, dmm):
    reference = dmm.measure_dc_voltage()
    counts = dut.read_adc_counts()
    gain = reference / counts if counts else 0.0

    measurements.reference_voltage = reference
    measurements.adc_counts = counts
    measurements.adc_gain = gain

    dut.write_calibration(gain)
```

Storing the coefficient per serial means a field failure can be checked against what was actually programmed into that unit.

## Reading the Results

| Question | Where it is answered |
|----------|---------------------|
| What is our yield this shift? | First pass yield on the procedure |
| Which step costs us the most units? | [Pareto of failing measurements](/guides/how-to-use-pareto-analysis-for-test-failures-with-tofupilot) |
| Is a measurement drifting toward its limit? | [Control chart](/guides/how-to-use-control-charts-for-tests-with-tofupilot) on that measurement |
| Are our limits too tight? | [Cpk per measurement](/guides/what-is-cpk-process-capability-index) |
| Is one fixture worse than the others? | Filter by slot or station |
| What did this returned board measure? | The unit history for that serial |

## If You Already Run OpenHTF

Existing OpenHTF suites run natively on the TofuPilot CLI with no rewrite:

```python filename="fct.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class DutPlug(BasePlug):
    def read_voltage(self) -> float:
        return 3.31


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
@htf.plug(dut=DutPlug)
def test_power_rails(test, dut):
    test.measurements.rail_3v3 = dut.read_voltage()


test = htf.Test(test_power_rails, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Recording pass/fail only | No Cpk, no drift detection, no limit review | Record the measured value with limits |
| Renaming measurements | History splits, trends break | Fix names early and keep them |
| One giant phase | No retry granularity, no parallelism, poor diagnostics | Split into phases along dependency lines |
| Powering the board before the interlock check | Safety incident, damaged fixture | Gate power on a safety phase with `depends_on` |
| Limits copied from the datasheet forever | Either scrapping good boards or shipping marginal ones | Review limits against the measured distribution |
| Not recording the fixture or station | Cannot tell a bad fixture from a bad lot | Record station and slot on every run |

### How to Compare Yield Between Lines
URL: https://www.tofupilot.com/guides/how-to-compare-yield-between-production-lines-with-tofupilot
Compare first-pass yield across production lines to identify underperforming equipment, operator training gaps, and process variation.

Two production lines running the same product should produce the same yield. When they don't, the gap points to equipment issues, process variation, or operator training problems. TofuPilot lets you compare lines side by side without exporting data or building reports manually.

## Why Yield Comparison Matters

A 2% yield difference between lines might look small. On 10,000 units per month, that's 200 extra failures, each costing rework time, replacement components, and delayed shipments. Catching the gap early and fixing the root cause pays for itself quickly.

Common causes of line-to-line yield variation:

| Cause | What to look for |
|-------|-----------------|
| Equipment calibration | One line's measurements consistently offset from the other |
| Fixture wear | Increasing contact resistance on older test fixtures |
| Operator technique | Manual steps done differently between shifts or lines |
| Component lots | Different reels or batches feeding each line |
| Environment | Temperature or humidity differences between areas |

## Tag Runs with Line Information

Use `station_id` to encode which line produced each unit. TofuPilot uses this field to group and filter results.

```python filename="test_power_board.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("rail_3v3_voltage")
    .in_range(minimum=3.25, maximum=3.40)
    .with_units(units.VOLT),
    htf.Measurement("rail_5v0_voltage")
    .in_range(minimum=4.90, maximum=5.10)
    .with_units(units.VOLT),
    htf.Measurement("standby_current")
    .in_range(maximum=0.015)
    .with_units(units.AMPERE),
)
def test_power_rails(test):
    test.measurements.rail_3v3_voltage = 3.31
    test.measurements.rail_5v0_voltage = 5.02
    test.measurements.standby_current = 0.0114

def main():
    test = htf.Test(test_power_rails)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PWR-2026-07823")

if __name__ == "__main__":
    main()
```

Each station on Line A gets an ID like `LINE-A-FCT-01`, `LINE-A-FCT-02`. Line B uses `LINE-B-FCT-01`, `LINE-B-FCT-02`. This lets you filter at both the line level and the individual station level.

## Run Identical Tests on Every Line

Yield comparison is only valid when every line runs the exact same test procedure with the same limits. Version-control your test scripts and deploy the same commit to all lines.

```python filename="test_comms_board.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("wifi_rssi")
    .in_range(minimum=-55.0),
    htf.Measurement("ble_rssi")
    .in_range(minimum=-65.0),
    htf.Measurement("antenna_impedance")
    .in_range(minimum=45.0, maximum=55.0)
    .with_units(units.OHM),
)
def test_wireless(test):
    test.measurements.wifi_rssi = -42.3
    test.measurements.ble_rssi = -51.7
    test.measurements.antenna_impedance = 49.8

def main():
    test = htf.Test(test_wireless)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "COMMS-2026-11045")

if __name__ == "__main__":
    main()
```

If one line uses different firmware or a modified fixture, document that difference. Otherwise your yield comparison is measuring two different things.

## Compare Lines in TofuPilot

With runs uploading from both lines, TofuPilot's filtering and analytics give you direct comparison:

- **FPY by station** shows yield for each station grouped by line prefix. If Line B consistently trails Line A, the problem is systemic to that line.
- **Measurement histograms** reveal whether one line's values are shifted or have wider spread. A shifted mean suggests calibration offset. Wider spread suggests fixture or process inconsistency.
- **Failure Pareto by line** shows which specific tests fail more often on each line. If Line B fails `antenna_impedance` three times more than Line A, start investigating that fixture.
- **Trend charts** show whether the gap is constant, growing, or appeared suddenly after a change.

## Investigate and Act on Differences

When you find a yield gap, narrow down the cause systematically:

1. **Check measurement distributions.** If one line's values are offset, it's likely calibration or equipment. If they're wider, it's process variation.
2. **Check by time of day.** Yield drops on night shifts point to operator training or environmental changes.
3. **Check by component lot.** If you track lot numbers, filter by lot to see if specific batches drive the difference.
4. **Check individual stations.** Sometimes the "line" problem is actually one bad station dragging down the average.

Fix the root cause, then watch TofuPilot's trend data to confirm the gap closes. The measurement-level detail in each run gives you the evidence to verify the fix worked.

### What Is First Article Inspection (FAI)
URL: https://www.tofupilot.com/guides/what-is-fai-with-tofupilot
First article inspection (FAI) verifies the first production unit meets all specifications. Learn how FAI works and how to track it with TofuPilot.

First article inspection (FAI) is a detailed verification of the first production unit to confirm that the manufacturing process can produce parts that meet all design specifications. It's required by aerospace (AS9102), automotive (PPAP), and medical device standards. This guide covers what FAI involves, when it's required, and how to track FAI results with TofuPilot.

## What FAI Verifies

FAI checks every characteristic on the drawing and specification, not just the critical ones. It's a full accounting of:

| What's Checked | Example |
|---------------|---------|
| Dimensional characteristics | All dimensions on the drawing |
| Material certifications | Raw material CoC, RoHS compliance |
| Process records | Solder profile, torque records, adhesive cure |
| Functional test results | All measurements against specification |
| Visual and cosmetic | Surface finish, marking, label placement |
| Traceability | Serial numbers, lot codes, date codes |

FAI is not sampling. It's a complete inspection of one unit (or a set of units) against every requirement.

## When FAI Is Required

| Trigger | Example |
|---------|---------|
| New product introduction | First production run of a new design |
| Design change | Engineering change order (ECO) affecting form, fit, or function |
| Process change | New supplier, new tooling, new factory location |
| Extended production break | No production for 2+ years (per AS9102) |
| Corrective action | After a nonconformance that affected product quality |

## FAI Documentation (AS9102)

AS9102 defines three standard FAI report forms:

| Form | Content |
|------|---------|
| Form 1: Part Number Accountability | Part number, name, revision, serial number, materials |
| Form 2: Product Accountability (Raw Material, Special Process, Functional Test) | Material certs, process records, test data |
| Form 3: Characteristic Accountability | Every dimension and characteristic with actual measured value |

Form 3 is the most labor-intensive. Every dimension on the drawing gets a balloon number, and the corresponding measured value goes in the FAI report.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Define FAI Test Phases

The functional test portion of FAI runs the same test procedure as production, but you record every measurement, not just pass/fail.

```python filename="fai_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("ripple_voltage_mV")
    .in_range(maximum=50)
    .with_units(units.MILLIVOLT),
    htf.Measurement("efficiency_percent")
    .in_range(minimum=90)
    .with_units(units.PERCENT),
    htf.Measurement("leakage_current_uA")
    .in_range(maximum=500)
    .with_units(units.MICROAMPERE),
)
def phase_fai_electrical(test):
    """FAI Form 2: All electrical characteristics."""
    test.measurements.output_voltage_V = 5.02
    test.measurements.ripple_voltage_mV = 18.3
    test.measurements.efficiency_percent = 93.7
    test.measurements.leakage_current_uA = 65.0


@htf.measures(
    htf.Measurement("firmware_version").equals("2.1.0"),
    htf.Measurement("boot_time_ms")
    .in_range(maximum=2000)
    .with_units(units.MILLISECOND),
    htf.Measurement("self_test").equals("PASS"),
)
def phase_fai_functional(test):
    """FAI Form 2: Functional characteristics."""
    test.measurements.firmware_version = "2.1.0"
    test.measurements.boot_time_ms = 1120
    test.measurements.self_test = "PASS"
```

## Step 2: Log FAI Results to TofuPilot

The FAI test runs once on the first article. TofuPilot stores every measurement with its limit, unit, and result.

```python filename="fai_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_fai_electrical,
    phase_fai_functional,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan first article serial: "))
```

## Step 3: Generate the FAI Report

TofuPilot stores the complete test record for the first article. Open the unit's test history to get:

- **All measurements** with actual values against specification limits
- **Pass/fail status** for each characteristic
- **Timestamps** for audit trail
- **Test reports** exportable for AS9102 Form 2 documentation

This data feeds directly into the FAI report. Instead of transcribing measurements from a bench multimeter into a spreadsheet, the test data is already structured and auditable.

## FAI vs Production Test

| Aspect | FAI | Production Test |
|--------|-----|-----------------|
| Scope | Every characteristic on the drawing | Critical functional characteristics |
| Sample | First unit (or first from each cavity/tool) | Every unit |
| Documentation | Formal FAI report (AS9102 forms) | Internal test records |
| Frequency | Once per trigger event | Every unit, every run |
| Goal | Prove the process can make a conforming part | Verify each unit conforms |

FAI proves the process is capable. Production test proves each unit is conforming. Both are necessary. TofuPilot tracks both using the same infrastructure.

### What Is Automated Test Equipment (ATE)
URL: https://www.tofupilot.com/guides/what-is-ate-with-tofupilot
Automated test equipment combines instruments, a fixture and software to test units without manual measurement. What ATE is and how to build one.

Automated test equipment is the combination of instruments, a fixture and software that tests a unit without an engineer taking readings by hand. The term covers everything from a bench with a multimeter and a script to a semiconductor tester costing millions. What they share is the same three layers.

## The Three Layers

| Layer | What it is | Examples |
|-------|-----------|----------|
| **Instruments** | The things that stimulate and measure | DMM, power supply, oscilloscope, DAQ, RF analyser |
| **Fixture** | The mechanical and electrical interface to the unit | Pogo-pin bed, clamshell, cable harness, handler |
| **Software** | Sequencing, limits, records, operator interaction | The test executive and your test code |

Most ATE problems that look like software problems are fixture problems, and most that look like fixture problems are contact problems. That ordering is worth remembering when yield drops on one station and not another.

## Types of ATE

| Type | Tests | Typical cost |
|------|-------|-------------|
| Bench ATE | One board, one operator, low volume | Thousands |
| Functional test fixture (FCT) | Assembled PCBAs at end of line | Tens of thousands |
| In-circuit tester (ICT) | Components and nets on a bare-populated board | Hundreds of thousands |
| Flying probe | Same as ICT, no dedicated fixture, slower | Hundreds of thousands |
| Semiconductor ATE | Wafers and packaged die | Millions |
| System-level test | Fully assembled products | Varies |

## What the Software Layer Has to Do

This is the part teams underestimate, because instruments and fixtures are visible and software is not.

| Responsibility | Why it matters |
|---|---|
| Sequencing | Steps run in a defined order, with dependencies |
| Limit evaluation | Pass/fail derived from the measurement, not hand-coded |
| Instrument lifecycle | Sessions opened once, shared, closed even on failure |
| Operator interaction | Serial scan, prompts, clear pass/fail indication |
| Result records | The measured value survives the shift |
| Traceability | Which unit, which station, which limits, which code |
| Recovery | A failure stops safely, not halfway through energised |

A test that prints PASS to a console has done the first two and none of the rest.

## Building a Simple ATE

The three layers map directly onto a procedure: instruments are plugs, the sequence and limits are the procedure, and the test logic is phases.

```python filename="plugs/psu.py"
class PowerSupply:
    """Instrument layer: programmable supply."""

    def configure(self, volts: float, current_limit: float):
        pass  # Replace with SCPI writes

    def output(self, on: bool):
        pass

    def measure_current_ma(self) -> float:
        return 41.3

    def close(self):
        self.output(False)
```

```python filename="plugs/dmm.py"
class Multimeter:
    def dc_voltage(self) -> float:
        return 3.31  # Replace with a PyVISA query
```

```python filename="plugs/fixture.py"
class Fixture:
    """Fixture layer: interlock and contact sensing."""

    def lid_closed(self) -> bool:
        return True

    def contact_ok(self) -> bool:
        return True
```

```python filename="phases/safety.py"
def safety(measurements, fixture):
    """Nothing is energised until the fixture says it is safe."""
    measurements.lid_closed = fixture.lid_closed()
    measurements.contact_ok = fixture.contact_ok()
```

```python filename="phases/power_up.py"
def power_up(measurements, psu, dmm):
    psu.configure(volts=5.0, current_limit=0.5)
    psu.output(True)
    measurements.rail_3v3 = dmm.dc_voltage()
    measurements.inrush_current = psu.measure_current_ma()
```

```yaml filename="procedure.yaml"
name: Bench ATE Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: psu
    python: plugs.psu:PowerSupply
  - name: dmm
    python: plugs.dmm:Multimeter
  - name: fixture
    python: plugs.fixture:Fixture

main:
  - name: Safety
    python: phases.safety
    measurements:
      - name: Lid Closed
        validators:
          - operator: "=="
            expected_value: true
      - name: Contact Ok
        validators:
          - operator: "=="
            expected_value: true

  - name: Power Up
    python: phases.power_up
    depends_on: [safety]
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Inrush Current
        unit: mA
        validators:
          - operator: "<="
            expected_value: 450
```

Run it with `tofupilot run ./procedure.yaml`.

Two structural points that make this ATE rather than a script:

**Safety gates power.** `depends_on: [safety]` means the supply is never enabled on an open fixture. That is a hard requirement of any ATE that energises anything.

**Power-down lives in the plug.** `close()` runs at teardown regardless of how the run ended, so a failure does not leave a board energised on the bench.

## Contact Resistance Is the Most Common ATE Failure

Pogo pins wear, oxidise and collect flux. A worn fixture produces intermittent failures on measurements that have nothing to do with the board, and the usual response is to blame the product.

The defence is to measure the interface itself rather than inferring it:

```python filename="phases/contact_check.py"
def contact_check(measurements, dmm):
    """Four-wire resistance across a known short on the fixture."""
    measurements.contact_resistance = dmm.four_wire_resistance()
```

```yaml filename="procedure.yaml"
  - name: Contact Check
    python: phases.contact_check
    depends_on: [safety]
    measurements:
      - name: Contact Resistance
        unit: "mΩ"
        description: Across the fixture's calibration short
        validators:
          - operator: "<="
            expected_value: 50
```

Trending that one measurement tells you when to service the fixture, before it starts failing good boards.

## Scaling to Multiple Units

When the fixture holds several boards, declare slots. Each gets its own unit, measurements and outcome, and they run concurrently:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: Slot 1
    - name: Slot 2
    - name: Slot 3
    - name: Slot 4
```

A shared resource, such as a single rack supply feeding all slots, belongs in an execution-scoped setup and teardown rather than per-slot, otherwise one slot finishing will power down its neighbours.

## Recording the Station

An ATE fleet needs to know which station produced a result:

```python filename="phases/record_station.py"
def record_station(run):
    import socket
    run.metadata["station"] = socket.gethostname()
    run.metadata["fixture_id"] = "FIX-4417-02"
```

Without it, a fixture-specific problem is indistinguishable from a product problem in the aggregate yield.

## Buying vs Building

| Situation | Direction |
|-----------|-----------|
| Standard boards, moderate volume | Build a bench ATE; the cost is mostly your time |
| High volume, tight cycle time | Buy a fixture, build the software |
| Component-level coverage needed | Buy ICT or use flying probe |
| Regulated product with qualification burden | Buy, and budget for the qualification |
| Prototype and NPI | Build; requirements change weekly |

The software layer is worth building even when you buy the hardware, because it is where the traceability and the analytics live, and it is the part that has to outlast any individual fixture.

## If You Already Run OpenHTF

```python filename="ate.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class FixturePlug(BasePlug):
    def lid_closed(self) -> bool:
        return True


@htf.measures(htf.Measurement("lid_closed").equals(True))
@htf.plug(fixture=FixturePlug)
def test_safety(test, fixture):
    test.measurements.lid_closed = fixture.lid_closed()


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
def test_power(test):
    test.measurements.rail_3v3 = 3.31


test = htf.Test(test_safety, test_power, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| No interlock gating power | Fixture damage, safety incident | Gate energising phases on a safety phase |
| Power-down as the last phase | Board left energised after a failure | Put it in the plug's `close()` |
| No contact resistance measurement | Fixture wear looks like product failure | Measure and trend the interface |
| No station or fixture ID | Cannot separate a bad fixture from a bad lot | Record both on every run |
| Recording pass/fail only | No drift detection, no limit review | Record the measured values |
| Shared instrument in per-slot teardown | One slot powers down its neighbours | Use execution scope for shared resources |

### What Is HALT and HASS Testing
URL: https://www.tofupilot.com/guides/what-is-halt-and-hass-testing-with-tofupilot
HALT finds design limits by stressing to destruction, HASS screens production units. How both work, when to use each, and how to record results.

HALT and HASS are stress testing methods that sound similar and do opposite things. HALT destroys a handful of units on purpose to find where the design breaks. HASS screens production units with a stress profile chosen to be survivable. Confusing them is expensive in both directions.

## The Difference in One Table

| | HALT | HASS |
|---|------|------|
| **Full name** | Highly Accelerated Life Test | Highly Accelerated Stress Screen |
| **When** | Design phase | Production |
| **Sample** | A few units (3-10) | Every unit, or a large sample |
| **Goal** | Find the design's destruct limits | Precipitate latent defects |
| **Stress level** | Increased until failure, beyond spec | Fixed, below the destruct limit |
| **Expected outcome** | Every unit eventually fails | Every unit survives |
| **A failure means** | Useful information about margin | This unit had a defect |
| **Output** | Operating and destruct limits | Pass/fail per unit |

The single most important relationship: **HALT defines the stress levels that HASS is allowed to use.** You cannot design a HASS profile without HALT data, because you would be guessing at how much stress the product survives.

## HALT: Finding the Limits

HALT applies increasing stress until the product stops working, in steps, one stress type at a time and then combined.

| Step | Stress | What you learn |
|------|--------|----------------|
| 1 | Cold step stress | Lower operating and destruct limits |
| 2 | Hot step stress | Upper operating and destruct limits |
| 3 | Rapid thermal transitions | Sensitivity to thermal shock |
| 4 | Vibration step stress | Mechanical resonances, solder joint weakness |
| 5 | Combined thermal + vibration | Interaction failures neither finds alone |

Two thresholds come out of each stress type:

- **Operating limit.** The product stops working but recovers when the stress is removed.
- **Destruct limit.** The product is permanently damaged.

The margin between the specification and the operating limit is the number that matters. A product specified to 60 °C with an operating limit of 65 °C has almost no margin; one with an operating limit of 110 °C has plenty.

HALT is not a pass/fail test and has no meaningful yield. Every unit is expected to fail eventually. Stopping HALT because a unit broke defeats the purpose.

## Recording HALT

HALT results are limits, not verdicts. The useful record is the stress level at which each failure mode appeared.

```python filename="phases/thermal_step.py"
def thermal_step(measurements, chamber, dut):
    """One step of a cold or hot step-stress sequence."""
    measurements.chamber_temperature = chamber.read_temperature()
    measurements.dut_functional = dut.self_test_passes()
    measurements.supply_current = dut.supply_current_ma()
```

```yaml filename="procedure.yaml"
name: HALT Thermal Step
version: 1.0.0
description: One step in a HALT step-stress sequence; failures are expected

unit:
  serial_number:
    default_value: "HALT-001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: chamber
    python: plugs.chamber:Chamber
  - name: dut
    python: plugs.dut:Dut

main:
  - name: Thermal Step
    python: phases.thermal_step
    measurements:
      - name: Chamber Temperature
        unit: "°C"
        description: Actual soak temperature, recorded not asserted
      - name: DUT Functional
        description: Whether the unit still operates at this step
      - name: Supply Current
        unit: mA
```

Note what is missing: validators. A HALT step has no limits, because the point is to record where behaviour changes, not to judge it. A measurement with no validator is recorded and reported without a pass/fail verdict, which is exactly right here.

The failure step is then visible in the unit's history: the temperature at which `dut_functional` went false is the operating limit, and whether it recovered on the way back tells you if that was also the destruct limit.

```python filename="phases/record_halt_context.py"
def record_halt_context(run):
    run.metadata["stress_type"] = "cold_step"
    run.metadata["step_number"] = 7
    run.metadata["recovered_after_step"] = True
```

## HASS: Screening Production

HASS uses a fixed profile derived from the HALT results, typically set well inside the operating limits, to precipitate latent defects without damaging good units.

| Parameter | Typical derivation |
|-----------|-------------------|
| Temperature extremes | Inside the operating limits, outside the spec range |
| Ramp rate | Aggressive, often 30-60 °C/min |
| Vibration level | A fraction of the vibration operating limit |
| Cycles | Few; enough to precipitate, not to consume life |
| Functional test | During and after stress, not only after |

The critical constraint is that HASS must not consume meaningful life from good units. That is verified with a **Safety of Screen** study: repeat the screen on the same units many times and confirm they still pass. If units start failing after twenty screens, the profile is too aggressive for a product expected to survive years in the field.

## Recording HASS

HASS is a normal pass/fail production test, so it has limits like any other:

```python filename="phases/hass_functional.py"
def hass_functional(measurements, chamber, dut):
    measurements.chamber_temperature = chamber.read_temperature()
    measurements.output_voltage = dut.output_voltage()
    measurements.self_test_passed = dut.self_test_passes()
```

```yaml filename="procedure.yaml"
name: HASS Screen
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: chamber
    python: plugs.chamber:Chamber
  - name: dut
    python: plugs.dut:Dut

main:
  - name: HASS Functional
    python: phases.hass_functional
    measurements:
      - name: Chamber Temperature
        unit: "°C"
        description: Verify the screen was actually applied
        validators:
          - operator: ">="
            expected_value: -38.0
          - operator: "<="
            expected_value: 82.0
      - name: Output Voltage
        unit: V
        description: Same limits as the ambient functional test
        validators:
          - operator: ">="
            expected_value: 4.75
          - operator: "<="
            expected_value: 5.25
      - name: Self Test Passed
        validators:
          - operator: "=="
            expected_value: true
```

Run it with `tofupilot run ./procedure.yaml --upload`.

Two deliberate choices:

**The chamber temperature is validated.** A screen that was never actually applied, because the chamber did not reach setpoint, produces units that passed a test that did not happen. Verifying the stress was delivered is part of the test.

**The functional limits match the ambient test.** That makes drift across the screen visible as a comparison against the same unit's earlier run, rather than two unrelated numbers.

## HALT and HASS in Context

| Method | Phase | Units | Purpose |
|--------|-------|-------|---------|
| **HALT** | Design | A few, destroyed | Find design margin |
| **HASS** | Production | All or a large sample | Screen out latent defects |
| [**ORT**](/guides/what-is-ort-with-tofupilot) | Ongoing production | Small continuous sample | Detect reliability drift over time |
| [**Burn-in**](/guides/burn-in-testing-for-electronics-a-complete-guide) | Production | All or a sample | Precipitate infant mortality by duration |

HASS and burn-in overlap in intent. HASS is generally faster and more aggressive, using thermal cycling and vibration; burn-in is longer and gentler, usually powered soak at elevated temperature.

## When HASS Is Not Worth It

HASS requires a chamber, a fixture that survives the profile, the HALT data to design it, and a Safety of Screen study. For low-volume or low-criticality products, that investment may exceed the cost of the field failures it prevents. The honest calculation is the expected cost of escapes against the cost of screening.

For high-reliability products, the calculation almost always favours screening, which is why it is standard in aerospace, medical and automotive.

## If You Already Run OpenHTF

```python filename="hass.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("chamber_temperature_C")
    .in_range(-38.0, 82.0)
    .with_units(units.DEGREE_CELSIUS),
    htf.Measurement("output_voltage_V")
    .in_range(4.75, 5.25)
    .with_units(units.VOLT),
)
def phase_hass(test):
    test.measurements.chamber_temperature_C = 81.2
    test.measurements.output_voltage_V = 5.02


test = htf.Test(phase_hass, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Designing HASS without HALT data | Either damaging good units or screening nothing | Run HALT first; derive the profile from its limits |
| Stopping HALT at the first failure | No destruct limit, no margin figure | Continue to destruct on every stress type |
| Putting pass/fail limits on HALT steps | The data becomes a verdict instead of a limit | Record HALT measurements without validators |
| No Safety of Screen study | HASS consumes field life from good units | Repeat the screen and confirm survival |
| Not verifying the chamber reached setpoint | Units pass a screen that never happened | Validate the chamber temperature |
| Different functional limits than the ambient test | Drift across the screen is invisible | Reuse the ambient measurement names and limits |

### What Is Test Data Management
URL: https://www.tofupilot.com/guides/what-is-test-data-management-with-tofupilot
Learn what test data management is, why spreadsheets fail at scale, and how TofuPilot structures your electronics test data automatically.

Test data management (TDM) is the practice of collecting, storing, and analyzing every measurement from your production test stations in a single structured system. Without it, test results end up scattered across CSVs, local databases, and shared drives, and you lose the ability to trace failures, track yields, or prove compliance.

## The Problem with Spreadsheets and CSVs

Most hardware teams start with CSV exports or shared spreadsheets. This works for a few units on one station. It breaks down fast.

Common failure modes:

- **No standard schema.** Each station logs different columns. Merging data requires manual cleanup every time.
- **No unit traceability.** You can't look up the full test history for a single serial number without searching multiple files.
- **No real-time visibility.** By the time someone opens a spreadsheet, the data is stale. Yield drops go unnoticed for hours or days.
- **No audit trail.** Overwritten cells, deleted rows, renamed files. Regulators and customers won't accept this as evidence.

At 100 units per day across two stations, you're already generating thousands of rows per week. At 10,000 units per day, spreadsheets aren't just inconvenient. They're a liability.

## What Structured Test Data Looks Like

A proper TDM system stores every test run with a consistent structure:

| Field | Example |
|-------|---------|
| Serial number | `SN-2026-00421` |
| Part number | `PCB-RevC-Main` |
| Station | `Station-3-Final` |
| Outcome | PASS / FAIL |
| Measurements | Voltage: 3.31V (limit: 3.0–3.6V) |
| Timestamp | 2026-03-12T14:32:00Z |
| Operator | Line 2, Shift A |
| Attachments | Waveform capture, log file |

Every measurement carries its name, value, unit, and limits. Every run links to a unit, a station, and a point in time. This structure is what makes FPY calculations, Cpk analysis, and failure Pareto charts possible.

## How TofuPilot Captures Test Data Automatically

If you're using OpenHTF, TofuPilot plugs directly into your test script. You don't need to build a logging layer or manage database connections. Write your test, add the TofuPilot output, and every run uploads automatically.

```python filename="voltage_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("output_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=3.0, maximum=3.6),
    htf.Measurement("current_draw")
    .with_units(units.AMPERE)
    .in_range(minimum=0.1, maximum=0.5),
)
def test_power_rail(test):
    test.measurements.output_voltage = 3.31
    test.measurements.current_draw = 0.25

def main():
    test = htf.Test(test_power_rail)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-2026-00421")

if __name__ == "__main__":
    main()
```

Each run produces a structured record with the serial number, measurements, limits, units, and pass/fail outcome. TofuPilot stores it all and links it to the unit's full history.

## What You Get from Structured Data

Once your test data flows into TofuPilot, the analytics dashboard gives you immediate visibility without writing a single query:

- **First pass yield** trends per station, per product, per time window
- **Cpk and control charts** for any measurement, so you catch drift before it causes failures
- **Failure Pareto** showing which test phases fail most often, ranked by frequency
- **Unit history** for any serial number, showing every test run from EVT through PVT and production
- **Measurement histograms** showing distribution against limits

You don't need to build dashboards or write scripts to compute these metrics. The structured data makes them automatic.

## TDM vs. MES vs. Custom Databases

TDM isn't an MES. A manufacturing execution system tracks work orders, routing, and inventory. TDM focuses specifically on test results, measurements, and quality metrics.

Building a custom database works until you need analytics, access control, or integrations. Most teams that start with PostgreSQL and a few Python scripts spend months maintaining infrastructure instead of improving their tests.

TofuPilot sits between these approaches. It handles the storage, structure, and analytics so your test engineers can focus on writing better tests.

### Pareto Analysis for Test Failures
URL: https://www.tofupilot.com/guides/how-to-use-pareto-analysis-for-test-failures-with-tofupilot
Learn how the 80/20 Pareto principle applies to test failures, how to write tests that produce actionable failure data, and how to use TofuPilot's Pareto chart.

The Pareto principle says 80% of your test failures come from 20% of your failure modes. If you fix the right two or three issues, most of your yield problems disappear. But you need structured failure data to find those issues, and that starts with how you write your tests.

## What Pareto Analysis Tells You

A Pareto chart ranks failure modes by frequency, highest to lowest, with a cumulative line showing how much of total failures each mode accounts for. It answers one question: where should you spend your time?

Without a Pareto chart, teams chase whatever failure they saw most recently. With one, you can see that "clock frequency out of range" causes 40% of all failures while "communication timeout" causes 3%. Fixing the clock issue first gives you 13x more impact.

## Writing Tests with Clear Failure Modes

Pareto analysis only works if your test phases map to distinct, diagnosable failure modes. A single phase that checks 15 things produces one failure label when any of them breaks. You can't tell which sub-check caused it.

Split your test into phases that each cover one functional area.

```python filename="multi_phase_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("input_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=4.75, maximum=5.25),
    htf.Measurement("inrush_current")
    .in_range(maximum=500),
)
def test_power_input(test):
    test.measurements.input_voltage = 5.02
    test.measurements.inrush_current = 320

@htf.measures(
    htf.Measurement("rail_3v3_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=3.135, maximum=3.465),
    htf.Measurement("rail_3v3_ripple")
    .in_range(maximum=50),
)
def test_3v3_regulator(test):
    test.measurements.rail_3v3_voltage = 3.28
    test.measurements.rail_3v3_ripple = 22

@htf.measures(
    htf.Measurement("clock_frequency")
    .in_range(minimum=7.99, maximum=8.01),
    htf.Measurement("clock_jitter")
    .in_range(maximum=100),
)
def test_clock(test):
    test.measurements.clock_frequency = 8.003
    test.measurements.clock_jitter = 45

@htf.measures(
    htf.Measurement("spi_loopback_pass"),
    htf.Measurement("spi_throughput")
    .in_range(minimum=900),
)
def test_spi_interface(test):
    test.measurements.spi_loopback_pass = True
    test.measurements.spi_throughput = 980

@htf.measures(
    htf.Measurement("adc_offset_error")
    .in_range(minimum=-2, maximum=2),
    htf.Measurement("adc_gain_error_pct")
    .in_range(minimum=-0.5, maximum=0.5),
    htf.Measurement("adc_snr")
    .in_range(minimum=60),
)
def test_adc_accuracy(test):
    test.measurements.adc_offset_error = 0.8
    test.measurements.adc_gain_error_pct = -0.12
    test.measurements.adc_snr = 67.3

def main():
    test = htf.Test(
        test_power_input,
        test_3v3_regulator,
        test_clock,
        test_spi_interface,
        test_adc_accuracy,
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-2026-01580")

if __name__ == "__main__":
    main()
```

Each phase covers one functional block: power input, voltage regulation, clock, SPI communication, and ADC accuracy. When a unit fails, the failure is tagged to a specific phase and measurement. TofuPilot uses this structure to build the Pareto chart.

## Reading TofuPilot's Failure Pareto Chart

TofuPilot's analytics dashboard includes a failure Pareto chart that updates in real time as test data flows in. Here's how to use it.

**Select your scope.** Filter by product, station, or time range. A Pareto chart for all products combined isn't useful because different products have different failure profiles.

**Read the bars.** Each bar represents a failure mode (a test phase or specific measurement that failed). The tallest bar on the left is your biggest problem.

**Follow the cumulative line.** The line shows what percentage of total failures you've covered as you move left to right. If the first two bars reach 70% on the cumulative line, fixing those two issues eliminates 70% of your failures.

**Compare over time.** After fixing the top failure mode, check the Pareto chart again. The distribution will shift. What was previously the second-biggest issue might now be the top one, or a previously hidden issue might surface.

## Turning Pareto Data into Action

A Pareto chart gives you priorities. Turning those priorities into fixes requires a systematic approach.

### For the Top Failure Mode

1. Pull up the measurement histogram in TofuPilot for the failing measurement. Check whether failures cluster at one limit or spread across the range.
2. Check the Cpk. If Cpk is below 1.0, your process isn't capable of meeting the test limits consistently. You either need to improve the process or review whether the limits are too tight.
3. Look at the unit history for several failed serial numbers. Do they fail once and pass on retest (fixture issue), or do they fail consistently (real defect)?
4. Check for station correlation. If failures concentrate on one station, the problem is likely the fixture or test environment, not the product.

### For Recurring Low-Level Failures

Some failure modes sit at 2-5% individually but collectively account for 20-30% of failures. These are often fixture wear, environmental drift, or operator handling issues. Address them as a group by improving fixture maintenance schedules and adding fixture validation phases to your tests.

## What Good Looks Like

A healthy Pareto chart has no single failure mode dominating above 15-20%. The bars taper gradually from left to right, indicating no single systemic issue.

If your chart shows one bar towering over the rest, you have a known problem that hasn't been addressed. If the chart is nearly flat with many small bars, your failures are distributed and likely random, which means your process is well-controlled and further improvement requires tightening tolerances or upgrading components.

Review the Pareto chart weekly during DVT and PVT. In production, a monthly review catches new trends before they become costly.

### OpenHTF vs pytest for Hardware Testing
URL: https://www.tofupilot.com/guides/openhtf-vs-pytest-for-hardware-testing-which-should-you-choose
Compare OpenHTF and pytest for hardware test, and see where the TofuPilot Framework fits, with real code and guidance on when to use each.

OpenHTF and pytest are the two most common Python frameworks engineers weigh up for hardware test automation. They solve different problems. OpenHTF was built by Google for manufacturing test. pytest was built for software testing and adapted by hardware teams. A third option, the TofuPilot Framework, is also purpose-built for hardware and covered below. This guide compares all three with real code so you can pick the right one.

## Feature Comparison

| Feature | OpenHTF | pytest | TofuPilot Framework |
|---------|---------|--------|--------------------|
| **Designed for** | Manufacturing/production test | Software testing (adapted for hardware) | Manufacturing/production test |
| **Test structure** | Phases (ordered sequence) | Functions (unordered by default) | YAML procedure + Python phases |
| **Measurements** | Built-in: name, value, limits, units | Manual: assert statements or custom fixtures | Built-in: declared in YAML with validators |
| **Serial number input** | Built-in (operator prompt) | Manual implementation | Built-in (`unit.serial_number`) |
| **Operator UI** | Built-in web UI | None (third-party or custom) | Declarative, generated from the procedure |
| **Instrument lifecycle** | Plugs (auto setup/teardown) | Fixtures (similar, more flexible) | Plugs (plain Python classes) |
| **Attachments** | Built-in (files, images, logs) | Manual (save to disk or custom plugin) | Built-in |
| **Parallel phases** | No (sequential) | Via pytest-xdist | Native (Rust engine) |
| **Parallel DUT testing** | Limited (single test executor) | Native (pytest-xdist) | Native (multi-slot fixtures) |
| **Waveform aggregations** | No | Manual | Built-in (mean, min, max, std) |
| **Per-axis validators** | No | Manual | Built-in |
| **Deployment to stations** | Your problem | Your problem | Git push, instant rollback |
| **Community size** | Small (~640 GitHub stars) | Massive (11K+ stars, huge ecosystem) | Newer, smaller |
| **Plugin ecosystem** | Minimal | Thousands of plugins | Plugs, growing |
| **Learning curve** | Steeper (less documentation) | Gentler (extensive documentation) | Low (YAML + plain Python) |
| **License** | Apache 2.0 | MIT | MIT |
| **TofuPilot integration** | Native on the CLI, or `upload()` callback | Via Python SDK | Native |

## The Same Test in Each Framework

Here's a functional test that verifies a PCBA's power rail and firmware version, written in all three.

### OpenHTF Version

```python filename="comparison/openhtf_test.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class DutPlug(BasePlug):
    """Manage DUT connection lifecycle."""

    def setUp(self):
        self.connected = True  # Replace with real connection

    def read_voltage(self) -> float:
        return 3.31  # Replace with instrument read

    def query_firmware(self) -> str:
        return "2.1.0"  # Replace with DUT query

    def tearDown(self):
        self.connected = False


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT)
    .doc("3.3V supply rail voltage"),
)
@htf.PhaseOptions(timeout_s=10)
@htf.plug(dut=DutPlug)
def test_power(test, dut):
    test.measurements.rail_3v3 = dut.read_voltage()


@htf.measures(
    htf.Measurement("firmware_version")
    .equals("2.1.0")
    .doc("Expected firmware version string"),
)
@htf.PhaseOptions(timeout_s=5)
@htf.plug(dut=DutPlug)
def test_firmware(test, dut):
    test.measurements.firmware_version = dut.query_firmware()


def main():
    test = htf.Test(
        test_power,
        test_firmware,
        procedure_id="FCT-001",
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial number: "))


if __name__ == "__main__":
    main()
```

Install the OpenHTF extra with `pip install "tofupilot[openhtf]"`. `upload()` reads `TOFUPILOT_API_KEY` from the environment. Alternatively, run the same suite with `tofupilot run` and the CLI streams and uploads it with no callback at all.

### pytest Version

```python filename="comparison/pytest_test.py"
import pytest
from tofupilot import TofuPilotClient


@pytest.fixture(scope="session")
def tofupilot():
    """Initialize TofuPilot client for the test session."""
    return TofuPilotClient()


@pytest.fixture(scope="session")
def serial_number():
    """Prompt operator for serial number before tests run."""
    return input("Scan serial number: ")


@pytest.fixture
def dut():
    """Manage DUT connection. Equivalent to an OpenHTF Plug."""
    # Setup
    connection = {"connected": True}  # Replace with real connection
    yield connection
    # Teardown
    connection["connected"] = False


def read_voltage() -> float:
    """Read voltage from instrument."""
    return 3.31  # Replace with real read


def query_firmware() -> str:
    """Query firmware version from DUT."""
    return "2.1.0"  # Replace with real query


def test_power_rail(dut):
    """Verify 3.3V supply rail is within spec."""
    voltage = read_voltage()
    assert 3.2 <= voltage <= 3.4, f"3.3V rail out of range: {voltage}V"


def test_firmware_version(dut):
    """Verify firmware version matches expected."""
    version = query_firmware()
    assert version == "2.1.0", f"Unexpected firmware: {version}"
```

### TofuPilot Framework Version

The procedure YAML declares the sequence and the limits. Phases are plain functions with no decorators, and the plug is a plain class.

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dut
    python: plugs.dut:Dut

main:
  - name: Power
    python: phases.power
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
  - name: Firmware
    python: phases.firmware
    measurements:
      - name: Firmware Version
        validators:
          - operator: "=="
            expected_value: "2.1.0"
```

```python filename="phases/power.py"
def power(measurements, dut):
    measurements.rail_3v3 = dut.read_voltage()
```

```python filename="phases/firmware.py"
def firmware(measurements, dut):
    measurements.firmware_version = dut.query_firmware()
```

```python filename="plugs/dut.py"
class Dut:
    def read_voltage(self) -> float:
        return 3.31  # Replace with an instrument read

    def query_firmware(self) -> str:
        return "2.1.0"  # Replace with a DUT query
```

Run it with `tofupilot run ./procedure.yaml`. The measurement key is generated from the name in snake_case (`Rail 3V3` becomes `rail_3v3`), and plugs are injected into phases by parameter name. Because the two phases are independent, the engine runs them in parallel.

### Key Differences in the Code

| Aspect | OpenHTF | pytest | TofuPilot Framework |
|--------|---------|--------|--------------------|
| **Measurements** | Declarative with `@htf.measures`. Name, limits, units defined upfront. | Implicit via `assert`. No structured metadata. | Declared in YAML, stored as data. |
| **Serial number** | Built into `test.execute()`. | Custom fixture or input call. | `unit.serial_number` in the procedure. |
| **DUT lifecycle** | Plug class with `setUp`/`tearDown`. Injected via `@htf.plug()` decorator. | Fixture with `yield`. Manually passed. | Plain class, injected by parameter name. |
| **Limits** | `.in_range(3.2, 3.4)` stored as structured data. | `assert 3.2 <= v <= 3.4` is code, not data. | `validators` in YAML, separate from the logic. |
| **Output** | Structured protobuf with all metadata. | Pass/fail only (unless you add custom reporting). | Streamed to the dashboard by the CLI. |
| **Timeouts** | `@htf.PhaseOptions(timeout_s=10)` | `@pytest.mark.timeout(10)` (requires plugin) | `timeout` on the phase in YAML. |

## When to Use OpenHTF

OpenHTF is the better choice when:

- **You already run it.** Existing OpenHTF suites run natively on the TofuPilot CLI with no rewrite. If it works, the migration cost isn't worth it.
- **You want structured test data in one pure-Python paradigm.** Every measurement carries its name, value, limits, units, and pass/fail status, all declared in Python decorators with no separate YAML layer.
- **Maturity matters most.** Years of production use at scale and a larger body of public examples and patterns than newer options.
- **Apache 2.0 fits your legal review.** Its explicit patent grant can matter in some procurement contexts.

## When to Use pytest

pytest is the better choice when:

- **You're doing R&D or validation testing.** pytest's flexibility shines when test requirements change frequently. No strict phase ordering, easy to skip or select tests, rich assertion messages.
- **Your team knows pytest.** Most Python developers already know it. Both hardware frameworks have a learning curve.
- **You want a plugin ecosystem.** pytest-timeout, pytest-repeat, pytest-html, pytest-cov. Thousands of plugins for every need.
- **You're testing firmware/software on hardware.** If the "test" is really a software test that happens to run on hardware (flashing firmware, running integration tests on a dev board), pytest is the natural fit.

## When to Use the TofuPilot Framework

The TofuPilot Framework is the better choice when:

- **Your line is throughput-bound.** Parallel phases and multi-slot fixtures test several boards at once on one fixture and cut cycle time. OpenHTF executes sequentially.
- **Operators need guided workflows.** Text inputs, checklists, images, sliders and progress bars are declared in the procedure and render on the station with no frontend work.
- **You validate waveforms or sweeps.** Built-in aggregations and per-axis validators check curve shape, not only endpoints.
- **You want the test plan separate from the logic.** The YAML procedure is readable and diffable independently of the Python phases.
- **You are setting up new stations.** Deploy-on-push, instant rollback, real-time streaming and offline queueing are built into the CLI.

## Using Them Together

Many teams use more than one. pytest for firmware validation and CI, a hardware framework for production functional test:

| Test Stage | Framework | Why |
|-----------|-----------|-----|
| Firmware CI | pytest | Runs in CI pipeline, parallel execution, software-style testing |
| EVT/DVT validation | pytest | Flexible, exploratory, requirements change frequently |
| PVT/Production FCT | TofuPilot Framework or OpenHTF | Structured measurements, operator UI, production data logging |
| Incoming inspection | TofuPilot Framework or OpenHTF | Repeatable, operator-driven, needs traceability |

All three work with TofuPilot. The TofuPilot Framework and OpenHTF both run natively on the CLI; pytest works through the Python SDK.

## TofuPilot Integration Comparison

### TofuPilot Framework: nothing to add

```bash filename="comparison/framework_run.sh"
tofupilot run ./procedure.yaml --upload
```

Phases, measurements, logs, attachments and the operator UI stream to the dashboard with no code.

### OpenHTF: native on the CLI, or two lines

```python filename="comparison/tofupilot_openhtf.py"
from tofupilot.openhtf import upload

# Wrap your test execution
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Serial: "))
# Measurements, limits, units, phases, attachments
# all logged automatically.
```

Or skip the callback entirely and run the suite with `tofupilot run`.

### pytest: Python SDK

```python filename="comparison/tofupilot_pytest.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# After collecting results, create a run
client.create_run(
    procedure_id="FCT-001",
    unit_under_test={"serial_number": "SN-001", "part_number": "PCBA-100"},
    run_passed=True,
    steps=[
        {
            "name": "test_power_rail",
            "step_passed": True,
            "measurements": [
                {
                    "name": "rail_3v3",
                    "value": 3.31,
                    "unit": "V",
                    "lower_limit": 3.2,
                    "upper_limit": 3.4,
                },
            ],
        },
    ],
)
```

More code, but you get the same analytics in TofuPilot: FPY, Cpk, control charts, failure Pareto, traceability.

## Decision Matrix

| Question | If yes → |
|----------|----------|
| Are you doing firmware CI or R&D validation? | pytest |
| Do you need a rich plugin ecosystem? | pytest |
| Do you already run OpenHTF in production? | OpenHTF, keep it |
| Do you want one pure-Python paradigm, no YAML? | OpenHTF |
| Is your line throughput-bound, several boards per fixture? | TofuPilot Framework |
| Do operators need guided workflows on the station? | TofuPilot Framework |
| Do you validate waveform or sweep shape? | TofuPilot Framework |
| Are you setting up new stations across sites? | TofuPilot Framework |

If you are starting production test from scratch, the TofuPilot Framework is the shortest path: `tofupilot run ./procedure.yaml` executes on your machine without an account. If you are doing software-style testing on hardware, pytest. If you already have OpenHTF in production, keep it. All three upload to the same dashboard.

For a detailed comparison of the two purpose-built options, see [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf).

### Environmental Testing for Hardware
URL: https://www.tofupilot.com/guides/environmental-testing-with-tofupilot
Learn how to log thermal cycling, humidity, HALT, and environmental screening test results in TofuPilot for reliability tracking.

Products fail in the field because of temperature, humidity, vibration, and altitude. Environmental testing simulates these conditions in the lab. TofuPilot tracks every environmental test result so you can build a reliability picture across your entire product lifecycle.

## Types of Environmental Tests

| Test | What it simulates | Standards |
|------|------------------|-----------|
| Thermal cycling | Temperature extremes during shipping and operation | IEC 60068-2-14, MIL-STD-810 |
| Thermal shock | Rapid temperature transitions | IEC 60068-2-14 |
| Humidity testing | Moisture exposure, condensation | IEC 60068-2-78 |
| HALT (Highly Accelerated Life Test) | Find design weaknesses fast | No formal standard |
| HASS (Highly Accelerated Stress Screen) | Production screening | No formal standard |
| Vibration | Mechanical stress during transport and operation | IEC 60068-2-6, IEC 60068-2-64 |
| Altitude | Low pressure environments | IEC 60068-2-13 |
| Salt spray | Corrosion resistance | IEC 60068-2-11 |

## Logging Thermal Cycling Results

A thermal cycling test runs a product through temperature extremes for hundreds of cycles. After each cycle (or batch of cycles), perform a functional check and log results.

```python filename="thermal_cycling_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

def log_thermal_cycle_check(serial, cycle_number, temp_profile, functional_results):
    """Log post-cycle functional check results."""
    measurements = [
        {"name": "cycle_number", "value": cycle_number, "unit": "cycles"},
        {"name": "temp_high_c", "value": temp_profile["high"], "unit": "°C"},
        {"name": "temp_low_c", "value": temp_profile["low"], "unit": "°C"},
        {"name": "dwell_time_min", "value": temp_profile["dwell"], "unit": "min"},
    ]

    # Add functional measurements taken after thermal exposure
    for name, result in functional_results.items():
        measurements.append({
            "name": name,
            "value": result["value"],
            "unit": result["unit"],
            "limit_low": result.get("limit_low"),
            "limit_high": result.get("limit_high"),
        })

    all_pass = all(
        r.get("limit_low", float("-inf")) <= r["value"] <= r.get("limit_high", float("inf"))
        for r in functional_results.values()
    )

    client.create_run(
        procedure_id="THERMAL-CYCLING-500",
        unit_under_test={"serial_number": serial},
        run_passed=all_pass,
        steps=[{
            "name": f"Post-Cycle {cycle_number} Functional Check",
            "step_type": "measurement",
            "status": all_pass,
            "measurements": measurements,
        }],
    )

# Example: check after every 50 cycles
for cycle in range(50, 501, 50):
    functional = {
        "vcc_3v3": {"value": measure_voltage(), "unit": "V", "limit_low": 3.25, "limit_high": 3.35},
        "current_ma": {"value": measure_current() * 1000, "unit": "mA", "limit_low": 30, "limit_high": 60},
        "comm_check": {"value": 1 if uart_ok() else 0, "unit": "bool", "limit_low": 1},
    }
    log_thermal_cycle_check(
        serial="EVT-UNIT-003",
        cycle_number=cycle,
        temp_profile={"high": 85, "low": -40, "dwell": 15},
        functional_results=functional,
    )
```

## Tracking Degradation Over Cycles

The key question in environmental testing: "At what point does the product start degrading?"

TofuPilot's measurement trending shows how each parameter changes across thermal cycles:

- **Cycle 50**: vcc_3v3 = 3.31V (nominal)
- **Cycle 200**: vcc_3v3 = 3.30V (still nominal)
- **Cycle 350**: vcc_3v3 = 3.28V (drifting)
- **Cycle 450**: vcc_3v3 = 3.26V (approaching limit)
- **Cycle 500**: vcc_3v3 = 3.24V (below limit, FAIL)

This degradation curve tells you the product survives 400+ cycles before the 3.3V rail drifts out of spec. If your spec requires 500 cycles, you need a design change.

## HALT Testing

HALT finds design weaknesses by pushing products beyond their specifications. There are no pass/fail limits. The goal is to find the breaking point.

```python filename="halt_test.py"
# HALT step stress results
halt_steps = [
    {"stress": "Cold Step", "temp_c": -40, "functional": True},
    {"stress": "Cold Step", "temp_c": -50, "functional": True},
    {"stress": "Cold Step", "temp_c": -60, "functional": False},  # Lower operating limit found
    {"stress": "Hot Step", "temp_c": 85, "functional": True},
    {"stress": "Hot Step", "temp_c": 95, "functional": True},
    {"stress": "Hot Step", "temp_c": 105, "functional": False},  # Upper operating limit found
    {"stress": "Vibration", "grms": 10, "functional": True},
    {"stress": "Vibration", "grms": 20, "functional": True},
    {"stress": "Vibration", "grms": 30, "functional": False},  # Vibration limit found
]

steps = []
for h in halt_steps:
    steps.append({
        "name": f"{h['stress']} - {'temp_c' in h and f\"{h['temp_c']}°C\" or f\"{h['grms']}Grms\"}",
        "step_type": "measurement",
        "status": h["functional"],
        "measurements": [{
            "name": "functional_after_stress",
            "value": 1 if h["functional"] else 0,
            "unit": "pass/fail",
        }],
    })

client.create_run(
    procedure_id="HALT-STEP-STRESS",
    unit_under_test={"serial_number": "EVT-UNIT-001"},
    run_passed=True,  # HALT always "passes" - it's about finding limits
    steps=steps,
)
```

## Environmental Test Matrix

Track which environmental tests have been completed across your product variants.

| Product variant | Thermal cycle | Humidity | Vibration | HALT | ESD |
|----------------|---------------|----------|-----------|------|-----|
| Rev A | 500 cycles ✓ | 85/85 ✓ | 10G ✓ | Done ✓ | 8kV ✓ |
| Rev B | 250/500 | Pending | Pending | Done ✓ | 8kV ✓ |
| Rev C (new layout) | Not started | Not started | Not started | Pending | Not started |

TofuPilot tracks completion automatically. If `THERMAL-CYCLING-500` has a passing run for "Rev A," it shows as complete. No spreadsheet maintenance needed.

## Production Screening (HASS/ESS)

Environmental stress screening in production catches infant mortality failures. Run every unit through a short thermal cycle and vibration profile, then functional test.

```python filename="production_screen.py"
# Quick production screen: 10 thermal cycles + vibration
client.create_run(
    procedure_id="ESS-PRODUCTION-SCREEN",
    unit_under_test={"serial_number": "PROD-UNIT-4521"},
    run_passed=True,
    steps=[
        {
            "name": "Post-Thermal Functional",
            "step_type": "measurement",
            "status": True,
            "measurements": [
                {"name": "vcc_3v3", "value": 3.31, "unit": "V", "limit_low": 3.25, "limit_high": 3.35},
                {"name": "current_ma", "value": 44, "unit": "mA", "limit_low": 30, "limit_high": 60},
            ],
        },
        {
            "name": "Post-Vibration Functional",
            "step_type": "measurement",
            "status": True,
            "measurements": [
                {"name": "vcc_3v3", "value": 3.30, "unit": "V", "limit_low": 3.25, "limit_high": 3.35},
                {"name": "current_ma", "value": 45, "unit": "mA", "limit_low": 30, "limit_high": 60},
            ],
        },
    ],
)
```

Track ESS yield separately from functional test yield. If ESS catches 0.5% of units that passed functional test, it's paying for itself by preventing field failures.


### Getting Started with PyVISA
URL: https://www.tofupilot.com/guides/getting-started-with-pyvisa-control-test-instruments-from-python
Control test instruments from Python with PyVISA: discovery, connection, SCPI basics, error handling, and wiring a DMM into a test procedure.

PyVISA lets you control test instruments (multimeters, oscilloscopes, power supplies, signal generators) from Python using the same VISA protocol that LabVIEW and TestStand use. If your instrument has a USB, GPIB, Ethernet, or serial interface, you can talk to it with PyVISA. This guide covers setup, connection, SCPI commands, and turning instrument reads into recorded measurements.

## Prerequisites

- Python 3.8+
- A test instrument with USB-TMC, GPIB, Ethernet (LXI), or serial interface
- Optional: NI-VISA runtime (for GPIB and some USB instruments)

## Installation

```bash filename="install.sh"
pip install pyvisa pyvisa-py
```

Two backends are available:

| Backend | Install | When to Use |
|---------|---------|-------------|
| **pyvisa-py** | `pip install pyvisa-py` | Pure Python. Works on Linux/macOS/Windows. Supports USB-TMC, Ethernet (TCP/IP), serial. No NI software needed. |
| **NI-VISA** | Install from ni.com | Required for GPIB. Also supports USB-TMC and Ethernet. Windows and Linux only. |

Start with pyvisa-py. Switch to NI-VISA only if you need GPIB or encounter compatibility issues.

## Finding Your Instruments

Every VISA instrument has a resource string that identifies it. PyVISA can discover connected instruments automatically.

```python filename="pyvisa_setup/discover.py"
import pyvisa

rm = pyvisa.ResourceManager("@py")  # Use pyvisa-py backend
# rm = pyvisa.ResourceManager()     # Use NI-VISA backend

# List all connected instruments
resources = rm.list_resources()
print(f"Found {len(resources)} instrument(s):")
for r in resources:
    print(f"  {r}")
```

Common resource string formats:

| Interface | Resource String | Example |
|-----------|----------------|---------|
| USB-TMC | `USB0::VID::PID::SERIAL::INSTR` | `USB0::0x2A8D::0x1301::MY59001234::INSTR` |
| Ethernet (LXI) | `TCPIP::IP::INSTR` | `TCPIP::192.168.1.100::INSTR` |
| GPIB | `GPIB0::ADDR::INSTR` | `GPIB0::22::INSTR` |
| Serial | `ASRL/dev/ttyUSB0::INSTR` | `ASRL/dev/ttyUSB0::INSTR` |

## Connecting to an Instrument

Once you have the resource string, open a connection and verify identity with the standard `*IDN?` query.

```python filename="pyvisa_setup/connect.py"
import pyvisa

rm = pyvisa.ResourceManager("@py")
dmm = rm.open_resource("TCPIP::192.168.1.100::INSTR")

# Set timeout (milliseconds)
dmm.timeout = 5000

# Query instrument identity
idn = dmm.query("*IDN?")
print(f"Connected to: {idn.strip()}")
# Output: Keysight Technologies,34461A,MY59001234,A.03.01
```

The `query()` method sends a command and reads the response. For commands that don't return data, use `write()`.

## SCPI Command Basics

SCPI (Standard Commands for Programmable Instruments) is the language most modern instruments speak. Commands follow a tree structure.

### Reading a DC Voltage

```python filename="pyvisa_setup/measure_voltage.py"
import pyvisa

rm = pyvisa.ResourceManager("@py")
dmm = rm.open_resource("TCPIP::192.168.1.100::INSTR")
dmm.timeout = 5000

# Configure for DC voltage measurement, auto-range
dmm.write(":CONF:VOLT:DC AUTO")

# Trigger a measurement and read the result
dmm.write(":INIT")
voltage = float(dmm.query(":FETCH?"))

print(f"Voltage: {voltage:.4f} V")

dmm.close()
rm.close()
```

### Common SCPI Commands

| Command | Purpose | Example |
|---------|---------|---------|
| `*IDN?` | Identify instrument | Returns manufacturer, model, serial, firmware |
| `*RST` | Reset to factory defaults | Good practice at test start |
| `*CLS` | Clear error queue | Clear any previous errors |
| `*OPC?` | Operation complete query | Returns "1" when last command finishes |
| `:CONF:VOLT:DC` | Configure DC voltage | `:CONF:VOLT:DC 10` for 10V range |
| `:CONF:CURR:DC` | Configure DC current | `:CONF:CURR:DC AUTO` for auto-range |
| `:CONF:RES` | Configure resistance | `:CONF:RES AUTO` |
| `:MEAS:VOLT:DC?` | Measure DC voltage (configure + trigger + read) | Returns float |
| `:INIT` | Trigger measurement | Use with `:FETCH?` for separate trigger/read |
| `:FETCH?` | Read last measurement | Returns float |
| `:SYST:ERR?` | Read error queue | Returns error code and message |

The `:MEAS:` shorthand configures, triggers, and reads in one command. Use `:CONF:` + `:INIT` + `:FETCH?` when you need more control over timing.

## Controlling a Power Supply

Power supplies use similar SCPI commands but add output control.

```python filename="pyvisa_setup/power_supply.py"
import pyvisa
import time

rm = pyvisa.ResourceManager("@py")
psu = rm.open_resource("TCPIP::192.168.1.101::INSTR")
psu.timeout = 5000

# Reset to known state
psu.write("*RST")
psu.write("*CLS")

# Configure channel 1: 5V, 500mA current limit
psu.write(":INST:SEL CH1")
psu.write(":VOLT 5.0")
psu.write(":CURR 0.5")

# Enable output
psu.write(":OUTP ON")
time.sleep(0.5)  # Wait for output to stabilize

# Read actual voltage and current
actual_voltage = float(psu.query(":MEAS:VOLT?"))
actual_current = float(psu.query(":MEAS:CURR?"))
print(f"Output: {actual_voltage:.3f}V, {actual_current:.4f}A")

# Disable output
psu.write(":OUTP OFF")

psu.close()
rm.close()
```

## Error Handling

Instruments communicate errors through the SCPI error queue. Always check for errors after a sequence of commands.

```python filename="pyvisa_setup/error_handling.py"
import pyvisa


def check_instrument_errors(instr):
    """Read and report all errors from the instrument error queue."""
    errors = []
    while True:
        err = instr.query(":SYST:ERR?").strip()
        code = int(err.split(",")[0])
        if code == 0:
            break
        errors.append(err)
    return errors


rm = pyvisa.ResourceManager("@py")
dmm = rm.open_resource("TCPIP::192.168.1.100::INSTR")
dmm.timeout = 5000

# Send commands
dmm.write("*RST")
dmm.write(":CONF:VOLT:DC AUTO")
dmm.write(":INIT")
voltage = float(dmm.query(":FETCH?"))

# Check for errors
errors = check_instrument_errors(dmm)
if errors:
    print(f"Instrument errors: {errors}")
else:
    print(f"Voltage: {voltage:.4f} V (no errors)")

dmm.close()
```

## From Script to Test

A script that prints a voltage is a good first step, but a production test needs the value recorded against limits, tied to a serial number, and kept. That means two changes: the instrument session moves into a plug, and the limits move into a procedure.

A plug is a plain Python class that owns the VISA session:

```python filename="plugs/dmm.py"
import pyvisa


class Multimeter:
    """PyVISA multimeter. Opens once, shared across phases."""

    RESOURCE = "TCPIP::192.168.1.100::INSTR"

    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 5000
        self.instr.write("*RST")
        self.instr.write("*CLS")

    def measure_dc_voltage(self, range_v: str = "AUTO") -> float:
        self.instr.write(f":CONF:VOLT:DC {range_v}")
        return float(self.instr.query(":MEAS:VOLT:DC?"))

    def measure_dc_current(self, range_a: str = "AUTO") -> float:
        self.instr.write(f":CONF:CURR:DC {range_a}")
        return float(self.instr.query(":MEAS:CURR:DC?"))

    def measure_resistance(self, range_ohm: str = "AUTO") -> float:
        self.instr.write(f":CONF:RES {range_ohm}")
        return float(self.instr.query(":MEAS:RES?"))

    def identify(self) -> str:
        return self.instr.query("*IDN?").strip()

    def close(self):
        self.instr.close()
```

The phase is a plain function that assigns readings. TofuPilot injects the plug by matching the parameter name:

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.supply_3v3 = dmm.measure_dc_voltage()
    measurements.supply_5v0 = dmm.measure_dc_voltage()
    measurements.idle_current = dmm.measure_dc_current()
```

```python filename="phases/pullups.py"
def pullups(measurements, dmm):
    measurements.pullup_resistance = dmm.measure_resistance()
```

The limits live in the procedure as data:

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Supply 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Supply 5V0
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
      - name: Idle Current
        unit: A
        validators:
          - operator: ">="
            expected_value: 0.01
          - operator: "<="
            expected_value: 0.15

  - name: Pullups
    python: phases.pullups
    depends_on: [power_rails]
    measurements:
      - name: Pullup Resistance
        unit: "Ω"
        description: I2C pullup, expect 4.7k
        validators:
          - operator: ">="
            expected_value: 4500
          - operator: "<="
            expected_value: 5500
```

Run it with `tofupilot run ./procedure.yaml`. Every measurement is stored with its name, value, unit, limits and outcome, which is what makes first pass yield, Cpk and control charts work without extra code.

One caution specific to instruments: independent phases run in parallel, and a single VISA session is not safe to use from two phases at once. Either give each phase its own instrument, or serialise the sharing phases with `depends_on`, as above.

It is also worth recording which instrument produced a reading:

```python filename="phases/record_equipment.py"
def record_equipment(run, dmm):
    run.metadata["dmm_id"] = dmm.identify()
```

## If You Already Run OpenHTF

The same session belongs in a `BasePlug` with `setUp` and `tearDown`:

```python filename="pyvisa_setup/openhtf_test.py"
import pyvisa
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class MultimeterPlug(BasePlug):
    RESOURCE = "TCPIP::192.168.1.100::INSTR"

    def setUp(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 5000
        self.instr.write("*RST")
        self.instr.write("*CLS")

    def measure_dc_voltage(self, range_v: str = "AUTO") -> float:
        self.instr.write(f":CONF:VOLT:DC {range_v}")
        return float(self.instr.query(":MEAS:VOLT:DC?"))

    def tearDown(self):
        self.instr.close()


@htf.measures(
    htf.Measurement("supply_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
@htf.plug(dmm=MultimeterPlug)
def test_power_rails(test, dmm):
    test.measurements.supply_3v3 = dmm.measure_dc_voltage()


test = htf.Test(test_power_rails, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial number: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Troubleshooting

| Problem | Cause | Fix |
|---------|-------|-----|
| `VisaIOError: VI_ERROR_RSRC_NFOUND` | Instrument not found | Check cable, verify IP/USB, run `rm.list_resources()` |
| `VisaIOError: VI_ERROR_TMO` | Timeout | Increase `instr.timeout`, check if instrument is busy |
| `VisaIOError: VI_ERROR_CONN_LOST` | Connection dropped | Check network, retry connection |
| Empty response from `query()` | Instrument didn't respond | Add `time.sleep()` after write, check command syntax |
| USB instrument not detected | Missing driver | Install NI-VISA runtime or check USB-TMC kernel module (Linux) |
| `ValueError: could not convert string to float` | Unexpected response format | Print raw response, check for error messages mixed in |
| Intermittent errors under parallel phases | One session used concurrently | Serialise with `depends_on`, or one instrument per phase |

### Manage Multiple SKUs on a Test Station
URL: https://www.tofupilot.com/guides/how-to-manage-multiple-skus-on-a-test-station-with-tofupilot
Learn how to run multiple product SKUs on a single test station using config-driven phase selection, SKU-specific limits, and per-SKU yield tracking.

One test station, multiple products: configure your OpenHTF test to detect the SKU at runtime, load SKU-specific limits and phases from a config file, and let TofuPilot track yield per part number automatically.

## Why Multi-SKU Stations Matter

Running separate stations per SKU wastes floor space and fixtures. A single station that handles a product family reduces:

- Fixture count (shared bed-of-nails or probe card)
- Operator training (one workflow, not five)
- Maintenance surface (one station to calibrate)

## SKU Detection Methods

| Method | When to Use | Reliability |
|--------|-------------|-------------|
| Barcode scan | Operator-scanned label, no electrical contact needed | High |
| Resistor ID | PCB has a resistor divider encoding SKU | Medium |
| I2C EEPROM | PCB stores part number in onboard memory | High |

### Barcode Scan

```python filename="station/plugs/barcode_plug.py"
import openhtf as htf


class BarcodePlug(htf.plugs.BasePlug):
    """Reads SKU from operator barcode scan."""

    def setUp(self):
        pass

    def scan(self) -> str:
        raw = input("Scan barcode: ").strip()
        if not raw:
            raise ValueError("Empty barcode scan")
        return raw

    def tearDown(self):
        pass
```

### Resistor ID

```python filename="station/plugs/resistor_id_plug.py"
import openhtf as htf

_VOLTAGE_MAP = {
    (0.0, 0.4): "SKU-A",
    (0.4, 0.8): "SKU-B",
    (0.8, 1.2): "SKU-C",
    (1.2, 1.6): "SKU-D",
}


class ResistorIdPlug(htf.plugs.BasePlug):
    """Identifies SKU from resistor divider voltage."""

    def setUp(self):
        pass

    def read_sku(self, adc_voltage: float) -> str:
        for (low, high), sku in _VOLTAGE_MAP.items():
            if low <= adc_voltage < high:
                return sku
        raise ValueError(f"Unrecognised voltage: {adc_voltage:.3f} V")

    def tearDown(self):
        pass
```

## Configuration-Driven Test Selection

Store SKU definitions in a YAML file. This separates limits from test logic.

```yaml filename="station/config/skus.yaml"
SKU-A:
  part_number: "PCB-001-A"
  description: "Standard 5 V variant"
  phases:
    - power_on
    - voltage_check
    - current_check
  limits:
    supply_voltage:
      min: 4.85
      max: 5.15
    supply_current:
      min: 0.080
      max: 0.120

SKU-B:
  part_number: "PCB-001-B"
  description: "Low-power 3.3 V variant"
  phases:
    - power_on
    - voltage_check
    - current_check
    - sleep_current_check
  limits:
    supply_voltage:
      min: 3.2
      max: 3.4
    supply_current:
      min: 0.030
      max: 0.060
    sleep_current:
      min: 0
      max: 0.000050
```

Load the config at station startup:

```python filename="station/config/loader.py"
from pathlib import Path
import yaml


def load_sku_config(path: str = "station/config/skus.yaml") -> dict:
    config_path = Path(path)
    if not config_path.exists():
        raise FileNotFoundError(f"SKU config not found: {config_path}")
    with config_path.open() as f:
        return yaml.safe_load(f)
```

## SKU-Specific Measurement Limits with OpenHTF

Use a phase factory to bake SKU-specific limits into OpenHTF measurements:

```python filename="station/phases/factory.py"
import openhtf as htf
from openhtf.util import units


def make_voltage_phase(min_v: float, max_v: float):
    """Returns a voltage phase with baked-in limits."""

    @htf.measures(
        htf.Measurement("supply_voltage")
        .in_range(minimum=min_v, maximum=max_v)
        .with_units(units.VOLT),
    )
    def voltage_check(test):
        test.measurements.supply_voltage = read_voltage()

    return voltage_check


def make_current_phase(min_a: float, max_a: float):
    """Returns a current phase with baked-in limits."""

    @htf.measures(
        htf.Measurement("supply_current")
        .in_range(minimum=min_a, maximum=max_a)
        .with_units(units.AMPERE),
    )
    def current_check(test):
        test.measurements.supply_current = read_current()

    return current_check
```

## Full Working Example

```python filename="station/main.py"
import openhtf as htf
from tofupilot.openhtf import upload

from station.config.loader import load_sku_config
from station.phases.factory import make_voltage_phase, make_current_phase

SKU_CONFIG = load_sku_config()


def power_on(test):
    """Powers on the DUT and waits for it to stabilise."""
    import time
    time.sleep(0.5)


def run_station():
    # 1. Detect SKU
    sku_id = input("Scan barcode: ").strip()

    if sku_id not in SKU_CONFIG:
        raise ValueError(f"Unknown SKU '{sku_id}'. Available: {list(SKU_CONFIG)}")

    sku = SKU_CONFIG[sku_id]
    limits = sku["limits"]
    part_number = sku["part_number"]

    # 2. Compose phases from config
    phases = [power_on]
    if "supply_voltage" in limits:
        lim = limits["supply_voltage"]
        phases.append(make_voltage_phase(lim["min"], lim["max"]))
    if "supply_current" in limits:
        lim = limits["supply_current"]
        phases.append(make_current_phase(lim["min"], lim["max"]))

    # 3. Build and run test
    test = htf.Test(*phases, test_name=f"Station ({sku_id})")

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial number: "))


if __name__ == "__main__":
    run_station()
```

## Tracking Per-SKU Yield in TofuPilot

TofuPilot groups yield statistics by part number. Each SKU gets its own yield chart without extra configuration.

| Part Number | SKU | Runs | Pass Rate |
|-------------|-----|------|-----------|
| PCB-001-A | SKU-A | 240 | 97.5% |
| PCB-001-B | SKU-B | 180 | 94.4% |

Filter by part number in the TofuPilot dashboard to compare SKU trends side by side.

### Run Parallel Tests on Multiple DUTs
URL: https://www.tofupilot.com/guides/how-to-run-parallel-tests-on-multiple-duts-with-tofupilot
Learn how to run independent OpenHTF test instances in parallel across multiple DUTs using Python multiprocessing, with per-DUT result logging to TofuPilot.

Running tests on multiple devices under test simultaneously cuts cycle time and reduces the cost per unit. This guide shows how to structure a Python test station that runs independent OpenHTF test instances in parallel, each logging results to TofuPilot.

## Why Parallel DUT Testing Matters

Sequential testing on a multi-fixture station leaves test capacity on the table. If your fixture holds four DUTs but your software tests one at a time, you're paying for hardware you're not using.

| Strategy | 4-DUT Fixture, 30s Test | Throughput (units/hour) |
|---|---|---|
| Sequential | 4 x 30s = 120s per cycle | 30 |
| Parallel (4 workers) | ~32s per cycle (overhead) | ~112 |

Parallel testing is most valuable when:
- Test time is dominated by I/O waits (serial reads, power supply settling, RF measurements)
- Your fixture physically holds multiple DUTs simultaneously
- Takt time is a bottleneck in manufacturing

## Architecture Overview

Each DUT gets its own OpenHTF test instance running in a separate process. Results for each DUT are logged independently to TofuPilot.

```text filename="architecture.txt"
Station Controller
├── DUT 0 → Process 0 → OpenHTF Test → TofuPilot (serial: SN-001)
├── DUT 1 → Process 1 → OpenHTF Test → TofuPilot (serial: SN-002)
├── DUT 2 → Process 2 → OpenHTF Test → TofuPilot (serial: SN-003)
└── DUT 3 → Process 3 → OpenHTF Test → TofuPilot (serial: SN-004)
```

## Choosing a Parallelism Model

| Model | Best For | Shared State | GIL Impact |
|---|---|---|---|
| `multiprocessing` | CPU-bound or instrument-isolated DUTs | Requires IPC (Queue, Manager) | None |
| `threading` | I/O-bound tests, shared instrument objects | Direct (with locks) | Applies |
| `ProcessPoolExecutor` | Simple parallel dispatch, result collection | Queue / Manager | None |
| `ThreadPoolExecutor` | Lightweight I/O-bound test phases | Direct (with locks) | Applies |

For most hardware test stations, `multiprocessing` with one process per DUT is the safest choice. It isolates faults, avoids GIL contention, and keeps each DUT's logs and state fully independent.

## Step 1: Define Test Phases

```python filename="station/test_phases.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units


class PowerSupplyPlug(BasePlug):
    """Mock power supply for a single DUT slot."""

    def setUp(self):
        self._voltage = 5.01
        self._current = 0.042

    def measure_voltage(self) -> float:
        return self._voltage

    def measure_current(self) -> float:
        return self._current

    def tearDown(self):
        pass


class SerialPortPlug(BasePlug):
    """Mock serial connection to a single DUT."""

    def setUp(self):
        self._version = "2.4.1"

    def query(self, command: str) -> str:
        return self._version

    def tearDown(self):
        pass


@htf.measures(
    htf.Measurement("supply_voltage")
    .in_range(minimum=4.8, maximum=5.2)
    .with_units(units.VOLT)
)
@htf.plug(psu=PowerSupplyPlug)
def measure_supply_voltage(test, psu):
    test.measurements.supply_voltage = psu.measure_voltage()


@htf.measures(
    htf.Measurement("idle_current")
    .in_range(minimum=0.010, maximum=0.150)
    .with_units(units.AMPERE)
)
@htf.plug(psu=PowerSupplyPlug)
def measure_idle_current(test, psu):
    test.measurements.idle_current = psu.measure_current()


@htf.measures(
    htf.Measurement("firmware_version").equals("2.4.1")
)
@htf.plug(serial=SerialPortPlug)
def check_firmware(test, serial):
    version = serial.query("VERSION?").strip()
    test.measurements.firmware_version = version
```

## Step 2: Create the Worker Process

Each worker creates its own OpenHTF `Test` object and runs it to completion.

```python filename="station/worker.py"
import openhtf as htf
from tofupilot.openhtf import upload

from station.test_phases import (
    measure_supply_voltage,
    measure_idle_current,
    check_firmware,
)


def run_test_for_dut(slot_index: int, serial_number: str) -> str:
    """Entry point for a single DUT worker process."""
    test = htf.Test(
        measure_supply_voltage,
        measure_idle_current,
        check_firmware,
        test_name=f"Production Test - Slot {slot_index}",
    )

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: serial_number)

    return serial_number
```

## Step 3: Launch All DUTs in Parallel

The controller scans DUT serials from the fixture, then dispatches one process per slot.

```python filename="station/controller.py"
import multiprocessing
from concurrent.futures import ProcessPoolExecutor, as_completed

from station.worker import run_test_for_dut


def scan_serials() -> dict[int, str]:
    """Read serial numbers from the fixture."""
    return {
        0: "SN-2024-001",
        1: "SN-2024-002",
        2: "SN-2024-003",
        3: "SN-2024-004",
    }


def run_parallel_cycle():
    serials = scan_serials()
    slots = list(serials.keys())

    print(f"Starting parallel test cycle for {len(slots)} DUTs...")

    with ProcessPoolExecutor(max_workers=len(slots)) as executor:
        futures = {
            executor.submit(run_test_for_dut, slot, serials[slot]): slot
            for slot in slots
        }

        for future in as_completed(futures):
            slot = futures[future]
            try:
                sn = future.result()
                print(f"Slot {slot} ({sn}): complete")
            except Exception as exc:
                print(f"Slot {slot}: FAILED with {exc}")


if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")
    run_parallel_cycle()
```

## Handling Partial Fixture Occupancy

Not all fixture slots may be occupied on every cycle. Guard against empty slots before submitting futures.

```python filename="station/controller.py"
def detect_present_duts() -> dict[int, str]:
    """Return only slots where a DUT is physically present."""
    present = {}
    for slot in range(4):
        if fixture_sense_pin_active(slot):
            present[slot] = read_barcode(slot)
    return present
```

## Comparison of Parallel Strategies

| Approach | Fault Isolation | Shared Instrument Access | Recommended For |
|---|---|---|---|
| `ProcessPoolExecutor` | Full (separate processes) | Requires IPC or per-channel locks | Most production stations |
| `ThreadPoolExecutor` | None (shared process) | Direct (locks required) | Lightweight I/O-bound tests |
| `asyncio` with async VISA | None | Cooperative (no preemption) | Fully async instrument drivers |
| One process per DUT (manual) | Full | Requires IPC | Custom fixture controllers |

## Common Pitfalls

**Process start method on macOS and Windows.** Always call `multiprocessing.set_start_method("spawn")` before launching workers. The default `fork` method on macOS can deadlock when forking after VISA or serial port initialization.

**Plug tearDown on test failure.** OpenHTF calls `tearDown()` on plugs even when a phase raises an exception. Make sure `tearDown` is idempotent and doesn't raise.

**Serial number collision.** If two processes receive the same serial number, TofuPilot creates two runs for the same unit. Validate serial uniqueness in `scan_serials()` before dispatching.

**Instrument timeout under load.** Shared instruments may time out when multiple channels are queried simultaneously. Tune per-channel lock timeouts and instrument read timeouts to match your slowest measurement.

### Test Power Rails on a PCBA
URL: https://www.tofupilot.com/guides/how-to-test-power-rails-on-a-pcba-with-python-and-tofupilot
Learn how to test PCBA power rails with Python, including voltage regulator limits, current draw checks, sequencing, and automated logging via OpenHTF.

Power rail testing validates that voltage regulators produce correct output before any downstream components are exercised. Run these checks first: a failed rail contaminates every subsequent measurement and risks damaging the device under test.

## Prerequisites

- TofuPilot account and API key
- OpenHTF and TofuPilot installed (`pip install openhtf tofupilot`)
- Bench power supply or fixture with controllable power enable line
- Digital multimeter or DAQ accessible from Python

## Common Power Rails on a PCBA

| Rail | Typical Source | Nominal (V) | Common Tolerance | Load Range |
|------|---------------|-------------|-----------------|------------|
| 3.3V | LDO or buck (e.g. TPS62840) | 3.3 | +/-2% | 0-500 mA |
| 5V | Boost or USB VBUS | 5.0 | +/-5% | 0-2 A |
| 1.8V | LDO (e.g. TLV75518) | 1.8 | +/-2% | 0-300 mA |
| 1.2V | Core LDO or buck | 1.2 | +/-3% | 0-1 A |
| 12V | Boost or input direct | 12.0 | +/-5% | 0-3 A |
| VBAT | Li-ion cell | 3.0-4.2 | varies | N/A |

## Setting Limits from Datasheets

Pull limits directly from the voltage regulator datasheet.

**TPS62840 (3.3V buck, Texas Instruments)**
- Typical output accuracy: +/-1.5% at room temperature
- Over temperature: +/-2.5%
- Production test at room temp, use +/-2%: 3.234 V to 3.366 V

**TLV75518 (1.8V LDO, Texas Instruments)**
- Initial accuracy: +/-0.75%
- Add +/-0.5% for resistor divider tolerance
- Production test, use +/-1.5%: 1.773 V to 1.827 V

Always add margin for your measurement system's accuracy. A 16-bit ADC with a 5 V reference introduces roughly +/-0.08 mV of quantization.

## Measurement Techniques

| Measurement | Instrument | When to Use |
|-------------|-----------|-------------|
| DC voltage | DMM / ADC | Every unit |
| No-load current | Current clamp or sense resistor | First article, incoming inspection |
| Full-load current | Power supply readback | Functional test |
| Ripple (peak-to-peak) | Oscilloscope or RMS ADC | Design validation, process escapes |

## Power Sequencing

1. Enable bench supply or fixture relay
2. Wait for the slowest regulator to settle (typically 1-5 ms)
3. Measure all rails
4. Execute functional tests
5. Disable rails in reverse order

## Complete OpenHTF Test with TofuPilot

```python filename="tests/power_rails_test.py"
import time
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


class PowerFixturePlug(htf.plugs.BasePlug):
    """Controls fixture relay and reads voltages from a DAQ."""

    def enable_power(self):
        # Assert fixture relay to apply supply voltage to DUT
        pass

    def disable_power(self):
        # De-assert fixture relay
        pass

    def read_voltage(self, channel: str) -> float:
        # Return DC voltage in volts on the named channel
        # Replace with your actual instrument driver
        voltages = {
            "rail_3v3": 3.312,
            "rail_1v8": 1.796,
            "rail_5v0": 4.985,
        }
        return voltages.get(channel, 0.0)

    def read_current_amps(self, channel: str) -> float:
        # Return current draw in amps via sense resistor
        currents = {
            "rail_3v3_current": 0.142,
            "rail_1v8_current": 0.085,
            "rail_5v0_current": 0.310,
        }
        return currents.get(channel, 0.0)

    def tearDown(self):
        self.disable_power()


@htf.plug(fixture=PowerFixturePlug)
def enable_and_settle(test, fixture):
    """Enable DUT power and wait for all regulators to settle."""
    fixture.enable_power()
    time.sleep(0.010)  # 10 ms covers TPS62840 and TLV75518 soft-start


@htf.plug(fixture=PowerFixturePlug)
@htf.measures(
    htf.Measurement("rail_3v3_voltage")
    .in_range(minimum=3.234, maximum=3.366)
    .with_units(units.VOLT)
    .doc("3.3V rail: TPS62840 +/-2% at room temp"),
    htf.Measurement("rail_3v3_current")
    .in_range(minimum=0.0, maximum=0.500)
    .with_units(units.AMPERE)
    .doc("3.3V rail current draw"),
)
def check_3v3_rail(test, fixture):
    test.measurements.rail_3v3_voltage = fixture.read_voltage("rail_3v3")
    test.measurements.rail_3v3_current = fixture.read_current_amps("rail_3v3_current")


@htf.plug(fixture=PowerFixturePlug)
@htf.measures(
    htf.Measurement("rail_1v8_voltage")
    .in_range(minimum=1.773, maximum=1.827)
    .with_units(units.VOLT)
    .doc("1.8V rail: TLV75518 +/-1.5% including divider tolerance"),
    htf.Measurement("rail_1v8_current")
    .in_range(minimum=0.0, maximum=0.300)
    .with_units(units.AMPERE)
    .doc("1.8V rail current draw"),
)
def check_1v8_rail(test, fixture):
    test.measurements.rail_1v8_voltage = fixture.read_voltage("rail_1v8")
    test.measurements.rail_1v8_current = fixture.read_current_amps("rail_1v8_current")


@htf.plug(fixture=PowerFixturePlug)
@htf.measures(
    htf.Measurement("rail_5v0_voltage")
    .in_range(minimum=4.750, maximum=5.250)
    .with_units(units.VOLT)
    .doc("5V boost rail +/-5%"),
    htf.Measurement("rail_5v0_current")
    .in_range(minimum=0.0, maximum=2.000)
    .with_units(units.AMPERE)
    .doc("5V rail current draw"),
)
def check_5v0_rail(test, fixture):
    test.measurements.rail_5v0_voltage = fixture.read_voltage("rail_5v0")
    test.measurements.rail_5v0_current = fixture.read_current_amps("rail_5v0_current")


@htf.plug(fixture=PowerFixturePlug)
def disable_power_phase(test, fixture):
    """Disable DUT power after all measurements complete."""
    fixture.disable_power()


def main():
    test = htf.Test(
        enable_and_settle,
        check_3v3_rail,
        check_1v8_rail,
        check_5v0_rail,
        disable_power_phase,
        test_name="Power Rails Test",
    )

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Enter serial number: ").strip())


if __name__ == "__main__":
    main()
```

## Power Rail Failure Diagnostics

| Symptom | Likely Cause | First Check |
|---------|-------------|-------------|
| Voltage reads 0 V | Regulator not enabled; blown input fuse | Check enable pin logic level; measure input voltage |
| Voltage below minimum | Current limit reached; input undervoltage | Measure current draw; check input supply voltage drop |
| Voltage above maximum | Feedback resistor open or wrong value | Measure feedback voltage divider; confirm BOM |
| Voltage oscillates / noisy | Output cap missing or wrong type | Check ESR of output cap; inspect for tombstoned parts |
| Correct voltage, excessive current | Short on downstream net | Disconnect loads one by one to isolate short |
| Soft-start too slow | Large output cap; wrong soft-start resistor | Compare measured rise time to datasheet spec |

### Handle Test Failures and Retries
URL: https://www.tofupilot.com/guides/how-to-handle-test-failures-and-retries-with-tofupilot
Learn how to control OpenHTF phase failure behavior, implement retry logic, and ensure clean teardown in production tests with TofuPilot.

When a measurement fails in OpenHTF, the default behavior is to mark the phase as failed and continue running subsequent phases. In production, you need deliberate control: when to stop early, when to retry, and how to preserve failure data in TofuPilot for analytics.

## Prerequisites

- TofuPilot Python client installed: `pip install tofupilot`
- OpenHTF installed: `pip install openhtf`
- Basic familiarity with OpenHTF phases and measurements

## Phase Failure Behavior by Default

OpenHTF phases return a `PhaseResult`. When a measurement fails, the phase is marked `FAIL` but execution continues unless you explicitly stop it.

```python filename="tests/voltage_check.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement('rail_3v3_voltage').in_range(3.2, 3.4).with_units(units.VOLT),
    htf.Measurement('rail_5v_voltage').in_range(4.85, 5.15).with_units(units.VOLT),
)
def check_power_rails(test):
    test.measurements.rail_3v3_voltage = 3.1  # fails: below minimum
    test.measurements.rail_5v_voltage = 5.02  # passes

@htf.measures(
    htf.Measurement('uart_echo').equals('OK'),
)
def check_uart(test):
    # This phase still runs even though check_power_rails failed
    test.measurements.uart_echo = 'OK'

def main():
    test = htf.Test(check_power_rails, check_uart)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: 'SN-001')
```

Both phases run. The run is uploaded to TofuPilot as FAIL with the specific failing measurement recorded.

## Stop on First Failure

For hardware tests where a failed power rail makes downstream tests meaningless, stop early using `PhaseResult.STOP`.

```python filename="tests/production_sequence.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement('rail_3v3_voltage').in_range(3.2, 3.4).with_units(units.VOLT),
)
def check_power_rails(test):
    test.measurements.rail_3v3_voltage = read_adc_channel(0)

    if not test.measurements.rail_3v3_voltage.is_pass:
        test.logger.error('Power rail out of spec, aborting sequence')
        return htf.PhaseResult.STOP

@htf.measures(
    htf.Measurement('uart_echo').equals('OK'),
)
def check_uart(test):
    response = send_uart_command('PING')
    test.measurements.uart_echo = response

def main():
    test = htf.Test(check_power_rails, check_uart)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: 'SN-001')
```

When `check_power_rails` returns `PhaseResult.STOP`, `check_uart` is skipped. TofuPilot records the run as FAIL with `check_uart` phases marked as not executed.

## Retry a Phase on Failure

Retries are useful for transient failures: communication timeouts, settling voltages, or flaky contacts. Use `PhaseResult.REPEAT` with a counter to avoid infinite loops.

```python filename="tests/comms_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

MAX_RETRIES = 3

@htf.measures(
    htf.Measurement('i2c_device_present').equals(True),
)
def check_i2c_device(test, retries=[0]):
    found = probe_i2c_address(0x48)
    test.measurements.i2c_device_present = found

    if not found and retries[0] < MAX_RETRIES:
        retries[0] += 1
        test.logger.warning('I2C device not found, retry %d/%d', retries[0], MAX_RETRIES)
        return htf.PhaseResult.REPEAT

    retries[0] = 0  # reset for future runs

@htf.measures(
    htf.Measurement('i2c_temperature').in_range(-10, 85).with_units(units.DEGREE_CELSIUS),
)
def read_i2c_temperature(test):
    temp = read_temperature_sensor(0x48)
    test.measurements.i2c_temperature = temp

def main():
    test = htf.Test(check_i2c_device, read_i2c_temperature)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: 'SN-002')
```

Using a mutable default argument (`retries=[0]`) persists the counter across `REPEAT` calls within the same phase execution. Reset it before returning to avoid stale state on the next DUT.

## Retry with Backoff Using a Plug

For production lines with many fixtures, a plug keeps retry logic reusable and keeps phase code clean.

```python filename="plugs/comms_plug.py"
import time
import openhtf as htf
from openhtf.plugs import BasePlug

class CommPlug(BasePlug):
    def setUp(self):
        import serial
        self._port = serial.Serial('/dev/ttyUSB0', 115200, timeout=1)

    def send_with_retry(self, command, expected, attempts=3, delay=0.5):
        response = ''
        for attempt in range(attempts):
            self._port.write(f'{command}\n'.encode())
            response = self._port.readline().decode().strip()
            if response == expected:
                return response, attempt + 1
            time.sleep(delay)
        return response, attempts

    def tearDown(self):
        if self._port:
            self._port.close()
```

```python filename="tests/serial_test.py"
import openhtf as htf
from tofupilot.openhtf import upload
from plugs.comms_plug import CommPlug

@htf.plug(comm=CommPlug)
@htf.measures(
    htf.Measurement('firmware_version').matches_regex(r'v\d+\.\d+\.\d+'),
    htf.Measurement('version_retry_count').in_range(minimum=1),
)
def check_firmware_version(test, comm):
    response, attempts = comm.send_with_retry('VERSION', expected='v2.1.0', attempts=3)
    test.measurements.firmware_version = response
    test.measurements.version_retry_count = attempts

def main():
    test = htf.Test(check_firmware_version)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: 'SN-003')
```

Recording `version_retry_count` as a measurement lets TofuPilot surface retry frequency across your production fleet. A spike in retries on a specific station points to a fixture problem before it becomes a yield problem.

## Clean Teardown After Failures

Plugs with a `tearDown` method are always called by OpenHTF, even when a phase fails or the test stops early. Use this to release hardware resources reliably.

```python filename="plugs/fixture_plug.py"
import openhtf as htf
from openhtf.plugs import BasePlug

class FixturePlug(BasePlug):
    def setUp(self):
        self._power_on = False
        self._relay_closed = False

    def power_on_dut(self):
        enable_power_rail()
        self._power_on = True

    def close_test_relay(self):
        close_relay(1)
        self._relay_closed = True

    def tearDown(self):
        # Always runs, even on STOP or exception
        if self._relay_closed:
            open_relay(1)
        if self._power_on:
            disable_power_rail()
```

```python filename="tests/powered_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload
from plugs.fixture_plug import FixturePlug

@htf.plug(fixture=FixturePlug)
@htf.measures(
    htf.Measurement('leakage_current').in_range(maximum=50).with_units(units.AMPERE),
)
def measure_leakage(test, fixture):
    fixture.power_on_dut()
    fixture.close_test_relay()

    current_ua = read_current_meter() * 1e6
    test.measurements.leakage_current = current_ua

    if not test.measurements.leakage_current.is_pass:
        test.logger.error('Leakage %.1f uA exceeds 50 uA limit', current_ua)
        return htf.PhaseResult.STOP
    # fixture.tearDown() called automatically by OpenHTF

def main():
    test = htf.Test(measure_leakage)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: 'SN-004')
```

## Failure Handling Strategy Comparison

| Strategy | Use when | PhaseResult | Downstream phases |
|---|---|---|---|
| Continue on fail | Failures are independent, collect all data | (default) | Run |
| Stop on first fail | Downstream tests are invalid after failure | `STOP` | Skipped |
| Retry transient | Failure may be a fixture fluke | `REPEAT` | Run after retry |
| Retry with limit | Same, but cap attempts | `REPEAT` + counter | Run or STOP |

## Logging Failures for Analytics

TofuPilot captures every failed measurement automatically. To make failures actionable in analytics, add structured context using `test.logger`.

```python filename="tests/full_sequence.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement('oscillator_freq_hz').in_range(7_990_000, 8_010_000).with_units(units.HERTZ),
)
def check_oscillator(test):
    freq = measure_frequency(channel=2)
    test.measurements.oscillator_freq_hz = freq

    if not test.measurements.oscillator_freq_hz.is_pass:
        deviation_ppm = abs(freq - 8_000_000) / 8_000_000 * 1e6
        test.logger.error(
            'Oscillator %.0f Hz (%.1f ppm from 8 MHz)',
            freq, deviation_ppm,
        )

@htf.measures(
    htf.Measurement('adc_offset_mv').in_range(-5, 5),
    htf.Measurement('adc_gain_error_pct').within_percent(1.0, 0.1),
)
def calibrate_adc(test):
    offset = measure_adc_offset()
    gain = measure_adc_gain()

    test.measurements.adc_offset_mv = offset * 1000
    test.measurements.adc_gain_error_pct = gain

def main():
    test = htf.Test(check_oscillator, calibrate_adc, name='PCB-Rev-C')
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: 'SN-005')
```

Log messages appear in the TofuPilot run detail view alongside the failing measurement. Quantitative context (deviation in ppm, not just pass/fail) makes root cause analysis faster when reviewing failures across a batch.

### How to Write a Hardware Test Plan
URL: https://www.tofupilot.com/guides/how-to-write-a-hardware-test-plan-with-tofupilot
Structure a hardware test plan from EVT to production: map requirements to stages, set limits from data, and implement it as a TofuPilot procedure.

A hardware test plan defines what gets tested, at which stage, with what equipment, and against which criteria. Without one, test coverage depends on whoever wrote the last script. This guide walks through structuring a test plan from EVT through production and implementing it in Python with TofuPilot.

## What a Test Plan Covers

| Section | Content |
|---------|---------|
| Product overview | What the product does, key specifications |
| Test stages | Which tests run at EVT, DVT, PVT, production |
| Test procedures | Step-by-step instructions per test |
| Equipment list | Instruments, fixtures, software |
| Pass/fail criteria | Measurement limits per test |
| Data requirements | What gets recorded, where, retention period |
| Roles | Who writes tests, who runs them, who reviews results |

The test plan is a living document. It starts rough in EVT and gets refined through each build stage.

## Step 1: Map Requirements to Test Stages

Start with the product requirements and decide when each one gets verified.

| Requirement | EVT | DVT | PVT | Production |
|-------------|-----|-----|-----|------------|
| Output voltage within 5% | X | X | X | X |
| Survives 1m drop | | X | | |
| Operates -20C to 60C | | X | X | |
| Firmware boots in under 2s | X | X | X | X |
| Leakage current below 500uA | | X | | X |
| FPY above 95% | | | X | X |

Not every requirement needs testing at every stage. EVT focuses on basic functionality. DVT covers environmental and stress. PVT validates the manufacturing process. Production tests run on every unit.

## Step 2: Write the Plan as a Procedure

The most common failure mode is a test plan in a document and limits in code, drifting apart within weeks. Keep the limits in one place that is both the plan and the executable.

A procedure `.yaml` declares the sequence, the measurements and the limits as data. It reads like the plan and runs like the test.

```yaml filename="procedure.yaml"
name: Production Functional Test
version: 1.0.0
description: Production acceptance per TP-100 rev B

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

main:
  - name: Production Checks
    python: phases.production_checks
    measurements:
      - name: Output Voltage
        unit: V
        description: TP-100 rev B section 3.1
        validators:
          - operator: ">="
            expected_value: 4.75
          - operator: "<="
            expected_value: 5.25
      - name: Boot Time
        unit: ms
        description: TP-100 rev B section 3.4
        validators:
          - operator: "<="
            expected_value: 2000
      - name: Leakage Current
        unit: uA
        description: TP-100 rev B section 5.2
        validators:
          - operator: "<="
            expected_value: 500
```

```python filename="phases/production_checks.py"
def production_checks(measurements):
    """Production test: voltage, boot time, safety."""
    measurements.output_voltage = 5.02
    measurements.boot_time = 1350
    measurements.leakage_current = 85.0
```

The `description` field on each measurement is where the test plan paragraph reference goes, so the document and the data point at each other. The `version` is recorded with every run, so results taken before a limit change stay interpretable.

## Step 3: Specify Equipment and Fixtures

Document every instrument in the test plan. This prevents "it works on my bench" problems.

| Instrument | Model | Purpose | Calibration |
|-----------|-------|---------|-------------|
| DMM | Keysight 34461A | Voltage and current | Annual |
| Power supply | Rigol DP832 | DUT power | Annual |
| Fixture | Custom PCB jig v2.1 | Pogo pin contact | Per shift check |
| PC | Any, Python 3.10+ | Test execution | N/A |

In the procedure, each instrument is a plug, so the equipment list has a direct counterpart in code:

```yaml filename="procedure.yaml"
plugs:
  - name: dmm
    python: plugs.dmm:Multimeter
  - name: psu
    python: plugs.psu:PowerSupply
```

Record the instrument identity on the run so a result can be traced back to the bench that produced it:

```python filename="phases/record_equipment.py"
def record_equipment(run, dmm):
    run.metadata["dmm_id"] = dmm.identify()
```

## Step 4: Set Limits from Data, Not Guesses

Start with limits from the datasheet or design spec. Tighten them after DVT data shows the actual distribution.

The practical move is one procedure per stage, with the same measurement names and progressively tighter validators. Same names means the distributions are comparable across stages.

```yaml filename="evt/procedure.yaml"
main:
  - name: Voltage
    python: phases.voltage
    measurements:
      - name: Output Voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.5
          - operator: "<="
            expected_value: 5.5
```

```yaml filename="production/procedure.yaml"
main:
  - name: Voltage
    python: phases.voltage
    measurements:
      - name: Output Voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.75
          - operator: "<="
            expected_value: 5.25
```

EVT runs wide while you learn the design. Production runs tight, set from what the DVT and PVT distributions actually showed. Because the measurement name is identical, TofuPilot shows one distribution across all stages and the Cpk tells you whether the tighter limit is survivable before you impose it on the line.

See [how to set measurement limits using production data](/guides/how-to-set-measurement-limits-using-production-data-with-tofupilot) for the statistics.

## Step 5: Run It and Collect From the Start

```bash filename="run.sh"
tofupilot run ./procedure.yaml --upload
```

Log every run from the first prototype. EVT data informs DVT limits, DVT data informs production limits, and none of that works if the early data was never captured.

TofuPilot tracks results across all test stages. Open the Analytics tab to see measurement distributions, yield trends, and failure patterns. This data feeds back into the test plan as you refine limits and add or remove test steps.

## If You Already Run OpenHTF

Existing OpenHTF suites run natively on the TofuPilot CLI with no rewrite. The same production checks expressed with decorators:

```python filename="production_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=4.75, maximum=5.25)
    .with_units(units.VOLT),
    htf.Measurement("boot_time_ms")
    .in_range(maximum=2000)
    .with_units(units.MILLISECOND),
    htf.Measurement("leakage_current_uA")
    .in_range(maximum=500)
    .with_units(units.MICROAMPERE),
)
def phase_production_checks(test):
    """Production test: voltage, boot time, safety."""
    test.measurements.output_voltage_V = 5.02
    test.measurements.boot_time_ms = 1350
    test.measurements.leakage_current_uA = 85.0


test = htf.Test(phase_production_checks)
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Fix |
|---------|-----|
| Testing everything at every stage | Map requirements to specific stages |
| Limits from the datasheet only | Refine from actual production data |
| Test plan in a Word doc, code elsewhere | Declare limits in the procedure; reference the plan section in `description` |
| Renaming measurements between stages | Keep names identical so distributions are comparable |
| No calibration tracking | Record instrument identity and calibration state on the run |
| Skipping EVT data collection | Start logging from the first prototype |

A test plan is not a one-time document. Review it after every build stage and update limits, procedures, and equipment based on what the data shows.

### How to Control a Keysight DMM in Python
URL: https://www.tofupilot.com/guides/how-to-control-a-keysight-multimeter-with-python-and-tofupilot
Control a Keysight 34461A or 34465A from Python: SCPI setup, NPLC and range selection, four-wire resistance, and recording readings as measurements.

The Keysight 34461A and 34465A are the workhorse bench DMMs on most test benches. They speak SCPI over USB or LAN, so Python talks to them without vendor software. This guide covers connection, the settings that actually affect your numbers, and turning readings into recorded measurements.

## Prerequisites

- Python 3.10+
- A Keysight 34460A/34461A/34465A/34470A on USB or Ethernet
- `pip install pyvisa pyvisa-py`

## Connecting

```python filename="keysight/connect.py"
import pyvisa

rm = pyvisa.ResourceManager("@py")
print(rm.list_resources())

dmm = rm.open_resource("TCPIP::192.168.1.100::INSTR")
dmm.timeout = 10000
print(dmm.query("*IDN?").strip())
# Keysight Technologies,34461A,MY59001234,A.03.01-02.40-03.01-00.52-03-01
```

On USB the resource looks like `USB0::0x2A8D::0x1301::MY59001234::INSTR`. The 2A8D vendor ID is Keysight; 1301 is the 34461A.

## Reading a Voltage

The shortest correct sequence:

```python filename="keysight/measure.py"
dmm.write("*RST")
dmm.write("*CLS")

voltage = float(dmm.query(":MEAS:VOLT:DC? 10,0.0001"))
print(f"{voltage:.6f} V")
```

The two arguments to `:MEAS:VOLT:DC?` are range and resolution. Passing them explicitly matters more than it looks, and that is the next section.

## The Three Settings That Change Your Numbers

**Range.** Autorange is convenient and slow, because the meter hunts on every reading. In a production test where the expected value is known, fix the range:

```python filename="keysight/range.py"
dmm.write(":CONF:VOLT:DC 10")     # 10 V range, fixed
dmm.write(":VOLT:DC:RANG:AUTO OFF")
```

Fixing the range typically takes a reading from tens of milliseconds to a few milliseconds, which is significant when a test takes a hundred readings.

**NPLC (integration time).** This is the single biggest lever on noise versus speed. NPLC is measured in power line cycles:

| NPLC | Reading time (50 Hz) | Noise rejection | Use for |
|------|---------------------|-----------------|---------|
| 0.02 | ~0.4 ms | Poor | Fast screening |
| 0.2 | ~4 ms | Moderate | General production |
| 1 | ~20 ms | Good, rejects line noise | Default for accuracy |
| 10 | ~200 ms | Very good | Low-level, calibration |
| 100 | ~2 s | Best | Metrology |

```python filename="keysight/nplc.py"
dmm.write(":VOLT:DC:NPLC 1")
```

NPLC of 1 or above rejects mains hum, because the integration window is a whole number of line cycles. Below 1, line noise starts appearing in your readings and people blame the board.

**Autozero.** The meter periodically measures its own offset. Accurate, but it roughly halves throughput. For a fast production test where you have characterised the drift:

```python filename="keysight/autozero.py"
dmm.write(":VOLT:DC:ZERO:AUTO ONCE")  # Take one zero, then hold
```

## Four-Wire Resistance

Two-wire resistance includes your lead resistance, which is fine at kilohms and useless at milliohms. For anything below about 100 Ω, use four-wire:

```python filename="keysight/four_wire.py"
dmm.write(":CONF:FRES 100")        # Four-wire, 100 Ω range
dmm.write(":FRES:NPLC 10")         # Slow and quiet for low values
dmm.write(":FRES:OCOM ON")         # Offset compensation, for thermal EMF

resistance = float(dmm.query(":READ?"))
print(f"{resistance:.6f} Ω")
```

Offset compensation takes two readings with opposite current polarity and cancels thermoelectric voltages at the junctions. On a weld or busbar measurement it is the difference between a repeatable number and a drifting one.

## A Reusable Plug

```python filename="plugs/dmm.py"
import pyvisa


class KeysightDmm:
    """Keysight 34461A/34465A over SCPI."""

    RESOURCE = "TCPIP::192.168.1.100::INSTR"

    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 10000
        self.instr.write("*RST")
        self.instr.write("*CLS")

    def dc_voltage(self, range_v: float = 10, nplc: float = 1) -> float:
        self.instr.write(f":CONF:VOLT:DC {range_v}")
        self.instr.write(f":VOLT:DC:NPLC {nplc}")
        return float(self.instr.query(":READ?"))

    def dc_current(self, range_a: float = 1, nplc: float = 1) -> float:
        self.instr.write(f":CONF:CURR:DC {range_a}")
        self.instr.write(f":CURR:DC:NPLC {nplc}")
        return float(self.instr.query(":READ?"))

    def four_wire_resistance(self, range_ohm: float = 100, nplc: float = 10) -> float:
        self.instr.write(f":CONF:FRES {range_ohm}")
        self.instr.write(f":FRES:NPLC {nplc}")
        self.instr.write(":FRES:OCOM ON")
        return float(self.instr.query(":READ?"))

    def identify(self) -> str:
        return self.instr.query("*IDN?").strip()

    def errors(self) -> list[str]:
        out = []
        while True:
            err = self.instr.query(":SYST:ERR?").strip()
            if err.startswith("+0,") or err.startswith("0,"):
                break
            out.append(err)
        return out

    def close(self):
        self.instr.close()
```

## Using It in a Procedure

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.dc_voltage(range_v=10)
    measurements.rail_1v8 = dmm.dc_voltage(range_v=10)
```

```python filename="phases/shunt_resistance.py"
def shunt_resistance(measurements, dmm):
    measurements.shunt_resistance = dmm.four_wire_resistance(range_ohm=1)
```

```python filename="phases/instrument_health.py"
def instrument_health(measurements, dmm):
    measurements.dmm_errors = len(dmm.errors())
```

```yaml filename="procedure.yaml"
name: Board Electrical Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:KeysightDmm

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Rail 1V8
        unit: V
        validators:
          - operator: ">="
            expected_value: 1.75
          - operator: "<="
            expected_value: 1.85

  - name: Shunt Resistance
    python: phases.shunt_resistance
    depends_on: [power_rails]
    measurements:
      - name: Shunt Resistance
        unit: "Ω"
        description: Four-wire with offset compensation
        validators:
          - operator: ">="
            expected_value: 0.0095
          - operator: "<="
            expected_value: 0.0105

  - name: Instrument Health
    python: phases.instrument_health
    depends_on: [shunt_resistance]
    measurements:
      - name: DMM Errors
        validators:
          - operator: "=="
            expected_value: 0
```

Run it with `tofupilot run ./procedure.yaml`.

The phases are chained because they share one VISA session. Independent phases run concurrently, and a single meter cannot serve two at once.

## Recording the Instrument

```python filename="phases/record_equipment.py"
def record_equipment(run, dmm):
    run.metadata["dmm_id"] = dmm.identify()
```

When two benches disagree, the first question is which meter took the reading. `*IDN?` includes the serial number, so this answers it without anyone typing an asset tag.

## Speed

For a production test taking many readings of the same quantity, configure once and read repeatedly rather than reconfiguring each time:

```python filename="keysight/fast_loop.py"
dmm.write(":CONF:VOLT:DC 10")
dmm.write(":VOLT:DC:NPLC 0.2")
dmm.write(":VOLT:DC:RANG:AUTO OFF")
dmm.write(":TRIG:SOUR IMM")
dmm.write(":SAMP:COUN 100")

readings = [float(x) for x in dmm.query(":READ?").split(",")]
print(f"{len(readings)} readings, mean {sum(readings)/len(readings):.6f} V")
```

`:SAMP:COUN` takes the whole burst inside the meter and returns them in one transfer, which is far faster than a hundred round trips.

## If You Already Run OpenHTF

```python filename="keysight/openhtf_test.py"
import pyvisa
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class KeysightPlug(BasePlug):
    def setUp(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource("TCPIP::192.168.1.100::INSTR")
        self.instr.timeout = 10000
        self.instr.write("*RST")

    def dc_voltage(self) -> float:
        self.instr.write(":CONF:VOLT:DC 10")
        self.instr.write(":VOLT:DC:NPLC 1")
        return float(self.instr.query(":READ?"))

    def tearDown(self):
        self.instr.close()


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
@htf.plug(dmm=KeysightPlug)
def test_rail(test, dmm):
    test.measurements.rail_3v3 = dmm.dc_voltage()


test = htf.Test(test_rail, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Troubleshooting

| Problem | Cause | Fix |
|---------|-------|-----|
| Readings noisy at the last digits | NPLC too low | Raise to 1 or 10 |
| Readings drift with room temperature | Autozero disabled | Re-enable, or `ZERO:AUTO ONCE` more often |
| Low-value resistance reads high and unstable | Two-wire, lead resistance included | Use `:CONF:FRES` with four leads |
| Milliohm reading wanders | Thermal EMF at the junctions | `:FRES:OCOM ON` |
| Each reading takes ~100 ms | Autoranging on every read | Fix the range, turn autorange off |
| `-420, "Query UNTERMINATED"` | Reading a response that was never produced | Check the command actually returns data |
| Timeouts under parallel phases | One session used concurrently | Serialise with `depends_on` |

### Build Your First Hardware Test
URL: https://www.tofupilot.com/guides/how-to-build-your-first-hardware-test-with-python-and-tofupilot
Build your first hardware test in Python: a working procedure with limits, a plug for your instrument, and results on the dashboard in minutes.

This guide takes you from nothing to a working hardware test that records structured measurements, evaluates limits, and uploads results. It runs on your laptop first, with a fake instrument, so you can see the whole shape before any hardware is involved.

## Prerequisites

- Python 3.10+
- The TofuPilot CLI

```bash filename="install.sh"
curl -fsSL https://www.tofupilot.app/install | sh
```

On Windows:

```powershell filename="install.ps1"
$p = "$env:TEMP\tp-install.ps1"; irm https://www.tofupilot.app/install.ps1 -OutFile $p; powershell -ExecutionPolicy Bypass -File $p; ri $p -EA 0
```

No account is needed to run a procedure locally.

## Step 1: Create the Files

A procedure is a directory with a `.yaml` file, a `phases/` folder for your test logic, and a `plugs/` folder for your instrument drivers.

```bash filename="scaffold.sh"
mkdir -p first-test/phases first-test/plugs
cd first-test
```

## Step 2: Write the Plug

A plug is a plain Python class that owns a connection to something: an instrument, a fixture, a DUT. Start with a fake one that returns a fixed value, so the test runs before the hardware exists.

```python filename="plugs/dmm.py"
class Multimeter:
    """Fake multimeter. Replace the body with real instrument calls."""

    def read_voltage(self) -> float:
        return 3.31
```

There is no base class to inherit and no decorator to apply. When you are ready for real hardware, the change is contained here:

```python filename="plugs/dmm.py"
import pyvisa


class Multimeter:
    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource("TCPIP::192.168.1.100::INSTR")
        self.instr.timeout = 5000

    def read_voltage(self) -> float:
        return float(self.instr.query(":MEAS:VOLT:DC?"))

    def close(self):
        self.instr.close()
```

## Step 3: Write the Phase

A phase is a plain function. It receives `measurements` and any plug it names as a parameter, injected by name.

```python filename="phases/power_rail.py"
def power_rail(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
```

That is the whole test logic. No limit checking, because limits are not logic: they are data, and they live in the procedure.

## Step 4: Declare the Procedure

```yaml filename="procedure.yaml"
name: My First Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rail
    python: phases.power_rail
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

Three things to notice:

- **The measurement key comes from the name.** `Rail 3V3` becomes `rail_3v3`, which is what the phase assigns to.
- **The plug is matched by name.** `dmm` in the `plugs` list is the `dmm` parameter in the phase.
- **The limits are data.** Two validators, greater-or-equal and less-or-equal, make a range.

## Step 5: Run It

```bash filename="run.sh"
tofupilot run ./procedure.yaml
```

The first run creates a virtual environment for the procedure. You will be asked to confirm the serial and part number, then the phase runs and the result appears.

To see it fail, change the fake reading to something out of range:

```python filename="plugs/dmm.py"
    def read_voltage(self) -> float:
        return 9.9
```

Run it again and the phase fails with the reason, naming the validator that rejected the value. The measured value is still recorded; a failure is data, not an absence of data.

## Step 6: Add a Second Measurement

Most real tests measure several things in one phase.

```python filename="phases/power_rail.py"
def power_rail(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
    measurements.rail_5v0 = dmm.read_voltage()
```

```yaml filename="procedure.yaml"
  - name: Power Rail
    python: phases.power_rail
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Rail 5V0
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
```

Measurements are not only numeric. A string check and a boolean check look like this:

```yaml filename="procedure.yaml"
      - name: Firmware Version
        validators:
          - operator: "=="
            expected_value: "2.1.0"
      - name: Label Present
        validators:
          - operator: "=="
            expected_value: true
```

## Step 7: Add a Second Phase

Phases are the unit of sequencing and of retry. Split the test when the steps are independent, or when you want them to fail separately.

```python filename="phases/firmware.py"
def firmware(measurements, dmm):
    measurements.firmware_version = "2.1.0"
```

```yaml filename="procedure.yaml"
main:
  - name: Power Rail
    python: phases.power_rail
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
  - name: Firmware
    python: phases.firmware
    measurements:
      - name: Firmware Version
        validators:
          - operator: "=="
            expected_value: "2.1.0"
```

Independent phases run in parallel by default. If one must follow another, say so:

```yaml filename="procedure.yaml"
  - name: Firmware
    python: phases.firmware
    depends_on: [power_rail]
```

## Step 8: Send Results to the Dashboard

Create a free account, then link the procedure and run it with `--upload`:

```bash filename="upload.sh"
tofupilot login
tofupilot link
tofupilot run ./procedure.yaml --upload
```

Every measurement arrives with its name, value, unit, limits and outcome, which is what makes first pass yield, Cpk and control charts work without any extra code.

## What to Do Next

| Goal | Where to go |
|------|-------------|
| Talk to a real instrument | [Getting started with PyVISA](/guides/getting-started-with-pyvisa-control-test-instruments-from-python) |
| Prompt the operator for input | [How to add operator prompts to hardware tests](/guides/how-to-add-operator-prompts-to-hardware-tests) |
| Test several boards at once | [How to run parallel tests on multiple DUTs](/guides/how-to-run-parallel-tests-on-multiple-duts-with-tofupilot) |
| Put it on a station | [How to deploy Python test scripts to production stations](/guides/how-to-deploy-python-test-scripts-to-production-stations) |
| Set limits from real data | [How to set measurement limits using production data](/guides/how-to-set-measurement-limits-using-production-data-with-tofupilot) |

## If You Prefer Everything in Python

OpenHTF is the other purpose-built option, with the sequence and limits expressed as decorators instead of YAML. The same first test:

```python filename="first_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
def power_rail(test):
    test.measurements.rail_3v3 = 3.31


test = htf.Test(power_rail, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`. Both frameworks run on the same CLI and upload to the same dashboard, so this is a style choice rather than a lock-in. See [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf).

## Common First-Test Mistakes

| Mistake | Symptom | Fix |
|---------|---------|-----|
| Phase parameter does not match the plug name | Plug is not injected | The parameter name must equal the `name` in `plugs` |
| Assigning a key that is not declared | Measurement never appears | The key comes from the measurement `name`, in snake_case |
| Checking limits inside the phase | Limits are invisible to analytics | Declare validators in the procedure instead |
| One phase for the entire test | Everything fails together, no retry granularity | Split independent steps into phases |
| Sharing one instrument across parallel phases | Intermittent instrument errors | Use `depends_on`, or give each phase its own instrument |

### TestStand Alternatives for Manufacturing Test
URL: https://www.tofupilot.com/guides/teststand-alternatives-with-tofupilot
NI TestStand alternatives for manufacturing test: what replaces sequences, process models and step types, and what a migration actually involves.

NI TestStand costs around $4,310 per seat per year, runs only on Windows, and stores sequences as binary `.seq` files that resist code review and merging. Teams look for alternatives when they add stations, when hiring gets hard, or when they want test code in the same Git workflow as the rest of engineering. This guide covers the realistic options and what a migration actually involves.

## What You Are Replacing

TestStand is a test executive. It is worth being precise about which parts you depend on, because the alternatives cover them unevenly.

| TestStand capability | How much teams actually use it |
|---|---|
| Sequence execution with step types | Always |
| Numeric and string limit steps | Always |
| Process models (Sequential, Parallel, Batch) | Often, for multi-UUT fixtures |
| Operator interfaces | Always, though usually a stock one |
| Database logging | Often, into a schema someone maintains |
| Report generation | Often |
| Callbacks (PreUUT, PostUUT, and friends) | Sometimes, and this is where the surprises hide |
| LabVIEW / C / .NET step modules | Depends entirely on the shop |

The last row decides the size of the job. If your step modules are Python or C, they port. If they are LabVIEW VIs doing real instrument work, the migration includes those too.

## The Options

| Alternative | Language | License | Operator UI | Parallel UUT | Notes |
|---|---|---|---|---|---|
| **TofuPilot Framework** | Python | MIT | Declarative, from the procedure | Native (slots) | YAML procedure, plain Python phases |
| **OpenHTF** | Python | Apache 2.0 | Station Server | Limited | Decorator-based, proven at scale |
| **OpenTAP** | C#/.NET | MPL 2.0 | Operator Panel | Native | Closest architecture to TestStand |
| **pytest** | Python | MIT | None | Via xdist | Not built for production test |
| **Keep TestStand** | - | Commercial | Built in | Native | Sometimes correct |

## Mapping TestStand Concepts

The mapping to the TofuPilot Framework is direct, because both separate the sequence definition from the step logic. TestStand does it with a binary `.seq` file and step modules; the framework does it with a text `.yaml` file and Python functions.

| TestStand | TofuPilot Framework |
|---|---|
| Sequence file (`.seq`) | `procedure.yaml` |
| Sequence | `main:` list |
| Step | Entry under `main:` |
| Step module (code) | Phase function in `phases/` |
| Numeric Limit Test | `measurements` with `>=` / `<=` validators |
| String Value Test | `validators` with `operator: "=="` |
| Multiple Numeric Limit Test | Several measurements in one phase |
| Pass/Fail Test | Boolean measurement, `== true` |
| Action step | Phase with no measurements |
| Precondition | `enable_if` on the phase |
| Step dependency / flow | `depends_on` |
| Loop / retry on failure | `retry` on the phase |
| Sequential process model | Default execution |
| Parallel / Batch process model | `execution.slots` |
| UUT serial number | `unit.serial_number` |
| PreUUT / PostUUT callback | `setup` / `teardown` phases |
| Station globals | Plug state, or run metadata |
| Database logging | Automatic upload |
| Report generation | Dashboard and exports |

## The Same Test, Both Ways

A TestStand sequence with two Numeric Limit Tests and a String Value Test, expressed as a procedure:

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Rail 5V0
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2

  - name: Firmware
    python: phases.firmware
    depends_on: [power_rails]
    measurements:
      - name: Firmware Version
        validators:
          - operator: "=="
            expected_value: "2.1.0"
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage(1)
    measurements.rail_5v0 = dmm.read_voltage(2)
```

```python filename="phases/firmware.py"
def firmware(measurements, dmm):
    measurements.firmware_version = dmm.query_firmware()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def read_voltage(self, channel: int) -> float:
        return {1: 3.31, 2: 5.02}[channel]  # Replace with a PyVISA query

    def query_firmware(self) -> str:
        return "2.1.0"
```

The limits are data in a text file, which is the practical difference: a limit change is a reviewable diff instead of an opaque binary edit.

## Replacing the Process Models

This is the part teams worry about most, and it maps more cleanly than expected.

**Sequential** is the default: one UUT, phases in dependency order.

**Parallel and Batch** become fixture slots. Declare them and the engine runs each slot independently, with its own unit, measurements and outcome:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: Slot 1
    - name: Slot 2
    - name: Slot 3
    - name: Slot 4
```

A failure in one slot stops that slot and leaves the others running, which is the behaviour Batch model users expect. For a fixture with many identical positions, give a count instead of a list:

```yaml filename="procedure.yaml"
execution:
  slots: 16
```

## Replacing the Callbacks

PreUUT and PostUUT become setup and teardown phases. The important distinction, which TestStand also makes, is between per-UUT and per-execution work:

- A **slot-scoped** teardown touches that slot only, and runs while other slots are still testing.
- An **execution-scoped** teardown runs once after every slot is done, and is where a shared instrument, a rack supply or a fixture lock belongs.

Getting that scope wrong is the classic multi-slot bug: a teardown that powers down a shared supply while a neighbouring slot is mid-measurement.

See [replace TestStand callbacks in Python](/guides/replace-teststand-callbacks-in-python) for the detail.

## What a Migration Actually Costs

| Work item | Effort | Notes |
|---|---|---|
| Port step modules to Python | Small if C/Python, large if LabVIEW | The real variable |
| Re-express sequences as procedures | Small | Mostly mechanical, and readable afterwards |
| Rebuild the operator interface | Small | Declared in the procedure, not coded |
| Replace database logging | Small | Automatic upload replaces the schema you maintained |
| Re-validate against known units | Medium, and non-negotiable | Run both systems on the same boards and compare |
| Retrain the team | Small | Python is a smaller step than TestStand was |

The re-validation step is the one to insist on. Run TestStand and the new procedure on the same set of units, including known-bad ones, and compare measurement by measurement before you decommission anything.

## When to Stay on TestStand

- **Deep NI hardware integration.** PXI chassis, switching, FPGA targets, and NI-specific instrument drivers you would otherwise have to rewrite.
- **A large working sequence library with no budget.** If it works and you are not adding stations, the cost may not be justified.
- **Heavy LabVIEW step modules.** The migration includes porting those, and that is the expensive part.

## If You Prefer Decorators

OpenHTF is the other purpose-built Python option and does the same job without a YAML layer, with a larger installed base. Existing OpenHTF suites and the TofuPilot Framework both run natively on the TofuPilot CLI and upload to the same dashboard, so the choice is about authoring style rather than lock-in. See [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf) and [how to migrate from NI TestStand to Python](/guides/how-to-migrate-from-ni-teststand-to-python-with-tofupilot).

## Common Migration Mistakes

| Mistake | Consequence | Fix |
|---|---|---|
| Porting sequences before step modules | Nothing runs, morale drops | Port and unit-test the modules first |
| Skipping the parallel re-validation | A limit transcription error ships | Run both systems on the same units |
| Recreating TestStand's structure exactly | You inherit workarounds you no longer need | Re-express the intent, not the implementation |
| Slot-scoped teardown on shared equipment | Intermittent failures under multi-slot | Use execution scope for shared resources |
| Migrating everything at once | One big risky cutover | One station, one product, then expand |

### Structure a Production Test Script
URL: https://www.tofupilot.com/guides/how-to-structure-a-python-test-script-for-production-with-tofupilot
Structure a Python test script for production: separate limits from logic, instruments from steps, and make the layout survive a year of changes.

A test script that works on your bench and a test script that survives two years on a production line are different artefacts. The difference is almost entirely structural: what is separated from what. This guide covers the layout that holds up, and the specific couplings that cause trouble later.

## The Four Things to Keep Apart

| Concern | Where it belongs | Why separate |
|---|---|---|
| **Sequence and limits** | Procedure file | Reviewable as data, changeable without touching logic |
| **Test logic** | Phase functions | Plain Python, unit-testable |
| **Instrument access** | Plug classes | Swappable, mockable, lifecycle-managed |
| **Configuration** | Environment or config file | Differs per station, not per product |

Most scripts that become unmaintainable have collapsed two or more of these into one file.

## The Layout

```
fct-pcba100/
├── procedure.yaml
├── phases/
│   ├── power_rails.py
│   ├── firmware.py
│   └── calibration.py
├── plugs/
│   ├── dmm.py
│   ├── psu.py
│   └── dut.py
└── tests/
    └── test_phases.py
```

One directory per procedure. Phases and plugs are ordinary Python modules, which means the `tests/` directory can exercise them with pytest and no hardware.

## Limits Belong in the Procedure, Not the Code

This is the separation that pays off most.

```python filename="bad/inline_limits.py"
def test_power(dmm):
    v = dmm.read_voltage()
    if not 3.2 <= v <= 3.4:          # Limits buried in a comparison
        raise AssertionError(f"Rail out of range: {v}")
    return v
```

The value is checked and then discarded. The limits exist only inside an expression, so nothing can report them, compare against them, or tell you they changed in March.

```yaml filename="procedure.yaml"
main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        description: TP-100 rev B section 3.1
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
```

The phase does one thing: get a number and record it. Evaluation is the framework's job, and because the limits are data, a limit change is a reviewable diff rather than a code change.

## Phases Should Be Boring

A good phase reads like a description of the step:

```python filename="phases/firmware.py"
def firmware(measurements, dut):
    measurements.firmware_version = dut.query_version()
    measurements.bootloader_version = dut.query_bootloader()
```

Things that do not belong in a phase:

- **Limit checks.** Declare validators instead.
- **Instrument setup.** That is the plug's constructor.
- **Retry loops.** Use `retry` with a `then:` trigger in the procedure.
- **Sleep-and-poll for an instrument.** Put it in the plug's method.
- **Deciding whether to run.** Use `enable_if` or `depends_on`.

Each of those, left in a phase, becomes copy-pasted into the next twenty phases.

## Plugs Own Lifecycle, Including Failure

```python filename="plugs/psu.py"
import pyvisa


class PowerSupply:
    RESOURCE = "TCPIP::192.168.1.101::INSTR"

    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 5000
        self.instr.write("*RST")

    def apply(self, channel: int, volts: float, limit: float):
        self.instr.write(f":APPL CH{channel},{volts},{limit}")
        self.instr.write(f":OUTP CH{channel},ON")

    def close(self):
        """Runs at teardown regardless of how the run ended."""
        for ch in (1, 2, 3):
            try:
                self.instr.write(f":OUTP CH{ch},OFF")
            except Exception:
                pass
        self.instr.close()
```

Putting power-down in `close()` rather than a final phase is the difference between a failed test that de-energises the board and one that leaves it powered on the fixture until someone notices.

## Make Configuration External

The resource string above is hard-coded, which is fine until the second station has a different IP.

```python filename="plugs/psu.py"
import os


class PowerSupply:
    def __init__(self):
        resource = os.environ.get("PSU_RESOURCE", "TCPIP::192.168.1.101::INSTR")
        ...
```

Station-specific values are environment, not code. Product-specific values, such as limits, are the procedure. Keeping those two straight prevents the situation where a station has its own fork of the test.

## Make Phases Testable Without Hardware

Because plugs are plain classes and phases are plain functions, both are testable in CI:

```python filename="tests/test_phases.py"
from phases.power_rails import power_rails


class FakeDmm:
    def read_voltage(self) -> float:
        return 3.31


class Recorder:
    def __init__(self):
        self.values = {}

    def __setattr__(self, name, value):
        if name == "values":
            super().__setattr__(name, value)
        else:
            self.values[name] = value


def test_power_rails_records_voltage():
    measurements = Recorder()
    power_rails(measurements, FakeDmm())
    assert measurements.values["rail_3v3"] == 3.31
```

This catches the class of bug that otherwise only appears on a station at 6 a.m.: a renamed measurement key, a typo'd attribute, a phase that silently records nothing.

## Naming Is a Long-Term Commitment

Measurement names are the identity of your data across every run you will ever take. Renaming `Rail 3V3` to `3V3 Rail` splits the history in two and breaks every trend and capability calculation that spans the change.

Pick names early, make them specific (`rail_3v3` not `voltage`), and leave them alone. If a name is genuinely wrong, changing it is a deliberate decision with a known cost, not a tidy-up.

## Version the Procedure

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.2.0
description: FCT for PCBA-100, per TP-100 rev B
```

Every run records the version. Without it, a yield drop caused by a tightened limit is indistinguishable from a process problem, and that is a diagnosis that can take weeks.

## Keep One Procedure for Attended and Unattended Runs

Maintaining a separate copy for CI guarantees the two drift. Use the same procedure and supply the operator answers externally:

```bash filename="ci/run.sh"
tofupilot run ./procedure.yaml --json --ui-values ci/ui.json --ui-timeout 300
```

## If You Already Run OpenHTF

The same separations apply, expressed with decorators. Keep plugs in their own module, phases in another, and the assembly in a thin entry point:

```python filename="main.py"
import openhtf as htf
from tofupilot.openhtf import upload

from phases import test_power_rails, test_firmware


def main():
    test = htf.Test(
        test_power_rails,
        test_firmware,
        procedure_id="FCT-001",
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial: "))


if __name__ == "__main__":
    main()
```

Install with `pip install "tofupilot[openhtf]"`, or run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---|---|---|
| Limits inside comparisons | No reporting, no trending, no review trail | Declare validators in the procedure |
| Instrument setup inside phases | Slow, leaky, duplicated | Plug constructor |
| Power-down as a final phase | Board energised after a failure | Plug's `close()` |
| Hard-coded station config | Each station forks the test | Environment variables |
| Renaming measurements | History splits, trends break | Choose names once |
| No procedure version | Limit changes look like process changes | Set and bump `version` |
| Separate CI copy of the procedure | The two drift apart | One procedure, `--ui-values` |
| Phases untestable without hardware | Bugs found on the line at 6 a.m. | Fake plugs, pytest in CI |

### Upgrade a Station from LabVIEW 2019 to 2024
URL: https://www.tofupilot.com/guides/upgrade-a-station-from-labview-2019-to-2024
Learn the order to upgrade a LabVIEW 2019 station to 2024, from mass compile and Git tags to the RTE and driver install on each station, and when to skip it.

Upgrading a station from LabVIEW 2019 to 2024 is a mass compile on the development machine plus a Run-Time Engine and driver upgrade on every station that will run the new build. The first half takes an afternoon. The second half is where builds get stranded, because a 2024 executable won't start on a 2019 RTE, and the drivers on a five-year-old station have their own supported version ranges.

## Before You Start

Write down what's on the floor before you touch the development machine. One row per station, filled in by walking to it, not from memory.

| Station | Current RTE and bitness | DAQmx and VISA versions | Who can rebuild | Runs per week |
|---|---|---|---|---|
| FCT-01 | 2019 32-bit | DAQmx 19.x, VISA 19.x | Marc | 420 |
| FCT-02 | 2019 32-bit | DAQmx 19.x, VISA 19.x | Marc | 380 |
| EOL-01 | 2019 32-bit | DAQmx 18.x, VISA 19.x | Marc | 150 |
| CAL-01 | 2018 32-bit | DAQmx 18.x, VISA 18.x | nobody | 60 |

Three things fall out of the table. The pilot station is the one with the highest volume and a complete row (FCT-01 here): it gives you a comparison in days, not months. Any station on a different bitness than your target (all of them, if you're going from 32-bit to 64-bit) needs the new RTE and new 64-bit drivers, not just the RTE. And a row with "nobody" in the rebuild column is a decision, not an upgrade: see the section on when not to upgrade.

The full inventory method, including how to read the RTE version off a running station, is in [which LabVIEW version runs on each station](/guides/which-labview-version-runs-on-each-station).

## The Upgrade Order

The order matters more than any single step, because each one is cheap to undo only until the next one.

1. Development machine first. Install LabVIEW 2024 next to 2019, not instead of it. You'll need 2019 to rebuild the old executable if the pilot fails.
2. Tag the last 2019 commit, then open the project in 2024 and run Tools, Advanced, Mass Compile on the project folder. Save everything.
3. Commit the mass compile as one commit with no functional change in it, and say so in the message. Tag it. The next section is why.
4. Build the executable on 2024. Fix the things the build reports (usually a deprecated VI or a changed driver palette), each in its own commit after the mass compile, never mixed into it.
5. One pilot station. Install the 2024 RTE and matching DAQmx and VISA on FCT-01 only. Leave the 2019 RTE installed; RTEs coexist. Deploy the new build next to the old one.
6. Side by side for two weeks. Run the 2024 build and the 2019 build on the same units, compare the measurements and the pass/fail per phase. Not just FPY: a limit that behaves differently on a new driver shows as a shift in a distribution, not always as a failure.
7. Then the rest, one station at a time, in the order of your table.

Two weeks is the smallest interval where a shift in a measurement is visible against normal variation on a 400-run-a-week station. On a 60-run-a-week station, wait longer.

## What Mass Compile Does to Your Git History

Mass compile rewrites every VI it recompiles, and after a version change that's every VI in the project. The repository shows as fully modified in one commit. That's unavoidable and it's fine, as long as you make it findable.

```bash filename="tag-mass-compile.sh"
# Bracket the mass compile with two tags so history on either side stays reachable.
git tag -a lv2019-last -m "Last commit built with LabVIEW 2019 32-bit"

# Open the project in LabVIEW 2024, run Tools > Advanced > Mass Compile, save all.

git add -A
git commit -m "Mass compile for LabVIEW 2024 64-bit. No functional change; every VI rewritten by the compiler."
git tag -a lv2024-first -m "First commit built with LabVIEW 2024 64-bit"
git push --tags
```

The two tags do two jobs. `lv2019-last` is where you check out to rebuild the old executable if the pilot fails. `lv2024-first` tells everyone reading `git log` later that nothing between those two tags is a real change, so they stop looking there for the commit that moved a limit.

Keep the mass compile commit pure. If you also fix a broken VI in it, that fix is now invisible, buried in a diff of a thousand rewritten binaries. Fix it in the next commit. [Separate compiled code in LabVIEW, explained](/guides/separate-compiled-code-in-labview-explained) covers the setting that reduces noise between mass compiles; it doesn't reduce the mass compile itself, and LabVIEW classes in particular can still show as modified after one.

The mass compile is also the moment the LVCompare story ends for the old history. Comparing a VI from before the tag with one from after shows every node as changed. That's expected, and it's why [why LabVIEW version control is so painful](/guides/why-labview-version-control-is-so-painful) puts mass compile among the four sources of pain.

## Driver Compatibility

Each NI driver (DAQmx, VISA, and any instrument-specific driver you use) supports a range of LabVIEW versions and a bitness. A 2019 32-bit station running a DAQmx from that era won't necessarily serve a 2024 64-bit build, and the version that does serve 2024 may drop support for an older card in the chassis.

Don't guess from version numbers. Check NI's version compatibility tables for DAQmx and VISA against the LabVIEW version and bitness you're targeting, and against every card and interface in the station, before you install anything on the pilot. Write the versions you chose into the inventory table. That table is the one document the next person will need.

Two cases deserve a separate look. A station with a legacy card that the new DAQmx no longer lists is a station you can't upgrade without replacing hardware. And a PXI station with FPGA code compiled for 2019 needs the FPGA module and a recompile of the bitfile, which is its own project.

## When Not to Upgrade

The upgrade is worth it for a station that will run for years and needs new instruments or new code. It's not worth it in three cases.

A station that will be replaced. If the fixture is due for a rebuild next year, leave it on 2019 and put the effort into the replacement. RTE 2019 keeps running.

A product near end-of-life. The last 5,000 units of a product don't justify the two-week side-by-side and the driver risk. Freeze the station, document it, and note the RTE and driver versions in the inventory so it can be rebuilt once if it has to be.

A PXI station with FPGA timing you can't retest. If the acceptance test for the timing needs equipment or units you no longer have, an upgrade that touches the FPGA bitfile is a change you can't validate. Don't make it.

In all three, the honest line in the inventory is "stays on 2019, rebuild from tag `lv2019-last` on the 2019 dev VM". Keep that VM.

## The Contrast

A TofuPilot Framework station pins its runtime in text. `pyproject.toml` names the Python version and the packages, and the deployment built from a commit carries the resolved runtime with it.

```toml filename="pyproject.toml"
# The runtime is pinned here; the deployment built from this commit carries it.
[project]
name = "controller-pcba-fct"
version = "2.4.1"
requires-python = ">=3.12,<3.13"
dependencies = ["tofupilot", "pyvisa", "pyvisa-py"]
```

Bumping the runtime is a one-line change in a pull request, reviewed like any other. `tofupilot deploy --prod` builds one immutable deployment with an id like `dep_abc123`, and `tofupilot pull` puts that same build on every station, so there's no per-station RTE to install and no station on a version the others aren't. Every run uploaded from a pulled deployment carries the `deployment_id`, so the side-by-side comparison is a filter in TofuPilot rather than a spreadsheet. And if the pilot fails, rollback re-pins every production station to the previous deployment with no rebuild, while the runs already uploaded keep the id of the build that produced them.

[How to deploy Python test scripts to production stations](/guides/how-to-deploy-python-test-scripts-to-production-stations) covers the deployment flow, and [run a mixed LabVIEW and Python test team](/guides/run-a-mixed-labview-and-python-test-team) covers the period when both kinds of station report to the same procedure.

## Start With One Station

When you do move a station, the pilot from your inventory table is the one to move. Rebuild it as a `procedure.yaml` with phases and plugs in Python with the TofuPilot Framework, keep the same limits, and run it side by side with the LabVIEW build on the same units for two weeks, the same protocol as the upgrade.

```bash filename="install-and-run.sh"
# Install the CLI, run the station locally (no account needed), or run with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Deploy Test Scripts to Stations with Git
URL: https://www.tofupilot.com/guides/how-to-deploy-python-test-scripts-to-production-stations
Push to a branch and every linked station runs the new test procedure, with immutable artifacts and instant rollback. Plus the manual path if you need it.

Getting a test script onto production stations reliably is one of the least discussed problems in manufacturing test. The common approaches, copying files over SSH, a network share everyone reads from, or a USB stick, all share the same weakness: nobody can say with certainty which station is running which version.

The advice engineers usually converge on is to keep a stable configuration in Git and have stations check it out. That instinct is correct, and it is what the deployment model below automates.

## Why manual deployment goes wrong

The failure modes are consistent across teams:

- **Version drift.** Station 3 was offline during the last rollout and is still running last month's limits. Nobody notices until its yield diverges.
- **Partial rollouts.** A copy loop fails halfway. Some stations updated, some did not, and the script does not report which.
- **No rollback.** A bad release reaches every station simultaneously and reverting means repeating the whole process under pressure.
- **Untracked local edits.** Someone fixed something directly on a station. That fix exists nowhere else and disappears at the next deployment.
- **Python version mismatch.** The script works on the development machine and fails on a station with a different interpreter.

The underlying problem is that the deployed artifact is not identifiable. If you cannot name exactly what is running, you cannot reason about it.

## Deploying from a Git push

Connect a repository to a procedure, and every push to a tracked branch builds a deployment. Pushes to the production branch roll out to every linked station; pushes to other branches produce preview deployments you can pin to a single station for testing.

GitHub, GitLab and Bitbucket Data Center are supported.

A procedure is a `.yaml` file declaring the sequence, the measurements and their limits. Phases are plain Python functions, and plugs connect instruments as Python classes.

```yaml filename="procedure.yaml"
name: PCBA Functional Test

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-200"

plugs:
  - name: Multimeter
    python: plugs.dmm:Multimeter

main:
  - name: Power Rail Test
    python: phases.power_rail
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/power_rail.py"
def power_rail(measurements, multimeter):
    measurements.rail_3v3 = multimeter.read_voltage()
```

Because the limits live in the YAML rather than in Python, tightening a limit is a one-line diff that reviews cleanly and is visible in the deployment history.

Commit, push, and the stations pick it up:

```bash filename="deploy.sh"
git add procedure.yaml phases/power_rail.py
git commit -m "Tighten 3V3 rail limits to 3.2-3.4V"
git push origin main
```

The build produces an immutable artifact carrying the executable, the resolved runtime, and every input the station needs. A station running `dep_abc123` always runs exactly that code. There is no ambiguity about what is on which station.

### Rolling out gradually

For a change you want to validate before it reaches the whole floor, pin a preview deployment to one station:

```bash filename="preview.sh"
git checkout -b tighter-limits
git push origin tighter-limits
# Pin the resulting preview deployment to one station from the dashboard
```

Run a shift on that station, compare the yield against the others, then merge to production when you are satisfied.

### Rolling back

Because artifacts are immutable and retained, rollback is selecting a previous deployment rather than rebuilding anything. The station picks it up on its next check. This matters most at the moment you least want to be running a build.

### Other ways to create a deployment

Git push is the usual path, but not the only one:

- **Manual trigger** from the procedure's Deployments page, by commit SHA or branch.
- **CLI**, which builds from your local working tree with or without a connected repo. Useful for a fix you have not committed yet.

## Deploying without the platform

If you are not using TofuPilot's deployment, the same principles apply and you can build them yourself. The rest of this guide covers that path.

### Pin your dependencies

Unpinned dependencies are the most common source of "works on my machine" failures on a station.

```text filename="requirements.txt"
openhtf==1.6.1
tofupilot[openhtf]==2.16.0
pyserial==3.5
numpy==1.26.4
pyinstaller==6.10.0
```

```bash filename="install_deps.sh"
#!/bin/bash
pip install -r requirements.txt
pip freeze > requirements.lock.txt  # capture transitive deps for audit
```

Use `requirements.lock.txt` for auditing, `requirements.txt` for installs. Never pin transitive dependencies manually.

### Bundle into a single executable

PyInstaller packages the script and its dependencies into one file, which removes the station Python version from the equation.

OpenHTF uses dynamic imports that PyInstaller cannot detect, so the spec file needs explicit hidden imports:

```python filename="station_tests.spec"
# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import collect_data_files, collect_submodules

block_cipher = None

openhtf_datas = collect_data_files("openhtf")
openhtf_hiddenimports = collect_submodules("openhtf")

a = Analysis(
    ["test_main.py"],
    pathex=["."],
    binaries=[],
    datas=openhtf_datas + [
        ("plugs/", "plugs/"),
        ("config/", "config/"),
        ("VERSION", "."),
    ],
    hiddenimports=openhtf_hiddenimports + [
        "openhtf.output.callbacks",
        "openhtf.output.callbacks.json_factory",
        "openhtf.util.logs",
        "tofupilot",
        "tofupilot.openhtf",
    ],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    cipher=block_cipher,
    noarchive=False,
)

pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.zipfiles,
    a.datas,
    [],
    name="station_tests",
    debug=False,
    strip=False,
    upx=False,
    runtime_tmpdir=None,
    console=True,
)
```

```bash filename="build.sh"
#!/bin/bash
set -e

VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
echo "$VERSION" > VERSION
echo "Building version $VERSION"

pyinstaller station_tests.spec --clean --noconfirm

mv dist/station_tests "dist/station_tests_v${VERSION}"
rm VERSION
echo "Built: dist/station_tests_v${VERSION}"
```

**PyInstaller troubleshooting:**

| Symptom | Cause | Fix |
|---|---|---|
| `ModuleNotFoundError: openhtf.util.logs` | Missing hidden import | Add to `hiddenimports` in spec |
| Blank frontend at `localhost:4444` | OpenHTF web assets not bundled | Check `collect_data_files("openhtf")` |
| `OSError: [Errno 2]` on config file | Data file not included | Add config dir to `datas` |
| Crash on second run | Leftover `_MEIPASS` temp dir | Set `runtime_tmpdir` to a fixed path |

### Keep station identity out of the code

Each station needs its own identity and credentials. Never hardcode either.

```bash filename="station_env.sh"
#!/bin/bash
# Place in /etc/profile.d/tofupilot.sh for persistent configuration
export TOFUPILOT_API_KEY="tp_live_xxxxxxxxxxxxxxxxxxxx"
export TOFUPILOT_STATION_ID="station-floor-1-cell-3"
export STATION_SERIAL_PORT="/dev/ttyUSB0"
export STATION_BAUD_RATE="115200"
```

For stations where environment variables are impractical, fall back to a config file:

```python filename="config/loader.py"
import json
import os
from pathlib import Path


def load_station_config() -> dict:
    if os.environ.get("TOFUPILOT_API_KEY"):
        return {
            "api_key": os.environ["TOFUPILOT_API_KEY"],
            "station_id": os.environ.get("TOFUPILOT_STATION_ID", "unknown"),
            "serial_port": os.environ.get("STATION_SERIAL_PORT", "/dev/ttyUSB0"),
            "baud_rate": int(os.environ.get("STATION_BAUD_RATE", "115200")),
        }

    config_path = Path("/etc/tofupilot/station.json")
    if not config_path.exists():
        raise FileNotFoundError(
            f"Station config not found at {config_path}. "
            "Set TOFUPILOT_API_KEY or create the config file."
        )

    with config_path.open() as f:
        return json.load(f)
```

### Push to stations

```bash filename="deploy_manual.sh"
#!/bin/bash
set -e

VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
EXECUTABLE="dist/station_tests_v${VERSION}"
DEPLOY_PATH="/opt/tofupilot/station_tests"

STATIONS=(
    "operator@station-01.local"
    "operator@station-02.local"
    "operator@station-03.local"
)

for STATION in "${STATIONS[@]}"; do
    echo "Deploying v${VERSION} to ${STATION}..."
    ssh "$STATION" "[ -f ${DEPLOY_PATH} ] && cp ${DEPLOY_PATH} ${DEPLOY_PATH}.bak || true"
    scp "$EXECUTABLE" "${STATION}:${DEPLOY_PATH}"
    ssh "$STATION" "chmod +x ${DEPLOY_PATH}"
    ssh "$STATION" "${DEPLOY_PATH} --version 2>&1 | head -1"
    echo "  Done: ${STATION}"
done

echo "Deployed v${VERSION} to ${#STATIONS[@]} stations."
```

Note the `.bak` copy before overwriting. That is your rollback, and it only goes back one version.

### Record the version on every run

Whichever method you use, record which version produced each result. Without it you cannot correlate a yield change with a code release.

The version is a property of the run, so record it where the rest of your run metadata lives. With OpenHTF, the test record's metadata carries it:

```python filename="test_main.py"
import importlib.metadata
import openhtf as htf
from tofupilot.openhtf import upload


def get_version() -> str:
    try:
        return importlib.metadata.version("station-tests")
    except importlib.metadata.PackageNotFoundError:
        import os, sys
        base = getattr(sys, "_MEIPASS", os.path.dirname(__file__))
        with open(os.path.join(base, "VERSION")) as f:
            return f.read().strip()


def record_version(test):
    test.metadata["software_version"] = get_version()


def main():
    test = htf.Test(
        record_version,
        phase_power_on,
        phase_voltage_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        part_number="PCBA-200",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: input("Enter serial number: ").strip())
```

With Git-push deployment you get this for free: the deployment identifier already names the exact artifact, so there is nothing to record by hand.

## Comparing the options

| Method | Setup | Station Python required | Rollback | Version certainty |
|---|---|---|---|---|
| Git push deployment | Connect a repo | No | Select a previous deployment | Exact, per station |
| PyInstaller over SSH | Medium | No | Restore `.bak`, one version | Manual tracking |
| Virtual environment | Low | Yes | Replace the venv | Manual tracking |
| Docker container | High | No | `docker pull` previous tag | Exact, by tag |
| Network share | Low | Yes | Revert the file | None |

The network share row is worth avoiding. It looks simplest and it is the only method where a station can silently pick up a half-written file mid-shift.

## What to check after any rollout

1. **Every station reports the expected version.** If one does not, it did not update.
2. **The first runs after rollout pass.** A broken deployment is most visible in the first few units.
3. **Yield after matches yield before.** If limits changed deliberately, expect a step; if they did not, a step means something went wrong.

Deployment is one of the few places in production test where being able to undo quickly matters more than getting it right first time.

### Track PCBA Results Across Lines
URL: https://www.tofupilot.com/guides/how-to-track-pcba-test-results-across-production-lines-with-tofupilot
Learn how to tag OpenHTF test runs with station, line, factory, and shift metadata, then compare first pass yield across production lines using TofuPilot.

TofuPilot associates every test run with a named station. By encoding line, factory, and shift into the station name and run metadata, you get a single query surface that spans every production line and factory without custom tooling.

## Why Multi-Line Traceability Matters

When a PCBA defect appears in the field, you need to answer three questions fast:

- Which production line built the affected units?
- Which shift was running at the time?
- Is the failure isolated to one station or systemic across a line?

Without structured metadata on each run, answering these questions means cross-referencing operator logs, shift schedules, and test CSV exports by hand. TofuPilot solves this by making station identity and run metadata first-class fields on every test record.

| Without structured metadata | With TofuPilot station and run metadata |
|---|---|
| Manual log cross-referencing | Single dashboard filter |
| Shift data in spreadsheets | Encoded in run at test time |
| Per-line export files | Unified API query |
| Yield comparison in Excel | Built-in per-station yield chart |

## Prerequisites

- Python 3.9+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot client installed (`pip install tofupilot`)
- A TofuPilot account with at least one procedure created

## Setting Up Station Identity in TofuPilot

Each physical test station maps to a named station in TofuPilot. The naming convention carries all the traceability context you need.

### Station Naming Convention

Use a structured name that encodes factory, line, and station number:

```
{FACTORY}-{LINE}-FCT{STATION_NUMBER}
```

Examples:

| Station name | Factory | Line | Station |
|---|---|---|---|
| `SZX-L1-FCT01` | Shenzhen | Line 1 | FCT station 1 |
| `SZX-L2-FCT01` | Shenzhen | Line 2 | FCT station 1 |
| `TXL-L1-FCT01` | Toulouse | Line 1 | FCT station 1 |
| `TXL-L1-FCT02` | Toulouse | Line 1 | FCT station 2 |

### Environment-Based Station Configuration

Store the station identity in environment variables on each machine, not in the test script:

```bash filename="/etc/environment"
TOFUPILOT_API_KEY=tp_station_xxxxxxxxxxxxx
STATION_ID=SZX-L1-FCT01
FACTORY=SZX
LINE=L1
```

Load them in your test script at runtime:

```python filename="config.py"
import os

STATION_ID = os.environ["STATION_ID"]
FACTORY = os.environ["FACTORY"]
LINE = os.environ["LINE"]
```

This approach means the same test script binary deploys to every station. Only the environment differs.

## Tagging Runs with Line, Factory, and Shift Metadata

### Determining the Current Shift

```python filename="shift.py"
from datetime import datetime

def get_current_shift() -> str:
    """Return shift label based on local wall clock."""
    hour = datetime.now().hour
    if 6 <= hour < 14:
        return "morning"
    elif 14 <= hour < 22:
        return "afternoon"
    else:
        return "night"
```

### Full OpenHTF Test with Station Metadata

This example tests a PCBA power supply board. The station identity, line, factory, and shift are injected at test setup time and attached to every run.

```python filename="pcba_fct_test.py"
import os

import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

import config
from shift import get_current_shift


class PowerSupplyPlug(htf.plugs.BasePlug):
    """Controls the bench PSU over USB-serial."""

    def setUp(self):
        import serial
        self._port = serial.Serial("/dev/ttyUSB0", 9600, timeout=1)

    def set_voltage(self, volts: float, channel: int = 1):
        self._port.write(f":APPL CH{channel},{volts},1.0\n".encode())

    def measure_voltage(self, channel: int = 1) -> float:
        self._port.write(f":MEAS:VOLT? CH{channel}\n".encode())
        return float(self._port.readline().strip())

    def tearDown(self):
        self._port.close()


@htf.plug(psu=PowerSupplyPlug)
@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(minimum=3.235, maximum=3.365)
    .with_units(units.VOLT)
    .doc("3.3 V rail under 100 mA load"),
    htf.Measurement("rail_5v0")
    .in_range(minimum=4.900, maximum=5.100)
    .with_units(units.VOLT)
    .doc("5.0 V rail under 200 mA load"),
)
def test_power_rails(test, psu):
    psu.set_voltage(3.3, channel=1)
    test.measurements.rail_3v3 = psu.measure_voltage(channel=1)

    psu.set_voltage(5.0, channel=2)
    test.measurements.rail_5v0 = psu.measure_voltage(channel=2)


@htf.plug(psu=PowerSupplyPlug)
@htf.measures(
    htf.Measurement("idle_current")
    .in_range(minimum=0, maximum=0.350)
    .with_units(units.AMPERE)
    .doc("Total board current draw at idle"),
)
def test_idle_current(test, psu):
    psu.set_voltage(5.0, channel=2)
    current_a = float(psu._port.readline().strip())
    test.measurements.idle_current = current_a


def main():
    serial_number = input("Scan DUT serial number: ").strip()

    test = htf.Test(
        test_power_rails,
        test_idle_current,
        procedure_id="PCBA-FCT-001",
    )

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: serial_number)


if __name__ == "__main__":
    main()
```

Every run in TofuPilot carries the station identity as queryable metadata alongside the standard pass/fail result and measurements.

## Comparing Results Across Lines in TofuPilot

With runs uploading from all lines, TofuPilot's filtering and analytics give you direct comparison:

- **FPY by station** shows yield for each station grouped by line prefix. If Line B consistently trails Line A, the problem is systemic to that line.
- **Measurement histograms** reveal whether one line's values are shifted or have wider spread. A shifted mean suggests calibration offset.
- **Failure Pareto by station** shows which specific tests fail more often on each station. If one station accounts for most failures, start investigating that fixture.
- **Trend charts** show whether yield gaps are constant, growing, or appeared suddenly after a change.

## Investigating Yield Differences

When you find a yield gap between lines, narrow down the cause systematically:

1. **Check measurement distributions.** If one line's values are offset, it's likely calibration or equipment. If they're wider, it's process variation.
2. **Check by time of day.** Yield drops on night shifts point to operator training or environmental changes.
3. **Check individual stations.** Sometimes the "line" problem is actually one bad station dragging down the average.
4. **Check by component lot.** If you track lot numbers as metadata, filter by lot to see if specific batches drive the difference.

A station with FPY below 95% while neighboring stations are at 97-98% typically indicates a fixture contact issue, cable degradation, or calibration drift.

## Deployment Checklist

| Step | Action |
|---|---|
| Station naming | Follow `{FACTORY}-{LINE}-FCT{N}` convention |
| API keys | One key per station, stored in environment |
| Procedure ID | Same ID across all lines for unified yield view |
| Dashboard | Verify station names appear on first run before full rollout |

### What Is OEE for Test Stations
URL: https://www.tofupilot.com/guides/what-is-oee-with-tofupilot
Overall equipment effectiveness (OEE) measures manufacturing productivity. Learn how to calculate OEE for test stations and track it with TofuPilot.

Overall equipment effectiveness (OEE) measures how productively a machine or station is used. It combines three factors: availability, performance, and quality. For test stations, OEE reveals how much of your capacity is actually producing good tested units. This guide covers how to calculate OEE for test stations and how to track its components with TofuPilot.

## The OEE Formula

OEE = Availability x Performance x Quality

| Factor | What It Measures | Formula |
|--------|-----------------|---------|
| Availability | How much of planned time the station is running | Run time / Planned production time |
| Performance | How fast it runs compared to ideal | (Ideal cycle time x Total units) / Run time |
| Quality | How many units pass first time | Good units / Total units |

Each factor is a percentage. OEE is their product.

### Example

| Factor | Value | Calculation |
|--------|-------|------------|
| Planned time | 8 hours (480 min) | Shift duration |
| Downtime | 45 min (setup + fixture change) | |
| Run time | 435 min | 480 - 45 |
| Availability | 90.6% | 435 / 480 |
| Ideal cycle time | 1 min/unit | |
| Total units tested | 400 | |
| Performance | 92.0% | (1 x 400) / 435 |
| Good units (first pass) | 380 | |
| Quality | 95.0% | 380 / 400 |
| **OEE** | **79.2%** | 90.6% x 92.0% x 95.0% |

World-class OEE is 85%+. Most manufacturing operations run 60-75%.

## The Six Big Losses

OEE breaks down into six loss categories. Each maps to one of the three OEE factors:

| Loss | OEE Factor | Test Station Example |
|------|-----------|---------------------|
| Unplanned stops | Availability | Fixture jam, instrument error, PC crash |
| Setup and adjustments | Availability | Product changeover, calibration, fixture swap |
| Small stops | Performance | Operator absent, DUT loading delay |
| Slow cycles | Performance | Instrument settling, retests, slow communication |
| Production rejects | Quality | Units that fail test and go to scrap |
| Startup rejects | Quality | First units after setup that need retesting |

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Capture Test Data

Every test run logged to TofuPilot contributes data to the three OEE factors. The test duration feeds performance. The pass/fail result feeds quality. Station uptime feeds availability.

```python filename="production_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("current_draw_mA")
    .in_range(minimum=90, maximum=110)
    .with_units(units.MILLIAMPERE),
)
def phase_functional(test):
    """Production functional test."""
    test.measurements.output_voltage_V = 5.01
    test.measurements.current_draw_mA = 99.2
```

```python filename="production_test.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_functional)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Step 2: Track OEE Components in TofuPilot

TofuPilot tracks the data you need for each OEE factor:

| OEE Factor | TofuPilot Data |
|-----------|---------------|
| Availability | Station uptime, gaps between test runs |
| Performance | Test duration per unit, throughput per hour |
| Quality | First pass yield per station |

Open the Analytics tab to monitor:

- **Station throughput** (units per hour) tracks performance
- **First pass yield** tracks quality
- **Test gaps** (time between consecutive runs) reveal availability losses

## Improving OEE

Focus on the lowest factor first. If availability is 70% but quality is 98%, fixing availability gives the biggest return.

| Low Factor | Where to Look | Action |
|-----------|--------------|--------|
| Availability below 85% | Downtime log, changeover time | Reduce setup time, improve fixture reliability |
| Performance below 90% | Phase durations in TofuPilot | Optimize instrument settings, reduce settling time |
| Quality below 95% | Failure Pareto in TofuPilot | Fix top failure modes, tighten process controls |

Small improvements compound. Improving each factor by 5% (from 85% to 90%) increases OEE from 61.4% to 72.9%. That's 19% more capacity from the same equipment.

### How to Optimize Manufacturing Test Stations
URL: https://www.tofupilot.com/guides/how-to-optimize-test-stations-with-tofupilot
Test station optimization reduces cycle time and increases throughput. Learn how to identify bottlenecks and improve station performance with TofuPilot.

Test station cycle time directly limits production throughput. A 60-second test on two stations means 120 units per hour. Cut that to 45 seconds and you're at 160 without adding hardware. This guide covers how to find bottlenecks, reduce cycle time, and track station performance with TofuPilot.

## Where Time Goes

A typical test station cycle breaks down into:

| Phase | Typical Share | Example |
|-------|--------------|---------|
| Fixture load/unload | 20-30% | Operator places DUT, closes clamp |
| Instrument settling | 10-20% | DMM auto-range, power supply ramp |
| Measurement acquisition | 30-40% | Read voltage, current, frequency |
| Data logging and reporting | 5-10% | Write results to database |
| Idle/wait | 5-15% | Operator between units |

The biggest gains usually come from instrument settling and measurement acquisition, not from faster code.

## Step 1: Measure Your Baseline

You can't optimize what you don't measure. OpenHTF records the duration of every phase automatically. TofuPilot displays these timings in the test step performance view.

```python filename="optimized_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("supply_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
)
def phase_power_check(test):
    """Measure supply voltage after power-up."""
    test.measurements.supply_voltage_V = 5.01


@htf.measures(
    htf.Measurement("signal_amplitude_V")
    .in_range(minimum=1.8, maximum=2.2)
    .with_units(units.VOLT),
)
def phase_signal_check(test):
    """Measure signal output amplitude."""
    test.measurements.signal_amplitude_V = 2.01


@htf.measures(
    htf.Measurement("self_test_result").equals("PASS"),
)
def phase_firmware_self_test(test):
    """Run firmware self-test and check result."""
    test.measurements.self_test_result = "PASS"
```

Run this on 50 units and review the phase durations in TofuPilot. The longest phase is your bottleneck.

## Step 2: Reduce Instrument Settling Time

Instrument settling is often the hidden time sink. Auto-ranging, input impedance switching, and filter settling add hundreds of milliseconds per measurement.

| Optimization | Time Saved | How |
|-------------|-----------|-----|
| Fixed range instead of auto-range | 100-500ms per reading | Set range in SCPI before measuring |
| Reduce integration time (NPLC) | 50-200ms per reading | Lower NPLC from 10 to 1 (trades accuracy) |
| Minimize channel switching | 50-100ms per switch | Group measurements by channel |
| Parallel instrument commands | Varies | Send setup commands while another instrument reads |

Only reduce accuracy where the measurement margin allows it. If your limit is 4.9V to 5.1V and units read 5.0V with 1mV spread, you have room to reduce NPLC.

## Step 3: Optimize Test Sequence

Phase order matters. Put the fastest-failing tests first to reduce average cycle time on bad units.

| Strategy | Effect |
|----------|--------|
| Fail-fast ordering | Tests that catch 80% of defects run first |
| Power-up checks before functional | Catch dead boards before slow tests |
| Skip downstream phases on failure | Don't test analog if power-up failed |

```python filename="optimized_test.py"
from openhtf import PhaseResult


@htf.measures(
    htf.Measurement("power_good").equals("PASS"),
)
def phase_power_good(test):
    """Quick power check. If this fails, skip everything else."""
    result = "PASS"
    test.measurements.power_good = result
    if result != "PASS":
        return PhaseResult.STOP
```

## Step 4: Track Station Performance

Connect the test to TofuPilot and monitor station-level metrics over time.

```python filename="optimized_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_power_good,
    phase_power_check,
    phase_signal_check,
    phase_firmware_self_test,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

TofuPilot tracks station performance automatically. Open the Analytics tab to see:

- **Throughput per station** (units per hour)
- **Phase duration breakdown** showing where time is spent
- **Yield per station** to catch fixture or equipment issues
- **Station comparison** to identify underperforming stations

## Optimization Checklist

| Check | Action |
|-------|--------|
| Longest phase identified | Review in TofuPilot step performance |
| Auto-range disabled where possible | Set fixed range via SCPI |
| NPLC reduced where margin allows | Check measurement distributions |
| Fail-fast order implemented | Move high-failure phases earlier |
| Phase skip on critical failure | Return PhaseResult.STOP |
| Fixture load time measured | Time the operator, not just the code |

Station optimization is iterative. Make one change, measure the effect in TofuPilot, then move to the next bottleneck. Small gains compound: five 10% improvements give you a 40% faster station.

### Scale from 1 to 100 Test Stations
URL: https://www.tofupilot.com/guides/how-to-scale-from-1-to-100-test-stations-with-tofupilot
Architecture guide for scaling test infrastructure from 1 to 100 stations, covering naming, script distribution, config management, networking, and monitoring.

Run one station or a hundred: the architecture decisions you make at 10 stations determine whether you succeed at 100. This guide walks through the changes needed at each scale threshold.

## Scaling Challenges by Stage

| Stations | What breaks | Root cause |
|----------|-------------|------------|
| 1 | Nothing | You're fine |
| 5-10 | Config drift | Manual setup per station |
| 10-25 | Script version mismatch | No deployment pipeline |
| 25-50 | Debugging blindness | No centralized logs |
| 50-100 | Network saturation | All stations hit cloud simultaneously |
| 100+ | Identity collision | Non-unique station names |

## Architecture at Each Scale

### 1 Station

A single machine running your test script. No special infrastructure needed.

```
[DUT] -> [Test Station] -> [TofuPilot Cloud]
```

### 10 Stations

At 10 stations, the bottleneck is configuration. You need a shared config source, consistent naming, and a way to push script updates.

```
[10 DUTs] -> [10 Test Stations] -> [TofuPilot Cloud]
                    |
              [Git repo for scripts]
```

### 100 Stations

At 100, you need a deployment pipeline, centralized monitoring, and staggered upload scheduling.

```
[100 DUTs] -> [100 Test Stations] -> [TofuPilot Cloud]
                    |                       |
             [CI/CD pipeline]        [Health dashboard]
             [Config server]
```

## Station Naming and Registration

Stations are identified by name. A collision corrupts per-station analytics.

```
{site}-{line}-{station_number}
# Examples:
taipei-line1-001
munich-eol-001
```

Set the station name as an environment variable:

```bash filename="/etc/environment"
TOFUPILOT_STATION_ID=taipei-line1-001
TOFUPILOT_API_KEY=tp_live_xxxxxxxxxxxx
```

Pick the scheme before the second station exists. Renaming later splits one station's history into two identities, and the old runs keep the old name forever.

## Test Script Distribution

### Option 1: Git Pull (1-20 stations)

```bash filename="/etc/cron.d/update-test-script"
0 6 * * * testuser cd /opt/testscripts && git pull origin main
```

Pros: Simple. Cons: Requires network access to Git host. Fails silently.

That silent failure is the real cost. A station that missed the pull keeps testing with old limits and reports nothing unusual, so you find out from diverging yield rather than from an error.

### Option 2: PyInstaller Binary (20-50 stations)

```bash filename="build_and_deploy.sh"
#!/bin/bash
pyinstaller --onefile test_main.py --name test_runner

for station in taipei-line1-{001..020}; do
  rsync -az dist/test_runner "${station}:/opt/testscripts/test_runner_new"
  ssh "${station}" "mv /opt/testscripts/test_runner_new /opt/testscripts/test_runner"
done
```

Pros: No Python on stations. Cons: Slower iteration, larger binary.

Note the write-then-rename: copying directly onto the running path can leave a station with a half-written binary. The loop also has no error handling, so a station that is offline is skipped without comment. Check the exit status per station if you use this.

### Option 3: Docker (50-100 stations)

```dockerfile filename="Dockerfile"
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY test_main.py .
CMD ["python", "test_main.py"]
```

```yaml filename="docker-compose.yml"
version: "3.9"
services:
  test_runner:
    image: your-registry/test-runner:latest
    restart: unless-stopped
    environment:
      - TOFUPILOT_STATION_ID
      - TOFUPILOT_API_KEY
    devices:
      - /dev/ttyUSB0:/dev/ttyUSB0
```

Update all stations:

```bash filename="deploy.sh"
#!/bin/bash
parallel-ssh -h stations.txt "docker compose pull && docker compose up -d"
```

Pin an explicit image tag rather than `latest` once you are past a handful of stations. With `latest`, two stations that pulled on different days run different code and nothing records which.

### Distribution Method Comparison

| Method | Stations | Python on station | Update speed | Rollback |
|--------|----------|-------------------|-------------|----------|
| Git pull | 1-20 | Yes | Fast | `git revert` |
| PyInstaller | 20-50 | No | Medium | Replace binary |
| Docker | 50-100 | No | Fast | Previous image tag |

If you deploy procedures from a Git push, this table is mostly moot: the build produces an immutable artifact per commit, every linked station picks it up, and rollback is selecting a previous deployment.

## Centralized Configuration

Never hardcode values that vary per station or product.

| Config tier | Scope | Example |
|-------------|-------|---------|
| Environment variable | Per station | `TOFUPILOT_STATION_ID`, `TOFUPILOT_API_KEY` |
| Config file | Per product | Voltage limits, timing thresholds |
| Test script | Per test phase | OpenHTF phase logic |

```yaml filename="config/product_v2.yaml"
voltage_rail_3v3:
  min: 3.2
  max: 3.4
voltage_rail_5v:
  min: 4.8
  max: 5.2
boot_time_ms:
  min: 0
  max: 3000
```

```python filename="test_main.py"
import yaml
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

with open("config/product_v2.yaml") as f:
    cfg = yaml.safe_load(f)


@htf.measures(
    htf.Measurement("voltage_3v3").in_range(
        cfg["voltage_rail_3v3"]["min"],
        cfg["voltage_rail_3v3"]["max"]
    ).with_units(units.VOLT)
)
def phase_power_check(test):
    voltage = read_voltage_rail("3v3")
    test.measurements.voltage_3v3 = voltage


def main():
    test = htf.Test(
        phase_power_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: input("Scan serial number: "))


if __name__ == "__main__":
    main()
```

Version the config file alongside the script. Limits that change without a version marker make results from before and after the change silently incomparable.

## Network Topology

Stations communicate only outbound. No inbound ports required.

| Destination | Port | Purpose |
|-------------|------|---------|
| `tofupilot.app` | 443 | Test result upload |
| Your Git host | 443 | Script updates |
| Your container registry | 443 | Image pulls |

At 100 stations, stagger uploads to avoid thundering herd:

```python filename="upload_utils.py"
import time
import random

def upload_with_jitter(upload_fn, max_jitter_seconds=10):
    time.sleep(random.uniform(0, max_jitter_seconds))
    upload_fn()
```

Jitter costs cycle time on every unit, so reach for it only if you actually see contention. Runs queue locally and sync when the network returns, which absorbs most bursts without help.

## Monitoring Station Health

Per-station yield, throughput, and test duration are tracked automatically. Filter analytics by station to spot degradation.

The single most useful query at scale is failure rate grouped by station. If one station sits at 12% while the rest are at 2%, the problem is that station's fixture or instrument, not your product.

### Station Health Checklist

| Check | Interval | Action on failure |
|-------|----------|-------------------|
| Heartbeat | 1 min | Alert on-call, check network |
| Test script version | On start | Auto-update via pipeline |
| Disk space | 1 hour | Archive old logs, alert if under 5 GB |
| USB fixture connectivity | Before each run | Fail test with clear error |

## Station Bootstrap Script

```bash filename="bootstrap.sh"
#!/bin/bash
set -e

STATION_ID=$1
API_KEY=$2

if [ -z "$STATION_ID" ] || [ -z "$API_KEY" ]; then
  echo "Usage: bootstrap.sh <station-id> <api-key>"
  exit 1
fi

echo "TOFUPILOT_STATION_ID=${STATION_ID}" >> /etc/environment
echo "TOFUPILOT_API_KEY=${API_KEY}" >> /etc/environment

curl -fsSL https://get.docker.com | sh

mkdir -p /opt/testrunner
cat > /opt/testrunner/docker-compose.yml << EOF
version: "3.9"
services:
  test_runner:
    image: your-registry/test-runner:latest
    restart: unless-stopped
    env_file: /etc/environment
EOF

docker compose -f /opt/testrunner/docker-compose.yml up -d
echo "Station ${STATION_ID} is running."
```

```bash filename="terminal"
sudo bash bootstrap.sh taipei-line1-042 tp_live_xxxxxxxxxxxx
```

Two things to tighten before using this on a real floor. The API key is passed as a command-line argument, which puts it in shell history and in the process list; read it from a file or a prompt instead. And appending to `/etc/environment` on a re-run duplicates the entries rather than replacing them.

### What Is Hardware Reliability Testing
URL: https://www.tofupilot.com/guides/what-is-hardware-reliability-testing-with-tofupilot
Hardware reliability testing predicts product lifespan under stress. Learn what it covers, common methods, and how to track results with TofuPilot.

Hardware reliability testing stresses a product beyond normal conditions to find failure modes before customers do. It predicts how long a product will last and which components fail first. This guide covers the main reliability test methods, how to build reliability test scripts in Python, and how to track results with TofuPilot.

## Why Reliability Testing Matters

Production test tells you a unit works right now. Reliability testing tells you it will still work in six months, two years, or ten years. The cost difference between finding a failure mode in the lab versus in the field is typically 10x to 100x.

Reliability testing is required or expected in:

- Medical devices (FDA 21 CFR Part 820)
- Aerospace (DO-160, MIL-STD-810)
- Automotive (AEC-Q100/Q200, IATF 16949)
- Consumer electronics (IEC 60068, JESD22)

## Common Reliability Test Methods

| Method | What It Does | Typical Duration |
|--------|-------------|-----------------|
| HALT (Highly Accelerated Life Test) | Ramps temperature and vibration until failure | 1-2 weeks |
| ALT (Accelerated Life Test) | Runs at elevated stress to predict field life | 2-8 weeks |
| Temperature cycling | Alternates hot/cold to stress solder joints | Days to weeks |
| Thermal shock | Rapid temperature transitions | Hours to days |
| Vibration (random/sine) | Simulates shipping and operational vibration | Hours to days |
| Humidity/bias | Powered operation at high humidity | 500-2000 hours |
| MTBF verification | Statistical sampling to validate predicted failure rate | Varies |

## HALT vs ALT

| Aspect | HALT | ALT |
|--------|------|-----|
| Goal | Find failure modes | Predict field life |
| Stress level | Beyond spec, until failure | Above spec, controlled acceleration |
| Sample size | 5-15 units | 20-50 units |
| Output | Failure modes and margins | Life prediction with confidence interval |
| When | EVT/DVT | DVT/PVT |

HALT is qualitative: you're looking for weak spots. ALT is quantitative: you're predicting how long the product lasts.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Define Reliability Check Phases

Reliability tests often run a functional check between stress cycles. This detects degradation over time.

```python filename="reliability_check.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(
        minimum=3.0, maximum=3.6,
        marginal_minimum=3.1, marginal_maximum=3.5,
    )
    .with_units(units.VOLT),
    htf.Measurement("current_draw_mA")
    .in_range(minimum=40, maximum=65)
    .with_units(units.MILLIAMPERE),
)
def phase_functional_check(test):
    """Run after each stress cycle to detect degradation."""
    test.measurements.output_voltage_V = 3.31
    test.measurements.current_draw_mA = 49.8


@htf.measures(
    htf.Measurement("insulation_resistance_MOhm")
    .in_range(minimum=100)
    .with_units(units.OHM),
)
def phase_insulation_check(test):
    """Verify insulation resistance hasn't degraded."""
    test.measurements.insulation_resistance_MOhm = 450.0
```

## Step 2: Log Each Cycle to TofuPilot

Run the functional check after every stress cycle and log it as a separate test run. This creates a timeline of measurements per unit that shows degradation trends.

```python filename="reliability_check.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_functional_check,
    phase_insulation_check,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan unit serial: "))
```

Run this script after each temperature cycle, vibration session, or time interval. Each execution creates a new test run linked to the same serial number.

## Step 3: Track Degradation in TofuPilot

TofuPilot tracks every run per serial number. Open the unit history to see:

- **Measurement trends** over time for each unit (voltage drift, current increase)
- **Marginal results** flagged before hard failures occur
- **Control charts** showing when a parameter starts trending toward a limit
- **Failure analysis** identifying which phase fails first across the test lot

This data feeds directly into reliability reports. Instead of pulling numbers from chamber logs and multimeter screenshots, the full test history lives in one place.

## When to Use Each Method

| Situation | Recommended Method |
|-----------|--------------------|
| Early prototype, need to find weak spots fast | HALT |
| Pre-production, need life prediction for datasheet | ALT |
| Qualifying a new supplier or component | Temperature cycling + functional check |
| Regulatory submission (medical, aerospace) | Per-standard test (DO-160, IEC 60068) |
| Field return investigation | Reproduce failure conditions, run functional checks |

Reliability testing is an investment. Start with HALT in EVT to find the obvious failures, then run ALT in DVT to quantify the margins. By PVT, your design should survive the stress levels your customers will see.

### How to Configure Test Environments
URL: https://www.tofupilot.com/guides/how-to-configure-test-environments-for-dev-qa-and-production
Learn how to manage separate test environments with environment variables, config files, and TofuPilot workspace separation.

Your test code shouldn't change between environments. The same script that runs on your dev bench needs to run in QA validation and on the production floor. What changes is the configuration: instrument addresses, measurement limits, TofuPilot workspace, logging levels.

This guide shows three approaches to environment-specific config and how to point each environment at its own TofuPilot workspace.

## The Problem

A single test script typically needs different values for:

- **Instrument addresses.** Your dev bench DMM is at `TCPIP::192.168.1.10` but production uses `GPIB0::22`.
- **Measurement limits.** Dev uses wider tolerances for debugging. Production uses datasheet limits.
- **TofuPilot workspace.** Dev runs shouldn't pollute production data.
- **Logging verbosity.** Debug logging in dev, errors only in production.
- **Fixture IDs.** Each station has its own fixture with unique calibration data.

Hardcoding any of these means editing code for every deployment. That's how bugs reach the production floor.

## Approach 1: Environment Variables with python-dotenv

The simplest approach. Create a `.env` file per environment and load it at startup.

```python filename=".env.dev"
TOFUPILOT_API_KEY=tp_dev_abc123
DMM_ADDRESS=TCPIP::192.168.1.10::INSTR
PSU_ADDRESS=TCPIP::192.168.1.11::INSTR
VOLTAGE_MIN=3.0
VOLTAGE_MAX=3.6
LOG_LEVEL=DEBUG
STATION_ID=DEV-BENCH-01
```

```python filename=".env.production"
TOFUPILOT_API_KEY=tp_prod_xyz789
DMM_ADDRESS=GPIB0::22::INSTR
PSU_ADDRESS=GPIB0::5::INSTR
VOLTAGE_MIN=3.2
VOLTAGE_MAX=3.4
LOG_LEVEL=ERROR
STATION_ID=PROD-LINE1-STN04
```

Load the right file based on an environment flag:

```python filename="config.py"
import os
from dotenv import load_dotenv

# Set TEST_ENV=dev|qa|production before running
env = os.getenv("TEST_ENV", "dev")
load_dotenv(f".env.{env}")

class Config:
    TOFUPILOT_API_KEY = os.getenv("TOFUPILOT_API_KEY")
    DMM_ADDRESS = os.getenv("DMM_ADDRESS")
    PSU_ADDRESS = os.getenv("PSU_ADDRESS")
    VOLTAGE_MIN = float(os.getenv("VOLTAGE_MIN", "3.2"))
    VOLTAGE_MAX = float(os.getenv("VOLTAGE_MAX", "3.4"))
    LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")
    STATION_ID = os.getenv("STATION_ID", "UNKNOWN")
```

Use it in your test:

```python filename="test_power.py"
import openhtf as htf
from tofupilot.openhtf import upload
from config import Config

@htf.measures(
    htf.Measurement("voltage_3v3").in_range(Config.VOLTAGE_MIN, Config.VOLTAGE_MAX),
)
def test_power_rail(test):
    # Use Config.DMM_ADDRESS to connect to the right instrument
    test.measurements.voltage_3v3 = 3.29

def main():
    test = htf.Test(test_power_rail)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "DUT-001")

if __name__ == "__main__":
    main()
```

Run with: `TEST_ENV=production python test_power.py`

## Approach 2: YAML Config Files

For more complex configurations with nested structures, YAML is cleaner than flat env vars.

```python filename="config/production.yaml"
instruments:
  dmm:
    address: "GPIB0::22::INSTR"
    timeout_ms: 5000
  psu:
    address: "GPIB0::5::INSTR"
    timeout_ms: 3000

limits:
  voltage_3v3:
    min: 3.2
    max: 3.4
  current_idle:
    max: 0.050

station:
  id: "PROD-LINE1-STN04"
  log_level: "ERROR"
```

```python filename="config/dev.yaml"
instruments:
  dmm:
    address: "TCPIP::192.168.1.10::INSTR"
    timeout_ms: 10000
  psu:
    address: "TCPIP::192.168.1.11::INSTR"
    timeout_ms: 10000

limits:
  voltage_3v3:
    min: 3.0
    max: 3.6
  current_idle:
    max: 0.100

station:
  id: "DEV-BENCH-01"
  log_level: "DEBUG"
```

Load with a simple helper:

```python filename="config_loader.py"
import os
import yaml
from pathlib import Path

def load_config(env: str = None) -> dict:
    """Load YAML config for the given environment."""
    env = env or os.getenv("TEST_ENV", "dev")
    config_path = Path(__file__).parent / "config" / f"{env}.yaml"

    if not config_path.exists():
        raise FileNotFoundError(f"No config file for environment: {env}")

    with open(config_path) as f:
        return yaml.safe_load(f)
```

```python filename="test_with_yaml.py"
import openhtf as htf
from tofupilot.openhtf import upload
from config_loader import load_config

cfg = load_config()
limits = cfg["limits"]["voltage_3v3"]

@htf.measures(
    htf.Measurement("voltage_3v3").in_range(limits["min"], limits["max"]),
)
def test_power_rail(test):
    test.measurements.voltage_3v3 = 3.29

def main():
    test = htf.Test(test_power_rail)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "DUT-001")

if __name__ == "__main__":
    main()
```

## Point to Different TofuPilot Workspaces

Each environment should write to its own TofuPilot workspace. This keeps dev noise out of production data.

Set the `TOFUPILOT_API_KEY` environment variable per environment. Each API key is scoped to a workspace. No code changes needed.

```python filename="run_test.sh"
# Dev
export TOFUPILOT_API_KEY="tp_dev_abc123"
python test_power.py

# QA
export TOFUPILOT_API_KEY="tp_qa_def456"
python test_power.py

# Production
export TOFUPILOT_API_KEY="tp_prod_xyz789"
python test_power.py
```

The TofuPilot client reads `TOFUPILOT_API_KEY` from the environment automatically. You don't need to pass it in code.

## Comparison: Env Vars vs Config Files vs CLI Args

| Approach | Best for | Pros | Cons |
|----------|----------|------|------|
| Environment variables | Simple configs, CI/CD, secrets | No files to deploy, works with Docker/systemd, secret-friendly | Flat structure only, no nesting |
| YAML config files | Complex configs, nested structures | Readable, supports comments, version-controllable | Need to deploy files, secrets in plaintext |
| CLI arguments | One-off overrides, debugging | No files needed, easy to test | Verbose for many params, not persistent |
| Combined (env + YAML) | Production deployments | Secrets in env, structure in YAML | Two systems to maintain |

The pragmatic approach: use YAML for instrument addresses and limits (things that vary by station), and environment variables for secrets (API keys, credentials). CLI args for quick overrides during debugging.

### Control Multiple Instruments in a Test
URL: https://www.tofupilot.com/guides/how-to-control-multiple-instruments-in-a-test-with-tofupilot
Control several instruments in one test: a plug per instrument, shared sessions, and how to avoid concurrency bugs when phases run in parallel.

A real test bench has more than one instrument: a supply, a meter, a scope, maybe a chamber and a bus interface. Getting several of them into one test is mostly about lifecycle and ownership, not about the SCPI. This guide covers the structure that scales, and the concurrency mistake that produces intermittent failures nobody can reproduce.

## One Plug per Instrument

Each instrument gets its own plug: a plain Python class that owns its session. The procedure lists them, and phases receive the ones they name.

```python filename="plugs/psu.py"
import pyvisa


class PowerSupply:
    RESOURCE = "TCPIP::192.168.1.101::INSTR"

    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 5000
        self.instr.write("*RST")

    def apply(self, channel: int, volts: float, limit: float):
        self.instr.write(f":APPL CH{channel},{volts},{limit}")
        self.instr.write(f":OUTP CH{channel},ON")

    def measure_current(self, channel: int) -> float:
        return float(self.instr.query(f":MEAS:CURR? CH{channel}"))

    def close(self):
        """Always de-energise, whatever happened during the run."""
        for ch in (1, 2, 3):
            try:
                self.instr.write(f":OUTP CH{ch},OFF")
            except Exception:
                pass
        self.instr.close()
```

```python filename="plugs/dmm.py"
import pyvisa


class Multimeter:
    RESOURCE = "TCPIP::192.168.1.100::INSTR"

    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 10000
        self.instr.write("*RST")

    def dc_voltage(self, range_v: float = 10, nplc: float = 1) -> float:
        self.instr.write(f":CONF:VOLT:DC {range_v}")
        self.instr.write(f":VOLT:DC:NPLC {nplc}")
        return float(self.instr.query(":READ?"))

    def close(self):
        self.instr.close()
```

```python filename="plugs/scope.py"
import pyvisa


class Oscilloscope:
    RESOURCE = "TCPIP::192.168.1.102::INSTR"

    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 15000
        self.instr.write("*RST")

    def measure_frequency(self, channel: int = 1) -> float:
        return float(self.instr.query(f":MEAS:FREQ? CHAN{channel}"))

    def close(self):
        self.instr.close()
```

Three separate classes, three separate sessions, three independent lifecycles. None of them knows about the others.

## Declaring Them

```yaml filename="procedure.yaml"
name: Multi-Instrument Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: psu
    python: plugs.psu:PowerSupply
  - name: dmm
    python: plugs.dmm:Multimeter
  - name: scope
    python: plugs.scope:Oscilloscope

main:
  - name: Power Up
    python: phases.power_up
    measurements:
      - name: Inrush Current
        unit: A
        validators:
          - operator: "<="
            expected_value: 0.45

  - name: Rail Measurement
    python: phases.rail_measurement
    depends_on: [power_up]
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4

  - name: Clock Check
    python: phases.clock_check
    depends_on: [power_up]
    measurements:
      - name: Clock Frequency
        unit: Hz
        validators:
          - operator: ">="
            expected_value: 7_990_000
          - operator: "<="
            expected_value: 8_010_000
```

A phase declares the instruments it needs by parameter name:

```python filename="phases/power_up.py"
def power_up(measurements, psu):
    psu.apply(channel=1, volts=5.0, limit=0.5)
    measurements.inrush_current = psu.measure_current(1)
```

```python filename="phases/rail_measurement.py"
def rail_measurement(measurements, dmm):
    measurements.rail_3v3 = dmm.dc_voltage()
```

```python filename="phases/clock_check.py"
def clock_check(measurements, scope):
    measurements.clock_frequency = scope.measure_frequency(1)
```

Run it with `tofupilot run ./procedure.yaml`.

Note what the phases do not do: no opening, no closing, no passing handles around. Each phase asks for what it needs and the engine supplies it.

## The Concurrency Mistake

Independent phases run in parallel. In the procedure above, `rail_measurement` and `clock_check` both depend on `power_up` but not on each other, so they run concurrently, and the run is as fast as the slower of the two.

That is a real benefit and also the trap. **Two phases running concurrently must not share a VISA session.** A single meter cannot serve two phases at once; you get interleaved commands, mismatched responses and timeouts that appear once a week.

Above, it is safe because the two concurrent phases use different instruments. If both needed the meter, you have two correct options.

**Serialise them.** Make one depend on the other:

```yaml filename="procedure.yaml"
  - name: Clock Check
    python: phases.clock_check
    depends_on: [rail_measurement]
```

**Or give each its own instrument.** Two meters, two plugs, no contention.

What does not work is sharing one session and hoping. The symptom is intermittent `VI_ERROR_TMO` or a reading that belongs to the other phase's query, which is far worse than a timeout because it silently records a wrong number.

## Instruments With Long Setup

A chamber that takes twenty minutes to reach setpoint should not be re-initialised per phase. The plug's `__init__` runs once for the whole run, so expensive setup naturally belongs there:

```python filename="plugs/chamber.py"
import time


class Chamber:
    def __init__(self):
        self._connect()
        self.set_temperature(25.0)
        self._wait_for_setpoint()

    def _connect(self):
        pass  # Replace with the real interface

    def set_temperature(self, celsius: float):
        self._setpoint = celsius

    def read_temperature(self) -> float:
        return 25.1

    def _wait_for_setpoint(self, timeout_s: float = 1800):
        deadline = time.time() + timeout_s
        while time.time() < deadline:
            if abs(self.read_temperature() - self._setpoint) < 1.0:
                return
            time.sleep(5)
        raise TimeoutError("Chamber did not reach setpoint")

    def close(self):
        self.set_temperature(25.0)
```

## Shared Instruments Across Fixture Slots

When one rack supply feeds four fixture slots, scope matters. A slot-scoped teardown powering down a shared supply while a neighbouring slot is still measuring is the classic multi-slot bug.

Shared equipment belongs in execution-scoped setup and teardown, which run once for the whole execution rather than per slot. Per-slot resources stay slot-scoped.

## Recording What Produced the Numbers

When benches disagree, the first question is which instruments were involved:

```python filename="phases/record_equipment.py"
def record_equipment(run, psu, dmm, scope):
    run.metadata["psu_id"] = psu.identify()
    run.metadata["dmm_id"] = dmm.identify()
    run.metadata["scope_id"] = scope.identify()
```

Reading `*IDN?` rather than typing asset tags means the record cannot drift from reality.

## Failure Isolation

One instrument failing to initialise fails the run, which is usually right: a test that silently skips the scope is worse than one that stops. If an instrument is genuinely optional, handle that inside the plug and record it as a measurement rather than letting the phase raise:

```python filename="phases/optional_scope.py"
def optional_scope(measurements, scope):
    measurements.scope_available = scope.is_connected()
```

That way the absence is data rather than an exception, and you can decide later whether it should have failed the unit.

## If You Already Run OpenHTF

Each instrument is a `BasePlug`, and phases declare what they need:

```python filename="multi_instrument.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class PsuPlug(BasePlug):
    def setUp(self):
        self.connected = True

    def measure_current(self) -> float:
        return 0.41

    def tearDown(self):
        self.connected = False


class DmmPlug(BasePlug):
    def dc_voltage(self) -> float:
        return 3.31


@htf.measures(
    htf.Measurement("inrush_current").in_range(maximum=0.45).with_units(units.AMPERE),
)
@htf.plug(psu=PsuPlug)
def test_power_up(test, psu):
    test.measurements.inrush_current = psu.measure_current()


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
@htf.plug(dmm=DmmPlug)
def test_rail(test, dmm):
    test.measurements.rail_3v3 = dmm.dc_voltage()


test = htf.Test(test_power_up, test_rail, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

OpenHTF phases run sequentially, so the concurrency caution does not apply there. Install with `pip install "tofupilot[openhtf]"`, or run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---|---|---|
| One session shared by concurrent phases | Interleaved commands, wrong readings | Serialise with `depends_on`, or one instrument each |
| Opening the instrument inside each phase | Slow, and leaks sessions on failure | Open once in the plug's `__init__` |
| Power-down as a final phase | Board stays energised when a phase raises | Put it in the plug's `close()` |
| Slot-scoped teardown on shared equipment | One slot powers down its neighbours | Execution scope for shared resources |
| Expensive setup repeated per phase | Cycle time dominated by chamber ramps | Do it once in the plug |
| No instrument identity on the run | Bench disagreements cannot be traced | Record `*IDN?` as run metadata |

### What Is Functional Testing for Hardware
URL: https://www.tofupilot.com/guides/what-is-functional-testing-for-hardware-with-tofupilot
Functional testing validates that hardware behaves correctly as a complete system. Learn the difference from ICT, how to build functional tests in Python.

Functional testing validates that a hardware product behaves correctly as a complete system. Unlike in-circuit testing (ICT), which checks individual components and solder joints, functional testing treats the board or product as a black box and verifies its inputs and outputs. This guide covers what functional testing involves, how it compares to other test methods, and how to build functional tests in Python with TofuPilot.

## Functional Testing vs Other Methods

| Method | What It Tests | How | Finds |
|--------|--------------|-----|-------|
| ICT (In-Circuit Test) | Individual components | Bed-of-nails fixture, powered | Opens, shorts, wrong values |
| Flying probe | Individual components | Moving probes, unpowered/powered | Same as ICT, no fixture needed |
| Functional test (FCT) | System behavior | Powered, exercised through interfaces | Integration bugs, firmware issues, performance |
| Boundary scan (JTAG) | Digital IC connections | Through JTAG chain | Digital connectivity |

ICT tells you the right parts are soldered correctly. Functional testing tells you the product actually works.

## What Functional Tests Cover

A functional test exercises the product through its real interfaces: power input, communication ports, sensors, actuators, and user controls.

| Category | Example Checks |
|----------|---------------|
| Power | Startup current, voltage regulation, power sequencing |
| Communication | UART, SPI, I2C, CAN, Ethernet respond correctly |
| Analog | ADC readings match known inputs within tolerance |
| Digital I/O | GPIO states match expected logic levels |
| Actuators | Motors, relays, LEDs respond to commands |
| Firmware | Version check, self-test pass, boot time |
| RF (if applicable) | Transmit power, receive sensitivity, frequency accuracy |

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Define Functional Test Phases

Each functional check becomes an OpenHTF phase. Group related checks together but keep phases focused enough that a failure points to a specific subsystem.

```python filename="functional_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("boot_time_ms")
    .in_range(maximum=2000)
    .with_units(units.MILLISECOND),
    htf.Measurement("firmware_version").equals("1.3.0"),
)
def phase_power_and_boot(test):
    """Power up the DUT and verify boot sequence."""
    test.measurements.boot_time_ms = 1240
    test.measurements.firmware_version = "1.3.0"


@htf.measures(
    htf.Measurement("uart_loopback").equals("PASS"),
    htf.Measurement("i2c_sensor_id").equals("0x68"),
)
def phase_communication(test):
    """Verify communication interfaces respond correctly."""
    test.measurements.uart_loopback = "PASS"
    test.measurements.i2c_sensor_id = "0x68"


@htf.measures(
    htf.Measurement("adc_channel_0_V")
    .in_range(minimum=2.45, maximum=2.55)
    .with_units(units.VOLT),
    htf.Measurement("adc_channel_1_V")
    .in_range(minimum=1.60, maximum=1.70)
    .with_units(units.VOLT),
)
def phase_analog_inputs(test):
    """Apply known voltages and verify ADC readings."""
    test.measurements.adc_channel_0_V = 2.50
    test.measurements.adc_channel_1_V = 1.65
```

## Step 2: Run and Log Results

Connect the test to TofuPilot. Every unit's functional test result uploads automatically with full measurement detail.

```python filename="functional_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_power_and_boot,
    phase_communication,
    phase_analog_inputs,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial number: "))
```

## Step 3: Track Results in TofuPilot

TofuPilot tracks functional test results automatically. Open the Analytics tab to see:

- **First pass yield** for each functional test procedure
- **Failure Pareto** showing which phases fail most often (power? communication? analog?)
- **Measurement distributions** with limit overlays to spot tightening trends
- **Station comparison** if you run the same test on multiple stations

This data helps you decide where to invest. If 80% of functional test failures come from one phase, that's where to focus your design or process improvement.

## When to Use Functional Testing

| Scenario | Use FCT? |
|----------|----------|
| PCBA with firmware | Yes, after SMT and programming |
| Simple passive board | No, ICT or flying probe is sufficient |
| Assembled product with sensors/actuators | Yes, as EOL test |
| Safety-critical product | Yes, with documented test procedure |
| High-volume, low-complexity | Maybe, depends on field failure cost |

Functional testing adds cycle time, but it catches the integration failures that ICT cannot. For products with firmware, communication interfaces, or analog circuits, it's the test stage that matters most.

### What Is Acceptance Testing for Hardware
URL: https://www.tofupilot.com/guides/what-is-acceptance-testing-with-tofupilot
Acceptance testing verifies a product meets agreed-upon criteria before delivery. Learn how to structure acceptance tests in Python and track results with.

Acceptance testing verifies that a product meets the criteria agreed upon between supplier and customer before delivery. It's the contractual gate: if the unit passes, the customer accepts it. This guide covers what acceptance testing involves for hardware, how to build acceptance test procedures in Python, and how to generate auditable records with TofuPilot.

## Types of Acceptance Testing

| Type | Who Runs It | Where | Purpose |
|------|------------|-------|---------|
| FAT (Factory Acceptance Test) | Supplier | Supplier's facility | Prove it works before shipping |
| SAT (Site Acceptance Test) | Customer or supplier | Customer's facility | Prove it works after installation |
| ATP (Acceptance Test Procedure) | Either | Either | The documented test procedure itself |

FAT and SAT often run the same test procedure but in different environments. The ATP defines the exact steps, measurements, and pass/fail criteria both parties agreed to.

## What Makes Acceptance Testing Different

Acceptance testing is not exploratory. Every step is pre-defined, and the customer typically reviews and approves the ATP before testing begins.

| Aspect | Acceptance Test | Production Test |
|--------|----------------|-----------------|
| Scope | Per contract requirements | Per design spec |
| Documentation | Formal report required | Internal records |
| Witness | Customer may be present | Internal only |
| Criteria | Contractually agreed limits | Engineering limits |
| Frequency | Per batch or per unit | Every unit |

The key difference: acceptance test limits come from the contract, not from engineering margins. They're often tighter than production limits because the customer specifies what they need, not what the design can do.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Define the Acceptance Criteria

Map each ATP requirement to an OpenHTF measurement with the contractual limits.

```python filename="acceptance_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_power_W")
    .in_range(minimum=95, maximum=105)
    .with_units(units.WATT),
    htf.Measurement("efficiency_percent")
    .in_range(minimum=92)
    .with_units(units.PERCENT),
)
def phase_power_output(test):
    """ATP Section 4.1: Output power and efficiency."""
    test.measurements.output_power_W = 100.3
    test.measurements.efficiency_percent = 94.7


@htf.measures(
    htf.Measurement("ripple_voltage_mV")
    .in_range(maximum=50)
    .with_units(units.MILLIVOLT),
)
def phase_output_quality(test):
    """ATP Section 4.2: Output ripple at full load."""
    test.measurements.ripple_voltage_mV = 32.1


@htf.measures(
    htf.Measurement("leakage_current_uA")
    .in_range(maximum=500)
    .with_units(units.MICROAMPERE),
)
def phase_safety(test):
    """ATP Section 4.3: Leakage current per IEC 60950."""
    test.measurements.leakage_current_uA = 120.0
```

## Step 2: Run the Acceptance Test

Connect the test to TofuPilot so every result is logged with the unit's serial number. This creates the traceability record the customer needs.

```python filename="acceptance_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_power_output,
    phase_output_quality,
    phase_safety,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan unit serial: "))
```

## Step 3: Generate Acceptance Records

TofuPilot stores every measurement, limit, and pass/fail result per serial number. Open the unit's test history to see:

- **Full test record** with all measurements against contractual limits
- **Pass/fail status** for each ATP section
- **Timestamps** showing exactly when each test ran
- **Test reports** exportable for customer review and audits

This replaces the manual process of filling out ATP datasheets and scanning signed paper forms. The customer gets the same data in a format that's searchable and auditable.

## Writing a Good ATP

A strong acceptance test procedure includes:

| Section | Content |
|---------|---------|
| Scope | What product, what revision, what contract |
| Equipment list | Instruments, fixtures, software versions |
| Environment | Temperature, humidity, power conditions |
| Test steps | Numbered, with exact setup and expected result |
| Pass/fail criteria | Numeric limits traceable to contract requirements |
| Data recording | What gets recorded, where, and in what format |

Keep the ATP stable. Changes require customer approval. Version-control the ATP alongside your test scripts in Git so the test procedure and the code stay in sync.

### What Is Process Capability (Cp and Cpk)
URL: https://www.tofupilot.com/guides/what-is-process-capability-cp-and-cpk-with-tofupilot
Understand Cp and Cpk formulas, what different Cpk values mean for your process, and how TofuPilot calculates them from your test data.

Process capability tells you whether your manufacturing process can consistently produce units within specification limits. Cp measures the potential, Cpk measures the reality. Together they reveal whether your process is capable and centered.

## The Formulas

**Cp** measures how much of the specification window your process variation uses:

**Cp = (USL - LSL) / 6σ**

Where USL is the upper specification limit, LSL is the lower specification limit, and σ is the process standard deviation.

Cp only looks at spread. It ignores whether your process is centered between the limits. That's where Cpk comes in.

**Cpk = min((USL - μ) / 3σ, (μ - LSL) / 3σ)**

Where μ is the process mean. Cpk takes the worse side. If your process drifts toward one limit, Cpk drops even if Cp stays high.

## What Cpk Values Mean

| Cpk Value | Interpretation | Defect Rate (approx.) |
|-----------|---------------|----------------------|
| < 1.0 | Process is not capable. Units are falling outside limits. | > 2,700 ppm |
| 1.0 | Barely capable. The 3-sigma edge touches the spec limit. | ~2,700 ppm |
| 1.0 - 1.33 | Marginally capable. Acceptable for non-critical dimensions. | 2,700 - 63 ppm |
| 1.33 | Commonly accepted minimum for production processes. | ~63 ppm |
| 1.33 - 1.67 | Good capability. Reasonable margin for process drift. | 63 - 0.6 ppm |
| > 1.67 | Excellent. Process has significant margin within spec. | < 0.6 ppm |

A practical way to think about it: Cpk 1.33 means your process uses 75% of the specification window, leaving 25% as buffer for drift and variation. Most automotive and aerospace standards require Cpk >= 1.33 for production, with 1.67 for safety-critical parameters.

## The Relationship Between Cp and Cpk

When Cp equals Cpk, your process is perfectly centered between the specification limits. When Cpk is significantly lower than Cp, your process has drifted off-center.

| Scenario | Cp | Cpk | Diagnosis |
|----------|-----|------|-----------|
| Centered and capable | 1.5 | 1.5 | Ideal state |
| Off-center but capable spread | 1.5 | 0.9 | Process mean has drifted. Re-center it. |
| Wide variation, centered | 0.8 | 0.8 | Too much variation. Reduce σ. |
| Wide variation, off-center | 0.8 | 0.4 | Both problems. Fix variation first. |

This distinction matters for corrective action. If Cp is fine but Cpk is low, you need to adjust the process mean (a calibration issue). If Cp itself is low, you need to reduce variation (a fundamental process issue).

## Feeding Measurement Data to TofuPilot

For TofuPilot to calculate Cpk, your test code needs to define measurements with upper and lower limits. Here's an OpenHTF test that measures critical parameters on a sensor module:

```python filename="test_sensor_calibration.py"
# Sensor calibration test with limits for Cpk analysis
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("output_voltage")
        .in_range(minimum=2.45, maximum=2.55)
        .with_units(units.VOLT),
    htf.Measurement("offset_voltage")
        .in_range(minimum=-5.0, maximum=5.0),
    htf.Measurement("sensitivity")
        .in_range(minimum=195, maximum=205),
    htf.Measurement("noise_floor")
        .in_range(maximum=500),
    htf.Measurement("response_time")
        .in_range(maximum=10.0),
)
def sensor_validation(test):
    test.measurements.output_voltage = 2.501
    test.measurements.offset_voltage = 0.8
    test.measurements.sensitivity = 200.3
    test.measurements.noise_floor = 320
    test.measurements.response_time = 4.7

def main():
    test = htf.Test(sensor_validation)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SENSOR-2024-1587")

if __name__ == "__main__":
    main()
```

The `in_range()` calls define USL and LSL. When you set `minimum` and `maximum`, TofuPilot gets both limits and can compute Cp and Cpk. When you set only `maximum`, TofuPilot computes a one-sided capability index.

## Reading Cpk in TofuPilot

Once enough test runs accumulate (typically 30+ data points for statistical significance), TofuPilot's measurement analytics show you:

- **Cpk value per measurement** calculated from your production data. You can see at a glance which measurements are capable and which aren't.
- **Measurement histograms** overlaid with specification limits, so you can visually assess distribution shape, centering, and spread.
- **Control charts** that track measurement values over time, revealing drift before it causes Cpk to drop below your threshold.

You don't need to export data and calculate Cpk in a spreadsheet. Define your limits in OpenHTF, run your tests, and TofuPilot computes the rest.

## Common Pitfalls

**Not enough data.** Cpk calculated from 10 units is unreliable. Wait for at least 30 data points, ideally 50+, before making process decisions based on Cpk.

**Confusing specification limits with control limits.** Specification limits (USL/LSL) come from your product requirements. Control limits come from your process data. Cpk uses specification limits.

**Ignoring non-normal distributions.** The standard Cpk formula assumes a normal distribution. If your measurement data is skewed (common with one-sided specs like noise floor), the standard formula may overestimate capability.

**Mixing populations.** If you calculate Cpk across data from two different test stations with different calibrations, the combined σ will be inflated. Check per-station Cpk first.

### Monitor Test Stations Across Factories
URL: https://www.tofupilot.com/guides/how-to-monitor-test-stations-across-factories-with-tofupilot
Track test station performance across multiple factories from a single dashboard. Compare yield, throughput, and measurement drift by site and station.

When the same test procedure runs on dozens of stations across multiple factories, you need a single place to see what's happening everywhere. TofuPilot gives you cross-factory visibility without building custom dashboards or aggregating CSVs.

## Why Multi-Site Monitoring Matters

A test that passes 98% at your Shenzhen factory and 94% at your Guadalajara factory tells you something is wrong. The problem could be equipment calibration, operator training, environmental conditions, or component lot variation. You can't fix what you can't see.

Station-level data also helps you spot individual machines drifting before they start producing false passes or unnecessary failures.

## Tag Each Station with Metadata

OpenHTF lets you attach metadata to every test run. Use `station_id` to identify the physical station and add factory or site information so TofuPilot can group results.

```python filename="test_board.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("supply_voltage")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT)
)
def test_supply_voltage(test):
    voltage = 5.03  # Read from instrument
    test.measurements.supply_voltage = voltage

def main():
    test = htf.Test(
        test_supply_voltage,
        station_id="SMT-LINE3-ICT-02",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PCB-2026-00451")

if __name__ == "__main__":
    main()
```

The `station_id` value should be unique and descriptive. A good convention is `{SITE}-{LINE}-{FUNCTION}-{NUMBER}`, like `SZ-L3-ICT-02` for Shenzhen, Line 3, In-Circuit Test, Station 2.

## Standardize Station Naming Across Sites

Consistent naming is critical. If one factory uses `ICT_01` and another uses `ict-station-1`, you can't filter or compare them reliably.

Define a naming convention before deployment:

| Field | Format | Example |
|-------|--------|---------|
| Site code | 2-3 letter abbreviation | `SZ`, `GDL`, `AUS` |
| Line number | `L` + number | `L1`, `L3` |
| Test type | Standard abbreviation | `ICT`, `FCT`, `EOL` |
| Station number | Zero-padded number | `01`, `02`, `12` |

This gives you station IDs like `SZ-L3-ICT-02` or `GDL-L1-FCT-05` that are readable and sortable.

## Run the Same Procedure Everywhere

Multi-site monitoring only works if every factory runs the same test procedure. Version-control your test scripts in Git and deploy them to all sites from a single repository.

```python filename="test_final.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("output_current")
    .in_range(minimum=0.95, maximum=1.05)
    .with_units(units.AMPERE),
    htf.Measurement("firmware_checksum")
    .equals("a3f8c2d1")
)
def test_output_and_firmware(test):
    test.measurements.output_current = 1.01  # Read from load
    test.measurements.firmware_checksum = "a3f8c2d1"  # Read from DUT

def main():
    test = htf.Test(
        test_output_and_firmware,
        station_id="AUS-L2-FCT-01",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "UNIT-88210")

if __name__ == "__main__":
    main()
```

When every station uploads results from the same procedure, TofuPilot automatically groups them. You can then filter by station, site, or time range to compare performance.

## View Cross-Factory Data in TofuPilot

Once runs are uploading from multiple sites, TofuPilot's station dashboard shows:

- **Per-station yield** so you can spot underperforming machines instantly
- **Measurement distributions by station** to catch calibration drift before it causes failures
- **Throughput by station** to identify bottlenecks and downtime
- **Failure Pareto by site** to see whether failure modes differ between factories

Filter by station ID prefix (like `SZ-` or `GDL-`) to compare entire factories side by side. Drill into individual stations to investigate anomalies.

## Handle Station Offline Detection

A station that stops reporting is a problem. It might be down for maintenance, or it might be running tests that aren't uploading. TofuPilot tracks when each station last reported, so gaps in data are visible immediately.

Build a habit of checking the station overview at the start of each shift. If a station hasn't reported in the expected window, investigate before production continues.

### Track Rolled Throughput Yield
URL: https://www.tofupilot.com/guides/how-to-track-rolled-throughput-yield-with-tofupilot
Learn how to measure Rolled Throughput Yield (RTY) across multi-step manufacturing processes and track it in TofuPilot's dashboard.

Final yield only tells you whether a unit passed at the end of the line. Rolled Throughput Yield (RTY) tells you the probability that a unit passes every step without rework or retesting. It's the metric that exposes hidden factory losses.

## Why RTY Matters More Than Final Yield

Final yield counts a unit as "pass" even if it failed three times before someone reworked it. RTY doesn't. It multiplies the first-pass yield of every process step together:

**RTY = FPY₁ x FPY₂ x ... x FPYₙ**

Consider a five-step process where each step has 95% FPY. Final yield might show 99% because rework catches most failures. But RTY tells the real story:

**RTY = 0.95 x 0.95 x 0.95 x 0.95 x 0.95 = 77.4%**

That means nearly 1 in 4 units needed rework somewhere. Each rework event costs time, labor, and materials that final yield hides completely.

## RTY Interpretation

| RTY Range | What It Means |
|-----------|---------------|
| > 95% | Process is well-controlled, minimal hidden rework |
| 85-95% | Some steps need attention, review failure Pareto |
| 70-85% | Significant hidden losses, prioritize worst FPY steps |
| < 70% | Process is unstable, rework costs are likely substantial |

The power of RTY is that it pinpoints which step drags down the whole line. If Step 3 has 88% FPY while others sit at 98%, you know exactly where to focus.

## Structuring Your Tests for RTY Tracking

Each process step should be a separate test procedure in TofuPilot. This gives you per-step FPY that feeds directly into RTY calculations. Here's a multi-phase OpenHTF test that represents a PCB assembly process with distinct steps:

```python filename="test_smt_placement.py"
# Step 1: SMT component placement verification
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("placement_offset_x")
        .in_range(minimum=-0.05, maximum=0.05)
        .with_units(units.MILLIMETRE),
    htf.Measurement("placement_offset_y")
        .in_range(minimum=-0.05, maximum=0.05)
        .with_units(units.MILLIMETRE),
    htf.Measurement("component_rotation")
        .in_range(minimum=-2.0, maximum=2.0),
)
def smt_placement_check(test):
    test.measurements.placement_offset_x = 0.02
    test.measurements.placement_offset_y = -0.01
    test.measurements.component_rotation = 0.5

def main():
    test = htf.Test(smt_placement_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PCB-2024-0042")

if __name__ == "__main__":
    main()
```

```python filename="test_reflow_solder.py"
# Step 2: Reflow soldering thermal profile check
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("peak_temperature")
        .in_range(minimum=235, maximum=250)
        .with_units(units.DEGREE_CELSIUS),
    htf.Measurement("time_above_liquidus")
        .in_range(minimum=60, maximum=120),
    htf.Measurement("cooling_rate")
        .in_range(maximum=3.0),
)
def reflow_profile_check(test):
    test.measurements.peak_temperature = 242
    test.measurements.time_above_liquidus = 85
    test.measurements.cooling_rate = 2.1

def main():
    test = htf.Test(reflow_profile_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PCB-2024-0042")

if __name__ == "__main__":
    main()
```

```python filename="test_functional.py"
# Step 3: Functional test after assembly
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("supply_voltage")
        .in_range(minimum=4.9, maximum=5.1)
        .with_units(units.VOLT),
    htf.Measurement("quiescent_current")
        .in_range(maximum=15.0)
        .with_units(units.AMPERE),
    htf.Measurement("clock_frequency")
        .in_range(minimum=7.99, maximum=8.01)
        .with_units(units.HERTZ),
)
def functional_check(test):
    test.measurements.supply_voltage = 5.02
    test.measurements.quiescent_current = 0.0113
    test.measurements.clock_frequency = 8003000

def main():
    test = htf.Test(functional_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PCB-2024-0042")

if __name__ == "__main__":
    main()
```

Each test procedure runs independently and uploads results to TofuPilot with the same serial number. TofuPilot links all three procedures to the same unit, giving you a complete process history.

## Tracking RTY in TofuPilot

Once your test procedures upload results, TofuPilot's dashboard gives you what you need to compute and track RTY:

- **Per-procedure FPY trends** show first-pass yield for each process step over time. You can spot degradation in a single step before it affects final yield.
- **Failure Pareto charts** break down which measurements fail most often within each step, so you can prioritize root cause analysis.
- **Unit history** shows the complete journey of any serial number across all procedures, including retests. This is the ground truth for whether a unit passed first time at every step.

You don't need to build RTY calculations in Python. Structure your tests as separate procedures, upload results with consistent serial numbers, and TofuPilot gives you the per-step visibility that makes RTY actionable.

## Key Practices for Accurate RTY

1. **One procedure per process step.** Don't combine SMT placement and reflow soldering into a single test. Separate procedures give you separate FPY numbers.

2. **Use consistent serial numbers.** Every procedure for the same unit must reference the same serial number. This is how TofuPilot links the full process chain.

3. **Don't filter out retests.** Upload every test execution, including failures and retests. RTY only works when you have the complete picture.

4. **Set measurement limits in code.** Limits defined in OpenHTF flow directly into TofuPilot's analytics. Don't rely on post-hoc analysis to determine pass/fail.

### Track Serial Numbers and Sub-Assemblies
URL: https://www.tofupilot.com/guides/how-to-track-serial-numbers-and-sub-assemblies-with-tofupilot
Record parent/child serial number relationships in your test data to trace sub-assemblies across boards, modules, and final products.

Most hardware products contain sub-assemblies, each with its own serial number. The `sub_units` field lets you record which components went into which parent unit, giving you full BOM traceability from a single test script.

## Why Sub-Assembly Tracking Matters

A finished product might contain a power supply board, a compute module, and a sensor array. Each has its own serial number and its own test history. When a field failure points to a specific component, you need to know which parent units contain that component.

This is standard practice for IPC-1782 traceability and required in regulated industries. Parent and child serial numbers link automatically when you include them in your test runs.

This is also the hardest thing to add retroactively. Reconstructing which board went into which assembly after the fact means reading build records by hand, if they exist at all. Model it from the first build even if you do not query it yet.

## Recording Sub-Unit Serial Numbers

`sub_units` is a field on `htf.Test(...)`, alongside `procedure_id` and `part_number`. It takes a list of `{"serial_number": "..."}` entries.

```python filename="test_assembly.py"
# Record sub-assembly serial numbers during final assembly test
import openhtf as htf
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("system_power_on").in_range(minimum=1, maximum=1),
    htf.Measurement("communication_check").in_range(minimum=1, maximum=1),
)
def system_integration_test(test):
    test.measurements.system_power_on = 1
    test.measurements.communication_check = 1


def main():
    test = htf.Test(
        system_integration_test,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="ASSY-100",
        sub_units=[
            {"serial_number": "PSU-2026-0441"},
            {"serial_number": "CPU-2026-1187"},
            {"serial_number": "SNS-2026-0893"},
        ],
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "ASSY-2026-0072")


if __name__ == "__main__":
    main()
```

After this test uploads, the unit page for `ASSY-2026-0072` shows three linked sub-assemblies. Click any sub-unit serial to see its own test history.

## Scanning Sub-Unit Serials During Assembly

In practice, operators scan component serial numbers as they install them. Because `sub_units` is a constructor argument, collect the scans before building the test:

```python filename="test_scan_subunits.py"
# Operator scans sub-assembly serials during assembly
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("power_rail_5v").in_range(minimum=4.8, maximum=5.2).with_units(units.VOLT),
    htf.Measurement("power_rail_3v3").in_range(minimum=3.1, maximum=3.5).with_units(units.VOLT),
)
def power_validation(test):
    test.measurements.power_rail_5v = 5.02
    test.measurements.power_rail_3v3 = 3.31


def main():
    psu = input("Scan PSU serial: ")
    cpu = input("Scan CPU serial: ")

    test = htf.Test(
        power_validation,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        part_number="ASSY-100",
        sub_units=[
            {"serial_number": psu},
            {"serial_number": cpu},
        ],
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: input("Scan assembly serial: "))


if __name__ == "__main__":
    main()
```

Scanning before the test starts also means a mis-scan is caught before you spend cycle time on measurements. Validate the serial format at the prompt rather than discovering a typo in the traceability record months later.

For a station with a real operator interface, replace `input()` with `openhtf.plugs.user_input` so the prompts appear on the operator screen instead of a console.

## Multi-Level Assemblies

For products with nested assemblies (a module inside a board inside a chassis), test each level separately with its own sub-units. The hierarchy builds automatically.

```python filename="test_nested_assembly.py"
# Test a module, then test the board that contains it
import openhtf as htf
from tofupilot.openhtf import upload

MODULE_PROCEDURE_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
BOARD_PROCEDURE_ID = "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"


@htf.measures(
    htf.Measurement("module_self_test").in_range(minimum=1, maximum=1),
)
def module_test(test):
    test.measurements.module_self_test = 1


@htf.measures(
    htf.Measurement("board_communication").in_range(minimum=1, maximum=1),
)
def board_test(test):
    test.measurements.board_communication = 1


def main():
    # First: test the module by itself
    module_serial = "MOD-2026-0551"
    t1 = htf.Test(
        module_test,
        procedure_id=MODULE_PROCEDURE_ID,
        part_number="MOD-100",
    )
    t1.add_output_callbacks(upload())
    t1.execute(lambda: module_serial)

    # Second: test the board, declaring the module as a sub-unit
    board_serial = "BRD-2026-0112"
    t2 = htf.Test(
        board_test,
        procedure_id=BOARD_PROCEDURE_ID,
        part_number="BRD-100",
        sub_units=[{"serial_number": module_serial}],
    )
    t2.add_output_callbacks(upload())
    t2.execute(lambda: board_serial)


if __name__ == "__main__":
    main()
```

Searching for the board serial shows the module as a sub-unit. Searching for the module serial shows its own test results plus the parent board it was installed in.

In production these two tests are usually separate scripts on separate stations, run hours apart. They are together here only to show both halves of the link.

## BOM Traceability Use Cases

Once sub-assembly links exist, you can answer questions that matter in production:

- **Field failure investigation.** A sensor module fails in the field. Search its serial number to find every parent unit that contains the same module type from the same production batch.
- **Component recall.** A supplier flags a batch of capacitors. If your test phases record component lot numbers as measurements, you can trace which assemblies used them.
- **Yield by component.** First pass yield broken down by sub-assembly batch shows whether a specific component lot is driving failures.

The recall case is the one that justifies the whole exercise. Without the links you scrap or recall everything built in a date range; with them you scrap only the units that actually contain the suspect lot.

## Viewing Sub-Assembly Links

Open any unit's page to see its sub-assembly tree:

- Direct child components with their serial numbers
- Each child's own test history (click through to view)
- The parent assembly, if this unit is itself a sub-component
- All test runs for the unit, across every production stage

This gives quality engineers and auditors a single place to trace any component through the entire product hierarchy.

### Use Control Charts for Tests
URL: https://www.tofupilot.com/guides/how-to-use-control-charts-for-tests-with-tofupilot
Learn how to use control charts to monitor test process stability, distinguish control limits from spec limits, and detect out-of-control conditions with.

Control charts show whether your test process is stable or drifting out of control. They're the foundation of Statistical Process Control (SPC) and let you catch problems before they cause failures.

## What Control Charts Show

A control chart plots individual measurements (or subgroup statistics) over time against three lines: a center line (process mean), an Upper Control Limit (UCL), and a Lower Control Limit (LCL). These limits are calculated from your actual data, typically at 3-sigma from the mean.

Points inside the control limits mean your process is behaving normally. Points outside, or patterns like seven consecutive points on one side of the center line, signal something has changed.

## Control Limits vs. Spec Limits

This distinction matters. Spec limits (USL/LSL) define what's acceptable for the product. Control limits (UCL/LCL) describe what your process actually does.

A process can be in control but out of spec (consistently producing bad parts). It can also be in spec but out of control (passing today, but unpredictably). Control charts catch the second case, which spec limits alone miss.

## Writing Tests That Feed Control Charts

Good control chart data starts with well-structured measurements. Each measurement needs a name, a unit, and consistent conditions across runs.

```python filename="control_chart_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("supply_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=4.75, maximum=5.25),
    htf.Measurement("supply_current")
    .with_units(units.AMPERE)
    .in_range(minimum=0.090, maximum=0.110),
    htf.Measurement("output_frequency")
    .with_units(units.HERTZ)
    .in_range(minimum=999.5, maximum=1000.5),
)
def power_supply_check(test):
    test.measurements.supply_voltage = 5.02
    test.measurements.supply_current = 0.0987
    test.measurements.output_frequency = 1000.1

@htf.measures(
    htf.Measurement("signal_amplitude")
    .in_range(minimum=-1.0, maximum=1.0),
    htf.Measurement("signal_noise_floor")
    .in_range(maximum=-60.0),
)
def signal_integrity_check(test):
    test.measurements.signal_amplitude = 0.3
    test.measurements.signal_noise_floor = -72.5

def main():
    test = htf.Test(power_supply_check, signal_integrity_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PCB-001")

if __name__ == "__main__":
    main()
```

Each run uploads measurements to TofuPilot. Over time, these build the dataset that control charts need.

## Choosing the Right Chart Type

For individual measurements (one value per run), use an I-MR chart (Individual and Moving Range). This is the most common case in electronics testing where each DUT produces one reading per measurement.

For subgrouped data (multiple samples per batch), X-bar and R charts track the subgroup mean and range. This applies when you test multiple units from the same production lot and want to monitor lot-to-lot variation.

## Reading Control Charts in TofuPilot

TofuPilot's measurement analytics automatically generates control charts from your test data. Open any measurement's detail view to see the time-series plot with calculated control limits.

Watch for these signals:

- A single point beyond UCL or LCL
- Seven or more consecutive points above or below the center line
- Six or more consecutive points trending in one direction
- Alternating patterns (up-down-up-down) suggesting measurement system issues

When you spot any of these, investigate the root cause before the process produces out-of-spec parts.

## Keeping Charts Useful

Recalculate control limits periodically, especially after process changes. If you've fixed a root cause and the process has genuinely improved, update the baseline. Stale limits hide real shifts and generate false alarms.

Group your charts by test station. Mixing data from different stations, fixtures, or environments inflates variation and makes the charts less sensitive to real changes on any single station.

### Log HIL Results for Regression Analysis
URL: https://www.tofupilot.com/guides/how-to-log-hil-test-results-to-tofupilot-for-regression-analysis
Structure HIL test results with firmware versions and environmental data for regression tracking in TofuPilot.

HIL tests generate a lot of data: measurements, waveform captures, firmware versions, environmental conditions. Without structured logging, spotting regressions across firmware releases turns into a spreadsheet archaeology project. This guide shows how to tag, attach, and query HIL results so regressions surface automatically.

## Prerequisites

- Python 3.9+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python client (`pip install "tofupilot[openhtf]"`)
- A working HIL test (see [How to Set Up HIL Testing for Embedded Systems with TofuPilot](/guides/how-to-set-up-hil-testing-for-embedded-systems-with-python-and-tofupilot))

## Step 1: Tag Runs with Firmware Version and Hardware Revision

Every HIL test run should carry the firmware version and hardware revision of the DUT. Record them as measurements so you can filter and query them later.

Query the firmware version from the DUT at the start of the test.

```python filename="hil_regression/main.py"
import openhtf as htf
from tofupilot.openhtf import upload

from hil_regression.phases import (
    read_firmware_info,
    test_adc_accuracy,
    test_pwm_output,
    test_sleep_current,
)


def main():
    test = htf.Test(
        read_firmware_info,
        test_adc_accuracy,
        test_pwm_output,
        test_sleep_current,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="DUT-100",
    )

    test.add_output_callbacks(upload())
    test.execute(lambda: input("Scan DUT serial number: "))


if __name__ == "__main__":
    main()
```

The firmware version and hardware revision get recorded as measurements, making them searchable across all runs.

```python filename="hil_regression/phases.py"
import openhtf as htf
from hil_regression.serial_plug import SerialCommandPlug


@htf.plug(serial=SerialCommandPlug)
@htf.measures(
    htf.Measurement("firmware_version"),
    htf.Measurement("hardware_revision"),
    htf.Measurement("bootloader_version"),
)
def read_firmware_info(test, serial):
    """Record firmware and hardware identifiers for traceability."""
    test.measurements.firmware_version = serial.send_command("VERSION")
    test.measurements.hardware_revision = serial.send_command("HWREV")
    test.measurements.bootloader_version = serial.send_command("BLVER")
```

This gives you three filterable fields on every run. When first pass yield drops after a firmware update, filter by `firmware_version` to isolate the change.

Put this phase first. If it fails, you have a run whose results cannot be attributed to a firmware version, which is worse than no run at all for regression work.

## Step 2: Attach Waveform and Log Files

HIL tests often capture oscilloscope waveforms, logic analyzer traces, or DUT console logs. Attach these to the test run so they're available for post-mortem analysis without digging through file shares.

```python filename="hil_regression/capture.py"
import csv
import time
import openhtf as htf
from openhtf.plugs import BasePlug


class WaveformCapturePlug(BasePlug):
    """Captures analog samples and saves to CSV for attachment."""

    def setUp(self):
        pass

    def capture_waveform(self, adc, channel: int, duration_s: float, sample_rate_hz: int) -> str:
        """Sample an ADC channel and write results to a CSV file."""
        filepath = f"/tmp/waveform_ch{channel}_{int(time.time())}.csv"
        samples = []
        interval = 1.0 / sample_rate_hz

        for i in range(int(duration_s * sample_rate_hz)):
            voltage = adc.read_voltage(channel)
            samples.append((i * interval, voltage))
            time.sleep(interval)

        with open(filepath, "w", newline="") as f:
            writer = csv.writer(f)
            writer.writerow(["time_s", "voltage_v"])
            writer.writerows(samples)

        return filepath

    def tearDown(self):
        pass
```

Then attach the file in your test phase:

```python filename="hil_regression/phases.py"
import openhtf as htf
from openhtf.util import units
from hil_regression.capture import WaveformCapturePlug
from hil_regression.analog_plug import AnalogInputPlug
from hil_regression.serial_plug import SerialCommandPlug


@htf.plug(serial=SerialCommandPlug, adc=AnalogInputPlug, capture=WaveformCapturePlug)
@htf.measures(
    htf.Measurement("pwm_voltage_mean").in_range(minimum=1.6, maximum=1.7).with_units(units.VOLT),
)
def test_pwm_output(test, serial, adc, capture):
    """Command 50% PWM and capture the output waveform."""
    serial.send_command("PWM SET 50")

    # Capture 1 second of data at 1 kHz
    waveform_path = capture.capture_waveform(adc, channel=3, duration_s=1.0, sample_rate_hz=1000)

    # Attach the waveform CSV to the test run
    test.attach("pwm_waveform", waveform_path, "text/csv")

    # Also record the mean voltage as a measurement
    import csv
    with open(waveform_path) as f:
        reader = csv.DictReader(f)
        voltages = [float(row["voltage_v"]) for row in reader]
    test.measurements.pwm_voltage_mean = sum(voltages) / len(voltages)
```

Attachments show up in the run detail view. You can download them later for offline analysis or comparison across firmware versions.

Note that the mean is the measurement and the waveform is the attachment. That split matters: the number is what trends and alerts, while the raw capture is what you open once a trend has already told you where to look. Attaching a full capture on every unit at volume will outgrow its usefulness quickly, so consider attaching only on failure or on a sample.

## Step 3: Use Sub-Units for Multi-Board HIL Setups

Many products contain multiple boards (main CPU board, power board, sensor board) that get tested together in a HIL fixture. Sub-units let you track each board independently while keeping them linked to the parent assembly.

`sub_units` is a field on `htf.Test(...)`, alongside `procedure_id` and `part_number`:

```python filename="hil_regression/multi_board.py"
import openhtf as htf
from tofupilot.openhtf import upload

from hil_regression.phases import (
    test_cpu_board,
    test_power_board,
    test_sensor_board,
)


def main():
    test = htf.Test(
        test_cpu_board,
        test_power_board,
        test_sensor_board,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        part_number="ASSY-100",
        sub_units=[
            {"serial_number": "CPU-BRD-0042", "part_number": "PCB-CPU-R3"},
            {"serial_number": "PWR-BRD-0019", "part_number": "PCB-PWR-R2"},
            {"serial_number": "SNS-BRD-0088", "part_number": "PCB-SNS-R1"},
        ],
    )

    test.add_output_callbacks(upload())
    test.execute(lambda: input("Scan assembly serial number: "))


if __name__ == "__main__":
    main()
```

Each sub-unit gets its own traceability record. If the sensor board starts failing after a hardware revision change, you can filter by `PCB-SNS-R1` and see exactly when failures started.

The serial numbers are hardcoded here to keep the example short. On a real fixture they come from barcode scans collected before the test is built, since `sub_units` is a constructor argument.

```python filename="hil_regression/board_phases.py"
import openhtf as htf
from openhtf.util import units
from hil_regression.analog_plug import AnalogInputPlug
from hil_regression.serial_plug import SerialCommandPlug


@htf.plug(serial=SerialCommandPlug)
@htf.measures(
    htf.Measurement("cpu_clock_mhz").in_range(minimum=79, maximum=81).with_units(units.HERTZ),
    htf.Measurement("cpu_temp").in_range(maximum=85).with_units(units.DEGREE_CELSIUS),
)
def test_cpu_board(test, serial):
    """Verify CPU board clock and temperature."""
    test.measurements.cpu_clock_mhz = float(serial.send_command("CLOCK?"))
    test.measurements.cpu_temp = float(serial.send_command("TEMP?"))


@htf.plug(adc=AnalogInputPlug)
@htf.measures(
    htf.Measurement("pwr_12v_rail").in_range(minimum=11.4, maximum=12.6).with_units(units.VOLT),
    htf.Measurement("pwr_efficiency_pct").in_range(minimum=85),
)
def test_power_board(test, adc):
    """Verify power board output rails and efficiency."""
    test.measurements.pwr_12v_rail = adc.read_voltage(channel=0)
    vin = adc.read_voltage(channel=4)
    vout = adc.read_voltage(channel=5)
    test.measurements.pwr_efficiency_pct = (vout / vin) * 100 if vin > 0 else 0


@htf.plug(serial=SerialCommandPlug, adc=AnalogInputPlug)
@htf.measures(
    htf.Measurement("sensor_offset_mv").in_range(minimum=-5, maximum=5),
    htf.Measurement("sensor_gain_error_pct").in_range(minimum=-1, maximum=1),
)
def test_sensor_board(test, serial, adc):
    """Verify sensor board calibration."""
    serial.send_command("SENSOR CAL_CHECK")
    test.measurements.sensor_offset_mv = float(serial.send_command("SENSOR OFFSET?"))
    test.measurements.sensor_gain_error_pct = float(serial.send_command("SENSOR GAIN_ERR?"))
```

## Detecting Regressions

Every measurement value is tracked across all runs. To detect firmware regressions:

1. **Filter by firmware version.** Filter runs by the `firmware_version` measurement and compare pass rates and measurement distributions between versions.
2. **Check measurement trends.** Trend charts show measurement values over time. A sudden shift after a firmware update is a clear regression signal.
3. **Compare Cpk by version.** If a measurement's Cpk drops after a firmware change, the process capability has degraded.

One caveat on comparing versions: firmware releases rarely land on a clean boundary. A version often overlaps with a fixture change, a new component lot, or a different shift pattern. Before attributing a shift to firmware, check whether the other dimensions moved at the same time.

## Manual Tracking vs Structured Logging

| Aspect | Spreadsheet / Manual | Structured logging |
|--------|---------------------|-----------|
| Firmware version tagging | Copy-paste into a column | Recorded as a measurement per run |
| Waveform storage | Shared drive with naming conventions | Attached to the run, always findable |
| Multi-board traceability | Separate sheets or tabs | Sub-units linked to parent assembly |
| Regression detection | Manual chart inspection | Filter by firmware version, compare trends |
| Cross-station comparison | Merge files from different PCs | All stations upload to one workspace |
| Historical lookup | "Which folder was that in?" | Search by serial number, part number, or date |
| Audit trail | Hope nobody deleted a row | Immutable records with timestamps |

### Version Control Hardware Tests with Git
URL: https://www.tofupilot.com/guides/how-to-version-control-hardware-tests-with-git-and-tofupilot
Learn how to version control hardware test scripts with Git and link test results to code versions in TofuPilot.

When a test starts failing on the production floor, the first question is always: "What changed?" If your test scripts aren't version controlled, you can't answer that. Git gives you a complete history of every change to every test. Recording the code version on every run lets you link a result back to the exact script that produced it.

This guide covers Git workflow for test projects, recording commit hashes on your runs, and a practical .gitignore for hardware test repos.

## Why Version Control Matters for Hardware Tests

Hardware test scripts aren't throwaway code. They're part of your quality system. Regulators, auditors, and your future self need to know:

- Which version of the test script produced a given result
- Who changed a measurement limit, and when
- Whether the test that ran in DVT is the same one running in PVT

Without Git, you get folders named `test_v2_final_FINAL_john.py`. With Git, you get a clean audit trail.

## Git Workflow for Test Scripts

A simple branching strategy works well for test projects. You don't need GitFlow. You need something your team will actually use.

```python filename="git_workflow.txt"
main          ──●──●──●──●──●──●──  (production-ready tests)
                    \        /
feature/add-  ──●──●──●──●          (new test or change)
thermal-test
```

**main** is always deployable to production stations. Every commit on main has been reviewed and tested.

**Feature branches** are where you develop new tests or modify existing ones. Name them descriptively: `feature/add-thermal-cycling`, `fix/dmm-timeout-handling`, `update/voltage-limits-rev-c`.

Basic workflow:

```python filename="workflow_commands.sh"
# Start a new test change
git checkout -b feature/add-current-measurement

# Make your changes, test locally
git add tests/power_rail_fct.py
git commit -m "Add idle current measurement to power rail FCT"

# Push and create a pull request for review
git push -u origin feature/add-current-measurement

# After review, merge to main
git checkout main
git pull
git merge feature/add-current-measurement
git push
```

Tag releases when you deploy to production stations:

```python filename="tagging.sh"
# Tag a release before deploying to production
git tag -a v1.2.0 -m "Add thermal cycling test, update voltage limits for Rev C"
git push origin v1.2.0
```

## Read the Git Commit Hash at Runtime

Link every test run to the exact code that produced it. This is the key traceability link between your test results and your test code.

```python filename="git_version.py"
import subprocess

def get_git_info() -> dict:
    """Get current Git commit hash and tag."""
    info = {}
    try:
        info["commit"] = subprocess.check_output(
            ["git", "rev-parse", "HEAD"],
            stderr=subprocess.DEVNULL,
        ).decode().strip()

        info["commit_short"] = subprocess.check_output(
            ["git", "rev-parse", "--short", "HEAD"],
            stderr=subprocess.DEVNULL,
        ).decode().strip()

        # Check for uncommitted changes
        status = subprocess.check_output(
            ["git", "status", "--porcelain"],
            stderr=subprocess.DEVNULL,
        ).decode().strip()
        info["dirty"] = len(status) > 0

        # Get latest tag if available
        try:
            info["tag"] = subprocess.check_output(
                ["git", "describe", "--tags", "--abbrev=0"],
                stderr=subprocess.DEVNULL,
            ).decode().strip()
        except subprocess.CalledProcessError:
            info["tag"] = None

    except subprocess.CalledProcessError:
        info["commit"] = "unknown"
        info["commit_short"] = "unknown"
        info["dirty"] = True
        info["tag"] = None

    return info
```

This relies on the station having Git installed and the script running inside the working tree. If you deploy a packaged executable instead, capture the version at build time and read it from a file, because `git rev-parse` will fail on the station.

## Record the Code Version on Every Run

Record the version as a **measurement**. Measurements are stored per run and are filterable, so you can compare passing and failing runs by code version.

```python filename="test_with_version.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload
from git_version import get_git_info

git_info = get_git_info()

# Warn if running with uncommitted changes in production
if git_info["dirty"]:
    print("WARNING: Running tests with uncommitted changes")


@htf.measures(
    htf.Measurement("script_version").doc("Git tag or short commit of the test script"),
    htf.Measurement("script_dirty").doc("True if the working tree had uncommitted changes"),
)
def record_script_version(test):
    test.measurements.script_version = git_info.get("tag") or git_info["commit_short"]
    test.measurements.script_dirty = git_info["dirty"]


@htf.measures(
    htf.Measurement("voltage_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
def test_power_rail(test):
    test.measurements.voltage_3v3 = 3.29


def main():
    test = htf.Test(
        record_script_version,
        test_power_rail,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "DUT-001")


if __name__ == "__main__":
    main()
```

Now every run carries the code version that produced it. When a test starts failing after a deployment, filter by `script_version` and compare the passing runs against the failing ones to isolate the change.

Recording `script_dirty` alongside it is worth the extra line. A run produced from an uncommitted working tree cannot be reproduced from any commit, so you want that flagged in the data rather than only printed to a console nobody reads.

A note on where this belongs: the fields promoted onto the run itself are a fixed set (`procedure_id`, `part_number`, `revision`, `batch_number`, `sub_units`, `operated_by`). Anything else you put on `test.metadata` stays on the Python object and is not persisted, which is why the version goes in as a measurement rather than as run metadata.

If you deploy procedures from a Git push, this is already handled for you: each deployment is an immutable artifact tied to a commit, so the run records which deployment produced it without any code on your side.

## .gitignore Template for Test Projects

Hardware test repos have specific files you don't want to track. Here's a practical starting point.

```python filename=".gitignore"
# Python
__pycache__/
*.py[cod]
*.so
*.egg-info/
dist/
build/
venv/
.venv/

# Environment and secrets
.env
.env.*
!.env.example

# IDE
.vscode/
.idea/
*.swp
*.swo

# Test artifacts (logs, reports, captures)
logs/
test_output/
*.log
*.csv
captures/
screenshots/

# Instrument calibration data (station-specific)
cal_data/
fixture_cal/

# OS files
.DS_Store
Thumbs.db

# PyInstaller (if you package tests)
*.spec

# Local config overrides
config/local.yaml
```

A few notes on what's excluded:

- **`.env` files** contain API keys. Never commit these. Include a `.env.example` with placeholder values so new team members know what to configure.
- **`logs/` and `test_output/`** are generated per run. They belong with your test results, not in Git.
- **`cal_data/`** is station-specific calibration data. It doesn't belong in the test script repo.
- **`config/local.yaml`** is for per-developer overrides. The shared configs (`config/dev.yaml`, `config/production.yaml`) stay tracked.

One caveat on the `*.csv` line: it is there to catch generated test output, but it will also silently ignore a CSV you intended to track, such as a limits table or a calibration reference. If you keep data files in the repo, narrow that pattern to the directories that actually hold output.

### Python vs LabVIEW for Manufacturing Test
URL: https://www.tofupilot.com/guides/python-vs-labview-for-manufacturing-test-a-practical-comparison
Compare Python and LabVIEW for manufacturing test: cost analysis, instrument control, version control, and what replaces the sequencer and TDMS.

Python and LabVIEW are the two most common languages for manufacturing test. LabVIEW has been the default for 30 years. Python is replacing it. This guide compares them on the things that actually matter for production test: cost, instrument support, version control, deployment, and analytics.

## Head-to-Head Comparison

| Feature | Python | LabVIEW |
|---------|--------|---------|
| **License cost** | Free | $3,160-4,840/seat/year |
| **Platform** | Linux, macOS, Windows | Windows only (runtime on Linux/macOS) |
| **Language type** | Text-based | Graphical (dataflow) |
| **Version control** | Git-native (text files) | Difficult (binary .vi files) |
| **Code review** | Standard pull requests | Requires LabVIEW to view |
| **Package ecosystem** | 400K+ packages (PyPI) | NI packages + limited community |
| **Instrument control** | PyVISA (open source) | NI drivers (proprietary) |
| **Test framework** | TofuPilot Framework, OpenHTF, pytest | Built-in test tools + TestStand |
| **Operator UI** | Declared in the procedure | Front panel (built by hand) |
| **Parallel UUT testing** | Fixture slots, declared | Supported, hand-orchestrated |
| **CI/CD** | Native (GitHub Actions, Jenkins, etc.) | Complex to integrate |
| **Learning curve** | Days (for programmers) | Weeks (unique paradigm) |
| **Hiring pool** | Very large | Small and shrinking |
| **Execution speed** | Interpreted (fast enough for test) | Compiled (slightly faster) |
| **Data analytics** | TofuPilot, pandas, numpy | TDMS + custom tools |

## Cost Analysis

For a team of 10 test engineers running 20 test stations:

| Item | Python | LabVIEW |
|------|--------|---------|
| Development licenses (10) | $0 | $31,600-48,400/year |
| Runtime licenses (20 stations) | $0 | Included (but limited without dev license) |
| Test framework | $0 (TofuPilot Framework or OpenHTF, both MIT/Apache) | $4,310/seat (TestStand) |
| Analytics platform | TofuPilot pricing | Custom development cost |
| **Year 1 total** | TofuPilot only | $75,000-90,000+ |
| **Year 5 total** | TofuPilot only | $375,000-450,000+ |

The savings compound. Every new test station with Python costs $0 in licensing. Every new station with LabVIEW adds license fees.

## Instrument Control: PyVISA vs NI Drivers

Both languages can control the same instruments. The approach differs.

### Python (PyVISA)

```python filename="comparison/python_instrument.py"
import pyvisa

rm = pyvisa.ResourceManager("@py")
dmm = rm.open_resource("TCPIP::192.168.1.100::INSTR")
dmm.timeout = 5000

voltage = float(dmm.query(":MEAS:VOLT:DC?"))
print(f"Voltage: {voltage:.4f} V")

dmm.close()
```

### LabVIEW

In LabVIEW, you'd use an instrument driver VI:
1. Open VISA session (VISA Open)
2. Write SCPI command (VISA Write)
3. Read response (VISA Read)
4. Parse string to number (Fract/Exp String to Number)
5. Close session (VISA Close)
6. Wire error cluster through everything

Five nodes and error wires vs. four lines of Python. The SCPI commands are identical. The instrument doesn't care which language sends them.

| Aspect | PyVISA | NI LabVIEW Drivers |
|--------|--------|-------------------|
| GPIB support | Via NI-VISA backend | Native |
| USB-TMC support | Native (pyvisa-py) | Native |
| Ethernet/LXI support | Native | Native |
| Serial support | Native | Native |
| Driver availability | Generic SCPI (works with any vendor) | Vendor-specific instrument drivers |
| Custom instruments | Write a Python class | Create a .vi driver |

## What Replaces the Sequencer

In LabVIEW, most teams build a state machine and a limit-checking layer by hand. In Python you do not: a hardware test framework provides sequencing, structured measurements and pass/fail evaluation.

With the TofuPilot Framework, the sequence and the limits are declared in a `procedure.yaml`, and each step is a plain Python function:

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def read_voltage(self) -> float:
        return 3.31  # Replace with a PyVISA query
```

Each `validators` block is the equivalent of a LabVIEW Numeric Limit Test, stored as data instead of wired into the diagram. Run it with `tofupilot run ./procedure.yaml`, which works on your machine without an account.

Two LabVIEW features migrating teams ask about, both covered:

- **The front panel.** Operator inputs, checklists, images, sliders and progress bars are declared in the procedure, so there is no UI to rebuild.
- **Parallel UUT testing.** Declare fixture slots and the engine runs them concurrently, each slot producing its own run and outcome:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: Slot 1
    - name: Slot 2
    - name: Slot 3
    - name: Slot 4
```

If you would rather keep everything in Python with no YAML layer, OpenHTF does the same job with decorators, and both run natively on the TofuPilot CLI. See [OpenHTF vs pytest](/guides/openhtf-vs-pytest-for-hardware-testing-which-should-you-choose) and [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf) for the choice between them.

## Test Data and Analytics

This is where Python with TofuPilot has a clear advantage over LabVIEW.

| Capability | Python + TofuPilot | LabVIEW + TDMS |
|-----------|-------------------|----------------|
| Data logging | Automatic | Manual TDMS write |
| FPY tracking | Automatic dashboard | Build it yourself |
| Cpk analysis | Automatic per measurement | Build it yourself |
| Control charts | Automatic | Build it yourself |
| Multi-station view | Automatic aggregation | Manual file aggregation |
| Failure Pareto | Automatic | Build it yourself |
| API access | REST API | Parse TDMS files |
| Historical analysis | Cloud storage, instant queries | TDMS files on file server |

With LabVIEW, you write the test and then spend weeks building analytics tools. With Python and TofuPilot, analytics are automatic from the first test run:

```bash filename="run_and_upload.sh"
tofupilot run ./procedure.yaml --upload
```

## Version Control

This is LabVIEW's biggest weakness for team collaboration.

| Scenario | Python | LabVIEW |
|----------|--------|---------|
| View changes | `git diff` in any terminal | Need LabVIEW open + LV Merge tool |
| Code review | GitHub/GitLab pull request | Can't see .vi diffs in browser |
| Merge conflict | Text-based merge (standard) | Often requires manual rebuild |
| Branch strategy | Standard Git flow | Risky (binary merge conflicts) |
| Blame/history | `git blame` per line | Per-file only (binary) |
| Repo size | Small (text files) | Large (binary VIs with front panels) |

Because a procedure is text, the deployment story follows from Git too: the TofuPilot CLI deploys a procedure to a station on a push, with immutable build artifacts and instant rollback.

## When to Stay with LabVIEW

LabVIEW is still the right choice in some scenarios:

- **NI hardware-only environment.** If your test system is entirely NI PXI/cDAQ/FPGA, the LabVIEW integration is unmatched. Python wrappers exist (nidaqmx) but LabVIEW has deeper access to NI hardware features.
- **Real-time and FPGA targets.** LabVIEW RT and LabVIEW FPGA have no Python equivalent for NI hardware. If you deploy to NI real-time controllers or FPGAs, you need LabVIEW.
- **Large existing codebase with no migration budget.** If it works and you're not adding test stations, the cost of migration may not be justified.
- **Non-programmer team.** If your test engineers are purely EE/ME with no programming background, LabVIEW's visual approach may be easier to learn. (Though most engineers learn Python quickly, and a YAML procedure is readable without being a programmer.)

## When to Switch to Python

Python is the better choice when:

- **You're starting a new test system.** Zero reason to start with LabVIEW licensing costs today.
- **You're scaling beyond 5 test stations.** License costs add up fast, and deploy-on-push scales better than copying VIs.
- **You need cross-platform.** Python runs on any OS. LabVIEW development is Windows-only.
- **You want CI/CD.** Python tests run in any CI pipeline. LabVIEW CI requires NI CLI tools and Windows runners.
- **Your team knows Python.** Most EEs learn Python in school. LabVIEW is niche.
- **You need analytics.** TofuPilot gives you FPY, Cpk, and control charts out of the box. With LabVIEW, you build everything.

## Migration Strategy

Don't rewrite everything at once. Migrate incrementally:

1. **New tests in Python.** Any new product gets a Python procedure.
2. **Wrap LabVIEW with Python.** Call existing LabVIEW code from Python using the `subprocess` module or NI's Python API.
3. **Convert high-maintenance tests first.** Tests that change frequently benefit most from Python's version control.
4. **Convert one station at a time.** Validate Python results match LabVIEW results on the same units before switching.
5. **Keep LabVIEW for FPGA/RT.** If you need NI real-time or FPGA targets, keep LabVIEW for those specific subsystems.

For the step-by-step concept mapping, see [how to migrate from LabVIEW to Python](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot).

### Design for Testability: PCB Best Practices
URL: https://www.tofupilot.com/guides/design-for-testability-pcb-best-practices-with-tofupilot
Design for testability on PCBs: test points, boundary scan, debug access and firmware hooks that make a board cheap to test in production.

Design for testability is the set of decisions made during layout that determine how much a board costs to test for the rest of its life. Most of them cost nothing at design time and are impossible to add later.

## Why It Is a Layout-Time Decision

A board that reaches production without test access leaves three options, all bad: build an expensive fixture that probes awkward places, skip coverage and ship risk, or respin the PCB. The cost curve is steep and one-directional, which is why DFT reviews belong before layout is released, not after the first prototypes fail on the bench.

## Test Points

| Practice | Recommendation |
|----------|---------------|
| Diameter | 0.9-1.0 mm pad, 0.6 mm minimum for fine-pitch probes |
| Spacing | 2.54 mm preferred, 1.27 mm minimum between adjacent points |
| Edge clearance | At least 3 mm from the board edge for fixture tooling |
| Side | All on one side if possible; double-sided fixtures roughly double cost |
| Finish | Solderable finish, not solder-masked |
| Labelling | Silkscreen the net name; a bench technician will thank you |
| Tooling holes | Two non-plated holes, diagonally opposite, for fixture alignment |

What to bring out as a test point:

- Every power rail, on the load side of any filtering
- Ground, several times, close to the rails they return
- Reset, boot mode and strap pins
- I2C/SPI bus lines that a fixture may need to drive or observe
- Crystal or clock output, if a frequency check is planned
- Any net the test plan intends to measure

The last line is the actual rule. DFT works when the test plan exists before layout is released, even in draft.

## Power Rail Access

Rails are the highest-value test points because nearly every failure mode shows up there first. Bring out each rail separately rather than relying on a single point downstream of all of them, so a measurement can localise the fault.

Consider a way to isolate rails from their loads, such as a zero-ohm link or a solder jumper, to distinguish a shorted regulator from a shorted load. A board that fails with a collapsed rail is otherwise a diagnostic dead end.

## Programming and Debug Access

| Interface | What to expose |
|-----------|----------------|
| SWD/JTAG | Clock, data, reset, ground, and target voltage reference |
| UART console | TX, RX, ground, at a documented baud rate |
| Boot mode | A strap or test point that can force the bootloader |
| Reset | A test point the fixture can drive, not only a button |

Programming at end of line is normal, so the fixture needs to flash firmware, set the boot mode and reset the target without human intervention. If the boot strap is only a physical jumper, someone is moving jumpers on every board.

## Boundary Scan

For dense BGA designs where physical probing cannot reach, IEEE 1149.1 boundary scan tests interconnects through the device itself.

| Requirement | Note |
|-------------|------|
| TCK, TMS, TDI, TDO, TRST | Bring out to test points or a connector |
| Chain order | Document it; the tooling needs it |
| Non-scan devices | Break the chain; note what is not covered |
| Pull-ups | Per the standard, so the chain is deterministic at power-up |

See [boundary scan JTAG test](/guides/boundary-scan-jtag-test).

## Firmware Hooks Are Part of DFT

The most under-used testability feature is firmware that cooperates with the test.

| Hook | Value in production test |
|------|-------------------------|
| Self-test command | Exercises peripherals the fixture cannot reach |
| Version query | Confirms the right image is installed |
| ADC raw readback | Verifies the analog front end without external stimulus |
| GPIO set/get | Enables loopback testing without extra hardware |
| Calibration write | Stores coefficients computed by the test |
| Fault injection | Verifies protection circuits deliberately |

A `selftest` command returning structured output turns a whole class of expensive fixture hardware into a serial query. It costs a day of firmware work and saves fixture cost on every station.

Design the output to be machine-readable from the start:

```python filename="plugs/dut.py"
import json


class Dut:
    """Serial console on the board under test."""

    def selftest(self) -> dict:
        """Firmware returns JSON, so the test does not parse prose."""
        raw = self._query("selftest --json")
        return json.loads(raw)

    def _query(self, cmd: str) -> str:
        return '{"adc_counts": 2048, "gpio_ok": true, "temp_c": 41.2}'
```

```python filename="phases/selftest.py"
def selftest(measurements, dut):
    result = dut.selftest()
    measurements.adc_counts = result["adc_counts"]
    measurements.gpio_ok = result["gpio_ok"]
    measurements.die_temperature = result["temp_c"]
```

```yaml filename="procedure.yaml"
main:
  - name: Selftest
    python: phases.selftest
    measurements:
      - name: ADC Counts
        validators:
          - operator: ">="
            expected_value: 2000
          - operator: "<="
            expected_value: 2096
      - name: GPIO Ok
        validators:
          - operator: "=="
            expected_value: true
      - name: Die Temperature
        unit: "°C"
        validators:
          - operator: "<="
            expected_value: 85
```

Structured output matters more than it looks. A firmware self-test that prints human prose forces every test script to parse it, and every firmware wording change breaks production.

## Fixture Considerations

| Practice | Why |
|----------|-----|
| Keep tall components away from probe areas | Probes need clearance |
| Avoid test points under connectors or shields | Unreachable once assembled |
| Provide a ground plane reference near high-speed probes | Otherwise the measurement is the fixture |
| Add a fixture calibration short | Lets the test measure its own contact resistance |
| Standardise tooling hole positions across products | One fixture base serves many boards |

The calibration short is worth singling out. A known short across two pins lets the test measure contact resistance directly, so fixture wear is detected as a trend rather than as mysterious intermittent failures on good boards.

## Panel and Multi-Up Considerations

If boards are tested in panel, decide early whether the fixture tests all positions at once. Testing four boards in parallel needs four sets of probes and four instrument paths, or a switching matrix. Both affect the layout, and retrofitting either is expensive.

## The DFT Review

Before layout release, walk the test plan against the layout:

1. Every measurement in the plan has an accessible test point.
2. Every rail can be measured, and ideally isolated.
3. Programming and reset are driveable by the fixture.
4. Boot mode can be forced without human intervention.
5. Test points are one side, correctly spaced, clear of the edge.
6. Tooling holes exist and are documented.
7. Firmware exposes the hooks the plan depends on.
8. Dense areas have boundary scan or accept documented coverage gaps.

An hour of this review is routinely worth more than a week of fixture engineering later.

## If You Already Run OpenHTF

```python filename="selftest.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("adc_counts").in_range(2000, 2096),
    htf.Measurement("gpio_ok").equals(True),
    htf.Measurement("die_temperature")
    .in_range(maximum=85)
    .with_units(units.DEGREE_CELSIUS),
)
def phase_selftest(test):
    test.measurements.adc_counts = 2048
    test.measurements.gpio_ok = True
    test.measurements.die_temperature = 41.2


test = htf.Test(phase_selftest, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Test points added after layout | Awkward fixture, or a respin | DFT review before layout release |
| Test points on both sides | Fixture cost roughly doubles | Consolidate to one side |
| Solder-masked test pads | Probes cannot make contact | Specify a solderable finish |
| Boot strap only as a physical jumper | Manual intervention per board | Make it fixture-driveable |
| Firmware self-test printing prose | Every wording change breaks production | Structured output, JSON |
| No fixture calibration short | Fixture wear looks like product failure | Add one and trend contact resistance |
| No tooling holes | Alignment by hand, inconsistent contact | Two non-plated holes, diagonal |

### HIL Testing for Embedded Systems
URL: https://www.tofupilot.com/guides/how-to-set-up-hil-testing-for-embedded-systems-with-python-and-tofupilot
Set up a HIL bench for embedded systems with Python: fixture plugs, firmware version tracking, CI integration and unattended runs.

Hardware-in-the-loop testing runs your real firmware on the real target, with the environment around it simulated. This guide is the practical setup: what the bench needs, how to structure the test so it survives firmware churn, and how to run it unattended in CI.

For the conceptual comparison against SIL and MIL, see [HIL vs SIL vs MIL](/guides/hil-vs-sil-vs-mil-testing-when-to-use-each-with-tofupilot).

## What the Bench Needs

| Component | Role | Typical choice |
|-----------|------|----------------|
| Target board | Runs the firmware under test | STM32, ESP32, RP2040 dev board or prototype |
| Debug probe | Flash and reset the target | J-Link, ST-Link, CMSIS-DAP |
| GPIO adapter | Stimulate inputs, read outputs | LabJack, Arduino, RPi GPIO |
| DAQ | Analog stimulus and capture | NI DAQ, LabJack, ADS1115 |
| Programmable supply | Power cycling, current measurement | Rigol DP800, Keysight E36xx |
| Bus interface | CAN, LIN, I2C, SPI as needed | PCAN, Kvaser, FTDI |
| Test host | Runs the procedure | Mini PC or Raspberry Pi at the bench |

The host lives at the bench permanently. That is what makes HIL a CI target rather than a manual exercise.

## Structuring the Test

A HIL suite has a shape that differs from production FCT: the unit under test is the same board every time, and what changes is the firmware. Everything follows from that.

### Plugs for the bench hardware

```python filename="plugs/programmer.py"
import subprocess


class Programmer:
    """Flashes the target and reports what is on it."""

    def flash(self, image: str) -> bool:
        result = subprocess.run(
            ["pyocd", "flash", "--target", "stm32f407vg", image],
            capture_output=True,
        )
        return result.returncode == 0

    def reset(self):
        subprocess.run(["pyocd", "reset", "--target", "stm32f407vg"])
```

```python filename="plugs/gpio.py"
class Gpio:
    """GPIO adapter. Replace with LabJack, Arduino or RPi.GPIO calls."""

    def __init__(self):
        self._state = {}

    def set_pin(self, pin: int, value: bool):
        self._state[pin] = value

    def read_pin(self, pin: int) -> bool:
        return self._state.get(pin, False)
```

```python filename="plugs/dut_console.py"
class DutConsole:
    """Serial console on the target."""

    def query_version(self) -> str:
        return "2.1.0"  # Replace with a real serial exchange

    def measure_boot_ms(self) -> int:
        return 840

    def free_heap_bytes(self) -> int:
        return 48120
```

### Phases

```python filename="phases/flash_firmware.py"
def flash_firmware(measurements, programmer):
    measurements.flash_ok = programmer.flash("build/firmware.bin")
    programmer.reset()
```

```python filename="phases/boot_check.py"
def boot_check(measurements, dut_console):
    measurements.firmware_version = dut_console.query_version()
    measurements.boot_time = dut_console.measure_boot_ms()
```

```python filename="phases/gpio_loopback.py"
def gpio_loopback(measurements, gpio):
    gpio.set_pin(1, True)
    measurements.gpio_loopback = gpio.read_pin(1)
```

```python filename="phases/memory_check.py"
def memory_check(measurements, dut_console):
    measurements.free_heap = dut_console.free_heap_bytes()
```

### The procedure

```yaml filename="procedure.yaml"
name: ECU HIL Regression
version: 1.0.0
description: Per-build HIL regression on the bench target

unit:
  serial_number:
    default_value: "BENCH-01"
  part_number:
    default_value: "ECU-100"

plugs:
  - name: programmer
    python: plugs.programmer:Programmer
  - name: gpio
    python: plugs.gpio:Gpio
  - name: dut_console
    python: plugs.dut_console:DutConsole

main:
  - name: Flash Firmware
    python: phases.flash_firmware
    measurements:
      - name: Flash Ok
        validators:
          - operator: "=="
            expected_value: true

  - name: Boot Check
    python: phases.boot_check
    depends_on: [flash_firmware]
    measurements:
      - name: Firmware Version
        validators:
          - operator: "=="
            expected_value: "2.1.0"
      - name: Boot Time
        unit: ms
        validators:
          - operator: "<="
            expected_value: 2000

  - name: GPIO Loopback
    python: phases.gpio_loopback
    depends_on: [boot_check]
    measurements:
      - name: GPIO Loopback
        validators:
          - operator: "=="
            expected_value: true

  - name: Memory Check
    python: phases.memory_check
    depends_on: [boot_check]
    measurements:
      - name: Free Heap
        unit: bytes
        validators:
          - operator: ">="
            expected_value: 32768
```

Flash runs first and everything waits on it. Boot check follows. GPIO and memory both depend on boot but not on each other, so they run concurrently.

Run it with `tofupilot run ./procedure.yaml`.

## The Unit Is the Bench, the Variable Is the Build

This is the structural decision that makes a HIL suite useful over time. In production the serial number identifies a board; on a HIL bench the same board runs thousands of times, and what you actually want to compare across runs is the firmware build.

Keep the serial as the bench identifier, and put the build on the run as metadata:

```python filename="phases/record_build.py"
def record_build(run):
    import os
    run.metadata["firmware_version"] = os.environ.get("FW_VERSION", "dev")
    run.metadata["git_sha"] = os.environ.get("GIT_SHA", "")
    run.metadata["branch"] = os.environ.get("GIT_BRANCH", "")
    run.metadata["build_number"] = os.environ.get("BUILD_NUMBER", "")
```

With that in place, the bench's run history is a regression history: boot time and free heap per build, and a commit to point at when either moves.

## Running It in CI

HIL has no operator, so the identify prompt must resolve from the procedure and any UI phases need pre-baked values:

```bash filename="ci/run_hil.sh"
set -euo pipefail

export FW_VERSION="$(cat build/version.txt)"
export GIT_SHA="${GITHUB_SHA:-}"
export GIT_BRANCH="${GITHUB_REF_NAME:-}"
export BUILD_NUMBER="${GITHUB_RUN_NUMBER:-}"

tofupilot run ./procedure.yaml \
  --upload \
  --json \
  --ui-values ci/ui.json \
  --ui-timeout 300
```

`ui.json` can be an empty map when there are no operator inputs to answer:

```json filename="ci/ui.json"
{}
```

The CLI exits non-zero when the run fails, so the job fails on a regression without extra scripting.

The runner must be a self-hosted runner at the bench. A cloud runner has no target board attached.

## What to Measure

| Category | Measurement | Why it catches regressions |
|----------|-------------|---------------------------|
| Boot | Boot time in ms | Initialisation creeping up |
| Memory | Free heap, stack high-water | Leaks, stack growth |
| Timing | Interrupt latency in µs | Scheduling regressions |
| Peripherals | GPIO, ADC, PWM, bus reads | Driver or pinmux breakage |
| Power | Sleep current, wake latency | Power management regressions |
| Watchdog | Recovery after induced hang | Safety behaviour |
| Long-run | Uptime before fault | Counter rollovers, slow leaks |

Boot time and free heap are the two that pay for themselves fastest. Both drift slowly, both are invisible in unit tests, and both are trivially measurable here.

## Keeping the Bench Honest

**Power-cycle between runs.** A target that was never reset carries state from the previous build, which produces intermittent results nobody can reproduce.

**Fail the flash step loudly.** If flashing silently fails, every later measurement describes the *previous* firmware, and the suite reports green on a build that was never tested. That is why `flash_ok` is a measurement with a validator rather than an unchecked side effect.

**Record the bench identity** when you have more than one, so a bench-specific fixture problem does not look like a firmware regression.

## If You Already Run OpenHTF

Existing suites run natively on the TofuPilot CLI:

```python filename="hil_main.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("boot_time_ms")
    .in_range(maximum=2000)
    .with_units(units.MILLISECOND),
    htf.Measurement("free_heap").in_range(minimum=32768),
)
def phase_boot(test):
    test.measurements.boot_time_ms = 840
    test.measurements.free_heap = 48120
    test.metadata["firmware_version"] = "2.1.0"


test = htf.Test(phase_boot, part_number="ECU-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: "BENCH-01")
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Not verifying the flash succeeded | Green results describing the old firmware | Make `flash_ok` a validated measurement |
| No power cycle between runs | Intermittent, unreproducible failures | Reset or power-cycle in a setup phase |
| Firmware version not recorded | No regression history, only scattered runs | Put build metadata on every run |
| Cloud CI runner | No hardware attached | Self-hosted runner at the bench |
| Prompts left in a CI procedure | The job hangs forever | `--ui-values` plus `--ui-timeout` |
| One phase for the whole suite | A single failure hides everything downstream | Split phases along dependency lines |

### Generate Test Reports for Auditors
URL: https://www.tofupilot.com/guides/how-to-generate-test-reports-for-auditors-with-tofupilot
Structure your OpenHTF tests so TofuPilot captures audit-ready records with serial numbers, measurements, limits, and metadata.

Auditors want to see that every unit was tested, what was measured, what the limits were, and whether it passed. TofuPilot captures all of this automatically when you structure your OpenHTF tests correctly.

## What Auditors Need to See

Regardless of the standard (ISO 9001, ISO 13485, IATF 16949, AS9100), audit requirements for test records share the same core elements.

| Element | Why Auditors Ask for It |
|---|---|
| Serial number (DUT ID) | Proves traceability to a specific unit |
| Timestamp | Proves when the test happened |
| Station ID | Proves which equipment was used |
| Operator | Proves who ran the test |
| Measurements with limits | Proves acceptance criteria were applied |
| Pass/fail verdict | Proves disposition was recorded |
| Firmware or software version | Proves configuration at time of test |

If your test records are missing any of these, you'll get a finding. TofuPilot stores all of them when you include them in your test run.

## Writing an Audit-Ready Test

A good test record starts with a well-structured OpenHTF test. Include every measurement with explicit limits, and pass metadata through your test run.

```python filename="audit_ready_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("input_resistance")
    .in_range(minimum=950, maximum=1050)
    .with_units(units.OHM),
    htf.Measurement("leakage_current")
    .in_range(maximum=0.000005)
    .with_units(units.AMPERE),
    htf.Measurement("dielectric_strength_pass")
    .equals(True),
)
def electrical_safety_test(test):
    test.measurements.input_resistance = 1002
    test.measurements.leakage_current = 0.0000013
    test.measurements.dielectric_strength_pass = True

@htf.measures(
    htf.Measurement("output_power")
    .in_range(minimum=48.0, maximum=52.0)
    .with_units(units.WATT),
    htf.Measurement("efficiency_percent")
    .in_range(minimum=89.0),
)
def performance_test(test):
    test.measurements.output_power = 50.1
    test.measurements.efficiency_percent = 92.4

def main():
    test = htf.Test(
        electrical_safety_test,
        performance_test,
        station_id="STATION-EOL-02",
    )

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PSU-2026-00512")

if __name__ == "__main__":
    main()
```

Every measurement gets stored with its value, limits, and verdict in TofuPilot. The serial number links the record to the unit's full history.

## What Makes a Good Test Record

Three things separate a record that satisfies auditors from one that raises questions.

**Consistent measurement names.** Use the same name for the same measurement everywhere. If `leakage_current` is called `leak_curr` on one station and `leakage_current` on another, you can't demonstrate consistent process monitoring. Pick one name and stick with it.

**Explicit limits on every measurement.** A measured value without limits is just data. Auditors want to see that you defined acceptance criteria before production, not after. Always use `.in_range()` or `.equals()` on every measurement.

**Meaningful metadata.** Include the operator, station, firmware version, and any other context your quality system defines. When an auditor asks "who tested unit X on what equipment with what software?", you need an answer.

## Viewing and Exporting Records

Once your tests run, TofuPilot stores the complete record. You can access it in three ways.

**Run detail page.** Search for any unit by serial number in TofuPilot's dashboard. The run detail page shows every measurement, its value, its limits, the verdict, timestamp, station, and all metadata. This is your primary view during an audit walkthrough.

**CSV export.** Export run data from the dashboard for offline review or to attach to audit documentation. The export includes all measurements, limits, and metadata.

**API access.** Pull run records programmatically through TofuPilot's REST API at `https://tofupilot.app` for integration with your QMS or document control system. This is useful if your quality team needs to generate batch records or certificates of conformance.

## Metadata Checklist

Before going into production, verify that your tests capture everything your quality system requires. Here's a starting checklist.

- [ ] DUT serial number (unique per unit)
- [ ] Test station ID (matches calibration records)
- [ ] Operator ID (matches training records)
- [ ] Firmware/software version (matches release records)
- [ ] All measurements have limits defined
- [ ] Measurement names are consistent across stations
- [ ] Test procedure name or ID is included
- [ ] Work order or batch number (if applicable)

If you're missing any of these, add them to your measurement definitions before production starts. It's much harder to backfill traceability data after the fact.

### The True Cost of Poor Quality in Electronics
URL: https://www.tofupilot.com/guides/the-true-cost-of-poor-quality-in-electronics-with-tofupilot
Understand COPQ categories, the 1-10-100 rule for defect costs, and how proper test coverage with TofuPilot catches issues before they become expensive.

Every defective unit has a price tag, and it grows the further that defect travels. The Cost of Poor Quality (COPQ) captures everything you spend because something wasn't built right the first time. For most electronics manufacturers, COPQ runs between 15% and 25% of revenue. Most of it is invisible.

## What COPQ Includes

COPQ splits into two categories: internal failures (caught before shipping) and external failures (caught by the customer).

### Internal Failure Costs

These happen inside your factory. They're painful but controllable.

| Cost Type | Example | Typical Impact |
|-----------|---------|---------------|
| Scrap | PCB with tombstoned components sent to recycling | Full material + labor cost lost |
| Rework | Re-soldering a BGA after X-ray finds voids | Labor + equipment time + retest |
| Retesting | Unit fails functional test, gets retested after fix | Station time blocked, throughput drops |
| Downgrading | Unit doesn't meet Grade A spec, sold as Grade B | Revenue loss per unit |
| Failure analysis | Engineering time to diagnose root cause | Hours of skilled labor diverted |

### External Failure Costs

These happen after the product ships. They're where the real damage lives.

| Cost Type | Example | Typical Impact |
|-----------|---------|---------------|
| Warranty claims | Customer returns a dead power supply | Replacement unit + shipping + processing |
| Field service | Technician dispatched to replace a failed module | Travel + labor + downtime penalty |
| Recalls | Batch of units with defective firmware update | Logistics + PR + regulatory reporting |
| Customer churn | Customer switches to competitor after repeated issues | Lifetime revenue lost |
| Brand damage | Negative reviews and lost referrals | Hard to quantify, slow to recover |

## The 1-10-100 Rule

This rule, originally from quality management literature, puts a ratio on when you catch a defect:

- **$1 for prevention.** Design the test, validate the process, set the limits. A well-written OpenHTF test with proper measurement limits costs almost nothing per unit to run.
- **$10 for detection.** Catch the defect on the line. You've already spent material and labor, but you can rework or scrap before shipping. This is where test stations earn their keep.
- **$100 for failure.** The defect reaches the customer. Now you're paying for returns, field service, warranty processing, and the trust you can't invoice for.

The ratios vary by industry. In medical devices, external failure costs can be 1000x prevention costs when you factor in regulatory consequences. In consumer electronics, it's closer to the 1-10-100 model. The principle is always the same: catch it earlier, pay less.

## Where Test Coverage Fits

Test coverage is your primary tool for converting $100 problems into $10 problems, and $10 problems into $1 investments.

### Coverage Gaps Cost Money

Consider a Bluetooth speaker manufacturer running only a final functional test. They check audio output and pairing. What they don't catch:

- Cold solder joints that pass at room temperature but fail after thermal cycling
- Battery cells with slightly low capacity that die early in the field
- Antenna impedance mismatch that causes range issues in certain orientations

Each gap is a field failure waiting to happen. Adding targeted tests at earlier stages closes these gaps.

### Structuring Tests to Catch Defects Early

A well-structured test strategy pushes detection upstream. Here's what that looks like for an IoT sensor module:

```python filename="test_incoming_inspection.py"
# Incoming inspection: catch component issues before assembly
import openhtf as htf
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("crystal_frequency")
        .in_range(minimum=31.990, maximum=32.010),
    htf.Measurement("sensor_ic_id_register")
        .equals(0xB5),
    htf.Measurement("flash_chip_capacity_mb")
        .equals(16),
)
def incoming_component_check(test):
    test.measurements.crystal_frequency = 32.001
    test.measurements.sensor_ic_id_register = 0xB5
    test.measurements.flash_chip_capacity_mb = 16

def main():
    test = htf.Test(incoming_component_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "IOT-2024-4401")

if __name__ == "__main__":
    main()
```

```python filename="test_post_assembly.py"
# Post-assembly test: validate solder quality and basic function
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("supply_current_sleep")
        .in_range(maximum=0.000050)
        .with_units(units.AMPERE),
    htf.Measurement("supply_current_active")
        .in_range(minimum=0.015, maximum=0.035)
        .with_units(units.AMPERE),
    htf.Measurement("i2c_sensor_ack")
        .equals(True),
    htf.Measurement("spi_flash_read_write_ok")
        .equals(True),
)
def post_assembly_validation(test):
    test.measurements.supply_current_sleep = 0.000028
    test.measurements.supply_current_active = 0.0243
    test.measurements.i2c_sensor_ack = True
    test.measurements.spi_flash_read_write_ok = True

def main():
    test = htf.Test(post_assembly_validation)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "IOT-2024-4401")

if __name__ == "__main__":
    main()
```

```python filename="test_final_calibration.py"
# Final calibration and functional test
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("temperature_accuracy")
        .in_range(minimum=-0.5, maximum=0.5)
        .with_units(units.DEGREE_CELSIUS),
    htf.Measurement("humidity_accuracy_pct")
        .in_range(minimum=-3.0, maximum=3.0),
    htf.Measurement("ble_rssi_at_1m")
        .in_range(minimum=-55, maximum=-35),
    htf.Measurement("battery_voltage")
        .in_range(minimum=3.0, maximum=4.2)
        .with_units(units.VOLT),
    htf.Measurement("ota_update_success")
        .equals(True),
)
def final_calibration_test(test):
    test.measurements.temperature_accuracy = 0.12
    test.measurements.humidity_accuracy_pct = -1.4
    test.measurements.ble_rssi_at_1m = -42
    test.measurements.battery_voltage = 3.85
    test.measurements.ota_update_success = True

def main():
    test = htf.Test(final_calibration_test)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "IOT-2024-4401")

if __name__ == "__main__":
    main()
```

Three test stages, each catching a different class of defect. Incoming inspection catches bad components before you solder them (cheapest fix). Post-assembly catches process defects before calibration effort is wasted. Final test catches system-level issues before shipping.

## Using TofuPilot to Track Quality Costs

You can't reduce COPQ without measuring it. TofuPilot gives you the data foundation:

- **FPY per procedure** shows your first-pass yield at each test stage. A drop in FPY at post-assembly means your process is generating rework. That's internal failure cost climbing.
- **Failure Pareto analysis** ranks which measurements fail most often. This tells you where to invest prevention dollars. If 60% of failures are solder-related, that's a clear signal to improve your reflow profile or paste deposition.
- **Yield trends over time** reveal whether quality is improving or degrading. A slow downward trend in FPY is COPQ increasing before anyone notices in the financial reports.
- **Unit traceability** connects field failures back to test data. When a customer returns a unit, you can pull its complete test history and see whether it passed marginally or had anomalies that were within spec but near the edge.

## Reducing COPQ in Practice

The path from high COPQ to low COPQ follows a predictable pattern:

1. **Instrument your process.** Add tests at each major manufacturing step. Upload everything to TofuPilot with consistent serial numbers and measurement names.
2. **Identify the biggest losses.** Use failure Pareto to find the top 3 failure modes. These typically account for 60-80% of your internal failures.
3. **Push detection upstream.** If final test catches a defect, ask whether an earlier test could have caught it before more value was added to the unit.
4. **Tighten limits proactively.** Use Cpk data from TofuPilot to identify measurements that are technically passing but drifting toward limits. Tightening process controls before failures occur is prevention, the cheapest category.
5. **Measure the result.** Track FPY improvement over time. Every percentage point of FPY improvement translates directly to lower scrap, less rework, and fewer field failures.

### How to Set Up Kiosk Mode for Test Stations
URL: https://www.tofupilot.com/guides/how-to-set-up-kiosk-mode-for-test-stations
Kiosk mode locks a test station to the operator UI. Learn how to configure Chrome kiosk mode, auto-start, and barcode scanner integration.

Kiosk mode locks a test station's display to the operator interface. The operator sees only the test UI. No address bar, no taskbar, no desktop, no way to accidentally close the browser or open other applications. This guide covers how to set up kiosk mode on Windows and Linux for manufacturing test stations.

## Why Kiosk Mode

| Problem | Kiosk Mode Solves It |
|---------|---------------------|
| Operator closes the browser | Browser restarts automatically |
| Operator navigates away from the test UI | Address bar is hidden |
| Operator opens other applications | Taskbar is hidden, Alt+Tab is disabled |
| Station boots to a desktop | Browser opens automatically on startup |
| Display is hard to read at arm's length | Browser zoom is preset to 150% |

Kiosk mode turns a general-purpose PC into a dedicated test terminal.

## Prerequisites

- Chrome or Chromium installed on the station PC
- TofuPilot streaming URL for the station
- Admin access to the station PC

## Step 1: Launch Chrome in Kiosk Mode

### Windows

Create a shortcut or batch script that launches Chrome in kiosk mode pointing to the TofuPilot streaming URL.

```bat filename="start_operator_ui.bat"
@echo off
REM Launch Chrome in kiosk mode for operator UI
start "" "C:\Program Files\Google\Chrome\Application\chrome.exe" ^
    --kiosk ^
    --disable-pinch ^
    --overscroll-history-navigation=0 ^
    --noerrdialogs ^
    --disable-translate ^
    --no-first-run ^
    --fast ^
    --fast-start ^
    --disable-features=TranslateUI ^
    --disk-cache-dir=nul ^
    "https://tofupilot.app/streaming/your-station-room-id"
```

### Linux

Create a shell script for kiosk mode on Linux stations.

```bash filename="start_operator_ui.sh"
#!/bin/bash
# Launch Chromium in kiosk mode for operator UI
chromium-browser \
    --kiosk \
    --disable-pinch \
    --overscroll-history-navigation=0 \
    --noerrdialogs \
    --disable-translate \
    --no-first-run \
    --disable-features=TranslateUI \
    "https://tofupilot.app/streaming/your-station-room-id"
```

## Step 2: Auto-Start on Boot

### Windows

Place the batch script in the Startup folder:

```
C:\Users\<operator>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\
```

Or create a scheduled task that runs the script at logon.

### Linux (systemd)

Create a systemd service that starts the browser after the display server is ready.

```ini filename="/etc/systemd/system/operator-ui.service"
[Unit]
Description=Operator UI Kiosk
After=graphical.target

[Service]
Type=simple
User=operator
Environment=DISPLAY=:0
ExecStart=/home/operator/start_operator_ui.sh
Restart=on-failure
RestartSec=5

[Install]
WantedBy=graphical.target
```

Enable the service:

```bash filename="terminal"
sudo systemctl enable operator-ui.service
```

The `Restart=on-failure` setting means the browser relaunches automatically if it crashes or gets closed.

## Step 3: Configure the Display

| Setting | Value | How |
|---------|-------|-----|
| Browser zoom | 125-150% | Set in Chrome settings or via `--force-device-scale-factor=1.5` flag |
| Screen timeout | Never | Disable sleep/screensaver in OS power settings |
| Screen rotation | Landscape | Match the UI layout |
| Resolution | Native | Don't downscale, the UI is responsive |

## Step 4: Set Up Barcode Scanner

Most USB barcode scanners work as HID keyboard devices. When the operator scans a barcode, the characters are typed into the active input field.

| Setting | Recommendation |
|---------|---------------|
| Scanner mode | USB HID Keyboard (default on most scanners) |
| Suffix | Configure the scanner to send Enter after each scan |
| Prefix | None (or configure to match your serial format) |
| Focus | The TofuPilot operator UI auto-focuses the serial input field |

The workflow becomes: operator scans barcode, Enter is sent automatically, test starts. No clicking, no typing.

## Step 5: Lock Down the Station

### Disable Keyboard Shortcuts (Windows)

Use Group Policy or a third-party tool to disable:

| Shortcut | Why |
|----------|-----|
| Alt+Tab | Prevents switching away from the operator UI |
| Alt+F4 | Prevents closing the browser |
| Ctrl+W | Prevents closing the tab |
| Windows key | Prevents opening the Start menu |

### Disable Keyboard Shortcuts (Linux)

Configure the window manager to ignore these shortcuts, or run Chrome in a minimal window manager (like Openbox) that doesn't bind them.

## Troubleshooting

| Issue | Fix |
|-------|-----|
| Browser shows "No internet" on boot | The browser starts before the network is ready. Add a 10-second delay to the startup script, or use `Restart=on-failure` in systemd. |
| Screen goes black after inactivity | Disable screen timeout in OS power settings and screensaver. |
| Scanner types into the wrong field | The TofuPilot UI auto-focuses the serial input. If focus is lost, clicking anywhere on the page restores it. |
| Browser updates break kiosk mode | Pin the Chrome/Chromium version or disable auto-updates on station PCs. |

### How to Migrate from WATS to TofuPilot
URL: https://www.tofupilot.com/guides/how-to-migrate-from-wats-to-tofupilot
Step-by-step migration from WATS to TofuPilot, covering concept mapping, data migration, and parallel running.

If you're evaluating a WATS alternative, this guide walks through the migration path. Common triggers include cost (WATS Analytics runs at EUR 297/mo per module), a preference for Python-first workflows, or wanting to build on an open-source test framework instead of proprietary data converters.

TofuPilot pairs with OpenHTF (Google's open-source test framework) to give you structured test data, real-time analytics, and a REST API without vendor lock-in on the test execution side.

## Concept Mapping

WATS and TofuPilot use different terminology for similar concepts. Here's how they map:

| WATS Concept | TofuPilot Equivalent | Notes |
|---|---|---|
| UUT Report | Test Run | One execution of a test sequence on a unit |
| Test Step | Phase | A phase in OpenHTF, containing measurements |
| Numeric Limit Test | Measurement with limits | Defined via `htf.Measurement().in_range()` |
| String Value Test | Measurement (string) | Any measured value, not just numeric |
| WATS Client / Data Converter | TofuPilot Python SDK | `from tofupilot.openhtf import upload` |
| WATS Dashboard | TofuPilot Dashboard | FPY, Cpk, Pareto, control charts at [tofupilot.app](https://tofupilot.app) |
| Test Sequence File | OpenHTF Test | Python script defining phases and measurements |
| Station Registration | Automatic | TofuPilot detects stations from test metadata |

## Replacing the WATS Data Converter

With WATS, you typically use a data converter (TestStand, LabVIEW, or custom) to serialize test results and push them to the WATS API. With TofuPilot, you write your test logic in OpenHTF and the SDK handles reporting directly.

Here's a typical test that would replace a WATS data converter workflow:

```python filename="functional_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("supply_voltage")
        .with_units(units.VOLT)
        .in_range(4.75, 5.25),
    htf.Measurement("clock_frequency")
        .with_units(units.HERTZ)
        .in_range(minimum=7.99e6),
    htf.Measurement("firmware_version"),
)
def power_and_clock_check(test):
    test.measurements.supply_voltage = 5.02
    test.measurements.clock_frequency = 8.00e6
    test.measurements.firmware_version = "v3.2.1"

@htf.measures(
    htf.Measurement("signal_amplitude")
        .with_units(units.VOLT)
        .in_range(0.9, 1.1),
    htf.Measurement("signal_thd")
        .in_range(maximum=0.05),
)
def signal_integrity_check(test):
    test.measurements.signal_amplitude = 1.01
    test.measurements.signal_thd = 0.03

def main():
    test = htf.Test(power_and_clock_check, signal_integrity_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-10042")

if __name__ == "__main__":
    main()
```

No data converter needed. The TofuPilot SDK serializes the OpenHTF test record and uploads it automatically. Each phase maps to what WATS calls a test step, and each measurement carries its value, units, and limits.

## Migrating Historical Data

You likely have test history in WATS that you want to preserve. TofuPilot's REST API accepts historical runs with explicit timestamps, so you can backfill without losing chronological context.

First, export your data from WATS (CSV export or API). Then import it:

```python filename="import_wats_history.py"
import csv
from datetime import datetime
from tofupilot import TofuPilotClient

client = TofuPilotClient()

with open("wats_export.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        client.create_run(
            procedure_id="functional-test",
            unit_under_test={
                "serial_number": row["SerialNumber"],
                "part_number": row["PartNumber"],
            },
            run_passed=row["Status"] == "Passed",
            started_at=datetime.fromisoformat(row["StartTime"]),
            duration=float(row["Duration"]),
            steps=[
                {
                    "name": row["StepName"],
                    "step_passed": row["StepStatus"] == "Passed",
                    "measurements": [
                        {
                            "name": row["MeasurementName"],
                            "measured_value": float(row["Value"]),
                            "unit": row["Unit"],
                            "lower_limit": float(row["LowLimit"]) if row["LowLimit"] else None,
                            "upper_limit": float(row["HighLimit"]) if row["HighLimit"] else None,
                        }
                    ],
                }
            ],
        )
```

For complex WATS exports with nested test steps, flatten them into TofuPilot's phase/measurement structure. Each WATS numeric limit test becomes a measurement with `lower_limit` and `upper_limit`.

## Parallel Running Strategy

Don't cut over in one shot. Run both systems side by side to validate the migration:

1. **Week 1.** Pick one test station. Add the TofuPilot integration to its tests while keeping the WATS data converter active. Both systems receive the same data.
2. **Week 2.** Compare results in both dashboards. Verify that measurement values, pass/fail outcomes, and station attribution match.
3. **Week 3.** If everything aligns, disable the WATS data converter on that station. Roll out to the next station.
4. **Repeat** until all stations report to TofuPilot only.

Keep your WATS subscription active during this period. You can export a final data snapshot before canceling.

## What You Get After Migration

Once your tests report to TofuPilot, you'll find the analytics at [tofupilot.app](https://tofupilot.app):

- **FPY trends** by station, product, and time range.
- **Cpk and control charts** for every measurement, computed automatically.
- **Failure Pareto** to identify your top failure modes.
- **Measurement histograms** showing distribution against limits.
- **Full traceability** per serial number across all test runs.
- **REST API** for integrating with your MES, ERP, or custom tooling.

These replace the WATS Analytics module. The difference is that your test execution layer is now open-source (OpenHTF), and you're not locked into a proprietary test framework to get analytics.

### Export Test Data to Power BI
URL: https://www.tofupilot.com/guides/how-to-export-test-data-to-power-bi-with-tofupilot
Learn how to export test data from TofuPilot to Power BI using CSV exports and the REST API for custom management reports and cross-system analysis.

TofuPilot's built-in dashboards cover most manufacturing analytics needs. But sometimes you need to combine test data with ERP, MES, or supply chain data in a single report for management. That's when Power BI comes in.

## When You Need External BI

TofuPilot handles FPY trends, Cpk, control charts, failure Pareto, and throughput natively. You don't need Power BI for any of those.

External BI tools make sense when you need to correlate test results with data from other systems. Examples: mapping yield by supplier lot (ERP data), comparing test costs to warranty claims (finance data), or building executive summaries that pull from five different sources.

## Option 1: CSV Export

The fastest way to get data into Power BI. TofuPilot lets you export test runs and measurements as CSV files directly from the dashboard.

In Power BI Desktop, use **Get Data > Text/CSV** to import the file. This works well for one-time reports or periodic snapshots.

For recurring reports, save the CSV to a shared folder and configure Power BI to refresh from that location on a schedule.

## Option 2: REST API

For automated, always-fresh data, connect Power BI directly to TofuPilot's REST API. This pulls the latest test data every time the report refreshes.

### Getting Your API Key

Generate an API key at [https://tofupilot.app](https://tofupilot.app) under your workspace settings. Store it securely. You'll pass it as a header in every request.

### API Endpoints

TofuPilot's API provides endpoints for the data you'll need in Power BI:

- `GET /api/v1/runs` returns test runs with status, timestamps, DUT ID, and procedure info
- `GET /api/v1/runs/{id}` returns a single run with full measurement data

The API returns JSON. Paginate through results using the `offset` and `limit` query parameters.

```python filename="fetch_runs_example.py"
import requests

API_KEY = "your-api-key"
BASE_URL = "https://tofupilot.app/api/v1"

response = requests.get(
    f"{BASE_URL}/runs",
    headers={"Authorization": f"Bearer {API_KEY}"},
    params={"limit": 100, "offset": 0},
)

runs = response.json()
```

### Connecting Power BI to the API

In Power BI Desktop, use **Get Data > Web** and configure the request:

1. Select **Advanced** in the Web dialog
2. Set the URL to `https://tofupilot.app/api/v1/runs?limit=100`
3. Add the header `Authorization` with value `Bearer your-api-key`
4. Click **OK** to load the JSON response

Power BI's Power Query editor will show the raw JSON. Use **To Table > Expand Columns** to flatten the nested structure into rows and columns.

### Handling Pagination

TofuPilot's API paginates results. To load all runs, create a parameterized query in Power Query that iterates through pages:

```text filename="power_query_pagination.m"
let
    BaseUrl = "https://tofupilot.app/api/v1/runs",
    PageSize = 100,
    GetPage = (offset) =>
        let
            Response = Web.Contents(
                BaseUrl,
                [
                    Query = [limit = Text.From(PageSize), offset = Text.From(offset)],
                    Headers = [Authorization = "Bearer your-api-key"]
                ]
            ),
            Json = Json.Document(Response)
        in
            Json,
    AllPages = List.Generate(
        () => [Page = GetPage(0), Offset = 0],
        each List.Count([Page]) > 0,
        each [Page = GetPage([Offset] + PageSize), Offset = [Offset] + PageSize],
        each [Page]
    ),
    Combined = List.Combine(AllPages),
    AsTable = Table.FromList(Combined, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
    AsTable
```

This query fetches pages until it gets an empty result, then combines everything into one table.

## Building Useful Reports

Once the data is in Power BI, keep the reports focused on what TofuPilot doesn't already show.

**Cross-system correlation.** Join test runs with supplier lot data from your ERP. Build a matrix of yield by supplier, by component, by date range. This answers "which supplier's parts cause the most failures?" without manual spreadsheet work.

**Executive summaries.** Combine yield metrics with production volume and cost data. Show units shipped, test cost per unit, and warranty return rate on a single page.

**Multi-site comparison.** If you run TofuPilot across multiple factories, Power BI can pull from all instances and normalize the data into a single cross-site view.

## Keeping Data Fresh

Set up scheduled refresh in Power BI Service (cloud) to pull new data daily or hourly. For the API connection, store credentials in Power BI's data source settings so refreshes run unattended.

For CSV-based reports, automate the export with a script that runs on a schedule and drops the file where Power BI expects it.

### What Is First Pass Yield and How to Track It
URL: https://www.tofupilot.com/guides/what-is-first-pass-yield-and-how-to-track-it-with-tofupilot
First pass yield is the share of units passing without rework. How to define it honestly, what corrupts it, and how to track it per stage.

First pass yield (FPY) is the proportion of units that pass a test stage on their first attempt, with no rework and no retest. It is the most quoted number in manufacturing test and the most commonly miscounted.

## The Definition

```
FPY = units passing on the first attempt / units entering the stage
```

The denominator is units that entered, not units that eventually passed. The numerator counts first attempts only. A board that fails, gets reworked, and passes on the second run is a failure for FPY purposes, even though it ships.

That last sentence is where most disagreements start, and it is the whole point of the metric. FPY measures whether the process produces good boards. Final yield measures whether you managed to ship them. Both are useful; conflating them hides the cost of rework.

## FPY vs the Other Yields

| Metric | Counts | Answers |
|--------|--------|---------|
| **First pass yield** | Passed first time / entered | Is the process right the first time? |
| **Final yield** | Passed eventually / entered | What fraction shipped? |
| **Rolled throughput yield (RTY)** | Product of FPY across all stages | What fraction passes every stage untouched? |
| **Throughput** | Units per hour | Is the line fast enough? |

A line with 92 % FPY and 99 % final yield is spending real money on rework that the final number conceals. See [rolled throughput yield](/guides/how-to-track-rolled-throughput-yield-with-tofupilot) for the multi-stage view.

## What Corrupts the Number

**Retests overwriting the original.** If a retest replaces the failing record, FPY silently becomes final yield. The failing run has to survive.

**Operators re-running until it passes.** Three attempts and a pass is not a pass; without the first two recorded, it looks like one.

**Blending stages.** ICT, FCT and final inspection have different failure modes. One combined number tells you the line is at 94 % and nothing about where to look.

**Excluding "not real" failures.** Fixture contact failures and operator errors are real production events that cost time. Excluding them from FPY produces a number that flatters the line and hides a fixture problem.

**Counting reworked units as new units.** A reworked board re-entering the stage inflates the denominator and makes yield look better with every rework cycle.

## Getting It Right in the Data

FPY is a property of your records, not a calculation you apply afterwards. Three things have to be true.

**One run per attempt.** Every execution uploads, whether it passed or failed. Failures are data.

**A stable unit identity.** Every run carries the serial number, so attempt order per unit is reconstructable.

**One procedure per stage.** Separate procedures for ICT, FCT and final gives you three FPY numbers that mean something.

With those, FPY is a query rather than a spreadsheet exercise.

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Quiescent Current
        unit: mA
        validators:
          - operator: "<="
            expected_value: 45.0
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
    measurements.quiescent_current = dmm.read_current_ma()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def read_voltage(self) -> float:
        return 3.31  # Replace with a real instrument read

    def read_current_ma(self) -> float:
        return 41.3
```

Run it with `tofupilot run ./procedure.yaml --upload`. Each attempt on a serial becomes its own run, in order, pass or fail.

## Marking Retests Honestly

When a board is reworked and re-tested, say so on the run. That makes the distinction between first attempts and retests explicit rather than inferred:

```python filename="phases/record_attempt.py"
def record_attempt(run, attempt_type):
    """attempt_type comes from an operator prompt: 'first' or 'retest'."""
    run.metadata["attempt_type"] = attempt_type
    run.metadata["rework_ref"] = ""
```

If the rework had a work order, record its reference too. A month later, "which rework action actually fixed anything" becomes answerable.

## Reading FPY

| Question | What to look at |
|----------|----------------|
| What is FPY this shift? | First pass yield on the procedure, filtered by date |
| Which measurement costs the most units? | [Pareto of failing measurements](/guides/how-to-use-pareto-analysis-for-test-failures-with-tofupilot) |
| Is one station worse? | FPY filtered by station or slot |
| Is a measurement drifting toward its limit? | [Control chart](/guides/how-to-use-control-charts-for-tests-with-tofupilot) on that measurement |
| Are limits too tight for the process? | [Cpk](/guides/what-is-cpk-process-capability-index) per measurement |
| How much rework are we doing? | FPY compared against final yield |

The gap between FPY and final yield is the rework bill. If FPY is 92 % and final is 99.5 %, 7.5 % of production is being touched twice, and that cost is invisible in the final number.

## A Falling FPY Is a Diagnosis Problem, Not a Number Problem

When FPY drops, the number itself tells you nothing about why. The useful sequence is:

1. **Which measurement started failing?** The Pareto usually names one.
2. **Did the values shift, or did the spread widen?** A shifted mean is usually a process or supplier change; a widened spread is usually a fixture or measurement problem.
3. **Is it one station or all of them?** Filter by station. One station means fixture, cabling, or instrument calibration.
4. **When did it start?** Line up the change against build records, paste lots, stencil changes, firmware releases.
5. **Were the limits changed?** A tightened limit produces a yield drop with no process change at all, which is why procedure versioning matters.

## If You Already Run OpenHTF

Existing OpenHTF suites produce the same per-attempt records on the TofuPilot CLI:

```python filename="fct.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
def phase_power(test):
    test.measurements.rail_3v3 = 3.31


test = htf.Test(phase_power, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Overwriting a failed run with the retest | FPY silently becomes final yield | Keep every attempt as its own run |
| One blended yield across stages | No idea which stage is failing | One procedure per stage |
| Excluding fixture failures | A fixture problem stays invisible | Count them; they cost real time |
| Counting reworked units in the denominator again | Yield improves with every rework cycle | Denominator is units entering, once |
| No serial number on the run | Attempt order cannot be reconstructed | Always identify the unit |
| Chasing the number instead of the measurement | Yield meetings with no root cause | Start from the failing measurement's distribution |

### Open Source Test Executive Alternatives
URL: https://www.tofupilot.com/guides/open-source-test-executive-alternatives
Compare open source test executives: TofuPilot Framework, OpenHTF, OpenTAP, pytest and HardPy. Features, operator UI, cost, and when to use each.

NI TestStand costs $3-5K per seat and runs only on Windows. Engineers are searching for open source alternatives that run on Linux, integrate with Git, and don't require proprietary licenses. This guide compares the main open source test executives for manufacturing: the TofuPilot Framework, OpenHTF, OpenTAP, pytest (with hardware plugins), and HardPy.

## What a Test Executive Does

A test executive manages the execution of test sequences. It handles:

| Function | What It Does |
|----------|-------------|
| Sequencing | Runs test steps in order with branching and looping |
| Measurement collection | Records values with units and limits |
| Pass/fail decisions | Compares measurements to limits, determines result |
| Operator interaction | Prompts for input, displays status |
| Data logging | Stores results for traceability and analytics |
| Plug/driver management | Initializes and tears down instrument connections |

TestStand does all of this in a proprietary Windows application. The open source alternatives do it in code.

## Comparison Table

| Feature | NI TestStand | TofuPilot Framework | OpenHTF | OpenTAP | pytest + plugins | HardPy |
|---------|-------------|--------------------|---------|---------|-----------------|--------|
| Language | LabVIEW/C#/VB | Python (Rust engine) | Python | C#/.NET | Python | Python |
| License | Proprietary ($3-5K) | MIT | Apache 2.0 | MPL 2.0 | MIT | Apache 2.0 |
| Platform | Windows only | Windows/Linux/macOS | Windows/Linux/macOS | Windows/Linux | Windows/Linux/macOS | Windows/Linux/macOS |
| Origin | National Instruments | TofuPilot | Google | Keysight | Community | EverPin |
| Measurements with limits | Yes | Yes (YAML validators) | Yes | Yes | Via plugins | Yes |
| Phase/step sequencing | Visual editor | YAML procedure | Python decorators | C# classes | Test functions | Test functions |
| Operator UI | Built-in OIs | Declarative, from the procedure | Station Server | Operator Panel | None built-in | Browser panel |
| Parallel phases | Yes | Yes (Rust engine) | No | Yes | Via xdist | No |
| Multi-slot fixtures | Yes | Yes | Limited | Yes | Manual | No |
| Instrument plugins | NI drivers | Plugs (plain classes) | Plugs system | DUT/Instrument plugins | Fixtures | Fixtures |
| Deploy to stations | Manual | Git push, instant rollback | Manual | Manual | Manual | Manual |
| Data export | ATML, custom | JSON, dashboard | JSON, protobuf | CSV, custom | JUnit XML | JSON |
| Version control | Difficult (binary) | Git-native | Git-native | Git-native | Git-native | Git-native |
| Community | Large (NI forums) | Newer, smaller | Small (GitHub) | Small (forum) | Large (general pytest) | Tiny |

## TofuPilot Framework

The TofuPilot Framework is an open source test framework purpose-built for hardware, released under the MIT license. A procedure is a YAML file that declares the sequence and the limits; phases are plain Python functions and plugs are plain Python classes.

### Strengths

- **Declarative procedure.** The test plan lives in YAML as data, readable and diffable separately from the logic.
- **Parallel execution.** A Rust engine runs independent phases and multiple fixture slots concurrently while your code stays plain Python. This is the one that cuts cycle time on a throughput-bound line.
- **Operator UI from the procedure.** Text inputs, checklists, images, sliders and progress bars are declared in the YAML, with no frontend work.
- **Rich measurements.** Aggregations (mean, min, max, std over arrays) and per-axis validators that check the shape of a curve, not just its endpoints.
- **Deployment built in.** Procedures deploy to stations on a Git push with instant rollback, and runs queue offline and sync when the network returns.

### Weaknesses

- **Newer project.** Smaller community and fewer public examples than OpenHTF.
- **Two files, not one.** The YAML layer is an extra concept to learn if you want everything in Python.

### Example

```yaml filename="procedure.yaml"
name: Supply Voltage Check
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Voltage Check
    python: phases.voltage_check
    measurements:
      - name: Supply Voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.9
          - operator: "<="
            expected_value: 5.1
```

```python filename="phases/voltage_check.py"
def voltage_check(measurements, dmm):
    measurements.supply_voltage = dmm.measure_voltage()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def measure_voltage(self) -> float:
        return 5.01  # Replace with a real instrument read
```

Run it with `tofupilot run ./procedure.yaml`, which executes on your machine without an account.

## OpenHTF

OpenHTF is Google's open source test framework for manufacturing. It was built for testing consumer electronics at scale.

### Strengths

- **Python-native.** Tests are Python functions decorated with measurements. No visual editor, no XML, no binary files.
- **Plug system.** Instruments are injected into test phases automatically. Plugs handle initialization and teardown.
- **Measurement validators.** Built-in support for ranges, tolerances, regex, and exact match.
- **Station Server.** Web-based operator interface at `localhost:12000`.
- **Proven at scale.** Years of production use, and a larger body of public examples than newer options.

### Weaknesses

- **Documentation.** Official docs from Google don't exist. The community (via openhtf.org) fills the gap.
- **Community size.** ~640 GitHub stars, low Stack Overflow presence.
- **Sequential execution.** Phases run in order against one unit. Parallel phases and multi-slot fixtures are orchestrated outside the framework.
- **Built-in UI.** The Station Server is functional but basic.

### Example

```python filename="openhtf_example.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
)
def phase_voltage_check(test):
    """Measure and validate supply voltage."""
    test.measurements.voltage_V = 5.01


test = htf.Test(phase_voltage_check)
test.execute(test_start=lambda: input("Scan serial: "))
```

Existing OpenHTF suites run natively on the TofuPilot CLI with no rewrite, so choosing a platform does not force a framework migration.

## OpenTAP

OpenTAP is Keysight's open source test automation framework. It's written in C# and runs on .NET.

### Strengths

- **Keysight backing.** Active development, commercial support available.
- **Plugin ecosystem.** DUT, instrument, and result listener plugins.
- **Test plan editor.** GUI for building test sequences without code.
- **Cross-platform.** Runs on Windows and Linux via .NET.

### Weaknesses

- **C#/.NET.** Most test engineers prefer Python. .NET adds deployment complexity on Linux.
- **Operator panel.** Basic, requires Keysight Test Automation license for full features.
- **Learning curve.** Plugin architecture is powerful but complex for simple tests.

### Example

```csharp filename="OpenTapExample.cs"
[Display("Voltage Check", Description = "Measure supply voltage")]
public class VoltageCheck : TestStep
{
    [Display("Voltage")]
    [Unit("V")]
    public double Voltage { get; set; }

    public override void Run()
    {
        Voltage = 5.01;
        if (Voltage < 4.9 || Voltage > 5.1)
            UpgradeVerdict(Verdict.Fail);
        else
            UpgradeVerdict(Verdict.Pass);
    }
}
```

## pytest with Hardware Plugins

pytest is the standard Python testing framework. With the right plugins, it can work for hardware testing.

### Strengths

- **Massive community.** Most Python developers already know pytest.
- **Fixture system.** Handles instrument setup/teardown cleanly.
- **Plugin ecosystem.** Thousands of plugins for reporting, parallel execution, and CI integration.
- **Low learning curve.** If you know Python, you know pytest.

### Weaknesses

- **Not built for manufacturing.** No built-in measurement system with limits, units, or validators.
- **No operator UI.** No prompts, no serial number input, no pass/fail display.
- **No native data logging.** Requires custom code or third-party tools to store results.
- **Designed for software testing.** Concepts like fixtures, marks, and parametrize don't map cleanly to hardware test sequences.

### Example

```python filename="test_hardware.py"
def test_voltage_check(dmm_fixture):
    """Measure and validate supply voltage."""
    voltage = dmm_fixture.measure_voltage()
    assert 4.9 <= voltage <= 5.1, f"Voltage {voltage}V out of range"
```

## HardPy

HardPy adds a browser-based operator panel to pytest for hardware testing.

### Strengths

- **pytest-native.** Tests are regular pytest functions with hardware-specific additions.
- **Browser UI.** Operator panel with test hierarchy, dialogs, and real-time charts.
- **Measurement storage.** Built-in CouchDB integration for test results.
- **Modern stack.** React frontend, clean API.

### Weaknesses

- **New project.** Limited adoption, small community.
- **Basic features.** No measurement limit system, no marginal bands, no analytics.
- **CouchDB dependency.** Adds infrastructure complexity.

## When to Use Each

| Situation | Recommendation |
|-----------|---------------|
| Starting a new production test system in Python | TofuPilot Framework |
| Throughput-bound line, several boards per fixture | TofuPilot Framework (multi-slot) |
| Operators need guided workflows on the station | TofuPilot Framework (declarative UI) |
| Validating waveform or sweep shape | TofuPilot Framework (per-axis validators) |
| Already running OpenHTF in production | Keep OpenHTF; it runs natively on the TofuPilot CLI |
| Want one pure-Python paradigm, no YAML layer | OpenHTF |
| Already using pytest for hardware tests | Add the TofuPilot SDK for data logging and operator UI |
| .NET shop with Keysight instruments | OpenTAP |
| Need the simplest possible operator panel | HardPy |
| Migrating from TestStand, want minimal change | OpenTAP (closest architecture) |
| Migrating from TestStand, want Python | TofuPilot Framework |
| Budget is zero, need it working today | TofuPilot Framework (`tofupilot run`, no account needed) |
| Need analytics, traceability, and operator UI | Any framework + TofuPilot |

## Cost Comparison

| Solution | Software License | Per-Station Cost | Operator UI | Data Analytics |
|----------|-----------------|-----------------|-------------|---------------|
| NI TestStand | $3-5K/seat | $3-5K | Included | NI InsightCM (extra) |
| TofuPilot Framework | Free (MIT) | $0 | Included, declarative | Included |
| OpenHTF + TofuPilot | Free | $0 | Included (all tiers) | Included |
| OpenTAP | Free (core) | $0 | Plugin (may need license) | Custom |
| pytest + TofuPilot | Free | $0 | Via TofuPilot | Included |
| HardPy | Free | $0 | Included | Basic |

The open source alternatives eliminate per-seat licensing. The total cost of ownership depends on engineering time to set up and maintain the system, which is where the operator UI and the deployment story matter as much as the test logic.

For a direct comparison of the two purpose-built Python options, see [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf).

### Test Traceability for Medical Devices
URL: https://www.tofupilot.com/guides/test-traceability-for-medical-devices-fda-with-tofupilot
Map FDA 21 CFR Part 820 and ISO 13485 requirements to TofuPilot features for device history records, test documentation, and audit trails.

FDA 21 CFR Part 820 requires medical device manufacturers to maintain a Device History Record (DHR) for every unit produced. Capturing test results, serial numbers, measurements, and pass/fail determinations in an immutable record maps directly to FDA and ISO 13485 requirements.

## Regulatory Requirements Overview

Medical device test traceability sits at the intersection of two frameworks. Both require documented evidence that each device was manufactured and tested according to approved procedures.

| Regulation | Clause | Requirement | Where it maps |
|---|---|---|---|
| 21 CFR 820.184 | DHR | Records demonstrating device manufactured per DMR | Test records per serial number with procedure name |
| 21 CFR 820.80 | Receiving, in-process, finished device acceptance | Acceptance activities with acceptance/rejection documented | Pass/fail results with measurement limits |
| 21 CFR 820.90 | Nonconforming product | Documented investigation of nonconformances | Failed runs with measurement-level detail |
| 21 CFR 820.180 | General requirements (records) | Records maintained for device lifetime or 2 years | Immutable cloud storage with retention |
| 21 CFR 820.250 | Statistical techniques | Valid statistical techniques for process capability | Cpk, control charts, first pass yield |
| ISO 13485 7.5.3 | Traceability | Records per unit or batch through production | Serial tracking, unit history, sub-assembly links |
| ISO 13485 8.2.4 | Monitoring and measurement | Evidence of conformity to acceptance criteria | Measurements with units, limits, and pass/fail |

This table maps requirements to where the evidence lives. It is not a compliance claim: your quality system, validation, and procedures are what satisfy the regulation, and the test records are one input to that. Software used in production also falls under 21 CFR 820.70(i), so the tooling itself needs validation for its intended use.

## Structuring Tests for DHR Compliance

The DHR must include test results that prove each device was manufactured according to its Device Master Record (DMR). Structure your OpenHTF tests so each procedure maps to a documented test protocol.

```python filename="test_medical_device.py"
# Medical device production test aligned with DHR requirements
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("impedance_channel_1")
        .in_range(minimum=980, maximum=1020)
        .with_units(units.OHM)
        .doc("Electrode impedance per IEC 60601-1, channel 1"),
    htf.Measurement("impedance_channel_2")
        .in_range(minimum=980, maximum=1020)
        .with_units(units.OHM)
        .doc("Electrode impedance per IEC 60601-1, channel 2"),
    htf.Measurement("leakage_current")
        .in_range(maximum=0.00001)
        .with_units(units.AMPERE)
        .doc("Patient leakage current, Type BF applied part per IEC 60601-1"),
)
def electrical_safety(test):
    test.measurements.impedance_channel_1 = 1002.3
    test.measurements.impedance_channel_2 = 997.1
    test.measurements.leakage_current = 0.000003


@htf.measures(
    htf.Measurement("signal_accuracy_pct")
        .in_range(minimum=99.0, maximum=101.0)
        .doc("Measurement accuracy against NIST-traceable reference"),
    htf.Measurement("noise_floor")
        .in_range(maximum=0.000005)
        .with_units(units.VOLT)
        .doc("Input-referred noise, 0.05-150 Hz bandwidth"),
    htf.Measurement("common_mode_rejection")
        .in_range(minimum=100)
        .doc("CMRR at 50/60 Hz, in dB"),
)
def signal_performance(test):
    test.measurements.signal_accuracy_pct = 99.8
    test.measurements.noise_floor = 0.0000021
    test.measurements.common_mode_rejection = 112.4


@htf.measures(
    htf.Measurement("firmware_version")
        .doc("Installed firmware version, must match DMR revision"),
    htf.Measurement("self_test_result")
        .in_range(minimum=1, maximum=1)
        .doc("Device self-test pass (1) or fail (0)"),
    htf.Measurement("battery_capacity_pct")
        .in_range(minimum=95)
        .doc("Battery capacity relative to rated"),
)
def system_validation(test):
    test.measurements.firmware_version = "3.2.1-release"
    test.measurements.self_test_result = 1
    test.measurements.battery_capacity_pct = 98.7


def main():
    test = htf.Test(
        electrical_safety,
        signal_performance,
        system_validation,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="MD-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "MD-2026-04-00192")


if __name__ == "__main__":
    main()
```

Install it with `pip install "tofupilot[openhtf]"`. Each measurement references the applicable standard through `.doc()` and carries its acceptance limits, so the stored record shows what was required as well as what was measured.

Version the procedure whenever limits change. A DHR is only defensible if you can say which acceptance criteria applied to a given unit on the day it was tested, and that becomes impossible once limits have moved without a version marker.

## Acceptance Activities (21 CFR 820.80)

The FDA requires documented acceptance activities at receiving, in-process, and finished device stages. Run separate procedures for each stage, all using the same serial number.

```python filename="test_acceptance_stages.py"
# Three-stage acceptance testing for FDA 820.80 compliance
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

RECEIVING_PROCEDURE_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
IN_PROCESS_PROCEDURE_ID = "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"
FINISHED_PROCEDURE_ID = "zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz"


@htf.measures(
    htf.Measurement("incoming_visual_pass")
        .in_range(minimum=1, maximum=1),
    htf.Measurement("incoming_dimension_check")
        .in_range(minimum=24.9, maximum=25.1)
        .with_units(units.MILLIMETRE),
)
def receiving_inspection(test):
    test.measurements.incoming_visual_pass = 1
    test.measurements.incoming_dimension_check = 25.02


@htf.measures(
    htf.Measurement("solder_paste_height")
        .in_range(minimum=0.10, maximum=0.15)
        .with_units(units.MILLIMETRE),
    htf.Measurement("component_presence")
        .in_range(minimum=1, maximum=1),
)
def in_process_inspection(test):
    test.measurements.solder_paste_height = 0.12
    test.measurements.component_presence = 1


@htf.measures(
    htf.Measurement("final_functional_test")
        .in_range(minimum=1, maximum=1),
    htf.Measurement("label_verification")
        .in_range(minimum=1, maximum=1),
)
def finished_device_acceptance(test):
    test.measurements.final_functional_test = 1
    test.measurements.label_verification = 1


def run_stage(phase, procedure_id, serial):
    test = htf.Test(
        phase,
        procedure_id=procedure_id,
        part_number="MD-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: serial)


def main():
    serial = "MD-2026-04-00192"
    run_stage(receiving_inspection, RECEIVING_PROCEDURE_ID, serial)
    run_stage(in_process_inspection, IN_PROCESS_PROCEDURE_ID, serial)
    run_stage(finished_device_acceptance, FINISHED_PROCEDURE_ID, serial)


if __name__ == "__main__":
    main()
```

The three stages run on separate procedures so each can be reported independently, while the shared serial number keeps them on one unit history. In production these are three scripts on three stations, days apart; they are shown together here only to make the pattern visible.

The unit page for this serial shows all three acceptance stages in order, which is what an inspector asks to see.

## Nonconforming Product (21 CFR 820.90)

When a device fails testing, the FDA requires documented investigation. The record holds the exact measurement that triggered the failure, its value, and the limit it exceeded.

Failed runs stay visible in the unit's history alongside every retest, so the investigation, rework, and retest cycle is traceable. That covers the record-keeping side of 820.90. The investigation and disposition decisions themselves live in your CAPA system; the test data is evidence feeding it, not a substitute for it.

## Component Traceability with Sub-Units

ISO 13485 clause 7.5.3 requires traceability of components used in each device. `sub_units` is a field on `htf.Test(...)` that links component serial numbers to the finished device.

```python filename="test_component_traceability.py"
# Record component serials for ISO 13485 traceability
import openhtf as htf
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("assembly_complete")
        .in_range(minimum=1, maximum=1),
)
def final_assembly(test):
    test.measurements.assembly_complete = 1


def main():
    test = htf.Test(
        final_assembly,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        part_number="MD-100",
        sub_units=[
            {"serial_number": "SENSOR-2026-08812"},
            {"serial_number": "PCBA-2026-04410"},
            {"serial_number": "BATT-2026-11023"},
        ],
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "MD-2026-04-00192")


if __name__ == "__main__":
    main()
```

The serials are hardcoded here for brevity. On a real line they come from barcode scans collected before the test is constructed, since `sub_units` is a constructor argument.

If a component supplier issues a recall, the unit hierarchy is what tells you which finished devices contain the affected parts, rather than quarantining everything built in a date range.

## Audit Trail and Exports

The records support what FDA and ISO 13485 auditors ask for:

- **Device History Record.** Every test run for a serial number, with procedure, station, operator, measurements, and result. This forms the test portion of the DHR.
- **Process capability.** Cpk values and control charts for any measurement, supporting statistical process control per 21 CFR 820.250.
- **Failure analysis.** Failure Pareto and measurement histograms to support CAPA investigations.
- **Record retention.** Immutable storage, with historical records not alterable through the standard interface.

Export test data from the dashboard for inclusion in your formal DHR documentation package. Records include timestamps, station identifiers, and all measurement values with their acceptance criteria.

Confirm retention periods against your own device lifetime requirement, and keep an export path that does not depend on a single vendor. 820.180 obligations outlive any particular tool.

### How to Build a Test Operator UI in Python
URL: https://www.tofupilot.com/guides/how-to-build-a-test-operator-ui-in-python
Build a production-ready operator interface for hardware tests using Python, OpenHTF, and TofuPilot. No frontend code required.

Most test engineers start with terminal output. That works during development, but production operators need a dedicated screen with pass/fail indicators, prompts, and input fields. This guide shows how to build an operator UI using OpenHTF and TofuPilot without writing any frontend code.

## What the Operator Sees

The finished operator interface runs in a browser and shows:

- Serial number entry (barcode scanner or manual input)
- Live phase progress as the test runs
- Operator prompts with input fields (text, numbers, dropdowns, image-based choices)
- Pass/fail result with color-coded display
- Measurement values with limit status

The operator never sees a terminal, a file browser, or Python code.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK 1.11.0+ installed (`pip install tofupilot`)

## Step 1: Create Test Phases with Prompts

Use OpenHTF's `user_input` plug to add operator interactions. Each prompt pauses the test until the operator responds.

```python filename="operator_ui_test.py"
import openhtf as htf
from openhtf.plugs import user_input
from openhtf.util import units


@htf.plug(prompts=user_input.UserInput)
def phase_load_dut(test, prompts):
    """Wait for the operator to load the DUT into the fixture."""
    prompts.prompt(
        "Place the board in the test fixture and close the clamp. "
        "Press Enter when ready."
    )


@htf.plug(prompts=user_input.UserInput)
@htf.measures(
    htf.Measurement("label_present").with_args(docstring="Label inspection result"),
)
def phase_visual_check(test, prompts):
    """Ask the operator to verify the label is present and correct."""
    result = prompts.prompt(
        "Is the product label present and correctly aligned?",
        text_input=True,
    )
    test.measurements.label_present = result
```

## Step 2: Add Automated Measurements

Mix operator prompts with automated instrument measurements. The operator sees both types of phases in the same interface.

```python filename="operator_ui_test.py"
@htf.measures(
    htf.Measurement("supply_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("boot_time_ms")
    .in_range(maximum=2000)
    .with_units(units.MILLISECOND),
)
def phase_power_and_boot(test):
    """Automated: measure supply voltage and boot time."""
    test.measurements.supply_voltage_V = 5.02
    test.measurements.boot_time_ms = 1340


@htf.measures(
    htf.Measurement("wifi_rssi_dBm")
    .in_range(minimum=-70, maximum=-20)
    .with_units(units.DBM),
)
def phase_wireless_check(test):
    """Automated: verify WiFi signal strength."""
    test.measurements.wifi_rssi_dBm = -42
```

## Step 3: Add a Final Operator Step

After automated tests complete, ask the operator to unload the DUT and apply a pass/fail sticker.

```python filename="operator_ui_test.py"
@htf.plug(prompts=user_input.UserInput)
def phase_unload(test, prompts):
    """Instruct the operator to remove the DUT and apply the label."""
    prompts.prompt(
        "Remove the board from the fixture. "
        "Apply a PASS sticker if the test passed. "
        "Press Enter when done."
    )
```

## Step 4: Connect to TofuPilot and Run

Wire all phases together and connect to TofuPilot. The operator UI streams automatically to a browser URL.

```python filename="operator_ui_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_load_dut,
    phase_visual_check,
    phase_power_and_boot,
    phase_wireless_check,
    phase_unload,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

When you run this script, TofuPilot prints a URL in the console. Open it in a browser on the operator's station. The operator sees each phase in sequence, responds to prompts, and gets a clear pass/fail result.

## Step 5: Configure the Station for Production

| Setting | How |
|---------|-----|
| Kiosk mode | Launch Chrome with `--kiosk --app=<streaming-url>` |
| Auto-start | Add the command to the OS startup script |
| Barcode scanner | Configure as USB HID keyboard (scan goes into the active input field) |
| Large text | Set browser zoom to 150% |
| Touchscreen | TofuPilot prompts work with touch input, no mouse needed |
| Multiple stations | Each station gets its own streaming URL |

## Input Types Available

TofuPilot's operator UI supports structured input types beyond simple text:

| Input Type | Use Case |
|-----------|----------|
| Text | Serial numbers, operator notes |
| Number | Manual measurements, counts |
| Slider | Analog adjustments, subjective ratings |
| Radio buttons | Choose one option from a list |
| Checkbox | Confirm multiple items (checklist) |
| Dropdown | Select from a long list of options |
| Image choice | Pick from photos (component orientation, defect type) |
| Toggle | Yes/no decisions |

These inputs render automatically in the browser. The operator fills them in, and the values flow back to the test script as measurement data.

## Common Patterns

### Fail-Fast with Operator Notification

```python filename="operator_ui_test.py"
from openhtf import PhaseResult


@htf.measures(
    htf.Measurement("power_good").equals("PASS"),
)
def phase_power_check(test):
    """Stop early if power-up fails. No point testing a dead board."""
    result = "PASS"
    test.measurements.power_good = result
    if result != "PASS":
        return PhaseResult.STOP
```

When this phase fails, the operator sees the failure immediately in the browser with the failing measurement highlighted. They don't wait for the remaining phases to time out.

### Conditional Operator Steps

```python filename="operator_ui_test.py"
@htf.plug(prompts=user_input.UserInput)
@htf.measures(
    htf.Measurement("rework_action").with_args(docstring="Rework performed"),
)
def phase_rework_prompt(test, prompts):
    """If a previous phase flagged an issue, ask operator to rework."""
    action = prompts.prompt(
        "The solder inspection flagged pad U3. "
        "Rework the joint and press Enter when done.",
        text_input=True,
    )
    test.measurements.rework_action = action
```

### Test Traceability for Aerospace (AS9100)
URL: https://www.tofupilot.com/guides/test-traceability-for-aerospace-as9100-with-tofupilot
Build AS9100 test traceability for aerospace: what records auditors ask for, and how to capture them automatically with a TofuPilot procedure.

AS9100 is the aerospace quality management standard, built on ISO 9001 with additional requirements for configuration management, risk, and traceability. For a test engineer, the practical question is narrower: when an auditor picks a shipped serial number, can you show what was tested, with which equipment, against which revision of the limits, by whom, and when? This guide covers the records AS9100 expects from test, and how to capture them without manual paperwork.

## What AS9100 Asks of Test Data

| Clause area | What it means for test | Record you must produce |
|---|---|---|
| Configuration management (8.1.2) | The test that ran matches the released revision | Procedure version recorded on every run |
| Traceability (8.5.2) | Every unit is identifiable through the process | Serial number on every run, linked to sub-assemblies |
| Monitoring equipment (7.1.5) | Instruments were in calibration when used | Instrument ID and calibration due date per run |
| Nonconforming output (8.7) | Failures were detected, segregated, dispositioned | Failed runs retained, with disposition |
| Retention of records (7.5.3) | Records survive and stay readable | Immutable run history, exportable |
| Competence (7.2) | The operator was qualified | Operator identity on the run |

The theme is that a pass/fail verdict is not a record. The record is the measured value, its limits, the revision of those limits, the equipment, the operator and the timestamp.

## Step 1: Version the Procedure, Not Just the Code

AS9100 configuration management cares that the test which ran is the released one. Put an explicit version in the procedure and let every run carry it.

```yaml filename="procedure.yaml"
name: LRU Acceptance Test
version: 2.3.0
description: Acceptance test for the 28V power conditioning LRU, per ATP-4417 rev C

unit:
  serial_number:
    default_value: "LRU000001"
  part_number:
    default_value: "LRU-4417"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Output Voltage
    python: phases.output_voltage
    measurements:
      - name: Output Voltage
        unit: V
        description: Per ATP-4417 rev C section 4.2
        validators:
          - operator: ">="
            expected_value: 27.5
          - operator: "<="
            expected_value: 28.5
      - name: Ripple
        unit: mV
        description: Per ATP-4417 rev C section 4.3
        validators:
          - operator: "<="
            expected_value: 150
```

```python filename="phases/output_voltage.py"
def output_voltage(measurements, dmm):
    measurements.output_voltage = dmm.measure_dc_voltage()
    measurements.ripple = dmm.measure_ripple_mv()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def measure_dc_voltage(self) -> float:
        return 28.02  # Replace with a real instrument read

    def measure_ripple_mv(self) -> float:
        return 96.0  # Replace with a real instrument read
```

The `version` field is recorded with every run, so results taken before a limit change stay interpretable afterwards. The `description` on each measurement is where the acceptance test procedure paragraph goes, so the auditor's document and your data line up without a translation step.

Deployments add the other half: the CLI records the exact commit that ran on the station, so "which code produced this result" has an answer that does not depend on anyone's memory.

## Step 2: Record the Equipment and Its Calibration

Clause 7.1.5 wants evidence that monitoring equipment was fit for use. Read the identity from the instrument rather than typing it, and record the calibration due date alongside it.

```python filename="phases/record_equipment.py"
def record_equipment(run, dmm):
    """Capture instrument identity and calibration status on the run."""
    run.metadata["dmm_id"] = dmm.identify()
    run.metadata["dmm_cal_due"] = dmm.calibration_due()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def identify(self) -> str:
        return "Keysight 34461A,MY59001234"  # Replace with *IDN? query

    def calibration_due(self) -> str:
        return "2027-04-18"  # Replace with your calibration system lookup
```

Making the calibration date a measurement rather than metadata turns it into a gate, which is stronger: a station running an out-of-calibration instrument fails the run instead of quietly producing records an auditor will later reject.

```yaml filename="procedure.yaml"
  - name: Equipment Check
    python: phases.equipment_check
    measurements:
      - name: Calibration Valid
        description: Instrument calibration in date at time of test
        validators:
          - operator: "=="
            expected_value: true
```

```python filename="phases/equipment_check.py"
from datetime import date


def equipment_check(measurements, dmm):
    due = date.fromisoformat(dmm.calibration_due())
    measurements.calibration_valid = due >= date.today()
```

## Step 3: Identify the Operator

Competence (7.2) is about qualified people. The run needs to say who ran it. On a station, prompt for a badge scan as an operator input, and record it.

```python filename="phases/record_operator.py"
def record_operator(run, operator_badge):
    run.metadata["operator_badge"] = operator_badge
```

For unattended or CI-driven runs there is no operator, and the honest record says so rather than inventing one.

## Step 4: Link Sub-Assemblies

Traceability (8.5.2) rarely stops at the top-level unit. An LRU contains boards, and a board contains a screened component lot. Model the child units so an auditor pulling a shipped serial can walk down, and a containment action can walk up from a suspect lot to every aircraft it reached.

See [how to build full unit traceability](/guides/how-to-build-full-unit-traceability-with-tofupilot) for the sub-assembly setup.

## Step 5: Keep the Failures

A common and expensive mistake is deleting or re-running over failed results to keep a yield number clean. AS9100 clause 8.7 expects nonconforming output to be identified and controlled, which means the failed run is evidence and must survive.

Keep the failing run, record the disposition, and let the retest be its own run under the same serial. The history then shows the failure, the decision and the outcome in order, which is exactly what an auditor is looking for.

## Step 6: Run and Retain

```bash filename="run.sh"
tofupilot run ./procedure.yaml --upload
```

Each run stores the measured values with their limits, the procedure version, the deployment commit, the equipment metadata, the operator, and timestamps. Records are retained and exportable, which covers 7.5.3 without a file server full of spreadsheets nobody can open in five years.

## If You Already Run OpenHTF

Existing OpenHTF suites run natively on the TofuPilot CLI with no rewrite, and produce the same records. The equivalent of the version field and the equipment metadata looks like this:

```python filename="lru_acceptance.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(27.5, 28.5)
    .with_units(units.VOLT)
    .doc("Per ATP-4417 rev C section 4.2"),
    htf.Measurement("ripple_mV")
    .in_range(maximum=150)
    .with_units(units.MILLIVOLT)
    .doc("Per ATP-4417 rev C section 4.3"),
)
def phase_output(test):
    test.measurements.output_voltage_V = 28.02
    test.measurements.ripple_mV = 96.0
    test.metadata["procedure_revision"] = "2.3.0"
    test.metadata["dmm_id"] = "Keysight 34461A,MY59001234"


test = htf.Test(phase_output, part_number="LRU-4417")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Audit Readiness Checklist

| Question an auditor asks | Where the answer lives |
|---|---|
| What was tested on this serial? | The run history for that unit |
| Against which limits? | The measurements stored with the run, including limits |
| Which revision of the test? | The procedure `version` and the deployment commit |
| With what equipment? | Equipment metadata, or a calibration gate measurement |
| Was the equipment in calibration? | The calibration measurement outcome |
| Who ran it? | Operator metadata on the run |
| What happened to the failures? | Retained failing runs plus the disposition and retest |
| What else used that component lot? | Sub-assembly links from the child serial upward |

## Common Mistakes

| Mistake | Consequence | Fix |
|---|---|---|
| Recording pass/fail only | Cannot show margin or drift to an auditor | Store the measured value with its limits |
| Unversioned procedures | Old results become uninterpretable after a limit change | Set `version` and change it deliberately |
| Typing instrument IDs by hand | Transcription errors in the audit trail | Read `*IDN?` from the instrument |
| Calibration tracked in a spreadsheet | Out-of-cal equipment produces records you must scrap | Gate the run on a calibration measurement |
| Deleting failed runs | Nonconformance evidence destroyed | Retain the failure, record the disposition |
| No sub-assembly links | Containment means scrapping everything | Model child units from day one |

### Run-over-Run Test Comparison
URL: https://www.tofupilot.com/guides/run-over-run-test-comparison-with-tofupilot
Learn how to compare hardware test runs side by side in TofuPilot to diagnose failures and track measurement changes across units.

When a unit fails, the first question is always: "What's different from a passing unit?" Run-over-run comparison in TofuPilot lets you put two or more test runs side by side and see exactly where they diverge.

## When to Use Run Comparison

- **Diagnosing a failure**: Compare a failed run to a recent passing run for the same procedure
- **Investigating a retest**: Compare first test to retest for the same unit
- **Validating a fix**: Compare runs before and after a corrective action
- **Tracking unit history**: Compare the same unit's results across different test stages
- **Benchmarking stations**: Compare the same unit tested on different stations

## How Run Comparison Works

TofuPilot stores every measurement from every run. When you compare runs, the system aligns measurements by name and shows the values side by side with their limits.

```
                    Run A (Pass)      Run B (Fail)
                    ────────────      ────────────
vcc_3v3             3.30 V    ✓      3.28 V    ✓
vcc_1v8             1.81 V    ✓      1.74 V    ✗
clk_freq            24.00 MHz ✓      23.98 MHz ✓
boot_time           320 ms    ✓      1,240 ms  ✗
current_idle        45 mA     ✓      78 mA     ✗
```

Three measurements differ significantly. The 1.8V rail is low, boot time is 4x longer, and idle current is 73% higher. These symptoms together point to a partial short on the 1.8V power rail causing excess current draw and slow boot.

## Comparing Runs in TofuPilot

### Step 1: Find the Runs

Navigate to the procedure page and filter to find the runs you want to compare. Common filters:

| Filter | Use case |
|--------|----------|
| Status: Failed | Find failing runs to diagnose |
| Serial number | Find all runs for a specific unit |
| Date range | Narrow to a time period |
| Station | Compare across stations |

### Step 2: Select Runs for Comparison

Select two or more runs from the run list. TofuPilot aligns their measurements by step name and measurement name.

### Step 3: Read the Comparison

Focus on measurements where the values differ significantly. Small variations (3.30V vs. 3.31V) are normal measurement noise. Large deviations (1.81V vs. 1.74V) indicate a real difference.

Color coding helps:
- **Green**: Both values within limits
- **Red**: Value outside limits
- **Yellow**: Value within limits but significantly different from the reference

## Common Comparison Patterns

### Pattern 1: Single Measurement Failure

One measurement fails, everything else is identical. This usually means:
- Component value out of tolerance
- Solder defect on that specific circuit
- Test probe contact issue (retest to confirm)

### Pattern 2: Correlated Failures

Multiple related measurements fail together (e.g., voltage low + current high + boot slow). This points to a systemic issue:
- Power rail problem affecting multiple circuits
- Firmware crash causing downstream test failures
- Fixture contact issue on a shared connection

### Pattern 3: All Measurements Shifted

Every measurement is slightly different from the reference, but most are still within limits. This suggests:
- Different environmental conditions (temperature affecting all measurements)
- Different station (instrument calibration differences)
- Different component lot (systematic parameter shift)

### Pattern 4: Intermittent Failure

Same unit, same station, same procedure. Sometimes passes, sometimes fails. Compare the passing and failing runs:
- If the failing measurement is always the same one, it's a marginal value near a limit
- If different measurements fail each time, it's likely a contact issue (pogo pin, cable)
- If the pattern is time-dependent, check for thermal effects

## Comparing Across Production Batches

Run comparison isn't just for debugging. Use it to validate that a new production batch matches the previous one.

1. Select a representative passing run from batch N
2. Select the first runs from batch N+1
3. Compare measurement distributions

If batch N+1 measurements are systematically shifted (even if still within limits), investigate before the full batch runs through production.

## Using the API for Programmatic Comparison

```python filename="compare_runs.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get two runs to compare
run_pass = client.get_run(run_id="run-id-pass")
run_fail = client.get_run(run_id="run-id-fail")

# Compare measurements
for step_p, step_f in zip(run_pass["steps"], run_fail["steps"]):
    for m_p, m_f in zip(step_p["measurements"], step_f["measurements"]):
        diff = abs(m_p["value"] - m_f["value"])
        if diff > 0:
            pct = diff / m_p["value"] * 100 if m_p["value"] != 0 else float("inf")
            status = "DIFF" if pct > 5 else "ok"
            print(f"{m_p['name']:30s} {m_p['value']:10.3f} {m_f['value']:10.3f} {pct:6.1f}% {status}")
```

This script highlights measurements that differ by more than 5%, giving you a quick programmatic way to identify where two runs diverge.


### Collaborative Test Data Analysis
URL: https://www.tofupilot.com/guides/collaborative-test-analysis-with-tofupilot
Learn how to share hardware test data across teams using TofuPilot's centralized platform for collaborative debugging and quality analysis.

Hardware debugging is a team sport. The test engineer sees the failure. The design engineer understands the circuit. The manufacturing engineer knows the process. But they're all looking at different data in different tools. TofuPilot puts everyone on the same page.

## The Collaboration Problem

When a test fails, the investigation usually goes like this:

1. Test engineer sees the failure on the station PC
2. Test engineer screenshots the data or exports a CSV
3. Test engineer emails the CSV to the design engineer
4. Design engineer asks for more context ("What station? What lot? What were the other measurements?")
5. Test engineer goes back to the station, pulls more data, sends another email
6. Manufacturing engineer gets looped in, asks for different data
7. Repeat

Every handoff loses context. Every email is a snapshot that's already outdated. The investigation takes days instead of hours.

## How TofuPilot Enables Collaboration

### Single Source of Truth

Every test result from every station lives in TofuPilot. When someone asks "What happened to unit UNIT-5501?" everyone looks at the same data.

No more "Which version of the spreadsheet are you looking at?" No more "Can you re-export with the timestamps included?"

### Shared Dashboards

TofuPilot's dashboards are accessible to everyone on the team. The test engineer, design engineer, manufacturing engineer, and quality manager all see the same metrics, the same trends, the same failure paretos.

Share a dashboard link instead of attaching a report. The recipient sees live data, can filter and drill down, and can explore on their own without asking the test engineer for help.

### Investigation Workflow

When a quality issue surfaces, the collaborative investigation looks like this:

1. Quality engineer notices a yield drop on the TofuPilot dashboard
2. They filter to see which station, which time period, which failure mode
3. They share the filtered view link with the test engineer: "Station 3 started failing Power Rail Check at 2 PM"
4. Test engineer opens the link, sees the exact measurements, compares with passing runs
5. Design engineer opens the same link, recognizes the measurement pattern: "That 1.8V drop looks like a decoupling cap issue"
6. Manufacturing engineer checks the BOM: "New capacitor lot arrived this morning"

Same data, different expertise, one platform. The issue is identified in 30 minutes, not 3 days.

## Team Roles in TofuPilot

| Role | What they look at | How they use it |
|------|------------------|----------------|
| Test engineer | Individual run results, station status | Debug test failures, maintain stations |
| Design engineer | Measurement trends, distributions | Identify design margin issues |
| Manufacturing engineer | Yield trends, failure paretos | Process optimization, supplier issues |
| Quality manager | FPY dashboards, compliance records | Release decisions, audit evidence |
| Field engineer | Unit history by serial number | Diagnose field returns |

Everyone uses the same platform but focuses on different views.

## Cross-Team Visibility

### Test to Design

Design engineers often don't see production test data until something goes wrong. With TofuPilot, they can proactively monitor measurement distributions for their circuits.

A design engineer who sees that their 3.3V rail measurements are clustering at 3.34V (near the 3.35V limit) can adjust the design or tighten the component spec before failures start.

### Test to Manufacturing

Manufacturing engineers need to correlate test failures with process variables: which solder profile, which pick-and-place program, which component lot. TofuPilot's structured data makes these correlations possible without manual data alignment.

### Test to Field

When a customer reports an issue, the field engineer searches by serial number and sees every test the unit ever passed. If the unit's production measurements were marginal, that's a likely root cause. If they were nominal, the issue is probably from field conditions or aging.

## Replacing Email-Based Debugging

| Email workflow | TofuPilot workflow |
|---------------|-------------------|
| "Can you send me the data for UNIT-5501?" | Search by serial number |
| "What's the yield been this week?" | Open the procedure dashboard |
| "Which station is failing?" | Station comparison view |
| "Here's my analysis (attached Excel)" | Share a filtered dashboard link |
| "Can you re-export with more columns?" | The recipient filters the data themselves |

Every email in the left column is a context switch and a delay. Every action in the right column takes seconds and is self-service.

## Getting Started

1. **Connect all stations**: Every test station pushes results to TofuPilot. No data silos.
2. **Invite the team**: Give access to everyone who touches test data: test, design, manufacturing, quality, and field teams.
3. **Share dashboard links**: Replace email attachments with live links.
4. **Build the habit**: When someone asks a test data question, the answer starts with "Open TofuPilot" instead of "Let me pull a report."

The biggest impact isn't the tool itself. It's that everyone can answer their own questions about test data without waiting for someone else to extract it for them.


### Set Up Alerts for Yield Drops
URL: https://www.tofupilot.com/guides/how-to-set-up-alerts-for-yield-drops-with-tofupilot
Learn how TofuPilot alerts on yield drops automatically against each line's own baseline, and when to add a custom rule with your own numbers.

A yield drop that goes unnoticed for a full shift can mean hundreds of scrapped units. TofuPilot watches first-pass yield on every procedure and notifies you when it falls out of that line's normal range, so you can intervene before the damage spreads.

## Why Real-Time Alerting Matters

Production problems rarely announce themselves. A solder paste machine runs low and starts producing weak joints. A fixture contact wears down and adds resistance to every measurement. A new component lot has slightly different characteristics.

In all these cases, the yield drop starts small and grows. The earlier you catch it, the fewer units are affected. Waiting for end-of-shift reports or weekly quality reviews is too slow.

## What TofuPilot Can Alert On

TofuPilot monitors your production data and covers six conditions.

| Alert type | What it catches | Automatic |
|------------|-----------------|-----------|
| Yield drop | First-pass yield falls out of the procedure's normal range | Yes |
| Measurement drift | A measurement's mean shifts away from its own baseline | Yes |
| Unit retest threshold | One unit is retested far more than the procedure's norm | Yes |
| Golden sample failed | A known-good reference unit starts failing | Yes |
| Failing sample passed | A known-bad reference unit starts passing | Yes |
| Run failed | A streak of consecutive failed runs | Custom rule only |

The two trend types catch broad degradation. The reference-sample types catch a test that has stopped discriminating at all, which is the failure mode that silently invalidates everything else. Retest thresholds catch the unit nobody wants to admit was retested nine times.

## Automatic Detection Sets No Threshold

This is the part most people expect to configure and do not have to.

There is no yield floor to enter. A floor that is right for a line running at 99% is wrong for a burn-in test that normally runs at 55%, and both are healthy. So TofuPilot measures what each procedure normally does — including how much that procedure normally *wobbles* — and alerts when the current smoothed yield departs from that baseline by more than the line's own variation.

| Severity | Fires at |
|----------|----------|
| Info | ≥ 2σ |
| Warning | ≥ 3σ |
| Critical | ≥ 5σ |

σ (sigma) here means "multiples of this line's own normal swing". A line that habitually moves between 50% and 60% has a wide normal swing, so wandering inside it does not alert. A line that sits rock-steady at 55% has a narrow one, and the same swing does alert.

Two consequences are worth knowing. A procedure needs roughly **275 first-runs over the past 90 days** before automatic yield grading applies, so a brand-new line is silent until it has history. And on near-perfect lines, severity stays at info while fewer than three actual failures sit in recent memory, because above roughly 99.6% yield a single isolated failure is statistically dramatic and practically nothing. The full method is documented in [automatic detection](/docs/alerts#how-alerts-trigger).

## When to Add a Custom Rule

Automatic detection answers "is this unusual for this line?". A custom rule answers a different question: "is this below the number I promised someone?" Use one when the number comes from outside the data — a customer commitment, a ramp target, a line qualification gate.

Custom yield rules are still expressed as a **drop in percentage points against the line's own recent baseline**, not as an absolute floor, and you pick which yield they grade:

| Metric | What it counts |
|--------|----------------|
| First Pass Yield | Units whose first-ever run on the procedure passed. Retests cannot improve it. |
| Last Pass Yield | Units whose last run passed. The state after rework. |
| Run pass rate | Passing runs over all runs. Retest storms drag it down fast. |

You set the drop that matters at each severity, and the scope — a whole procedure, one station, one part. See [alert rules](/docs/alerts/configure-alerts) for the full configuration.

## Produce Clean, Structured Test Data

Alerts are only as good as the data behind them. Write your OpenHTF tests with clear, measurable outputs so TofuPilot can track trends and trigger alerts on the right signals.

```python filename="test_charger_board.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("charge_voltage")
    .in_range(minimum=4.15, maximum=4.25)
    .with_units(units.VOLT),
    htf.Measurement("charge_current")
    .in_range(minimum=0.450, maximum=0.550)
    .with_units(units.AMPERE),
    htf.Measurement("thermal_shutdown_temp")
    .in_range(minimum=80, maximum=90)
    .with_units(units.DEGREE_CELSIUS),
)
def test_charging_circuit(test):
    test.measurements.charge_voltage = 4.19
    test.measurements.charge_current = 0.498
    test.measurements.thermal_shutdown_temp = 85.2

def main():
    test = htf.Test(
        test_charging_circuit,
        station_id="LINE-A-FCT-02",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "CHG-2026-03318")

if __name__ == "__main__":
    main()
```

Each measurement with defined limits feeds TofuPilot's analytics. The tighter and more consistent your naming, the more useful your alerts will be — a measurement renamed halfway through the quarter starts a new series with no history, and a series with no history cannot be graded.

## Design Tests for Better Alerting

You can make alerts more useful by how you structure your tests.

```python filename="test_led_driver.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    # Separate measurements for each channel make alerts specific
    htf.Measurement("led_current_ch1")
    .in_range(minimum=0.018, maximum=0.022)
    .with_units(units.AMPERE),
    htf.Measurement("led_current_ch2")
    .in_range(minimum=0.018, maximum=0.022)
    .with_units(units.AMPERE),
    htf.Measurement("led_current_ch3")
    .in_range(minimum=0.018, maximum=0.022)
    .with_units(units.AMPERE),
    htf.Measurement("led_current_ch4")
    .in_range(minimum=0.018, maximum=0.022)
    .with_units(units.AMPERE),
)
def test_led_channels(test):
    test.measurements.led_current_ch1 = 0.0201
    test.measurements.led_current_ch2 = 0.0198
    test.measurements.led_current_ch3 = 0.0203
    test.measurements.led_current_ch4 = 0.0200

def main():
    test = htf.Test(
        test_led_channels,
        station_id="LINE-B-FCT-01",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "LED-2026-05519")

if __name__ == "__main__":
    main()
```

Splitting measurements by channel (instead of a single pass/fail for all channels) means TofuPilot can alert you that channel 3 is drifting specifically. That's actionable. A generic "LED test failing more" is not.

## Respond to Alerts

When an alert fires, act on it:

1. **Check the data** in TofuPilot. The alert carries the numbers that fired it and a chart of the metric; look at the trend and the distribution to understand what changed.
2. **Narrow down the scope.** Is it one station or all stations? One shift or all shifts? One part or the whole procedure? This points to the root cause faster than any single measurement will.
3. **Fix the root cause.** Recalibrate the equipment, replace the fixture, retrain the operator, quarantine the component lot.
4. **Let it close itself.** A yield alert resolves automatically once the metric recovers well inside the band. Resolving by hand instead snoozes that alert for an hour, which is what you want when you already know and are working on it — and not what you want as a habit.

Alerts are a starting point, not the answer. They tell you something changed. The data in TofuPilot tells you what.

### Robotics Test Data Management
URL: https://www.tofupilot.com/guides/robotics-test-data-management-with-tofupilot
Learn how to manage test data for autonomous vehicles and robotics systems using TofuPilot's multi-stage test tracking and sensor data storage.

Robots go through more test stages than almost any other product category. Board-level tests, motor characterization, sensor calibration, firmware validation, system integration, environmental screening, and field simulation. TofuPilot tracks all of it under one serial number.

## The Robotics Testing Challenge

A typical autonomous robot has:

- 3-5 PCBAs, each with its own functional test
- 6-12 motors or actuators, each needing characterization
- Multiple sensor suites (LiDAR, cameras, IMUs) requiring calibration
- Firmware that needs validation at each revision
- System-level integration tests
- Environmental tests (thermal, vibration, IP rating)
- Field simulation or HIL testing

Each test stage generates data. Without a centralized system, that data lives in different tools, different formats, on different machines. When a robot fails in the field, tracing back to which test stage missed the issue is nearly impossible.

## Setting Up Multi-Stage Test Tracking

### Define Your Test Procedures

Create a procedure in TofuPilot for each test stage in your manufacturing process.

| Procedure ID | Stage | What it tests |
|-------------|-------|--------------|
| `PCBA-MOTOR-CTRL` | Board test | Motor controller PCBA functional test |
| `MOTOR-CHAR` | Subassembly | Motor torque/speed characterization |
| `IMU-CAL` | Sensor cal | IMU offset and sensitivity calibration |
| `SYS-INTEGRATION` | System | Full robot integration checks |
| `THERMAL-CYCLE` | Environmental | Thermal cycling qualification |
| `HIL-NAV` | HIL | Navigation algorithm validation |

### Link Tests to the Same Unit

Every test run references the robot's serial number. TofuPilot automatically links all runs for a given serial, creating a complete test history.

```python filename="motor_controller_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Board-level test for the motor controller
client.create_run(
    procedure_id="PCBA-MOTOR-CTRL",
    unit_under_test={
        "serial_number": "ROBO-2025-0142",
        "part_number": "MC-BOARD-R3",
    },
    run_passed=True,
    steps=[{
        "name": "H-Bridge Driver",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "gate_drive_voltage",
            "value": 12.1,
            "unit": "V",
            "limit_low": 11.5,
            "limit_high": 12.5,
        }],
    }, {
        "name": "Current Sense",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "current_sense_gain",
            "value": 50.2,
            "unit": "mV/A",
            "limit_low": 48.0,
            "limit_high": 52.0,
        }],
    }],
)
```

```python filename="motor_characterization.py"
# Motor characterization for the same robot
client.create_run(
    procedure_id="MOTOR-CHAR",
    unit_under_test={
        "serial_number": "ROBO-2025-0142",
        "part_number": "DRIVE-ASSEMBLY-R2",
    },
    run_passed=True,
    steps=[{
        "name": "Stall Torque",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "stall_torque_nm",
            "value": 2.45,
            "unit": "Nm",
            "limit_low": 2.2,
            "limit_high": 2.8,
        }],
    }, {
        "name": "No-Load Speed",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "no_load_rpm",
            "value": 5820,
            "unit": "RPM",
            "limit_low": 5500,
            "limit_high": 6100,
        }],
    }],
)
```

### Track Sensor Calibration Data

Sensor calibration produces arrays of data. TofuPilot handles multi-dimensional measurements natively.

```python filename="imu_calibration.py"
import numpy as np

# IMU calibration with multi-axis data
client.create_run(
    procedure_id="IMU-CAL",
    unit_under_test={"serial_number": "ROBO-2025-0142"},
    run_passed=True,
    steps=[{
        "name": "Accelerometer Offset",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "accel_offset_x", "value": 0.012, "unit": "g", "limit_low": -0.05, "limit_high": 0.05},
            {"name": "accel_offset_y", "value": -0.008, "unit": "g", "limit_low": -0.05, "limit_high": 0.05},
            {"name": "accel_offset_z", "value": 0.003, "unit": "g", "limit_low": -0.05, "limit_high": 0.05},
        ],
    }, {
        "name": "Gyro Bias",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "gyro_bias_x", "value": 0.15, "unit": "deg/s", "limit_low": -0.5, "limit_high": 0.5},
            {"name": "gyro_bias_y", "value": -0.22, "unit": "deg/s", "limit_low": -0.5, "limit_high": 0.5},
            {"name": "gyro_bias_z", "value": 0.08, "unit": "deg/s", "limit_low": -0.5, "limit_high": 0.5},
        ],
    }],
)
```

## Tracing Field Failures Back to Production

When a robot fails in the field, search by serial number in TofuPilot. You'll see every test it went through:

1. Did the motor controller board pass its functional test? What were the exact measurements?
2. Was the motor characterization nominal, or was it marginal?
3. Were the IMU calibration offsets within spec?
4. Did the system integration test pass cleanly, or were there retests?

This trace often reveals the root cause. A motor that passed characterization at the edge of its torque spec is more likely to fail under real-world loads.

## Fleet-Level Analytics

For robotics companies deploying fleets, TofuPilot's analytics work across your entire production.

- Compare calibration distributions across production batches
- Track motor characterization trends over time
- Identify which test stage catches the most defects
- Correlate field failure rates with production test margins

If robots from batch 47 have a higher field failure rate, pull their production test data and compare measurement distributions against batch 46. The difference in the data points to the root cause.


### Test Data Visualization for Hardware Teams
URL: https://www.tofupilot.com/guides/test-data-visualization-for-hardware-teams
Learn how to build dashboards and visualizations for hardware test data using TofuPilot's built-in analytics and charting tools.

Numbers in a spreadsheet don't show trends. Charts do. TofuPilot turns your raw test measurements into interactive dashboards that surface yield problems, measurement drift, and station issues at a glance.

## Why Visualize Test Data

Hardware test data has three properties that make visualization essential:

1. **Volume.** A production line running 500 units/day across 5 test procedures generates 2,500 test runs daily. You can't read that in a table.
2. **Patterns.** Drift, clustering, bimodal distributions, and outliers are invisible in raw numbers but obvious in a chart.
3. **Context.** A measurement of 3.31V means nothing alone. Plotted against the last 10,000 readings with spec limits overlaid, it tells a story.

## Dashboard Views in TofuPilot

### Procedure Dashboard

Every test procedure in TofuPilot gets an automatic dashboard showing:

| Widget | What it shows |
|--------|--------------|
| FPY trend | First-pass yield over time (daily, weekly, monthly) |
| Run history | Recent runs with pass/fail status and timing |
| Failure pareto | Top failure modes ranked by frequency |
| Measurement histograms | Distribution of each measurement across all runs |
| Measurement trends | Each measurement plotted over time with limits |

No setup required. Upload test data and the dashboards populate automatically.

### Unit History

Search by serial number to see every test a unit has been through. This view shows:

- All test procedures the unit has completed
- Pass/fail status for each procedure
- Full measurement data for every run
- Timeline of when each test was executed

### Station Comparison

Compare performance across test stations. This is the fastest way to find station-specific issues:

- FPY by station
- Measurement distributions by station
- Cycle time by station
- Failure modes by station

If Station 3 has 5% lower yield than the others, the station comparison view makes it obvious.

## Building Effective Test Dashboards

### Start with FPY

First-pass yield is the single most important metric for a production test operation. It tells you what percentage of units pass all tests on the first attempt.

Track FPY at three levels:

1. **Overall FPY** across all procedures. This is your headline number.
2. **FPY per procedure.** Which test is causing the most failures?
3. **FPY per station.** Is one station dragging down the overall number?

### Add Measurement Distributions

For each critical measurement, check the histogram. A healthy process looks like a tight bell curve centered well within the spec limits.

Warning signs in the histogram:

- **Skewed distribution**: Process is biased toward one limit
- **Wide distribution**: High variance, process not well controlled
- **Bimodal distribution**: Two peaks, suggesting mixed populations (different component lots, different operators, different stations)
- **Flat distribution**: No central tendency, process is essentially random within limits

### Use Trend Charts for Drift Detection

Plot critical measurements over time. The trend chart shows each individual reading as a data point, with spec limits drawn as horizontal lines.

Look for:

- **Gradual slope**: Measurement drifting toward a limit (fixture wear, calibration drift)
- **Step change**: Sudden shift in the baseline (new component lot, process change)
- **Increasing scatter**: Variance growing over time (loss of process control)
- **Periodic pattern**: Oscillation tied to time of day or production cycle

### Failure Pareto

The failure pareto chart ranks failure modes by frequency. Focus improvement efforts on the top bar. Fixing the #1 failure mode improves overall yield more than fixing #3, #4, and #5 combined (usually).

## Code: Exporting Data for Custom Visualizations

TofuPilot's built-in dashboards cover most needs. For custom analysis, export data via the API.

```python filename="export_measurements.py"
from tofupilot import TofuPilotClient
import csv

client = TofuPilotClient()

runs = client.get_runs(
    procedure_id="MOTOR-PERFORMANCE",
    limit=1000,
)

with open("motor_data.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["serial", "date", "torque_nm", "current_a", "status"])
    for run in runs:
        for step in run.get("steps", []):
            for m in step.get("measurements", []):
                if m["name"] == "peak_torque":
                    writer.writerow([
                        run["unit_under_test"]["serial_number"],
                        run["created_at"],
                        m["value"],
                        None,
                        "pass" if run["run_passed"] else "fail",
                    ])
```

## Dashboard Anti-Patterns

| Anti-pattern | Why it's bad | What to do instead |
|-------------|-------------|-------------------|
| Tracking only pass/fail | Misses drift and anomalies | Track individual measurements |
| Weekly summary reports | Too slow to catch issues | Use live dashboards |
| Separate dashboards per station | Can't compare across stations | Use station comparison view |
| No spec limits on charts | Can't assess margin | Always overlay limits |
| Too many metrics on one screen | Information overload | Focus on FPY + top 5 measurements |


### What Is Adaptive Testing for Manufacturing
URL: https://www.tofupilot.com/guides/what-is-adaptive-testing-for-manufacturing
Adaptive testing uses real-time data to adjust test sequences, skip redundant checks, and reduce cycle time. Learn how it works and where it applies.

Adaptive testing adjusts the test sequence in real time based on data from earlier test steps, historical results, or process conditions. Instead of running every unit through every test, adaptive testing decides which tests to run, which to skip, and which limits to apply based on what the data says. This guide covers how adaptive testing works, the different levels of adaptation, and where it produces the most value.

## Static vs Adaptive Testing

| Aspect | Static Testing | Adaptive Testing |
|--------|---------------|-----------------|
| Test sequence | Same for every unit | Adjusted per unit based on data |
| Test time | Fixed | Variable (shorter for good units) |
| Limits | Same for every unit | Can be tightened or relaxed based on context |
| Decision logic | Pass/fail per measurement | Pass/fail plus risk score |
| Data dependency | None (runs blind) | Requires historical data and inline analysis |

In semiconductor testing, adaptive test has been deployed for over a decade. Advantest and Teradyne offer platforms that skip tests based on upstream wafer data, reducing test time by 10-50%. In discrete manufacturing, the concept is newer but the same principles apply.

## Levels of Adaptive Testing

| Level | What Adapts | Example | Complexity |
|-------|-----------|---------|-----------|
| 1. Fail-fast | Test order | Run the highest-failure-rate test first, stop early on failure | Low |
| 2. Skip on pass | Test coverage | If power-up passes, skip the detailed voltage rail test | Medium |
| 3. Data-driven skip | Test selection | ML model predicts this unit will pass based on upstream data, skip test | High |
| 4. Dynamic limits | Pass/fail criteria | Tighten limits for a batch from a new supplier, relax for a proven lot | High |
| 5. Autonomous closure | Test termination | AI determines sufficient testing has been performed for this unit | Very high |

Most manufacturing teams operate at level 1 (fail-fast ordering) without calling it adaptive testing. Levels 2-3 deliver the biggest cycle time reductions. Levels 4-5 are emerging capabilities.

## Where Adaptive Testing Produces Value

| Scenario | Static Test Time | Adaptive Test Time | Savings |
|----------|-----------------|-------------------|---------|
| High-yield product (98% FPY) | 60 seconds | 45 seconds (skip redundant checks on passing units) | 25% |
| Multi-variant product | 90 seconds (all tests) | 50-70 seconds (test only the variant-specific steps) | 22-44% |
| Mature product, stable process | 60 seconds | 30 seconds (skip tests with 0% historical failure rate) | 50% |
| New product, unstable process | 60 seconds | 60 seconds (don't skip anything, need the data) | 0% |

Adaptive testing delivers the most value when yield is high and the product is mature. When you're still learning, run everything and collect data.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Implement Fail-Fast Ordering

The simplest form of adaptive testing: put the phases that fail most often first. When a unit fails early, skip the remaining phases.

```python filename="adaptive_test.py"
import openhtf as htf
from openhtf.util import units
from openhtf import PhaseResult


@htf.measures(
    htf.Measurement("power_good").equals("PASS"),
)
def phase_power_check(test):
    """Highest failure rate test. Run first, stop on fail."""
    result = "PASS"
    test.measurements.power_good = result
    if result != "PASS":
        return PhaseResult.STOP


@htf.measures(
    htf.Measurement("firmware_version").equals("3.1.0"),
)
def phase_firmware(test):
    """Second highest failure rate. Run next."""
    test.measurements.firmware_version = "3.1.0"


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
)
def phase_output(test):
    """Rarely fails. Run last."""
    test.measurements.output_voltage_V = 5.01
```

## Step 2: Skip Redundant Tests

If a test phase has had 0% failure rate for the last 10,000 units, consider removing it from the sequence. Use TofuPilot's failure Pareto to identify these phases.

```python filename="adaptive_test.py"
@htf.measures(
    htf.Measurement("communication_check").equals("PASS"),
)
def phase_communication(test):
    """0.01% failure rate. Candidate for skip on mature products."""
    test.measurements.communication_check = "PASS"
```

The decision to skip should be data-driven. TofuPilot tracks failure rates per test step. Open the Analytics tab to see which phases have zero or near-zero failure rates across thousands of units.

## Step 3: Log Results for Continuous Learning

Adaptive testing requires continuous data collection. Even when you skip a test on most units, run the full sequence on a sample to validate that the skipped tests are still passing.

```python filename="adaptive_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_power_check,
    phase_firmware,
    phase_output,
    phase_communication,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Risks and Safeguards

| Risk | Safeguard |
|------|----------|
| Skipping a test that would have caught a defect | Run full sequence on 5-10% of units as audit sample |
| Process change invalidates skip decisions | Re-evaluate after any ECO, supplier change, or process change |
| Over-optimization on stable data | Set a minimum test coverage floor that can't be reduced |
| Regulatory requirements for 100% testing | Some industries (medical, aerospace) require every test on every unit |

Adaptive testing is not about testing less. It's about testing smarter. The goal is to maintain the same defect detection rate with less time and cost.

### Analyze Test Step Performance
URL: https://www.tofupilot.com/guides/how-to-analyze-test-step-performance-with-tofupilot
Learn how to break down test performance by step to find which phases fail most, run slowest, or have the lowest Cpk using TofuPilot's step analysis.

A test that fails tells you something went wrong. Step-level analysis tells you exactly where. TofuPilot breaks down every run by phase so you can pinpoint which steps fail most, which take longest, and which measurements are closest to their limits.

## Why Step-Level Analysis Matters

Overall pass/fail rates hide problems. A test with 95% FPY sounds fine until you discover one phase accounts for 80% of all failures. Without step-level data, you're guessing where to focus improvement efforts.

Step analysis answers three questions: Which phases have the lowest pass rate? Which measurements have the lowest Cpk? Which steps take the most time?

## Structuring Tests for Good Step Data

Each OpenHTF phase becomes a step in TofuPilot. Name your phases clearly and keep each one focused on a single functional area. This makes the analysis meaningful.

```python filename="multi_phase_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("input_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=11.5, maximum=12.5),
    htf.Measurement("input_current")
    .with_units(units.AMPERE)
    .in_range(maximum=2.0),
)
def power_input_check(test):
    """Verify power supply input within spec."""
    test.measurements.input_voltage = 12.03
    test.measurements.input_current = 1.45

@htf.measures(
    htf.Measurement("reg_3v3_output")
    .with_units(units.VOLT)
    .in_range(minimum=3.25, maximum=3.35),
    htf.Measurement("reg_5v_output")
    .with_units(units.VOLT)
    .in_range(minimum=4.90, maximum=5.10),
    htf.Measurement("reg_ripple")
    .in_range(maximum=50.0),
)
def voltage_regulator_test(test):
    """Test all onboard voltage regulators."""
    test.measurements.reg_3v3_output = 3.30
    test.measurements.reg_5v_output = 4.98
    test.measurements.reg_ripple = 22.4

@htf.measures(
    htf.Measurement("i2c_ack_received").with_allowed_values(True),
    htf.Measurement("spi_loopback_ok").with_allowed_values(True),
    htf.Measurement("uart_baud_error_pct")
    .in_range(maximum=2.0),
)
def communication_bus_test(test):
    """Verify I2C, SPI, and UART interfaces."""
    test.measurements.i2c_ack_received = True
    test.measurements.spi_loopback_ok = True
    test.measurements.uart_baud_error_pct = 0.8

@htf.measures(
    htf.Measurement("wifi_rssi")
    .in_range(minimum=-70, maximum=-20),
    htf.Measurement("bluetooth_pair_time")
    .in_range(maximum=5.0),
)
def wireless_connectivity_test(test):
    """Test WiFi and Bluetooth connections."""
    test.measurements.wifi_rssi = -45
    test.measurements.bluetooth_pair_time = 2.3

@htf.measures(
    htf.Measurement("flash_write_speed")
    .in_range(minimum=10.0),
    htf.Measurement("flash_read_speed")
    .in_range(minimum=20.0),
    htf.Measurement("eeprom_verify_ok").with_allowed_values(True),
)
def memory_test(test):
    """Verify flash and EEPROM read/write performance."""
    test.measurements.flash_write_speed = 14.2
    test.measurements.flash_read_speed = 28.7
    test.measurements.eeprom_verify_ok = True

def main():
    test = htf.Test(
        power_input_check,
        voltage_regulator_test,
        communication_bus_test,
        wireless_connectivity_test,
        memory_test,
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "BOARD-042")

if __name__ == "__main__":
    main()
```

This test has five distinct phases, each covering a different subsystem. TofuPilot tracks pass rates, measurement distributions, and timing for each one independently.

## Finding Your Weakest Steps

In TofuPilot's step analysis view, sort phases by pass rate to find the weakest links. A phase with 92% pass rate in a five-phase test dominates your overall yield loss.

Look at the measurements within that phase. One measurement with a Cpk below 1.0 means your process variation is too wide for the spec. Either tighten the process or widen the spec if the product can tolerate it.

## Identifying Slow Steps

Step timing data reveals bottlenecks. If your wireless connectivity test takes 8 seconds out of a 12-second total, that's where to focus cycle time reduction.

Common fixes for slow steps: reduce retry counts, parallelize independent measurements, or optimize instrument communication (batch queries instead of sequential reads).

## Using Step Data to Prioritize

Combine pass rate, Cpk, and timing into a prioritized list. A step that fails often and runs slowly is the highest-impact target. A step with high Cpk and fast execution needs no attention.

TofuPilot's measurement analytics shows all of this per phase. Review it weekly to track whether your process improvements are actually moving the numbers.

### Real-Time Production Test Monitoring
URL: https://www.tofupilot.com/guides/real-time-test-monitoring-with-tofupilot
Learn how to monitor hardware test operations in real time using TofuPilot's live dashboards, yield tracking, and instant failure alerts.

By the time you open last week's test report, the damage is done. Real-time monitoring means seeing test results as they happen, so you can react to problems in minutes instead of days.

## Why Real-Time Matters

A test station starts failing at 10 AM. Without real-time monitoring, nobody notices until the quality engineer pulls the weekly report on Friday. That's 4.5 days of potentially bad units moving downstream.

With TofuPilot's live dashboards, the yield drop shows up immediately. The engineer investigates at 10:15 AM, finds a worn pogo pin on Station 3, and fixes it by 11 AM. Three bad units instead of three hundred.

## Setting Up Live Monitoring

### Step 1: Connect Your Test Stations

Every test station pushes results to TofuPilot as runs complete. The data appears on dashboards within seconds.

```python filename="station_with_monitoring.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Results appear on the dashboard immediately after this call
client.create_run(
    procedure_id="FINAL-FUNCTIONAL",
    unit_under_test={"serial_number": "UNIT-9921"},
    run_passed=True,
    steps=[{
        "name": "Power On Self Test",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "boot_time_ms",
            "value": 342,
            "unit": "ms",
            "limit_high": 500,
        }],
    }],
)
```

### Step 2: Open the Live Dashboard

The procedure dashboard in TofuPilot shows:

- **Run feed**: Latest runs appear at the top as they come in. Green for pass, red for fail.
- **Rolling FPY**: Yield calculated over a sliding window (last 50 runs, last 24 hours, etc.)
- **Measurement trends**: Live-updating charts for each measurement parameter.

Put this on a monitor near the production floor. When the feed turns red, everyone sees it.

### Step 3: Monitor Multiple Stations

If you run multiple stations for the same procedure, the dashboard shows results from all of them. Color-code or filter by station to spot station-specific issues.

| View | What it tells you |
|------|------------------|
| All stations combined | Overall line performance |
| Single station filtered | Whether one station is underperforming |
| Station comparison | Side-by-side FPY and measurement distributions |

## What to Monitor

### Yield (FPY)

The first number to watch. Track it at three time scales:

- **Hourly**: Catches sudden problems (fixture failure, bad component reel)
- **Daily**: Shows shift-to-shift variation
- **Weekly**: Reveals longer-term trends (gradual fixture wear, seasonal effects)

### Failure Modes

When a run fails, TofuPilot records which step and measurement caused the failure. The failure pareto updates in real time, showing you the current top failure modes.

If a new failure mode suddenly appears at the top of the pareto, something changed. Investigate immediately.

### Measurement Values

Even for passing units, track the actual measured values. A measurement that's drifting toward its limit is a future failure waiting to happen.

TofuPilot's measurement trend chart shows each reading as a dot, with spec limits drawn as horizontal lines. When dots start clustering near a limit line, it's time to act.

### Cycle Time

Test cycle time tells you about station health. If a test that normally takes 45 seconds starts taking 90 seconds, something is wrong (instrument communication timeout, retry loop, slow fixture actuation).

## Responding to Real-Time Alerts

When the dashboard shows a problem:

1. **Check the failure mode.** What measurement is failing? Is it one measurement or multiple?
2. **Check the station.** Is the problem on one station or all stations? One station means a fixture or instrument issue. All stations means a product or process issue.
3. **Check the timing.** When did it start? What changed at that time (new component lot, shift change, fixture service)?
4. **Act.** If it's station-specific, take the station offline and inspect. If it's product-wide, hold the batch and investigate the component or process change.

## Real-Time vs. Batch Monitoring

| Aspect | Batch (weekly reports) | Real-time (TofuPilot) |
|--------|----------------------|----------------------|
| Detection time | Days to weeks | Minutes |
| Units at risk | Hundreds to thousands | Single digits |
| Data format | Static PDF/Excel | Interactive dashboard |
| Drill-down ability | Limited (aggregated data) | Full (individual runs) |
| Station comparison | Manual cross-reference | Automatic |

## Production Floor Setup

For maximum visibility, set up a dedicated monitoring display:

1. Mount a screen visible from the test area
2. Open TofuPilot's procedure dashboard in a browser
3. Set the time filter to "last 24 hours"
4. Auto-refresh keeps the data current

Operators and engineers can glance at the screen between test runs. When something goes wrong, the red indicators are unmistakable.


### Root Cause Analysis for Hardware Test Failures
URL: https://www.tofupilot.com/guides/root-cause-analysis-for-hardware-test-failures
Learn how to use TofuPilot's test data to trace hardware failures back to their root cause using measurement trends and run comparisons.

A test fails. The question is never just "what failed" but "why did it fail, and will it fail again?" TofuPilot stores every measurement from every run, so you can trace failures back to their source instead of guessing.

## The Root Cause Analysis Problem in Hardware

Software bugs leave stack traces. Hardware failures leave measurements. The difference: a stack trace points to one line of code, but an out-of-spec voltage reading could mean a bad solder joint, a drifting power supply, a faulty test fixture, or a component lot issue.

Root cause analysis in hardware means correlating test data across multiple dimensions: time, station, component lot, operator, and environmental conditions.

## Step 1: Identify the Failure Pattern

Start by filtering failed runs in TofuPilot's dashboard.

| Filter | Purpose |
|--------|---------|
| Procedure | Narrow to the specific test |
| Date range | Find when failures started |
| Station | Check if failures cluster on one station |
| Status: Failed | See only failed runs |

Look for clustering. If failures concentrate on one station, one shift, or one date range, you've already narrowed the search.

## Step 2: Compare Failing vs. Passing Runs

TofuPilot lets you compare runs side by side. Select a failing run and a passing run from the same procedure, then compare their measurements.

```
Passing run (UNIT-3021):          Failing run (UNIT-3045):
┌─────────────────────────┐      ┌─────────────────────────┐
│ vcc_3v3:  3.30 V  ✓     │      │ vcc_3v3:  3.28 V  ✓     │
│ vcc_1v8:  1.81 V  ✓     │      │ vcc_1v8:  1.76 V  ✗     │
│ clk_freq: 24.00 MHz ✓   │      │ clk_freq: 23.95 MHz ✓   │
│ current:  45 mA  ✓      │      │ current:  62 mA  ✗      │
└─────────────────────────┘      └─────────────────────────┘
```

In this example, the 1.8V rail is low and current draw is high. These two symptoms together point to a short or partial short on the 1.8V rail.

## Step 3: Check the Measurement Trend

A single comparison shows you the "what." The trend shows you the "when." Open the measurement timeline for `vcc_1v8` across all runs.

If the 1.8V reading was stable at 1.81V for weeks and then started dropping on March 3rd, something changed on March 3rd. Check:

- Was a new component lot introduced?
- Was the test fixture serviced?
- Did the station's power supply get recalibrated?

## Step 4: Correlate with Component Lots

TofuPilot tracks unit metadata including component information. If failures cluster around units built with a specific component lot, you've found a supplier quality issue.

```python filename="lot_analysis.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get all failed runs for a specific procedure
runs = client.get_runs(
    procedure_id="BOARD-FUNCTIONAL",
    run_passed=False,
    limit=100,
)

# Check if failures correlate with component lots
lot_counts = {}
for run in runs:
    lot = run.get("unit_under_test", {}).get("batch", "unknown")
    lot_counts[lot] = lot_counts.get(lot, 0) + 1

for lot, count in sorted(lot_counts.items(), key=lambda x: -x[1]):
    print(f"Lot {lot}: {count} failures")
```

## Step 5: Isolate the Variable

Root cause analysis is a process of elimination. TofuPilot helps you hold variables constant while changing one at a time:

| Variable | How to control it |
|----------|-------------------|
| Station | Filter by station ID |
| Operator | Filter by shift/time |
| Component lot | Filter by batch/lot |
| Fixture | Check station metadata |
| Environment | Compare with temperature logs |

When you can reproduce the failure by controlling one variable (e.g., "all failures are from Station 3"), you've isolated the root cause.

## Common Root Cause Patterns

### Station-Specific Failures
Failures cluster on one test station. Usually caused by:
- Worn pogo pins or test probes
- Loose cable connections
- Calibration drift on station instruments

### Lot-Specific Failures
Failures cluster around units from a specific component lot. Usually caused by:
- Supplier quality escape
- Component parameter shift
- Wrong component revision

### Time-Correlated Failures
Failures start at a specific date and continue. Usually caused by:
- Process change (solder profile, firmware version)
- Environmental change (humidity, temperature)
- Fixture wear reaching a threshold

### Intermittent Failures
Failures appear randomly across stations and lots. Usually caused by:
- Marginal design (values close to limits)
- Test measurement noise
- Environmental sensitivity

## From Root Cause to Corrective Action

Once you've identified the root cause, TofuPilot's data helps you verify the fix. Run the same tests after the corrective action and compare the measurement distributions before and after. If the 1.8V rail readings shift back to 1.81V and current draw returns to normal, your fix worked.

Track the corrective action's effectiveness over time. TofuPilot's trend views show whether the fix holds or if the problem returns.


### Manufacturing Test Frameworks Compared
URL: https://www.tofupilot.com/guides/manufacturing-test-frameworks-compared-openhtf-vs-pytest-vs-opentap-vs-teststand
Compare TofuPilot Framework, OpenHTF, pytest, OpenTAP and NI TestStand for manufacturing test, with real code, feature matrix and cost analysis.

Five frameworks dominate manufacturing test automation: the TofuPilot Framework, OpenHTF, pytest, OpenTAP, and NI TestStand. Each makes different tradeoffs between structure, flexibility, cost, and ecosystem. This guide compares them with real code, concrete metrics, and decision criteria so you can pick the right one.

## Framework Overview

| Framework | Language | License | Origin | Focus |
|-----------|---------|---------|--------|-------|
| **TofuPilot Framework** | Python (Rust engine) | MIT (free) | TofuPilot | Manufacturing test with parallel execution |
| **OpenHTF** | Python | Apache 2.0 (free) | Google | Manufacturing/production test |
| **pytest** | Python | MIT (free) | Community | Software testing, adapted for hardware |
| **OpenTAP** | C# / Python | MPL 2.0 (free) | Keysight | Instrument-heavy test automation |
| **NI TestStand** | LabVIEW / C / Python | Commercial ($4,310/seat) | NI (Emerson) | Enterprise manufacturing test |

## Feature Comparison Matrix

| Feature | TofuPilot Framework | OpenHTF | pytest | OpenTAP | TestStand |
|---------|--------------------|---------|--------|---------|-----------|
| **Structured measurements** | Built-in (YAML validators) | Built-in (name, value, limits, units) | Manual (assert only) | Plugin-based | Built-in |
| **Serial number input** | Built-in | Built-in prompt | Manual | Plugin | Built-in |
| **Operator UI** | Declarative, from the procedure | Built-in web UI | None | Built-in (GUI editor) | Built-in (Sequence Editor) |
| **Test sequencing** | YAML procedure | Phase ordering | Function ordering (plugins) | Step ordering (GUI) | Sequence files (GUI) |
| **Parallel phases** | Native (Rust engine) | No | Via xdist | Native | Native |
| **Multi-slot fixtures** | Native | Limited | Manual | Native | Native |
| **Waveform aggregations** | Built-in (mean, min, max, std) | No | Manual | Plugin | Manual |
| **Per-axis validators** | Built-in | No | Manual | No | No |
| **Instrument drivers** | Plugs (plain Python classes) | Plugs (Python) | Fixtures (Python) | Plugins (C#/Python) | NI drivers (LabVIEW) |
| **Deployment to stations** | Git push, instant rollback | Your problem | Your problem | Manual | Manual |
| **Offline handling** | Queues and syncs | Queues and syncs (via CLI) | Your problem | Manual | Database-dependent |
| **Version control** | Git (YAML + Python) | Git (Python files) | Git (Python files) | Git (XML + code) | Difficult (binary .seq files) |
| **CI/CD integration** | Native (Python) | Native (Python) | Native (Python) | Possible | Difficult |
| **Cross-platform** | Linux, macOS, Windows | Linux, macOS, Windows | Linux, macOS, Windows | Linux, Windows | Windows only |
| **Community** | Newer, smaller | Small (~640 stars) | Massive (11K+ stars) | Small (~200 stars) | Large (NI forums) |
| **Learning curve** | Low (YAML + plain Python) | Medium | Low | Medium | High |
| **Annual cost (5 seats)** | $0 | $0 | $0 | $0 | ~$21,550 |

## The Same Test in Each Framework

A simple functional test: measure a 3.3V rail voltage, check it's within 3.2V to 3.4V.

### TofuPilot Framework

The procedure YAML declares the sequence and the limits. Phases are plain Python functions with no decorators, and plugs are plain Python classes.

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dut
    python: plugs.dut:Dut

main:
  - name: Power Rail Test
    python: phases.power_rail
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/power_rail.py"
def power_rail(measurements, dut):
    measurements.rail_3v3 = dut.read_voltage()
```

```python filename="plugs/dut.py"
class Dut:
    def read_voltage(self) -> float:
        return 3.31  # Replace with an instrument read
```

Run it from the procedure directory with `tofupilot run`.

**Strengths:** Limits live in the YAML as data, so the test plan is readable and diffable separately from the logic. Phases and fixture slots run in parallel on a Rust engine while your code stays plain Python. Operator UI is generated from the procedure with no frontend work. Procedures deploy to stations on a Git push, with instant rollback, and runs queue offline when the network drops.

**Weakness:** Newer than OpenHTF, so a smaller community and fewer public examples.

### OpenHTF

```python filename="comparison/openhtf_example.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


class DutPlug(htf.plugs.BasePlug):
    def setUp(self):
        self.voltage = 3.31  # Replace with instrument read

    def tearDown(self):
        pass


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
@htf.plug(dut=DutPlug)
def test_power(test, dut):
    test.measurements.rail_3v3 = dut.voltage


def main():
    test = htf.Test(test_power, procedure_id="FCT-001", part_number="PCBA-100")
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Serial: "))


if __name__ == "__main__":
    main()
```

Install the OpenHTF extra with `pip install "tofupilot[openhtf]"`. `upload()` reads `TOFUPILOT_API_KEY` from the environment. Alternatively, run the same suite with `tofupilot run` and the CLI handles streaming and upload with no callback at all.

**Strengths:** Measurements are structured data (name, value, limits, units). Years of production use at scale. Serial number prompting is built in. Runs natively on the TofuPilot CLI.

**Weakness:** Phases execute sequentially against one unit. Parallel phases and multi-slot fixtures are orchestrated outside the framework.

### pytest

```python filename="comparison/pytest_example.py"
import pytest


@pytest.fixture
def dut():
    connection = {"voltage": 3.31}  # Replace with real connection
    yield connection


def test_power_rail(dut):
    voltage = dut["voltage"]
    assert 3.2 <= voltage <= 3.4, f"3.3V rail: {voltage}V"
```

**Strengths:** Familiar to every Python developer. Huge plugin ecosystem. Flexible fixture system. Great for R&D and validation.

**Weakness:** Measurements are implicit (assert statements). The measured value lives inside an assert and then disappears, so there is no structured data for analytics without extra code.

### OpenTAP

```csharp filename="comparison/opentap_example.cs"
// OpenTAP C# test step
using OpenTap;

[Display("Power Rail Test", Group: "FCT")]
public class PowerRailStep : TestStep
{
    [Display("Lower Limit")]
    public double LowerLimit { get; set; } = 3.2;

    [Display("Upper Limit")]
    public double UpperLimit { get; set; } = 3.4;

    public override void Run()
    {
        double voltage = 3.31; // Replace with instrument read
        Results.Publish("rail_3v3", new { Voltage = voltage });

        if (voltage < LowerLimit || voltage > UpperLimit)
            UpgradeVerdict(Verdict.Fail);
    }
}
```

**Strengths:** GUI step editor for non-programmers. Strong Keysight instrument integration. Plugin architecture for test plans.

**Weakness:** C# primary language (Python plugin exists but is secondary). Smaller community.

### NI TestStand

TestStand uses a visual sequence editor. The equivalent test is a sequence file (.seq) with a "Numeric Limit Test" step configured via GUI: test value = voltage reading, low limit = 3.2, high limit = 3.4.

**Strengths:** Mature, enterprise-grade. Deep NI hardware integration. Built-in report generation, database logging, parallel execution.

**Weakness:** $4,310/seat/year. Windows only. Binary sequence files don't version-control well. Tied to NI ecosystem.

## Cost Analysis

| | TofuPilot Framework | OpenHTF | pytest | OpenTAP | TestStand |
|--|--------------------|---------|--------|---------|-----------|
| License (5 seats) | $0 | $0 | $0 | $0 | $21,550/year |
| License (20 seats) | $0 | $0 | $0 | $0 | $86,200/year |
| Runtime deployment | Free | Free | Free | Free | Additional runtime licenses |
| Training | Self-taught (docs) | Self-taught (docs) | Self-taught (docs) | Self-taught (docs) | NI training courses ($2K+) |
| Vendor lock-in | None (MIT, open source) | None | None | Low (Keysight-adjacent) | High (NI ecosystem) |
| Support | Community + TofuPilot | Community (GitHub) | Community (massive) | Community + Keysight | NI support contract |

## When to Use Each Framework

| Scenario | Best Choice | Why |
|----------|------------|-----|
| Production test, throughput-bound line | **TofuPilot Framework** | Parallel phases and multi-slot fixtures cut cycle time |
| Production test, operators need guided workflows | **TofuPilot Framework** | Declarative operator UI, no frontend work |
| Validating waveforms or sweeps | **TofuPilot Framework** | Aggregations and per-axis validators check curve shape |
| New stations, many sites | **TofuPilot Framework** | Deploy on Git push, instant rollback, offline queueing |
| Existing OpenHTF suites in production | **OpenHTF** | Runs natively on the TofuPilot CLI; migration cost isn't worth it |
| Single pure-Python paradigm, no YAML layer | **OpenHTF** | Everything in Python decorators |
| R&D validation, firmware CI | **pytest** | Flexible, fast iteration, CI/CD native, huge ecosystem |
| Multi-vendor instruments, enterprise | **OpenTAP** | Plugin architecture, GUI editor, Keysight integration |
| Existing NI hardware, large enterprise | **TestStand** | Deep NI integration, enterprise support, existing infrastructure |
| Mixed (R&D + production) | **pytest** + **TofuPilot Framework** | pytest for validation, Framework for production |

## Migration Paths

### From TestStand to the TofuPilot Framework

| TestStand Concept | TofuPilot Framework Equivalent |
|-------------------|-------------------------------|
| Sequence file (.seq) | `procedure.yaml` |
| Step types | Phase functions in `phases/` |
| Numeric Limit Test | `measurements` with `validators` in YAML |
| String Value Test | `validators` with `operator: "=="` |
| Step module (code) | Plug class in `plugs/` |
| Process model | Built into the CLI |
| Report generation | TofuPilot dashboard |
| UUT serial number | `unit.serial_number` in the procedure |
| Parallel sequence execution | `execution.slots` |

### From OpenHTF to the TofuPilot Framework

| OpenHTF Concept | TofuPilot Framework Equivalent |
|-----------------|-------------------------------|
| Phase function with `@htf.measures` | Plain function, measurements declared in YAML |
| `.in_range(3.2, 3.4)` | `validators` with `>=` and `<=` |
| `BasePlug` subclass | Plain Python class |
| `@htf.plug(dut=DutPlug)` | Plug injected by parameter name |
| `test.measurements.x = v` | `measurements.x = v` |
| `htf.Test(phase1, phase2)` | `main:` list in the procedure |

### From pytest to the TofuPilot Framework

| pytest Concept | TofuPilot Framework Equivalent |
|---------------|-------------------------------|
| Test function | Phase function |
| Fixture | Plug |
| Assert statement | `validators` in the procedure YAML |
| conftest.py | `plugs/` directory |
| JUnit XML | TofuPilot dashboard |

## TofuPilot Integration

All five frameworks work with TofuPilot:

| Framework | Integration Method | Effort |
|-----------|-------------------|--------|
| TofuPilot Framework | Native (`tofupilot run --upload`) | None |
| OpenHTF | Native on the CLI, or `upload()` callback | None, or 2 lines |
| pytest | Python SDK (`TofuPilotClient`) | ~10 lines per test |
| OpenTAP | REST API or Python SDK | Medium |
| TestStand | REST API | Medium |

The TofuPilot Framework and OpenHTF both run natively on the CLI, which streams phases, measurements, logs, attachments, and the operator UI to the dashboard with no extra code on either side. For a self-managed pipeline that skips the CLI, OpenHTF also has the `upload()` output callback shown above.

## Decision Flowchart

1. **Do you have existing TestStand infrastructure?** Yes, and it works well: stay with TestStand. Yes, but you want to migrate: move to the TofuPilot Framework.
2. **Do you already run OpenHTF in production?** Yes: keep it. It runs natively on the TofuPilot CLI, and the migration cost isn't worth it unless you need parallel slots.
3. **Are you building production tests on new stations?** Yes: the TofuPilot Framework. Parallel execution, declarative operator UI, and deploy-on-push are built in.
4. **Are you doing R&D or firmware validation?** Yes: pytest. More flexible, better CI/CD.
5. **Do you need a GUI for non-programmers?** Yes: OpenTAP. Visual step editor.
6. **Unsure?** Start with the TofuPilot Framework. `tofupilot run ./procedure.yaml` executes a procedure on your machine without an account.

For a deeper comparison of the two purpose-built Python options, see [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf).

### Offline Test Data Sync
URL: https://www.tofupilot.com/guides/offline-test-data-sync-with-tofupilot
Learn how to collect hardware test data in offline or air-gapped environments and sync it to TofuPilot when connectivity is restored.

Not every test station has an internet connection. Cleanrooms, field deployments, secure facilities, and factory floors with spotty WiFi all need to capture test data reliably. TofuPilot's Python client handles offline scenarios by letting you store results locally and sync when connectivity returns.

## When Offline Testing Happens

| Scenario | Why it's offline |
|----------|-----------------|
| Air-gapped facility | Security requirements prohibit internet access |
| Field testing | Remote location with no connectivity |
| Cleanroom | No network access inside the controlled environment |
| Factory floor | Unreliable WiFi, can't depend on it for every test |
| Mobile test station | Traveling between sites |

In all these cases, the test must run and record data regardless of network status. You can't tell a production line to stop because the WiFi is down.

## Architecture for Offline-First Testing

```
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Test Station │────▶│  Local Store  │────▶│   TofuPilot  │
│  (runs tests) │     │  (JSON files) │     │   (cloud)    │
└──────────────┘     └──────────────┘     └──────────────┘
                      Offline              When connected
```

The test station writes results to local storage. A sync process uploads them to TofuPilot when a connection is available.

## Step 1: Store Results Locally

Write test results to JSON files on the local machine. Each file represents one test run.

```python filename="offline_test.py"
import json
import os
from datetime import datetime

OFFLINE_DIR = "/data/test-results/pending"
os.makedirs(OFFLINE_DIR, exist_ok=True)

def run_test(serial_number):
    """Run the test and store results locally."""
    # Run your actual test measurements
    vcc = measure_voltage(channel=1)
    current = measure_current()

    result = {
        "procedure_id": "BOARD-FUNCTIONAL-V3",
        "unit_under_test": {"serial_number": serial_number},
        "run_passed": 3.25 <= vcc <= 3.35 and 30 <= current <= 60,
        "tested_at": datetime.utcnow().isoformat(),
        "steps": [{
            "name": "Power Rail Check",
            "step_type": "measurement",
            "status": 3.25 <= vcc <= 3.35,
            "measurements": [{
                "name": "vcc_3v3",
                "value": vcc,
                "unit": "V",
                "limit_low": 3.25,
                "limit_high": 3.35,
            }],
        }, {
            "name": "Current Draw",
            "step_type": "measurement",
            "status": 30 <= current <= 60,
            "measurements": [{
                "name": "idle_current_ma",
                "value": current,
                "unit": "mA",
                "limit_low": 30,
                "limit_high": 60,
            }],
        }],
    }

    filename = f"{serial_number}_{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}.json"
    filepath = os.path.join(OFFLINE_DIR, filename)
    with open(filepath, "w") as f:
        json.dump(result, f, indent=2)

    print(f"Result saved locally: {filepath}")
    return result

# Run tests regardless of network status
run_test("UNIT-5501")
run_test("UNIT-5502")
```

## Step 2: Sync When Connected

A separate sync script uploads pending results to TofuPilot and moves them to a "synced" directory.

```python filename="sync_results.py"
import json
import os
import shutil
from tofupilot import TofuPilotClient

PENDING_DIR = "/data/test-results/pending"
SYNCED_DIR = "/data/test-results/synced"
os.makedirs(SYNCED_DIR, exist_ok=True)

def sync_pending_results():
    """Upload all pending results to TofuPilot."""
    client = TofuPilotClient()
    pending_files = sorted(os.listdir(PENDING_DIR))

    if not pending_files:
        print("No pending results to sync.")
        return

    print(f"Syncing {len(pending_files)} pending results...")

    for filename in pending_files:
        filepath = os.path.join(PENDING_DIR, filename)
        with open(filepath) as f:
            result = json.load(f)

        try:
            client.create_run(
                procedure_id=result["procedure_id"],
                unit_under_test=result["unit_under_test"],
                run_passed=result["run_passed"],
                steps=result["steps"],
            )
            # Move to synced directory
            shutil.move(filepath, os.path.join(SYNCED_DIR, filename))
            print(f"Synced: {filename}")
        except Exception as e:
            print(f"Failed to sync {filename}: {e}")
            # Leave in pending for retry

    remaining = len(os.listdir(PENDING_DIR))
    print(f"Sync complete. {remaining} files still pending.")

sync_pending_results()
```

## Step 3: Automate the Sync

Run the sync script automatically when connectivity is available.

### Option A: Cron Job

```bash filename="crontab"
# Try to sync every 15 minutes
*/15 * * * * /usr/bin/python3 /opt/test-station/sync_results.py >> /var/log/test-sync.log 2>&1
```

### Option B: Network Event Trigger

```python filename="network_watcher.py"
import subprocess
import time

def has_internet():
    try:
        subprocess.check_call(
            ["curl", "-s", "--max-time", "5", "https://app.tofupilot.com/api/health"],
            stdout=subprocess.DEVNULL,
        )
        return True
    except subprocess.CalledProcessError:
        return False

while True:
    if has_internet():
        subprocess.run(["python3", "sync_results.py"])
    time.sleep(300)  # Check every 5 minutes
```

## Data Integrity

Offline sync introduces a risk: what if a file gets corrupted or a sync partially fails?

| Risk | Mitigation |
|------|-----------|
| File corruption | Write to temp file first, then rename (atomic write) |
| Duplicate upload | TofuPilot deduplicates by procedure + serial + timestamp |
| Partial sync failure | Files stay in pending until successfully uploaded |
| Clock drift on offline machines | Use monotonic timestamps for ordering, UTC for absolute time |

## When to Use This Pattern

Use offline-first testing when:

- Your test station can't guarantee network connectivity
- Network latency would slow down test cycle time
- You need tests to run even during network outages
- Security requirements prevent direct cloud access from the test floor

For stations with reliable connectivity, push results directly to TofuPilot from within the test script. The offline pattern adds complexity that's only justified when connectivity is unreliable.


### Test Sequencing and Orchestration
URL: https://www.tofupilot.com/guides/test-sequencing-and-orchestration-with-tofupilot
Learn how to build repeatable hardware test sequences using TofuPilot with OpenHTF and Python for structured, automated test orchestration.

A hardware test isn't one check. It's a sequence: power up, wait for boot, measure voltages, run calibration, check communication, stress test, power down. Capturing each step with its measurements and timing gives you a structured record of the full test flow.

## Why Test Sequencing Matters

Running tests manually from a bench instrument works for prototypes. It doesn't work for production. Production testing needs:

- **Repeatability**: Every unit goes through the exact same steps in the exact same order
- **Speed**: No waiting for an operator to click "next"
- **Data capture**: Every measurement recorded automatically with limits and pass/fail status
- **Traceability**: A record of exactly what was tested, in what order, with what results

Test orchestration means defining that sequence once and running it the same way every time.

## Test Sequence Architecture

A well-structured test sequence follows a pattern:

```
┌─────────────┐
│  Setup       │  Power on, initialize instruments, identify DUT
├─────────────┤
│  Step 1      │  Measurement or action with pass/fail criteria
├─────────────┤
│  Step 2      │  Next measurement or action
├─────────────┤
│  ...         │  Additional steps as needed
├─────────────┤
│  Teardown    │  Power off, release instruments, upload results
└─────────────┘
```

Each step produces measurements. Each measurement has limits. The sequence stops on critical failures or continues through all steps depending on your strategy.

## Building a Test Sequence with OpenHTF

OpenHTF is a Python framework designed for hardware test sequencing. TofuPilot integrates as an output callback.

Plugs subclass `BasePlug` with `setUp` and `tearDown` (note the capitalisation, OpenHTF will not call `setup`/`teardown`), and are injected into a phase with `@htf.plug(name=PlugClass)`.

```python filename="production_test_sequence.py"
import time

import openhtf as htf
from openhtf.plugs import BasePlug
from tofupilot.openhtf import upload


class PowerSupplyPlug(BasePlug):
    """Controls the bench power supply."""

    def setUp(self):
        self.psu = connect_power_supply()

    def set_voltage(self, voltage):
        self.psu.write(f"VOLT {voltage}")

    def enable_output(self):
        self.psu.write("OUTP ON")

    def disable_output(self):
        self.psu.write("OUTP OFF")

    def tearDown(self):
        self.disable_output()


class DMMPlug(BasePlug):
    """Reads from the digital multimeter."""

    def setUp(self):
        self.dmm = connect_dmm()

    def measure_voltage(self):
        return float(self.dmm.query("MEAS:VOLT:DC?"))

    def measure_current(self):
        return float(self.dmm.query("MEAS:CURR:DC?"))


# Step 1: Power rail verification
@htf.PhaseOptions(name="Power Rail Verification")
@htf.plug(psu=PowerSupplyPlug, dmm=DMMPlug)
@htf.measures(
    htf.Measurement("vcc_3v3").in_range(3.25, 3.35).with_units("V"),
    htf.Measurement("vcc_1v8").in_range(1.75, 1.85).with_units("V"),
    htf.Measurement("vcc_5v0").in_range(4.90, 5.10).with_units("V"),
)
def power_rail_check(test, psu, dmm):
    psu.set_voltage(12.0)
    psu.enable_output()
    time.sleep(0.5)  # Wait for rails to stabilize

    test.measurements.vcc_3v3 = dmm.measure_voltage()
    # Switch DMM channel and measure other rails
    test.measurements.vcc_1v8 = dmm.measure_voltage()
    test.measurements.vcc_5v0 = dmm.measure_voltage()


# Step 2: Current consumption
@htf.PhaseOptions(name="Current Consumption")
@htf.plug(psu=PowerSupplyPlug, dmm=DMMPlug)
@htf.measures(
    htf.Measurement("idle_current_ma").in_range(30, 60).with_units("mA"),
    htf.Measurement("active_current_ma").in_range(80, 150).with_units("mA"),
)
def current_check(test, psu, dmm):
    test.measurements.idle_current_ma = dmm.measure_current() * 1000
    trigger_active_mode()
    time.sleep(0.2)
    test.measurements.active_current_ma = dmm.measure_current() * 1000


# Step 3: Communication check
@htf.PhaseOptions(name="Communication Interfaces")
@htf.measures(
    htf.Measurement("uart_loopback").equals(True),
    htf.Measurement("spi_whoami").equals(0x68),
)
def comm_check(test):
    test.measurements.uart_loopback = verify_uart_loopback()
    test.measurements.spi_whoami = read_spi_register(0x75)


def main():
    test = htf.Test(
        power_rail_check,
        current_check,
        comm_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: input("Scan DUT serial: "))


if __name__ == "__main__":
    main()
```

Install it with `pip install "tofupilot[openhtf]"`. Every phase runs in order, every measurement is captured, and the full sequence uploads when the test completes.

One thing the example glosses over: the three rails are read from the same DMM call without switching channels. On a real fixture you would either switch the mux between reads or use separate channels, otherwise all three measurements record the same rail.

## Building a Test Sequence with the Python SDK

If you're not using OpenHTF, create the run directly. Each step becomes a phase with its own measurements and outcome.

```python filename="sequence_with_client.py"
import os
import time
from datetime import datetime, timezone

from tofupilot.v2 import TofuPilot

serial = input("Scan DUT serial: ")
phases = []

# Step 1: Power rails
started = datetime.now(timezone.utc)
psu.enable(12.0)
time.sleep(0.5)
vcc_3v3 = dmm.measure_voltage(channel=1)
vcc_1v8 = dmm.measure_voltage(channel=2)
rails_ok = 3.25 <= vcc_3v3 <= 3.35 and 1.75 <= vcc_1v8 <= 1.85

phases.append({
    "name": "Power Rail Verification",
    "outcome": "PASS" if rails_ok else "FAIL",
    "started_at": started,
    "ended_at": datetime.now(timezone.utc),
    "measurements": [
        {
            "name": "VCC 3V3",
            "measured_value": vcc_3v3,
            "units": "V",
            "validators": [
                {"operator": ">=", "expected_value": 3.25},
                {"operator": "<=", "expected_value": 3.35},
            ],
        },
        {
            "name": "VCC 1V8",
            "measured_value": vcc_1v8,
            "units": "V",
            "validators": [
                {"operator": ">=", "expected_value": 1.75},
                {"operator": "<=", "expected_value": 1.85},
            ],
        },
    ],
})

# Step 2: Functional check
started = datetime.now(timezone.utc)
boot_ok = wait_for_boot(timeout=5)
phases.append({
    "name": "Boot Sequence",
    "outcome": "PASS" if boot_ok else "FAIL",
    "started_at": started,
    "ended_at": datetime.now(timezone.utc),
    "measurements": [
        {"name": "Boot Success", "measured_value": boot_ok},
    ],
})

# Upload the complete sequence
run_outcome = "PASS" if all(p["outcome"] == "PASS" for p in phases) else "FAIL"

with TofuPilot(api_key=os.getenv("TOFUPILOT_API_KEY")) as client:
    client.runs.create(
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        serial_number=serial,
        part_number="PCBA-100",
        outcome=run_outcome,
        phases=phases,
    )
```

Note that the limits travel with the measurement as validators rather than living only in the `if` statement above. That is what keeps the result interpretable later, when the limits have changed and you are looking at old data.

## Sequence Design Best Practices

| Practice | Why |
|----------|-----|
| Test cheap things first | If a power rail is shorted, don't waste time running communication checks |
| Group related measurements | Keep all voltage checks in one step, all current checks in another |
| Use consistent step names | "Power Rail Verification" across all procedures, not "Voltage Check" in one and "Rail Test" in another |
| Include setup/teardown | Always power down the DUT at the end, even if the test fails |
| Set limits on every measurement | A measurement without limits can't be trended or analyzed |

The teardown point deserves emphasis. A phase that raises leaves the supply enabled unless the plug's `tearDown` cuts it, which is exactly why the power-down belongs in the plug rather than at the end of a phase body.

## Handling Sequence Failures

Two strategies for what happens when a step fails:

**Fail-fast**: Stop the sequence immediately on the first failure. Use this for safety-critical tests or when a failure in step 1 makes later steps meaningless (a shorted rail means there is no point testing communication).

**Run-all**: Continue through all steps even if one fails. Use this when you want complete diagnostic data, since knowing which 3 of 20 measurements failed helps root cause analysis.

In OpenHTF a phase returns `htf.PhaseResult.STOP` to halt the sequence, or `CONTINUE` to carry on. Most production sequences are run-all with a fail-fast gate on the first few safety-relevant phases.

## What Gets Stored for Each Sequence

Every test run captures:

| Data | Purpose |
|------|---------|
| Procedure ID | Which test sequence was run |
| Serial number | Which unit was tested |
| Overall outcome | Did the sequence pass? |
| Phases with measurements | Every step, every measurement, every limit |
| Timestamps | When the test started and ended |
| Station ID | Which station ran the test |
| Duration | How long the sequence took |

This structured data is what enables trending, comparison, and analytics across your entire production history.

### Manage Subcontractor Test Quality
URL: https://www.tofupilot.com/guides/how-to-manage-subcontractor-test-quality-with-tofupilot
Give subcontractors your test procedures, collect their results in your TofuPilot workspace, and maintain full visibility over outsourced production quality.

You design the product. Your contract manufacturer builds it. But when quality problems slip through, your brand takes the hit. TofuPilot lets you share test procedures with subcontractors and collect every result in your workspace, so you have the same visibility as if you were on the factory floor.

## The OEM/CM Visibility Problem

Most OEMs get a weekly yield report from their CM, maybe a spreadsheet, maybe an email with a number. That's not enough. You need measurement-level data for every unit: what was tested, what passed, what failed, and what the actual values were.

Without this data, you can't tell whether a 96% yield means the process is healthy or whether it's hiding a measurement that's drifting toward the limit. By the time the weekly report shows a yield drop, hundreds of bad units may have shipped.

## Set Up the Workflow

The setup is straightforward. You write the test procedures, the subcontractor runs them, and results upload to your TofuPilot workspace.

1. **You** write and version-control the OpenHTF test scripts
2. **You** provide the subcontractor with the scripts and a TofuPilot API key scoped to your workspace
3. **The subcontractor** installs the scripts on their test stations and runs production
4. **TofuPilot** collects every run with station metadata identifying the subcontractor's site

The subcontractor doesn't need access to your TofuPilot dashboard. They just run the tests. You see everything.

## Write Tests the Subcontractor Will Run

Keep test scripts self-contained. The subcontractor's operators shouldn't need to modify anything. Use `station_id` to identify the CM's stations in your data.

```python filename="test_motor_driver.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("vbus_voltage")
    .in_range(minimum=23.5, maximum=24.5)
    .with_units(units.VOLT),
    htf.Measurement("phase_current")
    .in_range(minimum=0.9, maximum=1.1)
    .with_units(units.AMPERE),
    htf.Measurement("pwm_frequency")
    .in_range(minimum=19500, maximum=20500)
    .with_units(units.HERTZ),
)
def test_motor_output(test):
    test.measurements.vbus_voltage = 24.01
    test.measurements.phase_current = 1.02
    test.measurements.pwm_frequency = 20010

def main():
    test = htf.Test(
        test_motor_output,
        station_id="CM-ACME-SZ-FCT-01",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "MDR-2026-04417")

if __name__ == "__main__":
    main()
```

Prefix the station ID with the subcontractor's name or code (like `CM-ACME-SZ`) so you can filter by CM in TofuPilot.

## Provide Deployment Instructions

Package everything the subcontractor needs:

| Item | Purpose |
|------|---------|
| Test scripts (Git repo or zip) | The exact procedures to run |
| `requirements.txt` | Python dependencies including `openhtf` and `tofupilot` |
| Environment setup guide | How to set the `TOFUPILOT_API_KEY` environment variable |
| Station naming convention | How to set `station_id` for each machine |
| Instrument configuration | VISA addresses, serial ports, fixture pin maps |

The subcontractor sets the API key as an environment variable. The TofuPilot client picks it up automatically.

```bash filename="setup_env.sh"
# The subcontractor sets this once per station
export TOFUPILOT_API_KEY=tp_live_xxxxxxxxxxxx

# The TofuPilot client reads it automatically
# No code changes needed
```

## Handle Multiple Subcontractors

If you work with more than one CM, use distinct station ID prefixes for each:

```python filename="test_sensor_board.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("temp_accuracy")
    .in_range(minimum=-0.5, maximum=0.5)
    .with_units(units.DEGREE_CELSIUS),
    htf.Measurement("humidity_accuracy_pct")
    .in_range(minimum=-3.0, maximum=3.0),
)
def test_sensor_accuracy(test):
    test.measurements.temp_accuracy = 0.12
    test.measurements.humidity_accuracy_pct = -1.3

def main():
    test = htf.Test(
        test_sensor_accuracy,
        # CM-BETA identifies this subcontractor's facility
        station_id="CM-BETA-GDL-EOL-03",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SNS-2026-09102")

if __name__ == "__main__":
    main()
```

This gives you clean filtering: see all of CM-BETA's results, compare CM-ACME vs CM-BETA yield, or drill into a specific station.

## Monitor Subcontractor Quality in TofuPilot

With all CMs uploading to your workspace, TofuPilot gives you:

- **Yield by subcontractor** so you can compare CM performance at a glance
- **Measurement distributions per CM** to catch process differences. If CM-ACME's voltage readings are shifted compared to CM-BETA, one of them has a calibration issue.
- **Failure Pareto per site** to see whether different CMs struggle with different tests
- **Real-time run feed** so you see results as they happen, not in a weekly report

## Keep Procedures in Sync

When you update a test procedure, every CM needs the new version. Use Git tags or release versions so you can verify which version each site is running. TofuPilot tracks the procedure used for each run, so you can filter by procedure version and confirm the rollout.

Don't let CMs modify test limits or skip tests. The procedures you ship are the single source of truth. If a CM reports that a test is failing too often, investigate the root cause instead of loosening limits.

### What Is a Test Copilot
URL: https://www.tofupilot.com/guides/what-is-a-test-copilot
A test copilot is an AI assistant that helps engineers write tests, analyze failures, and optimize limits. Learn what it does and where the technology is.

A test copilot is an AI assistant purpose-built for test engineering. Like GitHub Copilot for software development, a test copilot helps engineers write test scripts, analyze failure data, suggest measurement limits, and debug test sequences. This guide covers what a test copilot does, how it differs from general-purpose AI, and where the technology is heading.

## What a Test Copilot Does

| Capability | What It Looks Like |
|-----------|-------------------|
| Test script generation | Describe what you want to test in plain English, get working OpenHTF phases |
| Limit suggestion | Analyze production data and recommend measurement limits with margins |
| Failure analysis | Ask "why are units failing phase_voltage_check?" and get root cause analysis |
| Code review | Flag common test script mistakes (wrong validators, missing units, bad plug patterns) |
| Documentation | Generate test procedure documents from code |
| Troubleshooting | Describe a symptom, get diagnostic steps based on test history |

## Why Test Engineering Needs a Specialized Copilot

General-purpose AI (ChatGPT, Claude, GitHub Copilot) can write Python code, but test engineering has domain-specific patterns that generic models get wrong:

| Pattern | General AI Gets It Wrong | Test Copilot Gets It Right |
|---------|------------------------|--------------------------|
| OpenHTF plug injection | Uses type hints (doesn't work) | Uses `@htf.plug()` decorator |
| Measurement validators | Invents `.at_least()` (doesn't exist) | Uses `.in_range(minimum=x)` |
| Test limits | Picks round numbers | Derives from datasheet specs or production data |
| Instrument control | Generic SCPI examples | Instrument-specific command sequences |
| Failure analysis | Generic debugging advice | Correlates with test data patterns |

A test copilot is trained on (or has access to) test frameworks, instrument documentation, measurement science, and production data. It speaks the language of FPY, Cpk, DUT, and SCPI.

## Current State of the Technology

### What Exists Today

| Product | What It Does | Scope |
|---------|-------------|-------|
| NI Nigel AI | AI assistant trained on NI hardware, software, and test methodologies | NI ecosystem only |
| Flux Copilot | AI assistant for PCB design (not test) | Hardware design, not test |
| GitHub Copilot | Code completion for any language | Generic, not test-aware |
| TofuPilot + Claude/ChatGPT | AI assistants with access to TofuPilot data via MCP | Open, framework-agnostic |

### What's Emerging

| Capability | Status |
|-----------|--------|
| AI-generated test plans from product specifications | Research phase |
| Automatic limit optimization from production data | Early products in semiconductor |
| Natural language test specification | Academic (ASE 2025 conference papers) |
| AI-driven root cause analysis from test data | Deployed in automotive (Acerta, QualityLine) |
| Agentic test execution (AI decides what to test next) | Concept phase, Forrester defined category Q3 2025 |

## How a Test Copilot Fits Into the Workflow

| Workflow Stage | Without Copilot | With Copilot |
|---------------|----------------|-------------|
| Writing test script | Engineer writes from scratch or copies from template | Describe the test, copilot generates phases with measurements and limits |
| Setting limits | Engineer reads datasheet, picks values | Copilot analyzes production data, suggests limits with 3-sigma margins |
| Debugging failures | Engineer reviews logs, guesses root cause | Copilot correlates failure patterns across thousands of runs |
| Reviewing test coverage | Engineer manually checks requirements vs test steps | Copilot flags requirements not covered by any test phase |
| Optimizing cycle time | Engineer profiles phases manually | Copilot identifies redundant tests and suggests skip conditions |

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Example: From Description to Test Code

Today, an engineer can describe a test to an AI assistant and get working code. The quality depends on the assistant's knowledge of the test framework.

A well-trained test copilot turns this description:

> "Test a 5V power supply. Check output voltage is 4.9-5.1V, ripple is under 50mV, and efficiency is above 90%."

Into this code:

```python filename="power_supply_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("ripple_mV")
    .in_range(maximum=50)
    .with_units(units.MILLIVOLT),
    htf.Measurement("efficiency_percent")
    .in_range(minimum=90)
    .with_units(units.PERCENT),
)
def phase_power_supply_validation(test):
    """Validate power supply output characteristics."""
    test.measurements.output_voltage_V = 5.02
    test.measurements.ripple_mV = 28.3
    test.measurements.efficiency_percent = 93.1
```

```python filename="power_supply_test.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_power_supply_validation)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

The copilot knows to use `.in_range()` (not `.at_least()`), to include `units`, and to structure the test with TofuPilot integration.

## Where This Is Heading

| Timeframe | Capability |
|-----------|-----------|
| Now | AI generates test scripts from descriptions, reviews code for common mistakes |
| Near-term | AI suggests limits based on production data, identifies root causes from failure patterns |
| Mid-term | AI optimizes test sequences (adaptive testing), generates test plans from product specs |
| Long-term | Autonomous test systems that design, execute, and optimize tests with minimal human input |

The test copilot won't replace test engineers. It will handle the repetitive parts (writing boilerplate, analyzing large datasets, setting initial limits) so engineers can focus on test strategy, fixture design, and solving the hard problems that require physical intuition.

### How to Manage NPI Test Data
URL: https://www.tofupilot.com/guides/how-to-manage-npi-testing-with-tofupilot
Structure your test procedures across EVT, DVT, and PVT phases. Refine measurement limits using early data and track procedure versions as your product matures.

New product introduction means your test procedures start loose and get tighter as the design matures. TofuPilot tracks every version of your tests across EVT, DVT, and PVT, so you can see how limits evolved and use real production data to set final specifications.

## Testing Needs at Each NPI Phase

Each phase has different goals, and your tests should reflect that.

| Phase | Goal | Test approach |
|-------|------|--------------|
| EVT | Validate the design works | Wide limits, characterization measurements, data collection |
| DVT | Prove reliability and compliance | Tightened limits, environmental stress tests, margin analysis |
| PVT | Confirm manufacturing readiness | Production-ready limits, yield optimization, process capability |

During EVT, you're learning what the product actually does. During PVT, you're proving it can be built at volume. The test procedures need to evolve accordingly.

## EVT: Characterize with Wide Limits

In EVT, you don't know the true distribution of your measurements yet. Set limits wide enough to collect data without rejecting units unnecessarily. The goal is characterization, not screening.

```python filename="test_amplifier_evt.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("gain")
    .in_range(minimum=18.0, maximum=26.0)
    .doc("EVT: wide limits for characterization"),
    htf.Measurement("noise_floor")
    .in_range(maximum=-60.0)
    .doc("EVT: upper bound only, collecting distribution data"),
    htf.Measurement("input_impedance")
    .in_range(minimum=40.0, maximum=60.0)
    .with_units(units.OHM)
    .doc("EVT: centered on 50 ohm nominal, wide tolerance"),
)
def test_amplifier_performance(test):
    test.measurements.gain = 21.4
    test.measurements.noise_floor = -72.3
    test.measurements.input_impedance = 49.1

def main():
    test = htf.Test(
        test_amplifier_performance,
        station_id="LAB-BENCH-01",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "AMP-EVT-0012")

if __name__ == "__main__":
    main()
```

Upload every EVT run to TofuPilot. Even with only 20-50 units, the measurement histograms show the natural distribution of your design. This data is what you'll use to set DVT limits.

## DVT: Tighten Limits Based on EVT Data

After EVT, you have real measurement distributions in TofuPilot. Use them to set informed limits. Look at the histogram for each measurement: the mean tells you where the design centers, the spread tells you how much to allow.

```python filename="test_amplifier_dvt.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("gain")
    .in_range(minimum=20.0, maximum=23.0)
    .doc("DVT: tightened from EVT data, Cpk target > 1.33"),
    htf.Measurement("noise_floor")
    .in_range(maximum=-65.0)
    .doc("DVT: tightened based on EVT distribution"),
    htf.Measurement("input_impedance")
    .in_range(minimum=45.0, maximum=55.0)
    .with_units(units.OHM)
    .doc("DVT: narrowed from 40-60 to 45-55 based on EVT Cpk"),
)
def test_amplifier_performance(test):
    test.measurements.gain = 21.2
    test.measurements.noise_floor = -73.1
    test.measurements.input_impedance = 50.3

def main():
    test = htf.Test(
        test_amplifier_performance,
        station_id="LAB-BENCH-02",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "AMP-DVT-0087")

if __name__ == "__main__":
    main()
```

The gain limit narrowed from 18-26 (EVT) to 20-23 (DVT) because EVT data showed the design naturally centers around 21 with low spread. TofuPilot's Cpk and measurement statistics make this analysis straightforward.

## PVT: Lock Production-Ready Limits

PVT is the final gate before mass production. Limits should be tight enough to catch defective units but not so tight that they kill yield on good units. Use DVT data to set limits that achieve your Cpk target.

```python filename="test_amplifier_pvt.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("gain")
    .in_range(minimum=20.2, maximum=22.8)
    .doc("PVT: production limits, Cpk > 1.67"),
    htf.Measurement("noise_floor")
    .in_range(maximum=-67.0)
    .doc("PVT: production limit with margin"),
    htf.Measurement("input_impedance")
    .in_range(minimum=46.0, maximum=54.0)
    .with_units(units.OHM)
    .doc("PVT: final production limits"),
    htf.Measurement("thd_pct")
    .in_range(maximum=0.5)
    .doc("PVT: added THD test for production screening"),
)
def test_amplifier_performance(test):
    test.measurements.gain = 21.3
    test.measurements.noise_floor = -71.8
    test.measurements.input_impedance = 50.1
    test.measurements.thd_pct = 0.12

def main():
    test = htf.Test(
        test_amplifier_performance,
        station_id="SZ-L1-FCT-01",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "AMP-PVT-0341")

if __name__ == "__main__":
    main()
```

Notice the PVT version also adds a new measurement (`thd_pct`) that wasn't in EVT or DVT. As you learn more about the product through testing, you add tests that catch failure modes discovered during validation.

## Version-Control Your Procedure Evolution

Keep every NPI phase in version control. Tag releases so you can trace which limits were active when any unit was tested.

```
v0.1.0  EVT - initial characterization limits
v0.2.0  EVT - added noise floor measurement
v1.0.0  DVT - tightened all limits from EVT data
v1.1.0  DVT - added environmental stress sequence
v2.0.0  PVT - production-ready limits, added THD
v2.0.1  PVT - adjusted impedance limit after line trial
```

TofuPilot tracks which procedure version produced each run. You can filter by version to compare yield between EVT and DVT limits, or to see the impact of a specific limit change.

## Use TofuPilot Data to Drive Limit Decisions

The key advantage of uploading every run from every phase is that TofuPilot builds a complete picture of your product's behavior over time.

- **EVT histograms** show the design's natural distribution before any limit optimization
- **DVT Cpk values** tell you whether your tightened limits have enough margin for volume production
- **PVT yield trends** confirm the limits work on real production lines, not just lab benches
- **Cross-phase comparison** shows how the measurement distribution changed as the design and process matured

Don't set limits in a spreadsheet and hope for the best. Use the actual measurement data from each NPI phase to make informed decisions. TofuPilot keeps all of it in one place, from your first EVT prototype through PVT line trials and into mass production.

## Transition to Mass Production

When PVT is complete and limits are locked, the same test procedure and TofuPilot integration carry straight into production. There's no handoff gap. The production team inherits:

- Validated test procedures with data-driven limits
- Historical measurement distributions for comparison
- Established station configurations and naming conventions
- Alert thresholds tuned during PVT line trials

The NPI data stays in TofuPilot alongside production data, giving you a continuous record from first prototype to millionth unit.

### What Is a Continuous Test Stack
URL: https://www.tofupilot.com/guides/what-is-a-continuous-test-stack
A continuous test stack connects test development, execution, data collection, and analytics into one integrated workflow. Learn what it includes and how.

A continuous test stack connects every stage of manufacturing test into one integrated workflow: writing tests, deploying them to stations, executing on the production floor, collecting data, and analyzing results. Instead of disconnected tools at each stage, a continuous stack feeds data from production back into test development, creating a loop that improves test quality over time.

## The Fragmented Stack Problem

Most manufacturing test operations look like this:

| Stage | Tool | Data Format | Connection to Next Stage |
|-------|------|------------|------------------------|
| Write tests | Text editor or IDE | Python/LabVIEW files | Manual copy to station |
| Deploy to stations | USB drive or shared folder | Executable or script | None |
| Execute tests | Test executive (OpenHTF, TestStand) | Internal format | CSV export or local DB |
| Store results | File server or spreadsheet | CSV, Excel | Manual report generation |
| Analyze | Excel, custom scripts | Charts, pivot tables | Email to engineering |
| Improve | Engineer reads report, updates test | Back to step 1 | Manual, delayed |

Every handoff between stages is manual. Data changes format. Context is lost. The feedback loop from production data back to test improvement takes weeks or months.

## The Continuous Stack

A continuous test stack eliminates the handoffs:

| Stage | How It Works | Feedback Loop |
|-------|-------------|--------------|
| Write tests | Python scripts in Git | Production failure data informs what to test next |
| Deploy to stations | pip install or Docker pull | Version-controlled, reproducible |
| Execute tests | OpenHTF runs phases, streams results | Real-time operator UI |
| Store results | Automatic upload to TofuPilot | Structured, searchable, traceable |
| Analyze | Live dashboards, automated alerts | Immediate visibility |
| Improve | Engineer sees data, updates test, pushes to Git | Hours, not weeks |

The key difference: data flows automatically from execution to analysis, and insights flow back to development. There's no manual export step, no Excel processing, no emailed reports.

## Components of a Continuous Test Stack

| Component | Purpose | Open Source Option |
|-----------|---------|-------------------|
| Test framework | Sequence and execute test phases | OpenHTF, pytest |
| Version control | Track test script changes | Git |
| Package management | Deploy test scripts to stations | pip, Docker |
| Data platform | Store and query test results | TofuPilot |
| Operator interface | Production floor UI | TofuPilot streaming |
| Analytics | Yield, SPC, Pareto, trends | TofuPilot Analytics |
| Alerting | Notify on yield drops or drift | TofuPilot alerts |

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)
- Git for version control

## Step 1: Write Tests in Git

Keep test scripts in a Git repository. Every change is tracked, reviewable, and reversible.

```python filename="tests/production_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("current_draw_mA")
    .in_range(minimum=90, maximum=110)
    .with_units(units.MILLIAMPERE),
)
def phase_electrical(test):
    """Production electrical test."""
    test.measurements.output_voltage_V = 5.01
    test.measurements.current_draw_mA = 99.3
```

## Step 2: Deploy with pip

Package your tests as a Python package. Stations install or update with one command.

```bash filename="terminal"
pip install --upgrade your-test-package
```

This replaces the USB drive workflow. Every station runs the same version. Updates are traceable.

## Step 3: Stream Results to TofuPilot

Connect the test to TofuPilot. Results flow automatically from every station to a central platform.

```python filename="tests/production_test.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_electrical)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Step 4: Close the Loop

The continuous part: use production data to improve tests.

| Signal in TofuPilot | Action |
|---------------------|--------|
| Phase with 0% failure rate for 10K units | Consider removing from production test |
| Measurement consistently at 99% of limit | Investigate design margin, adjust limit |
| Yield drop after test script update | Git blame the change, revert if needed |
| New failure mode appears | Add a test phase to catch it |
| Station 3 has lower yield than stations 1-2 | Investigate fixture or equipment on station 3 |

Each action results in a code change in Git, a new deployment to stations, and the cycle continues. The stack is continuous because data flows in a loop, not a line.

## Continuous Stack vs Point Solutions

| Approach | Pros | Cons |
|----------|------|------|
| Point solutions (TestStand + custom DB + Excel) | Each tool is best-in-class for its stage | No integration, manual handoffs, data silos |
| Continuous stack (OpenHTF + Git + TofuPilot) | Integrated data flow, fast feedback loops | Requires upfront architecture decisions |
| Custom-built everything | Perfectly tailored | Expensive to build and maintain |

The continuous stack approach trades some per-tool flexibility for end-to-end integration. The ROI comes from faster feedback loops: when you can go from "yield dropped" to "root cause identified" to "fix deployed" in hours instead of weeks, the investment pays for itself quickly.

### Data-Driven Manufacturing Testing
URL: https://www.tofupilot.com/guides/data-driven-manufacturing-testing-with-tofupilot
Learn how to replace opinion-based test decisions with data-driven manufacturing quality control using TofuPilot's analytics platform.

Most hardware test decisions are still made on gut feeling. "I think yield dropped last week." "This component lot seems worse." "The night shift probably isn't following the procedure." TofuPilot replaces opinions with measurements, so every quality decision is backed by data.

## The Problem with Opinion-Based Testing

Hardware test operations generate enormous amounts of data. But in most organizations, that data goes into local files and never gets analyzed systematically. Decisions are made based on:

- The last failure someone remembers
- Anecdotal reports from operators
- Monthly summary reports that are outdated by the time they're reviewed
- Institutional knowledge that walks out the door when engineers leave

Data-driven testing means every decision (change limits, adjust process, qualify a supplier, release a batch) is backed by actual measurement data.

## What Data-Driven Testing Looks Like

### Setting Test Limits

**Opinion-based**: "The spec says 3.3V +/- 5%, so let's use 3.135V to 3.465V as test limits."

**Data-driven**: Measure 1,000 units. The distribution is centered at 3.31V with a standard deviation of 0.015V. Set test limits at 3.25V to 3.37V (4-sigma). This gives a Cpk of 1.33, balancing quality with false failure rates.

```python filename="calculate_limits.py"
import numpy as np

# Production measurement data from TofuPilot
values = [3.31, 3.30, 3.32, 3.29, 3.31, 3.30, 3.33, 3.31, 3.28, 3.32]
# ... (1000 values in practice)

mean = np.mean(values)
std = np.std(values, ddof=1)

# Set limits at mean +/- 4 sigma for Cpk ~ 1.33
limit_low = round(mean - 4 * std, 3)
limit_high = round(mean + 4 * std, 3)

print(f"Mean: {mean:.3f} V")
print(f"Std:  {std:.4f} V")
print(f"Recommended limits: {limit_low} V to {limit_high} V")
```

### Qualifying a New Supplier

**Opinion-based**: "The samples looked fine. Let's approve the supplier."

**Data-driven**: Run 50 units with the new supplier's components through your full test suite. Compare measurement distributions against the baseline from your current supplier.

| Measurement | Current supplier (mean) | New supplier (mean) | Shift | Verdict |
|------------|------------------------|--------------------| ------|---------|
| vcc_3v3 | 3.310 V | 3.305 V | -0.005 V | OK |
| current_idle | 42.1 mA | 48.3 mA | +6.2 mA | Investigate |
| boot_time | 340 ms | 335 ms | -5 ms | OK |

The current draw shift is within limits but significant. Investigate the root cause before full qualification.

### Deciding When to Service a Fixture

**Opinion-based**: "It's been 6 months. Time for scheduled maintenance."

**Data-driven**: Track station-specific measurement variance over time. Service the fixture when variance exceeds a threshold, not on a calendar.

| Station | Variance (week 1) | Variance (week 8) | Variance (week 16) | Action |
|---------|-------------------|-------------------|--------------------| -------|
| STN-01 | 0.008 | 0.009 | 0.011 | OK |
| STN-02 | 0.007 | 0.012 | 0.025 | Service now |
| STN-03 | 0.009 | 0.010 | 0.010 | OK |

Station 2's variance tripled. Service it now. Stations 1 and 3 are fine. Skip their scheduled maintenance.

## Building a Data-Driven Test Culture

### Step 1: Centralize Everything

No data-driven decisions are possible when test data lives in 15 different places. TofuPilot centralizes all test results from all stations, all procedures, all sites.

```python filename="centralize.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Every station, every test, every site pushes to TofuPilot
client.create_run(
    procedure_id="ICT-BOARD-V4",
    unit_under_test={"serial_number": "PCB-20251087"},
    run_passed=True,
    steps=[...],
)
```

### Step 2: Define Key Metrics

Pick the metrics that matter most to your operation:

| Metric | What it tells you | How often to review |
|--------|------------------|-------------------|
| FPY by procedure | Which test is your bottleneck | Daily |
| Cpk by measurement | Process capability | Weekly |
| Failure pareto | Top quality issues | Daily |
| Station FPY comparison | Equipment health | Weekly |
| Yield trend | Direction of quality | Daily |

### Step 3: Make Decisions from Dashboards

Replace meeting-based decisions with dashboard-based decisions. Instead of a weekly quality review where someone presents slides, open TofuPilot's dashboard and look at the actual data.

Questions the dashboard answers directly:

- "Should we release this batch?" Check FPY and measurement distributions.
- "Is Station 3 performing?" Compare its metrics to other stations.
- "Did the process change improve yield?" Compare before and after.
- "Is this supplier's quality acceptable?" Compare measurement distributions.

### Step 4: Automate Decisions Where Possible

Some decisions can be fully automated. Use TofuPilot's API to build automated gates:

- Auto-release batches with FPY above 98%
- Auto-flag stations with yield below 95%
- Auto-reject units with any critical measurement failure
- Auto-escalate when a new failure mode appears in the top 3

## The Payoff

Data-driven testing doesn't require new test equipment or radical process changes. It requires centralizing the data you're already collecting and building the habit of making decisions from dashboards instead of opinions.

The typical result: yield improves 2-5% in the first quarter, not because you changed anything fundamental, but because you started seeing problems sooner and fixing them faster.


### Production Test Validation at Scale
URL: https://www.tofupilot.com/guides/production-test-validation-at-scale-with-tofupilot
Learn how to validate your production test process at scale using TofuPilot's statistical analysis, Cpk tracking, and yield monitoring.

Running a test that works for 10 prototypes is different from running it for 10,000 production units. At scale, you need to validate not just the product, but the test process itself. Are your limits correct? Is your test repeatable? Are you catching real defects without creating false failures? TofuPilot's analytics help answer these questions.

## What Production Test Validation Means

Production test validation (PVT) answers three questions:

1. **Are the test limits correct?** Limits that are too tight cause false failures. Limits that are too loose let defective units ship.
2. **Is the test repeatable?** The same unit tested twice should give the same result.
3. **Is the test effective?** Does it catch the defects it's supposed to catch?

## Step 1: Analyze Measurement Distributions

After running your test on the first 100-200 production units, analyze the measurement distributions in TofuPilot.

```python filename="distribution_analysis.py"
import numpy as np
from tofupilot import TofuPilotClient

client = TofuPilotClient()

runs = client.get_runs(
    procedure_id="FINAL-FUNCTIONAL-V3",
    limit=200,
)

# Extract measurement values
vcc_values = []
for run in runs:
    for step in run.get("steps", []):
        for m in step.get("measurements", []):
            if m["name"] == "vcc_3v3":
                vcc_values.append(m["value"])

values = np.array(vcc_values)
print(f"N:      {len(values)}")
print(f"Mean:   {np.mean(values):.4f} V")
print(f"Std:    {np.std(values, ddof=1):.4f} V")
print(f"Min:    {np.min(values):.4f} V")
print(f"Max:    {np.max(values):.4f} V")
print(f"Range:  {np.max(values) - np.min(values):.4f} V")
```

What to look for:

| Observation | Action |
|------------|--------|
| Distribution centered within limits, Cpk > 1.33 | Limits are well-set |
| Distribution skewed toward one limit | Investigate process bias |
| Distribution wider than expected | Tighten process or widen limits |
| Outliers beyond 3-sigma | Investigate those specific units |
| Bimodal distribution | Two populations, likely mixed lots |

## Step 2: Calculate Process Capability

Cpk tells you how well your process fits within the test limits. TofuPilot provides the measurement data; you calculate the Cpk.

```python filename="cpk_validation.py"
import numpy as np

def calculate_cpk(values, lsl, usl):
    mean = np.mean(values)
    std = np.std(values, ddof=1)
    cpu = (usl - mean) / (3 * std)
    cpl = (mean - lsl) / (3 * std)
    cpk = min(cpu, cpl)
    return cpk, mean, std

# From TofuPilot data
vcc_values = [3.30, 3.31, 3.29, 3.32, 3.30, 3.31, 3.29, 3.30, 3.33, 3.31]
lsl, usl = 3.25, 3.35

cpk, mean, std = calculate_cpk(vcc_values, lsl, usl)
print(f"Cpk: {cpk:.2f}")
print(f"Mean: {mean:.3f} V")
print(f"Std: {std:.4f} V")

if cpk >= 1.67:
    print("Excellent process capability")
elif cpk >= 1.33:
    print("Acceptable process capability")
elif cpk >= 1.0:
    print("Marginal. Consider tightening process or widening limits")
else:
    print("Poor capability. Action required")
```

| Cpk | Meaning | DPMO (approx) |
|-----|---------|---------------|
| 2.0 | Excellent | 0.002 |
| 1.67 | Very good | 0.6 |
| 1.33 | Good | 63 |
| 1.0 | Marginal | 2,700 |
| 0.67 | Poor | 45,500 |

## Step 3: Validate Test Repeatability (Gauge R&R)

Test the same unit multiple times to measure your test system's repeatability.

```python filename="repeatability_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Test the same unit 30 times
serial = "GRR-GOLDEN-001"
for i in range(30):
    vcc = measure_voltage()
    client.create_run(
        procedure_id="GRR-FUNCTIONAL-V3",
        unit_under_test={"serial_number": serial},
        run_passed=True,
        steps=[{
            "name": "Power Rail",
            "step_type": "measurement",
            "status": True,
            "measurements": [{
                "name": "vcc_3v3",
                "value": vcc,
                "unit": "V",
                "limit_low": 3.25,
                "limit_high": 3.35,
            }],
        }],
    )
```

After 30 runs, analyze the spread in TofuPilot:

| Metric | Target | Meaning |
|--------|--------|---------|
| GR&R % of tolerance | < 10% | Excellent measurement system |
| GR&R % of tolerance | 10-30% | Acceptable, monitor |
| GR&R % of tolerance | > 30% | Measurement system needs improvement |

If your test measurement varies by 0.04V on the same unit and your tolerance is 0.10V, that's 40% GR&R. Your test is too noisy to reliably distinguish good from bad units.

## Step 4: Optimize Test Limits

Use production data to optimize limits. The goal: catch real defects without rejecting good units.

### Tightening Limits

If Cpk > 2.0 and you're seeing no false failures, your limits might be too loose. Tighter limits catch marginal units before they become field failures.

### Widening Limits

If Cpk < 1.0 and you're seeing false failures (units that fail test but work fine in the field), your limits are too tight for your current process capability.

### Dynamic Limits

Some teams use TofuPilot data to set limits based on the production distribution:

```python filename="dynamic_limits.py"
# Calculate limits from production data
mean = 3.310
std = 0.015

# 4-sigma limits for Cpk = 1.33
dynamic_low = mean - 4 * std   # 3.250
dynamic_high = mean + 4 * std  # 3.370
```

## Step 5: Monitor at Scale

Once your test is validated, monitor it continuously. Scale introduces new variables:

- Different operators
- Different component lots across months
- Fixture wear over thousands of cycles
- Environmental changes (season, humidity)
- Equipment calibration drift

TofuPilot's trend dashboards surface these changes. Set up monitoring for:

1. **FPY trend**: Catch yield drops within hours
2. **Cpk trend**: Catch process capability degradation within days
3. **Measurement mean shift**: Catch drift before it causes failures
4. **Failure pareto changes**: Catch new failure modes early

## Validation Checklist

Before approving a test for production volume:

- [ ] Measurement distributions are normal (or expected shape)
- [ ] Cpk > 1.33 for all critical measurements
- [ ] GR&R < 30% for all measurements
- [ ] No false failures in the last 200 units
- [ ] Test catches known defect modes (verified with known-bad units)
- [ ] Test cycle time meets throughput requirements
- [ ] All stations produce equivalent results (cross-station correlation)


### What Is Generative Test Design
URL: https://www.tofupilot.com/guides/what-is-generative-test-design
Generative test design uses AI to create test plans, scripts, and measurement strategies from product specifications. Learn where it works today and where.

Generative test design uses AI to create test plans, test scripts, and measurement strategies from product specifications. Instead of an engineer manually translating a datasheet into test code, an AI system reads the requirements and generates the test structure. This guide covers what generative test design can do today, what it can't, and where it's heading.

## From Spec to Test

Traditional test development follows this path:

| Step | Who Does It | Time |
|------|-----------|------|
| Read product spec | Test engineer | Hours |
| Identify testable requirements | Test engineer | Hours |
| Choose measurements and limits | Test engineer | Hours |
| Write test code | Test engineer | Days |
| Debug and validate | Test engineer | Days |
| Document test procedure | Test engineer | Hours |

Generative test design compresses steps 1-4:

| Step | Who Does It | Time |
|------|-----------|------|
| Feed spec to AI | Test engineer | Minutes |
| AI generates test structure | AI | Seconds |
| Engineer reviews and refines | Test engineer | Hours |
| Validate on hardware | Test engineer | Hours |
| Document (auto-generated from code) | AI | Minutes |

The engineer's role shifts from writing to reviewing. The AI handles the translation from requirements to code. The engineer validates that the translation is correct and the tests work on real hardware.

## What Works Today

| Capability | Status | Quality |
|-----------|--------|---------|
| Generate OpenHTF phases from a text description | Works | Good for standard patterns (voltage check, communication test) |
| Suggest measurement limits from a datasheet | Works | Needs engineer review for margin decisions |
| Generate test scripts from similar existing tests | Works well | AI can adapt a power supply test to a different model |
| Create test procedure documentation from code | Works | Accurate but may need formatting adjustments |
| Generate fixture designs from board geometry | Early research | Not production-ready |

## What Doesn't Work Yet

| Capability | Why Not |
|-----------|---------|
| Generate tests for novel products with no precedent | AI needs examples to learn from |
| Choose between test methods (ICT vs FCT vs boundary scan) | Requires physical understanding of the product |
| Design test fixtures | Requires 3D geometry and mechanical knowledge |
| Set limits that account for manufacturing variation | Requires production data that doesn't exist yet for new products |
| Replace domain expertise for safety-critical tests | Regulatory requirements demand human judgment |

## Example: Spec to Test Code

Given this specification fragment:

> Output voltage: 12V +/- 5%
> Output current: 0-2A
> Ripple: < 50mV peak-to-peak
> Efficiency: > 85% at full load
> Operating temperature: -20C to 60C

An AI generates:

```python filename="generated_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=11.4, maximum=12.6)
    .with_units(units.VOLT),
    htf.Measurement("ripple_mVpp")
    .in_range(maximum=50)
    .with_units(units.MILLIVOLT),
    htf.Measurement("efficiency_percent")
    .in_range(minimum=85)
    .with_units(units.PERCENT),
)
def phase_output_validation(test):
    """Validate output characteristics per product specification."""
    test.measurements.output_voltage_V = 12.03
    test.measurements.ripple_mVpp = 28.4
    test.measurements.efficiency_percent = 91.2
```

The AI correctly computed 12V +/- 5% as 11.4-12.6V, chose appropriate units, and structured the code following OpenHTF patterns. The engineer still needs to:

- Add instrument control code (the AI generated placeholder values)
- Review the limits (should there be marginal bands?)
- Add phases for load testing at 0A and 2A
- Add temperature-dependent tests if required

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Start with a Template

Generative test design works best when the AI has examples to learn from. Start with a working test and ask the AI to adapt it for a new product.

```python filename="template_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=4.75, maximum=5.25)
    .with_units(units.VOLT),
)
def phase_voltage_check(test):
    """Template: measure output voltage against spec."""
    test.measurements.output_voltage_V = 5.01
```

From this template, the AI can generate variants for different voltage levels, add current measurements, include ripple checks, and create multi-phase test sequences.

## Step 2: Review and Refine

AI-generated test code needs human review before deployment. Check for:

| Check | What to Verify |
|-------|---------------|
| Limits | Do they match the spec exactly? Are margins appropriate? |
| Units | Are units correct and consistent? |
| Measurement names | Are they descriptive and consistent with your naming convention? |
| Phase structure | Are phases logically grouped? Is the sequence correct? |
| Missing tests | Did the AI miss any requirement from the spec? |
| Instrument control | Is the SCPI command sequence correct for your instruments? |

## Step 3: Log and Learn

Connect the generated test to TofuPilot. As production data accumulates, it feeds back into better test generation for the next product.

```python filename="template_test.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_voltage_check)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Where This Is Heading

| Timeframe | Capability |
|-----------|-----------|
| Now | Generate test code from text descriptions and adapt existing templates |
| Near-term | Generate test plans from product specs with requirement traceability |
| Mid-term | AI suggests which tests to add based on failure patterns in similar products |
| Long-term | End-to-end: spec in, validated test system out, including fixture design and instrument selection |

Generative test design won't eliminate the need for test engineers. It will make them faster. The engineers who learn to use AI as a design tool will build test systems in days instead of weeks.

### How to Automate Manufacturing Test Reports
URL: https://www.tofupilot.com/guides/automated-test-reporting-with-tofupilot
Learn how to generate automated hardware test reports from TofuPilot data, replacing manual Excel and PDF report creation.

Test reports are necessary. Building them manually is not. TofuPilot stores all your test data in a structured format, so reports can be generated automatically instead of assembled by hand in Excel every Friday.

## The Manual Reporting Problem

A typical weekly quality report requires:

1. Export data from each test station (15 min per station)
2. Merge exports into one spreadsheet (30 min)
3. Calculate FPY, failure pareto, and measurement stats (45 min)
4. Build charts (30 min)
5. Write summary and observations (30 min)
6. Format and distribute (15 min)

That's 3+ hours every week. For a quality engineer who should be analyzing data, not formatting cells.

## What TofuPilot Automates

TofuPilot replaces steps 1-4 entirely. The data is already centralized. The metrics are already computed. The charts are already drawn.

| Manual step | TofuPilot equivalent |
|------------|---------------------|
| Export from each station | Data uploads automatically |
| Merge into one spreadsheet | Already in one database |
| Calculate FPY | Computed in real time |
| Build charts | Built-in dashboards |
| Failure pareto | Auto-generated from test results |
| Measurement distributions | Histograms per measurement |

## Types of Test Reports

### Per-Unit Test Report

A complete record of one unit's test results. Useful for:
- Customer acceptance documentation
- Warranty records
- Regulatory submissions

Contents: serial number, all test procedures run, all measurements with limits, pass/fail status, timestamps.

### Batch Summary Report

Aggregate quality data for a production batch. Useful for:
- Production release decisions
- Supplier quality reviews
- Management dashboards

Contents: batch size, FPY, failure pareto, measurement distributions, Cpk values.

### Station Performance Report

How each test station is performing. Useful for:
- Maintenance planning
- Capacity planning
- Station qualification

Contents: FPY per station, cycle time, failure modes, measurement distributions.

## Generating Reports via the API

Use TofuPilot's API to pull data for custom report formats.

```python filename="batch_report.py"
from tofupilot import TofuPilotClient
from datetime import datetime, timedelta

client = TofuPilotClient()

# Pull all runs for a procedure in the last 7 days
runs = client.get_runs(
    procedure_id="FINAL-FUNCTIONAL-V3",
    limit=500,
)

# Calculate batch statistics
total = len(runs)
passed = sum(1 for r in runs if r["run_passed"])
fpy = passed / total if total > 0 else 0

# Failure mode breakdown
failure_modes = {}
for run in runs:
    if not run["run_passed"]:
        for step in run.get("steps", []):
            if not step["status"]:
                mode = step["name"]
                failure_modes[mode] = failure_modes.get(mode, 0) + 1

print(f"Batch Report - FINAL-FUNCTIONAL-V3")
print(f"Period: Last 7 days")
print(f"Total units: {total}")
print(f"Passed: {passed}")
print(f"FPY: {fpy:.1%}")
print()
print("Top failure modes:")
for mode, count in sorted(failure_modes.items(), key=lambda x: -x[1]):
    print(f"  {mode}: {count} ({count/total:.1%})")
```

### Generating a CSV Export

```python filename="export_csv.py"
import csv
from tofupilot import TofuPilotClient

client = TofuPilotClient()

runs = client.get_runs(
    procedure_id="FINAL-FUNCTIONAL-V3",
    limit=1000,
)

with open("test_report.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["Serial", "Date", "Status", "Step", "Measurement", "Value", "Unit", "Low Limit", "High Limit", "Pass"])

    for run in runs:
        serial = run["unit_under_test"]["serial_number"]
        date = run["created_at"]
        status = "PASS" if run["run_passed"] else "FAIL"

        for step in run.get("steps", []):
            for m in step.get("measurements", []):
                writer.writerow([
                    serial,
                    date,
                    status,
                    step["name"],
                    m["name"],
                    m["value"],
                    m.get("unit", ""),
                    m.get("limit_low", ""),
                    m.get("limit_high", ""),
                    "PASS" if step["status"] else "FAIL",
                ])

print(f"Exported {len(runs)} runs to test_report.csv")
```

## Report Scheduling

### Option A: Cron-Based Reports

```python filename="weekly_report.py"
# Run every Monday at 7 AM via cron
# 0 7 * * 1 python3 /opt/reports/weekly_report.py

from tofupilot import TofuPilotClient
import smtplib
from email.mime.text import MIMEText

client = TofuPilotClient()

# Generate report content
runs = client.get_runs(procedure_id="FINAL-FUNCTIONAL-V3", limit=500)
total = len(runs)
passed = sum(1 for r in runs if r["run_passed"])
fpy = passed / total if total > 0 else 0

report = f"""Weekly Test Report - FINAL-FUNCTIONAL-V3

Units tested: {total}
First-pass yield: {fpy:.1%}
"""

# Send via email
msg = MIMEText(report)
msg["Subject"] = f"Weekly Test Report - FPY {fpy:.1%}"
msg["From"] = "reports@yourcompany.com"
msg["To"] = "quality-team@yourcompany.com"

with smtplib.SMTP("smtp.yourcompany.com") as server:
    server.send_message(msg)
```

### Option B: Dashboard Links

Instead of generating PDF reports, share TofuPilot dashboard links. The recipient sees live data, not a snapshot from when the report was generated.

Advantages over static reports:
- Always up to date
- Interactive (filter, drill down, compare)
- No generation step needed
- No distribution step needed

## Replacing the Weekly Quality Meeting

Instead of spending the first 30 minutes of your quality meeting presenting data, open TofuPilot's dashboard and start the conversation from the data. The meeting shifts from "here's what happened" to "here's what we should do about it."


### Call a 32-bit DLL from a TofuPilot Plug
URL: https://www.tofupilot.com/guides/call-a-32-bit-dll-from-a-tofupilot-plug
Learn how to keep a 32-bit vendor DLL working on a 64-bit TofuPilot station by hosting it in a msl-loadlib 32-bit server behind a plug.

A test station running the TofuPilot CLI gets a 64-bit Python, and a 64-bit process cannot load a 32-bit DLL. Learn how to keep a 32-bit vendor library in service by hosting it in its own 32-bit process with msl-loadlib, and exposing it to your phases as an ordinary plug.

## Why a 64-bit Process Cannot Load a 32-bit DLL

A DLL is loaded into the memory space of the process that calls it. Pointers are 8 bytes in a 64-bit process and 4 bytes in a 32-bit one, so the two cannot share an address space. Windows refuses the load before any code runs:

```text filename="error.txt"
OSError: [WinError 193] %1 is not a valid Win32 application
```

The station interpreter is 64-bit by construction. The CLI provisions each procedure's virtual environment with `uv venv --python 3.12`, which takes the native architecture of the machine, and no setting changes it. Old vendor libraries, in-house DLLs whose source is lost and shared libraries built with an old LabVIEW are the usual 32-bit survivors.

Two ways out exist:

| Option | Needs | Result |
|---|---|---|
| Rebuild the DLL in 64-bit | The source and a 64-bit build of every dependency | The dependency disappears for good |
| Host the DLL in a 32-bit process | Nothing but the DLL | The sequencer stays 64-bit, the DLL keeps running |

This guide does the second. The isolation is the same idea as an optocoupler between two voltage domains: signals cross, wires do not.

## Prerequisites

- A Windows test station with the TofuPilot CLI installed
- The 32-bit DLL, with its header file or its documentation, so you know each function prototype and the calling convention (`__cdecl` or `__stdcall`)
- msl-loadlib 1.1 or later, which ships a prebuilt 32-bit server for Windows, so no 32-bit Python needs to be installed on the station

## Step 1: Add msl-loadlib to the Procedure Dependencies

The CLI installs the requirements of a procedure into its virtual environment on `tofupilot run` and on every deployment:

```text filename="requirements.txt"
msl-loadlib>=1.1
```

msl-loadlib is a thin wrapper around ctypes. The difference is where the ctypes code runs: inside a 32-bit server process that the library starts for you, instead of inside your 64-bit Python.

## Step 2: Move the ctypes Code into a 32-bit Server Class

The server module is imported by the 32-bit server, never by the station Python. Put every ctypes call in it, and every buffer the DLL writes into. The example wraps a fictional I2C library with four functions:

```python filename="plugs/busbridge_server32.py"
import ctypes

from msl.loadlib import Server32


class BusBridgeServer(Server32):
    def __init__(self, host, port, **kwargs):
        # kwargs come from the Client64 constructor. msl-loadlib turns
        # every value into a string, so convert them back here.
        dll_path = kwargs["dll_path"]
        device_index = int(kwargs.get("device_index", "0"))

        # "cdll" for a __cdecl DLL, "windll" for a __stdcall one.
        super().__init__(dll_path, "cdll", host, port)

        lib = self.lib
        lib.BB_Open.argtypes = [ctypes.c_int]
        lib.BB_Open.restype = ctypes.c_int
        lib.BB_I2CWrite.argtypes = [
            ctypes.c_int,
            ctypes.c_int,
            ctypes.POINTER(ctypes.c_ubyte),
            ctypes.c_int,
        ]
        lib.BB_I2CWrite.restype = ctypes.c_int
        lib.BB_I2CRead.argtypes = [
            ctypes.c_int,
            ctypes.c_int,
            ctypes.POINTER(ctypes.c_ubyte),
            ctypes.c_int,
        ]
        lib.BB_I2CRead.restype = ctypes.c_int
        lib.BB_Close.argtypes = [ctypes.c_int]
        lib.BB_Close.restype = None

        self._handle = lib.BB_Open(device_index)
        if self._handle < 0:
            raise RuntimeError(f"BB_Open({device_index}) failed with code {self._handle}")

    def i2c_write(self, address, data):
        buf = (ctypes.c_ubyte * len(data))(*data)
        rc = self.lib.BB_I2CWrite(self._handle, address, buf, len(data))
        if rc != 0:
            raise RuntimeError(f"BB_I2CWrite(0x{address:02X}) failed with code {rc}")

    def i2c_read(self, address, length):
        buf = (ctypes.c_ubyte * length)()
        count = self.lib.BB_I2CRead(self._handle, address, buf, length)
        if count < 0:
            raise RuntimeError(f"BB_I2CRead(0x{address:02X}) failed with code {count}")
        # A list of ints crosses the process boundary. The buffer does not.
        return list(buf[:count])

    def close(self):
        if self._handle >= 0:
            self.lib.BB_Close(self._handle)
            self._handle = -1
```

Three rules apply to every method of the server class:

- **Arguments and return values are plain Python values**: numbers, strings, lists, dicts. They are pickled across the two processes.
- **Pointers never cross.** A buffer the DLL fills is allocated in the method, read in the method, and returned as a list.
- **The library handle is `self.lib`.** It is a `ctypes.CDLL` for `cdll` and a `ctypes.WinDLL` for `windll`. Set `argtypes` and `restype` once in `__init__`, as you would with plain ctypes.

An exception raised in a server method does not cross as itself. The client raises `msl.loadlib.exceptions.Server32Error`, whose message carries the server-side traceback, including the text you wrote. The phase sees one exception with a readable cause, which is what a failing phase needs.

## Step 3: Write the Plug as a 64-bit Client

The plug is the class the engine instantiates. It inherits from `Client64`, starts the 32-bit server in its constructor, and forwards each method with `request32`:

```python filename="plugs/busbridge.py"
import os

from msl.loadlib import Client64

SERVER_MODULE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "busbridge_server32.py")


class BusBridge(Client64):
    def __init__(self, dll_path, device_index=0):
        # Starts the 32-bit server (a few seconds) and imports
        # SERVER_MODULE inside it. The extra keyword arguments are
        # handed to BusBridgeServer.__init__ as strings.
        super().__init__(SERVER_MODULE, dll_path=dll_path, device_index=device_index)
        self.dll_path = dll_path

    def i2c_write(self, address, data):
        self.request32("i2c_write", address, list(data))

    def i2c_read(self, address, length):
        return self.request32("i2c_read", address, length)

    def __del__(self):
        # The engine calls __del__ when the plug is destroyed. Without
        # the shutdown, a 32-bit server process would outlive each run.
        try:
            self.request32("close")
        except Exception:
            pass
        try:
            self.shutdown_server32()
        except AttributeError:
            pass  # the server never started, there is nothing to stop
```

`request32` takes the name of a server method and its positional arguments, and returns what the method returned. The keyword arguments passed to `Client64.__init__` reach the server constructor, with one caveat: `host`, `port`, `timeout`, `rpc_timeout`, `server32_dir`, `append_sys_path`, `append_environ_path` and `add_dll_directory` are msl-loadlib's own parameters, so name yours differently.

The engine guarantees the `__del__` call when the plug is destroyed. The method shuts down the 32-bit server, so no orphan process is left between runs.

## Step 4: Declare the Plug and Use It in a Phase

```yaml filename="procedure.yaml"
name: EEPROM readback through a 32-bit bus bridge DLL
version: 0.1.0

plugs:
  - name: Bus Bridge
    key: bus_bridge
    description: 32-bit vendor DLL hosted in a msl-loadlib 32-bit server
    python: plugs.busbridge:BusBridge
    scope: station
    config:
      dll_path: "C:\\BusBridge\\busbridge.dll"
      device_index: 0

main:
  - name: EEPROM header readback
    python: phases.i2c_readback
    measurements:
      - name: eeprom_header_byte0
        validators:
          - operator: "=="
            expected_value: 165
```

`scope: station` keeps one instance alive for the life of the station process, so the server start (two to three seconds measured on a Windows runner) and the `BB_Open` call are paid once, not on every run. The `config` block is passed to the plug constructor as keyword arguments, which is where the DLL path belongs: a station-specific path has no place in Python code.

The phase receives the plug by its key and calls it like any plug:

```python filename="phases/i2c_readback.py"
EEPROM_ADDRESS = 0x50


def i2c_readback(bus_bridge, measurements, log):
    # Plug arguments are positional and JSON-serializable: ints and
    # lists of ints here, never bytes objects or ctypes buffers.
    bus_bridge.i2c_write(EEPROM_ADDRESS, [0x00, 0x00])
    header = bus_bridge.i2c_read(EEPROM_ADDRESS, 4)
    log.info(f"EEPROM header: {[hex(b) for b in header]}")
    measurements.eeprom_header_byte0 = header[0]
```

Two process boundaries sit between the phase and the DLL: the engine's own plug process, then the 32-bit server. Both carry the same kind of values, so the rule is the same on both sides. Pass numbers, strings and lists, never buffers or handles.

## Step 5: Reuse Existing 32-bit Python Modules Without Rewriting Them

If the ctypes code already exists as Python modules written against a 32-bit Python, you do not have to move it into the server class. The server class can import those modules unchanged and delegate to them:

```python filename="plugs/busbridge_server32.py"
import sys

from msl.loadlib import Server32

# Drop the 64-bit site-packages the client passes along, so the import
# below cannot pick up a 64-bit build of numpy or comtypes.
Server32.remove_site_packages_64bit()

import legacy_busbridge  # the existing module, untouched


class BusBridgeServer(Server32):
    def __init__(self, host, port, **kwargs):
        super().__init__(kwargs["dll_path"], "cdll", host, port)
        self._dev = legacy_busbridge.BusBridge(int(kwargs["device_index"]))

    def i2c_read(self, address, length):
        return list(self._dev.read(address, length))
```

Two things change compared to the first version:

1. **The server must find the legacy modules.** Pass their directory with `append_sys_path` in the `Client64` constructor.
2. **The server must contain their dependencies.** The prebuilt server ships with the standard library only. If the legacy modules import numpy, pyserial or comtypes, build a custom server once from the 32-bit Python environment that already runs them:

```text filename="build-server32.txt"
# In the existing 32-bit Python environment (3.8 or later),
# run from the directory that holds the legacy modules
pip install msl-loadlib pyinstaller
freeze32 --imports numpy serial legacy_busbridge --dest server32-custom
```

The command writes a `server32-windows.exe` into `server32-custom`. Every module named after `--imports` is frozen into it, the legacy module included, so `append_sys_path` is no longer needed for them. Point the plug at that directory with the `server32_dir` argument of `Client64.__init__`, and the legacy code runs as it always did, inside its own interpreter. One trap: freeze32 reports a module it cannot import but still exits with status 0, so check that the executable exists before shipping it.

## Keep Large Data on the 32-bit Side

Each `request32` call pickles its arguments and its result through a local socket. A few hundred bytes cost nothing. A long acquisition does: a one-million-sample waveform in 16-bit is 2 MB per call, and a phase that pulls fifty of them to average a noise figure spends more time copying than measuring.

Move the computation to the data instead. Compute the measurement in the server method and return the numbers:

```python filename="plugs/daq_server32.py"
def rms_noise(self, acquisitions):
    """Acquire `acquisitions` waveforms and return RMS noise, computed here."""
    import math

    total = 0.0
    count = 0
    for _ in range(acquisitions):
        buf = (ctypes.c_int16 * self._samples)()
        self.lib.DAQ_Acquire(self._handle, buf, self._samples)
        mean = sum(buf) / self._samples
        total += sum((s - mean) ** 2 for s in buf)
        count += self._samples
    return math.sqrt(total / count) * self._volts_per_lsb
```

The phase receives one float. The samples never leave the 32-bit process.

## Common Pitfalls

**Passing a pointer or a ctypes object across `request32`.** The call fails to pickle, or worse, the 32-bit side reads a 64-bit address. Allocate in the server method, return values.

**Using the wrong calling convention.** A `__stdcall` DLL loaded as `cdll` corrupts the stack on the first call. The header says which, and `windll` is the fix.

**Forgetting that server kwargs are strings.** `device_index=0` arrives as `"0"`. Convert in the server constructor.

**Importing numpy in the server without a custom build.** The prebuilt server has no numpy, and the 64-bit site-packages it inherits holds the wrong one. Call `remove_site_packages_64bit()` and freeze a server that contains the packages you need.

**Declaring the plug with `scope: slot` on a shared bus.** Several slots would start several 32-bit servers against one DLL and one physical interface. Use `station` or `execution` so calls serialize through one instance.

**Dropping the `shutdown_server32()` call.** Each run leaves a 32-bit process behind. Keep it in `__del__`.

**Sending whole acquisitions through the boundary.** Compute in the server, return numbers.

## Key Points

- A 64-bit Python cannot load a 32-bit DLL, and the station interpreter is 64-bit with no override.
- msl-loadlib runs your ctypes code in a prebuilt 32-bit server and gives the 64-bit side a client class.
- The plug is the client. Phases see an ordinary plug with ordinary methods.
- Pointers and buffers stay in the server. Only numbers, strings, lists and dicts cross.
- Existing 32-bit Python modules can be hosted unchanged, with a custom frozen server when they need packages.
- Keep heavy data on the 32-bit side and return the measurement, not the raw samples.

### DO-178C Test Traceability
URL: https://www.tofupilot.com/guides/do-178c-test-traceability-with-tofupilot
Learn how to maintain DO-178C compliant test traceability for airborne software using TofuPilot's structured test records and requirement mapping.

DO-178C (Software Considerations in Airborne Systems and Equipment Certification) requires complete traceability from requirements to test cases to test results. For hardware-software integrated systems, this means linking every test measurement to the requirement it verifies. TofuPilot provides the structured test data layer.

## DO-178C and Testing

DO-178C defines five Design Assurance Levels (DAL), from A (catastrophic) to E (no safety effect). Higher levels require more rigorous testing and documentation.

| DAL | Failure condition | Testing rigor |
|-----|-------------------|---------------|
| A | Catastrophic | Full MC/DC coverage, independence |
| B | Hazardous | Full decision coverage |
| C | Major | Full statement coverage |
| D | Minor | Basic testing |
| E | No effect | Minimal |

Regardless of DAL, all levels require traceability between requirements, test cases, and test results.

## The Traceability Chain

```
Requirements → Test Cases → Test Procedures → Test Results
     ↓              ↓              ↓                ↓
   DOORS         Test Plan      TofuPilot       TofuPilot
   or Jama        (doc)        Procedure       Run Results
```

TofuPilot handles the right half: test procedures (defined as procedure IDs with steps and measurements) and test results (actual run data with pass/fail).

## Mapping Requirements to Test Procedures

Use a consistent naming convention that links TofuPilot procedures to requirements.

```python filename="do178c_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Procedure ID includes the requirement reference
client.create_run(
    procedure_id="TC-SW-REQ-042-AIRSPEED-COMPUTATION",
    unit_under_test={
        "serial_number": "ADC-UNIT-007",
        "part_number": "AIR-DATA-COMPUTER-V2",
    },
    run_passed=True,
    steps=[{
        "name": "Airspeed Computation Accuracy",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "indicated_airspeed_error_kts", "value": 0.8, "unit": "kts", "limit_high": 2.0},
            {"name": "true_airspeed_error_kts", "value": 1.2, "unit": "kts", "limit_high": 3.0},
            {"name": "mach_number_error", "value": 0.002, "unit": "Mach", "limit_high": 0.005},
        ],
    }, {
        "name": "Airspeed Range Verification",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "min_airspeed_kts", "value": 30, "unit": "kts", "limit_high": 40},
            {"name": "max_airspeed_kts", "value": 450, "unit": "kts", "limit_low": 400},
        ],
    }],
)
```

### Requirements Traceability Matrix

| Req ID | Requirement | Test Procedure | TofuPilot Measurement | Limit |
|--------|------------|----------------|----------------------|-------|
| SW-REQ-042 | Airspeed computation accuracy < 2 kts | TC-SW-REQ-042 | indicated_airspeed_error_kts | < 2.0 kts |
| SW-REQ-043 | True airspeed accuracy < 3 kts | TC-SW-REQ-042 | true_airspeed_error_kts | < 3.0 kts |
| SW-REQ-044 | Mach computation accuracy < 0.005 | TC-SW-REQ-042 | mach_number_error | < 0.005 |
| SW-REQ-045 | Operate from 40 to 400 kts | TC-SW-REQ-042 | min/max_airspeed_kts | 40/400 |

## Hardware-Software Integration Testing

DO-178C Section 6.4 covers hardware/software integration testing. These tests verify that the software works correctly on the target hardware.

```python filename="hwsw_integration.py"
# Hardware/Software integration test
client.create_run(
    procedure_id="HWSW-INT-ADC-ARINC429",
    unit_under_test={"serial_number": "ADC-UNIT-007"},
    run_passed=True,
    steps=[{
        "name": "ARINC 429 Output Verification",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "label_airspeed_rate_hz", "value": 25.0, "unit": "Hz", "limit_low": 24.5, "limit_high": 25.5},
            {"name": "label_altitude_rate_hz", "value": 12.5, "unit": "Hz", "limit_low": 12.0, "limit_high": 13.0},
            {"name": "ssm_status_normal", "value": 1, "unit": "bool", "limit_low": 1},
            {"name": "data_latency_ms", "value": 18.2, "unit": "ms", "limit_high": 40.0},
        ],
    }, {
        "name": "Watchdog Timer Verification",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "watchdog_timeout_ms", "value": 50, "unit": "ms", "limit_low": 45, "limit_high": 55},
            {"name": "reset_recovery_ms", "value": 120, "unit": "ms", "limit_high": 200},
        ],
    }],
)
```

## Regression Testing

When software is modified, DO-178C requires regression testing to verify no unintended effects. TofuPilot makes regression tracking straightforward:

1. Run the full test suite on the new software version
2. Compare results against the baseline (previous version) in TofuPilot
3. Flag any measurements that changed beyond expected tolerance

```python filename="regression_check.py"
# Compare test results across software versions
baseline_runs = client.get_runs(
    procedure_id="TC-SW-REQ-042-AIRSPEED-COMPUTATION",
    limit=10,
)

# Filter by software version through unit metadata or date range
# Compare measurement values between versions
```

If a measurement that was 0.8 kts error in version 2.1 becomes 1.9 kts error in version 2.2, the regression test catches it before certification submission.

## DER/Auditor Evidence Package

When presenting test evidence to a DER (Designated Engineering Representative) or certification authority:

| Document | Source |
|----------|--------|
| Requirements Traceability Matrix | Your requirements tool + TofuPilot procedure mapping |
| Test Procedures | TofuPilot procedure definitions with steps and limits |
| Test Results | TofuPilot run data with measurements and pass/fail |
| Test Coverage Analysis | Map of requirements to TofuPilot procedures |
| Regression Test Report | TofuPilot comparison between software versions |

TofuPilot provides the structured, timestamped test evidence. Your certification package references TofuPilot data as the authoritative test record.

## DO-254 Hardware Testing

For DO-254 (hardware assurance), the same traceability principles apply. Hardware requirements map to hardware test procedures, which map to test results in TofuPilot.

Common DO-254 test types tracked in TofuPilot:
- FPGA functional verification
- Environmental qualification
- EMI/EMC compliance
- Power supply characterization
- Timing and performance verification

Use the same procedure naming convention (requirement ID in the procedure ID) for consistent traceability across hardware and software testing.


### ISO 13485 Test Data Requirements
URL: https://www.tofupilot.com/guides/iso-13485-test-data-requirements-with-tofupilot
Learn how to meet ISO 13485 test documentation requirements for medical devices using TofuPilot's structured data and traceability features.

ISO 13485 is the quality management standard for medical device manufacturers. It requires rigorous control of production processes, including complete test records for every device produced. TofuPilot provides the structured, traceable test data storage that ISO 13485 demands.

## What ISO 13485 Requires for Test Data

ISO 13485 Section 7.5.1 (Control of Production and Service Provision) and Section 8.2.4 (Monitoring and Measurement of Product) require:

| Requirement | ISO 13485 clause | What it means for testing |
|------------|------------------|--------------------------|
| Process validation | 7.5.2 | Prove your test process produces consistent results |
| Identification and traceability | 7.5.3 | Link every test result to a specific device serial number |
| Monitoring and measurement | 8.2.4 | Define acceptance criteria and record conformity evidence |
| Control of records | 4.2.5 | Maintain test records, ensure they're retrievable and protected |
| Nonconforming product | 8.3 | Document what happens to devices that fail testing |

## How TofuPilot Maps to ISO 13485

### Traceability (7.5.3)

Every test run in TofuPilot is linked to a unit serial number. Search by serial to see the complete test history of any device.

```python filename="medical_device_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Every run ties to a unique device identifier (UDI)
client.create_run(
    procedure_id="FINAL-INSPECTION-IEC60601",
    unit_under_test={
        "serial_number": "MD-2025-00421",  # UDI or serial
        "part_number": "PULSE-OX-V3",
    },
    run_passed=True,
    steps=[{
        "name": "Electrical Safety",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "earth_leakage_ua", "value": 287, "unit": "uA", "limit_high": 500},
            {"name": "patient_leakage_ua", "value": 8.3, "unit": "uA", "limit_high": 10},
            {"name": "insulation_resistance_mohm", "value": 520, "unit": "Mohm", "limit_low": 100},
        ],
    }, {
        "name": "Functional Performance",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "spo2_accuracy_pct", "value": 1.2, "unit": "%", "limit_high": 2.0},
            {"name": "heart_rate_accuracy_bpm", "value": 1.5, "unit": "BPM", "limit_high": 3.0},
        ],
    }],
)
```

### Acceptance Criteria (8.2.4)

ISO 13485 requires documented acceptance criteria for every inspection and test. In TofuPilot, limits on measurements serve as your acceptance criteria.

| Measurement | Acceptance criteria | Standard reference |
|------------|--------------------|--------------------|
| Earth leakage | < 500 uA | IEC 60601-1 |
| Patient leakage | < 10 uA (Type BF) | IEC 60601-1 |
| Insulation resistance | > 100 Mohm | IEC 60601-1 |
| SpO2 accuracy | < 2% ARMS | ISO 80601-2-61 |

These limits are defined in the test procedure and enforced by TofuPilot on every run.

### Control of Records (4.2.5)

ISO 13485 requires records to be:

| Requirement | How TofuPilot meets it |
|------------|----------------------|
| Legible | Structured data, not handwritten |
| Readily identifiable | Searchable by serial, procedure, date |
| Retrievable | API and dashboard access |
| Protected from damage | Cloud-hosted, backed up |
| Protected from loss | Redundant storage |
| Retained for defined period | Data retention per your QMS policy |

### Nonconforming Product (8.3)

When a device fails testing, ISO 13485 requires documentation of:

1. The nature of the nonconformity (which measurement failed)
2. Actions taken (rework, scrap, use-as-is disposition)
3. Re-inspection results after rework

TofuPilot captures #1 automatically (failed measurements with values and limits). For #2 and #3, the retest flow creates additional run records linked to the same serial number.

```python filename="retest_after_rework.py"
# Device failed initial test
# After rework, retest and record
client.create_run(
    procedure_id="FINAL-INSPECTION-IEC60601-RETEST",
    unit_under_test={
        "serial_number": "MD-2025-00421",
    },
    run_passed=True,
    steps=[{
        "name": "Post-Rework Electrical Safety",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "earth_leakage_ua", "value": 312, "unit": "uA", "limit_high": 500},
            {"name": "patient_leakage_ua", "value": 7.1, "unit": "uA", "limit_high": 10},
        ],
    }],
)
```

The device now has two test records in TofuPilot: the initial failure and the post-rework pass. Both are traceable, timestamped, and immutable.

## Device History Record (DHR)

The DHR is a compilation of records that documents the production history of a finished device. Test records are a key component.

TofuPilot contributes to the DHR by providing:

- All test procedures executed on the device
- All measurement results with pass/fail status
- Dates and times of each test
- Which station/tester ran each test
- Any retests after rework

Pull a device's complete test history by serial number for DHR compilation.

## Audit Readiness

When an auditor asks for test records, you need to produce them quickly and completely.

Common audit requests and how to answer them with TofuPilot:

| Auditor request | TofuPilot response |
|----------------|-------------------|
| "Show me test records for lot 2025-03" | Filter by date range, export all runs |
| "What's the acceptance criteria for leakage testing?" | Show measurement limits in the procedure |
| "Show me all devices that failed and were reworked" | Filter for failed runs, check retests by serial |
| "Prove your test equipment is calibrated" | Station metadata with calibration dates |
| "What's your first-pass yield for this product?" | Procedure dashboard FPY metric |

## IEC 60601 Electrical Safety Tests

Medical devices connected to patients require IEC 60601-1 electrical safety testing. Key measurements:

| Test | Type BF limit | Type CF limit |
|------|--------------|---------------|
| Earth leakage (normal) | 500 uA | 500 uA |
| Earth leakage (single fault) | 1000 uA | 1000 uA |
| Patient leakage (normal) | 100 uA | 10 uA |
| Patient leakage (single fault) | 500 uA | 50 uA |

Log all of these in TofuPilot with limits matching your device classification. The structured records serve as evidence of conformity for regulatory submissions.

## Software Validation

If you're using TofuPilot as part of your quality system, ISO 13485 Section 7.5.2.1 requires validation of software used in production and quality management. Work with your quality team to document TofuPilot's role in your QMS and include it in your software validation plan.


### Video and Log Sync for Hardware Tests
URL: https://www.tofupilot.com/guides/video-and-log-synchronization-for-hardware-tests
Learn how to attach video recordings, log files, and supplementary data to hardware test runs in TofuPilot for synchronized review.

Measurements tell you what happened. Videos and logs tell you why. A failed vibration test makes more sense when you can watch the unit rattle loose. A firmware crash is easier to debug when the serial console log is right next to the test result. TofuPilot stores attachments alongside measurements.

## Why Attachments Matter

A measurement value of "FAIL" tells you something broke. But it doesn't tell you:

- What the unit physically looked like during the test
- What the firmware logged before the crash
- What the oscilloscope waveform looked like
- What the operator saw on their screen

Attachments bridge this gap. They turn a test result from a number into a complete record of what happened.

## Types of Test Attachments

| Attachment type | Use case | Format |
|----------------|----------|--------|
| Video recording | Visual inspection, mechanical tests | MP4, AVI |
| Serial console log | Firmware debug, boot sequence | TXT, LOG |
| Oscilloscope capture | Waveform analysis, timing | PNG, CSV |
| Thermal image | Hot spot detection | JPEG, PNG |
| Test station log | Test software debug | TXT, LOG |
| Configuration file | DUT or fixture settings | JSON, YAML |
| Photo | Physical defect documentation | JPEG, PNG |

## Attaching Files to Test Runs

### With the Python Client

```python filename="test_with_attachments.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

client.create_run(
    procedure_id="VIBRATION-SCREENING",
    unit_under_test={"serial_number": "UNIT-7832"},
    run_passed=False,
    steps=[{
        "name": "Random Vibration 20-2000Hz",
        "step_type": "measurement",
        "status": False,
        "measurements": [{
            "name": "resonance_freq_hz",
            "value": 847,
            "unit": "Hz",
            "limit_low": 900,
            "limit_high": 1500,
        }],
    }],
    attachments=[
        "recordings/unit-7832-vibration.mp4",
        "logs/unit-7832-accel.csv",
        "captures/unit-7832-spectrum.png",
    ],
)
```

### Capturing Video Automatically

Record video during the test and attach it to the run.

```python filename="video_capture.py"
import subprocess
import os

def start_recording(output_path):
    """Start recording from a USB camera."""
    proc = subprocess.Popen([
        "ffmpeg", "-f", "v4l2", "-i", "/dev/video0",
        "-t", "120",  # max 2 minutes
        "-y", output_path,
    ], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    return proc

def stop_recording(proc):
    """Stop the recording."""
    proc.terminate()
    proc.wait()

# Usage in a test
video_path = f"/tmp/test_{serial}.mp4"
recorder = start_recording(video_path)

# Run the test...
run_test()

stop_recording(recorder)

# Attach to TofuPilot run
client.create_run(
    procedure_id="MECHANICAL-TEST",
    unit_under_test={"serial_number": serial},
    run_passed=passed,
    steps=steps,
    attachments=[video_path],
)
```

### Capturing Serial Console Logs

```python filename="serial_logger.py"
import serial
import threading

class SerialLogger:
    def __init__(self, port, baudrate=115200):
        self.ser = serial.Serial(port, baudrate, timeout=1)
        self.log = []
        self.running = False

    def start(self):
        self.running = True
        self.log = []
        self.thread = threading.Thread(target=self._read_loop)
        self.thread.start()

    def stop(self):
        self.running = False
        self.thread.join()
        self.ser.close()

    def _read_loop(self):
        while self.running:
            line = self.ser.readline().decode("utf-8", errors="replace")
            if line:
                self.log.append(line)

    def save(self, path):
        with open(path, "w") as f:
            f.writelines(self.log)

# Usage
logger = SerialLogger("/dev/ttyUSB0")
logger.start()

# Run the test...
run_test()

logger.stop()
log_path = f"/tmp/console_{serial}.log"
logger.save(log_path)

# Attach to TofuPilot
client.create_run(
    procedure_id="FIRMWARE-VALIDATION",
    unit_under_test={"serial_number": serial},
    run_passed=passed,
    steps=steps,
    attachments=[log_path],
)
```

## Synchronized Review

When you open a test run in TofuPilot, all attachments are available alongside the measurements. This synchronized view lets you:

1. See that `resonance_freq_hz` failed at 847 Hz (below the 900 Hz limit)
2. Watch the video to see the enclosure flexing during the test
3. Open the accelerometer CSV to see the raw vibration data
4. Check the spectrum plot to identify the resonant mode

All in one place, for one run, with one click.

## When to Attach What

Not every test needs video and logs. Attachments add storage and complexity. Use them where they add diagnostic value.

| Test type | Recommended attachments |
|-----------|------------------------|
| Vibration/shock | Video, accelerometer data |
| Firmware validation | Serial console log |
| Visual inspection | Photo of DUT |
| Power testing | Oscilloscope captures |
| Environmental testing | Temperature/humidity logs |
| Burn-in | All logs (long-duration, hard to reproduce) |

For routine production tests (ICT, basic functional), measurements alone are usually sufficient. Save attachments for tests where visual or log context helps debugging.

## Storage Considerations

| Practice | Why |
|----------|-----|
| Compress videos before attaching | Reduce storage costs |
| Keep attachments under 50MB per run | Practical upload/download speed |
| Use PNG for plots, JPEG for photos | PNG for clarity, JPEG for size |
| Only attach to failing runs (optionally) | Save storage on passing runs where attachments are rarely reviewed |
| Include timestamps in log files | Correlate log events with test step timing |


### What Is Autonomous Test Closure
URL: https://www.tofupilot.com/guides/what-is-autonomous-test-closure
Autonomous test closure uses AI to determine when a unit has been tested enough. Learn how it works, where it applies, and what data it needs.

Autonomous test closure is the concept that an AI system can determine when sufficient testing has been performed on a unit, and stop the test sequence early without human intervention. Instead of running every unit through every test step, the system evaluates test results in real time and decides: this unit has been tested enough, we're confident it's good. This guide covers how the concept works, where it applies, and what data it needs.

## The Problem: Over-Testing

Most manufacturing test sequences are static. Every unit runs through every test phase regardless of how clearly it's passing. The result: good units are tested longer than necessary.

| Scenario | Test Time | Units Per Hour |
|----------|----------|----------------|
| Full test (all phases) | 60 seconds | 60 |
| Early closure on high-confidence units | 35 seconds (average) | 103 |
| Improvement | 42% faster | 72% more throughput |

In semiconductor testing, this concept (called "adaptive test" or "predictive binning") has been deployed for over a decade, reducing test time by 10-50%. In discrete manufacturing, it's an emerging concept.

## How It Works

Autonomous test closure evaluates two things after each test phase:

1. **Confidence level**: Based on measurements so far, how confident are we this unit is good?
2. **Remaining risk**: What's the probability that a remaining test phase would catch a defect the previous phases missed?

| After Phase | Measurements So Far | Confidence | Decision |
|-------------|-------------------|-----------|----------|
| Phase 1: Power-up | Current within spec, voltage nominal | 70% | Continue |
| Phase 2: Communication | All buses responding correctly | 85% | Continue |
| Phase 3: Analog check | All channels within 2% of nominal | 97% | Close early (skip phases 4-5) |
| Phase 4: Stress test | (Skipped) | - | - |
| Phase 5: Final validation | (Skipped) | - | - |

The confidence threshold is configurable. Higher thresholds mean fewer tests are skipped. Lower thresholds mean faster throughput but higher risk.

## Prerequisites

| Requirement | Why |
|-------------|-----|
| Historical test data (10,000+ units) | Train the confidence model |
| Low correlation between skipped and kept tests | If phases are redundant, skipping is safe |
| Stable, mature product | New products need full test data |
| No regulatory requirement for 100% testing | Some industries can't skip tests |

## Where It Applies

| Applicable | Not Applicable |
|-----------|---------------|
| High-volume consumer electronics | Medical devices (FDA requires defined test protocol) |
| Mature products with stable yield | New product introduction (need all data) |
| Tests with redundant coverage | Safety-critical tests (hipot, leakage) |
| Products with high FPY (>97%) | Products with unstable processes |

## Levels of Autonomous Closure

| Level | How It Decides | Risk |
|-------|---------------|------|
| 1. Rule-based | "If phases 1-3 pass, skip phase 5" | Low (engineer defines rules) |
| 2. Statistical | "If all measurements are within 2-sigma, skip remaining" | Medium (data-driven threshold) |
| 3. ML-based | Trained model predicts pass probability from partial test data | Medium-high (model accuracy dependent) |
| 4. Fully autonomous | System continuously learns and adjusts skip decisions | High (requires robust safeguards) |

## Safeguards

Autonomous test closure increases throughput but introduces risk. Safeguards are essential:

| Safeguard | Purpose |
|-----------|---------|
| Audit sampling | Run full test on 5-10% of early-closed units |
| Escape monitoring | Track field returns for early-closed vs full-tested units |
| Confidence floor | Never close below a minimum confidence threshold |
| Phase lock-in | Some phases (safety, regulatory) can never be skipped |
| Automatic reversion | If audit failures increase, revert to full testing |

## The Data Foundation

Autonomous test closure depends on structured test data. Every measurement, every limit, every pass/fail result from every unit builds the dataset the confidence model learns from.

```python filename="test_with_data.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("supply_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("current_mA")
    .in_range(minimum=90, maximum=110)
    .with_units(units.MILLIAMPERE),
)
def phase_power(test):
    """Phase 1: Power-up check."""
    test.measurements.supply_voltage_V = 5.01
    test.measurements.current_mA = 99.5


@htf.measures(
    htf.Measurement("comm_status").equals("PASS"),
)
def phase_communication(test):
    """Phase 2: Communication check."""
    test.measurements.comm_status = "PASS"


@htf.measures(
    htf.Measurement("analog_ch1_V")
    .in_range(minimum=2.4, maximum=2.6)
    .with_units(units.VOLT),
)
def phase_analog(test):
    """Phase 3: Analog measurement. High-confidence units may close here."""
    test.measurements.analog_ch1_V = 2.50
```

```python filename="test_with_data.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_power,
    phase_communication,
    phase_analog,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

TofuPilot stores every measurement from every run. This structured dataset is what autonomous closure models need: thousands of units with complete measurement profiles and known pass/fail outcomes.

## The Path Forward

Autonomous test closure is at the intersection of adaptive testing, predictive quality, and AI. The technology exists in semiconductor testing. Applying it to discrete manufacturing requires:

1. **Structured test data** (measurements with units, limits, serial traceability)
2. **Historical depth** (thousands of units with complete test profiles)
3. **Correlation analysis** (which early phases predict overall pass/fail)
4. **Confidence modeling** (statistical or ML-based pass prediction)
5. **Safeguard infrastructure** (audit sampling, escape monitoring, automatic reversion)

Start by collecting the data. The intelligence comes later.

### Hipot and Dielectric Withstand Testing
URL: https://www.tofupilot.com/guides/hipot-and-dielectric-withstand-testing-with-tofupilot
Learn how to automate hipot and dielectric withstand safety tests, log results to TofuPilot, and maintain compliance records.

Hipot (high-potential) testing verifies that a product's electrical insulation can withstand voltage stress without breakdown. It's required by UL, IEC, and most safety standards for any product connected to mains power. TofuPilot logs every hipot result for production traceability and compliance.

## What Hipot Testing Verifies

Hipot testing applies a high voltage between isolated circuits (typically line-to-ground or primary-to-secondary) and measures the leakage current. If current stays below the threshold and no breakdown occurs, the insulation is adequate.

| Parameter | Typical value | Standard |
|-----------|--------------|----------|
| Test voltage (AC) | 1000-3000 Vrms | IEC 60950, IEC 62368 |
| Test voltage (DC) | 1414-4243 Vdc | IEC 60601 (medical) |
| Duration | 1-60 seconds | Varies by standard |
| Leakage current limit | 0.5-10 mA | Varies by product class |
| Ramp time | 0.5-3 seconds | Prevents voltage spikes |

## Automating Hipot Tests

### Connecting to a Hipot Tester

Most programmable hipot testers (Chroma, Associated Research, GW Instek) support SCPI commands over GPIB, RS-232, or LAN.

```python filename="hipot_test.py"
import pyvisa
from tofupilot import TofuPilotClient

rm = pyvisa.ResourceManager()
hipot = rm.open_resource("GPIB::5::INSTR")
client = TofuPilotClient()

def run_hipot_test(serial, test_voltage_v=1500, duration_s=60, leakage_limit_ma=5.0):
    # Configure hipot tester
    hipot.write(f"VOLT {test_voltage_v}")
    hipot.write(f"TIME {duration_s}")
    hipot.write(f"CURR:LIM {leakage_limit_ma}")
    hipot.write("RAMP 2")  # 2 second ramp

    # Run test
    hipot.write("TEST")
    hipot.query("*OPC?")  # Wait for completion

    # Read results
    result = hipot.query("MEAS:RES?")  # "PASS" or "FAIL"
    leakage = float(hipot.query("MEAS:CURR?")) * 1000  # Convert to mA

    passed = result.strip() == "PASS"

    client.create_run(
        procedure_id="HIPOT-SAFETY-TEST",
        unit_under_test={"serial_number": serial},
        run_passed=passed,
        steps=[{
            "name": "Dielectric Withstand",
            "step_type": "measurement",
            "status": passed,
            "measurements": [
                {"name": "test_voltage_v", "value": test_voltage_v, "unit": "V"},
                {"name": "duration_s", "value": duration_s, "unit": "s"},
                {"name": "leakage_current_ma", "value": leakage, "unit": "mA", "limit_high": leakage_limit_ma},
                {"name": "breakdown", "value": 0 if passed else 1, "unit": "bool", "limit_high": 0},
            ],
        }],
    )

    hipot.write("VOLT 0")  # Discharge
    return passed
```

### Multi-Point Hipot Testing

Some products require hipot tests between multiple isolation boundaries.

```python filename="multi_point_hipot.py"
# Test multiple isolation boundaries
isolation_tests = [
    {"name": "Line-to-Ground", "voltage": 1500, "limit_ma": 5.0},
    {"name": "Line-to-Secondary", "voltage": 3000, "limit_ma": 5.0},
    {"name": "Secondary-to-Ground", "voltage": 500, "limit_ma": 1.0},
]

steps = []
for test in isolation_tests:
    # Configure and run each test
    leakage = run_hipot_measurement(test["voltage"])
    passed = leakage < test["limit_ma"]

    steps.append({
        "name": test["name"],
        "step_type": "measurement",
        "status": passed,
        "measurements": [
            {"name": f"voltage_{test['name'].lower().replace('-', '_')}", "value": test["voltage"], "unit": "V"},
            {"name": f"leakage_{test['name'].lower().replace('-', '_')}", "value": leakage, "unit": "mA", "limit_high": test["limit_ma"]},
        ],
    })

all_pass = all(s["status"] for s in steps)
client.create_run(
    procedure_id="HIPOT-MULTI-POINT",
    unit_under_test={"serial_number": serial},
    run_passed=all_pass,
    steps=steps,
)
```

## Leakage Current Trending

Even for passing units, track leakage current over production. A healthy process shows consistent leakage values. Changes indicate:

| Trend | Possible cause |
|-------|---------------|
| Gradual increase | Contamination buildup on PCB, flux residue |
| Step increase | New board revision, different conformal coating |
| High variance | Inconsistent manufacturing (solder splash, debris) |
| Bimodal distribution | Two different PCB suppliers or manufacturing lines |

TofuPilot's measurement histogram for leakage current shows these patterns across your production.

## Compliance Records

Safety agencies require per-unit hipot test records. TofuPilot stores:

- Serial number of the tested unit
- Test voltage applied
- Duration of the test
- Measured leakage current
- Pass/fail result
- Timestamp
- Station/tester identification

When an auditor asks for hipot records for a specific serial number or date range, pull them from TofuPilot in seconds.

## Safety Considerations

Hipot testing involves lethal voltages. Always:

- Use proper safety interlocks on the test fixture
- Ensure the DUT is fully discharged after the test
- Never bypass safety mechanisms on the hipot tester
- Log test equipment calibration dates
- Train operators on hipot safety procedures

TofuPilot is the data management layer. The safety of the test setup is your responsibility.

## Ground Continuity Testing

Ground continuity is often tested alongside hipot. It verifies that the safety ground connection has low enough resistance to carry fault current.

```python filename="ground_continuity.py"
# Ground continuity test (typically 25A for 2 seconds)
ground_resistance = measure_ground_resistance(current_a=25, duration_s=2)

client.create_run(
    procedure_id="GROUND-CONTINUITY",
    unit_under_test={"serial_number": serial},
    run_passed=ground_resistance < 0.1,
    steps=[{
        "name": "Ground Bond",
        "step_type": "measurement",
        "status": ground_resistance < 0.1,
        "measurements": [
            {"name": "ground_resistance_ohm", "value": ground_resistance, "unit": "ohm", "limit_high": 0.1},
            {"name": "test_current_a", "value": 25, "unit": "A"},
        ],
    }],
)
```

Most safety test workflows run ground continuity first, then hipot. If the ground bond is bad, there's no point running the hipot test. TofuPilot links both results to the same serial number for a complete safety test record.


### Burn-In Testing for Electronics
URL: https://www.tofupilot.com/guides/burn-in-testing-for-electronics-a-complete-guide
Learn how to set up burn-in tests for electronic assemblies, log results to TofuPilot, and use burn-in data to catch infant mortality defects.

Burn-in testing operates products under stress for an extended period to catch infant mortality failures before they reach customers. It's the hardware equivalent of "let it run overnight and see if it crashes." TofuPilot tracks burn-in results alongside your other production tests for complete unit traceability.

## What Burn-In Testing Is

The bathtub curve describes electronic failure rates over time:

1. **Infant mortality** (early life): High failure rate, decreasing. Caused by manufacturing defects, weak solder joints, marginal components.
2. **Useful life**: Low, constant failure rate.
3. **Wear-out**: Increasing failure rate at end of life.

Burn-in targets the infant mortality region. By operating the product under stress for hours or days, you accelerate the aging process and force weak units to fail in the factory instead of in the field.

## Burn-In Parameters

| Parameter | Typical range | Purpose |
|-----------|--------------|---------|
| Temperature | 55-85°C (elevated) | Accelerate thermally-activated failures |
| Voltage | 10-20% above nominal | Stress power supply and regulators |
| Duration | 24-168 hours | Enough time for weak units to fail |
| Monitoring interval | Every 1-4 hours | Catch failures as they happen |
| Functional check | Before and after, plus periodic | Verify unit still works |

## Setting Up Burn-In with TofuPilot

### Pre-Burn-In Functional Test

Always run a functional test before burn-in to establish a baseline.

```python filename="burn_in_workflow.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

def pre_burn_in_test(serial):
    vcc = measure_voltage()
    current = measure_current()
    temp = measure_temperature()

    client.create_run(
        procedure_id="BURN-IN-PRE-CHECK",
        unit_under_test={"serial_number": serial},
        run_passed=True,
        steps=[{
            "name": "Pre-Burn-In Baseline",
            "step_type": "measurement",
            "status": True,
            "measurements": [
                {"name": "vcc_3v3", "value": vcc, "unit": "V", "limit_low": 3.25, "limit_high": 3.35},
                {"name": "current_ma", "value": current * 1000, "unit": "mA", "limit_low": 30, "limit_high": 60},
                {"name": "board_temp_c", "value": temp, "unit": "°C"},
            ],
        }],
    )
```

### Periodic Monitoring During Burn-In

Log measurements at regular intervals during the burn-in period.

```python filename="burn_in_monitor.py"
import time

def burn_in_monitor(serial, duration_hours=48, check_interval_hours=4):
    checks = int(duration_hours / check_interval_hours)

    for check in range(1, checks + 1):
        elapsed = check * check_interval_hours

        # Measure without removing unit from burn-in chamber
        vcc = measure_voltage()
        current = measure_current()
        temp = measure_temperature()

        passed = 3.20 <= vcc <= 3.40 and current * 1000 <= 80

        client.create_run(
            procedure_id="BURN-IN-MONITOR",
            unit_under_test={"serial_number": serial},
            run_passed=passed,
            steps=[{
                "name": f"Hour {elapsed} Check",
                "step_type": "measurement",
                "status": passed,
                "measurements": [
                    {"name": "hours_elapsed", "value": elapsed, "unit": "h"},
                    {"name": "vcc_3v3", "value": vcc, "unit": "V", "limit_low": 3.20, "limit_high": 3.40},
                    {"name": "current_ma", "value": current * 1000, "unit": "mA", "limit_high": 80},
                    {"name": "chamber_temp_c", "value": temp, "unit": "°C"},
                ],
            }],
        )

        if not passed:
            print(f"FAIL at hour {elapsed}: {serial}")
            return False

        time.sleep(check_interval_hours * 3600)

    return True
```

### Post-Burn-In Functional Test

After burn-in, run the same functional test as before. Compare pre and post measurements.

```python filename="post_burn_in.py"
def post_burn_in_test(serial):
    vcc = measure_voltage()
    current = measure_current()

    client.create_run(
        procedure_id="BURN-IN-POST-CHECK",
        unit_under_test={"serial_number": serial},
        run_passed=True,
        steps=[{
            "name": "Post-Burn-In Verification",
            "step_type": "measurement",
            "status": True,
            "measurements": [
                {"name": "vcc_3v3", "value": vcc, "unit": "V", "limit_low": 3.25, "limit_high": 3.35},
                {"name": "current_ma", "value": current * 1000, "unit": "mA", "limit_low": 30, "limit_high": 60},
            ],
        }],
    )
```

## Analyzing Burn-In Data

### Pre vs. Post Comparison

TofuPilot lets you compare pre-burn-in and post-burn-in measurements for the same unit. Look for:

| Change | What it means |
|--------|--------------|
| No change | Unit is stable, burn-in passed |
| Small shift (< 1%) | Normal thermal aging, acceptable |
| Large shift (> 5%) | Component degradation, investigate |
| Failure during burn-in | Infant mortality defect caught |

### Burn-In Failure Rate Tracking

Track the percentage of units that fail during burn-in over time.

| Month | Units burned in | Failures | Burn-in failure rate |
|-------|----------------|----------|---------------------|
| Jan | 1,000 | 8 | 0.8% |
| Feb | 1,200 | 12 | 1.0% |
| Mar | 1,100 | 25 | 2.3% |

If the burn-in failure rate increases, something changed in your manufacturing process. The burn-in is doing its job by catching these defects, but you need to find and fix the root cause.

### When to Skip Burn-In

Burn-in costs time and money. Not every product needs it.

| Factor | Burn-in recommended | Burn-in optional |
|--------|--------------------|-----------------| 
| Safety-critical product | Yes | |
| High field failure cost | Yes | |
| Mature, high-yield process | | Yes |
| Consumer electronics (cost-sensitive) | | Yes |
| Medical/aerospace/defense | Yes | |

Use TofuPilot data to make this decision. If burn-in catches 2% of units and field failure costs $5,000 per incident, burn-in pays for itself. If burn-in catches 0.01% and field failure costs $50, it doesn't.

## Burn-In Rack Monitoring

For high-volume burn-in, you may have racks with dozens of units burning in simultaneously. Monitor them all through TofuPilot.

Each slot in the rack uploads periodic measurements. The dashboard shows:
- Which slots currently have units
- Current temperature and power consumption per slot
- Any failures that occurred during the burn-in period
- Historical burn-in yield per rack position (catches rack-specific issues like bad power connections)

If rack position 12 has a higher failure rate than other positions, the rack needs maintenance, not the product.


### Build a Device History Record
URL: https://www.tofupilot.com/guides/how-to-build-a-device-history-record-with-tofupilot
Learn how to compile FDA-compliant Device History Records using TofuPilot's per-unit test data, traceability, and structured measurement storage.

The Device History Record (DHR) is a mandatory document for FDA-regulated medical devices. It's the complete production record for each finished device: what was built, how it was tested, and what the results were. TofuPilot provides the test data component of the DHR automatically.

## What the FDA Requires

21 CFR 820.184 (Device History Record) requires documentation of:

| DHR component | What it includes | TofuPilot's role |
|--------------|-----------------|------------------|
| Production dates | When the device was manufactured | Test run timestamps |
| Quantity manufactured | How many in the lot/batch | Run count by batch |
| Acceptance records | Test results showing conformity | All measurements with limits |
| Primary identification label | Serial number, UDI | Unit under test serial |
| Labeling | Labels used on the device | (Outside TofuPilot scope) |
| Equipment used | Test stations, calibration status | Station identification |

## Test Data as DHR Evidence

Every test run in TofuPilot is a DHR evidence record. For each device serial number, TofuPilot stores:

```python filename="dhr_test_record.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# ICT (In-Circuit Test)
client.create_run(
    procedure_id="ICT-MEDICAL-BOARD",
    unit_under_test={
        "serial_number": "MD-2025-00421",
        "part_number": "CARDIAC-MONITOR-PCB-R5",
    },
    run_passed=True,
    steps=[{
        "name": "Component Verification",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "r1_resistance_kohm", "value": 10.02, "unit": "kohm", "limit_low": 9.5, "limit_high": 10.5},
            {"name": "c1_capacitance_uf", "value": 4.72, "unit": "uF", "limit_low": 4.23, "limit_high": 5.17},
            {"name": "u1_continuity", "value": 1, "unit": "bool", "limit_low": 1},
        ],
    }],
)

# Functional test
client.create_run(
    procedure_id="FUNC-CARDIAC-MONITOR",
    unit_under_test={"serial_number": "MD-2025-00421"},
    run_passed=True,
    steps=[{
        "name": "ECG Signal Chain",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "ecg_gain_db", "value": 60.2, "unit": "dB", "limit_low": 59.0, "limit_high": 61.0},
            {"name": "ecg_cmrr_db", "value": 112, "unit": "dB", "limit_low": 100},
            {"name": "ecg_noise_uv_rms", "value": 8.3, "unit": "uV", "limit_high": 15},
        ],
    }],
)

# Electrical safety
client.create_run(
    procedure_id="SAFETY-IEC60601",
    unit_under_test={"serial_number": "MD-2025-00421"},
    run_passed=True,
    steps=[{
        "name": "Patient Leakage",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "patient_leakage_normal_ua", "value": 4.2, "unit": "uA", "limit_high": 10},
            {"name": "patient_leakage_sfc_ua", "value": 22.1, "unit": "uA", "limit_high": 50},
        ],
    }],
)

# Final inspection
client.create_run(
    procedure_id="FINAL-INSPECTION",
    unit_under_test={"serial_number": "MD-2025-00421"},
    run_passed=True,
    steps=[{
        "name": "Cosmetic and Label Check",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "label_present", "value": 1, "unit": "bool", "limit_low": 1},
            {"name": "cosmetic_pass", "value": 1, "unit": "bool", "limit_low": 1},
            {"name": "udi_barcode_readable", "value": 1, "unit": "bool", "limit_low": 1},
        ],
    }],
)
```

## Compiling the DHR

Search by serial number in TofuPilot to retrieve all test records for a device.

For device `MD-2025-00421`, the DHR test section shows:

| Test | Date | Result | Station |
|------|------|--------|---------|
| ICT-MEDICAL-BOARD | 2025-03-15 09:22 | PASS | STN-ICT-02 |
| FUNC-CARDIAC-MONITOR | 2025-03-15 10:15 | PASS | STN-FUNC-01 |
| SAFETY-IEC60601 | 2025-03-15 11:30 | PASS | STN-SAFETY-01 |
| FINAL-INSPECTION | 2025-03-15 14:00 | PASS | STN-FINAL-01 |

Each row links to the complete measurement data: every value, every limit, every pass/fail status.

## Handling Nonconformities in the DHR

When a device fails a test and is reworked, the DHR must show:

1. The original failure (which measurement, what value)
2. The disposition decision (rework, scrap, use-as-is)
3. The retest result after rework

TofuPilot captures #1 and #3 automatically. For #2, document the disposition in your QMS and cross-reference the TofuPilot run ID.

```python filename="dhr_retest.py"
# Original test failed
# Run ID: run-001 (patient leakage at 12.3 uA, limit 10 uA)

# After rework (replaced connector), retest
client.create_run(
    procedure_id="SAFETY-IEC60601-RETEST",
    unit_under_test={"serial_number": "MD-2025-00421"},
    run_passed=True,
    steps=[{
        "name": "Patient Leakage (Post-Rework)",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "patient_leakage_normal_ua", "value": 3.8, "unit": "uA", "limit_high": 10},
        ],
    }],
)
```

The DHR for this device now shows: original fail, rework disposition (from QMS), and passing retest.

## Batch DHR Reports

For batch release, pull all devices in a production lot and verify complete test records.

```python filename="batch_dhr_check.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Required procedures for a complete DHR
required_procedures = [
    "ICT-MEDICAL-BOARD",
    "FUNC-CARDIAC-MONITOR",
    "SAFETY-IEC60601",
    "FINAL-INSPECTION",
]

# Check all devices in the batch
batch_serials = [f"MD-2025-{i:05d}" for i in range(400, 450)]

incomplete = []
for serial in batch_serials:
    runs = client.get_runs(unit_serial=serial)
    completed_procedures = {r["procedure_id"] for r in runs if r["run_passed"]}
    missing = set(required_procedures) - completed_procedures
    if missing:
        incomplete.append({"serial": serial, "missing": missing})

if incomplete:
    print(f"{len(incomplete)} devices have incomplete DHRs:")
    for d in incomplete:
        print(f"  {d['serial']}: missing {d['missing']}")
else:
    print("All devices have complete DHRs. Batch ready for release.")
```

## Retention and Retrieval

FDA requires DHR retention for the lifetime of the device plus additional years. TofuPilot stores all test data with timestamps and audit trails. Configure your data retention policy to match your regulatory requirements.

When FDA asks for a specific device's DHR during an inspection, search by serial number and export the complete test history. Structured, timestamped, immutable records replace paper binders and Excel files.

## Integration with Your QMS

TofuPilot handles the test data component of the DHR. Your QMS (quality management system) handles:

- Bill of materials and component traceability
- Manufacturing work orders
- Rework and disposition records
- Labeling and packaging records
- Release and distribution records

Together, TofuPilot and your QMS provide the complete DHR. The key is using consistent serial numbers across both systems so records can be cross-referenced.


### Track Battery Cycling Test Data
URL: https://www.tofupilot.com/guides/how-to-track-battery-cycling-test-data-with-tofupilot
Learn how to log charge/discharge cycling data, track capacity fade, and monitor cell performance across production using TofuPilot.

Battery cycling tests generate thousands of data points per cell: voltage curves, current profiles, temperature readings, capacity measurements across hundreds of cycles. TofuPilot stores all of it in a structured, queryable format so you can track capacity fade, compare cells, and catch quality issues before they reach the pack.

## What Battery Cycling Data Looks Like

A single cell cycling test produces:

| Data point | Per cycle | Over 500 cycles |
|-----------|-----------|-----------------|
| Voltage vs. time | ~1000 samples | 500,000 samples |
| Current vs. time | ~1000 samples | 500,000 samples |
| Temperature | ~100 samples | 50,000 samples |
| Charge capacity (Ah) | 1 value | 500 values |
| Discharge capacity (Ah) | 1 value | 500 values |
| Coulombic efficiency | 1 value | 500 values |

Multiply that by hundreds or thousands of cells in production, and you're looking at a data management problem that spreadsheets can't handle.

## Logging Cycling Data to TofuPilot

### Per-Cycle Upload

Upload results after each charge/discharge cycle completes. This gives you real-time visibility into cell performance.

```python filename="battery_cycle_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

def log_cycle(cell_serial, cycle_number, charge_ah, discharge_ah, temp_max, voltage_curve):
    coulombic_eff = discharge_ah / charge_ah * 100 if charge_ah > 0 else 0
    capacity_retention = discharge_ah / nominal_capacity * 100

    client.create_run(
        procedure_id=f"CELL-CYCLING-C{cycle_number}",
        unit_under_test={"serial_number": cell_serial},
        run_passed=capacity_retention > 80 and temp_max < 45,
        steps=[{
            "name": f"Cycle {cycle_number}",
            "step_type": "measurement",
            "status": capacity_retention > 80,
            "measurements": [
                {"name": "charge_capacity_ah", "value": charge_ah, "unit": "Ah", "limit_low": 2.8},
                {"name": "discharge_capacity_ah", "value": discharge_ah, "unit": "Ah", "limit_low": 2.8},
                {"name": "coulombic_efficiency_pct", "value": coulombic_eff, "unit": "%", "limit_low": 99.0},
                {"name": "capacity_retention_pct", "value": capacity_retention, "unit": "%", "limit_low": 80.0},
                {"name": "max_temperature_c", "value": temp_max, "unit": "°C", "limit_high": 45.0},
                {"name": "voltage_curve_v", "value": voltage_curve, "unit": "V"},
            ],
        }],
    )
```

### End-of-Life Summary Upload

After cycling completes, upload a summary with key aging metrics.

```python filename="battery_eol_summary.py"
client.create_run(
    procedure_id="CELL-CYCLING-SUMMARY",
    unit_under_test={"serial_number": cell_serial},
    run_passed=final_capacity_retention > 80,
    steps=[{
        "name": "Cycling Summary",
        "step_type": "measurement",
        "status": final_capacity_retention > 80,
        "measurements": [
            {"name": "total_cycles", "value": 500, "unit": "cycles"},
            {"name": "initial_capacity_ah", "value": 3.2, "unit": "Ah"},
            {"name": "final_capacity_ah", "value": 2.72, "unit": "Ah"},
            {"name": "capacity_retention_pct", "value": 85.0, "unit": "%", "limit_low": 80.0},
            {"name": "avg_coulombic_efficiency", "value": 99.7, "unit": "%", "limit_low": 99.0},
            {"name": "max_temperature_observed", "value": 42.3, "unit": "°C", "limit_high": 45.0},
        ],
    }],
)
```

## Tracking Capacity Fade Across Production

The real value of centralized cycling data is comparing cells across production batches. TofuPilot's measurement trending shows:

- **Capacity retention distribution**: Are all cells aging at the same rate?
- **Batch-to-batch variation**: Does the new electrolyte formulation change the fade curve?
- **Outlier detection**: Which cells are degrading faster than expected?

If batch 47 cells show 90% retention at 300 cycles while batch 46 showed 94%, something changed in the manufacturing process. The data in TofuPilot tells you immediately.

## Cell Grading from Cycling Data

Not all cells are created equal. Cycling data helps grade cells into bins for different applications.

| Grade | Criteria | Application |
|-------|----------|-------------|
| A | Capacity > 3.1 Ah, retention > 95% at 200 cycles | EV packs |
| B | Capacity > 2.9 Ah, retention > 90% at 200 cycles | Energy storage |
| C | Below Grade B | Second-life applications |

TofuPilot's measurement filters let you query cells by any combination of cycling metrics to assign grades automatically.

## Integration with Battery Cyclers

Most battery cyclers (Arbin, Maccor, Neware, BioLogic) export data in CSV or proprietary formats. Parse the cycler output and upload to TofuPilot.

```python filename="arbin_import.py"
import csv
from tofupilot import TofuPilotClient

client = TofuPilotClient()

def import_arbin_data(csv_path, cell_serial):
    with open(csv_path) as f:
        reader = csv.DictReader(f)
        for row in reader:
            if row["Step_Type"] == "Discharge":
                client.create_run(
                    procedure_id="CELL-CYCLING",
                    unit_under_test={"serial_number": cell_serial},
                    run_passed=float(row["Discharge_Capacity(Ah)"]) > 2.8,
                    steps=[{
                        "name": f"Cycle {row['Cycle_Index']}",
                        "step_type": "measurement",
                        "status": True,
                        "measurements": [
                            {"name": "discharge_capacity_ah", "value": float(row["Discharge_Capacity(Ah)"]), "unit": "Ah"},
                            {"name": "charge_capacity_ah", "value": float(row["Charge_Capacity(Ah)"]), "unit": "Ah"},
                            {"name": "max_voltage_v", "value": float(row["Voltage(V)"]), "unit": "V"},
                        ],
                    }],
                )
```

## Safety Monitoring

Battery testing has unique safety requirements. TofuPilot helps track safety-critical measurements:

- **Maximum temperature**: Cells approaching thermal runaway thresholds
- **Voltage anomalies**: Cells that don't reach full charge voltage or drop too fast
- **Capacity jumps**: Sudden capacity changes that indicate internal shorts
- **Impedance growth**: Rising internal resistance indicating degradation

Set tight limits on these safety measurements. A cell that passes capacity specs but shows abnormal temperature behavior needs investigation before it goes into a pack.


### What Is Predictive Quality in Mfg Test
URL: https://www.tofupilot.com/guides/what-is-predictive-quality-in-manufacturing-test
Predictive quality uses production data to catch defects before they happen. Learn how it works, what data it needs, and how test results feed prediction.

Predictive quality uses historical test data and process data to forecast defects before they occur. Instead of catching bad units at end-of-line testing, predictive quality identifies the conditions that produce bad units and flags them earlier in the process. This guide covers how predictive quality works, what data it needs, and how structured test results feed prediction models.

## Reactive vs Predictive Quality

| Approach | When Defects Are Found | Cost |
|----------|----------------------|------|
| Reactive (inspect and reject) | After the unit is built | High (scrap, rework, field returns) |
| Statistical (SPC) | When a process drifts out of control | Medium (catches trends, not individual units) |
| Predictive (ML-based) | Before the defect occurs | Low (prevents defective units from being built) |

Traditional quality control catches defects after they happen. SPC catches trends before they produce defects. Predictive quality goes further: it uses patterns in upstream data to predict which specific units or batches will fail downstream tests.

## How Predictive Quality Works

The core idea: upstream test data contains signals that correlate with downstream failures.

| Step | What Happens |
|------|-------------|
| 1. Collect data | Every measurement, at every test stage, for every unit |
| 2. Find correlations | ML models identify which upstream measurements predict downstream failures |
| 3. Build prediction model | Train a model on historical pass/fail data with upstream features |
| 4. Deploy inline | Run predictions on live test data as units move through production |
| 5. Act on predictions | Flag high-risk units for additional inspection or route to rework |

### Example

A power supply fails output ripple testing at end-of-line. Analysis of historical data shows that units with input capacitor ESR above 45 milliohms at incoming inspection are 8x more likely to fail ripple testing. A predictive model flags these units at IQC before they're assembled.

## What Data Feeds Predictive Quality

| Data Source | What It Provides | Stage |
|------------|-----------------|-------|
| Incoming quality (IQC) | Component measurements, supplier lot data | Before assembly |
| In-process quality (IPQC) | SPI paste volume, AOI defect counts, reflow profile | During assembly |
| Functional test (FCT) | Electrical measurements with limits | After assembly |
| End-of-line test (EOL) | Final pass/fail, measurement values | Before shipping |
| Environmental data | Temperature, humidity on the production floor | Continuous |
| Equipment data | Fixture cycle count, instrument calibration age | Continuous |

The more structured your test data, the better the predictions. Measurements with units, limits, and serial number traceability are the foundation.

## Predictive Quality Use Cases

| Use Case | Input Data | Prediction | Benefit |
|----------|-----------|-----------|---------|
| Skip testing | Upstream measurements | Unit will pass downstream test | 10-50% test time reduction |
| Early warning | Process trends | Batch will have high failure rate | Catch before full batch is built |
| Supplier quality | IQC measurements by vendor | Lot will cause downstream failures | Reject lots at incoming |
| Limit optimization | Measurement distributions | Current limits are too wide or too tight | Reduce false failures and escapes |
| Field failure prediction | Production test data | Unit will fail within warranty period | Tighten limits or add screening |

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Capture Structured Test Data

Predictive quality starts with clean, consistent test data. Every measurement needs a name, value, unit, and limits.

```python filename="production_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("input_capacitor_esr_mOhm")
    .in_range(maximum=50)
    .with_units(units.OHM),
    htf.Measurement("output_ripple_mV")
    .in_range(maximum=30)
    .with_units(units.MILLIVOLT),
    htf.Measurement("efficiency_percent")
    .in_range(minimum=90)
    .with_units(units.PERCENT),
)
def phase_electrical_test(test):
    """Capture measurements that feed predictive models."""
    test.measurements.input_capacitor_esr_mOhm = 38.2
    test.measurements.output_ripple_mV = 22.1
    test.measurements.efficiency_percent = 93.4
```

## Step 2: Log Everything to TofuPilot

Every run uploads to TofuPilot with serial number, measurements, limits, and pass/fail status. This structured data is what predictive models need.

```python filename="production_test.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_electrical_test)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Step 3: Use TofuPilot Data for Analysis

TofuPilot tracks measurement distributions, correlations, and trends across all test stages. Open the Analytics tab to identify:

- **Measurement correlations** between upstream and downstream tests
- **Failure patterns** by supplier lot, station, or time period
- **Distribution shifts** that precede failure rate increases
- **Marginal results** that predict future failures

This data is the starting point for building predictive models. The structured format (measurements with units, limits, and serial traceability) eliminates the data cleaning step that typically consumes 80% of ML project time.

## Predictive Quality vs Traditional Quality

| Aspect | Traditional | Predictive |
|--------|------------|-----------|
| When defects are found | After they happen | Before they happen |
| Test strategy | Test everything the same way | Adapt testing based on risk |
| Data use | Compliance reporting | Real-time decision making |
| Limit setting | From datasheet or engineering judgment | From production data and ML models |
| ROI measurement | Defect rate reduction | Cost avoidance (prevented scrap, skipped tests) |

## Getting Started

Predictive quality doesn't require a massive ML infrastructure on day one. The progression is:

| Level | What You Do | What You Need |
|-------|------------|--------------|
| 1. Collect | Log all measurements with units and limits | OpenHTF + TofuPilot |
| 2. Visualize | Review distributions, correlations, and trends | TofuPilot Analytics |
| 3. Correlate | Identify upstream predictors of downstream failures | Export data, run correlation analysis |
| 4. Predict | Build and deploy ML models on live data | Data science team + production data |
| 5. Automate | Act on predictions inline (skip tests, flag units) | Model deployment infrastructure |

Most teams get significant value from levels 1-3 alone. Just visualizing the correlation between incoming component measurements and end-of-line failures reveals actionable insights without building any ML models.

### What Is End-of-Line Testing
URL: https://www.tofupilot.com/guides/what-is-end-of-line-testing-with-tofupilot
End-of-line testing is the last gate before a product ships. What it covers, how it differs from FCT, and how to structure it for cycle time.

End-of-line (EOL) test is the last verification a product receives before it is packed. Everything upstream tested parts of it; EOL tests the thing the customer will actually receive, fully assembled, in its enclosure, behaving as a product rather than as a board.

## What EOL Covers

| Area | Typical checks |
|------|---------------|
| Power-on behaviour | Boots, correct current draw, no fault indications |
| Functional | Every feature the datasheet claims, exercised |
| Safety | Hipot, ground bond, leakage where applicable |
| Calibration | Final coefficients measured and written |
| Interfaces | Connectors, buttons, displays, LEDs, audio |
| Wireless | RF power, sensitivity, pairing, antenna presence |
| Sensors | Readings against a reference, in the final enclosure |
| Identity | Serial number, MAC, firmware version, labels |
| Cosmetic | Scratches, gaps, label placement |

The enclosure matters more than it looks. A board that passed FCT can fail EOL because an antenna is detuned by the case, a sensor reads differently once enclosed, or a connector does not align with its cutout. Those are precisely the defects EOL exists to find.

## EOL vs FCT

| | FCT | EOL |
|--|-----|-----|
| **Unit** | Bare PCBA | Fully assembled product |
| **Tests** | Board-level function | Product-level function |
| **Catches** | Assembly and component faults | Integration, mechanical, enclosure faults |
| **Access** | Test points, pogo fixture | Product connectors only |
| **Cycle time** | Seconds to a minute | Often minutes |
| **Rework** | Cheap; board is accessible | Expensive; disassembly required |

The access difference drives the design. At EOL you generally cannot probe a test point, so the test talks to the product the way a customer would: over USB, the console, or the radio. This is why firmware self-test hooks matter so much at this stage.

## Cycle Time Is the Constraint

EOL usually sits at the end of a line with a takt time, and a test that overruns it becomes the bottleneck for the whole factory. Two structural levers matter.

**Run independent checks concurrently.** Most EOL checks do not depend on each other. Declaring them as independent phases lets them run in parallel rather than in sequence.

**Test several units at once.** When the fixture holds more than one product, fixture slots test them concurrently, each producing its own run and outcome.

```yaml filename="procedure.yaml"
name: Product EOL Test
version: 1.0.0
description: End-of-line test for the assembled product

unit:
  serial_number:
    default_value: "PRD000001"
  part_number:
    default_value: "PRD-100"

plugs:
  - name: fixture
    python: plugs.fixture:Fixture
  - name: dut
    python: plugs.dut:Dut
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Safety
    python: phases.safety
    measurements:
      - name: Enclosure Closed
        validators:
          - operator: "=="
            expected_value: true

  - name: Power On
    python: phases.power_on
    depends_on: [safety]
    measurements:
      - name: Inrush Current
        unit: A
        validators:
          - operator: "<="
            expected_value: 1.5
      - name: Boot Time
        unit: ms
        validators:
          - operator: "<="
            expected_value: 3000

  - name: Identity
    python: phases.identity
    depends_on: [power_on]
    measurements:
      - name: Firmware Version
        validators:
          - operator: "in"
            expected_value: ["2.1.0", "2.1.1"]
      - name: Serial Format
        validators:
          - operator: "matches"
            expected_value: "^PRD[0-9]{6}$"

  - name: Sensors
    python: phases.sensors
    depends_on: [power_on]
    measurements:
      - name: Temperature Reading
        unit: "°C"
        description: Against the fixture reference, in the enclosure
        validators:
          - operator: ">="
            expected_value: 20.0
          - operator: "<="
            expected_value: 30.0

  - name: Interfaces
    python: phases.interfaces
    depends_on: [power_on]
    measurements:
      - name: Button Response
        validators:
          - operator: "=="
            expected_value: true
      - name: LED Check
        validators:
          - operator: "=="
            expected_value: true

  - name: Wireless
    python: phases.wireless
    depends_on: [power_on]
    measurements:
      - name: TX Power
        unit: dBm
        validators:
          - operator: ">="
            expected_value: 8.0
          - operator: "<="
            expected_value: 14.0
```

Identity, sensors, interfaces and wireless all depend on power-on but not on each other, so the four run concurrently and the phase group costs the longest of them rather than their sum.

```python filename="phases/power_on.py"
def power_on(measurements, fixture, dut, dmm):
    fixture.power_on()
    measurements.inrush_current = dmm.peak_current()
    measurements.boot_time = dut.wait_for_boot_ms()
```

```python filename="phases/identity.py"
def identity(measurements, dut):
    measurements.firmware_version = dut.firmware_version()
    measurements.serial_format = dut.serial_number()
```

```python filename="phases/sensors.py"
def sensors(measurements, dut):
    measurements.temperature_reading = dut.read_temperature()
```

Run it with `tofupilot run ./procedure.yaml`.

For a multi-up fixture:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: Position 1
    - name: Position 2
    - name: Position 3
    - name: Position 4
```

A failure in one position stops that position and leaves the others testing.

## Firmware Self-Test Carries EOL

Because probe access is gone, the product has to report on itself. A structured self-test command is the highest-value EOL feature a firmware team can provide:

```python filename="plugs/dut.py"
import json


class Dut:
    def selftest(self) -> dict:
        raw = self._query("selftest --json")
        return json.loads(raw)

    def _query(self, cmd: str) -> str:
        return '{"flash_ok": true, "imu_ok": true, "adc_counts": 2048}'
```

```python filename="phases/selftest.py"
def selftest(measurements, dut):
    result = dut.selftest()
    measurements.flash_ok = result["flash_ok"]
    measurements.imu_ok = result["imu_ok"]
    measurements.adc_counts = result["adc_counts"]
```

JSON rather than prose, so a firmware wording change does not break production.

## Calibration at EOL

Final calibration usually belongs here, because the sensor is in its final mechanical and thermal environment. Record both the raw measurement and the coefficient written:

```python filename="phases/calibrate.py"
def calibrate(measurements, dut, fixture):
    reference = fixture.reference_temperature()
    raw = dut.read_raw_temperature()
    offset = reference - raw

    measurements.reference_temperature = reference
    measurements.raw_temperature = raw
    measurements.temperature_offset = offset

    dut.write_calibration(offset)
```

Storing the coefficient per serial means a field complaint can be checked against what was actually programmed into that unit.

## Labels and Identity

EOL is where the product gets its identity, and where mismatches are cheapest to catch. Verify that what is printed matches what is programmed:

```python filename="phases/label_check.py"
def label_check(measurements, dut, scanner):
    programmed = dut.serial_number()
    printed = scanner.read_label()
    measurements.label_matches = programmed == printed
```

A unit whose label and firmware disagree is untraceable for its entire life. This check costs one scan.

## What EOL Cannot Do

EOL is a short test on a finished product. It does not find defects that take time to appear, which is what [burn-in](/guides/burn-in-testing-for-electronics-a-complete-guide) and [ongoing reliability testing](/guides/what-is-ort-with-tofupilot) are for. A product can pass EOL and fail at 400 hours, and no amount of EOL coverage changes that.

It also cannot diagnose at component level. An EOL failure usually means disassembly and a trip back to a board-level test, which is why upstream coverage still matters.

## If You Already Run OpenHTF

```python filename="eol.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("boot_time_ms")
    .in_range(maximum=3000)
    .with_units(units.MILLISECOND),
    htf.Measurement("firmware_version").equals("2.1.0"),
)
def phase_power_on(test):
    test.measurements.boot_time_ms = 1850
    test.measurements.firmware_version = "2.1.0"


test = htf.Test(phase_power_on, part_number="PRD-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan product serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Sequential phases on a takt-time line | EOL becomes the bottleneck | Declare independent phases; use slots |
| Assuming FCT coverage is enough | Enclosure and integration faults ship | Test the assembled product as a product |
| Firmware self-test printing prose | Production breaks on a wording change | Structured JSON output |
| Not verifying label against firmware | Untraceable units in the field | Scan and compare |
| Calibration before final assembly | Coefficients wrong in the real environment | Calibrate at EOL |
| Treating EOL as a reliability test | Time-dependent failures still escape | Add burn-in or ORT |

### What Is an AI-Native Test Station
URL: https://www.tofupilot.com/guides/what-is-an-ai-native-test-station
An AI-native test station is built around data and inference from the start, not bolted on after. Learn what it means and how it changes manufacturing test.

An AI-native test station is designed from the ground up with data collection, real-time analytics, and machine learning as core capabilities, not as add-ons. The difference is architectural: instead of running tests and exporting CSVs for offline analysis, an AI-native station streams structured data, learns from every run, and adapts its behavior based on what the data shows.

## Traditional vs AI-Native

| Aspect | Traditional Station | AI-Native Station |
|--------|-------------------|-------------------|
| Data flow | Test runs, results saved to local file or database | Test runs, results stream to analytics platform in real time |
| Data format | CSV, proprietary binary, or unstructured logs | Structured measurements with units, limits, and metadata |
| Analytics | Offline, batch processing, manual export | Real-time dashboards, automated trend detection |
| Limit management | Hard-coded in test script, changed manually | Informed by production data, adjusted based on distributions |
| Failure response | Engineer reviews logs after the fact | Immediate alerts, automated root cause suggestions |
| Cross-station learning | Each station is isolated | All stations share data, models improve from fleet-wide patterns |
| Test sequence | Static, same for every unit | Can adapt based on upstream data or risk score |

## The Three Pillars

### 1. Structured Data by Default

Every measurement has a name, value, unit, and limits. Every run has a serial number, timestamp, station ID, and pass/fail result. This isn't optional. It's the foundation.

```python filename="ai_native_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("supply_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("boot_time_ms")
    .in_range(maximum=2000)
    .with_units(units.MILLISECOND),
)
def phase_functional_check(test):
    """Every measurement is structured and traceable."""
    test.measurements.supply_voltage_V = 5.01
    test.measurements.boot_time_ms = 1280
```

Without structured data, AI has nothing to learn from. The most common reason AI projects fail in manufacturing is not bad algorithms. It's bad data.

### 2. Real-Time Streaming

Test results flow to a central platform as they happen, not at the end of the shift or when someone remembers to export.

```python filename="ai_native_test.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_functional_check)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Real-time streaming enables:
- Immediate yield alerts when failures spike
- Live measurement distributions across all stations
- Cross-station comparison to detect fixture or equipment issues
- Operator UI that shows results as they happen

### 3. Feedback Loops

The station learns from its own data. Historical results inform future decisions:

| Feedback Loop | What It Does |
|--------------|-------------|
| Limit refinement | Production data shows the real distribution, limits get tightened or relaxed |
| Failure prioritization | Pareto analysis ranks which failures matter most |
| Station health monitoring | Throughput and yield trends detect equipment degradation |
| Measurement drift detection | Control charts flag when a parameter starts trending |

## What Changes in Practice

### For Test Engineers

| Before | After |
|--------|-------|
| Write test, deploy, forget | Write test, deploy, monitor, iterate |
| Set limits from datasheet | Set initial limits, refine from production data |
| Debug failures from logs | Query failure patterns across thousands of runs |
| Optimize one station at a time | Compare performance across all stations instantly |

### For Operators

| Before | After |
|--------|-------|
| Run test, read pass/fail on terminal | Run test, see results on streaming operator UI |
| Report failures verbally | Failures are logged automatically with full context |
| No visibility into trends | Dashboard shows yield and throughput in real time |

### For Quality Engineers

| Before | After |
|--------|-------|
| Export CSVs, build reports in Excel | Reports generated from live data |
| Monthly quality reviews with stale data | Real-time quality dashboards |
| Root cause analysis takes weeks | Failure correlations visible immediately |

## Building an AI-Native Station

You don't need to buy new hardware. An AI-native station is a software architecture choice:

| Component | Traditional | AI-Native |
|-----------|-----------|-----------|
| Test framework | Any (OpenHTF, pytest, custom) | Same, but with structured measurements |
| Data backend | Local files, local database | Cloud or self-hosted analytics platform |
| Operator interface | Terminal or custom GUI | Web-based streaming UI |
| Analytics | Excel, manual reports | Automated dashboards, alerts, trend detection |
| Integration | Point-to-point, custom scripts | API-based, standard data formats |

The key insight: making a station AI-native is not about adding AI features. It's about structuring the data and infrastructure so that AI features become possible. Get the data right, and the intelligence follows.

### Write a Functional Test Spec for PCBA
URL: https://www.tofupilot.com/guides/how-to-write-a-functional-test-spec-for-pcba-with-tofupilot
Learn how to build a functional test specification for PCBA production, map schematic nets to measurements with datasheet-derived limits, and translate to.

A functional test specification defines every measurement your board must pass before it ships. It maps schematic nets to physical test points, sets pass/fail limits from datasheets, and becomes the contract between hardware design and production test. This guide shows you how to build that document and turn it into executable OpenHTF phases tracked in TofuPilot.

## What a Functional Test Spec Is

A functional test spec is a structured table that answers three questions for every signal on your board:

1. What are you measuring? (net name, test point, measurement type)
2. What is the expected value? (nominal from schematic or datasheet)
3. What is the acceptable range? (min/max limits with tolerance)

It is not a test script. It is the source of truth that drives the test script.

## Mapping Schematic Nets to Test Points

Start with your schematic and export a net list. For each net that carries a testable signal, identify the physical access point on the board.

| Net Name | Test Point | Signal Type | Nominal | Notes |
|----------|-----------|------------|---------|-------|
| VCC_3V3 | TP1 | DC voltage | 3.3 V | LDO output |
| VCC_5V0 | TP2 | DC voltage | 5.0 V | USB VBUS |
| XTAL_OUT | TP3 | Frequency | 12 MHz | Crystal oscillator |
| I2C_SDA | TP4 | Logic high | 3.3 V | Pull-up to VCC_3V3 |
| VBAT | TP5 | DC voltage | 3.7 V | Li-ion nominal |
| LED_PWM | TP6 | Duty cycle | 50% | Default firmware state |

Rules for choosing test points:

- Prefer dedicated test vias over component pads
- Include a ground reference TP near each measurement cluster
- Number test points in physical order to reduce fixture wiring complexity

## Defining Limits from Datasheets

Limits come from two sources: component datasheets and system-level margins. Trace every limit to a document and a line item.

### Voltage Regulators

For an LDO like the TLV1117-3.3, the datasheet specifies:

- Output voltage accuracy: +/-1% over temperature
- Line regulation: 0.2% typical
- Load regulation: 0.4% typical

At 3.3 V nominal with +/-1% accuracy plus 0.4% load regulation:

| Net | Nominal | Min | Max | Source |
|-----|---------|-----|-----|--------|
| VCC_3V3 | 3.300 V | 3.234 V | 3.366 V | TLV1117-3.3 DS, Table 6.6 |

### Crystal Oscillators

A 12 MHz crystal with +/-20 ppm tolerance at 25 C:

- 20 ppm of 12 MHz = 240 Hz

| Net | Nominal | Min | Max | Source |
|-----|---------|-----|-----|--------|
| XTAL_OUT | 12 000 000 Hz | 11 999 760 Hz | 12 000 240 Hz | ABM8 DS, ppm spec |

### Logic Signals

Use the component's VIH and VIL thresholds, not 50% of VCC.

| Net | Type | Min | Max | Source |
|-----|------|-----|-----|--------|
| I2C_SDA (high) | V_OH | 2.97 V | 3.63 V | GPIO spec, 0.9*VCC to 1.1*VCC |
| I2C_SCL (low) | V_OL | 0 V | 0.33 V | GPIO spec, V_OL max 0.1*VCC |

## Test Specification Document Structure

Store the spec as a versioned table. One row per measurement.

| ID | Test Point | Net | Phase Name | Measurement Type | Nominal | Min | Max | Unit | Datasheet Ref |
|----|-----------|-----|-----------|-----------------|---------|-----|-----|------|---------------|
| M01 | TP1 | VCC_3V3 | power_rails | DC voltage | 3.300 | 3.234 | 3.366 | V | TLV1117-3.3 DS Table 6.6 |
| M02 | TP2 | VCC_5V0 | power_rails | DC voltage | 5.000 | 4.750 | 5.250 | V | USB 2.0 spec section 7.2.1 |
| M03 | TP3 | XTAL_OUT | clock_check | Frequency | 12000000 | 11999760 | 12000240 | Hz | ABM8 DS ppm spec |
| M04 | TP4 | I2C_SDA | i2c_bus | DC voltage | 3.300 | 2.970 | 3.630 | V | MCU GPIO spec |
| M05 | TP5 | VBAT | battery_voltage | DC voltage | 3.700 | 3.000 | 4.200 | V | Li-ion cell spec |
| M06 | TP6 | LED_PWM | led_driver | Duty cycle | 50.0 | 45.0 | 55.0 | % | Firmware requirement |

Keep this table in a CSV or spreadsheet under version control alongside the firmware. Tag each spec revision to the hardware revision it applies to.

## Translating the Spec into OpenHTF Phases

Each phase corresponds to a functional block in the spec. Measurements within a phase map to individual rows by ID.

```python filename="tests/pcba_functional_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

from plugs.dmm import DmmPlug
from plugs.counter import FrequencyCounterPlug


@htf.plug(dmm=DmmPlug)
@htf.measures(
    htf.Measurement("vcc_3v3_voltage")
    .in_range(minimum=3.234, maximum=3.366)
    .with_units(units.VOLT)
    .doc("M01: VCC_3V3 at TP1, TLV1117-3.3 DS Table 6.6"),
    htf.Measurement("vcc_5v0_voltage")
    .in_range(minimum=4.750, maximum=5.250)
    .with_units(units.VOLT)
    .doc("M02: VCC_5V0 at TP2, USB 2.0 spec 7.2.1"),
)
def power_rails(test, dmm):
    test.measurements.vcc_3v3_voltage = dmm.measure_dc_voltage(channel=1)
    test.measurements.vcc_5v0_voltage = dmm.measure_dc_voltage(channel=2)


@htf.plug(counter=FrequencyCounterPlug)
@htf.measures(
    htf.Measurement("xtal_frequency")
    .in_range(minimum=11_999_760, maximum=12_000_240)
    .with_units(units.HERTZ)
    .doc("M03: XTAL_OUT at TP3, ABM8 DS ppm spec"),
)
def clock_check(test, counter):
    test.measurements.xtal_frequency = counter.measure_frequency(channel=1)


@htf.plug(dmm=DmmPlug)
@htf.measures(
    htf.Measurement("i2c_sda_high_voltage")
    .in_range(minimum=2.970, maximum=3.630)
    .with_units(units.VOLT)
    .doc("M04: I2C_SDA at TP4, MCU GPIO spec"),
)
def i2c_bus(test, dmm):
    test.measurements.i2c_sda_high_voltage = dmm.measure_dc_voltage(channel=3)


@htf.plug(dmm=DmmPlug)
@htf.measures(
    htf.Measurement("vbat_voltage")
    .in_range(minimum=3.000, maximum=4.200)
    .with_units(units.VOLT)
    .doc("M05: VBAT at TP5, Li-ion cell spec"),
)
def battery_voltage(test, dmm):
    test.measurements.vbat_voltage = dmm.measure_dc_voltage(channel=4)


@htf.plug(counter=FrequencyCounterPlug)
@htf.measures(
    htf.Measurement("led_pwm_duty_cycle")
    .in_range(minimum=45.0, maximum=55.0)
    .doc("M06: LED_PWM at TP6, firmware requirement"),
)
def led_driver(test, counter):
    test.measurements.led_pwm_duty_cycle = counter.measure_duty_cycle(channel=2)


def main():
    test = htf.Test(
        power_rails,
        clock_check,
        i2c_bus,
        battery_voltage,
        led_driver,
        test_name="PCBA Functional Test",
    )

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Enter DUT serial number: "))


if __name__ == "__main__":
    main()
```

The `.doc()` string on each measurement links it back to the spec ID, making the test output self-auditing. When a measurement fails in TofuPilot, you can trace it directly to the datasheet row that defined the limit.

## Tracking Spec Coverage in TofuPilot

After your first run, cross-reference the spec table against the TofuPilot measurement list:

| Spec ID | Measurement Name | TofuPilot Status |
|---------|-----------------|-----------------|
| M01 | vcc_3v3_voltage | present |
| M02 | vcc_5v0_voltage | present |
| M03 | xtal_frequency | present |
| M04 | i2c_sda_high_voltage | present |
| M05 | vbat_voltage | present |
| M06 | led_pwm_duty_cycle | present |

A measurement that never fails across hundreds of units may indicate limits that are too loose. Review the distribution in TofuPilot's measurement analytics and tighten limits if the histogram shows all values clustered far from the edges.

### Updating Limits

When hardware revision B changes the LDO, update the spec table and code together:

```python filename="tests/pcba_functional_test.py"
from openhtf.util import units

# AP2112K-3.3: Vout accuracy +/-1.5%, load reg 0.3%
# Min: 3.300 * (1 - 0.015 - 0.003) = 3.2406
# Max: 3.300 * (1 + 0.015 + 0.003) = 3.3594
htf.Measurement("vcc_3v3_voltage")
    .in_range(minimum=3.2406, maximum=3.3594)
    .with_units(units.VOLT)
    .doc("M01: VCC_3V3 at TP1, AP2112K-3.3 DS rev B"),
```

Commit the spec table update and the code change in the same commit so the audit trail stays intact.

### Sensor Calibration at Scale
URL: https://www.tofupilot.com/guides/sensor-calibration-at-scale-with-tofupilot
Learn how to automate sensor calibration workflows, track calibration certificates, and manage multi-point calibration data with TofuPilot.

Calibrating 10 sensors is a manual job. Calibrating 10,000 requires a system. You need multi-point reference measurements, pass/fail against tolerance bands, calibration certificates, and traceability back to reference standards.

TofuPilot handles the data side so you can focus on the calibration procedure itself.

## What Sensor Calibration Involves

Production sensor calibration typically follows this flow:

1. Apply known reference stimuli (temperature, pressure, force, etc.)
2. Read the sensor output at each reference point
3. Calculate error, linearity, and hysteresis
4. Apply correction factors if needed
5. Verify corrected output meets specifications
6. Generate a calibration certificate

## Prerequisites

- Python 3.8+ with `openhtf` and `tofupilot` installed
- A reference standard (calibrated source or reference sensor)
- A data acquisition system or instrument to read sensor output

## Step 1: Define Multi-Point Calibration Measurements

A typical calibration uses 5 to 11 reference points across the sensor's range. Define measurements for each point:

```python filename="sensor_cal.py"
import openhtf as htf
from openhtf.util import units

CAL_POINTS = [0, 25, 50, 75, 100]  # Percent of full scale

measures = []
for pct in CAL_POINTS:
    measures.append(
        htf.Measurement(f"error_at_{pct}pct")
            .with_units(units.PERCENT)
            .in_range(-0.5, 0.5)
            .doc(f"Measurement error at {pct}% of full scale")
    )

measures.append(
    htf.Measurement("max_linearity_error")
        .with_units(units.PERCENT)
        .at_most(0.25)
        .doc("Maximum linearity deviation from best-fit line")
)

measures.append(
    htf.Measurement("hysteresis")
        .with_units(units.PERCENT)
        .at_most(0.1)
        .doc("Maximum hysteresis between up and down sweep")
)

@htf.measures(*measures)
def calibration_test(test, reference_source, sensor_reader):
    """Run multi-point calibration with up and down sweep."""
    full_scale = 100.0  # Adjust for your sensor range
    up_readings = {}
    down_readings = {}

    # Up sweep
    for pct in CAL_POINTS:
        ref_value = full_scale * pct / 100.0
        reference_source.set_output(ref_value)
        time.sleep(2.0)  # Settle time

        reading = sensor_reader.read()
        up_readings[pct] = reading
        error = (reading - ref_value) / full_scale * 100
        setattr(test.measurements, f"error_at_{pct}pct", error)

    # Down sweep for hysteresis
    for pct in reversed(CAL_POINTS):
        ref_value = full_scale * pct / 100.0
        reference_source.set_output(ref_value)
        time.sleep(2.0)

        reading = sensor_reader.read()
        down_readings[pct] = reading

    # Hysteresis: max difference between up and down readings
    max_hyst = max(
        abs(up_readings[p] - down_readings[p]) / full_scale * 100
        for p in CAL_POINTS
    )
    test.measurements.hysteresis = max_hyst

    # Linearity: deviation from best-fit line
    import numpy as np
    ref_vals = [full_scale * p / 100.0 for p in CAL_POINTS]
    read_vals = [up_readings[p] for p in CAL_POINTS]
    coeffs = np.polyfit(ref_vals, read_vals, 1)
    fit_vals = np.polyval(coeffs, ref_vals)
    linearity_errors = [(r - f) / full_scale * 100 for r, f in zip(read_vals, fit_vals)]
    test.measurements.max_linearity_error = max(abs(e) for e in linearity_errors)
```

## Step 2: Store Calibration Curves as Multi-Dimensional Data

Capture the full calibration curve, not just pass/fail:

```python filename="cal_curve_capture.py"
import openhtf as htf
from openhtf.util import units

@htf.measures(
    htf.Measurement("calibration_curve")
        .with_dimensions(units.PERCENT)
        .doc("Full calibration curve: reference input vs sensor output"),
    htf.Measurement("correction_curve")
        .with_dimensions(units.PERCENT)
        .doc("Correction factors at each calibration point"),
)
def capture_calibration_curve(test, reference_source, sensor_reader):
    """Capture dense calibration curve for post-processing."""
    full_scale = 100.0

    for pct in range(0, 101, 5):  # 5% increments
        ref_value = full_scale * pct / 100.0
        reference_source.set_output(ref_value)
        time.sleep(1.0)

        reading = sensor_reader.read()
        test.measurements.calibration_curve[pct] = reading

        # Correction factor: what to add to get the true value
        correction = ref_value - reading
        test.measurements.correction_curve[pct] = correction
```

TofuPilot stores the full curve. You can compare curves across sensors, detect batch variations, and track calibration drift over time.

## Step 3: Track Reference Standard Traceability

Every calibration is only as good as its reference. Track which reference standard was used:

```python filename="cal_traceability.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

result = client.create_run(
    procedure_id="pressure-sensor-cal",
    unit_under_test={
        "serial_number": sensor_sn,
        "part_number": "PS-500-A",
    },
    run_passed=True,
    properties={
        "reference_standard": "FLUKE-8845A-SN12345",
        "reference_cal_date": "2026-01-15",
        "reference_cal_due": "2027-01-15",
        "reference_cert_number": "CAL-2026-0042",
        "ambient_temp_c": 23.1,
        "ambient_humidity_pct": 45,
        "operator": "OP-003",
    },
)
```

When an auditor asks about your traceability chain, every calibration run links to the reference standard, its own calibration certificate, and the environmental conditions during the procedure.

## Step 4: Manage Recalibration Schedules

Sensors drift. Track when each unit was last calibrated and when it's due:

```python filename="recal_tracking.py"
from tofupilot import TofuPilotClient
from datetime import date, timedelta

client = TofuPilotClient()

# Get all calibration runs for a sensor type
cal_runs = client.get_runs(
    procedure_id="pressure-sensor-cal",
    limit=5000,
)

# Find latest calibration per serial number
latest_cal = {}
for run in cal_runs:
    sn = run.unit.serial_number
    if sn not in latest_cal or run.started_at > latest_cal[sn].started_at:
        latest_cal[sn] = run

# Check for overdue recalibrations (12-month interval)
cal_interval = timedelta(days=365)
today = date.today()

overdue = []
upcoming = []

for sn, run in latest_cal.items():
    cal_date = run.started_at.date()
    due_date = cal_date + cal_interval

    if due_date < today:
        overdue.append((sn, due_date))
    elif due_date < today + timedelta(days=30):
        upcoming.append((sn, due_date))

print(f"Overdue: {len(overdue)} sensors")
print(f"Due within 30 days: {len(upcoming)} sensors")
```

## Step 5: Detect Calibration Drift

Compare calibration results over time to catch sensors that are drifting toward their tolerance limits:

```python filename="drift_detection.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get calibration history for a specific sensor
cal_history = client.get_runs(
    procedure_id="pressure-sensor-cal",
    unit_serial_number="PS-500-A-0042",
    limit=10,
)

print(f"Calibration history for PS-500-A-0042:")
print(f"{'Date':<12} {'Error@50%':<12} {'Linearity':<12} {'Status'}")
print("-" * 48)

for run in reversed(cal_history):
    error_50 = run.measurements.get("error_at_50pct", {}).get("value", "N/A")
    linearity = run.measurements.get("max_linearity_error", {}).get("value", "N/A")
    status = "PASS" if run.passed else "FAIL"
    print(f"{run.started_at.date()!s:<12} {error_50:<12} {linearity:<12} {status}")
```

If the error at 50% is 0.1% this year and was 0.05% last year, you know which direction it's heading. Replace or adjust before it fails.


### Design an End-of-Line Operator Screen
URL: https://www.tofupilot.com/guides/how-to-design-an-end-of-line-test-operator-screen
An EOL operator screen must be fast, clear, and error-proof. Learn how to design the interface for end-of-line test stations.

An end-of-line (EOL) test operator runs the same test hundreds of times per shift. The screen they see must communicate three things instantly: what to do, whether it passed, and what to do next. This guide covers design principles for EOL operator screens and how to implement them with OpenHTF and TofuPilot.

## Design Principles

### 1. The 3-Second Rule

An operator should understand the screen state within 3 seconds of looking at it. This means:

| Element | Requirement |
|---------|------------|
| Pass/fail indicator | Full-screen green or red background, visible from 3 meters |
| Current step | Large text showing what's happening now |
| Operator action | Bold instruction when input is needed |
| Everything else | Secondary, smaller, or hidden |

### 2. Minimize Decisions

Every decision the operator makes is a chance for error. Reduce them:

| Bad | Better |
|-----|--------|
| Type the serial number | Scan the barcode |
| Choose the product variant | Auto-detect from barcode prefix |
| Select the test procedure | One station, one procedure |
| Click "Start Test" | Auto-start after barcode scan |
| Click "Print Label" | Auto-print on pass |

### 3. Error-Proof the Workflow

| Risk | Mitigation |
|------|-----------|
| Wrong serial format | Validate barcode format before starting |
| Testing the same unit twice | Warn if serial was already tested today |
| Skipping the test | Lock packaging station until test passes |
| Missing a failure | Audible alert on fail, require acknowledgment |

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Design the Test Flow

An EOL test should follow this sequence:

```
Scan barcode → Auto-start → Run phases → Show result → Ready for next unit
```

No menus, no configuration, no extra clicks.

```python filename="eol_operator.py"
import openhtf as htf
from openhtf.util import units
from openhtf import PhaseResult


@htf.measures(
    htf.Measurement("supply_current_mA")
    .in_range(minimum=90, maximum=110)
    .with_units(units.MILLIAMPERE),
)
def phase_power_up(test):
    """Apply power and verify current draw."""
    test.measurements.supply_current_mA = 101.2


@htf.measures(
    htf.Measurement("firmware_version").equals("3.1.0"),
)
def phase_firmware(test):
    """Check firmware version matches production release."""
    test.measurements.firmware_version = "3.1.0"


@htf.measures(
    htf.Measurement("self_test").equals("PASS"),
)
def phase_self_test(test):
    """Run the DUT's built-in self-test."""
    test.measurements.self_test = "PASS"


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=11.8, maximum=12.2)
    .with_units(units.VOLT),
)
def phase_output_check(test):
    """Measure main output voltage."""
    test.measurements.output_voltage_V = 12.03
```

## Step 2: Fail Fast

Put the most likely failure first. If power-up fails, skip everything else. The operator sees the failure in under 5 seconds instead of waiting for the full test cycle.

```python filename="eol_operator.py"
@htf.measures(
    htf.Measurement("power_good").equals("PASS"),
)
def phase_power_good(test):
    """Quick power check before running full test."""
    result = "PASS"
    test.measurements.power_good = result
    if result != "PASS":
        return PhaseResult.STOP
```

## Step 3: Connect and Stream

TofuPilot's operator UI handles the display. The operator sees each phase progress, measurements with limits, and a full-screen pass/fail result.

```python filename="eol_operator.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_power_good,
    phase_power_up,
    phase_firmware,
    phase_self_test,
    phase_output_check,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Screen Layout Recommendations

| Zone | Content | Size |
|------|---------|------|
| Top bar | Station name, current time, units tested today | Small |
| Center | Current phase name and status, or pass/fail result | Large (60% of screen) |
| Prompt area | Operator instruction when input is needed | Medium, highlighted |
| Bottom | Last 5 results (pass/fail per serial) | Small, scrolling |

### Color Coding

| State | Color | Meaning |
|-------|-------|---------|
| Idle | Gray or blue | Waiting for barcode scan |
| Running | Blue | Test in progress |
| Pass | Green | Unit passed all phases |
| Fail | Red | Unit failed one or more phases |
| Prompt | Yellow/amber | Waiting for operator input |

## Cycle Time Optimization for EOL

EOL tests run continuously. Every second of cycle time matters.

| Optimization | Effect |
|-------------|--------|
| Auto-start after scan | Saves 2-3 seconds per unit |
| No unnecessary prompts | Saves 5-10 seconds per prompt |
| Fail-fast phase ordering | Reduces average cycle time on bad units by 50%+ |
| Parallel instrument commands | Saves 100-500ms per phase |
| Kiosk mode (no browser chrome) | Prevents accidental navigation |

## What to Avoid

| Mistake | Why |
|---------|-----|
| Small pass/fail text | Operator must lean in to read it |
| Detailed measurement tables during test | Distracts from the current step |
| Multiple windows or tabs | Operator gets lost |
| Login screens | Operators share stations across shifts |
| Dark theme in bright factory lighting | Low contrast, hard to read |
| Animations or transitions | Slow down perception of state changes |

### Track Repair and Rework Data
URL: https://www.tofupilot.com/guides/how-to-track-repair-and-rework-data-with-tofupilot
Learn how to structure OpenHTF tests so repair loops, rework actions, and retests all link to the same serial number in TofuPilot.

Production testing catches defects, but the real value comes from closing the loop: diagnosing failures, repairing units, and retesting them. Every retest links to the original serial number, so you get a complete history of each unit's journey through your repair process.

## The Repair Loop

A typical repair workflow follows this cycle: a unit fails a test, a technician diagnoses the root cause, performs a repair, and the unit goes back through testing. Without structured data, this history gets lost in spreadsheets or paper logs.

The fix is to key everything to the serial number. Every test run against the same DUT automatically appears in its unit history. You don't need special configuration. Just use the same serial number when retesting.

## Structure Your Initial Test

Start with a standard OpenHTF test that measures your DUT and uploads results.

```python filename="functional_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("output_voltage")
    .in_range(minimum=4.8, maximum=5.2)
    .with_units(units.VOLT),
    htf.Measurement("current_draw")
    .in_range(minimum=0.095, maximum=0.105)
    .with_units(units.AMPERE),
)
def functional_check(test):
    test.measurements.output_voltage = 5.05
    test.measurements.current_draw = 0.1012


def main():
    test = htf.Test(
        functional_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-20260312-001")


if __name__ == "__main__":
    main()
```

When this test fails, the unit enters your repair queue.

## Record Repair Actions as Measurements

After a technician diagnoses and repairs the unit, capture that context in the retest. A dedicated phase records what was found and what was done.

```python filename="retest_after_repair.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("repair_code"),
    htf.Measurement("failure_category"),
    htf.Measurement("repair_action"),
)
def record_repair_info(test):
    # These values come from your repair technician's input
    test.measurements.repair_code = "RC-042"
    test.measurements.failure_category = "solder_bridge"
    test.measurements.repair_action = "reworked_U3_solder_joints"


@htf.measures(
    htf.Measurement("output_voltage")
    .in_range(minimum=4.8, maximum=5.2)
    .with_units(units.VOLT),
    htf.Measurement("current_draw")
    .in_range(minimum=0.095, maximum=0.105)
    .with_units(units.AMPERE),
)
def functional_recheck(test):
    test.measurements.output_voltage = 5.01
    test.measurements.current_draw = 0.1003


def main():
    test = htf.Test(
        record_repair_info,
        functional_recheck,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    # Same serial number links this retest to the original failure
    test.execute(lambda: "SN-20260312-001")


if __name__ == "__main__":
    main()
```

The `record_repair_info` phase stores the diagnosis and repair action alongside the retest measurements, tied to the same unit.

In practice the technician types these values rather than hardcoding them. Prompt for them with `openhtf.plugs.user_input` so the operator picks from your category list instead of typing free text, which is what keeps the data aggregatable later.

## Use Failure Categories Consistently

Define a standard set of failure categories and repair codes across your team. Consistent naming lets you filter and aggregate repair data later.

Common failure categories for PCBA testing:

| Category | Description |
|----------|-------------|
| `solder_bridge` | Unintended solder connection between pads |
| `cold_joint` | Insufficient solder wetting |
| `missing_component` | Component not placed during assembly |
| `wrong_value` | Incorrect component value populated |
| `damaged_component` | Component damaged during handling or reflow |
| `pcb_defect` | Board-level issue (trace crack, via failure) |

Store these as string measurements so they stay searchable.

A short list beats a thorough one. Categories nobody can tell apart get filled in at random, and the Pareto chart built from them is then worse than no chart at all.

## Track Multiple Repair Cycles

Some units need more than one repair attempt. Each retest creates a new run against the same serial number, and the unit history shows the full chain: initial fail, first repair attempt, second repair attempt, and final pass.

```python filename="second_repair_retest.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("repair_code"),
    htf.Measurement("failure_category"),
    htf.Measurement("repair_action"),
    htf.Measurement("repair_cycle"),
)
def record_repair_info(test):
    test.measurements.repair_code = "RC-043"
    test.measurements.failure_category = "cold_joint"
    test.measurements.repair_action = "reflowed_C12_pads"
    test.measurements.repair_cycle = 2


@htf.measures(
    htf.Measurement("output_voltage")
    .in_range(minimum=4.8, maximum=5.2)
    .with_units(units.VOLT),
    htf.Measurement("current_draw")
    .in_range(minimum=0.095, maximum=0.105)
    .with_units(units.AMPERE),
)
def functional_recheck(test):
    test.measurements.output_voltage = 4.98
    test.measurements.current_draw = 0.0998


def main():
    test = htf.Test(
        record_repair_info,
        functional_recheck,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-20260312-001")


if __name__ == "__main__":
    main()
```

Adding a `repair_cycle` measurement makes it easy to count how many attempts each unit needed. Set a threshold where a unit stops being repaired and gets scrapped: a board on its fourth rework has usually absorbed more technician time than it is worth, and repeated reflow damages the laminate.

## Analyze Repair Trends

Once repair data flows in, you can answer the questions that matter without writing analysis scripts.

**Unit history** shows every test run for a serial number in chronological order. You can see exactly when a unit failed, what was repaired, and whether the retest passed.

**Failure Pareto charts** rank your failure categories by frequency. If `solder_bridge` dominates, that's a signal to investigate your reflow profile or stencil design.

**First pass yield trends** reflect your repair effectiveness over time. A rising FPY after process changes confirms the fix is working. A unit that keeps failing the same test after multiple repairs may point to a deeper design issue.

Watch the gap between first pass yield and final yield. A widening gap means the process is degrading while rework absorbs it, which looks fine on a final-yield chart right up until the rework capacity runs out.

## Separate Procedures for Initial Test and Retest

For traceability, consider distinct procedures for initial tests and retests. Runs group by procedure, so this separation makes reporting cleaner.

Each procedure has its own UUID from the dashboard, so pointing a script at the retest procedure is a matter of changing `procedure_id`:

```python filename="retest_procedure.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

RETEST_PROCEDURE_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"  # "PCBA Functional Test - Retest"


@htf.measures(
    htf.Measurement("repair_code"),
    htf.Measurement("repair_action"),
)
def record_repair_info(test):
    test.measurements.repair_code = "RC-042"
    test.measurements.repair_action = "reworked_U3_solder_joints"


@htf.measures(
    htf.Measurement("output_voltage")
    .in_range(minimum=4.8, maximum=5.2)
    .with_units(units.VOLT),
)
def functional_recheck(test):
    test.measurements.output_voltage = 5.02


def main():
    test = htf.Test(
        record_repair_info,
        functional_recheck,
        procedure_id=RETEST_PROCEDURE_ID,
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-20260312-001")


if __name__ == "__main__":
    main()
```

This way you can compare yield between initial tests and retests independently, while the unit history still ties everything together under one serial number.

The tradeoff: splitting procedures means first pass yield on the initial procedure no longer counts the units that eventually passed on retest. That is usually what you want, but say which number you mean when reporting it.

### Test Fixture Design and Management
URL: https://www.tofupilot.com/guides/test-fixture-design-and-management-with-tofupilot
Learn how to track test fixture health, calibration, and lifecycle using TofuPilot's station monitoring and measurement trending.

The test fixture is the most underappreciated part of hardware test infrastructure. A worn pogo pin or a loose cable causes more false failures than bad product. TofuPilot helps you monitor fixture health through measurement trends and station-specific analytics.

## What Makes a Test Fixture

A test fixture connects the DUT (device under test) to the test instruments. It typically includes:

| Component | Purpose | Failure mode |
|-----------|---------|-------------|
| Pogo pins / spring probes | Electrical contact to DUT | Wear, contamination, broken springs |
| Alignment hardware | Position DUT consistently | Wear, loosening over time |
| Cable harness | Connect fixture to instruments | Broken wires, loose connectors |
| Pneumatic actuator | Clamp DUT against probes | Seal failure, pressure loss |
| Guide pins | Align PCB to probe positions | Wear, bending |

## Monitoring Fixture Health Through Test Data

You don't need dedicated fixture tests. Your production test data already contains fixture health signals.

### Contact Resistance as a Health Indicator

If your test includes a continuity or resistance measurement, track it per station.

```python filename="fixture_health.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# This measurement doubles as a DUT test AND a fixture health indicator
client.create_run(
    procedure_id="BOARD-FUNCTIONAL",
    unit_under_test={"serial_number": serial},
    run_passed=True,
    steps=[{
        "name": "Contact Verification",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "probe_contact_mohm", "value": 35, "unit": "mohm", "limit_high": 100},
        ],
    }],
)
```

Track `probe_contact_mohm` per station over time:

| Week | Station 1 | Station 2 | Station 3 |
|------|-----------|-----------|-----------|
| 1 | 25 mohm | 28 mohm | 30 mohm |
| 4 | 28 mohm | 30 mohm | 45 mohm |
| 8 | 30 mohm | 32 mohm | 72 mohm |
| 12 | 32 mohm | 35 mohm | 95 mohm |

Station 3's contact resistance is climbing fast. Service the pogo pins before it hits the 100 mohm limit and starts causing false failures.

### Measurement Variance as a Health Indicator

Even when measurements pass, increasing variance signals fixture problems.

A voltage measurement that bounces between 3.29V and 3.33V has more variance than one that reads 3.31V consistently. High variance on a specific station means something is intermittent: a loose wire, a dirty probe, or a worn alignment pin.

### Cycle Time as a Health Indicator

If a test normally takes 42 seconds and starts taking 55 seconds on one station, something changed:

- Pneumatic actuator slowing down (low air pressure, seal wear)
- Instrument communication timeout (loose cable, aging connector)
- Retry loops from intermittent contacts

## Fixture Lifecycle Tracking

### Maintenance Schedule Based on Data

Replace calendar-based maintenance with data-driven maintenance.

| Approach | Schedule | Basis |
|----------|----------|-------|
| Calendar-based | Every 3 months | Time elapsed |
| Count-based | Every 50,000 cycles | Insertion count |
| Condition-based | When data shows degradation | Measurement trends |

Condition-based maintenance costs less and catches problems earlier. TofuPilot's measurement trends tell you which fixtures need attention now and which are fine.

### Tracking Pogo Pin Life

Pogo pins have a rated cycle life (typically 100,000-1,000,000 cycles). Track cycles per fixture.

```python filename="fixture_tracking.py"
# Track fixture cycle count alongside DUT test
client.create_run(
    procedure_id="BOARD-FUNCTIONAL",
    unit_under_test={"serial_number": serial},
    run_passed=True,
    steps=[{
        "name": "Contact Verification",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {"name": "probe_contact_mohm", "value": contact_r, "unit": "mohm", "limit_high": 100},
            {"name": "fixture_cycle_count", "value": get_fixture_count(), "unit": "cycles"},
        ],
    }],
)
```

When contact resistance starts rising, check the cycle count. If you're at 80% of rated life, schedule replacement during the next planned downtime.

## Fixture Design Best Practices

| Practice | Why |
|----------|-----|
| Use keyed connectors | Prevent incorrect cable connections |
| Include a "golden board" test point | Verify fixture health independently from DUT |
| Minimize cable length | Reduces noise, improves measurement accuracy |
| Use separate ground pins | Don't rely on one ground contact for the whole fixture |
| Design for probe replacement | Make pogo pins individually replaceable |
| Add alignment features | Ensure consistent DUT positioning |
| Include interlock sensors | Detect when DUT is properly seated |

### Golden Board Verification

A "golden board" is a known-good reference board. Run it through the fixture periodically to verify the fixture is working correctly.

```python filename="golden_board_check.py"
# Weekly golden board verification
GOLDEN_SERIAL = "GOLDEN-001"
EXPECTED_VCC = 3.310  # Known value from initial characterization

vcc = measure_voltage()
drift = abs(vcc - EXPECTED_VCC)

client.create_run(
    procedure_id="FIXTURE-GOLDEN-BOARD-CHECK",
    unit_under_test={"serial_number": GOLDEN_SERIAL},
    run_passed=drift < 0.010,  # Less than 10mV drift
    steps=[{
        "name": "Golden Board Verification",
        "step_type": "measurement",
        "status": drift < 0.010,
        "measurements": [
            {"name": "vcc_3v3", "value": vcc, "unit": "V", "limit_low": 3.300, "limit_high": 3.320},
            {"name": "drift_from_reference_mv", "value": drift * 1000, "unit": "mV", "limit_high": 10},
        ],
    }],
)
```

If the golden board measurement drifts, the fixture or instrument has changed, not the board.

## Common Fixture Failures and Detection

| Failure | Detection in TofuPilot | Fix |
|---------|----------------------|-----|
| Worn pogo pins | Rising contact resistance trend | Replace affected pins |
| Loose cable | Intermittent measurement failures (random, no pattern) | Reseat or replace cable |
| Alignment wear | Increasing measurement variance across all channels | Replace guide pins |
| Contamination | Gradual measurement shift, contact resistance increase | Clean probes and PCB contacts |
| Pneumatic leak | Longer cycle times, intermittent contacts | Replace seals |

## Multi-Fixture Management

For production lines with multiple fixtures for the same test:

1. Give each fixture a unique identifier
2. Track which fixture ran each test (station ID in TofuPilot)
3. Compare performance across fixtures
4. Rotate golden board checks across all fixtures

If Fixture A has 98% yield and Fixture B has 94% yield on the same product, Fixture B needs attention. TofuPilot's station comparison view shows this at a glance.


### Calculate ROI of Test Automation
URL: https://www.tofupilot.com/guides/how-to-calculate-roi-of-test-automation-with-tofupilot
Learn how to build a business case for hardware test automation and measure ROI using TofuPilot's yield, cycle time, and defect data.

Test automation costs money upfront: fixtures, instruments, software, integration time. But it saves money continuously: fewer field failures, faster throughput, less manual labor. TofuPilot gives you the data to quantify both sides of the equation.

## The ROI Framework

```
ROI = (Annual Savings - Annual Cost) / Initial Investment × 100%
```

The challenge isn't the formula. It's getting accurate numbers for savings and costs. TofuPilot provides the data for the savings side.

## Cost of NOT Automating

### Manual Testing Costs

| Cost item | Manual test | Automated test |
|-----------|------------|----------------|
| Operator time per unit | 15-30 min | 1-3 min |
| Data recording | Manual entry (error-prone) | Automatic |
| Report generation | 3-4 hours/week | Zero (live dashboard) |
| Debug time per failure | 2-4 hours | 15-30 min |
| Operator training | Weeks | Days |

### Field Failure Costs

This is usually the biggest number. A field failure costs 10-100x more than catching the same defect in the factory.

| Stage where defect is caught | Typical cost |
|------------------------------|-------------|
| In-circuit test (ICT) | $1-5 per unit |
| Functional test | $5-20 per unit |
| Burn-in / ESS | $10-50 per unit |
| Customer site (warranty) | $100-5,000 per unit |
| Safety recall | $10,000+ per unit |

## Measuring Savings with TofuPilot Data

### Defect Escape Rate

Track how many defects your test catches vs. how many reach customers.

```python filename="escape_rate.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get production test data
runs = client.get_runs(
    procedure_id="FINAL-FUNCTIONAL",
    limit=10000,
)

total = len(runs)
caught = sum(1 for r in runs if not r["run_passed"])
catch_rate = caught / total * 100

print(f"Units tested: {total}")
print(f"Defects caught: {caught}")
print(f"Catch rate: {catch_rate:.2f}%")
print(f"Estimated escapes (if no test): {caught} defects reaching customers")
print(f"Field failure cost avoided: ${caught * 500:,.0f}")  # $500 per field failure
```

### Yield Improvement Value

Track FPY over time. Every percentage point of yield improvement has a dollar value.

```
Before TofuPilot: 94% FPY
After TofuPilot: 97% FPY
Improvement: 3 percentage points

Annual production: 50,000 units
Units saved from rework: 50,000 × 0.03 = 1,500 units
Rework cost per unit: $25
Annual rework savings: 1,500 × $25 = $37,500
```

### Debug Time Reduction

Track how long it takes to diagnose failures before and after TofuPilot.

| Metric | Before | After | Savings |
|--------|--------|-------|---------|
| Avg debug time per failure | 3 hours | 0.5 hours | 2.5 hours |
| Failures per week | 20 | 20 | - |
| Weekly debug hours | 60 hours | 10 hours | 50 hours |
| Engineer cost per hour | $75 | $75 | - |
| Weekly savings | - | - | $3,750 |
| Annual savings | - | - | $195,000 |

### Report Time Elimination

| Metric | Before | After | Savings |
|--------|--------|-------|---------|
| Weekly report preparation | 4 hours | 0 hours | 4 hours |
| Monthly quality review prep | 8 hours | 0 hours | 8 hours |
| Annual report hours saved | 304 hours | 0 hours | 304 hours |
| Cost saved | - | - | $22,800 |

## Building the Business Case

### Costs (Year 1)

| Item | Cost |
|------|------|
| Test fixtures (2 stations) | $20,000 |
| Test instruments | $30,000 |
| TofuPilot subscription | $X,000/year |
| Integration engineering (2 weeks) | $15,000 |
| **Total Year 1** | **$65,000 + subscription** |

### Annual Savings

| Savings category | Amount |
|-----------------|--------|
| Field failure prevention | $150,000 |
| Debug time reduction | $195,000 |
| Rework reduction (yield improvement) | $37,500 |
| Report elimination | $22,800 |
| Operator time reduction | $50,000 |
| **Total annual savings** | **$455,300** |

### ROI Calculation

```
Year 1 ROI = ($455,300 - $65,000) / $65,000 × 100% = 600%
Payback period = $65,000 / ($455,300 / 12) = 1.7 months
```

## Using TofuPilot to Track ROI Over Time

Once you're running with TofuPilot, track these metrics monthly:

| KPI | How to measure | Target |
|-----|---------------|--------|
| First-pass yield | TofuPilot procedure dashboard | > 95% |
| Defect escape rate | Field returns / units shipped | < 0.1% |
| Mean time to diagnosis | Avg debug time per failure | < 30 min |
| Test throughput | Units tested per shift | Increasing |
| Rework rate | Failed units reworked / total | Decreasing |

These KPIs make the ongoing ROI visible to management. When someone asks "Is the test system worth it?" point to the dashboard.

## Common Objections and Responses

| Objection | Response |
|-----------|----------|
| "We don't have enough volume" | Calculate: how many field failures pay for the test system? |
| "Manual testing is fine" | Ask: how many hours/week on reports, data gathering, debugging? |
| "It's too expensive" | Compare to the cost of one product recall |
| "We don't have time to set it up" | Start with one test, one station. Expand after proving value |


### How to Build a PPAP Test Package
URL: https://www.tofupilot.com/guides/how-to-build-a-ppap-test-package-with-tofupilot
Learn how to assemble PPAP test documentation from production test data using TofuPilot, including control plans, MSA, and process capability reports.

Your customer sent a PPAP request. You need control plans, Cpk reports, MSA studies, and dimensional data, all tied to specific part numbers and production runs. Most teams spend days pulling this together from spreadsheets and disconnected systems.

TofuPilot captures the test data you need for PPAP submissions as part of your normal production workflow. No extra data entry required.

## What Is PPAP

Production Part Approval Process (PPAP) is a standardized framework, defined in AIAG's PPAP manual, that proves your manufacturing process consistently produces parts meeting customer specifications. It's required across automotive (IATF 16949), and increasingly adopted in aerospace, medical devices, and industrial electronics.

A PPAP submission typically includes 18 elements. Several depend directly on test data:

| PPAP Element | What It Needs | TofuPilot Source |
|---|---|---|
| Control Plan | Test steps, limits, frequencies | Test procedure definitions |
| Process Capability Study | Cpk/Ppk for critical dimensions | Measurement history |
| MSA / Gage R&R | Measurement system analysis | Repeated measurement data |
| Dimensional Results | Measured vs. spec for each characteristic | Run measurements with limits |
| Material/Performance Test Results | Functional test pass/fail with data | Test run results |

## Prerequisites

- A TofuPilot account with production test data
- Python 3.8+ with the `tofupilot` client installed
- At least 30 production runs for process capability calculations

## Step 1: Structure Your Tests Around Critical Characteristics

PPAP reviewers care about Critical-to-Quality (CTQ) characteristics. Map each CTQ to a measurement in your test procedure.

```python filename="ppap_test.py"
import openhtf as htf
from openhtf.util import units

# Map each CTQ characteristic to a measurement with spec limits
@htf.measures(
    htf.Measurement("output_voltage_5v")
        .with_units(units.VOLT)
        .in_range(4.95, 5.05)
        .doc("CTQ-001: 5V rail accuracy"),
    htf.Measurement("current_draw_idle")
        .with_units(units.AMPERE)
        .in_range(0.0, 0.150)
        .doc("CTQ-002: Idle current consumption"),
    htf.Measurement("rise_time_ms")
        .with_units(units.MILLISECOND)
        .at_most(5.0)
        .doc("CTQ-003: Power-on rise time"),
)
def power_rail_test(test):
    # Your instrument control code here
    test.measurements.output_voltage_5v = 5.01
    test.measurements.current_draw_idle = 0.098
    test.measurements.rise_time_ms = 3.2
```

Use the `.doc()` field to store the CTQ identifier. This makes it easy to trace from PPAP documentation back to actual test results.

## Step 2: Collect Process Capability Data

PPAP Level 3 submissions require process capability studies. You need at least 30 consecutive parts (AIAG recommends 300 for Ppk).

TofuPilot calculates Cp and Cpk automatically from your measurement data. To pull capability data for your PPAP package:

```python filename="ppap_capability.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get runs for a specific part number and procedure
runs = client.get_runs(
    procedure_id="power-board-fct",
    unit_part_number="PWR-200-R3",
    limit=300,
)

# Extract measurement values for each CTQ
voltage_values = [
    r.measurements["output_voltage_5v"].value
    for r in runs
    if r.measurements.get("output_voltage_5v")
]

print(f"Sample size: {len(voltage_values)}")
print(f"Mean: {sum(voltage_values) / len(voltage_values):.4f}")
```

TofuPilot's dashboard shows Cpk directly on measurement charts. Screenshot those for your submission, or export the raw data.

## Step 3: Run Gage R&R Studies

Measurement System Analysis (MSA) proves your test equipment is capable. A typical Gage R&R study uses 10 parts, 3 operators, 3 trials each.

```python filename="gage_rr_study.py"
import openhtf as htf
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Tag each run with operator and trial info for MSA
for operator in ["OP-A", "OP-B", "OP-C"]:
    for trial in range(1, 4):
        for part_sn in part_serial_numbers:
            # Run the test
            result = run_test(part_sn)

            # Upload with metadata for MSA grouping
            client.create_run(
                procedure_id="gage-rr-voltage",
                unit_under_test={
                    "serial_number": part_sn,
                    "part_number": "PWR-200-R3",
                },
                run_passed=result.passed,
                measurements=result.measurements,
                properties={
                    "operator": operator,
                    "trial": trial,
                    "study": "MSA-2026-Q1",
                },
            )
```

Filter by the `study` property in TofuPilot to pull all Gage R&R data for analysis.

## Step 4: Generate the Control Plan

Your control plan documents what you test, how you test it, and what limits you use. Pull this directly from your test definitions:

```python filename="control_plan_export.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get procedure definition with all measurements
procedure = client.get_procedure("power-board-fct")

print("| CTQ | Measurement | Lower Limit | Upper Limit | Unit | Method |")
print("|-----|-------------|-------------|-------------|------|--------|")

for m in procedure.measurements:
    print(
        f"| {m.doc or '-'} | {m.name} | "
        f"{m.lower_limit or '-'} | {m.upper_limit or '-'} | "
        f"{m.unit or '-'} | Automated FCT |"
    )
```

This table goes directly into your PPAP control plan. When limits change, regenerate it from the same source of truth.

## Step 5: Package Dimensional and Performance Results

PPAP requires actual measured values for a sample of parts (typically 5 from a significant production run).

```python filename="ppap_sample_results.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get the first 5 passing runs from the initial production run
sample_runs = client.get_runs(
    procedure_id="power-board-fct",
    unit_part_number="PWR-200-R3",
    passed=True,
    limit=5,
)

for run in sample_runs:
    print(f"SN: {run.unit.serial_number}")
    for name, m in run.measurements.items():
        status = "PASS" if m.passed else "FAIL"
        print(f"  {name}: {m.value} {m.unit or ''} [{status}]")
```

## Maintaining Your PPAP Package

PPAP isn't a one-time exercise. When you change your process, you may need to resubmit.

Track what triggers a PPAP update:

- **Engineering change**: New part revision, updated test limits
- **Process change**: New test equipment, different station configuration
- **Supplier change**: New component source affecting test characteristics

Because TofuPilot versions your test procedures and tracks station configurations, you can always trace which PPAP submission maps to which test setup. When a customer asks "what changed since your last submission," the answer is in your run history.


### HIL vs SIL vs MIL Testing: When to Use Each
URL: https://www.tofupilot.com/guides/hil-vs-sil-vs-mil-testing-when-to-use-each-with-tofupilot
Compare MIL, SIL and HIL testing on cost, speed and fidelity, with Python examples where only the plug changes between the three levels.

MIL, SIL, and HIL are three levels of hardware-in-the-loop testing that trade cost and speed for fidelity. Use MIL during algorithm development, SIL to validate compiled code, and HIL to verify behavior against real hardware before production.

## What MIL, SIL, and HIL Mean

**Model-in-the-Loop (MIL)** runs your control algorithm as a simulation model against a simulated plant. No hardware involved. Fast iteration, low cost, but lowest fidelity.

**Software-in-the-Loop (SIL)** compiles your production code and runs it on a host machine against a simulated environment. Same binary, no hardware. Catches code generation bugs MIL misses.

**Hardware-in-the-Loop (HIL)** runs your production code on the target ECU or microcontroller, connected to a real-time simulator that emulates the physical environment. Highest fidelity, highest cost.

## Comparison Matrix

| Dimension | MIL | SIL | HIL |
|-----------|-----|-----|-----|
| Cost | Low | Low-medium | High |
| Speed | Fast (seconds) | Medium (minutes) | Slow (minutes-hours) |
| Fidelity | Low | Medium | High |
| Hardware required | None | None | Target ECU + simulator rack |
| Typical use | Algorithm design | Code verification | System validation |
| CI-friendly | Yes | Yes | No (dedicated bench) |
| Bug discovery | Design errors | Codegen errors | Integration, timing |

## When to Use Each

### MIL: Early Algorithm Development

Use MIL when the algorithm is still changing, you need rapid iteration, hardware is not yet available, or you're running parametric sweeps. MIL is wrong when you need to validate timing behavior, interrupt latency, or bus communication.

### SIL: Code Verification Before Hardware

Use SIL when code generation is complete, you want to catch regressions in CI, you're validating numerical equivalence between model and generated code, or hardware bench slots are scarce. SIL misses real-time timing faults and hardware-specific peripheral behavior.

### HIL: Pre-Production System Validation

Use HIL when the ECU firmware is frozen, you need to validate timing and bus behavior, certification requires hardware evidence, or you're running overnight regression suites. HIL is expensive to set up and slow to iterate.

## Example: Voltage Limit Test at Each Level

The same test logic runs at all three levels. Only the plug changes.

With the TofuPilot Framework, that property is structural: the procedure declares the measurement and its limits once, and each level supplies a different plug for the same `sensor` parameter.

### The shared procedure

```yaml filename="procedure.yaml"
name: Battery Voltage Limit
version: 1.0.0

unit:
  serial_number:
    default_value: "UNIT000001"
  part_number:
    default_value: "ECU-100"

plugs:
  - name: sensor
    python: plugs.sensor:VoltageSensor

main:
  - name: Measure Battery Voltage
    python: phases.measure_battery_voltage
    measurements:
      - name: Battery Voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 10.0
          - operator: "<="
            expected_value: 14.5
```

```python filename="phases/measure_battery_voltage.py"
def measure_battery_voltage(measurements, sensor):
    measurements.battery_voltage = sensor.read_voltage()
```

Record which level produced the run so the dashboard can separate them:

```python filename="phases/record_level.py"
def record_level(run):
    run.metadata["test_level"] = "HIL"  # MIL | SIL | HIL
```

### MIL: simulated sensor

```python filename="plugs/sensor.py"
class VoltageSensor:
    """MIL: returns a deterministic value from a simulation model."""

    def read_voltage(self) -> float:
        return 11.8
```

### SIL: compiled production code

```python filename="plugs/sensor.py"
import ctypes


class VoltageSensor:
    """SIL: calls the compiled production C library via ctypes."""

    LIB_PATH = "./build/battery_monitor.so"

    def __init__(self):
        self._lib = ctypes.CDLL(self.LIB_PATH)
        self._lib.get_battery_voltage.restype = ctypes.c_float

    def read_voltage(self) -> float:
        return float(self._lib.get_battery_voltage())
```

### HIL: real ECU over CAN

```python filename="plugs/sensor.py"
import can


class VoltageSensor:
    """HIL: reads voltage from the real ECU over the CAN bus."""

    CHANNEL = "can0"
    BUSTYPE = "socketcan"

    def __init__(self):
        self._bus = can.interface.Bus(channel=self.CHANNEL, bustype=self.BUSTYPE)

    def read_voltage(self) -> float:
        msg = self._bus.recv(timeout=1.0)
        raw = msg.data[2]
        return raw * 0.1

    def close(self):
        self._bus.shutdown()
```

Keep the three plugs in separate directories (or select one with an environment variable) and point the procedure at the level you are running. The phase, the measurement name and the limits never change, which is what makes a MIL/SIL/HIL discrepancy meaningful rather than an artifact of three differently-written tests.

### The same thing in OpenHTF

If you prefer everything in Python with no YAML layer, OpenHTF expresses the same pattern with a plug per level:

```python filename="tests/hil/test_voltage_limit.py"
import can
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


class HILVoltageSensor(htf.plugs.BasePlug):
    """Reads voltage from real ECU over CAN bus."""

    CHANNEL = "can0"
    BUSTYPE = "socketcan"

    def setUp(self):
        self._bus = can.interface.Bus(channel=self.CHANNEL, bustype=self.BUSTYPE)

    def read_voltage(self) -> float:
        msg = self._bus.recv(timeout=1.0)
        raw = msg.data[2]
        return raw * 0.1

    def tearDown(self):
        self._bus.shutdown()


@htf.plug(sensor=HILVoltageSensor)
@htf.measures(
    htf.Measurement("battery_voltage")
    .in_range(minimum=10.0, maximum=14.5)
    .with_units(units.VOLT),
)
def measure_battery_voltage(test, sensor):
    test.measurements.battery_voltage = sensor.read_voltage()


def main():
    test = htf.Test(measure_battery_voltage, test_name="battery-voltage-hil")
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "UNIT-HIL-001")


if __name__ == "__main__":
    main()
```

Swap `HILVoltageSensor` for a simulated or ctypes-backed plug to get the MIL and SIL variants. Install with `pip install "tofupilot[openhtf]"`, or run the suite with `tofupilot run` and skip the callback.

## How TofuPilot Tracks Results Across All Levels

Each run uploads with the same measurement name (`battery_voltage`) regardless of level. A metadata key or the procedure name distinguishes the environment.

| Level | Distinguishing field | Traceability |
|-------|---------------------|-------------|
| MIL | `test_level: MIL` | Algorithm baseline |
| SIL | `test_level: SIL` | Codegen regression |
| HIL | `test_level: HIL` | Hardware validation |

A unit that passes MIL but fails SIL indicates a code generation issue. A unit that passes SIL but fails HIL indicates a hardware integration issue. Because the limits are declared in one place, that comparison is valid.

## Running MIL and SIL in CI

MIL and SIL have no operator and no hardware, so they belong in the pipeline. Resolve the identify prompt from the procedure and pre-bake any UI values so nothing blocks:

```bash filename="ci/run_sil.sh"
tofupilot run ./procedure.yaml --json --ui-values ui.json --ui-timeout 300
```

HIL needs the bench, so it runs on a dedicated station wired to a self-hosted CI runner instead.

## Decision Framework

| Activity | Recommended Level |
|----------|-------------------|
| Tuning control gains | MIL |
| Regression testing in CI | SIL |
| Verifying code generation output | SIL |
| Validating CAN/LIN bus behavior | HIL |
| Interrupt and timing verification | HIL |
| Overnight fault injection suite | HIL |
| Parametric sweep (100+ cases) | MIL or SIL |
| Pre-release sign-off | HIL |
| Root-cause analysis of field issue | HIL |

For the bench-side detail (GPIO, ADC, PWM plugs and CI wiring), see [HIL testing with Python](/guides/hardware-in-the-loop-testing-with-python-a-practical-guide).

### Motor and Actuator Testing
URL: https://www.tofupilot.com/guides/motor-and-actuator-testing-with-tofupilot
Learn how to automate motor and actuator production tests, capture torque, speed, and vibration data, and track results with TofuPilot.

Every motor that leaves your line needs to spin at the right speed, draw the right current, and not vibrate itself apart. Manual spot checks don't scale. You need automated tests that capture torque curves, speed profiles, and vibration signatures for every unit.

TofuPilot and OpenHTF let you build those tests in Python and track every result across production.

## What Motor Tests Cover

Production motor testing typically includes:

| Test Type | What It Measures | Why It Matters |
|---|---|---|
| No-load speed | RPM at rated voltage, no torque | Winding and magnet integrity |
| Stall torque | Maximum torque at zero RPM | Mechanical strength |
| Current draw | Amps at no-load, rated load, stall | Efficiency, winding shorts |
| Back-EMF | Generated voltage when spun externally | Magnet and winding quality |
| Vibration | Acceleration spectrum at operating speed | Bearing health, balance |
| Insulation resistance | Megohm reading winding-to-case | Safety, dielectric integrity |

## Prerequisites

- Python 3.8+ with `openhtf` and `tofupilot` installed
- A dynamometer or torque sensor with serial/GPIB/USB interface
- A programmable power supply for motor drive
- Optional: accelerometer or vibration sensor

## Step 1: Set Up the No-Load Test

The no-load test is your first gate. A motor that can't reach rated speed with no load has a fundamental problem.

```python filename="motor_noload_test.py"
import openhtf as htf
from openhtf.util import units
import time

@htf.measures(
    htf.Measurement("no_load_speed_rpm")
        .with_units(units.HERTZ)  # RPM stored as value
        .in_range(2850, 3150)
        .doc("No-load speed at rated voltage"),
    htf.Measurement("no_load_current_a")
        .with_units(units.AMPERE)
        .at_most(0.5)
        .doc("No-load current draw"),
    htf.Measurement("startup_time_ms")
        .with_units(units.MILLISECOND)
        .at_most(200)
        .doc("Time to reach 90% rated speed"),
)
def no_load_test(test, psu, tachometer):
    """Run motor at rated voltage with no load attached."""
    psu.set_voltage(24.0)
    psu.output_on()

    start = time.time()
    # Wait for motor to reach steady state
    time.sleep(2.0)

    speed = tachometer.read_rpm()
    current = psu.measure_current()
    startup = tachometer.get_startup_time_ms()

    test.measurements.no_load_speed_rpm = speed
    test.measurements.no_load_current_a = current
    test.measurements.startup_time_ms = startup

    psu.output_off()
```

## Step 2: Capture Torque-Speed Curves

The torque-speed curve characterizes motor performance across its operating range. Capture it as a multi-dimensional measurement.

```python filename="motor_torque_curve.py"
import openhtf as htf
from openhtf.util import units
import numpy as np

@htf.measures(
    htf.Measurement("torque_speed_curve")
        .with_dimensions(units.HERTZ)  # RPM as dimension
        .doc("Torque vs speed characteristic curve"),
    htf.Measurement("rated_torque_nm")
        .in_range(0.45, 0.55)
        .doc("Torque at rated speed"),
    htf.Measurement("efficiency_at_rated")
        .in_range(0.80, 1.0)
        .doc("Efficiency at rated operating point"),
)
def torque_curve_test(test, psu, dyno):
    """Sweep load from no-load to stall, capture torque at each point."""
    psu.set_voltage(24.0)
    psu.output_on()
    time.sleep(1.0)

    torque_points = []
    speed_points = []

    for load_pct in range(0, 105, 5):
        dyno.set_load_percent(load_pct)
        time.sleep(0.5)  # Settle time

        torque = dyno.read_torque_nm()
        speed = dyno.read_rpm()

        torque_points.append(torque)
        speed_points.append(speed)

        test.measurements.torque_speed_curve[speed] = torque

    # Find rated operating point (closest to rated speed)
    rated_idx = np.argmin(np.abs(np.array(speed_points) - 3000))
    rated_torque = torque_points[rated_idx]
    rated_speed = speed_points[rated_idx]

    test.measurements.rated_torque_nm = rated_torque

    # Calculate efficiency: P_mech / P_elec
    p_mech = rated_torque * rated_speed * 2 * np.pi / 60
    p_elec = 24.0 * psu.measure_current()
    test.measurements.efficiency_at_rated = p_mech / p_elec

    dyno.set_load_percent(0)
    psu.output_off()
```

## Step 3: Add Vibration Analysis

Vibration testing catches bearing defects, rotor imbalance, and misalignment before they become field failures.

```python filename="motor_vibration_test.py"
import openhtf as htf
from openhtf.util import units
import numpy as np

@htf.measures(
    htf.Measurement("vibration_rms_g")
        .at_most(0.5)
        .doc("Overall vibration RMS in g"),
    htf.Measurement("vibration_spectrum")
        .with_dimensions(units.HERTZ)
        .doc("Vibration frequency spectrum"),
    htf.Measurement("dominant_frequency_hz")
        .with_units(units.HERTZ)
        .doc("Highest amplitude frequency component"),
)
def vibration_test(test, psu, accelerometer):
    """Measure vibration at rated speed."""
    psu.set_voltage(24.0)
    psu.output_on()
    time.sleep(3.0)  # Wait for steady state

    # Capture 1 second of acceleration data at 10 kHz
    raw_data = accelerometer.capture(duration_s=1.0, sample_rate=10000)

    # RMS vibration
    rms = np.sqrt(np.mean(raw_data ** 2))
    test.measurements.vibration_rms_g = rms

    # FFT for spectrum
    fft_vals = np.abs(np.fft.rfft(raw_data))
    freqs = np.fft.rfftfreq(len(raw_data), d=1/10000)

    for freq, amplitude in zip(freqs[1:500], fft_vals[1:500]):
        test.measurements.vibration_spectrum[freq] = amplitude

    # Dominant frequency
    peak_idx = np.argmax(fft_vals[1:]) + 1
    test.measurements.dominant_frequency_hz = freqs[peak_idx]

    psu.output_off()
```

## Step 4: Build the Full Test Sequence

Combine all motor tests into a single production sequence:

```python filename="motor_production_test.py"
import openhtf as htf
from tofupilot import TofuPilotClient

def main():
    test = htf.Test(
        no_load_test,
        torque_curve_test,
        vibration_test,
    )

    test.add_output_callbacks(
        TofuPilotClient().as_openhtf_callback(
            procedure_id="motor-fct",
            procedure_name="Motor Production FCT",
        )
    )

    test.execute(test_start=htf.PhaseDescriptor.wrap(
        lambda test: setattr(
            test, "dut_id", input("Scan motor serial number: ")
        )
    ))

if __name__ == "__main__":
    main()
```

## Step 5: Monitor Production Trends

Once tests are running, use TofuPilot to catch drift before it causes failures:

- **No-load speed trending down**: Possible demagnetization or increased friction
- **Current draw creeping up**: Winding insulation degradation or bearing wear
- **Vibration RMS increasing**: Bearing failure progression
- **Efficiency dropping**: Look at both electrical and mechanical subsystems

Set measurement limits with margin. If your spec is 2850 to 3150 RPM, consider tighter production limits of 2900 to 3100 to catch drift early.

TofuPilot's control charts show these trends automatically. A shift in the mean or increasing standard deviation shows up before parts start failing.

## Actuator-Specific Considerations

Linear actuators and servo motors need additional tests:

| Actuator Type | Additional Tests |
|---|---|
| Linear actuator | Stroke length, extension force, retraction speed |
| Servo motor | Position accuracy, settling time, step response |
| Stepper motor | Holding torque, step accuracy, resonance points |
| Solenoid | Pull-in voltage, drop-out voltage, response time |

The test structure is the same. Define measurements with limits, capture the data, let TofuPilot track everything.


### LVCompare and LVMerge: What They Cannot Do
URL: https://www.tofupilot.com/guides/lvcompare-and-lvmerge-what-they-cannot-do
Learn what LVCompare and LVMerge do, how to wire them into Git, the failure modes you'll hit, and the four things neither tool can do for a review.

LVCompare and LVMerge both work. LVCompare shows a node-by-node diff of two VIs, and LVMerge performs a real three-way merge into a new VI. Both need a LabVIEW Professional seat with every dependency resolvable on the machine that runs them, and neither runs where reviews happen: in a pull request on the web, or on a colleague's laptop without Professional.

## What Each Tool Does

| Tool | Input | Output | Needs |
|---|---|---|---|
| LVCompare | Two VIs (two files, or two revisions of one) | A window listing each difference in front panel, block diagram, attributes and connector pane, with both VIs open side by side | LabVIEW Professional, same version as the VIs, all subVIs and classes resolvable |
| LVMerge | Base VI, Theirs VI, Yours VI | A merged VI you accept or reject difference by difference, then save | Same as LVCompare, and both branches must load without a missing-dependency dialog |

Both tools open the VIs in a real LabVIEW instance. That's why the dependency rule exists: if `Power Rails.vi` calls `DMM Read.vi` from a library the diff machine doesn't have, LabVIEW searches, and the compare either stalls on a dialog or runs against a broken VI.

## Setting Them Up with Git

Git runs whatever you name as a difftool or mergetool, so wiring them in is a few config lines. The lines below are for a 64-bit LabVIEW on Windows, run from Git Bash. Change the install path for your LabVIEW version; the path, the tool version and the VIs all have to match.

```ini filename=".gitconfig"
# LabVIEW difftool and mergetool. Paths are per LabVIEW version and bitness.
# LVMerge argument order is Base, Theirs, Yours, Merged, and all four must be absolute.
[difftool "lvcompare"]
    cmd = \"C:/Program Files/National Instruments/Shared/LabVIEW Compare/LVCompare.exe\" \"$(cygpath -aw \"$LOCAL\")\" \"$(cygpath -aw \"$REMOTE\")\" -nobdcosm -nofppos
[mergetool "lvmerge"]
    cmd = \"C:/Program Files/National Instruments/Shared/LabVIEW Merge/LVMerge.exe\" \"$(cygpath -aw \"$BASE\")\" \"$(cygpath -aw \"$REMOTE\")\" \"$(cygpath -aw \"$LOCAL\")\" \"$(cygpath -aw \"$MERGED\")\"
    trustExitCode = false
[diff]
    tool = lvcompare
[merge]
    tool = lvmerge
```

Three details carry the whole setup.

The argument order. Git exposes the three merge inputs as `$BASE`, `$LOCAL` (yours) and `$REMOTE` (theirs), and the output as `$MERGED`. LVMerge wants Base, Theirs, Yours, Merged, so `$REMOTE` goes before `$LOCAL`. Swap them and the tool runs, opens, and merges the wrong way round.

The absolute paths. Git hands the tools relative paths into a temporary directory. LVCompare and LVMerge want absolute Windows paths, which is what `cygpath -aw` produces. Without it the tool reports a missing file or opens nothing.

The version. The executable path contains the LabVIEW install, and the tool inside it only opens VIs from a compatible version. A team on LabVIEW 2019 and 2024 keeps two sets of lines and switches by machine.

Tell Git the files are binary as well, so it never tries a text merge or line-ending conversion on them.

```text filename=".gitattributes"
# Never treat these as text. Git stores and reverts them; the tools above diff them.
*.vi binary
*.ctl binary
*.seq binary
```

Then `git difftool -- "FCT/Power Rails.vi"` opens LVCompare on the working copy against HEAD, and `git mergetool -- "FCT/Power Rails.vi"` opens LVMerge during a conflicted merge.

## The Failure Modes

These are the ones that show up in practice and have a known cause.

| Symptom | Cause | Workaround |
|---|---|---|
| LVCompare refuses to open the second file, or opens it as a copy | LabVIEW can't hold two VIs with the same name in memory, so one copy has to be renamed, and a renamed class member no longer belongs to its class | Compare a plain VI, or check the other revision out into a second folder and compare from there; for class members, accept that the class context is lost |
| A search dialog appears and the compare stalls | A subVI, typedef or class the VI depends on isn't on the diff machine, or isn't at the path the VI expects | Run the compare from the project's own development machine with the `.lvproj` open, never from a bare checkout |
| The merged VI contains a node twice | LVMerge duplicates a node that was deleted and re-added in one branch | Inspect the merged block diagram before saving, remove the duplicate, run the VI once |
| The tool opens the wrong revisions, or nothing happens | Argument order is wrong (LVMerge wants Base, Theirs, Yours, Merged), or a path is relative | Check the order in `.gitconfig`, wrap every path in `cygpath -aw` |
| The config works on one PC and fails on another | The install path and the tool version are per LabVIEW version and bitness | Keep one config block per LabVIEW version in the repository README and copy the matching one |
| `LVCompare.exe` isn't there at all | The seat is Base or Full, not Professional | None inside LabVIEW; the review moves to a machine with Professional |

## What They Cannot Do

Four gaps, and none of them is a bug. They follow from the tools being LabVIEW instances.

Review in a pull request on the web. GitHub, GitLab and Bitbucket render a diff of text. For a `.vi` they show "Binary file changed". The reviewer downloads both revisions, opens LVCompare locally, and writes the review from memory. Nobody comments on line 27 of a block diagram.

Run on a machine without Professional. Base and Full seats don't ship the tools. On a team where one person has Professional, that person is the only reviewer, whatever the org chart says.

Merge a class or a typedef reliably. The rename problem above breaks class membership during a compare, and a typedef change ripples into every VI that uses it, which LVMerge handles one VI at a time. Most teams merge classes by picking one side and redoing the other by hand, which [fix LabVIEW merge conflicts on binary VIs](/guides/fix-labview-merge-conflicts-on-binary-vis) walks through.

Diff a `.seq`. TestStand sequences aren't VIs. Saved as XML they diff a little better in plain Git, but they still aren't something you'd merge by hand, and LVCompare and LVMerge don't open them.

## When They Are Still the Right Tool

If you stay on LabVIEW, use them. LVCompare on the development machine before every commit is the closest thing to reading your own diff, and it catches the accidental wire and the constant you didn't mean to change. LVMerge is worth the argument-order fiddling when two engineers have both made real diagram changes to the same VI and neither wants to redo a day's work.

They're the right tool for the machine that has them. They were never meant to be the review process for a team, and treating them as one is where the pain in [why LabVIEW version control is so painful](/guides/why-labview-version-control-is-so-painful) comes from. Where they stop, the honest options are a convention (one person per VI at a time), a second Professional seat, or moving limits out of the diagram into a file Git can read. [Code review for LabVIEW tests: why it fails](/guides/code-review-for-labview-tests-why-it-fails) covers the team side.

## The Contrast

A TofuPilot Framework station is text: `procedure.yaml`, `phases/*.py`, `plugs/*.py`. The same conflict on the same limit, when two branches both changed the 3.3 V floor, looks like this in the file.

```diff filename="procedure.yaml"
       - name: rail_3v3
         unit: V
         validators:
           - operator: ">="
<<<<<<< HEAD
             expected_value: 3.15
=======
             expected_value: 3.1
>>>>>>> widen-3v3-limit
           - operator: "<="
             expected_value: 3.4
```

You read both values, pick one or ask the two authors, delete the markers, `git add`, commit. A minute, on any machine, with no licence. Everything else in the two branches merged on its own, because Git could read it. The reviewer on the pull request saw the two candidate values before the merge, and after the push a station pulls the deployment and every run it uploads to TofuPilot carries that deployment's id.

## Start With One Station

When you move a station, pick the one whose VIs conflict most often. Rebuild it as a `procedure.yaml` with phases and plugs in Python with the TofuPilot Framework, keep the limits identical, and run it side by side with the LabVIEW version on the same units for two weeks.

```bash filename="install-and-run.sh"
# Install the CLI, run the station locally (no account needed), or run with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Automate a Rigol Power Supply
URL: https://www.tofupilot.com/guides/how-to-automate-a-rigol-power-supply-with-python-and-tofupilot
Automate a Rigol power supply from Python: SCPI control, safe output sequencing, and recording load measurements in a test procedure.

Rigol DP800-series supplies (DP821, DP831, DP832) are common on test benches because they are programmable, multi-channel, and inexpensive. This guide covers controlling one from Python over SCPI, sequencing the output safely, and recording what you measure as structured test data.

## Prerequisites

- Python 3.10+
- A Rigol DP800-series supply on USB or Ethernet
- `pip install pyvisa pyvisa-py`

## Connecting

```python filename="rigol/connect.py"
import pyvisa

rm = pyvisa.ResourceManager("@py")
print(rm.list_resources())

psu = rm.open_resource("TCPIP::192.168.1.101::INSTR")
psu.timeout = 5000
print(psu.query("*IDN?").strip())
# RIGOL TECHNOLOGIES,DP832,DP8A1234567890,00.01.16
```

If the supply is on USB and is not discovered on Linux, the usual cause is udev permissions rather than PyVISA.

## The Commands You Actually Need

| Command | Purpose |
|---------|---------|
| `:INST:NSEL 1` | Select channel by number (1, 2, 3) |
| `:VOLT 5.0` | Set voltage on the selected channel |
| `:CURR 0.5` | Set current limit on the selected channel |
| `:OUTP CH1,ON` | Enable a channel's output |
| `:OUTP CH1,OFF` | Disable it |
| `:MEAS:VOLT? CH1` | Measure actual output voltage |
| `:MEAS:CURR? CH1` | Measure actual output current |
| `:MEAS:POWE? CH1` | Measure output power |
| `:OUTP:OVP:VAL CH1,6.0` | Over-voltage protection threshold |
| `:OUTP:OCP:VAL CH1,1.0` | Over-current protection threshold |
| `:OUTP:OVP CH1,ON` | Enable OVP |
| `:SYST:ERR?` | Read the error queue |

Rigol accepts both `:INST:NSEL n` followed by bare `:VOLT`, and the channel-qualified `:APPL CH1,5.0,0.5` form. The channel-qualified form is harder to get wrong in a script, because there is no hidden selected-channel state.

## A Minimal Session

```python filename="rigol/basic.py"
import pyvisa
import time

rm = pyvisa.ResourceManager("@py")
psu = rm.open_resource("TCPIP::192.168.1.101::INSTR")
psu.timeout = 5000

psu.write("*RST")
psu.write("*CLS")

# Configure channel 1 for 5 V with a 500 mA limit
psu.write(":APPL CH1,5.0,0.5")

# Protection before the output is ever enabled
psu.write(":OUTP:OVP:VAL CH1,6.0")
psu.write(":OUTP:OVP CH1,ON")

psu.write(":OUTP CH1,ON")
time.sleep(0.2)  # Let the rail settle before measuring

v = float(psu.query(":MEAS:VOLT? CH1"))
i = float(psu.query(":MEAS:CURR? CH1"))
print(f"{v:.3f} V, {i:.4f} A")

psu.write(":OUTP CH1,OFF")
psu.close()
```

Two details that matter on a bench with a real board attached:

**Set protection before enabling the output.** OVP and OCP configured after the fact protect nothing during the inrush.

**Measure after a settle delay.** The supply will happily answer `:MEAS:VOLT?` mid-ramp and give you a number that is real but meaningless.

## Wrapping It as a Plug

In a test, the supply belongs in a plug: opened once, shared by phases, and guaranteed to power down at the end even when a phase raises.

```python filename="plugs/psu.py"
import time

import pyvisa


class PowerSupply:
    """Rigol DP800-series supply over SCPI."""

    RESOURCE = "TCPIP::192.168.1.101::INSTR"
    SETTLE_S = 0.2

    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 5000
        self.instr.write("*RST")
        self.instr.write("*CLS")

    def configure(self, channel: int, volts: float, current_limit: float):
        self.instr.write(f":APPL CH{channel},{volts},{current_limit}")
        self.instr.write(f":OUTP:OVP:VAL CH{channel},{volts * 1.2:.3f}")
        self.instr.write(f":OUTP:OVP CH{channel},ON")

    def output(self, channel: int, on: bool):
        self.instr.write(f":OUTP CH{channel},{'ON' if on else 'OFF'}")
        if on:
            time.sleep(self.SETTLE_S)

    def measure_voltage(self, channel: int) -> float:
        return float(self.instr.query(f":MEAS:VOLT? CH{channel}"))

    def measure_current(self, channel: int) -> float:
        return float(self.instr.query(f":MEAS:CURR? CH{channel}"))

    def identify(self) -> str:
        return self.instr.query("*IDN?").strip()

    def errors(self) -> list[str]:
        out = []
        while True:
            err = self.instr.query(":SYST:ERR?").strip()
            if err.startswith("0,"):
                break
            out.append(err)
        return out

    def close(self):
        """Always de-energise, even if a phase failed."""
        for ch in (1, 2, 3):
            try:
                self.instr.write(f":OUTP CH{ch},OFF")
            except Exception:
                pass
        self.instr.close()
```

`close()` is the important method. It runs at teardown whatever happened during the run, which is what stops a failed test leaving a board powered on the fixture overnight.

## Using It in a Procedure

```python filename="phases/power_up.py"
def power_up(measurements, psu):
    """Energise the board and confirm the rail came up under load."""
    psu.configure(channel=1, volts=5.0, current_limit=0.5)
    psu.output(1, True)

    measurements.rail_5v = psu.measure_voltage(1)
    measurements.inrush_current = psu.measure_current(1)
```

```python filename="phases/idle_current.py"
def idle_current(measurements, psu):
    """Board is booted; measure the quiescent draw."""
    measurements.idle_current = psu.measure_current(1)
```

```python filename="phases/supply_health.py"
def supply_health(measurements, psu):
    measurements.supply_errors = len(psu.errors())
```

```yaml filename="procedure.yaml"
name: Board Power-Up Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: psu
    python: plugs.psu:PowerSupply

main:
  - name: Power Up
    python: phases.power_up
    measurements:
      - name: Rail 5V
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.9
          - operator: "<="
            expected_value: 5.1
      - name: Inrush Current
        unit: A
        validators:
          - operator: "<="
            expected_value: 0.45

  - name: Idle Current
    python: phases.idle_current
    depends_on: [power_up]
    measurements:
      - name: Idle Current
        unit: A
        validators:
          - operator: ">="
            expected_value: 0.01
          - operator: "<="
            expected_value: 0.12

  - name: Supply Health
    python: phases.supply_health
    depends_on: [idle_current]
    measurements:
      - name: Supply Errors
        description: SCPI error queue depth after the sequence
        validators:
          - operator: "=="
            expected_value: 0
```

Run it with `tofupilot run ./procedure.yaml`.

The phases are chained with `depends_on` deliberately. They share one VISA session, and independent phases run concurrently, so a shared instrument must be serialised or given one session per phase.

The current-limit setting is worth a comment: at 0.5 A limit with an inrush limit of 0.45 A, a board that trips into constant-current mode will read a voltage well below 4.9 V and fail the rail measurement rather than silently passing at a collapsed rail.

## Recording the Supply Identity

```python filename="phases/record_equipment.py"
def record_equipment(run, psu):
    run.metadata["psu_id"] = psu.identify()
```

Useful the first time two benches disagree and you need to know which supply produced which numbers.

## Multi-Channel Boards

For a board with several rails, configure each channel and measure them together:

```python filename="phases/all_rails.py"
RAILS = {1: (5.0, 0.5), 2: (3.3, 0.5), 3: (1.8, 0.3)}


def all_rails(measurements, psu):
    for ch, (volts, limit) in RAILS.items():
        psu.configure(ch, volts, limit)
        psu.output(ch, True)

    measurements.rail_5v = psu.measure_voltage(1)
    measurements.rail_3v3 = psu.measure_voltage(2)
    measurements.rail_1v8 = psu.measure_voltage(3)
```

Power-up order matters on many designs. If your board requires a sequence, enable the channels in that order with delays between them rather than all at once.

## If You Already Run OpenHTF

The same supply wraps into a `BasePlug`:

```python filename="rigol/openhtf_test.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class RigolPsu(BasePlug):
    def setUp(self):
        import pyvisa
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource("TCPIP::192.168.1.101::INSTR")
        self.instr.timeout = 5000
        self.instr.write("*RST")

    def apply(self, channel: int, volts: float, limit: float):
        self.instr.write(f":APPL CH{channel},{volts},{limit}")
        self.instr.write(f":OUTP CH{channel},ON")

    def measure_voltage(self, channel: int) -> float:
        return float(self.instr.query(f":MEAS:VOLT? CH{channel}"))

    def tearDown(self):
        for ch in (1, 2, 3):
            self.instr.write(f":OUTP CH{ch},OFF")
        self.instr.close()


@htf.measures(
    htf.Measurement("rail_5v").in_range(4.9, 5.1).with_units(units.VOLT),
)
@htf.plug(psu=RigolPsu)
def test_power_up(test, psu):
    psu.apply(1, 5.0, 0.5)
    test.measurements.rail_5v = psu.measure_voltage(1)


test = htf.Test(test_power_up, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Troubleshooting

| Problem | Cause | Fix |
|---------|-------|-----|
| Voltage reads far below setpoint | Board is drawing the current limit | Raise the limit, or treat it as a genuine failure |
| Reading taken mid-ramp | No settle delay after `:OUTP ON` | Sleep before measuring |
| `-113, "Undefined header"` | Command typo, or unsupported on this model | Check the model's programming guide |
| Output stays on after a failed run | Teardown did not execute | Disable outputs in the plug's `close()` |
| Intermittent timeouts under parallel phases | One VISA session used concurrently | Serialise with `depends_on` |
| USB device not found on Linux | udev permissions | Add a udev rule for the Rigol vendor ID |

### Build a Test Data Strategy
URL: https://www.tofupilot.com/guides/how-to-build-a-test-data-strategy-with-tofupilot
Define what test data to capture, how to name procedures and measurements, and how to organize everything in TofuPilot for long-term traceability.

A test data strategy decides what you capture, how you name it, and how you organize it before you write your first test. Getting this right early saves months of cleanup later. Getting it wrong means your analytics are noisy, your traceability has gaps, and your team wastes time decoding inconsistent data.

## What Data to Capture

Every test run should include four categories of data:

### Measurements with Limits

These are the quantitative values that determine pass/fail. Always define limits in code, not in post-processing. This ensures every run is evaluated against the same criteria, and it means the limit that applied travels with the result.

| Data Type | Example | Why It Matters |
|-----------|---------|---------------|
| Parametric measurements | Voltage, current, resistance | Enables Cpk analysis and trend detection |
| Functional checks | Communication response, boot time | Validates system-level behavior |
| Environmental readings | Temperature, humidity during test | Explains measurement variation |

Record the measured value, not just the verdict. A boolean tells you a unit failed; the value tells you it measured 3.19V against a 3.20V limit and had been drifting for three weeks. This is the single field that is impossible to reconstruct later.

### Unit Identity

Every unit needs a unique serial number. If your product has sub-assemblies, track those serial numbers too. This is what lets you trace a field failure back through every test it ever went through.

### Metadata

Context that doesn't have limits but matters for analysis: firmware version, hardware revision, operator, test station. When you're investigating why yield dropped on Tuesday, metadata is how you find the cause.

### Attachments

Log files, waveform captures, images from optical inspection. Not every run needs attachments, but when a failure investigation starts, you'll want them. Attaching on failures rather than on every unit keeps this from outgrowing its usefulness.

## Naming Conventions

Consistent naming is the difference between data you can query and data you have to dig through manually.

### Procedure Names

Use a clear hierarchy: `{product}_{stage}_{test_type}`. Keep names lowercase with underscores.

| Pattern | Example |
|---------|---------|
| `{product}_{stage}_{test}` | `sensor_v2_evt_functional` |
| `{product}_{stage}_{test}` | `motor_ctrl_pvt_burn_in` |
| `{product}_{stage}_{test}` | `power_supply_dvt_thermal` |

Don't embed dates or station IDs in procedure names. Those are tracked as separate fields.

### Measurement Names

Use `{component}_{parameter}` format. Be specific enough that someone unfamiliar with the test can understand what was measured.

```python filename="test_naming_example.py"
# Well-named measurements for a power supply test
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    # Good: specific component, clear parameter
    htf.Measurement("rail_3v3_voltage")
        .in_range(minimum=3.25, maximum=3.35)
        .with_units(units.VOLT),
    htf.Measurement("rail_3v3_ripple")
        .in_range(maximum=30)
        .with_units(units.MILLIVOLT),
    htf.Measurement("rail_5v_voltage")
        .in_range(minimum=4.9, maximum=5.1)
        .with_units(units.VOLT),
    htf.Measurement("rail_5v_load_regulation_pct")
        .in_range(maximum=2.0),
    htf.Measurement("input_current_idle")
        .in_range(maximum=0.050)
        .with_units(units.AMPERE),
    htf.Measurement("thermal_shutdown_temp")
        .in_range(minimum=145, maximum=155)
        .with_units(units.DEGREE_CELSIUS),
)
def power_supply_validation(test):
    test.measurements.rail_3v3_voltage = 3.301
    test.measurements.rail_3v3_ripple = 18.4
    test.measurements.rail_5v_voltage = 5.03
    test.measurements.rail_5v_load_regulation_pct = 1.2
    test.measurements.input_current_idle = 0.0321
    test.measurements.thermal_shutdown_temp = 150.2


def main():
    test = htf.Test(
        power_supply_validation,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="PSU-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "PSU-2024-0891")


if __name__ == "__main__":
    main()
```

Install it with `pip install "tofupilot[openhtf]"`. The `upload()` callback reads `TOFUPILOT_API_KEY` from the environment.

Measurement names are effectively permanent. Renaming one splits its history into two series, so the trend you wanted to look at stops at the rename. Settle the convention before the first production run.

Avoid these naming mistakes:

| Bad Name | Problem | Better Name |
|----------|---------|-------------|
| `test1` | Meaningless | `rail_3v3_voltage` |
| `voltage` | Which voltage? | `rail_5v_voltage` |
| `V_out_3.3V_meas` | Inconsistent casing and format | `rail_3v3_output_voltage` |
| `temperature` | Which temperature, what unit? | `thermal_shutdown_temp` |

### Serial Number Format

Pick a format and enforce it. Common patterns:

| Format | Example | Use Case |
|--------|---------|----------|
| `{PREFIX}-{YEAR}-{SEQ}` | `PCB-2024-00042` | General manufacturing |
| `{PRODUCT}-{LOT}-{SEQ}` | `SNS-L0287-015` | Lot-based production |
| `{SITE}-{LINE}-{DATE}-{SEQ}` | `SH-A3-240315-0001` | Multi-site tracking |

Validate the format at the scan prompt. A typo caught at the fixture costs seconds; the same typo found six months later in a traceability record is usually unrecoverable.

## Organizing Procedures by Production Phase

Structure your procedures to match your actual production flow. Each phase of testing should be a separate procedure.

```
EVT (Engineering Validation)
├── sensor_v2_evt_power_on
├── sensor_v2_evt_functional
├── sensor_v2_evt_environmental
└── sensor_v2_evt_reliability

DVT (Design Validation)
├── sensor_v2_dvt_incoming_inspection
├── sensor_v2_dvt_calibration
├── sensor_v2_dvt_functional
└── sensor_v2_dvt_burn_in

PVT (Production Validation)
├── sensor_v2_pvt_smt_inspection
├── sensor_v2_pvt_ict
├── sensor_v2_pvt_functional
└── sensor_v2_pvt_final_qc
```

This structure gives you per-phase first pass yield, lets you compare yield between EVT and PVT, and makes it obvious which test caught a failure.

## Recording Sub-Assembly Links

`sub_units` is a field on `htf.Test(...)`, alongside `procedure_id` and `part_number`. It records which components went into the unit being tested.

```python filename="test_with_metadata.py"
# Attaching sub-unit serial numbers for BOM traceability
import openhtf as htf
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("boot_time_ms").in_range(maximum=500),
    htf.Measurement("self_test_result").equals("PASS"),
)
def functional_check(test):
    test.measurements.boot_time_ms = 230
    test.measurements.self_test_result = "PASS"


def main():
    test = htf.Test(
        functional_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # UUID for sensor_v2_pvt_functional
        part_number="SENSOR-V2",
        sub_units=[
            {"serial_number": "WIFI-MOD-2024-0331"},
            {"serial_number": "BT-MOD-2024-0887"},
        ],
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SENSOR-2024-2210")


if __name__ == "__main__":
    main()
```

Note that `procedure_id` is the dashboard UUID, not the procedure name. The naming convention above is for humans reading the dashboard; the code references the UUID. A legacy external identifier also works, but only if one was explicitly set on the procedure.

Sub-unit serial numbers let you trace which specific components went into each assembly. When a component lot has issues, you can find every finished unit that contains affected parts instead of quarantining a date range. This is also the hardest field to add retroactively, so model it from the first build even if you do not query it yet.

## Data Retention Checklist

Before you start collecting data, answer these questions:

1. **What compliance standards apply?** ISO 13485 (medical), AS9100 (aerospace), and IATF 16949 (automotive) all have specific data retention requirements.
2. **How long do you need to keep data?** Product lifetime plus warranty period is a common baseline. Medical devices often require 15+ years.
3. **What data needs to be immutable?** Test results used for regulatory compliance should never be editable after the fact.
4. **Who needs access?** Define roles early. Test engineers, quality managers, and customers may need different views of the same data.

Add a fifth: **how do you get the data out?** Retention obligations outlive tooling choices, so keep an export path that does not depend on a single vendor remaining in business.

Test data is stored with full audit history. Every run is timestamped and linked to its procedure, station, and operator, so you do not need to build your own retention system.

### Set Measurement Limits from Prod Data
URL: https://www.tofupilot.com/guides/how-to-set-measurement-limits-using-production-data-with-tofupilot
Learn how to derive statistically sound measurement limits from production data using the 3-sigma method, detect distribution anomalies, and configure.

Datasheet limits are starting points, not production limits. Real boards have tolerances, assembly variation, and environmental drift that datasheets cannot capture. This guide shows you how to collect measurement data from your production line, apply the 3-sigma method to derive statistically sound limits, and configure those limits in OpenHTF with TofuPilot tracking.

## Prerequisites

- TofuPilot account with production run data
- Python 3.8+
- `numpy` and `scipy` installed
- At least 30 test samples (100+ recommended)

## Why Datasheet Limits Are Not Enough

| Source of Variation | Example |
|---------------------|---------|
| Component tolerance | Resistor +/-5% shifts output voltage |
| PCB trace impedance | Layout differences across panel positions |
| Assembly variation | Solder joint resistance changes |
| Environmental drift | Temperature at test station vs. field |
| Measurement noise | Probe contact, cable length |

A 3.3V rail with a datasheet range of +/-5% (3.135V-3.465V) may show actual production distribution centered at 3.31V with a std of 0.012V. Setting limits from the datasheet alone misses that your process is drifting high.

## Step 1: Collect Baseline Samples

You need a minimum of 30 samples from boards you have already validated as good. Use 100+ for stable sigma estimates.

```python filename="collect_samples.py"
import numpy as np

# Simulated collection from known-good boards
# In practice, export from TofuPilot or accumulate from live runs
vdd_samples = [
    3.312, 3.308, 3.315, 3.310, 3.307,
    3.319, 3.311, 3.314, 3.309, 3.316,
    3.313, 3.308, 3.312, 3.317, 3.310,
    3.306, 3.315, 3.311, 3.313, 3.309,
    3.318, 3.310, 3.312, 3.315, 3.308,
    3.311, 3.314, 3.307, 3.316, 3.313,
    3.310, 3.308, 3.315, 3.312, 3.311,
    3.309, 3.317, 3.314, 3.310, 3.313,
]

n = len(vdd_samples)
mean = np.mean(vdd_samples)
std = np.std(vdd_samples, ddof=1)  # sample std, not population

print(f"N={n}  mean={mean:.4f}V  std={std:.4f}V")
# N=40  mean=3.3119V  std=0.0031V
```

Use `ddof=1` for sample standard deviation when your sample size is less than the full population.

## Step 2: Check for Outliers and Bimodal Distributions

Before computing limits, inspect the distribution. A bimodal distribution signals two distinct populations.

```python filename="check_distribution.py"
import numpy as np
from scipy import stats

vdd_samples = [
    3.312, 3.308, 3.315, 3.310, 3.307,
    3.319, 3.311, 3.314, 3.309, 3.316,
    3.313, 3.308, 3.312, 3.317, 3.310,
    3.306, 3.315, 3.311, 3.313, 3.309,
    3.318, 3.310, 3.312, 3.315, 3.308,
    3.311, 3.314, 3.307, 3.316, 3.313,
    3.310, 3.308, 3.315, 3.312, 3.311,
    3.309, 3.317, 3.314, 3.310, 3.313,
]

# Z-score outlier detection
z_scores = np.abs(stats.zscore(vdd_samples))
outliers = [v for v, z in zip(vdd_samples, z_scores) if z > 3]
print(f"Outliers (|z| > 3): {outliers}")

# Shapiro-Wilk normality test
stat, p = stats.shapiro(vdd_samples)
print(f"Shapiro-Wilk p={p:.4f} ({'normal' if p > 0.05 else 'NOT normal'})")
```

If Shapiro-Wilk returns `p < 0.05`, investigate whether your samples came from two assembly batches or two component reels.

## Step 3: Compute 3-Sigma Limits

The 3-sigma rule covers 99.73% of a normal distribution (roughly 2700 DPMO at the limit edge).

```python filename="compute_limits.py"
import numpy as np

vdd_samples = [
    3.312, 3.308, 3.315, 3.310, 3.307,
    3.319, 3.311, 3.314, 3.309, 3.316,
    3.313, 3.308, 3.312, 3.317, 3.310,
    3.306, 3.315, 3.311, 3.313, 3.309,
    3.318, 3.310, 3.312, 3.315, 3.308,
    3.311, 3.314, 3.307, 3.316, 3.313,
    3.310, 3.308, 3.315, 3.312, 3.311,
    3.309, 3.317, 3.314, 3.310, 3.313,
]

mean = np.mean(vdd_samples)
std = np.std(vdd_samples, ddof=1)

sigma_levels = {
    "3s (99.73%)": 3,
    "4s (99.9937%)": 4,
    "6s (99.99966%)": 6,
}

print(f"Mean:  {mean:.4f} V")
print(f"Std:   {std:.4f} V\n")

for label, k in sigma_levels.items():
    lo = mean - k * std
    hi = mean + k * std
    print(f"{label:25s}  low={lo:.4f}V  high={hi:.4f}V")
```

| Sigma Level | Coverage | Hard limits (VDD example) |
|-------------|----------|---------------------------|
| 3-sigma | 99.73% | 3.3026V to 3.3212V |
| 4-sigma | 99.9937% | 3.2995V to 3.3243V |
| 6-sigma | 99.99966% | 3.2933V to 3.3305V |

## Step 4: Add Marginal Limits (Warning Zone)

Marginal limits create a warning zone between the soft limit and the hard fail. A board in the marginal zone passes but gets flagged for monitoring.

| Zone | Range | Result |
|------|-------|--------|
| Pass | mean +/- 2-sigma | Normal pass |
| Marginal | mean +/- 2-sigma to mean +/- 3-sigma | Pass with warning |
| Fail | outside mean +/- 3-sigma | Hard fail |

## Step 5: Configure Limits in OpenHTF with TofuPilot

```python filename="vdd_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

# Derived from 40 production samples: mean=3.3119V, std=0.0031V
VDD_HARD_LOW      = 3.3026  # mean - 3-sigma
VDD_HARD_HIGH     = 3.3212  # mean + 3-sigma
VDD_MARGINAL_LOW  = 3.3057  # mean - 2-sigma
VDD_MARGINAL_HIGH = 3.3181  # mean + 2-sigma


@htf.measures(
    htf.Measurement("vdd_rail_voltage")
    .in_range(
        minimum=VDD_HARD_LOW,
        maximum=VDD_HARD_HIGH,
        marginal_minimum=VDD_MARGINAL_LOW,
        marginal_maximum=VDD_MARGINAL_HIGH,
    )
    .with_units(units.VOLT)
    .doc("3.3V rail with marginal zone [2-sigma, 3-sigma] for drift monitoring.")
)
def measure_vdd(test):
    voltage = read_voltage_at_tp12()
    test.measurements.vdd_rail_voltage = voltage


def main():
    test = htf.Test(
        measure_vdd,
        test_name="VDD Rail Validation",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-001")


if __name__ == "__main__":
    main()
```

OpenHTF marginal outcome maps to `MARGINAL_PASS` in TofuPilot. You can filter by this outcome in the dashboard to track drift trends without stopping the line.

## Step 6: Track Limit Drift in TofuPilot

Re-run the sigma analysis monthly and compare:

```python filename="refresh_limits.py"
import numpy as np
from datetime import datetime


def compute_sigma_limits(samples: list[float], k: float = 3.0) -> dict:
    arr = np.array(samples)
    mean = np.mean(arr)
    std = np.std(arr, ddof=1)
    return {
        "n": len(arr),
        "mean": round(mean, 6),
        "std": round(std, 6),
        "low": round(mean - k * std, 6),
        "high": round(mean + k * std, 6),
        "sigma": k,
        "computed_at": datetime.utcnow().isoformat(),
    }
```

Compare new limits to current production limits before deploying. A shift in mean greater than 1-sigma warrants a process investigation.

| Review Trigger | Action |
|----------------|--------|
| Mean shift > 1-sigma | Investigate root cause before updating |
| Std increase > 20% | Check component reel change or station calibration |
| Marginal rate > 5% | Tighten limits or improve process |
| New sample N > 500 | Re-evaluate sigma level (consider 4-sigma) |

### Predictive Maintenance from Test Data
URL: https://www.tofupilot.com/guides/predictive-maintenance-from-test-data-in-tofupilot
Learn how to use production test data trends in TofuPilot to predict equipment failures and schedule maintenance before downtime occurs.

Your test data already contains the signals for predictive maintenance. A test fixture that's wearing out doesn't fail suddenly. Its measurements drift first. TofuPilot's trending tools let you see the drift before it causes failures.

## What Test Data Reveals About Equipment Health

Every test run is a health check on two things: the unit under test and the test equipment itself.

When a measurement starts drifting, it could be the product changing or the test station changing. Separating the two is the key to predictive maintenance.

| Signal | Product issue | Equipment issue |
|--------|--------------|-----------------|
| Measurement drift | Component lot variation | Fixture wear, calibration drift |
| Increased variance | Mixed component populations | Loose connections, intermittent contacts |
| Step change | Design revision, process change | Equipment swap, recalibration |
| Station-specific failures | N/A (affects all stations) | Worn probes, faulty instruments |

The distinguishing factor: if the drift or failure is specific to one station, it's the equipment.

## Using TofuPilot for Fixture Health Monitoring

### Track Key Measurements by Station

Filter measurements by station to isolate equipment-related trends from product-related trends.

```python filename="fixture_health_check.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get recent runs from a specific station
runs = client.get_runs(
    procedure_id="BOARD-FUNCTIONAL",
    limit=500,
)

# Track contact resistance trend (proxy for pogo pin wear)
contact_readings = []
for run in runs:
    for step in run.get("steps", []):
        for m in step.get("measurements", []):
            if m["name"] == "contact_resistance_mohm":
                contact_readings.append({
                    "date": run["created_at"],
                    "value": m["value"],
                    "station": run.get("station_id"),
                })

# Rising contact resistance = pogo pins wearing out
```

### Set Maintenance Thresholds

Don't wait for a fixture to fail. Set maintenance thresholds based on measurement trends.

| Measurement | Normal range | Maintenance threshold | Failure threshold |
|------------|-------------|----------------------|-------------------|
| Contact resistance | 10-50 mohm | > 80 mohm | > 150 mohm |
| Probe alignment (via measurement variance) | < 0.5% RSD | > 1% RSD | > 2% RSD |
| Cycle time | 40-45 s | > 55 s | > 70 s |

When a station's contact resistance trend crosses the maintenance threshold, schedule pin replacement. Don't wait for it to hit the failure threshold.

### Monitor Cycle Time as a Health Indicator

Test cycle time is an underused equipment health signal. A test that normally takes 42 seconds and starts taking 55 seconds is telling you something:

- Instrument communication delays (aging GPIB cables, USB hub issues)
- Actuator slowdown (pneumatic fixture, motorized probe)
- Retry loops (intermittent contacts causing measurement retries)
- Software issues (memory leaks in long-running test software)

TofuPilot tracks cycle time for every run. A sudden increase is an early warning.

## Calibration Drift Detection

Test instruments drift over time. A DMM that reads 3.300V today might read 3.305V next month. If your spec limit is 3.35V, that 5mV drift eats into your margin.

TofuPilot helps detect calibration drift by comparing measurement distributions over time:

1. Establish a baseline distribution during the first month after calibration
2. Monitor the distribution monthly
3. If the mean shifts by more than 10% of the spec range, recalibrate

This approach lets you move from calendar-based calibration (every 6 months regardless) to condition-based calibration (when the data shows drift). You save money on unnecessary calibrations and avoid the risk of running with a drifted instrument.

## From Reactive to Predictive

| Approach | When you fix it | Cost |
|----------|----------------|------|
| Reactive | After it breaks | Highest: downtime + scrap + emergency repair |
| Preventive | On a schedule | Medium: some unnecessary maintenance |
| Predictive | When data shows early signs | Lowest: targeted maintenance, zero downtime |

The data for predictive maintenance is already flowing through your test stations. TofuPilot stores it, trends it, and surfaces the early warning signs. You just need to look.

## Building a Maintenance Dashboard

Create a monitoring view focused on equipment health:

1. **Station FPY comparison**: Drop in one station's yield = equipment issue
2. **Contact resistance trend per station**: Rising trend = fixture wear
3. **Cycle time trend per station**: Increasing trend = equipment slowdown
4. **Measurement variance per station**: Increasing variance = intermittent contacts

Review this dashboard weekly. Most equipment issues give you days or weeks of warning before they cause production impact. The data is there. TofuPilot makes it visible.


### Test Station Monitoring and Performance
URL: https://www.tofupilot.com/guides/test-station-monitoring-track-uptime-and-performance-with-tofupilot
Learn how to monitor test station health, track uptime metrics, and detect performance degradation using TofuPilot analytics.

A test station that's silently degrading is worse than one that's down. Slow tests eat throughput. Dropping yields hide root causes. You need visibility into station health before problems hit your production line.

This guide covers what to monitor, how to capture station health metrics inside your OpenHTF tests, and how to detect drift using TofuPilot's analytics.

## What to Monitor

Four metrics tell you most of what you need to know about a test station.

**Test throughput.** Units per hour, per station. A drop means something changed: slower tests, more retests, or operator delays.

**Pass rate trends.** FPY over time, not just today's number. A slow decline from 97% to 93% over two weeks is easy to miss in daily reports.

**Average test duration.** Track per-phase and total. If your calibration phase went from 4s to 12s, the instrument connection is probably degrading.

**Station errors.** Uncaught exceptions, instrument timeouts, fixture faults. These don't always fail the DUT, but they signal trouble.

## Capture Station Health Metrics

You can log station health data (CPU, memory, disk) as OpenHTF measurements alongside your DUT tests. This gives you a per-run snapshot of station condition.

```python filename="station_health.py"
import openhtf as htf
import psutil

@htf.measures(
    htf.Measurement("cpu_percent").in_range(maximum=90),
    htf.Measurement("memory_percent").in_range(maximum=85),
    htf.Measurement("disk_percent").in_range(maximum=90),
    htf.Measurement("disk_read_mb"),
    htf.Measurement("cpu_temp"),
)
def station_health_check(test):
    """Capture station health metrics before running DUT tests."""
    test.measurements.cpu_percent = psutil.cpu_percent(interval=1)
    test.measurements.memory_percent = psutil.virtual_memory().percent
    test.measurements.disk_percent = psutil.disk_usage("/").percent
    test.measurements.disk_read_mb = psutil.disk_io_counters().read_bytes / (1024 * 1024)

    # CPU temperature (Linux only, returns empty list on other platforms)
    temps = psutil.sensors_temperatures()
    if temps and "coretemp" in temps:
        test.measurements.cpu_temp = temps["coretemp"][0].current
    else:
        test.measurements.cpu_temp = 0.0
```

Add this phase at the start of your test sequence. If CPU or memory is pegged, you'll see it in TofuPilot before it causes flaky test results.

```python filename="main_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload
from station_health import station_health_check

# Your DUT test phases
@htf.measures(htf.Measurement("voltage_3v3").in_range(3.1, 3.5).with_units(units.VOLT))
def test_power_rail(test):
    test.measurements.voltage_3v3 = 3.28

def main():
    test = htf.Test(
        station_health_check,
        test_power_rail,
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "DUT-001")

if __name__ == "__main__":
    main()
```

## Detect Performance Drift in TofuPilot

TofuPilot tracks test duration, pass rates, and measurement trends per station automatically. Use the Analytics tab to spot drift:

- **Test duration trend.** Filter by station and check whether average test time is increasing. A 15%+ increase over baseline signals instrument connection degradation, fixture wear, or background process interference.
- **FPY by station.** Compare yield across stations running the same procedure. A station with 3+ points lower FPY than its neighbors needs fixture inspection.
- **Measurement histograms.** Check whether station health measurements (CPU, memory, disk) are creeping toward their limits over time.
- **Failure Pareto.** If one station accounts for a disproportionate share of failures, investigate that station's fixture and connections.

## Monitoring Checklist

| Metric | Frequency | Threshold | Action |
|--------|-----------|-----------|--------|
| Test throughput (units/hr) | Hourly | Below 80% of target | Check for operator delays, instrument timeouts |
| First pass yield | Per shift | Below 95% (or your target) | Investigate top failing phases |
| Average test duration | Daily | More than 15% above baseline | Check instrument connections, fixture wear |
| CPU usage | Per run | Above 90% | Close background processes, check for memory leaks |
| Memory usage | Per run | Above 85% | Restart station, check for leaking test processes |
| Disk usage | Daily | Above 90% | Clean logs, archive old data |
| Station errors | Per run | Any uncaught exception | Fix root cause, add error handling |
| Instrument timeout rate | Daily | Above 1% | Check cables, GPIB/USB connections |

### Add Measurements with Limits
URL: https://www.tofupilot.com/guides/how-to-add-measurements-with-limits-in-tofupilot
Add measurements with limits in TofuPilot: numeric, string and boolean validators, one-sided limits, units, and what makes a measurement useful later.

A measurement is a named value with units and limits, recorded against a unit. Getting this right is the single highest-leverage decision in a test system, because every analysis you will ever want (yield, capability, drift, Pareto) is computed from measurements and impossible without them.

## The Shape of a Measurement

```yaml filename="procedure.yaml"
main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        description: TP-100 rev B section 3.1
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
```

Four parts, each doing a job:

| Field | Purpose |
|-------|---------|
| `name` | Display name, and the source of the Python key |
| `unit` | Displayed everywhere, and prevents ambiguity between mV and V |
| `description` | Where the spec paragraph reference goes |
| `validators` | The limits, stored as data |

The Python key is derived from the name in snake_case: `Rail 3V3` becomes `rail_3v3`. You can set `key` explicitly if you want a different one.

## Numeric Limits

A two-sided range is two validators:

```yaml filename="procedure.yaml"
      - name: Output Voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.75
          - operator: "<="
            expected_value: 5.25
```

The available numeric operators:

| Operator | Meaning |
|----------|---------|
| `==` | Equal to |
| `!=` | Not equal to |
| `>` | Greater than |
| `>=` | Greater than or equal |
| `<` | Less than |
| `<=` | Less than or equal |

## One-Sided Limits

Plenty of measurements have only one meaningful bound. Leakage current has a maximum; insulation resistance has a minimum. Declare exactly one validator:

```yaml filename="procedure.yaml"
      - name: Leakage Current
        unit: uA
        validators:
          - operator: "<="
            expected_value: 500

      - name: Insulation Resistance
        unit: "MΩ"
        validators:
          - operator: ">="
            expected_value: 100
```

Do not invent a second limit to make the measurement look symmetric. A fabricated lower bound on leakage current produces a Cpk figure that describes nothing. See [CPL and CPU](/guides/what-is-cpl-and-cpu-one-sided-capability).

## String Measurements

```yaml filename="procedure.yaml"
      - name: Firmware Version
        validators:
          - operator: "=="
            expected_value: "2.1.0"
```

String operators include set membership and regex, which is what you want for serial formats and multi-version acceptance:

| Operator | Expected value | Example |
|----------|---------------|---------|
| `==` | String | `"PASS"` |
| `!=` | String | `"ERROR"` |
| `in` | Array of strings | `["v1.0.0", "v1.0.1"]` |
| `not in` | Array of strings | `["FAIL", "ERROR"]` |
| `matches` | Regex | `"^SN[0-9]{8}$"` |

```yaml filename="procedure.yaml"
      - name: Accepted Firmware
        validators:
          - operator: "in"
            expected_value: ["2.1.0", "2.1.1", "2.2.0"]

      - name: Serial Format
        validators:
          - operator: "matches"
            expected_value: "^SN[0-9]{8}$"
```

## Boolean Measurements

```yaml filename="procedure.yaml"
      - name: Lid Closed
        validators:
          - operator: "=="
            expected_value: true
```

```python filename="phases/safety.py"
def safety(measurements, fixture):
    measurements.lid_closed = fixture.lid_closed()
```

Booleans are the right type for genuinely binary facts: an interlock, a link established, a self-test result. They are the wrong type for something that was measured and then thresholded in Python, which is the most common measurement mistake.

## Record the Value, Not Your Verdict

```python filename="bad/thresholded.py"
def power_rails(measurements, dmm):
    v = dmm.read_voltage()
    measurements.rail_ok = 3.2 <= v <= 3.4   # The number is gone
```

```python filename="good/measured.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()   # Keep the number
```

The boolean version passes and fails correctly and is worthless a month later. It cannot tell you the rail has drifted from 3.31 to 3.38, cannot produce a Cpk, and cannot justify a limit change. Let the validators do the comparison.

## Measurements Without Limits

A measurement with no validators is recorded and reported without a verdict. That is correct for context you want to keep but not judge:

```yaml filename="procedure.yaml"
      - name: Ambient Temperature
        unit: "°C"
      - name: Chamber Setpoint
        unit: "°C"
```

Useful for HALT steps, environmental context, and any diagnostic value whose acceptable range you do not yet know. Recording them now is what lets you set limits from data later.

## Several Measurements in One Phase

A phase records as many measurements as it needs:

```yaml filename="procedure.yaml"
  - name: Power Checks
    python: phases.power_checks
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Rail 5V0
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
      - name: Quiescent Current
        unit: mA
        validators:
          - operator: "<="
            expected_value: 45
      - name: Boot Time
        unit: ms
        validators:
          - operator: "<="
            expected_value: 2000
```

```python filename="phases/power_checks.py"
def power_checks(measurements, dmm, dut):
    measurements.rail_3v3 = dmm.dc_voltage(1)
    measurements.rail_5v0 = dmm.dc_voltage(2)
    measurements.quiescent_current = dmm.dc_current_ma()
    measurements.boot_time = dut.boot_time_ms()
```

The phase fails if any of its measurements fail, and the failure message names which one.

## Measurements From Operator Input

A value an operator types is still a measurement. Bind the UI component directly to it:

```yaml filename="procedure.yaml"
  - name: Torque Check
    python: phases.torque_check
    ui:
      components:
        - key: torque
          type: number_input
          label: "Screw torque (Nm)"
          required: true
          bind: measurements.screw_torque
    measurements:
      - name: Screw Torque
        unit: Nm
        validators:
          - operator: ">="
            expected_value: 0.45
          - operator: "<="
            expected_value: 0.55
```

A manually recorded torque then trends on a control chart like any instrument reading.

## Naming Is Permanent

The name is the identity of the measurement across every run you will ever take. Renaming splits the history and breaks every trend spanning the change.

| Poor name | Better | Why |
|-----------|--------|-----|
| `voltage` | `rail_3v3` | Which voltage? |
| `current1` | `quiescent_current` | Meaningless in six months |
| `test2` | `boot_time` | Describes nothing |
| `temp` | `chamber_temperature` | Ambient, chamber, or die? |

Specific names also survive the test growing a second, third and fourth voltage.

## If You Already Run OpenHTF

The same concepts, expressed with decorators:

```python filename="fct.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT)
    .doc("TP-100 rev B section 3.1"),
    htf.Measurement("leakage_current_uA")
    .in_range(maximum=500)
    .with_units(units.MICROAMPERE),
    htf.Measurement("firmware_version").equals("2.1.0"),
    htf.Measurement("lid_closed").equals(True),
)
def phase_checks(test):
    test.measurements.rail_3v3 = 3.31
    test.measurements.leakage_current_uA = 85.0
    test.measurements.firmware_version = "2.1.0"
    test.measurements.lid_closed = True


test = htf.Test(phase_checks, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Thresholding in Python into a boolean | The number is lost; no Cpk, no drift | Record the value, declare validators |
| Inventing a second limit for symmetry | Meaningless capability figures | One validator for one-sided limits |
| Generic names | Ambiguous in six months, unscalable | Specific names chosen once |
| Renaming later | History splits, trends break | Treat names as permanent |
| Omitting units | mV and V confusion on a real bench | Always set `unit` |
| Discarding operator-entered values | Manual readings never trend | `bind: measurements.<key>` |
| No `description` | Nobody can find the governing spec paragraph | Reference the document section |

### What Is Design Validation Testing
URL: https://www.tofupilot.com/guides/what-is-design-validation-testing-with-tofupilot
Design validation testing (DVT) confirms a product meets its design requirements. Learn how to structure DVT in Python and track results with TofuPilot.

Design validation testing (DVT) confirms that a product meets its design requirements under real-world conditions. It happens after prototyping and before production tooling. This guide explains where DVT fits in the product development cycle, how to build DVT scripts in Python, and how to track validation results with TofuPilot.

## Where DVT Fits

Hardware products typically go through three build stages:

| Stage | Units | Purpose |
|-------|-------|---------|
| EVT (Engineering Validation) | 5-20 | Does the concept work? |
| DVT (Design Validation) | 20-100 | Does the design meet requirements? |
| PVT (Production Validation) | 100-500 | Can manufacturing build it reliably? |

DVT answers a specific question: does this design meet the requirements in the spec? You test against the product requirements document (PRD), not against manufacturing tolerances. The units are built on near-final tooling, and the test conditions include environmental extremes, mechanical stress, and electrical corner cases.

## What DVT Covers

| Category | Example Tests |
|----------|--------------|
| Electrical | Power consumption, signal integrity, EMI/EMC |
| Mechanical | Drop, vibration, torque, insertion force |
| Environmental | Temperature cycling, humidity, altitude |
| Reliability | HALT, accelerated life testing, endurance |
| Safety | Leakage current, dielectric strength, flammability |
| Functional | All user-facing features under nominal and edge conditions |

DVT is not pass/fail in the EOL sense. It produces data that feeds design decisions. A measurement that lands close to a limit might pass DVT but prompt a design change to add margin before PVT.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Define Validation Measurements

DVT phases often measure the same parameter under different conditions. Use descriptive measurement names that include the test condition.

```python filename="dvt_thermal.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("current_draw_25C_mA")
    .in_range(minimum=40, maximum=60)
    .with_units(units.MILLIAMPERE),
    htf.Measurement("current_draw_85C_mA")
    .in_range(minimum=40, maximum=70)
    .with_units(units.MILLIAMPERE),
)
def phase_thermal_current(test):
    """Measure current draw at room temp and high temp."""
    test.measurements.current_draw_25C_mA = 48.3
    test.measurements.current_draw_85C_mA = 55.1
```

## Step 2: Add Marginal Limits

DVT benefits from marginal limits. A measurement inside the marginal band passes but flags a potential risk. TofuPilot tracks marginal results separately so you can spot trends before they become failures.

```python filename="dvt_thermal.py"
@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(
        minimum=3.0, maximum=3.6,
        marginal_minimum=3.1, marginal_maximum=3.5,
    )
    .with_units(units.VOLT),
)
def phase_output_voltage(test):
    """Validate output voltage is within spec with margin."""
    test.measurements.output_voltage_V = 3.28
```

## Step 3: Run and Track Results

Connect the DVT script to TofuPilot. Each unit in the DVT build gets its own test record, making it easy to compare performance across the build lot.

```python filename="dvt_thermal.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_thermal_current,
    phase_output_voltage,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan DVT unit serial: "))
```

## Step 4: Analyze in TofuPilot

TofuPilot tracks DVT results automatically. Open the Analytics tab to review:

- **Measurement distributions** across the DVT build lot
- **Marginal results** flagged separately from hard failures
- **Unit-level traceability** linking each prototype to its full test history
- **Trends over time** to catch drift across sequential builds

This data feeds directly into the DVT report. Instead of manually compiling spreadsheets, export the data from TofuPilot or share the dashboard link with your team.

## DVT vs Production Testing

| Aspect | DVT | Production Test |
|--------|-----|-----------------|
| Goal | Verify the design | Verify each unit |
| Sample size | 20-100 units | Every unit |
| Test depth | Deep, multi-condition | Fast, single-condition |
| Limits | Spec-derived, with margin bands | Tightened from production data |
| Output | Design report | Ship/no-ship decision |

DVT limits often start wider and tighten as you move into PVT and production. TofuPilot lets you track this evolution by comparing measurement distributions across test procedures for the same product.

### What Is an Operator Interface
URL: https://www.tofupilot.com/guides/what-is-an-operator-interface-for-manufacturing-test
An operator interface is the screen a technician uses to run a test. What it must show, what it must prevent, and how to build one without frontend work.

The operator interface is the screen a technician looks at all shift. It decides how fast the line runs, how many retests happen for no reason, and whether a real failure gets noticed or waved through. It is usually the last part of a test system to get attention and the first part to cause trouble.

## What It Has to Do

| Requirement | Why |
|---|---|
| Identify the unit | Scan a barcode; typing serial numbers produces typos and untraceable results |
| Show progress | A technician needs to know whether a 90-second test is running or hung |
| Show the verdict unambiguously | Pass and fail must be distinguishable across a noisy room at a glance |
| Show what failed | "FAIL" alone sends the board to a diagnostic bench that did not need to be involved |
| Collect operator input | Visual checks, torque values, cable connections, lot codes |
| Prevent unsafe actions | No energising with the fixture open |
| Work offline | The line does not stop because the network did |

The fourth row is the one most often missed. A screen that shows only a red bar trains operators to retest rather than investigate, and a retest that passes hides a marginal board.

## What Usually Gets Built Instead

Most teams start with a terminal running a Python script. It works for a prototype and fails on a line for predictable reasons: a scrolling console gives no glanceable state, operators learn to press Enter through prompts they stop reading, and there is no way to stop a run cleanly.

The next step is usually a small desktop app, and that is where the cost appears. A production-grade operator UI needs live state from the running test, safe interruption, input validation, and a build for every station. That is a frontend project with its own release cycle, competing for time with the tests themselves.

## Declaring the Interface Instead

The alternative is to declare what the operator sees as part of the procedure, so there is no frontend to build or deploy. Each phase can carry a `ui.components` list, and each component has a `key`, a `type` and a `label`.

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: fixture
    python: plugs.fixture:Fixture
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Fixture Setup
    python: phases.fixture_setup
    ui:
      components:
        - key: instructions
          type: text
          label: "Insert the board, close the lid, then continue."
        - key: preflight
          type: checklist
          label: "Confirm before starting"
          required: true
          options:
            - label: "Board seated flat"
              value: "seated"
            - label: "Lid latched"
              value: "latched"
            - label: "No debris on pogo pins"
              value: "clean"

  - name: Safety
    python: phases.safety
    depends_on: [fixture_setup]
    measurements:
      - name: Lid Closed
        validators:
          - operator: "=="
            expected_value: true

  - name: Power Rails
    python: phases.power_rails
    depends_on: [safety]
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/fixture_setup.py"
def fixture_setup():
    """The checklist blocks until the operator completes it."""
    pass
```

```python filename="phases/safety.py"
def safety(measurements, fixture):
    measurements.lid_closed = fixture.lid_closed()
```

Run it with `tofupilot run ./procedure.yaml`. The CLI renders the interface, and the same declaration works on a bench laptop and on a station in kiosk mode.

## The Components Worth Knowing

| Component | Use |
|---|---|
| `text` | Instructions, warnings, live values |
| `text_input` / `textarea` | Notes, lot codes, free text |
| `number_input` | Torque values, manual readings |
| `checklist` | Multi-step confirmations before a risky action |
| `switch` / `radio` / `select` / `multiselect` | Choices, variants, pass/fail judgements |
| `image` | Show the operator what a good joint looks like |
| `slider` | Bounded manual adjustment |
| `progress` | Long-running phases |

An image next to a visual inspection prompt is worth more than any amount of written instruction, and it costs a few lines in the procedure.

## Operator Input Is Data, Not Just Flow Control

The mistake that quietly destroys traceability is treating a prompt as a gate and discarding the answer. If an operator records a torque value, that value belongs in the record as a measurement with limits.

The `bind` field does exactly that: it routes the component's value straight into a measurement, with no phase code in between.

```yaml filename="procedure.yaml"
  - name: Torque Check
    python: phases.torque_check
    ui:
      components:
        - key: torque
          type: number_input
          label: "Screw torque (Nm)"
          required: true
          bind: measurements.screw_torque
    measurements:
      - name: Screw Torque
        unit: Nm
        validators:
          - operator: ">="
            expected_value: 0.45
          - operator: "<="
            expected_value: 0.55
```

```python filename="phases/torque_check.py"
def torque_check(measurements):
    """The bound UI value populates screw_torque; nothing to do here."""
    pass
```

Now a torque that drifts across a shift is visible on a control chart, rather than being a number somebody typed and nobody kept. If it falls outside the limits, the phase fails like any other measurement.

`bind` also routes to unit fields and run attribution:

```yaml filename="procedure.yaml"
  - name: Scan Unit
    python: phases.scan_unit
    ui:
      components:
        - key: serial_number
          type: text_input
          label: "Serial Number"
          bind: unit.serial_number
        - key: operator_badge
          type: text_input
          label: "Operator badge"
          bind: run.operated_by
```

A badge scanner filling `run.operated_by` is how competence records get captured without anyone maintaining a separate log.

## Running Without an Operator

The same procedure has to run unattended in CI or on a soak rack. Rather than maintaining a second copy, resolve the prompts externally with a JSON file keyed by phase and component:

```bash filename="ci/run.sh"
tofupilot run ./procedure.yaml --json --ui-values ci/ui.json --ui-timeout 300
```

```json filename="ci/ui.json"
{
  "fixture_setup": { "preflight": ["seated", "latched", "clean"] },
  "torque_check": { "torque": 0.5 }
}
```

`--ui-timeout` matters here. Without it, an unanswerable prompt blocks forever; with it, the phase fails and the job moves on.

## Kiosk Mode on a Station

On a production station, the interface runs full-screen with no browser chrome, so operators cannot navigate away or close it. Deployments update it on a Git push, and runs queue locally when the network drops. See [how to set up kiosk mode for test stations](/guides/how-to-set-up-kiosk-mode-for-test-stations).

## If You Already Run OpenHTF

OpenHTF has a built-in Station Server and user input plugs, and existing suites run natively on the TofuPilot CLI, which provides the operator UI:

```python filename="operator_test.py"
import openhtf as htf
from openhtf.plugs.user_input import UserInput
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
@htf.plug(prompts=UserInput)
def test_power(test, prompts):
    prompts.prompt("Close the fixture lid, then press OK.")
    test.measurements.rail_3v3 = 3.31


test = htf.Test(test_power, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or run it with `tofupilot run` for the richer interface. See [operator interface with OpenHTF](/guides/operator-interface-with-openhtf).

## Common Mistakes

| Mistake | Consequence | Fix |
|---|---|---|
| Showing FAIL with no detail | Boards go to diagnostics that did not need to | Show which measurement failed and its value |
| Typed serial numbers | Typos, untraceable results | `bind: unit.serial_number` with a scanner |
| Discarding operator input | Manual readings vanish from the record | `bind: measurements.<key>` with limits |
| Prompts an operator can click through blind | Confirmations stop meaning anything | Use checklists for genuinely risky steps |
| A separate unattended copy of the procedure | The two drift apart | One procedure, `--ui-values` for CI |
| No timeout on prompts | An unattended run hangs forever | `--ui-timeout` |
| Building a custom desktop app first | A frontend project competing with test work | Declare the UI in the procedure |

### DUT vs UUT: Is There Actually a Difference?
URL: https://www.tofupilot.com/guides/dut-vs-uut-is-there-actually-a-difference
DUT and UUT mean the same thing: the hardware being tested. Where each term comes from, who says which, and what actually matters: the serial number.

Short answer: same thing. DUT (device under test) and UUT (unit under test) both mean the piece of hardware sitting in the fixture right now, about to find out what it's really worth. Test engineers have used both for decades, often in the same meeting, sometimes in the same sentence.

So why two names? Mostly geography. Not countries, industries.

## Who Says What

| Term | Stands for | Where you'll hear it |
|------|-----------|----------------------|
| **DUT** | Device Under Test | Semiconductors, RF, lab benches, instrument manuals, OpenHTF |
| **UUT** | Unit Under Test | Aerospace, defense, MRO, NI TestStand, IEEE 1671 (ATML) |

A few ways to tell which camp you're in:

- Someone mentions the **DUT board** on a semiconductor tester: DUT camp.
- Your test executive has an entry point called **Test UUTs**: you're in TestStand country.
- OpenHTF, which started at Google, stores the ID of whatever it tested as `dut_id`. NI and Google never settled this, and the hardware never noticed.

## Is There Any Real Difference?

Officially, no. Unofficially, some teams use a size rule:

- **DUT** for something small: a chip, a sensor, a bare board
- **UUT** for something you'd rather not drop on your foot: an assembly, a line-replaceable unit, a finished product

That's a house convention, not a standard. If your team has one, put it in the test plan. If it doesn't, please don't open a Slack thread about it. (Someone already has.)

## The Rest of the Acronym Zoo

Once you spot the "under test" pattern, it's everywhere:

| Term | Stands for | Habitat |
|------|-----------|---------|
| EUT | Equipment Under Test | EMC and safety labs (IEC 61000-4, CISPR) |
| SUT | System Under Test | System-level and software testing |
| DUV | Design Under Verification | Chip verification, before there's any silicon |
| IUT | Implementation Under Test | Protocol conformance testing |
| BUT | Board Under Test | Nowhere official. Yet. |

And if you work with French teams: DUT is also a French two-year technology diploma, held by plenty of test technicians. It was replaced in 2021 by the BUT. So in France, a DUT really can test a DUT.

## What Actually Matters: the Serial Number

No yield has ever dropped because of an acronym. What does break is identity: the same board recorded under two IDs, or a repaired unit that comes back from rework as a stranger.

Whatever you call it, the thing under test needs three fields:

| Field | Why it matters |
|-------|---------------|
| Serial number | Ties every run, retest and repair to one physical unit |
| Part number | Groups units of the same design for yield and Cpk |
| Revision | Keeps REV-A data out of REV-B statistics |

In TofuPilot we skipped the debate and just called it a **unit**:

```yaml filename="procedure.yaml"
unit:
  serial_number:
    default_value: "SN-000001"
  part_number:
    default_value: "PCBA-100"
```

Every run attaches to a unit by serial number, so its full history lives in one place, whatever your team calls it. If you're coming from OpenHTF, the `dut_id` you return from `test_start` becomes the unit's serial number.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Debating DUT vs UUT in a design review | One lost afternoon | Pick one, write it in the test plan |
| `dut_sn` on one station, `uut_sn` on another | Two columns that never join | One identity field across every station |
| Testing before the serial is known | Retests look like new units, FPY lies | Scan or read the serial before the first phase |
| New serial after rework | Repair history split across two "units" | Keep the serial, record the rework as a run |

## FAQ

**Is DUT or UUT more correct?**
Neither. Use whatever your industry, your customer or your test executive already uses.

**Can I use both in the same document?**
You can. Your reviewer will circle it in red. Pick one per document.

**What do I call it in TofuPilot?**
A unit, identified by its serial number. That's the whole naming scheme.

### Track Equipment Calibration
URL: https://www.tofupilot.com/guides/how-to-track-equipment-calibration-with-tofupilot
Validate instrument calibration status before every test run. Track calibration dates as metadata and catch expired calibrations before they corrupt your data.

Test instruments drift over time. A multimeter that was accurate six months ago might now read 50mV high, pushing borderline units past their limits. Tracking calibration status in your test workflow prevents expired instruments from corrupting your production data.

## Why Calibration Tracking Matters

Regulatory frameworks like ISO 17025 and ISO 9001 require traceable calibration for test equipment. But compliance aside, the practical reason is simple: if your instruments aren't accurate, your pass/fail decisions aren't reliable.

An uncalibrated power supply might output 5.08V when it claims 5.00V. A DMM with drift might read 3.28V when the true value is 3.31V. These errors shift your measurement distributions and can cause both false passes (shipping bad units) and false failures (scrapping good ones).

## Validate Calibration Before Testing

Build a calibration check into your test sequence. Before running any DUT measurements, query the instrument's calibration date and compare it against the expiration window.

```python filename="test_with_cal_check.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload
from datetime import datetime, timedelta

CALIBRATION_INTERVAL_DAYS = 180

@htf.measures(
    htf.Measurement("dmm_cal_days_remaining")
    .in_range(minimum=0)
    .doc("Days until DMM calibration expires. Fails if expired.")
)
def check_dmm_calibration(test):
    # Read last calibration date from instrument or local config
    last_cal_date = datetime(2026, 1, 15)
    expiry_date = last_cal_date + timedelta(days=CALIBRATION_INTERVAL_DAYS)
    days_remaining = (expiry_date - datetime.now()).days
    test.measurements.dmm_cal_days_remaining = days_remaining

@htf.measures(
    htf.Measurement("output_voltage")
    .in_range(minimum=3.25, maximum=3.35)
    .with_units(units.VOLT),
    htf.Measurement("ripple")
    .in_range(maximum=50.0),
)
def test_voltage_regulator(test):
    test.measurements.output_voltage = 3.30
    test.measurements.ripple = 12.4

def main():
    test = htf.Test(
        check_dmm_calibration,
        test_voltage_regulator,
        station_id="SZ-L1-FCT-03",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "REG-2026-08841")

if __name__ == "__main__":
    main()
```

The `check_dmm_calibration` phase runs first. If the calibration has expired (days remaining is negative), the measurement fails its `minimum=0` limit and the entire test run fails. No DUT measurements are taken with an out-of-cal instrument.

## Track Multiple Instruments Per Station

Most test stations use several instruments. Check each one.

```python filename="test_with_multi_cal.py"
import openhtf as htf
from tofupilot.openhtf import upload
from datetime import datetime, timedelta

CAL_INTERVAL = timedelta(days=180)

def days_until_expiry(last_cal_date):
    return (last_cal_date + CAL_INTERVAL - datetime.now()).days

@htf.measures(
    htf.Measurement("cal_dmm_days_remaining").in_range(minimum=0),
    htf.Measurement("cal_scope_days_remaining").in_range(minimum=0),
    htf.Measurement("cal_psu_days_remaining").in_range(minimum=0),
)
def check_all_calibrations(test):
    # Read from instrument memory, config file, or calibration database
    test.measurements.cal_dmm_days_remaining = days_until_expiry(datetime(2026, 1, 15))
    test.measurements.cal_scope_days_remaining = days_until_expiry(datetime(2025, 11, 20))
    test.measurements.cal_psu_days_remaining = days_until_expiry(datetime(2026, 2, 1))

def main():
    test = htf.Test(
        check_all_calibrations,
        station_id="GDL-L2-FCT-01",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "UNIT-2026-12003")

if __name__ == "__main__":
    main()
```

Each instrument gets its own measurement. If the oscilloscope's calibration expires but the DMM and PSU are fine, TofuPilot shows exactly which instrument needs attention.

## Store Calibration Metadata

Beyond the pass/fail check, record calibration details so you have a full audit trail.

```python filename="test_with_cal_metadata.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("cal_dmm_days_remaining").in_range(minimum=0),
)
def check_calibration(test):
    test.measurements.cal_dmm_days_remaining = 47

@htf.measures(
    htf.Measurement("supply_voltage")
    .in_range(minimum=11.8, maximum=12.2)
    .with_units(units.VOLT),
)
def test_supply(test):
    test.measurements.supply_voltage = 12.03

def main():
    test = htf.Test(
        check_calibration,
        test_supply,
        station_id="AUS-L1-EOL-02",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PSU-2026-00219")

if __name__ == "__main__":
    main()
```

TofuPilot stores every run with its full measurement data. If a quality audit asks "was the DMM calibrated when unit PSU-2026-00219 was tested?", you can answer definitively by looking up that run.

## Use Fixture Validation Phases

Beyond instrument calibration, test fixtures themselves need validation. Add a fixture check phase that measures a known reference standard before testing DUTs.

```python filename="test_with_fixture_check.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("fixture_ref_resistance")
    .in_range(minimum=99.5, maximum=100.5)
    .with_units(units.OHM)
    .doc("Golden reference resistor. Validates fixture contacts and DMM.")
)
def validate_fixture(test):
    # Measure a known 100 ohm reference on the fixture
    test.measurements.fixture_ref_resistance = 100.1

@htf.measures(
    htf.Measurement("dut_resistance")
    .in_range(minimum=45.0, maximum=55.0)
    .with_units(units.OHM),
)
def test_dut_resistance(test):
    test.measurements.dut_resistance = 49.7

def main():
    test = htf.Test(
        validate_fixture,
        test_dut_resistance,
        station_id="SZ-L3-ICT-01",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "HTR-2026-06650")

if __name__ == "__main__":
    main()
```

The fixture validation phase measures a golden reference. If the reading is outside tolerance, the fixture contacts are worn, the DMM has drifted, or both. Either way, the test stops before producing unreliable DUT data.

## Monitor Calibration Health in TofuPilot

TofuPilot's measurement trends show your calibration health over time:

- **Track `cal_*_days_remaining` trends** to see when instruments are approaching expiration across all stations
- **Watch fixture validation measurements** for slow drift that indicates contact wear
- **Filter by station** to see which specific machines need calibration soon
- **Set up alerts** for when calibration days remaining drops below a threshold (e.g., 14 days) so you can schedule recalibration proactively

### What Is FMEA for Manufacturing Test
URL: https://www.tofupilot.com/guides/what-is-fmea-for-manufacturing-test-with-tofupilot
FMEA identifies potential failure modes before they happen. Learn how to apply FMEA to manufacturing test processes and track corrective actions with TofuPilot.

Failure Mode and Effects Analysis (FMEA) is a structured method for identifying what can go wrong, how severe the consequences are, and what controls exist to catch it. Applied to manufacturing test, FMEA helps you decide which test steps matter most and where to invest in detection. This guide covers how FMEA works, the difference between DFMEA and PFMEA, and how TofuPilot data feeds your FMEA process.

## How FMEA Works

FMEA scores each potential failure mode on three dimensions:

| Factor | What It Measures | Scale |
|--------|-----------------|-------|
| Severity (S) | How bad is it if this failure reaches the customer? | 1-10 |
| Occurrence (O) | How often does this failure mode happen? | 1-10 |
| Detection (D) | How likely is your current test to catch it? | 1-10 |

The Risk Priority Number (RPN) is S x O x D. Higher RPN means higher priority. A failure that's severe, frequent, and hard to detect gets the highest score.

### Example FMEA for a Power Supply

| Failure Mode | Effect | S | O | D | RPN | Action |
|-------------|--------|---|---|---|-----|--------|
| Solder bridge on output FET | Short circuit, potential fire | 10 | 3 | 2 | 60 | AOI + hipot test covers this |
| Wrong capacitor value | Output ripple out of spec | 6 | 2 | 4 | 48 | Add ripple measurement to FCT |
| Cold solder on connector | Intermittent connection in field | 8 | 4 | 7 | 224 | Add pull test or thermal cycling screen |
| Firmware flash failure | Unit DOA | 9 | 2 | 1 | 18 | Self-test catches this |

The cold solder on the connector has the highest RPN because it's hard to detect (D=7). That's where you invest in better testing.

## DFMEA vs PFMEA

| Type | Scope | Who Leads | When |
|------|-------|-----------|------|
| DFMEA (Design FMEA) | Product design failures | Design engineering | EVT/DVT |
| PFMEA (Process FMEA) | Manufacturing process failures | Process/test engineering | PVT/production |

DFMEA asks: what can fail in the design? PFMEA asks: what can go wrong during manufacturing? For test engineering, PFMEA is where you define which tests are needed and why.

## How TofuPilot Data Feeds FMEA

FMEA requires real data for the Occurrence and Detection scores. Guessing these numbers makes the analysis unreliable. TofuPilot provides the data you need:

| FMEA Input | TofuPilot Source |
|-----------|-----------------|
| Occurrence rate | Failure Pareto by test step |
| Detection effectiveness | Compare production test yield to field return rate |
| Failure mode distribution | Measurement distributions showing which parameters drift |
| Trend data | Control charts showing whether a failure mode is increasing |

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Map FMEA Actions to Test Phases

Each FMEA action item that requires a test becomes an OpenHTF phase. Include the FMEA reference in the phase docstring for traceability.

```python filename="fmea_driven_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_ripple_mV")
    .in_range(maximum=50)
    .with_units(units.MILLIVOLT),
)
def phase_ripple_check(test):
    """PFMEA item #2: Detect wrong capacitor via ripple measurement."""
    test.measurements.output_ripple_mV = 28.5


@htf.measures(
    htf.Measurement("connector_resistance_mOhm")
    .in_range(maximum=100)
    .with_units(units.OHM),
)
def phase_connector_check(test):
    """PFMEA item #3: Detect cold solder on output connector."""
    test.measurements.connector_resistance_mOhm = 42.0


@htf.measures(
    htf.Measurement("hipot_result").equals("PASS"),
)
def phase_hipot(test):
    """PFMEA item #1: Verify no shorts on output stage."""
    test.measurements.hipot_result = "PASS"
```

## Step 2: Log Results and Update FMEA

Connect the test to TofuPilot. As production data accumulates, update your FMEA occurrence and detection scores with real numbers.

```python filename="fmea_driven_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_hipot,
    phase_ripple_check,
    phase_connector_check,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Step 3: Review and Iterate

TofuPilot tracks results per test step. Open the Analytics tab to review:

- **Failure Pareto** shows which failure modes actually occur most often (updates your O score)
- **Measurement distributions** show whether your limits catch marginal units (validates your D score)
- **Yield trends** show whether corrective actions are working (RPN should decrease over time)

FMEA is a living document. Review it quarterly using TofuPilot data to update scores and reprioritize actions. A failure mode that was rare six months ago might be common now due to a supplier change.

### Build Full Unit Traceability
URL: https://www.tofupilot.com/guides/how-to-build-full-unit-traceability-with-tofupilot
Link every test result to a specific serial number and track each unit's complete test history through its production lifecycle.

Unit traceability means linking every test result to a specific serial number so you can reconstruct the full history of any device. TofuPilot captures this automatically when you run OpenHTF tests with serial numbers, giving you a searchable unit history page for every DUT.

## Why Unit Traceability Matters

When a field failure occurs, you need to answer: what tests did this unit pass, when, on which station, and with what measurements? Without traceability, you're digging through spreadsheets and log files. With it, you type a serial number and get the full picture.

Regulatory frameworks like ISO 13485, AS9100, and IPC-1782 all require linking test records to individual units. TofuPilot stores this mapping automatically.

## Capturing Serial Numbers in OpenHTF

OpenHTF uses a `test_start` trigger to capture the DUT serial number before any test phase runs. The simplest approach prompts the operator to scan or type the serial.

```python filename="test_traceability.py"
# Production test with serial number capture and TofuPilot upload
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("voltage_output").in_range(minimum=4.8, maximum=5.2).with_units(units.VOLT),
    htf.Measurement("current_draw").in_range(minimum=0.1, maximum=0.5).with_units(units.AMPERE),
)
def power_test(test):
    test.measurements.voltage_output = 5.01
    test.measurements.current_draw = 0.25

def main():
    test = htf.Test(power_test)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial: "))

if __name__ == "__main__":
    main()
```

Every time this test runs, TofuPilot associates the measurements and pass/fail result with the scanned serial number. Run the same serial again and TofuPilot appends to its history.

## Programmatic Serial Number Input

In automated lines, serial numbers come from barcode scanners or MES systems rather than operator input. You can pass the serial directly.

```python filename="test_automated_serial.py"
# Automated serial input from barcode scanner
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("resistance").in_range(minimum=95, maximum=105).with_units(units.OHM),
)
def resistance_check(test):
    test.measurements.resistance = 100.2

def main():
    # Serial comes from your barcode scanner or MES integration
    serial_number = read_barcode_scanner()

    test = htf.Test(resistance_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: serial_number)

if __name__ == "__main__":
    main()
```

## Adding Unit Metadata

You can attach metadata to test runs for clearer unit history organization by production stage.

```python filename="test_with_metadata.py"
# Tag runs with production stage for clearer unit history
import openhtf as htf
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("firmware_version"),
    htf.Measurement("boot_time").in_range(maximum=5.0),
)
def firmware_check(test):
    test.measurements.firmware_version = "2.1.0"
    test.measurements.boot_time = 1.8

def main():
    test = htf.Test(firmware_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-20260312-001")

if __name__ == "__main__":
    main()
```

## Multi-Stage Unit History

Real production involves multiple test stages: ICT, functional test, burn-in, final QA. Each stage runs as a separate OpenHTF test with the same serial number. TofuPilot stitches them together.

```python filename="test_multi_stage.py"
# Each production stage uploads to TofuPilot with the same serial
import openhtf as htf
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("ict_shorts").in_range(maximum=0),
)
def ict_test(test):
    test.measurements.ict_shorts = 0

@htf.measures(
    htf.Measurement("functional_pass"),
)
def functional_test(test):
    test.measurements.functional_pass = True

def run_stage(phase, serial):
    test = htf.Test(phase)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: serial)

def main():
    serial = "SN-20260312-001"
    run_stage(ict_test, serial)
    run_stage(functional_test, serial)

if __name__ == "__main__":
    main()
```

Open the unit's page in TofuPilot's dashboard to see every test run, measurement, and pass/fail result for that serial number in chronological order. You can filter by procedure, station, or date range.

## Viewing Unit History in TofuPilot

Once your tests upload results, go to TofuPilot's Units page and search by serial number. The unit detail page shows:

- Every test run tied to that serial, across all procedures and stations
- Individual measurements with pass/fail status and limits
- Timestamps and station identifiers for each run
- Links to related sub-assemblies (if configured)

This is the audit trail that regulators and quality engineers need. No extra code required.

### What Is Gage R&R and MSA
URL: https://www.tofupilot.com/guides/what-is-grr-and-msa-with-tofupilot
Gage R&R and MSA quantify measurement system variation. Learn how to assess your test equipment and track measurement reliability with TofuPilot.

Gage Repeatability and Reproducibility (GRR) measures how much of your test variation comes from the measurement system itself. Measurement System Analysis (MSA) is the broader evaluation that includes GRR plus bias, linearity, and stability. This guide covers how GRR and MSA work, how to run a GRR study, and how to use TofuPilot data to monitor measurement system health.

## Why Measurement Variation Matters

Every test measurement has three sources of variation:

| Source | What It Is |
|--------|-----------|
| Part-to-part | Real differences between units (this is what you want to measure) |
| Repeatability | Variation when the same operator measures the same part multiple times |
| Reproducibility | Variation when different operators or stations measure the same part |

If your measurement system contributes too much variation, you can't trust your pass/fail decisions. Good units get rejected (false failures). Bad units pass (escapes).

## GRR Acceptance Criteria

| GRR % of Tolerance | Assessment |
|--------------------|-----------|
| Below 10% | Measurement system is acceptable |
| 10% to 30% | May be acceptable depending on application |
| Above 30% | Measurement system needs improvement |

GRR is expressed as a percentage of the specification tolerance. A GRR of 20% means the measurement system uses up 20% of your tolerance band with its own noise.

## How a GRR Study Works

A crossed GRR study uses multiple operators, multiple parts, and multiple trials:

| Parameter | Typical Value |
|-----------|--------------|
| Operators (or stations) | 2-3 |
| Parts | 10 (spanning the range of production variation) |
| Trials per part per operator | 2-3 |
| Total measurements | 40-90 |

Each operator measures each part multiple times, in random order. The data is analyzed using ANOVA or the range method to separate repeatability and reproducibility.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Collect GRR Data

Run the same measurement on the same set of parts multiple times. Use TofuPilot to log every measurement with the part identifier and trial number.

```python filename="grr_study.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("voltage_reading_V")
    .with_units(units.VOLT),
    htf.Measurement("operator").with_units(units.UNITLESS),
    htf.Measurement("trial").with_units(units.UNITLESS),
)
def phase_grr_measurement(test):
    """Measure voltage for GRR study. Record operator and trial."""
    test.measurements.voltage_reading_V = 5.003
    test.measurements.operator = 1
    test.measurements.trial = 1
```

## Step 2: Log to TofuPilot

Each measurement becomes a test run. The serial number identifies the part. Run each part-operator-trial combination as a separate test execution.

```python filename="grr_study.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_grr_measurement)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan GRR part serial: "))
```

## Step 3: Analyze the Results

After collecting all measurements, export the data from TofuPilot and compute the GRR metrics. The key calculations:

| Metric | Formula |
|--------|---------|
| Repeatability (EV) | Average range within operator x d2 constant |
| Reproducibility (AV) | Range of operator averages x d2 constant, corrected for sample size |
| GRR | Square root of (EV squared + AV squared) |
| GRR % | (GRR / tolerance) x 100 |
| Number of distinct categories (ndc) | (Part variation / GRR) x 1.41 |

An ndc below 5 means the measurement system can't distinguish enough categories of parts. You need at least 5 for a capable system.

## Beyond GRR: Full MSA

MSA includes four additional evaluations beyond GRR:

| Study | What It Measures | When to Run |
|-------|-----------------|-------------|
| Bias | Difference between measured average and known reference | At calibration |
| Linearity | How bias changes across the measurement range | At calibration |
| Stability | How measurements drift over time | Ongoing monitoring |
| Resolution | Smallest increment the system can detect | At setup |

## Using TofuPilot for Ongoing MSA

You don't need to run formal GRR studies constantly. TofuPilot's measurement distribution data gives you ongoing visibility:

- **Measurement distributions** for the same part number show total variation over time
- **Station comparison** shows whether different stations give different results (reproducibility signal)
- **Control charts** detect when measurement drift begins (stability monitoring)
- **Marginal results** flag when measurements are close to limits, which could indicate measurement system issues rather than part issues

When you see unexpected variation in TofuPilot, run a formal GRR study to quantify the source. If the measurement system is the problem, recalibrate or upgrade the instrument before tightening limits.

### What Is AOI, SPI, and AXI in PCB Manufacturing
URL: https://www.tofupilot.com/guides/what-is-aoi-spi-and-axi-with-tofupilot
AOI, SPI and AXI are the automated inspection steps on an SMT line. What each one catches, where it sits, and how to trend the results.

SPI, AOI and AXI are the three automated inspection steps on an SMT line. They look at the board rather than testing it electrically, and together they catch most assembly defects before any power is applied. This guide covers what each one sees, where it belongs in the line, and how to get more out of the data than a pass/fail count.

## The Three Inspections

| Step | Full name | Where it sits | What it sees |
|------|-----------|---------------|--------------|
| **SPI** | Solder Paste Inspection | After paste printing, before placement | Paste volume, area, height, offset, bridging |
| **AOI** | Automated Optical Inspection | After reflow (often also after placement) | Presence, polarity, offset, tombstoning, visible solder defects |
| **AXI** | Automated X-ray Inspection | After reflow | Hidden joints: BGA, QFN, bottom-terminated parts, voiding |

The order is the logic. SPI checks the process input, AOI checks what the process produced where a camera can see it, and AXI checks what a camera cannot see at all.

## SPI: Solder Paste Inspection

Roughly 60-70 % of SMT defects originate at paste print. SPI is where they are cheapest to catch, because nothing has been placed yet and the board can simply be wiped and reprinted.

| Measurement | Typical spec | What a drift means |
|-------------|-------------|--------------------|
| Volume | 80-120 % of aperture | Stencil wear, squeegee pressure, paste viscosity |
| Height | Within ±25 % of stencil thickness | Stencil damage, board support |
| Area | 80-120 % | Aperture clogging |
| Offset X/Y | Under 25 % of pad width | Stencil-to-board alignment |
| Bridging | None | Excess paste, poor gasketing |

SPI is the strongest candidate on the line for [statistical process control](/guides/what-is-spc-statistical-process-control), because it produces a continuous numeric value per pad, thousands per board. A control chart on paste volume shows stencil wear days before AOI starts flagging opens.

## AOI: Automated Optical Inspection

AOI compares the board against a golden image or a rules model.

| Defect class | Detected reliably |
|--------------|-------------------|
| Missing component | Yes |
| Wrong component (visibly different) | Yes |
| Polarity / orientation | Yes, with good lighting and marking |
| Tombstoning, billboarding | Yes |
| Offset, rotation | Yes |
| Solder bridge (visible) | Yes |
| Insufficient / excess solder (visible fillet) | Usually |
| Cold joint under a part body | No |
| BGA / QFN joints | No |

AOI's weakness is a generic one: false calls. A machine tuned to catch every real defect will also stop good boards, and an operator who overrides 50 false calls an hour will eventually wave a real one through. The false-call rate is worth tracking as a number in its own right.

## AXI: Automated X-ray Inspection

AXI sees through the package. It is the only inspection that reaches BGA, QFN and other bottom-terminated joints.

| Measurement | Why it matters |
|-------------|----------------|
| Void percentage | Thermal and mechanical reliability, typically under 25 % for BGA |
| Joint shape and standoff | Cold joints, head-in-pillow, insufficient collapse |
| Open joints | Not visible by any optical method |
| Solder volume per ball | Paste and reflow consistency at the package level |

AXI is slower and more expensive per board than AOI, so many lines sample rather than inspect 100 %, or apply it only to the packages that need it.

## Where They Sit

```
Paste print → SPI → Place → Reflow → AOI → AXI → ICT/FCT
```

Each step is a decision point with its own yield. That is the single most useful thing to get right in the data: one procedure per inspection step, so a drop at SPI is visibly an SPI problem and not blended into a line-wide number.

## Getting the Data Out of the Machines

SPI, AOI and AXI machines all produce results, usually as CSV, XML, or a database row per board. The common mistake is to keep only the verdict. A board that passed SPI at 119 % paste volume is a different board from one that passed at 101 %, and only one of them is telling you the stencil needs cleaning.

The useful pattern is a thin importer per machine: read its output, map the values onto named measurements, and upload one run per board per step.

```python filename="plugs/spi_machine.py"
import csv


class SpiMachine:
    """Reads the SPI machine's CSV export for one board."""

    def __init__(self):
        self._rows = None

    def load(self, path: str):
        with open(path, newline="") as fh:
            self._rows = list(csv.DictReader(fh))

    def volume_stats(self) -> tuple[float, float, float]:
        vols = [float(r["volume_pct"]) for r in self._rows]
        return min(vols), sum(vols) / len(vols), max(vols)

    def offset_max(self) -> float:
        return max(float(r["offset_pct"]) for r in self._rows)

    def bridges(self) -> int:
        return sum(1 for r in self._rows if r["bridge"] == "1")
```

```python filename="phases/spi_import.py"
def spi_import(measurements, spi_machine):
    """One run per board, carrying the numbers rather than the verdict."""
    spi_machine.load("/data/spi/latest.csv")
    lo, mean, hi = spi_machine.volume_stats()

    measurements.paste_volume_min = lo
    measurements.paste_volume_mean = mean
    measurements.paste_volume_max = hi
    measurements.paste_offset_max = spi_machine.offset_max()
    measurements.bridge_count = spi_machine.bridges()
```

```yaml filename="procedure.yaml"
name: SPI Inspection
version: 1.0.0
description: Solder paste inspection results per board

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: spi_machine
    python: plugs.spi_machine:SpiMachine

main:
  - name: SPI Import
    python: phases.spi_import
    measurements:
      - name: Paste Volume Min
        unit: "%"
        validators:
          - operator: ">="
            expected_value: 80
      - name: Paste Volume Mean
        unit: "%"
        validators:
          - operator: ">="
            expected_value: 90
          - operator: "<="
            expected_value: 110
      - name: Paste Volume Max
        unit: "%"
        validators:
          - operator: "<="
            expected_value: 120
      - name: Paste Offset Max
        unit: "%"
        validators:
          - operator: "<="
            expected_value: 25
      - name: Bridge Count
        validators:
          - operator: "=="
            expected_value: 0
```

Run it with `tofupilot run ./procedure.yaml --upload`, triggered by the machine's export or a file watcher.

Record which machine and which stencil produced the board, so a drift has somewhere to point:

```python filename="phases/record_context.py"
def record_context(run):
    run.metadata["spi_machine"] = "SPI-02"
    run.metadata["stencil_id"] = "STN-4417-C"
    run.metadata["line"] = "L3"
```

Do the same for AOI and AXI with their own procedures: component-level defect counts for AOI, void percentage and joint measurements for AXI.

## What the Data Then Tells You

| Question | How the data answers it |
|----------|------------------------|
| Is the stencil wearing? | Control chart on paste volume mean, per stencil ID |
| Which component fails AOI most? | Pareto of AOI defect types |
| Are BGA voids trending up? | Control chart on void percentage |
| Is one line worse than another? | Filter by the `line` metadata |
| Did the new paste lot change anything? | Compare volume distributions before and after |
| Are our SPI limits too tight? | Cpk on paste volume against the current limits |

The last two are only possible because the numbers were stored. A pass/fail column cannot answer either.

## Inspection Is Not Test

A board that passes SPI, AOI and AXI has been inspected, not tested. None of the three applies power or exercises a function. The electrical gates are [ICT and FCT](/guides/ict-vs-fct-vs-flying-probe-when-to-use-each-pcba-test-method), and they catch a different class of problem entirely: a correctly-soldered board with a marginal design or a firmware fault passes all three inspections.

The complementary failure is also worth naming: a board that passes FCT can still have a 40 % BGA void that will fail in the field in eighteen months. That is precisely what AXI is for.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Storing pass/fail only | No SPC, no drift detection, no limit review | Store the measured values |
| One blended inspection yield | A paste problem looks like a placement problem | One procedure per inspection step |
| Ignoring the false-call rate | Operators start waving real defects through | Track false calls as a measurement |
| No machine or stencil ID on the run | A drift has nowhere to point | Record them as run metadata |
| Treating AOI as a substitute for AXI | Hidden joint defects ship | Use AXI for bottom-terminated packages |
| Treating inspection as a substitute for test | Functional and firmware faults ship | Keep ICT/FCT downstream |

### Migrate from NI TestStand to Python
URL: https://www.tofupilot.com/guides/how-to-migrate-from-ni-teststand-to-python-with-tofupilot
Migrate from NI TestStand to Python: map sequences, step types, process models and callbacks, and validate the port before you cut over.

TestStand is a capable test executive. The reasons teams leave it are usually commercial and operational rather than technical: about $4,310 per seat per year, Windows-only, and binary `.seq` files that resist code review. This guide is the concrete migration: what maps to what, what the work actually costs, and how to validate the port before anything is decommissioned.

For the shorter survey of options, see [TestStand alternatives](/guides/teststand-alternatives-with-tofupilot).

## Before You Start: Inventory What You Use

The size of the job is set almost entirely by your step modules.

| What your steps call | Migration effort |
|---|---|
| Python modules | Minimal; they port as-is |
| C/C++ DLLs | Small; call them with `ctypes` |
| .NET assemblies | Small to medium; `pythonnet`, or reimplement |
| LabVIEW VIs doing instrument work | Large; this is the real cost |
| ActiveX / COM | Medium; often easier to reimplement |

Also inventory the parts of TestStand you actually rely on. Most teams use sequences, limit steps and a stock operator interface. Fewer use callbacks in anger, and those are where the surprises live.

## The Concept Map

| TestStand | TofuPilot Framework |
|---|---|
| Sequence file (`.seq`) | `procedure.yaml` |
| MainSequence | `main:` list |
| Step | Entry under `main:` |
| Step module (code) | Phase function in `phases/` |
| Numeric Limit Test | `measurements` with `>=` / `<=` validators |
| Multiple Numeric Limit Test | Several measurements in one phase |
| String Value Test | `validators` with `operator: "=="` |
| Pass/Fail Test | Boolean measurement, `== true` |
| Action step | Phase with no measurements |
| Precondition | `enable_if` on the phase |
| Run mode: Skip | `enable_if: false` |
| Loop / retry on failure | `retry` on the phase |
| Sequential model | Default execution |
| Parallel / Batch model | `execution.slots` |
| UUT serial number | `unit.serial_number` |
| PreUUT / PostUUT | `setup` / `teardown` phases |
| Station globals | Plug state or run metadata |
| StationInfo, step results | Run metadata and measurements |
| Database logging step | Automatic upload |
| Report generation | Dashboard and exports |

## Porting a Sequence

A typical TestStand MainSequence: power up, three Numeric Limit Tests, a String Value Test, power down.

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0
description: Ported from FCT_PCBA100.seq rev D

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: psu
    python: plugs.psu:PowerSupply
  - name: dmm
    python: plugs.dmm:Multimeter
  - name: dut
    python: plugs.dut:Dut

main:
  - name: Power Up
    python: phases.power_up

  - name: Rail Measurements
    python: phases.rail_measurements
    depends_on: [power_up]
    measurements:
      - name: Rail 3V3
        unit: V
        description: FCT_PCBA100.seq step 3
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Rail 5V0
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
      - name: Quiescent Current
        unit: mA
        validators:
          - operator: "<="
            expected_value: 45.0

  - name: Firmware Version
    python: phases.firmware_version
    depends_on: [rail_measurements]
    measurements:
      - name: Firmware Version
        validators:
          - operator: "=="
            expected_value: "2.1.0"
```

```python filename="phases/power_up.py"
def power_up(psu):
    """Action step: no measurements, just side effects."""
    psu.configure(channel=1, volts=5.0, current_limit=0.5)
    psu.output(1, True)
```

```python filename="phases/rail_measurements.py"
def rail_measurements(measurements, dmm):
    """Three Numeric Limit Tests become three measurements in one phase."""
    measurements.rail_3v3 = dmm.dc_voltage()
    measurements.rail_5v0 = dmm.dc_voltage()
    measurements.quiescent_current = dmm.dc_current_ma()
```

```python filename="phases/firmware_version.py"
def firmware_version(measurements, dut):
    measurements.firmware_version = dut.query_firmware()
```

Keeping the original step reference in `description` is worth the small effort. During validation you will be comparing against the TestStand report line by line, and it makes that mechanical.

Power-down belongs in the plug's `close()`, not a final phase, so it happens even when a phase raises:

```python filename="plugs/psu.py"
class PowerSupply:
    def configure(self, channel: int, volts: float, current_limit: float):
        ...

    def output(self, channel: int, on: bool):
        ...

    def close(self):
        """Runs at teardown whatever happened during the run."""
        for ch in (1, 2, 3):
            self.output(ch, False)
```

## Porting the Process Models

**Sequential** is the default. Nothing to do.

**Parallel and Batch** become fixture slots:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: Slot 1
    - name: Slot 2
    - name: Slot 3
    - name: Slot 4
```

Each slot runs independently with its own unit, measurements and outcome, and a failure in one slot leaves the others running. For a fixture with many identical positions, a count works instead of a list:

```yaml filename="procedure.yaml"
execution:
  slots: 16
```

## Porting the Callbacks

PreUUT and PostUUT become setup and teardown phases. The distinction that matters, and that TestStand also forces you to think about, is scope:

- **Slot-scoped** teardown touches that slot only and may run while neighbouring slots are still testing.
- **Execution-scoped** teardown runs once after every slot is finished, and is where a shared rack supply, a fixture lock or a chamber belongs.

The classic multi-slot bug is a slot-scoped teardown switching off a shared supply while another slot is mid-measurement. See [replace TestStand callbacks in Python](/guides/replace-teststand-callbacks-in-python).

## Calling Existing Code

You rarely have to rewrite working instrument code immediately.

**C/C++ DLL:**

```python filename="plugs/legacy_dll.py"
import ctypes


class LegacyInstrument:
    def __init__(self):
        self._lib = ctypes.CDLL("./vendor/instrument.dll")
        self._lib.read_voltage.restype = ctypes.c_double

    def read_voltage(self) -> float:
        return float(self._lib.read_voltage())
```

**LabVIEW VI**, via a built executable or the LabVIEW Python API, wrapped the same way. This is the step worth measuring honestly: if most of your instrument logic is in VIs, that is the migration, and the sequencing part is the easy half.

## Validate Before You Cut Over

Non-negotiable, and cheap:

1. Pick 20-30 units including known-bad ones.
2. Run TestStand and the new procedure on each.
3. Compare measurement by measurement, not just pass/fail.
4. Investigate every disagreement before decommissioning anything.

A limit transcribed with the wrong sign or a factor of ten is the failure mode that ships bad boards, and it is invisible if you only compare verdicts.

Keep measurement names stable from day one of the port. They are the identity of the data across every run you will ever take.

## What It Costs

| Work item | Effort |
|---|---|
| Port step modules | Small if C/Python, large if LabVIEW |
| Re-express sequences as procedures | Small, mostly mechanical |
| Rebuild the operator interface | Small; declared in the procedure |
| Replace database logging | Small; upload replaces the schema |
| Parallel validation | Medium, and mandatory |
| Retraining | Small; Python is a smaller step than TestStand was |

## When Not to Migrate

- **Deep NI hardware integration.** PXI chassis, switching, FPGA targets and NI-specific drivers you would have to rewrite.
- **Large working sequence library, no new stations.** If it works and is not growing, the cost may not be justified.
- **Heavy LabVIEW step modules with no budget to port them.**

## If You Prefer Decorators

OpenHTF is the other purpose-built Python option, with sequencing and limits expressed in Python decorators rather than YAML:

```python filename="fct.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
def test_rail(test):
    test.measurements.rail_3v3 = 3.31


test = htf.Test(test_rail, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`. Both run on the same CLI and upload to the same dashboard. See [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf).

## Common Migration Mistakes

| Mistake | Consequence | Fix |
|---|---|---|
| Porting sequences before step modules | Nothing runs end to end | Port and test modules first |
| Comparing pass/fail only during validation | A wrong limit survives the port | Compare measurement values |
| Recreating TestStand structure exactly | You inherit workarounds you no longer need | Port the intent |
| Power-down as a final phase | Board stays energised when a phase raises | Put it in the plug's `close()` |
| Slot-scoped teardown on shared equipment | Intermittent multi-slot failures | Execution scope for shared resources |
| Big-bang cutover | One risky day | One station, one product, then expand |

### Calculate True First Pass Yield
URL: https://www.tofupilot.com/guides/how-to-calculate-true-first-pass-yield-with-tofupilot
Learn the FPY formula, how it differs from rework yield and rolled throughput yield, and how TofuPilot tracks it automatically.

First pass yield (FPY) is the percentage of units that pass testing on their first attempt, without rework or retesting. It's the single most important metric for understanding your production quality, and most teams measure it wrong.

## The FPY Formula

FPY = (units passing on first test) / (total unique units tested)

If you test 200 boards today and 180 pass on the first attempt, your FPY is 90%. The 20 that failed, even if they pass after rework and retest, don't count toward FPY.

This distinction matters. A line reporting 98% "yield" might actually have 85% FPY with the rest passing only after rework. Those numbers tell very different stories about process health.

## FPY vs. Other Yield Metrics

Several yield metrics exist, and confusing them leads to bad decisions.

| Metric | Formula | What it measures |
|--------|---------|------------------|
| **First pass yield (FPY)** | Pass on 1st attempt / Total units | Process capability at the test station |
| **Rework yield** | Pass after rework / Total reworked | Effectiveness of your rework process |
| **Output yield** | Total passing (including retests) / Total units | Final throughput, hides rework cost |
| **Rolled throughput yield (RTY)** | FPY₁ x FPY₂ x ... x FPYₙ | Probability a unit passes all stations without rework |

Output yield is the number most teams report. It looks good on slides. But it hides the cost of rework, the strain on test capacity, and the risk of shipping marginal units.

RTY is the most demanding metric. If you have three test stations each with 95% FPY, your RTY is 0.95 x 0.95 x 0.95 = 85.7%. That means only 86 out of 100 units flow through your entire line without touching rework.

## Writing Tests That Produce Accurate FPY Data

FPY accuracy depends on your test scripts producing clean, structured results. Every run needs a serial number so TofuPilot can distinguish first attempts from retests of the same unit.

```python filename="board_functional_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("supply_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=4.75, maximum=5.25),
    htf.Measurement("clock_frequency")
    .in_range(minimum=7.99, maximum=8.01),
    htf.Measurement("standby_current")
    .in_range(maximum=50),
)
def test_power_and_clock(test):
    test.measurements.supply_voltage = 5.03
    test.measurements.clock_frequency = 8.002
    test.measurements.standby_current = 32

@htf.measures(
    htf.Measurement("comms_loopback_pass"),
)
def test_communications(test):
    test.measurements.comms_loopback_pass = True

def main():
    test = htf.Test(test_power_and_clock, test_communications)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-2026-00842")

if __name__ == "__main__":
    main()
```

When this test runs, TofuPilot records the serial number, all measurements with their limits, and the pass/fail outcome. If `SN-2026-00842` fails and gets retested later, TofuPilot knows it's a retest, not a new unit.

## How TofuPilot Tracks FPY Automatically

You don't need to compute FPY yourself. TofuPilot's analytics dashboard calculates it in real time from your test data.

What the dashboard provides:

- **FPY trend over time.** See daily, weekly, or monthly FPY for any product or station. Spot drops the day they happen, not weeks later in a quarterly review.
- **FPY by station.** Compare stations running the same test. If Station 3 has 88% FPY while Station 1 and 2 are at 96%, the problem is the station, not the product.
- **FPY by product revision.** Track whether a new board revision actually improved yield or made it worse.
- **Unit history.** For any serial number, see every test attempt. Understand whether failures cluster on specific units or spread evenly.

## Common FPY Pitfalls

**Counting retests as first attempts.** If your system doesn't track serial numbers, every test looks like a first attempt. Your FPY will be artificially low because failures and their retests both count as unique units.

**Excluding known-bad units.** Some teams exclude units that "obviously" failed due to fixture issues or operator error. This inflates FPY and hides real problems.

**Measuring too late.** FPY at final test captures problems from every upstream process. Measuring FPY at each station (and computing RTY) gives you much better isolation.

**Setting limits too tight.** If your 3-sigma process capability doesn't fit inside your test limits, you'll see chronic low FPY even with a good process. Check your Cpk before blaming the line.

### EMC Pre-Compliance Testing: A Complete Guide
URL: https://www.tofupilot.com/guides/emc-pre-compliance-testing-a-complete-guide
Learn how to set up EMC pre-compliance tests, log radiated and conducted emissions data, and track EMC margins with TofuPilot.

Failing EMC certification costs weeks of delay and tens of thousands in retest fees. Pre-compliance testing in your own lab catches problems early, when they're cheap to fix. TofuPilot tracks your EMC test data so you can monitor margins, compare design revisions, and build confidence before the formal test.

## What EMC Testing Covers

| Test | Standard | What it measures |
|------|----------|-----------------|
| Radiated emissions | CISPR 32, FCC Part 15 | RF energy radiated from the product |
| Conducted emissions | CISPR 32, FCC Part 15 | RF noise on power lines |
| Radiated immunity | IEC 61000-4-3 | Resistance to external RF fields |
| ESD immunity | IEC 61000-4-2 | Resistance to electrostatic discharge |
| Surge immunity | IEC 61000-4-5 | Resistance to power line surges |
| Conducted immunity | IEC 61000-4-6 | Resistance to RF on cables |

Pre-compliance focuses on emissions (radiated and conducted) because they're the most common failure modes and can be tested with relatively affordable equipment.

## Pre-Compliance Equipment

| Equipment | Purpose | Typical cost |
|-----------|---------|-------------|
| Near-field probe set | Locate emission sources on PCB | $500-2,000 |
| Spectrum analyzer | Measure frequency content | $5,000-30,000 |
| LISN (Line Impedance Stabilization Network) | Conducted emissions measurement | $2,000-5,000 |
| EMC antenna (biconical + log-periodic) | Radiated emissions measurement | $3,000-8,000 |
| Shielded enclosure or open area test site | Controlled measurement environment | $10,000+ |

You don't need a fully equipped chamber for pre-compliance. A spectrum analyzer with near-field probes catches most issues at the board level.

## Logging EMC Data to TofuPilot

### Radiated Emissions Scan

```python filename="emc_radiated_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Peak emissions at key frequencies
emissions = [
    {"freq_mhz": 30, "level_dbuv_m": 28.5, "limit_dbuv_m": 40.0},
    {"freq_mhz": 100, "level_dbuv_m": 32.1, "limit_dbuv_m": 43.5},
    {"freq_mhz": 230, "level_dbuv_m": 35.8, "limit_dbuv_m": 46.0},
    {"freq_mhz": 500, "level_dbuv_m": 22.3, "limit_dbuv_m": 46.0},
    {"freq_mhz": 1000, "level_dbuv_m": 18.7, "limit_dbuv_m": 50.0},
]

measurements = []
all_pass = True
for e in emissions:
    margin = e["limit_dbuv_m"] - e["level_dbuv_m"]
    passed = margin > 0
    if not passed:
        all_pass = False
    measurements.append({
        "name": f"radiated_{e['freq_mhz']}mhz",
        "value": e["level_dbuv_m"],
        "unit": "dBuV/m",
        "limit_high": e["limit_dbuv_m"],
    })
    measurements.append({
        "name": f"margin_{e['freq_mhz']}mhz",
        "value": margin,
        "unit": "dB",
        "limit_low": 6.0,  # 6dB margin target
    })

client.create_run(
    procedure_id="EMC-RADIATED-PRECOMPLIANCE",
    unit_under_test={
        "serial_number": "PROTO-REV-C",
        "part_number": "PRODUCT-V2",
    },
    run_passed=all_pass,
    steps=[{
        "name": "Radiated Emissions 30MHz-1GHz",
        "step_type": "measurement",
        "status": all_pass,
        "measurements": measurements,
    }],
)
```

### Conducted Emissions

```python filename="emc_conducted_test.py"
# Conducted emissions on power input lines
conducted = [
    {"freq_mhz": 0.15, "level_dbuv": 52.3, "limit_avg": 56.0, "limit_qp": 66.0},
    {"freq_mhz": 0.5, "level_dbuv": 48.7, "limit_avg": 46.0, "limit_qp": 56.0},
    {"freq_mhz": 5.0, "level_dbuv": 38.2, "limit_avg": 46.0, "limit_qp": 56.0},
    {"freq_mhz": 30.0, "level_dbuv": 32.1, "limit_avg": 46.0, "limit_qp": 56.0},
]

measurements = []
for c in conducted:
    measurements.append({
        "name": f"conducted_avg_{c['freq_mhz']}mhz",
        "value": c["level_dbuv"],
        "unit": "dBuV",
        "limit_high": c["limit_avg"],
    })

client.create_run(
    procedure_id="EMC-CONDUCTED-PRECOMPLIANCE",
    unit_under_test={"serial_number": "PROTO-REV-C"},
    run_passed=True,
    steps=[{
        "name": "Conducted Emissions 150kHz-30MHz",
        "step_type": "measurement",
        "status": True,
        "measurements": measurements,
    }],
)
```

## Tracking EMC Margins

The goal of pre-compliance isn't just pass/fail. It's knowing your margin. A product that passes with 2dB of margin will likely fail at the test house (measurement uncertainty alone is 3-5dB).

| Margin | Risk level | Action |
|--------|-----------|--------|
| > 10 dB | Low | Proceed to certification |
| 6-10 dB | Medium | Proceed, but have mitigation plans ready |
| 3-6 dB | High | Fix before certification |
| < 3 dB | Very high | Will likely fail certification |

Track margins in TofuPilot across design revisions. If Rev A had 4dB margin at 230MHz and Rev B has 8dB after adding a filter, you can see the improvement quantitatively.

## Comparing Design Revisions

EMC performance changes with every board revision, layout change, and component swap. TofuPilot lets you compare emissions data across revisions side by side.

```
Rev A (no filter):          Rev B (added pi filter):
┌─────────────────────┐    ┌─────────────────────┐
│ 230MHz: 42.1 dBuV/m │    │ 230MHz: 35.8 dBuV/m │
│ Limit:  46.0 dBuV/m │    │ Limit:  46.0 dBuV/m │
│ Margin: 3.9 dB  ⚠   │    │ Margin: 10.2 dB  ✓  │
└─────────────────────┘    └─────────────────────┘
```

The pi filter bought 6.3dB of margin at 230MHz. This kind of data-driven design feedback is only possible when you track every test iteration.

## Common EMC Failure Modes

| Failure | Typical cause | Fix |
|---------|--------------|-----|
| Harmonics of switching frequency | DC-DC converter, clock oscillator | Add input/output filtering, spread-spectrum clocking |
| Cable-radiated emissions | Unshielded cables acting as antennas | Add common-mode chokes, use shielded cables |
| High-frequency broadband | Fast digital edges, poor grounding | Slow edge rates, improve ground plane |
| Low-frequency conducted | Switching power supply ripple | Add differential-mode filter on input |

Track which failure modes appear across products in TofuPilot. If every product fails at the 3rd harmonic of the switching frequency, standardize your input filter design.

## ESD Testing

ESD is the most common immunity test failure. Log ESD test results systematically.

```python filename="esd_test.py"
esd_levels = [
    {"mode": "contact", "voltage_kv": 4, "result": "pass"},
    {"mode": "contact", "voltage_kv": 6, "result": "pass"},
    {"mode": "contact", "voltage_kv": 8, "result": "fail_recoverable"},
    {"mode": "air", "voltage_kv": 8, "result": "pass"},
    {"mode": "air", "voltage_kv": 15, "result": "pass"},
]

measurements = []
for test in esd_levels:
    passed = test["result"] == "pass"
    measurements.append({
        "name": f"esd_{test['mode']}_{test['voltage_kv']}kv",
        "value": 1 if passed else 0,
        "unit": "pass/fail",
        "limit_low": 1,
    })

client.create_run(
    procedure_id="EMC-ESD-PRECOMPLIANCE",
    unit_under_test={"serial_number": "PROTO-REV-C"},
    run_passed=all(t["result"] == "pass" for t in esd_levels),
    steps=[{
        "name": "ESD Immunity per IEC 61000-4-2",
        "step_type": "measurement",
        "status": True,
        "measurements": measurements,
    }],
)
```

## From Pre-Compliance to Certification

Pre-compliance data in TofuPilot serves as your engineering notebook for EMC. When you arrive at the test house:

- You know your margins at every frequency
- You know which design changes improved or degraded performance
- You have a history of every test iteration
- If you fail, you have data to guide the fix instead of starting from scratch


### CI/CD for Hardware Testing
URL: https://www.tofupilot.com/guides/cicd-for-hardware-testing-with-tofupilot
Learn how to integrate hardware test automation into CI/CD pipelines using TofuPilot for continuous validation and gating.

Software teams have had CI/CD for years. Hardware teams still gate releases with spreadsheets and manual sign-offs. TofuPilot bridges that gap by treating hardware test results as pipeline artifacts, so you can automate validation the same way you automate builds.

## Why CI/CD Matters for Hardware

Hardware test data is the equivalent of a software test suite. Every unit that comes off the line runs through functional tests, calibration checks, and environmental screens. The question isn't whether you have tests. It's whether you can automatically block a shipment when test data says "stop."

Traditional hardware workflows rely on someone reviewing a report. CI/CD for hardware means the pipeline reviews the data and makes the call.

## Architecture

```
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Test Station │────▶│   TofuPilot  │────▶│  CI/CD Gate  │
│  (OpenHTF /   │     │   (Results   │     │  (Pass/Fail  │
│   pytest)     │     │    + API)    │     │   Decision)  │
└──────────────┘     └──────────────┘     └──────────────┘
```

Test stations push results to TofuPilot. Your CI/CD system queries TofuPilot's API to check whether a batch meets release criteria.

## Setting Up Continuous Hardware Validation

### Step 1: Push Results from Test Stations

Every test run uploads to TofuPilot automatically. If you're using OpenHTF, add the TofuPilot output callback. If you're using pytest or a custom framework, use the Python client.

```python filename="station_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

client.create_run(
    procedure_id="EVT-POWER-CYCLE",
    unit_under_test={"serial_number": "UNIT-4521"},
    run_passed=True,
    steps=[{
        "name": "Power Rail Check",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "vcc_3v3",
            "value": 3.31,
            "unit": "V",
            "limit_low": 3.25,
            "limit_high": 3.35,
        }],
    }],
)
```

### Step 2: Query Results in Your Pipeline

Use TofuPilot's API to check batch-level pass rates before allowing a release to proceed.

```python filename="ci_gate.py"
# CI/CD gate script: check batch FPY before release
import requests
import sys

API_KEY = os.environ["TOFUPILOT_API_KEY"]
PROCEDURE = "EVT-POWER-CYCLE"
MIN_FPY = 0.95
MIN_UNITS = 50

response = requests.get(
    "https://app.tofupilot.com/api/v1/runs",
    headers={"Authorization": f"Bearer {API_KEY}"},
    params={"procedure_id": PROCEDURE, "limit": 200},
)

runs = response.json()
passed = sum(1 for r in runs if r["run_passed"])
total = len(runs)
fpy = passed / total if total > 0 else 0

print(f"FPY: {fpy:.1%} ({passed}/{total} units)")

if total < MIN_UNITS:
    print(f"Not enough units tested ({total} < {MIN_UNITS})")
    sys.exit(1)

if fpy < MIN_FPY:
    print(f"FPY below threshold ({fpy:.1%} < {MIN_FPY:.0%})")
    sys.exit(1)

print("Gate passed. Batch approved for release.")
```

### Step 3: Integrate with GitHub Actions or GitLab CI

```yaml filename=".github/workflows/hardware-gate.yml"
name: Hardware Release Gate
on:
  workflow_dispatch:
    inputs:
      batch_id:
        description: "Batch to validate"
        required: true

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install requests
      - run: python ci_gate.py
        env:
          TOFUPILOT_API_KEY: ${{ secrets.TOFUPILOT_API_KEY }}
```

## Gating Strategies

| Strategy | When to use | How it works |
|----------|-------------|--------------|
| FPY threshold | Production release | Block if first-pass yield drops below target |
| Zero critical failures | Safety-critical products | Block if any critical measurement fails |
| Cpk minimum | Process validation | Block if process capability index is too low |
| Trend detection | Early warning | Alert if yield is declining over last N batches |

## Test Like You Fly

The aerospace principle of "test like you fly" means production tests should mirror real operating conditions. CI/CD for hardware makes this practical by automating the feedback loop:

1. Run the same tests in EVT, DVT, and PVT
2. Compare results across phases in TofuPilot
3. Gate each phase transition on measured data, not opinions

When your DVT thermal cycling shows a 3% yield drop compared to EVT, TofuPilot's comparison dashboards surface it immediately. No waiting for someone to pull a report.

## What This Replaces

Most hardware teams gate releases with a combination of Excel reports, email chains, and meeting sign-offs. CI/CD for hardware doesn't eliminate human judgment. It eliminates the manual data gathering that delays judgment. Engineers spend time analyzing instead of aggregating.


### What Is Zero-Defect Manufacturing
URL: https://www.tofupilot.com/guides/what-is-zero-defect-manufacturing
Zero-defect manufacturing aims to prevent defects rather than detect them. Learn what it means in practice, what it requires, and how test data supports it.

Zero-defect manufacturing (ZDM) is the principle that defects should be prevented, not just detected. Instead of accepting a defect rate and managing it, ZDM aims to eliminate defects at their source. This guide covers what zero-defect manufacturing means in practice, how it differs from traditional quality approaches, and how test data supports the goal.

## The Zero-Defect Principle

Philip Crosby introduced the zero-defect concept in the 1960s. The core idea: the performance standard should be zero defects, not "an acceptable quality level."

| Traditional Approach | Zero-Defect Approach |
|---------------------|---------------------|
| "Some defects are inevitable" | "Every defect has a preventable root cause" |
| Accept AQL of 0.1-1.0% | Target 0% defect rate |
| Focus on detection and sorting | Focus on prevention and process control |
| Quality is a cost center | Quality is a savings generator |
| Inspect quality in | Build quality in |

Zero defects doesn't mean perfection is achieved on day one. It means perfection is the target, and every defect is treated as a process failure to be investigated and corrected, not as an acceptable cost of doing business.

## Zero Defects in Practice

No production line truly has zero defects. The practical question is: how close can you get, and what does each step closer cost?

| Defect Rate | FPY | DPMO | Sigma Level | Typical Industry |
|------------|-----|------|-------------|-----------------|
| 10% | 90% | 100,000 | 2.8 | Early production, new product |
| 1% | 99% | 10,000 | 3.8 | Established consumer electronics |
| 0.1% | 99.9% | 1,000 | 4.6 | Automotive components |
| 0.01% | 99.99% | 100 | 5.2 | Medical devices, aerospace |
| 0.00034% | 99.99966% | 3.4 | 6.0 | Six Sigma target |

Moving from 99% to 99.9% FPY is harder than moving from 90% to 99%. Each order of magnitude requires deeper process understanding and more sophisticated detection.

## The Four Pillars of ZDM

### 1. Prevention

Design the product and process so defects can't occur.

| Prevention Method | Example |
|------------------|---------|
| Design for manufacturability (DFM) | Wider solder pads, larger component clearances |
| Poka-yoke (mistake-proofing) | Connectors that only fit one way |
| Process capability studies | Verify Cpk > 1.33 before production |
| Supplier qualification | Qualify components and vendors before use |

### 2. Detection

Catch defects as early as possible when prevention fails.

| Detection Method | Stage |
|-----------------|-------|
| SPI, AOI, AXI | In-process inspection |
| Functional test | After assembly |
| EOL test | Before shipping |
| ORT sampling | Ongoing reliability |

### 3. Prediction

Use data to forecast defects before they occur.

| Prediction Method | Data Source |
|------------------|-----------|
| SPC control charts | Process parameter trends |
| Measurement drift detection | Test result trends |
| Supplier quality tracking | IQC rejection rates |
| ML-based prediction | Historical test data correlations |

### 4. Correction

When defects occur, fix the root cause permanently.

| Correction Method | Purpose |
|------------------|---------|
| Root cause analysis | Find the true cause, not the symptom |
| CAPA | Document corrective and preventive actions |
| Process change validation | Verify the fix works without introducing new defects |
| Limit refinement | Tighten or add detection for the specific failure mode |

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## How Test Data Supports Zero Defects

Structured test data is the foundation of all four pillars. Without data, prevention is guesswork, detection is incomplete, prediction is impossible, and correction is temporary.

```python filename="zdm_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(
        minimum=4.9, maximum=5.1,
        marginal_minimum=4.92, marginal_maximum=5.08,
    )
    .with_units(units.VOLT),
    htf.Measurement("current_draw_mA")
    .in_range(minimum=90, maximum=110)
    .with_units(units.MILLIAMPERE),
)
def phase_production_test(test):
    """Production test with marginal bands for early warning."""
    test.measurements.output_voltage_V = 5.01
    test.measurements.current_draw_mA = 99.5
```

Marginal limits are important for ZDM. A unit that passes within the marginal band technically passes, but it's trending toward a failure. TofuPilot flags marginal results separately, giving you an early warning before defects start escaping.

```python filename="zdm_test.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_production_test)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

TofuPilot supports zero-defect goals by providing:

- **Measurement distributions** showing how close you are to limits
- **Marginal result tracking** for early warning
- **Failure Pareto** prioritizing which defects to eliminate next
- **Control charts** detecting process drift before it produces defects
- **Yield trends** measuring progress toward zero-defect targets
- **Root cause data** correlating failures with process variables

## The Cost of Zero Defects

| Investment | Return |
|-----------|--------|
| Better incoming inspection | Fewer defective components in assembly |
| In-process monitoring (SPI, AOI) | Catch defects before value is added |
| Structured test data collection | Enable prediction and correlation analysis |
| Root cause investigation for every failure | Permanent fixes, not band-aids |
| Marginal limit monitoring | Catch drift before it becomes failure |

Zero-defect manufacturing is a direction, not a destination. Every defect prevented is cheaper than a defect detected, which is cheaper than a defect shipped. The test system's job is to generate the data that makes prevention possible.

### Automated Verification and Validation
URL: https://www.tofupilot.com/guides/automated-verification-and-validation-with-tofupilot
Learn how to automate hardware verification and validation workflows using TofuPilot for structured test evidence and traceability.

Verification and validation (V&V) in hardware is still largely manual: engineers run tests, record results in documents, and someone reviews the evidence. TofuPilot automates the data capture side so engineers can focus on the analysis instead of the paperwork.

## V&V in Hardware Development

Verification answers: "Did we build the product right?" (Does it meet its specifications?)
Validation answers: "Did we build the right product?" (Does it meet user needs?)

Both require structured test evidence. In regulated industries (medical devices, aerospace, automotive), this evidence must be traceable, timestamped, and immutable.

| Phase | Purpose | Typical tests | Evidence needed |
|-------|---------|--------------|----------------|
| EVT | Engineering validation | Functional, environmental, reliability | Test reports, measurement data |
| DVT | Design verification | Full spec compliance, regulatory | Formal test records, traceability |
| PVT | Production validation | Manufacturing process qualification | Statistical data, Cpk analysis |
| Production | Ongoing verification | In-line test, final test | Per-unit records, yield data |

## The Manual V&V Problem

Traditional V&V workflows look like this:

1. Engineer writes a test protocol (Word document)
2. Technician runs the tests manually or semi-automatically
3. Results are recorded in a spreadsheet or paper form
4. Engineer reviews results and writes a test report
5. Quality reviews and approves the report
6. Documents are filed in a QMS

This process is slow, error-prone, and disconnected from the actual test data. The spreadsheet is a copy of the data, not the data itself. If someone transcribes a number wrong, the V&V record is wrong.

## Automating V&V with TofuPilot

### Step 1: Define Your Verification Test Procedures

Create a procedure in TofuPilot for each verification test in your protocol.

```python filename="evt_thermal.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# EVT thermal cycling test
client.create_run(
    procedure_id="EVT-THERMAL-CYCLE",
    unit_under_test={
        "serial_number": "EVT-PROTO-003",
        "part_number": "PRODUCT-V2.1",
    },
    run_passed=True,
    steps=[{
        "name": "Post-Cycle Functional Check",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {
                "name": "output_voltage",
                "value": 5.02,
                "unit": "V",
                "limit_low": 4.90,
                "limit_high": 5.10,
            },
            {
                "name": "output_ripple_mv",
                "value": 12.3,
                "unit": "mV",
                "limit_high": 50.0,
            },
            {
                "name": "efficiency_pct",
                "value": 91.2,
                "unit": "%",
                "limit_low": 88.0,
            },
        ],
    }, {
        "name": "Thermal Performance",
        "step_type": "measurement",
        "status": True,
        "measurements": [
            {
                "name": "junction_temp_max",
                "value": 82.4,
                "unit": "°C",
                "limit_high": 105.0,
            },
            {
                "name": "thermal_resistance",
                "value": 3.8,
                "unit": "°C/W",
                "limit_high": 5.0,
            },
        ],
    }],
)
```

### Step 2: Map Requirements to Measurements

Each measurement in TofuPilot corresponds to a requirement in your spec. Use consistent naming to create a clear mapping.

| Requirement ID | Requirement | TofuPilot Measurement | Limit |
|---------------|-------------|----------------------|-------|
| REQ-ELEC-001 | Output voltage: 5V +/- 2% | `output_voltage` | 4.90-5.10 V |
| REQ-ELEC-002 | Ripple: < 50mV | `output_ripple_mv` | < 50 mV |
| REQ-ELEC-003 | Efficiency: > 88% | `efficiency_pct` | > 88% |
| REQ-THERM-001 | Junction temp: < 105C | `junction_temp_max` | < 105 C |

This mapping turns your test data into verification evidence. Every measurement in TofuPilot is a timestamped, immutable record that a requirement was tested.

### Step 3: Run Tests Across V&V Phases

Use the same procedures across EVT, DVT, and PVT. TofuPilot stores all results, so you can compare performance across phases.

```
EVT (3 prototypes):     DVT (30 units):         PVT (300 units):
├── EVT-THERMAL         ├── DVT-THERMAL          ├── PVT-THERMAL
├── EVT-VIBRATION       ├── DVT-VIBRATION        ├── PVT-PROCESS-QUAL
├── EVT-FUNCTIONAL      ├── DVT-EMC              └── PVT-YIELD-ANALYSIS
└── EVT-RELIABILITY     ├── DVT-SAFETY
                        └── DVT-FUNCTIONAL
```

### Step 4: Generate V&V Evidence

TofuPilot's stored test data serves as your verification evidence:

- **Per-unit test records**: Every measurement for every unit, with timestamps and pass/fail status
- **Statistical summaries**: Cpk, mean, standard deviation, distribution across the test population
- **Traceability**: Link from requirement to test procedure to test result to specific unit
- **Trend data**: How measurements changed across EVT, DVT, and PVT phases

## Traceability for Audits

When an auditor asks "Show me the evidence that REQ-ELEC-001 was verified," you can:

1. Open TofuPilot
2. Filter by procedure (e.g., `DVT-FUNCTIONAL`)
3. Show all `output_voltage` measurements across the DVT population
4. Show the pass/fail status, limits, and statistical summary

No digging through binders. No searching shared drives. The data is structured, searchable, and timestamped.

## Comparing V&V Across Design Revisions

When you make a design change between EVT and DVT, TofuPilot lets you compare the same measurements across phases.

Did the thermal performance improve with the new heatsink design? Compare `junction_temp_max` distributions between EVT and DVT. The data answers the question objectively.

## Continuous Verification in Production

V&V doesn't end at PVT. Every unit tested in production is a data point that verifies the design continues to meet its specifications.

TofuPilot's production dashboards show ongoing verification:
- Are all measurements still within limits?
- Is the process capability (Cpk) maintained?
- Are there any new failure modes that weren't seen during V&V?

If a measurement that was comfortably within limits during DVT starts approaching a limit in production, you know the process or components have changed. Investigate before it becomes a field issue.


### Why LabVIEW Version Control Is So Painful
URL: https://www.tofupilot.com/guides/why-labview-version-control-is-so-painful
Learn why Git can store a LabVIEW VI but not diff, merge or blame it, what NI's tooling fixes and where it stops, and what a text station changes.

LabVIEW version control is painful because a VI is a binary file. Git stores it, versions it and reverts it, but it can't show you a line diff, can't merge two people's edits to the same VI, and can't tell you who changed a limit. NI's tooling narrows that gap, one licensed Professional seat at a time.

## What Git Gives a Text File and Not a VI

Git was built for text. Every feature you expect from source control on a Python file follows from Git being able to read lines, and a `.vi` gives it none to read.

| Capability | `phases/power_rails.py` | `Power Rails.vi` |
|---|---|---|
| Store and revert a version | Yes | Yes |
| Line diff (`git diff`) | Yes | No, Git prints "Binary files differ" |
| Blame (`git blame`) on one limit | Yes | No |
| Three-way merge of two edits | Yes, automatic when the edits don't overlap | No, Git stops with a conflict |
| Review in a pull request on the web | Yes | No, the reviewer downloads both versions and opens them |
| Works on any machine | Yes | Only with the same LabVIEW version installed |
| Licence needed to review | None | A LabVIEW seat, Professional for LVCompare |

The right column isn't a Git setting you can change. Git can't merge what it can't parse, and a block diagram is a graph serialised in a format only LabVIEW reads. The same holds for `.ctl`, `.lvclass` and `.lvlib` contents, and for a TestStand `.seq` (binary by default, and the XML save option diffs a little better but still isn't something you'd merge by hand).

## The Four Sources of Pain

Most "LabVIEW and Git" problems reduce to four mechanisms. Each has its own guide in this cluster. This section is the map.

### Binary VIs

A commit message is the only readable record of what changed in a VI. If the message says "fixed limits", that's all anyone will ever know about that commit, including you in six months. [LVCompare and LVMerge](/guides/lvcompare-and-lvmerge-what-they-cannot-do) show the diagram diff, on a machine that has them installed and licensed.

### One Person per VI at a Time

Because Git can't merge two edits to the same VI, two engineers who touch the same file on two branches produce a conflict Git can't resolve on its own. The usual answer is a convention: you say which VI you're in, and nobody else opens it until you push. It holds on a team of two. [Fix LabVIEW merge conflicts on binary VIs](/guides/fix-labview-merge-conflicts-on-binary-vis) covers what to do when it slips.

### Compiled Code and Mass Compile

By default a VI file holds its source and its compiled code together. Recompile a caller because a callee's connector pane changed, and the caller's file changes on disk even though you edited nothing in it. Separating compiled code from the source file (a setting since LabVIEW 2010) moves the compiled part to a local object cache and removes most of that noise.

It doesn't remove all of it. A mass compile, which you run after any LabVIEW version or bitness change, rewrites every VI it touches, so the whole repository shows as modified in one commit and the history on either side of it no longer diffs cleanly. [Separate compiled code in LabVIEW, explained](/guides/separate-compiled-code-in-labview-explained) covers the setting and its limits.

### A Run-Time Engine per Station

A built executable runs only on the LabVIEW Run-Time Engine of the same version and bitness. A station on RTE 2019 32-bit won't start a build from LabVIEW 2024 64-bit until someone installs that RTE, and the DAQmx and VISA drivers on that station have their own supported version ranges. So "which version of the test runs on station 3" has two answers: the commit the build came from, and the RTE and drivers that build needs. [Which LabVIEW version runs on each station](/guides/which-labview-version-runs-on-each-station) is the inventory.

## What NI Gives You

NI knows all this and has built real tooling around it. It's worth stating fairly, conditions included.

| Tool | What it does | Condition |
|---|---|---|
| LVCompare | Graphical diff of two VIs, front panel and block diagram, node by node | Professional Development System, every dependency resolvable on that machine |
| LVMerge | Three-way merge (Base, Theirs, Yours) into a merged VI | Same as LVCompare, plus both branches must load |
| Separate compiled code | Keeps compiled code out of the source file, so callers don't show as modified when a callee changes | Per-VI or per-project setting, still not a diff, and a mass compile still touches files |
| Project Explorer source control | Check out and commit from inside LabVIEW through a configured provider | Depends on the provider, and the diff still needs LVCompare |

The pattern is the same each time: the tool works, and it needs the same licensed LabVIEW installed on the machine where it runs. Reviews happen on a web page, and no NI tool runs there. NI's own page on [software configuration management and LabVIEW](https://www.ni.com/en/support/documentation/supplemental/06/software-configuration-management-and-labview.html) is the honest summary of what they recommend, and the doc on [separating compiled code from VIs](https://www.ni.com/docs/en-US/bundle/labview/page/separating-compiled-code-from-vis-and-other-file-types.html) covers the setting.

LabVIEW is a capable tool with a shrinking pool of people who read it. The versioning pain is specific to binary files, and it doesn't go away with a better convention. It goes away with text.

## What a Text Station Looks Like

A TofuPilot Framework station is a folder: `procedure.yaml`, `phases/*.py`, `plugs/*.py`. All of it is text. Here's a limit change on a 3.3 V rail, exactly as Git shows it in a pull request, with no plugin and no licence, on any machine.

```diff filename="git-diff.txt"
# git diff of one limit change, as any reviewer sees it
diff --git a/procedure.yaml b/procedure.yaml
index 3f2a1c7..8b9e0d4 100644
--- a/procedure.yaml
+++ b/procedure.yaml
@@ -2,7 +2,7 @@
 name: Controller PCBA FCT
-version: 2.4.0
+version: 2.4.1
 
 unit:
   serial_number:
     default_value: SN-000000
@@ -24,7 +24,7 @@ main:
         unit: V
         validators:
           - operator: ">="
-            expected_value: 3.2
+            expected_value: 3.15
           - operator: "<="
             expected_value: 3.4
```

Two lines changed, and the reviewer can see the old value, the new value and the author without opening anything. `git blame procedure.yaml` names who set 3.15 and when. Two engineers editing different phases on two branches merge automatically.

The phase behind that limit is a plain function. The limit isn't in it, which is the point: the code reads the instrument, and the procedure holds the number.

```python filename="phases/power_rails.py"
# The 3.3 V rail phase. The limit lives in procedure.yaml, not here.
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
```

The same property carries through to the floor. A Git push builds an immutable deployment with an id like `dep_abc123`, stations fetch it with `tofupilot pull`, and every run uploaded from that station carries the `deployment_id`. In TofuPilot, "which units ran 2.4.0 and which ran 2.4.1" is a filter on the run list, and rolling every station back to the previous deployment is one action with no rebuild. [Trace a test change to the units it shipped](/guides/trace-a-test-change-to-the-units-it-shipped) walks through it, and [how to version control hardware tests with Git and TofuPilot](/guides/how-to-version-control-hardware-tests-with-git-and-tofupilot) covers the repository layout.

## Start With One Station

Pick the station only one person can open, or the one with the highest volume. Rebuild it as a `procedure.yaml` with its phases and plugs in Python with the TofuPilot Framework, keep the same limits, and run it side by side with the LabVIEW version on the same units for two weeks. The first pull request on that station is the first reviewable limit change your plant has had.

```bash filename="install-and-run.sh"
# Install the CLI, run the station locally (no account needed), or run with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The step-by-step rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Fix LabVIEW Merge Conflicts on Binary VIs
URL: https://www.tofupilot.com/guides/fix-labview-merge-conflicts-on-binary-vis
Learn how to resolve a Git merge conflict on a binary VI by hand or with LVMerge, and the four habits that stop the same conflict coming back.

You can't merge the bytes of two edited VIs, so a LabVIEW merge conflict has two ways out. Pick one side with `git checkout --ours` or `--theirs`, then redo the other side's change by hand in LabVIEW. Or run LVMerge, if you have a Professional seat and both branches load on your machine. Either way you'll reopen the VI, because there's no text to resolve.

## What the Conflict Looks Like

Git tells you up front that it won't try. The warning line is the one to read.

```text filename="git-status.txt"
$ git merge widen-3v3-limit
warning: Cannot merge binary files: FCT/Power Rails.vi (HEAD vs. widen-3v3-limit)
Auto-merging FCT/Power Rails.vi
CONFLICT (content): Merge conflict in FCT/Power Rails.vi
Automatic merge failed; fix conflicts and then commit the result.

$ git status
On branch main
You have unmerged paths.
  (fix conflicts and run "git commit")
  (use "git merge --abort" to abort the merge)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
        both modified:   FCT/Power Rails.vi
```

The file on disk right now is one of the two versions, not a mix, and Git won't say which without asking. Don't open it in LabVIEW yet. Decide first.

## The Decision

| Situation | Do this |
|---|---|
| One side's change is small (a constant, a limit, a wire) | Keep the other side, redo the small change by hand |
| Both sides made real diagram changes, you have Professional, both branches load | LVMerge |
| The VI is a class member or a typedef | Manual, always. LVMerge and the rename it needs don't hold up on classes |
| No machine on the team has Professional | Manual, always |
| You're in a rebase, not a merge | Same steps, but `--ours` and `--theirs` swap meaning (see below) |
| You don't know what the other side changed | Ask, or run `git log -1 --format=%B <branch> -- "FCT/Power Rails.vi"` and read the commit message. That message is all the history you have |

That last row is the real cost. On a text file you'd read the diff. On a VI the commit message is the diff, which is why "fixed limits" as a message costs you an afternoon later. [Why LabVIEW version control is so painful](/guides/why-labview-version-control-is-so-painful) has the full mechanism.

## The Manual Path

This works on any seat and any machine that can open the VI.

```bash filename="resolve-manually.sh"
# Keep one side of the VI, then redo the other side's change in LabVIEW by hand.
git checkout --ours -- "FCT/Power Rails.vi"
# or, to keep the incoming branch's version instead:
# git checkout --theirs -- "FCT/Power Rails.vi"

# Open the VI in LabVIEW, redo the other side's change, save. Then:
git add "FCT/Power Rails.vi"
git commit -m "Merge widen-3v3-limit: redo the 3.15 V floor in Power Rails.vi by hand"
```

Step by step:

1. Pick the side with the larger change and check it out. In a merge, `--ours` is the branch you're on and `--theirs` is the branch you're merging in. In a rebase they swap: `--ours` is the upstream you're rebasing onto and `--theirs` is your own commit being replayed.
2. Open the VI in LabVIEW and redo the smaller change. Use the other branch's commit message, or ask the author.
3. Save. If the two branches were built on the same LabVIEW version and bitness, you're done with LabVIEW. If the merge crossed a version or bitness change, mass compile the project now, and only now. A mass compile rewrites every VI it touches, so doing it for a single-VI conflict adds noise to the commit for no reason.
4. `git add` the VI and commit. Put what you redid in the message, with the value. That's the only place it will ever be written down.

Run the VI once before you push. There's no diff to review, so the test is the review.

## The LVMerge Path

LVMerge does a real three-way merge: Base (the common ancestor), Theirs, Yours, into a merged VI you accept difference by difference. It needs the mergetool config from [LVCompare and LVMerge: what they cannot do](/guides/lvcompare-and-lvmerge-what-they-cannot-do), a Professional seat, and every subVI and class in both branches resolvable on this machine.

```bash filename="resolve-with-lvmerge.sh"
# Three-way merge with LVMerge. Needs Professional and the lvmerge mergetool config.
git config mergetool.keepBackup false
git mergetool --tool=lvmerge -- "FCT/Power Rails.vi"
# LVMerge opens Base, Theirs and Yours. Accept or reject each difference, then save.
git add "FCT/Power Rails.vi"
git commit
```

Step by step:

1. Run `git mergetool` on the one file. Git writes the three inputs to temporary absolute paths and calls LVMerge in Base, Theirs, Yours, Merged order.
2. Work through the difference list. Each entry is a front panel, block diagram or attribute change from one side.
3. Before saving, read the merged block diagram once. LVMerge can duplicate a node that one branch deleted and re-added. Remove the duplicate.
4. Save, `git add`, commit. `keepBackup false` stops Git leaving a `.orig` copy of the VI next to the real one, which LabVIEW would otherwise find on the next load.

If LVMerge stalls on a dependency dialog, abort, and take the manual path on the development machine that has the full project.

## How to Stop It Recurring

The conflict is a symptom of two people needing the same binary file at the same time. Four habits reduce how often that happens, in order of cost.

One person per VI at a time. Say which VIs you're in before you start, in the channel or in a `LOCKS.md` at the repository root, and nobody else opens them until you push. It's a convention, not a mechanism, and it holds on a small team.

Smaller VIs. A 40-node VI that does the whole test is one file two people will both need. Split by phase and the odds of two engineers in the same file drop.

Limits in a config file, not in the diagram. Most conflicts on test VIs are two people changing two different constants in one file. Move the constants out, and the VI stops changing when a limit does.

```ini filename="limits.ini"
; Read by the test VI with the Config File VIs. This file diffs and merges in plain Git.
[rail_3v3]
min = 3.15
max = 3.4

[rail_5v]
min = 4.85
max = 5.15
```

Separate compiled code from the source file. It doesn't prevent a real two-edit conflict, but it removes the fake ones, where a VI shows as modified because a callee changed. [Separate compiled code in LabVIEW, explained](/guides/separate-compiled-code-in-labview-explained) covers the setting. Together with the hand-over notes in [document a test station for hand-over](/guides/document-a-test-station-for-hand-over), it's what makes the manual path survivable when the author of the other branch has left.

## The Contrast

A TofuPilot Framework station is text: `procedure.yaml`, `phases/*.py`, `plugs/*.py`. The same conflict, two branches both changing the 3.3 V floor, looks like this.

```diff filename="procedure.yaml"
       - name: rail_3v3
         unit: V
         validators:
           - operator: ">="
<<<<<<< HEAD
             expected_value: 3.15
=======
             expected_value: 3.1
>>>>>>> widen-3v3-limit
           - operator: "<="
             expected_value: 3.4
```

You read both values, keep one, delete the markers, `git add`, commit. Every other change in the two branches merged automatically. Afterwards, `git blame` answers the question the LabVIEW commit message couldn't.

```text filename="git-blame.txt"
$ git blame -L 25,26 procedure.yaml
a41f9c2e (Priya Raman  2026-09-14 10:22:41 +0200 25)           - operator: ">="
7d03be51 (Marc Dubois  2026-09-18 16:05:09 +0200 26)             expected_value: 3.15
```

Who set the floor to 3.15, on what day, in which commit. And once that commit is pushed, the station pulls the deployment and every run it uploads to TofuPilot carries the deployment id, so the units tested under 3.15 are a filter on the run list.

## Start With One Station

Pick the station whose VIs conflict most often, or the one only one person can open. Rebuild it as a `procedure.yaml` with phases and plugs in Python with the TofuPilot Framework, keep the same limits, and run it side by side with the LabVIEW version on the same units for two weeks.

```bash filename="install-and-run.sh"
# Install the CLI, run the station locally (no account needed), or run with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Test Acceleration for Hardware Teams
URL: https://www.tofupilot.com/guides/test-acceleration-for-hardware-teams-with-tofupilot
Learn how to accelerate hardware test cadence using TofuPilot to identify bottlenecks, reduce debug time, and optimize test coverage.

Hardware teams don't need to test more. They need to test faster. The bottleneck isn't usually the test itself. It's the time spent searching for data, debugging failures manually, and waiting for reports. TofuPilot eliminates these delays.

## Where Hardware Teams Lose Time

| Activity | Typical time | With TofuPilot |
|----------|-------------|----------------|
| Finding test data for a specific unit | 15-30 min | 10 seconds |
| Comparing passing vs. failing runs | 1-2 hours | 2 minutes |
| Building a weekly quality report | 3-4 hours | Already done (live dashboard) |
| Correlating failures with component lots | 1-2 days | 15 minutes |
| Answering "What's our yield?" | 30 min (pull data, calculate) | Glance at dashboard |
| Debugging a field return | 2-4 hours | 5 minutes (search by serial) |

The test run itself might take 60 seconds. Everything around it takes hours or days. That's where acceleration happens.

## Accelerating Failure Debug

### Before TofuPilot

1. Unit fails on the test station
2. Operator calls the test engineer
3. Test engineer walks to the station, looks at the screen
4. Test engineer manually records the failing measurements
5. Test engineer goes back to their desk, opens old data files to compare
6. Test engineer emails the design team with findings
7. Back and forth continues

### With TofuPilot

1. Unit fails on the test station
2. Test engineer opens TofuPilot, filters to the failed run
3. Compares measurements against recent passing runs in one click
4. Identifies the anomalous measurement
5. Checks the measurement trend to see when it started
6. Shares the dashboard link with the design team

Step 2-6 takes 5 minutes. The old way takes half a day.

## Accelerating Test Development

### Data-Driven Limit Setting

Instead of guessing limits from datasheets, run 50 units and use the actual distribution to set limits.

```python filename="set_limits_from_data.py"
import numpy as np
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get pilot production data
runs = client.get_runs(
    procedure_id="PILOT-FUNCTIONAL",
    limit=50,
)

# Extract values for each measurement
measurements = {}
for run in runs:
    for step in run.get("steps", []):
        for m in step.get("measurements", []):
            if m["name"] not in measurements:
                measurements[m["name"]] = []
            measurements[m["name"]].append(m["value"])

# Calculate recommended limits (mean +/- 4 sigma)
for name, values in measurements.items():
    arr = np.array(values)
    mean = np.mean(arr)
    std = np.std(arr, ddof=1)
    print(f"{name}: mean={mean:.4f}, std={std:.4f}")
    print(f"  Recommended limits: [{mean - 4*std:.4f}, {mean + 4*std:.4f}]")
```

This replaces weeks of limit tuning with 30 minutes of data analysis.

### Identifying Redundant Tests

Not every measurement adds value. Some measurements never fail. Some always correlate perfectly with another measurement (testing the same thing twice).

Pull your measurement data from TofuPilot and check:

| Check | Action |
|-------|--------|
| Measurement never fails (100% pass rate over 1000+ units) | Consider removing or widening limits |
| Two measurements always fail together (r > 0.95) | One might be redundant |
| Measurement adds 10s to cycle time but catches 0.01% of defects | Consider removing from production test |

Cutting redundant measurements directly reduces cycle time and increases throughput.

## Accelerating Yield Improvement

### The Improvement Loop

```
   Measure → Analyze → Improve → Verify
      ↑                              │
      └──────────────────────────────┘
```

TofuPilot accelerates every step:

1. **Measure**: Automatic data collection from every station
2. **Analyze**: Dashboards show yield trends, failure paretos, and measurement distributions
3. **Improve**: Data points to the root cause, so fixes are targeted
4. **Verify**: Before/after comparison confirms the fix worked

Without centralized data, steps 1 and 2 take days. With TofuPilot, they're instant.

### Prioritizing Improvements

The failure pareto in TofuPilot shows you where to focus. Fix the #1 failure mode first. It has the biggest yield impact.

Don't try to fix everything at once. Fix one thing, verify the improvement in TofuPilot, then move to the next.

## Accelerating NPI (New Product Introduction)

During NPI, test development and product development happen in parallel. Every design revision needs updated tests. Every test result informs the next design revision.

TofuPilot accelerates this loop by:

- Storing results from every prototype build
- Comparing measurements across design revisions
- Showing which design changes improved (or regressed) specific measurements
- Providing immediate visibility into whether a new build meets specs

### Comparing Design Revisions

```python filename="revision_comparison.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get runs from two design revisions
rev_a_runs = client.get_runs(procedure_id="PROTO-FUNCTIONAL", limit=20)
rev_b_runs = client.get_runs(procedure_id="PROTO-FUNCTIONAL", limit=20)

# Compare measurement means
# Filter by part_number or date range to separate revisions
```

If Rev B has 15% lower idle current and 5% tighter voltage regulation, the design change worked. If thermal measurements regressed, the new layout needs attention. TofuPilot shows this in minutes, not days.

## The Compound Effect

Each individual acceleration seems small: 5 minutes saved here, 30 minutes there. But across a team of 5 test engineers running 10 debug sessions per week, the compound effect is significant.

| Savings per debug session | 2 hours |
|--------------------------|---------|
| Debug sessions per week | 50 (5 engineers x 10) |
| Weekly time saved | 100 hours |
| Monthly time saved | 400 hours |

That's 400 engineering hours per month redirected from data gathering to actual engineering work: improving designs, optimizing processes, and shipping better products.


### Time-Series Test Data Analysis
URL: https://www.tofupilot.com/guides/time-series-test-data-analysis-with-tofupilot
Learn how to capture, store, and analyze time-series measurement data from hardware tests using TofuPilot's multi-dimensional arrays.

Hardware tests produce waveforms, not just single numbers. A power supply ripple test captures thousands of voltage samples over time. A vibration test records acceleration spectra across frequency bands. Storing these natively lets you trend and compare waveforms across your production.

## Single Values vs. Time-Series

Most test systems store one number per measurement: "output voltage = 3.31V." But the full story is in the waveform. That 3.31V might be a clean DC signal or a noisy mess that happens to average out to 3.31V.

| Data type | Example | What it reveals |
|-----------|---------|----------------|
| Single value | Vout = 3.31V | Average output level |
| Time series | 1000 samples over 10ms | Ripple, noise, transient behavior |
| Frequency spectrum | FFT of output voltage | Switching noise frequency content |
| Multi-axis | X/Y/Z acceleration | Vibration in all directions |

Dimensioned measurements handle all of these.

## Capturing Time-Series Data

### With OpenHTF

Use dimensioned measurements to store time-series data.

```python filename="ripple_test_openhtf.py"
import openhtf as htf
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("output_ripple_mv")
        .with_dimensions("time_us")
        .with_units("mV"),
    htf.Measurement("ripple_pk_pk_mv")
        .in_range(0, 50)
        .with_units("mV"),
)
def ripple_test(test):
    # Capture 1000 samples at 100kHz (10us spacing)
    waveform = capture_oscilloscope(channel=1, samples=1000, rate_hz=100000)

    for i, sample in enumerate(waveform):
        test.measurements.output_ripple_mv[i * 10] = sample  # time in microseconds

    # Also store the scalar summary
    test.measurements.ripple_pk_pk_mv = max(waveform) - min(waveform)


def main():
    test = htf.Test(
        ripple_test,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="PSU-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "PSU-2025-0099")


if __name__ == "__main__":
    main()
```

Install it with `pip install "tofupilot[openhtf]"`. The `upload()` callback reads `TOFUPILOT_API_KEY` from the environment.

Note that the pass/fail limit is on the scalar (`ripple_pk_pk_mv`), not on the waveform. That split is deliberate and worth keeping: the scalar decides the verdict and drives trending, while the waveform is what you open when the scalar tells you something is wrong.

### With the Python SDK

If you are not running OpenHTF, create the run directly:

```python filename="ripple_test_client.py"
import os
from datetime import datetime, timedelta, timezone

import numpy as np
from tofupilot.v2 import TofuPilot

started = datetime.now(timezone.utc) - timedelta(seconds=5)
ended = datetime.now(timezone.utc)

waveform = capture_oscilloscope(channel=1, samples=1000, rate_hz=100000)
pk_pk = float(np.max(waveform) - np.min(waveform))
outcome = "PASS" if pk_pk < 50 else "FAIL"

with TofuPilot(api_key=os.getenv("TOFUPILOT_API_KEY")) as client:
    client.runs.create(
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        serial_number="PSU-2025-0099",
        part_number="PSU-100",
        outcome=outcome,
        started_at=started,
        ended_at=ended,
        phases=[{
            "name": "Output Ripple",
            "outcome": outcome,
            "started_at": started,
            "ended_at": ended,
            "measurements": [{
                "name": "Ripple Peak to Peak",
                "measured_value": pk_pk,
                "units": "mV",
                "validators": [
                    {"operator": "<=", "expected_value": 50},
                ],
            }],
        }],
    )
```

This uploads the extracted scalar rather than all 1000 samples. If you need the raw capture kept alongside the result, attach it as a file rather than expanding it into measurements.

## Types of Time-Series Test Data

### Voltage/Current Waveforms
Captured from oscilloscopes or DAQs during power-on sequences, ripple tests, or transient response tests. Store the raw waveform plus scalar summaries (peak-to-peak, RMS, rise time).

### Temperature Profiles
Recorded during thermal cycling, burn-in, or heat dissipation tests. Multiple sensors produce multi-channel time-series data over minutes or hours.

```python filename="thermal_profile.py"
# Thermal test with 4 temperature sensors sampled every second for 30 minutes
sensors = ["junction", "ambient", "heatsink", "case"]
duration_s = 1800
samples_per_sensor = duration_s  # 1 sample/second

for sensor_name in sensors:
    readings = read_thermal_sensor(sensor_name, samples=samples_per_sensor)
    measurements.append({
        "name": f"temp_{sensor_name}",
        "measured_value": max(readings),   # peak temperature is the gating value
        "units": "degC",
        "validators": [
            {"operator": "<=", "expected_value": 85},
        ],
    })
```

Four sensors at 1800 samples each is 7200 points per unit. At production volume, store the features that gate the result (peak, time above threshold, final steady-state) and keep the full profile as an attachment.

### Vibration Spectra
FFT data from accelerometers during vibration testing. Frequency on one axis, amplitude on the other.

### Pressure/Flow Curves
Time-series pressure and flow measurements during leak tests, pneumatic tests, or hydraulic validation.

## Analyzing Time-Series Across Production

The power of storing waveforms rather than only scalars is comparison across units.

### Waveform Overlay
Compare ripple waveforms from 100 units. If 99 look the same and one has an extra spike, that unit has a problem the peak-to-peak measurement alone might not catch.

### Statistical Bounds
Calculate the mean and standard deviation of your waveform at each time point across all units. This gives you an envelope of normal behavior. Any unit whose waveform falls outside the envelope is flagged.

```python filename="waveform_statistics.py"
import numpy as np

# All waveforms from production (each is a list of 1000 samples)
all_waveforms = np.array(waveforms_from_production)  # shape: (N_units, 1000)

mean_waveform = np.mean(all_waveforms, axis=0)
std_waveform = np.std(all_waveforms, axis=0)

upper_bound = mean_waveform + 3 * std_waveform
lower_bound = mean_waveform - 3 * std_waveform

# Check if a new unit's waveform is within bounds
new_waveform = np.array(new_unit_data)
is_anomalous = np.any(new_waveform > upper_bound) or np.any(new_waveform < lower_bound)
```

Two cautions before using this as a gate. With 1000 sample points and a 3-sigma band, roughly 3 points per unit will fall outside by chance alone, so `np.any` will flag a large share of good units. Require a run of consecutive out-of-band points, or compare against a tolerance band rather than a per-point sigma. And the envelope must be built from a period when the process was stable, otherwise you encode a drift as normal.

### Trend Analysis on Waveform Features
Extract features from each waveform (rise time, settling time, overshoot, RMS) and trend them over production. A gradual increase in rise time across units suggests a component or process drift.

This is usually the highest-value analysis of the three, and the cheapest to store: a handful of numbers per unit that trend and alert like any other measurement.

## Best Practices

| Practice | Why |
|----------|-----|
| Store both raw waveform and scalar summaries | Waveforms for deep analysis, scalars for dashboards and trending |
| Use consistent sample rates | Comparing waveforms requires the same number of points and timing |
| Include time axis metadata | Sample rate or time stamps so the waveform can be reconstructed |
| Set limits on scalar summaries | Scalars drive pass/fail, waveforms drive root cause analysis |
| Limit waveform size | Keep arrays under 10,000 points per measurement for practical storage |
| Capture raw traces on failures or a sample | Every unit at full rate outgrows its usefulness quickly |

### IPC-A-610 Inspection Tracking
URL: https://www.tofupilot.com/guides/ipc-a-610-inspection-tracking-with-tofupilot
Learn how to track IPC-A-610 visual inspection results for PCBA acceptability using TofuPilot's structured test records.

IPC-A-610 is the acceptability standard for electronic assemblies. It defines what "good" looks like for solder joints, component placement, cleanliness, and mechanical assembly. TofuPilot tracks inspection results systematically instead of relying on paper checklists.

## What IPC-A-610 Covers

IPC-A-610 classifies defects into three product classes:

| Class | Application | Acceptance criteria |
|-------|-------------|-------------------|
| Class 1 | General electronics (consumer) | Least strict |
| Class 2 | Dedicated service electronics (industrial) | Moderate |
| Class 3 | High-performance electronics (medical, aerospace, military) | Most strict |

Inspection criteria include:

| Category | Examples |
|----------|---------|
| Solder joints | Wetting, fillets, bridges, cold joints |
| Component placement | Alignment, orientation, polarity |
| Cleanliness | Flux residue, contamination, corrosion |
| Mechanical | Wire routing, strain relief, conformal coating |
| Marking | Labels, part numbers, date codes |

## Logging Inspection Results to TofuPilot

### Per-Board Inspection

```python filename="ipc610_inspection.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

def log_inspection(serial, inspector, defects):
    """Log IPC-A-610 inspection results."""
    measurements = [
        {"name": "inspector_id", "value": inspector, "unit": ""},
        {"name": "ipc_class", "value": 2, "unit": "class"},
        {"name": "total_defects", "value": len(defects), "unit": "count", "limit_high": 0},
    ]

    # Log each defect category
    defect_categories = {
        "solder": 0, "placement": 0, "cleanliness": 0,
        "mechanical": 0, "marking": 0,
    }
    for d in defects:
        if d["category"] in defect_categories:
            defect_categories[d["category"]] += 1

    for cat, count in defect_categories.items():
        measurements.append({
            "name": f"defects_{cat}",
            "value": count,
            "unit": "count",
            "limit_high": 0,
        })

    passed = len(defects) == 0

    client.create_run(
        procedure_id="IPC610-VISUAL-INSPECTION",
        unit_under_test={
            "serial_number": serial,
            "part_number": "MAIN-BOARD-V4",
        },
        run_passed=passed,
        steps=[{
            "name": "IPC-A-610 Class 2 Inspection",
            "step_type": "measurement",
            "status": passed,
            "measurements": measurements,
        }],
    )

# Example: board with two defects
log_inspection(
    serial="PCB-2025-04521",
    inspector="OP-012",
    defects=[
        {"category": "solder", "location": "U12-pin3", "type": "insufficient_wetting"},
        {"category": "cleanliness", "location": "J5-area", "type": "flux_residue"},
    ],
)
```

### Defect Classification

Log specific defect types to build a defect pareto.

```python filename="defect_logging.py"
# Detailed defect logging
defect_types = {
    "solder_bridge": "Solder bridge between adjacent pins",
    "cold_joint": "Cold or disturbed solder joint",
    "insufficient_wetting": "Insufficient solder wetting on pad or lead",
    "solder_ball": "Loose solder ball on board surface",
    "tombstone": "Component standing on end (tombstoning)",
    "misalignment": "Component offset from pad center",
    "wrong_polarity": "Polarized component installed backwards",
    "missing_component": "Component not placed",
    "flux_residue": "Excessive flux residue after cleaning",
    "damaged_component": "Component body cracked or chipped",
}

# Log each defect with its type
for defect in board_defects:
    measurements.append({
        "name": f"defect_{defect['type']}",
        "value": 1,
        "unit": "count",
        "limit_high": 0,
    })
```

## Defect Pareto Analysis

TofuPilot's analytics show you the most common defect types across production.

| Rank | Defect type | Count | Percentage |
|------|------------|-------|------------|
| 1 | Insufficient wetting | 45 | 32% |
| 2 | Solder bridge | 28 | 20% |
| 3 | Flux residue | 22 | 16% |
| 4 | Misalignment | 15 | 11% |
| 5 | Cold joint | 12 | 9% |
| - | All others | 18 | 13% |

Fix the top defect type first. Insufficient wetting (32%) is likely a solder paste issue (stencil wear, paste viscosity, or reflow profile).

## Inspector Consistency

Track defect detection rates by inspector to identify training needs.

```python filename="inspector_analysis.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

runs = client.get_runs(
    procedure_id="IPC610-VISUAL-INSPECTION",
    limit=1000,
)

# Group by inspector
inspector_stats = {}
for run in runs:
    for step in run.get("steps", []):
        for m in step.get("measurements", []):
            if m["name"] == "inspector_id":
                inspector = m["value"]
                if inspector not in inspector_stats:
                    inspector_stats[inspector] = {"total": 0, "rejected": 0}
                inspector_stats[inspector]["total"] += 1
                if not run["run_passed"]:
                    inspector_stats[inspector]["rejected"] += 1

for inspector, stats in inspector_stats.items():
    reject_rate = stats["rejected"] / stats["total"] * 100
    print(f"Inspector {inspector}: {stats['total']} boards, {reject_rate:.1f}% rejection rate")
```

If Inspector A rejects 8% and Inspector B rejects 2% on the same product, either A is too strict or B is missing defects. Calibrate inspectors using reference boards with known defects.

## Connecting Inspection to Electrical Test

The real power is correlating visual inspection results with electrical test results.

If boards that pass visual inspection but fail functional test at a specific measurement, there may be a defect type that visual inspection isn't catching. Conversely, if visually rejected boards are reworked and always pass functional test, the visual criteria may be too strict for your product class.

TofuPilot links both inspection and electrical test results to the same serial number, making this correlation straightforward.

## AOI Integration

Automated Optical Inspection (AOI) systems can push results to TofuPilot the same way manual inspections do.

```python filename="aoi_upload.py"
# Parse AOI machine output and upload to TofuPilot
import json

with open("aoi_results.json") as f:
    aoi_data = json.load(f)

for board in aoi_data["boards"]:
    defects = board.get("defects", [])
    client.create_run(
        procedure_id="AOI-INSPECTION",
        unit_under_test={"serial_number": board["serial"]},
        run_passed=len(defects) == 0,
        steps=[{
            "name": "AOI Scan",
            "step_type": "measurement",
            "status": len(defects) == 0,
            "measurements": [
                {"name": "defect_count", "value": len(defects), "unit": "count", "limit_high": 0},
                {"name": "scan_coverage_pct", "value": board.get("coverage", 100), "unit": "%"},
            ],
        }],
    )
```

Combine AOI data with manual inspection data and electrical test data in TofuPilot for a complete quality picture of every board.


### Semiconductor Wafer Test Data
URL: https://www.tofupilot.com/guides/semiconductor-wafer-test-data-with-tofupilot
Learn how to track wafer-level and final test data for semiconductors using TofuPilot for CP/FT yield analysis and bin mapping.

Semiconductor testing happens in two stages: wafer-level testing (CP, circuit probe) and final test (FT, packaged parts). Each stage generates massive datasets. TofuPilot stores per-die and per-part test results with full measurement data, enabling yield analysis, bin mapping, and lot traceability.

## Semiconductor Test Flow

```
Wafer Fab → Wafer Test (CP) → Packaging → Final Test (FT) → Binning → Ship
                 ↓                              ↓
            TofuPilot                       TofuPilot
         (per-die data)                 (per-part data)
```

### Wafer Test (Circuit Probe)

Probes contact each die on the wafer. Tests include:

| Test | What it checks |
|------|---------------|
| Continuity/opens | Bond pad connectivity |
| Leakage | Junction and oxide integrity |
| Parametric | Vth, Idsat, Rds(on), timing |
| Functional | Basic digital/analog operation |

### Final Test (FT)

After packaging, full parametric and functional testing at speed.

| Test | What it checks |
|------|---------------|
| DC parametric | Input/output levels, power consumption |
| AC parametric | Speed, setup/hold times, rise/fall times |
| Functional | Full device operation at target frequency |
| Burn-in (optional) | Infant mortality screening |

## Logging Wafer Test Data

### Per-Die Results

```python filename="wafer_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

def log_die_test(wafer_id, die_x, die_y, measurements, bin_code):
    """Log test results for a single die."""
    die_id = f"{wafer_id}-X{die_x}Y{die_y}"

    steps = [{
        "name": "Parametric",
        "step_type": "measurement",
        "status": bin_code == 1,  # Bin 1 = good die
        "measurements": [
            {"name": "vth_mv", "value": measurements["vth"], "unit": "mV", "limit_low": 350, "limit_high": 450},
            {"name": "idsat_ua_um", "value": measurements["idsat"], "unit": "uA/um", "limit_low": 500, "limit_high": 700},
            {"name": "ioff_pa_um", "value": measurements["ioff"], "unit": "pA/um", "limit_high": 100},
            {"name": "rdson_mohm", "value": measurements["rdson"], "unit": "mohm", "limit_high": 50},
        ],
    }]

    client.create_run(
        procedure_id="WAFER-CP-TEST",
        unit_under_test={
            "serial_number": die_id,
            "part_number": "IC-POWER-MGMT-V3",
        },
        run_passed=bin_code == 1,
        steps=steps,
    )

# Test all dies on a wafer
wafer_id = "LOT2025A-W07"
for x in range(20):
    for y in range(20):
        meas = probe_die(x, y)
        bin_code = classify_die(meas)
        log_die_test(wafer_id, x, y, meas, bin_code)
```

### Wafer-Level Summary

After testing all dies, upload a wafer summary.

```python filename="wafer_summary.py"
# Wafer-level summary
total_dies = 400
good_dies = 352
yield_pct = good_dies / total_dies * 100

client.create_run(
    procedure_id="WAFER-CP-SUMMARY",
    unit_under_test={
        "serial_number": "LOT2025A-W07",
        "part_number": "IC-POWER-MGMT-V3",
    },
    run_passed=yield_pct > 80,
    steps=[{
        "name": "Wafer Summary",
        "step_type": "measurement",
        "status": yield_pct > 80,
        "measurements": [
            {"name": "total_dies", "value": total_dies, "unit": "count"},
            {"name": "good_dies", "value": good_dies, "unit": "count"},
            {"name": "cp_yield_pct", "value": yield_pct, "unit": "%", "limit_low": 80},
            {"name": "bin1_count", "value": 352, "unit": "count"},
            {"name": "bin2_count", "value": 30, "unit": "count"},
            {"name": "bin3_count", "value": 18, "unit": "count"},
        ],
    }],
)
```

## Bin Mapping

Binning assigns each die or part to a category based on test results. Common bin structure:

| Bin | Meaning | Typical criteria |
|-----|---------|-----------------|
| 1 | Good, full spec | All parametrics within spec |
| 2 | Good, derated | Parametrics within wider limits |
| 3 | Functional fail | Failed functional test |
| 4 | Parametric fail | One or more parametric out of spec |
| 5 | Continuity fail | Open or short detected |
| 6 | Leakage fail | Excessive junction leakage |

Track bin distributions in TofuPilot across wafers, lots, and time periods.

## Yield Analysis

### Wafer-to-Wafer Yield

Compare CP yield across wafers in a lot.

| Wafer | Total dies | Good (Bin 1) | Yield |
|-------|-----------|--------------|-------|
| W01 | 400 | 360 | 90.0% |
| W02 | 400 | 355 | 88.8% |
| W03 | 400 | 312 | 78.0% |
| W04 | 400 | 358 | 89.5% |

W03 has significantly lower yield. Investigate wafer-level process issues (particle count, lithography focus, etch uniformity).

### Lot-to-Lot Yield

Track yield across production lots to monitor fab process stability.

```python filename="lot_yield_tracking.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get all wafer summary runs
runs = client.get_runs(
    procedure_id="WAFER-CP-SUMMARY",
    limit=100,
)

# Group by lot and calculate average yield
lot_yields = {}
for run in runs:
    serial = run["unit_under_test"]["serial_number"]
    lot = serial.split("-W")[0]  # Extract lot ID
    for step in run.get("steps", []):
        for m in step.get("measurements", []):
            if m["name"] == "cp_yield_pct":
                if lot not in lot_yields:
                    lot_yields[lot] = []
                lot_yields[lot].append(m["value"])

for lot, yields in sorted(lot_yields.items()):
    avg = sum(yields) / len(yields)
    print(f"{lot}: {avg:.1f}% avg yield ({len(yields)} wafers)")
```

### CP to FT Yield Correlation

Track yield at both test stages to find packaging-related issues.

| Lot | CP Yield | FT Yield | FT/CP Loss |
|-----|----------|----------|------------|
| LOT2025A | 89.2% | 87.1% | 2.1% |
| LOT2025B | 91.0% | 88.5% | 2.5% |
| LOT2025C | 90.5% | 82.3% | 8.2% |

LOT2025C shows excessive CP-to-FT yield loss (8.2% vs. typical 2-3%). This points to a packaging issue: wire bond failures, die attach problems, or moisture sensitivity during assembly.

## Parametric Distribution Monitoring

Track critical parametric distributions across production to catch process drift.

Key parameters to monitor:

| Parameter | Why it matters |
|-----------|---------------|
| Vth (threshold voltage) | Shifts indicate gate oxide or implant dose issues |
| Idsat (saturation current) | Indicates transistor drive strength |
| Ioff (off-state leakage) | Affects standby power, process control indicator |
| Rdson (on-resistance) | Critical for power devices |
| Timing margins | Speed grading, binning into speed grades |

TofuPilot's measurement histograms show the distribution of each parameter across all tested dies. A bimodal distribution suggests two different process conditions. A shifting mean suggests process drift.

## Traceability from Part to Wafer

When a packaged part fails in the field, trace back:

1. Part serial number → packaging lot
2. Packaging lot → wafer ID
3. Wafer ID → die position (X, Y)
4. Die position → original CP test data

All of this data lives in TofuPilot, linked by serial numbers. Was the die marginal at wafer test? Was it a Bin 2 (derated) part sold as Bin 1? The data answers these questions.


### What Is Shift-Left Quality in Manufacturing
URL: https://www.tofupilot.com/guides/what-is-shift-left-quality-in-manufacturing
Shift-left quality moves defect detection earlier in the manufacturing process. Learn how it reduces cost and how test data enables it.

Shift-left quality means moving defect detection earlier in the manufacturing process. The further left (earlier) you catch a problem, the cheaper it is to fix. A component defect caught at incoming inspection costs pennies to address. The same defect caught in the field costs hundreds of dollars. This guide covers how shift-left quality works, what it costs at each stage, and how test data enables the shift.

## The Cost of Finding Defects Late

| Stage | Relative Cost to Fix | Example |
|-------|---------------------|---------|
| Design (simulation) | 1x | Catch a voltage margin issue in SPICE |
| Incoming inspection (IQC) | 10x | Reject a bad component lot before assembly |
| In-process (IPQC/AOI) | 25x | Rework a solder bridge before final assembly |
| End-of-line test (EOL) | 50x | Scrap or rework a fully assembled unit |
| Field return | 500-1000x | Warranty repair, shipping, customer impact |

The 10x rule is well-established in manufacturing: every stage you delay detection, the cost increases by roughly an order of magnitude.

## What Shifting Left Looks Like

| Before (Right-Heavy) | After (Shift-Left) |
|----------------------|-------------------|
| Test everything at EOL | Add measurements at IQC and IPQC |
| Find solder defects at functional test | Catch them at AOI/SPI after reflow |
| Discover component issues during assembly | Screen components at incoming inspection |
| Learn about failure modes from field returns | Detect them during DVT and ORT |
| Set limits from engineering judgment | Set limits from upstream process data |

Shifting left doesn't mean removing end-of-line testing. It means adding detection points earlier so fewer defects reach the final test stage.

## The Data Requirement

Shift-left quality requires data from every stage. You can't correlate upstream measurements with downstream failures if you're only collecting data at EOL.

| Stage | Data Needed |
|-------|------------|
| IQC | Component measurements, supplier lot info, CoC data |
| IPQC | SPI paste volume, AOI defect counts, process parameters |
| FCT | Electrical measurements with limits and units |
| EOL | Full functional test results per serial number |
| Field | Return reason codes, failure analysis results |

When all stages feed data into one platform, correlations become visible. You can answer questions like: "Do units from supplier A's lot 2024-47 have higher EOL failure rates than supplier B's lot 2024-48?"

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Add Upstream Test Points

Don't wait until end-of-line to take measurements. Add test phases at earlier stages.

```python filename="iqc_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("capacitor_esr_mOhm")
    .in_range(maximum=45)
    .with_units(units.OHM),
    htf.Measurement("resistance_ohm")
    .within_percent(4700, 5)
    .with_units(units.OHM),
)
def phase_iqc_check(test):
    """Incoming component check. Catches bad parts before assembly."""
    test.measurements.capacitor_esr_mOhm = 32.1
    test.measurements.resistance_ohm = 4720
```

## Step 2: Log Every Stage to TofuPilot

Each stage uploads results independently. TofuPilot links them by serial number or lot code.

```python filename="iqc_test.py"
from tofupilot.openhtf import upload

test = htf.Test(phase_iqc_check)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan component lot: "))
```

## Step 3: Correlate Across Stages

TofuPilot tracks results across all test stages. Open the Analytics tab to find correlations:

- **Upstream predictors**: Which IQC measurements correlate with EOL failures?
- **Process indicators**: Which SPI/AOI measurements predict functional test failures?
- **Supplier quality**: Which vendors produce components with higher downstream failure rates?
- **Time correlations**: Did failures increase after a specific process change?

These correlations tell you where to add or tighten upstream detection. Each defect caught earlier is a defect that never reaches EOL or the field.

## Shift-Left Maturity Levels

| Level | What You Do | Defect Detection |
|-------|------------|-----------------|
| 1. EOL only | All testing at end-of-line | Late, expensive |
| 2. Multi-stage | IQC + FCT + EOL, data in separate systems | Earlier, but no correlation |
| 3. Connected | All stages in one platform, serial-level traceability | Can correlate upstream and downstream |
| 4. Predictive | ML models predict downstream failures from upstream data | Earliest, cheapest |

Most companies are at level 1 or 2. The jump from 2 to 3 (connecting data across stages) is where shift-left quality becomes actionable. Level 4 (predictive) is the long-term goal.

## Common Mistakes

| Mistake | Why It Fails |
|---------|-------------|
| Adding tests without data analysis | You add cost without knowing if the test catches real defects |
| Over-testing at every stage | Cycle time and cost increase without proportional quality gain |
| Shifting left without tracking results | No way to prove the upstream test is catching defects |
| Removing EOL tests too early | Until upstream detection is proven, keep the safety net |

Shift-left quality is data-driven. Add upstream detection, measure its effectiveness with downstream data, and iterate. The goal is catching defects at the cheapest point, not testing everything everywhere.

### ICT vs FCT vs Flying Probe: When to Use Each
URL: https://www.tofupilot.com/guides/ict-vs-fct-vs-flying-probe-when-to-use-each-pcba-test-method
Compare in-circuit test (ICT), functional test (FCT), and flying probe for PCBA manufacturing, with cost analysis, coverage tradeoffs, and decision criteria.

Three test methods dominate PCBA manufacturing: in-circuit test (ICT), functional test (FCT), and flying probe. Each catches different defects at different costs. Most production lines use two or three in combination. This guide compares them so you can choose the right test strategy for your product.

## Quick Comparison

| | ICT | FCT | Flying Probe |
|--|-----|-----|-------------|
| **What it tests** | Individual components | Board-level behavior | Individual components |
| **Defects caught** | Wrong value, missing, short, open | Power, comms, firmware, performance | Same as ICT |
| **Test time** | 5-30 seconds | 30 seconds - 5 minutes | 1-10 minutes |
| **Fixture cost** | $5,000-50,000 | $1,000-10,000 | None (fixtureless) |
| **NRE time** | 4-8 weeks | 1-2 weeks | Hours (program only) |
| **Volume sweet spot** | 10K+ units/year | Any volume | Prototypes, &lt; 5K units/year |
| **Automation** | Fully automated | Fully automated | Fully automated |
| **Board access** | Bed-of-nails (needs test pads) | Connectors + probes | Probes (no fixture) |
| **Programming** | Vendor software | Python/LabVIEW/custom | Vendor software |

## In-Circuit Test (ICT)

ICT presses a bed-of-nails fixture against the board and tests each component individually. It verifies that the right components are in the right places with the right values.

### What ICT Catches

| Defect | Detection Method |
|--------|-----------------|
| Wrong resistor value | Resistance measurement |
| Missing component | Open circuit detection |
| Solder short | Short circuit detection |
| Wrong polarity (capacitor, diode) | Capacitance/diode test |
| IC pin opens | Boundary scan (JTAG) |
| BGA connection issues | Limited (only accessible pins) |

### ICT Costs

| Item | Typical Cost |
|------|-------------|
| Fixture (bed-of-nails) | $5,000-50,000 depending on board complexity |
| Fixture lead time | 4-8 weeks |
| ICT machine | $50,000-500,000 (capital) |
| Test time per board | 5-30 seconds |
| Program development | $2,000-10,000 |

### When to Use ICT

- **High volume (10K+ boards/year).** The fixture cost amortizes over volume.
- **Complex boards with many passives.** ICT excels at verifying resistor/capacitor values.
- **SMT assembly with known defect rates.** ICT catches assembly defects before FCT.
- **When you need fast test time.** 5-30 seconds vs. minutes for FCT.

### When to Skip ICT

- **Low volume (&lt; 5K/year).** Fixture cost doesn't justify the volume.
- **Simple boards (&lt; 50 components).** FCT alone catches most defects.
- **Fast design iteration.** Fixture changes with every board revision.
- **BGA-heavy designs.** ICT can't access BGA pads without boundary scan.

## Functional Test (FCT)

FCT tests the board as a working system. You power it up, run the firmware, and verify behavior. FCT catches defects that ICT misses: firmware bugs, timing issues, analog performance, and system-level interactions.

### What FCT Catches

| Defect | Detection Method |
|--------|-----------------|
| Voltage regulator failure | Power rail measurement |
| Wrong firmware | Version query |
| Communication failure | UART/SPI/I2C response check |
| Excessive current draw (short) | Current measurement |
| Wrong crystal frequency | Frequency measurement |
| ADC/DAC calibration drift | Analog measurement |
| Failed self-test | Firmware self-test command |

### FCT Costs

| Item | Typical Cost |
|------|-------------|
| Test fixture (pogo pins + connectors) | $1,000-10,000 |
| Fixture lead time | 1-2 weeks |
| Instruments (DMM, PSU, etc.) | $2,000-20,000 (one-time, reusable) |
| Test time per board | 30 seconds - 5 minutes |
| Script development | Python + OpenHTF (free tools) |

### When to Use FCT

- **Every production board should get FCT.** It's the final check before shipping.
- **Any volume.** Low fixture cost makes it viable even for 100 units/year.
- **After ICT.** FCT verifies system behavior that ICT can't test.
- **As the only test for simple boards.** If the board has &lt; 50 components, FCT alone may suffice.

### FCT with Python and TofuPilot

```python filename="ict_vs_fct/fct_example.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class BoardPlug(BasePlug):
    """Board interface for FCT."""

    def setUp(self):
        self._voltages = iter([3.31, 5.01, 1.81])
        self._current = 0.12

    def read_voltage(self) -> float:
        return next(self._voltages)

    def read_current(self) -> float:
        return self._current

    def query_firmware(self) -> str:
        return "2.1.0"

    def tearDown(self):
        pass


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
    htf.Measurement("rail_5v0").in_range(4.8, 5.2).with_units(units.VOLT),
    htf.Measurement("rail_1v8").in_range(1.7, 1.9).with_units(units.VOLT),
)
@htf.plug(board=BoardPlug)
def test_power(test, board):
    test.measurements.rail_3v3 = board.read_voltage()
    test.measurements.rail_5v0 = board.read_voltage()
    test.measurements.rail_1v8 = board.read_voltage()


@htf.measures(
    htf.Measurement("firmware_version").equals("2.1.0"),
    htf.Measurement("idle_current").in_range(0.05, 0.20).with_units(units.AMPERE),
)
@htf.plug(board=BoardPlug)
def test_system(test, board):
    test.measurements.firmware_version = board.query_firmware()
    test.measurements.idle_current = board.read_current()


def main():
    test = htf.Test(
        test_power,
        test_system,
        procedure_id="FCT-001",
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial number: "))
```

Every measurement flows into TofuPilot with limits, units, and pass/fail status. You get FPY, Cpk, and failure Pareto automatically.

## Flying Probe

Flying probe machines use motorized probes that move to each test point. No fixture needed. The machine programs from your Gerber files and netlist.

### What Flying Probe Catches

Same defects as ICT: shorts, opens, wrong values, missing components. Some machines add:
- Capacitance measurement
- Inductance measurement
- Boundary scan (JTAG) integration
- Basic functional tests (limited)

### Flying Probe Costs

| Item | Typical Cost |
|------|-------------|
| Fixture | $0 (fixtureless) |
| Setup time | Hours (from Gerber + netlist) |
| Machine | $100,000-500,000 (capital) |
| Test time per board | 1-10 minutes (depends on test points) |
| Per-board cost at CM | $5-50 depending on program complexity |

### When to Use Flying Probe

- **Prototypes and first articles.** Zero fixture cost, fast setup.
- **Low volume (&lt; 5K/year).** Cheaper than ICT fixtures.
- **Design iteration.** No fixture to modify when the board changes.
- **Complex BGAs.** Some flying probe machines can test BGA pads via boundary scan.

### When to Skip Flying Probe

- **High volume.** Too slow (1-10 minutes vs. 5-30 seconds for ICT).
- **When you need FCT anyway.** Flying probe doesn't replace FCT for system-level tests.
- **Budget constraints at CM.** Some contract manufacturers charge premium for flying probe time.

## Test Strategy by Volume

| Volume (units/year) | Recommended Strategy | Why |
|---------------------|---------------------|-----|
| 1-100 (prototype) | FCT only | Low volume, simple fixture, catches most defects |
| 100-1,000 | Flying probe + FCT | Flying probe catches assembly defects, FCT catches system issues |
| 1,000-10,000 | Flying probe or ICT + FCT | Evaluate ICT fixture ROI at upper end |
| 10,000-100,000 | ICT + FCT | ICT fixture pays for itself, fast test time |
| 100,000+ | AOI + ICT + FCT | Full coverage, maximum throughput |

## Test Strategy by Board Complexity

| Board Complexity | Components | Recommended Tests |
|-----------------|------------|-------------------|
| Simple (LED driver, sensor board) | &lt; 50 | FCT only |
| Medium (MCU board, IoT device) | 50-200 | Flying probe + FCT |
| Complex (multi-rail, mixed signal) | 200-500 | ICT + FCT |
| Very complex (RF, high-speed digital) | 500+ | AOI + ICT + FCT + specialized |

## Coverage Comparison

| Defect Type | AOI | ICT | Flying Probe | FCT |
|------------|-----|-----|-------------|-----|
| Missing component | Yes | Yes | Yes | Sometimes |
| Wrong value | No | Yes | Yes | Sometimes |
| Solder short | Yes | Yes | Yes | Sometimes |
| Cold solder joint | Yes | No | No | Sometimes |
| Tombstoning | Yes | No | No | No |
| Wrong polarity | No | Yes | Yes | Sometimes |
| Firmware bug | No | No | No | Yes |
| Power rail issue | No | No | No | Yes |
| Communication failure | No | No | No | Yes |
| Performance degradation | No | No | No | Yes |

No single test method catches everything. The combination of ICT (or flying probe) for assembly defects plus FCT for system defects gives you the best coverage.

### Test Data Management for Electronics
URL: https://www.tofupilot.com/guides/test-data-management-for-electronics-a-developers-guide
Learn how to structure, store, and query electronics test data with TofuPilot for automatic traceability, yield tracking, and process control.

Every hardware test produces data. Voltages, pass/fail results, serial numbers, timestamps, operator IDs. The question isn't whether you have data. It's whether you can find it, trust it, and act on it six months later.

This guide covers how to structure, store, and query electronics test data with TofuPilot so you get automatic traceability, yield tracking, and process control without building any of it yourself.

## What Test Data Management Means

Test data management is the practice of organizing test results so they're queryable, traceable, and useful over time. Here's what it needs to answer:

| Question | What it requires |
|----------|-----------------|
| Did this unit pass? | A test run linked to a serial number with a clear pass/fail outcome |
| What failed and why? | Per-step measurements with limits, not just a top-level verdict |
| Is yield dropping? | Time-series aggregation of pass/fail across runs |
| Is this measurement drifting? | Historical measurement values with timestamps and limits |
| Can we trace this unit's full history? | Every test run linked to a unit, across stations and revisions |
| Which station has the worst yield? | Station-level metadata on every run |

If your current system can't answer all six, you've got a data problem, not a test problem.

## Why Spreadsheets and File Systems Fail at Scale

Most teams start with CSV exports or shared drives. That works for 10 units. It falls apart at 1,000.

| Capability | Spreadsheets / file system | Structured database (TofuPilot) |
|------------|---------------------------|-------------------------------|
| Schema consistency | No. Columns drift across files | Yes. Every run follows the same model |
| Query by serial number | Manual search | Instant lookup |
| Yield over time | Build it yourself in Excel | Built-in, automatic |
| Measurement traceability | Fragile, depends on naming | Enforced by data model |
| Multi-station aggregation | Copy-paste across files | Automatic, per-station metadata |
| Concurrent access | File locks, merge conflicts | Native multi-user |
| Audit trail | None | Immutable run history |

The core issue: flat files don't enforce relationships between units, runs, steps, and measurements. Without those relationships, every query is a one-off script.

## The Test Data Model

TofuPilot organizes test data into a hierarchy that maps directly to how hardware testing works:

| Entity | What it represents | Example |
|--------|-------------------|---------|
| Procedure | A test definition (what you're testing) | "FCT_PowerBoard_v2" |
| Run | A single execution of a procedure against a unit | Run #4821, serial SN-0042, PASS |
| Step | A phase or stage within a run | "measure_3v3_rail" |
| Measurement | A single data point within a step, with optional limits | 3.28 V (min: 3.13, max: 3.47) |
| Unit | A physical device identified by serial number | SN-0042 |
| Sub-unit | A component tracked within a parent unit | WiFi module WF-1122 inside SN-0042 |

Every run links to a procedure, a unit, and optionally a station. Steps and measurements nest inside runs. This structure means you can query in any direction: "show me all runs for this unit," "show me all measurements for this step across 10,000 runs," or "show me yield by station for this procedure."

## How TofuPilot Structures Test Data Automatically

If you're using OpenHTF, TofuPilot captures the full test structure (phases, measurements, limits, attachments) with zero extra code. Just wrap your test with TofuPilot:

```python filename="fct_power_board.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.13, 3.47).with_units(units.VOLT),
)
def measure_3v3_rail(test):
    voltage = read_voltage("3V3_RAIL")
    test.measurements.rail_3v3 = voltage


@htf.measures(
    htf.Measurement("rail_5v").in_range(4.75, 5.25).with_units(units.VOLT),
)
def measure_5v_rail(test):
    voltage = read_voltage("5V_RAIL")
    test.measurements.rail_5v = voltage


@htf.measures(
    htf.Measurement("current_draw").in_range(0.05, 0.5).with_units(units.AMPERE),
)
def measure_current_draw(test):
    current = read_current("VIN")
    test.measurements.current_draw = current


def main():
    test = htf.Test(
        measure_3v3_rail,
        measure_5v_rail,
        measure_current_draw,
        procedure_id="FCT_PowerBoard_v2",
    )

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-0042")


if __name__ == "__main__":
    main()
```

That single `TofuPilot(test)` wrapper sends the full run (phases, measurements, limits, units, outcome, duration) to TofuPilot. No serialization code, no API calls, no file management.

## Using the Python Client Directly

Not using OpenHTF? The TofuPilot Python client lets you log structured test data from any test framework or custom script:

```python filename="log_run.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

client.create_run(
    procedure_id="FCT_PowerBoard_v2",
    unit_under_test={"serial_number": "SN-0042"},
    run_passed=True,
    steps=[
        {
            "name": "measure_3v3_rail",
            "step_passed": True,
            "measurements": [
                {
                    "name": "rail_3v3",
                    "measured_value": 3.28,
                    "units": "V",
                    "lower_limit": 3.13,
                    "upper_limit": 3.47,
                }
            ],
        },
        {
            "name": "measure_5v_rail",
            "step_passed": True,
            "measurements": [
                {
                    "name": "rail_5v",
                    "measured_value": 5.01,
                    "units": "V",
                    "lower_limit": 4.75,
                    "upper_limit": 5.25,
                }
            ],
        },
        {
            "name": "measure_current_draw",
            "step_passed": True,
            "measurements": [
                {
                    "name": "current_draw",
                    "measured_value": 0.12,
                    "units": "A",
                    "lower_limit": 0.05,
                    "upper_limit": 0.5,
                }
            ],
        },
    ],
)
```

Same data model, same queryability. The client handles validation, batching, and retries.

## Built-in Analytics

TofuPilot computes FPY, Cpk, and failure Pareto charts automatically from your test data. Open the Analytics tab on any procedure to see yield trends, or create a custom Report for cross-procedure analysis.

**First Pass Yield (FPY)** shows the percentage of units that pass on the first attempt, plotted over time. You can filter by station, date range, or unit revision. A sudden FPY drop tells you something changed: a new component lot, a fixture problem, or a test script regression.

**Process Capability (Cpk)** is calculated per measurement across all runs. TofuPilot plots the measurement distribution against your spec limits and computes Cp and Cpk automatically. A Cpk below 1.33 means your process is too close to the limits.

**Failure Pareto** ranks test steps by failure count so you can focus on the biggest contributor first. TofuPilot builds this chart for any procedure, any time range.

**Custom Reports** let you combine FPY, Cpk, and failure data across multiple procedures into a single view. Use these for weekly quality reviews or to compare yield across production lines.

## Comparison: Approaches to Test Data Management

| Capability | CSV / shared drive | Custom database | TofuPilot |
|------------|-------------------|----------------|-----------|
| Structured data model | No | You build it | Built in |
| Query by serial number | Manual | SQL queries | Instant search |
| FPY tracking | Spreadsheet formulas | You build it | Automatic |
| Cpk analysis | Export to Minitab | You build it | Built in per measurement |
| Failure Pareto | Manual sorting | You build it | Automatic |
| Multi-station support | Separate files | You build it | Native, per-station metadata |
| Unit traceability | Fragile | You build it | Full history per serial number |
| Sub-unit tracking | Not practical | You build it | Built in |
| Audit trail | None | You build it | Immutable |
| Setup time | Minutes | Weeks to months | Minutes |
| Maintenance | Low (until it breaks) | Ongoing | Zero |

The pattern is clear. You can build all of this yourself, and many teams have. But every hour spent on test infrastructure is an hour not spent on the product you're actually testing.

### Hardware Telemetry Analysis
URL: https://www.tofupilot.com/guides/hardware-telemetry-analysis-with-tofupilot
Learn how to collect, store, and analyze hardware sensor telemetry data using TofuPilot's measurement arrays and dashboards.

Sensor data from hardware tests is only useful if you can find patterns across thousands of runs. Storing telemetry as structured, indexed measurements is what lets you surface trends before they become production issues.

## What Hardware Telemetry Looks Like in Practice

A single hardware test run can produce hundreds of sensor readings: temperature curves, voltage traces, vibration spectra, pressure waveforms. Without a structured system, this data ends up in CSV files on shared drives, impossible to query at scale.

Treat every sensor measurement as a first-class object. Each reading gets a name, unit, limits, and an optional array dimension for time-series or multi-axis data.

## Ingesting Sensor Data

### OpenHTF Users

OpenHTF measurements flow in automatically. Multi-dimensional arrays (1D waveforms, 2D matrices, ND tensors) are processed without code changes.

```python filename="telemetry_test.py"
import openhtf as htf
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("temperature_curve").with_dimensions("time_s"),
    htf.Measurement("vibration_spectrum").with_dimensions("freq_hz"),
)
def sensor_sweep(test):
    for t in range(100):
        test.measurements.temperature_curve[t] = read_thermocouple()
    for f in range(500):
        test.measurements.vibration_spectrum[f] = read_accelerometer_fft(f)


def main():
    test = htf.Test(
        sensor_sweep,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="DUT-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "DUT-001")


if __name__ == "__main__":
    main()
```

Install it with `pip install "tofupilot[openhtf]"`. The `upload()` callback reads `TOFUPILOT_API_KEY` from the environment.

### Python SDK Users

If you are not running OpenHTF, create the run directly. Dimensional data goes in as a series of measurements under a phase:

```python filename="upload_telemetry.py"
import os
from datetime import datetime, timedelta, timezone

import numpy as np
from tofupilot.v2 import TofuPilot

started = datetime.now(timezone.utc) - timedelta(seconds=30)
ended = datetime.now(timezone.utc)

waveform = np.sin(np.linspace(0, 2 * np.pi, 1000))
ripple_pk_pk = float(waveform.max() - waveform.min())

with TofuPilot(api_key=os.getenv("TOFUPILOT_API_KEY")) as client:
    client.runs.create(
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        serial_number="PSU-2025-0042",
        part_number="PSU-100",
        outcome="PASS",
        started_at=started,
        ended_at=ended,
        phases=[{
            "name": "Output Ripple",
            "outcome": "PASS",
            "started_at": started,
            "ended_at": ended,
            "measurements": [{
                "name": "Ripple Peak to Peak",
                "measured_value": ripple_pk_pk,
                "units": "mV",
                "validators": [
                    {"operator": "<=", "expected_value": 50},
                ],
            }],
        }],
    )
```

Note what this example does with the waveform: it uploads the extracted scalar, not the thousand raw samples. That is deliberate, and it is the single most useful habit in telemetry work. Store the features you will actually query on (peak-to-peak, rise time, overshoot) as measurements with limits, and keep the full capture as an attachment on failures or a sample. A test record platform is for results; it is not a time-series acquisition store, and uploading every sample from every unit will outgrow its usefulness fast.

## Querying Telemetry at Scale

Once ingested, every measurement is queryable. Filter by procedure, unit serial number, date range, or outcome.

| Query | What it shows |
|-------|--------------|
| All runs for a procedure over the last 7 days | Ripple voltage trends across production |
| Failed runs with `temperature_curve` out of limits | Units that exceeded thermal specs |
| Measurement distribution for `vibration_spectrum` | Histogram of vibration amplitudes across the fleet |

## Catching Telemetry Drift

Measurement distributions are tracked over time. When a sensor reading starts drifting toward its limits, you can catch it before it causes failures.

The practical approach is to alert on the trend rather than the threshold. By the time a reading crosses its limit, the drift has usually been present for weeks. Watch the mean and the spread separately: a moving mean with a tight spread points to something that shifted, such as a recalibrated instrument or a new component lot, while a stable mean with a widening spread points to something that became inconsistent, such as a worn fixture or temperature variation across a shift.

This matters most for temperature sensors, where gradual calibration drift goes unnoticed until units start failing in the field.

## Where This Pays Off

A thermal cycling test across 8 temperature zones produces 8 time-series measurements per run, one per zone. The manual version of this is downloading CSV files from each chamber controller, merging them in a spreadsheet, and checking limits by hand.

With the measurements uploaded per run, the limits are evaluated on ingest and the zone-by-zone history is queryable across the full production record. The gain is not that the analysis becomes possible; it is that it stops depending on someone remembering to do it.

### Hardware Test Infrastructure
URL: https://www.tofupilot.com/guides/hardware-test-infrastructure-with-tofupilot
Learn how to build scalable hardware test infrastructure using TofuPilot as the data backbone for test stations, instruments, and automation.

Test infrastructure is everything between the engineer's test script and the data that drives quality decisions. Instruments, fixtures, stations, software frameworks, data pipelines. TofuPilot sits at the center as the data backbone, connecting your test stations to dashboards, analytics, and traceability.

## What Hardware Test Infrastructure Looks Like

A production test setup has multiple layers:

```
┌────────────────────────────────────────────┐
│              TofuPilot (Cloud)              │
│  Dashboards · Analytics · Traceability     │
├────────────────────────────────────────────┤
│           Test Framework Layer              │
│  OpenHTF · pytest · Custom Python          │
├────────────────────────────────────────────┤
│           Instrument Layer                  │
│  DMMs · Oscilloscopes · Power Supplies     │
│  DAQs · Spectrum Analyzers · Load Banks    │
├────────────────────────────────────────────┤
│           Fixture Layer                     │
│  Pogo pin fixtures · Cable harnesses       │
│  Pneumatic actuators · Thermal chambers    │
├────────────────────────────────────────────┤
│           DUT (Device Under Test)           │
└────────────────────────────────────────────┘
```

Each layer has its own concerns. TofuPilot handles the top layer: collecting, storing, and analyzing the data that flows up from the test stack.

## Station Architecture

A well-designed test station separates concerns:

| Component | Responsibility | Tools |
|-----------|---------------|-------|
| Test logic | What to test and in what order | OpenHTF, pytest, custom Python |
| Instrument control | Communicating with test equipment | PyVISA, python-ivi, vendor drivers |
| Fixture control | Engaging/disengaging the DUT | GPIO, serial, pneumatic controllers |
| Data capture | Recording measurements | TofuPilot client |
| Operator interface | Scan serial, display results | OpenHTF UI, custom GUI |

### Minimal Station Setup

```python filename="minimal_station.py"
from tofupilot import TofuPilotClient
import pyvisa

# Instrument layer
rm = pyvisa.ResourceManager()
dmm = rm.open_resource("TCPIP::192.168.1.10::INSTR")
psu = rm.open_resource("TCPIP::192.168.1.11::INSTR")

# Test logic
def run_functional_test(serial_number):
    psu.write("OUTP ON")
    vcc = float(dmm.query("MEAS:VOLT:DC?"))
    current = float(dmm.query("MEAS:CURR:DC?"))
    psu.write("OUTP OFF")

    passed = 3.25 <= vcc <= 3.35 and 30 <= current <= 60

    # Data capture
    client = TofuPilotClient()
    client.create_run(
        procedure_id="BOARD-FUNCTIONAL",
        unit_under_test={"serial_number": serial_number},
        run_passed=passed,
        steps=[{
            "name": "Power Rails",
            "step_type": "measurement",
            "status": 3.25 <= vcc <= 3.35,
            "measurements": [
                {"name": "vcc_3v3", "value": vcc, "unit": "V", "limit_low": 3.25, "limit_high": 3.35},
                {"name": "idle_current_ma", "value": current * 1000, "unit": "mA", "limit_low": 30, "limit_high": 60},
            ],
        }],
    )
    return passed

# Operator interface
while True:
    serial = input("Scan serial (or 'q' to quit): ")
    if serial.lower() == "q":
        break
    result = run_functional_test(serial)
    print(f"{'PASS' if result else 'FAIL'}")
```

## Scaling from One Station to Many

### Identical Stations

When you add stations for the same test, keep the test code identical across all stations. Use the same procedure ID. TofuPilot distinguishes stations automatically.

```python filename="station_config.py"
import socket

# Station identification
STATION_ID = socket.gethostname()  # Each station PC has a unique hostname

# Same test code, same procedure, different station
client.create_run(
    procedure_id="BOARD-FUNCTIONAL",
    unit_under_test={"serial_number": serial},
    run_passed=passed,
    steps=steps,
)
# TofuPilot tracks which station ran each test
```

### Different Test Stages

Production flows typically have multiple test stages. Each stage gets its own procedure.

```
Assembly Line:
  Station 1-4: ICT (In-Circuit Test)        → procedure: "ICT-V2"
  Station 5-6: Functional Test               → procedure: "FUNC-TEST-V3"
  Station 7:   Burn-In                       → procedure: "BURN-IN-24H"
  Station 8-10: Final Test                   → procedure: "FINAL-TEST-V2"
```

TofuPilot links all test stages for a given serial number, creating a complete manufacturing test history.

## Instrument Management

### Common Instrument Types

| Instrument | What it measures | Interface |
|-----------|-----------------|-----------|
| DMM (Digital Multimeter) | Voltage, current, resistance | GPIB, LAN, USB |
| Oscilloscope | Waveforms, timing | LAN, USB |
| Power Supply | (Provides power, measures output) | GPIB, LAN, USB |
| DAQ (Data Acquisition) | Multi-channel analog/digital | USB, PCIe |
| Spectrum Analyzer | Frequency content | GPIB, LAN |
| Load Bank | (Simulates load conditions) | Serial, LAN |
| Environmental Chamber | Temperature, humidity | Serial, LAN |

### Instrument Connection Patterns

```python filename="instrument_pool.py"
import pyvisa

class InstrumentPool:
    """Manage instrument connections for a test station."""

    def __init__(self):
        self.rm = pyvisa.ResourceManager()
        self._instruments = {}

    def get(self, name, address):
        if name not in self._instruments:
            self._instruments[name] = self.rm.open_resource(address)
        return self._instruments[name]

    def close_all(self):
        for inst in self._instruments.values():
            inst.close()

# Usage
pool = InstrumentPool()
dmm = pool.get("dmm", "TCPIP::192.168.1.10::INSTR")
psu = pool.get("psu", "TCPIP::192.168.1.11::INSTR")
```

## Infrastructure Monitoring

Your test infrastructure is itself something that needs monitoring. TofuPilot's dashboards reveal infrastructure health:

| Metric | What it indicates |
|--------|------------------|
| FPY per station | Station hardware health |
| Cycle time per station | Instrument/fixture performance |
| Measurement variance per station | Fixture contact quality |
| Uptime per station | Reliability of test PC/software |

When one station's FPY drops while others stay stable, the problem is the station, not the product. Investigate the fixture, instruments, and cabling.

## Infrastructure Best Practices

| Practice | Why |
|----------|-----|
| Version your test code | Know exactly what test ran on each unit |
| Use the same procedure ID across identical stations | Enables cross-station comparison |
| Store instrument calibration dates | Know when calibration might affect results |
| Keep fixtures as simple as possible | Fewer moving parts = fewer failure modes |
| Monitor cycle time | It's a free health indicator |
| Separate test logic from instrument drivers | Swap instruments without rewriting tests |
| Use IP-based instrument connections | More reliable than USB for production |

## What TofuPilot Handles vs. What You Handle

| TofuPilot handles | You handle |
|------------------|-----------|
| Data storage and indexing | Test code and logic |
| Dashboards and analytics | Instrument drivers |
| Traceability and search | Fixture design and maintenance |
| Yield calculations | Station hardware setup |
| Measurement trending | Network connectivity |
| Cross-station comparison | Operator training |

TofuPilot is the data layer. You bring the test layer. Together they form a complete test infrastructure.


### Which LabVIEW Version Runs on Each Station?
URL: https://www.tofupilot.com/guides/which-labview-version-runs-on-each-station
Learn how to record the LabVIEW version, bitness and Run-Time Engine on every station, catch a stranded build before release, and keep the list true.

A built LabVIEW executable runs only on the Run-Time Engine of the same version and bitness, so the answer to "which LabVIEW version runs on each station" has to be written down, one row per station. Most plants can't produce that list in under a day, because the version lives in an installer someone ran three years ago and nowhere else. This guide gives you the inventory, a script that flags the mismatches, and a way to keep the list true.

## The Failure Mode: A Build That Won't Start

The development machine gets upgraded to LabVIEW 2024 64-bit. A limit change is built there, copied to a station that has run the same executable since 2019, and the operator double-clicks it. A dialog says the required LabVIEW Run-Time Engine can't be found, names the version, and points at ni.com.

Nothing is wrong with the build. The compiled code inside a LabVIEW executable targets one Run-Time Engine version and one bitness, and the station has RTE 2019 32-bit installed. RTE 2024 64-bit is a separate install, and so is RTE 2024 32-bit.

Sometimes the executable starts and fails later instead. NI-DAQmx and NI-VISA each support a range of LabVIEW versions, so a station on an old driver can load the new executable and then error on the first DAQ call. Either way the station is down until someone with admin rights installs the right RTE and drivers, on a floor PC that often has no internet connection.

## How to Find the RTE on a Station

You need four things from each station: the Run-Time Engine version, its bitness, the NI-DAQmx version and the NI-VISA version. Then you need the LabVIEW version and bitness the executable was built with, which comes from the development machine, not the station.

| What you need | Where it is |
|---|---|
| RTE version and bitness | Windows Programs and Features, entries named like "NI LabVIEW Runtime 2019 (32-bit)" |
| RTE on newer stations | NI Package Manager, Installed tab, filter on "Runtime" |
| NI-DAQmx and NI-VISA versions | Same two places, entries "NI-DAQmx" and "NI-VISA" |
| Version the executable was built with | The build specification on the development machine, or the About box if the author added one |
| Build date | The executable's Properties, Details tab, plus the version information from the build specification if it was filled in |

Check the bitness twice. A 32-bit LabVIEW on a 64-bit Windows produces a 32-bit executable that needs the 32-bit RTE, and Programs and Features lists both variants with the same year.

Write down the development machine's bitness as well. A 32-bit and a 64-bit LabVIEW of the same year install side by side, the build specification decides which one builds, and after an upgrade it's easy to build with the one the stations don't have.

## The Inventory Template

One row per station, one file for the plant, under Git next to the hand-over documents.

```csv filename="stations.csv"
station,product,executable_build_date,labview_version,bitness,rte_installed,daqmx,visa,last_verified
FCT-01,controller-pcba,2024-11-03,2019,32-bit,2019 32-bit,19.6,19.5,2026-09-01
FCT-02,controller-pcba,2026-08-14,2024,64-bit,2019 32-bit,19.6,19.5,2026-09-01
EOL-01,drone-assembly,2023-06-18,2020,32-bit,2020 32-bit,20.1,20.0,2026-03-12
CAL-01,imu-module,2021-02-09,2018,32-bit,2018 32-bit,18.6,18.0,
```

`labview_version` and `bitness` describe the build. `rte_installed` describes the station, in the same "year bitness" format so the two can be compared by a script. `last_verified` is the date someone stood at the station and checked, not the date the row was typed.

Leave `last_verified` empty when nobody has checked. An empty cell is more useful than a guess, because the script below treats it as a problem.

## A Script That Prints the Mismatches

This reads the file and prints every station where the build and the installed RTE disagree, or where the row is stale.

```python filename="rte_check.py"
# Prints stations whose installed Run-Time Engine doesn't match their build, or whose row is stale.
import csv
from datetime import date

MAX_AGE_DAYS = 90

with open("stations.csv", newline="") as f:
    rows = list(csv.DictReader(f))

for row in rows:
    needed = f"{row['labview_version']} {row['bitness']}"
    installed = row["rte_installed"].strip()
    problems = []
    if installed != needed:
        problems.append(f"needs RTE {needed}, has {installed or 'nothing recorded'}")
    if not row["last_verified"]:
        problems.append("never verified")
    else:
        age = (date.today() - date.fromisoformat(row["last_verified"])).days
        if age > MAX_AGE_DAYS:
            problems.append(f"last verified {age} days ago")
    if problems:
        print(f"{row['station']:8} {'; '.join(problems)}")
```

For the sample file it prints three lines. FCT-02 needs RTE 2024 64-bit and has 2019 32-bit, which is the stranded build from the first section. EOL-01 was last verified more than 90 days ago, and CAL-01 has never been verified at all.

Run it before every release. If it prints the station you're about to copy a build to, install the RTE first, or build with the version the station already has. The upgrade itself is covered in [upgrade a station from LabVIEW 2019 to 2024](/guides/upgrade-a-station-from-labview-2019-to-2024).

## Keeping It True

The list decays the day after you write it, unless updating it is part of releasing a build. Three habits keep it accurate.

Verify at every release. Whoever copies an executable to a station updates `executable_build_date`, `labview_version`, `bitness` and `last_verified` on the same day. No build leaves the development machine without its row changing.

Put the version in the executable. Fill the version information in the build specification, show it in an About box with the LabVIEW version and bitness, and have the build write a `version.txt` next to the executable. Then anyone at the station can read the build without a LabVIEW licence.

Add a row for the development machine. It's the one whose upgrade strands everything else, and the day it moves to a new version the script flags every station it builds for.

Give the file one owner and a home. It belongs in the same repository as the station hand-over documents, which [document a test station for hand-over](/guides/document-a-test-station-for-hand-over) describes, so the person who inherits the station inherits the list. The reason the list is needed at all, binary VIs and per-version toolchains, is covered in [why LabVIEW version control is so painful](/guides/why-labview-version-control-is-so-painful).

The dry part of this exercise is discovering that two stations with the same label have run different builds for a year.

## The Contrast: When the Station Is Text

A Python station under the TofuPilot Framework has no Run-Time Engine to match. The station is a folder of `procedure.yaml`, `phases/*.py` and `plugs/*.py`, and a Git push builds a deployment with an id like `dep_abc123`, immutable and tied to one commit.

The CSV becomes a query. `tofupilot deployments ls --procedure-ids <id> --environments production` lists what's built, the station page in TofuPilot shows which deployment each station runs, and every uploaded run carries the `deployment_id` that produced it. "Which version runs on FCT-02" is a lookup, and "which units ran the old version" is a filter on the run list, which [trace a test change to the units it shipped](/guides/trace-a-test-change-to-the-units-it-shipped) walks through.

## Start With One Station

Take the station the script flagged with the most runs per week, rebuild it in Python with the TofuPilot Framework with the same limits, and run it next to the LabVIEW version on the same units for two weeks. The Python side needs the CLI and nothing else, on any version of Windows the station already has.

```bash filename="install-and-run.sh"
# Install the CLI, then run the station locally (no account needed) or with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### RF Testing and Calibration
URL: https://www.tofupilot.com/guides/rf-testing-and-calibration-with-tofupilot
Learn how to log RF test and calibration data for wireless products using TofuPilot, covering output power, sensitivity, and frequency accuracy.

Every wireless product needs RF testing: output power, receiver sensitivity, frequency accuracy, spurious emissions. For products with WiFi, Bluetooth, LoRa, cellular, or custom RF, these measurements determine whether the product meets regulatory requirements and actually works in the field. TofuPilot stores RF test data for trending, calibration tracking, and compliance.

## RF Test Parameters

| Parameter | What it measures | Why it matters |
|-----------|-----------------|---------------|
| TX output power (dBm) | Transmitted signal strength | Too high: regulatory violation. Too low: poor range |
| RX sensitivity (dBm) | Minimum receivable signal | Determines receive range |
| Frequency error (ppm) | Crystal/oscillator accuracy | Affects interoperability |
| EVM (Error Vector Magnitude) | Modulation quality | Affects data throughput |
| Spurious emissions | Unintended RF output | Regulatory compliance |
| RSSI accuracy | Received signal strength indicator | Affects link management |

## Logging RF Test Results

### Basic RF Parametric Test

```python filename="rf_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

def rf_production_test(serial, tx_power, rx_sensitivity, freq_error_ppm, evm_pct):
    client.create_run(
        procedure_id="RF-PARAMETRIC-BLE",
        unit_under_test={
            "serial_number": serial,
            "part_number": "IOT-SENSOR-V3",
        },
        run_passed=True,
        steps=[{
            "name": "TX Performance",
            "step_type": "measurement",
            "status": 0 <= tx_power <= 4,
            "measurements": [
                {"name": "tx_power_dbm", "value": tx_power, "unit": "dBm", "limit_low": 0, "limit_high": 4},
                {"name": "freq_error_ppm", "value": freq_error_ppm, "unit": "ppm", "limit_low": -20, "limit_high": 20},
                {"name": "evm_pct", "value": evm_pct, "unit": "%", "limit_high": 30},
            ],
        }, {
            "name": "RX Performance",
            "step_type": "measurement",
            "status": rx_sensitivity <= -90,
            "measurements": [
                {"name": "rx_sensitivity_dbm", "value": rx_sensitivity, "unit": "dBm", "limit_high": -90},
            ],
        }],
    )
```

### Multi-Channel RF Test

Test across all operating channels to catch channel-specific issues.

```python filename="rf_multi_channel.py"
# BLE has 40 channels (0-39)
# Test a representative set
test_channels = [0, 12, 19, 20, 38, 39]

measurements = []
for ch in test_channels:
    set_channel(ch)
    power = measure_tx_power()
    measurements.append({
        "name": f"tx_power_ch{ch}_dbm",
        "value": power,
        "unit": "dBm",
        "limit_low": 0,
        "limit_high": 4,
    })

client.create_run(
    procedure_id="RF-MULTI-CHANNEL-BLE",
    unit_under_test={"serial_number": serial},
    run_passed=all(0 <= m["value"] <= 4 for m in measurements),
    steps=[{
        "name": "Multi-Channel TX Power",
        "step_type": "measurement",
        "status": True,
        "measurements": measurements,
    }],
)
```

## RF Calibration

Many wireless products require per-unit RF calibration to compensate for component tolerances. The calibration process:

1. Measure actual TX power at a reference level
2. Calculate the offset from target
3. Write a calibration value to the DUT's flash/EEPROM
4. Verify calibrated output matches target

```python filename="rf_calibration.py"
def calibrate_rf(serial):
    # Step 1: Measure uncalibrated power
    raw_power = measure_tx_power()

    # Step 2: Calculate calibration offset
    target_power = 0.0  # dBm
    cal_offset = target_power - raw_power

    # Step 3: Write calibration to DUT
    write_cal_value(cal_offset)

    # Step 4: Verify
    calibrated_power = measure_tx_power()

    client.create_run(
        procedure_id="RF-CALIBRATION-BLE",
        unit_under_test={"serial_number": serial},
        run_passed=abs(calibrated_power - target_power) < 1.0,
        steps=[{
            "name": "TX Calibration",
            "step_type": "measurement",
            "status": True,
            "measurements": [
                {"name": "raw_tx_power_dbm", "value": raw_power, "unit": "dBm"},
                {"name": "cal_offset_db", "value": cal_offset, "unit": "dB"},
                {"name": "calibrated_tx_power_dbm", "value": calibrated_power, "unit": "dBm",
                 "limit_low": -1.0, "limit_high": 1.0},
            ],
        }],
    )
```

## Tracking RF Performance Across Production

### Calibration Offset Distribution

The distribution of calibration offsets tells you about your RF hardware consistency. A tight distribution (small offsets) means consistent PCB manufacturing. A wide or drifting distribution suggests:

- Antenna placement variation
- PCB impedance variation
- Component lot changes (balun, matching network)

### TX Power Over Time

Plot TX power across production. Look for:

| Pattern | Cause |
|---------|-------|
| Gradual drift | Test equipment calibration drift |
| Step change | New component lot, PCB revision |
| Increased variance | Manufacturing process variation |
| Channel-dependent failures | Antenna or matching network issues |

### RX Sensitivity Monitoring

Receiver sensitivity is harder to measure in production (requires a shielded environment and calibrated signal source). Track it to catch:

- LNA gain degradation
- Filter mistuning
- Increased noise figure from layout changes

## Test Fixture Considerations

RF testing requires controlled RF environments:

| Setup | Use case | Isolation |
|-------|----------|-----------|
| Shielded box | Production testing | 60-80 dB |
| Anechoic chamber | Antenna characterization | 80+ dB |
| RF cable (conducted) | Board-level testing | Direct connection |

Log the fixture type and serial number with each test run. If RF test results shift, the fixture calibration or cable condition may have changed.

## Regulatory Compliance Data

RF certification (FCC, CE, IC, MIC) requires specific test data. While certification testing is done at accredited labs, pre-compliance data from TofuPilot helps you:

- Predict certification outcomes based on production margins
- Compare production units to the original certified unit
- Track any changes that could void certification


### What Is Ongoing Reliability Testing (ORT)
URL: https://www.tofupilot.com/guides/what-is-ort-with-tofupilot
Ongoing reliability testing pulls units from the line and stress-tests them on a schedule. How ORT works and how to track results over time.

Ongoing reliability testing (ORT) pulls a sample of units from production and stress-tests them continuously, to catch reliability problems that end-of-line functional test cannot see. FQC tells you a unit works today. ORT tells you whether this month's units still work after 500 hours at 55 °C.

## ORT vs the Other Reliability Tests

| Test | When | Sample | Purpose |
|------|------|--------|---------|
| **HALT** | Design phase | A few units | Find design margins by stressing to destruction |
| **HASS** | Production ramp | Every unit or a large sample | Screen out infant mortality with a proven stress profile |
| **ORT** | Ongoing production | Small sample, continuously | Detect reliability drift in the running process |
| **Burn-in** | Production | Every unit or a sample | Precipitate early-life failures before shipment |

The distinguishing feature of ORT is that it never ends. It runs for the life of the product, on units drawn from normal production, and it exists to answer one question: has anything about the product or the process changed in a way that will show up in the field in six months?

## What ORT Catches That FQC Does Not

- **Supplier substitutions.** A capacitor vendor change passes functional test and fails at temperature 300 hours in.
- **Process drift.** Reflow profile drifts, solder joints get marginal, thermal cycling finds them.
- **Wear-out mechanisms.** Connector insertion cycles, relay contacts, fan bearings, battery cycling.
- **Firmware regressions under duration.** Memory leaks and counter rollovers that no 90-second EOL test will ever reach.

## Designing an ORT Program

| Decision | Typical choice | Notes |
|----------|---------------|-------|
| Sample size | 2-10 units per week | Enough to detect a shift, small enough to afford |
| Sampling method | Random across shifts and lines | Systematic sampling hides line-specific problems |
| Duration | 500-1000 hours, continuous | Long enough to reach the wear-out region |
| Stress profile | Elevated temperature, thermal cycling, duty cycling | Representative of field use, accelerated |
| Measurement interval | Every 24 hours, plus at entry and exit | Frequent enough to see drift, not so frequent it dominates |
| Exit criteria | Zero failures, or a defined failure rate | Defined before the program starts, not after a failure |

## Implementing ORT as a Procedure

ORT has a shape that suits a declarative procedure: the same measurements, repeated on a schedule, on a unit that already exists in the system.

```yaml filename="procedure.yaml"
name: ORT Checkpoint
version: 1.0.0
description: 24-hour ORT checkpoint measurement, per REL-220

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: chamber
    python: plugs.chamber:Chamber
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Chamber Conditions
    python: phases.chamber_conditions
    measurements:
      - name: Chamber Temperature
        unit: "°C"
        description: Soak temperature at time of measurement
        validators:
          - operator: ">="
            expected_value: 54.0
          - operator: "<="
            expected_value: 56.0
  - name: Functional Checkpoint
    python: phases.functional_checkpoint
    measurements:
      - name: Output Voltage
        unit: V
        description: Same limits as EOL functional test
        validators:
          - operator: ">="
            expected_value: 4.75
          - operator: "<="
            expected_value: 5.25
      - name: Quiescent Current
        unit: mA
        validators:
          - operator: "<="
            expected_value: 45.0
      - name: Still Booting
        validators:
          - operator: "=="
            expected_value: true
```

```python filename="phases/chamber_conditions.py"
def chamber_conditions(measurements, chamber):
    """Record the actual soak conditions, not the setpoint."""
    measurements.chamber_temperature = chamber.read_temperature()
```

```python filename="phases/functional_checkpoint.py"
def functional_checkpoint(measurements, dmm):
    """Re-run the EOL measurements on a unit that has been soaking."""
    measurements.output_voltage = dmm.measure_dc_voltage()
    measurements.quiescent_current = dmm.measure_dc_current_ma()
    measurements.still_booting = True
```

```python filename="plugs/chamber.py"
class Chamber:
    def read_temperature(self) -> float:
        return 55.1  # Replace with a real chamber query
```

```python filename="plugs/dmm.py"
class Multimeter:
    def measure_dc_voltage(self) -> float:
        return 5.02  # Replace with a real instrument read

    def measure_dc_current_ma(self) -> float:
        return 41.3  # Replace with a real instrument read
```

Two deliberate choices in that procedure:

**The functional limits are identical to the end-of-line test.** ORT is only meaningful as a comparison against the unit's own EOL result. Same measurement names, same limits, so the drift is visible as a trend on one chart rather than two unrelated numbers.

**The chamber conditions are measured, not assumed.** A checkpoint taken while the chamber was recovering from a door opening is not comparable to one taken at soak. Recording the actual temperature lets you discard or annotate those points instead of chasing a phantom drift.

## Recording the Elapsed Hours

The independent variable in ORT is time under stress, so it has to be on the run.

```python filename="phases/record_elapsed.py"
def record_elapsed(run):
    """Hours since this unit entered the ORT chamber."""
    run.metadata["ort_hours"] = 168
    run.metadata["ort_lot"] = "2026-W38"
```

With `ort_hours` on every run and the same serial number across checkpoints, the unit's history is the reliability curve: open the serial and the measurements are in order from hour 0 to hour 1000.

## Running It on a Schedule

An ORT checkpoint has no operator. Resolve the identify prompt from the procedure and pre-bake any UI values so a cron job or scheduler can drive it:

```bash filename="ort/checkpoint.sh"
tofupilot run ./procedure.yaml --upload --json --ui-values ui.json --ui-timeout 300
```

## Reading the Results

| Signal | What it means | Action |
|--------|--------------|--------|
| A single unit fails at hour 700 | Possibly normal wear-out, possibly a real defect | Failure analysis on that unit; check whether its lot is over-represented |
| Every unit from one week drifts | Process or supplier change in that window | Pull the build records for that lot |
| Quiescent current climbs across all units | A wear-out mechanism reaching the population | Compare against the design margin; this is a product problem |
| Chamber temperature out of band | The stress was not applied | Discard the checkpoint, fix the chamber |
| No failures for a year | Either genuinely robust, or the stress is too gentle | Revisit the profile; ORT that never fails may not be testing anything |

That last row is the one most programs get wrong. An ORT program with zero findings over a long period is not automatically good news; it may mean the stress profile is below the level that provokes any real mechanism.

## If You Already Run OpenHTF

Existing OpenHTF suites run natively on the TofuPilot CLI. The same checkpoint with decorators:

```python filename="ort_checkpoint.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(4.75, 5.25)
    .with_units(units.VOLT),
    htf.Measurement("quiescent_current_mA")
    .in_range(maximum=45.0)
    .with_units(units.MILLIAMPERE),
)
def phase_checkpoint(test):
    test.measurements.output_voltage_V = 5.02
    test.measurements.quiescent_current_mA = 41.3
    test.metadata["ort_hours"] = 168


test = htf.Test(phase_checkpoint, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan ORT unit serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Different limits than the EOL test | Drift is invisible because the numbers are not comparable | Reuse the EOL measurement names and limits |
| Sampling only from one line or shift | Line-specific problems never enter the sample | Randomise across lines and shifts |
| Not recording elapsed hours | No reliability curve, just scattered pass/fail | Put `ort_hours` on every run |
| Assuming the chamber setpoint | Checkpoints taken during recovery pollute the trend | Measure the actual conditions |
| Stopping ORT when yield looks good | The program exists precisely to catch what yield cannot | Keep it running for the product life |
| Discarding failed ORT units without analysis | The most informative unit you have gets thrown away | Failure analysis on every ORT failure |

### SCPI Commands in Python for Test Engineers
URL: https://www.tofupilot.com/guides/scpi-commands-in-python-a-complete-guide-for-test-engineers
Reference for SCPI commands in Python with PyVISA: measurement types, triggering, error handling, and wiring instruments into a TofuPilot procedure.

SCPI (Standard Commands for Programmable Instruments) is the command language that most modern test instruments speak. If your multimeter, power supply, or oscilloscope was made after 1990, it almost certainly supports SCPI. This guide covers the commands you'll use most in manufacturing test, how to send them from Python with PyVISA, and how to wire them into a production test procedure.

## What Is SCPI

SCPI defines a standard set of text commands for controlling instruments. Instead of learning a different protocol for every vendor, you use the same command structure across Keysight, Rigol, Rohde &amp; Schwarz, Tektronix, and others.

Commands follow a tree structure:

```
:MEASure
  :VOLTage
    :DC?        → Measure DC voltage
    :AC?        → Measure AC voltage
  :CURRent
    :DC?        → Measure DC current
  :RESistance?  → Measure resistance
```

The colon separates levels. A question mark means "query" (read a value). No question mark means "command" (set something).

## Setup

```bash filename="install.sh"
pip install pyvisa pyvisa-py
```

Connect to an instrument:

```python filename="scpi/connect.py"
import pyvisa

rm = pyvisa.ResourceManager("@py")  # Pure Python backend
dmm = rm.open_resource("TCPIP::192.168.1.100::INSTR")
dmm.timeout = 5000  # 5 seconds

# Every SCPI instrument responds to *IDN?
idn = dmm.query("*IDN?")
print(f"Connected: {idn.strip()}")
```

## Common SCPI Commands

### IEEE 488.2 Mandatory Commands

Every SCPI instrument supports these:

| Command | Purpose | Returns |
|---------|---------|---------|
| `*IDN?` | Identify instrument | Manufacturer, model, serial, firmware |
| `*RST` | Reset to factory defaults | Nothing |
| `*CLS` | Clear status and error queue | Nothing |
| `*OPC?` | Operation complete query | "1" when done |
| `*TST?` | Self-test | 0 = pass, nonzero = fail |
| `*WAI` | Wait for pending operations | Nothing (blocks until done) |

Always start a test with `*RST` and `*CLS`. This puts the instrument in a known state.

```python filename="scpi/reset.py"
dmm.write("*RST")
dmm.write("*CLS")
```

### Measurement Commands

Two patterns: shorthand and explicit.

**Shorthand** (configure + trigger + read in one command):

```python filename="scpi/measure_shorthand.py"
# DC voltage, auto-range
voltage = float(dmm.query(":MEAS:VOLT:DC?"))

# DC current, auto-range
current = float(dmm.query(":MEAS:CURR:DC?"))

# Resistance, auto-range
resistance = float(dmm.query(":MEAS:RES?"))

# AC voltage
ac_voltage = float(dmm.query(":MEAS:VOLT:AC?"))
```

**Explicit** (separate configure, trigger, read for more control):

```python filename="scpi/measure_explicit.py"
# Configure for DC voltage, 10V range
dmm.write(":CONF:VOLT:DC 10")

# Trigger a measurement
dmm.write(":INIT")

# Wait for completion
dmm.query("*OPC?")

# Read the result
voltage = float(dmm.query(":FETCH?"))
```

Use the explicit pattern when you need precise timing control or when measuring multiple channels in sequence.

| Pattern | When to Use | Commands |
|---------|------------|----------|
| `:MEAS:...?` | Quick single reading | 1 command |
| `:CONF:` + `:INIT` + `:FETCH?` | Precise timing, multi-channel | 3+ commands |
| `:CONF:` + `:READ?` | Configure once, read repeatedly | 2 commands |

### Power Supply Commands

```python filename="scpi/power_supply.py"
import pyvisa
import time

rm = pyvisa.ResourceManager("@py")
psu = rm.open_resource("TCPIP::192.168.1.101::INSTR")
psu.timeout = 5000

psu.write("*RST")
psu.write("*CLS")

# Select channel and configure
psu.write(":INST:SEL CH1")
psu.write(":VOLT 5.0")          # Set 5V output
psu.write(":CURR 0.5")          # 500mA current limit

# Enable output
psu.write(":OUTP ON")
time.sleep(0.5)                  # Wait for stabilization

# Read actual values
actual_v = float(psu.query(":MEAS:VOLT?"))
actual_i = float(psu.query(":MEAS:CURR?"))
print(f"Output: {actual_v:.3f}V, {actual_i:.4f}A")

# Always disable output at the end
psu.write(":OUTP OFF")
psu.close()
```

Common power supply commands:

| Command | Purpose |
|---------|---------|
| `:INST:SEL CH1` | Select output channel |
| `:VOLT 5.0` | Set voltage |
| `:CURR 0.5` | Set current limit |
| `:OUTP ON` / `:OUTP OFF` | Enable/disable output |
| `:MEAS:VOLT?` | Read actual output voltage |
| `:MEAS:CURR?` | Read actual output current |
| `:VOLT:PROT 6.0` | Set over-voltage protection |
| `:CURR:PROT 1.0` | Set over-current protection |

### Oscilloscope Commands

```python filename="scpi/oscilloscope.py"
# Basic oscilloscope setup
scope = rm.open_resource("TCPIP::192.168.1.102::INSTR")
scope.timeout = 10000

scope.write("*RST")
scope.write(":CHAN1:DISP ON")           # Enable channel 1
scope.write(":CHAN1:SCAL 1.0")          # 1V/div
scope.write(":TIM:SCAL 0.001")         # 1ms/div
scope.write(":TRIG:EDGE:SOUR CHAN1")    # Trigger on channel 1
scope.write(":TRIG:EDGE:LEV 1.5")      # Trigger at 1.5V

# Measure frequency and amplitude
frequency = float(scope.query(":MEAS:FREQ? CHAN1"))
amplitude = float(scope.query(":MEAS:VAMP? CHAN1"))
```

## Error Handling

Instruments report errors through a queue. Always drain the queue after a command sequence.

```python filename="scpi/error_handling.py"
def check_errors(instr) -> list[str]:
    """Read all errors from the SCPI error queue."""
    errors = []
    while True:
        err = instr.query(":SYST:ERR?").strip()
        code = int(err.split(",")[0])
        if code == 0:
            break
        errors.append(err)
    return errors


# After a sequence of commands
errors = check_errors(dmm)
if errors:
    for e in errors:
        print(f"  Error: {e}")
    raise RuntimeError(f"Instrument errors: {errors}")
```

Common SCPI error codes:

| Code | Meaning | Typical Cause |
|------|---------|---------------|
| -100 | Command error | Typo in command string |
| -200 | Execution error | Invalid parameter value |
| -300 | Device-specific error | Hardware issue |
| -400 | Query error | Query without reading response |
| 0 | No error | Queue empty |

## Triggering Models

SCPI defines several trigger sources. The right choice depends on your timing requirements.

| Trigger Source | Command | Use Case |
|---------------|---------|----------|
| Immediate | `:TRIG:SOUR IMM` | Default. Measures as soon as configured. |
| Bus | `:TRIG:SOUR BUS` | Software trigger with `*TRG`. Precise timing. |
| External | `:TRIG:SOUR EXT` | Hardware trigger line. Synchronized with other instruments. |
| Timer | `:TRIG:SOUR TIM` | Periodic measurements at fixed intervals. |

Bus trigger example (precise timing control):

```python filename="scpi/bus_trigger.py"
dmm.write(":CONF:VOLT:DC 10")       # Configure range
dmm.write(":TRIG:SOUR BUS")         # Bus trigger mode
dmm.write(":INIT")                   # Arm the trigger system
# ... do other setup ...
dmm.write("*TRG")                   # Send the trigger
dmm.query("*OPC?")                  # Wait for completion
reading = float(dmm.query(":FETCH?"))
```

## Vendor-Specific Differences

SCPI is a standard, but vendors add their own extensions. Common differences:

| Feature | Keysight | Rigol | Rohde &amp; Schwarz |
|---------|----------|-------|-----------------|
| Channel select | `:INST:SEL CH1` | `:INST CH1` | `:INST:SEL 1` |
| Screenshot | `:DISP:DATA? PNG` | `:DISP:DATA?` | `:HCOP:DATA?` |
| Error query | `:SYST:ERR?` | `:SYST:ERR?` | `:SYST:ERR:ALL?` |
| Beeper | `:SYST:BEEP` | `:SYST:BEEP:STAT ON` | `:SYST:BEEP:IMM` |

Always check your instrument's programming manual for exact syntax. The core measurement commands (`:MEAS:`, `:CONF:`, `:INIT`, `:FETCH?`) are consistent across vendors.

## Using SCPI in a Production Test

In a production test, the SCPI session belongs in a plug so the connection opens once, is shared across phases, and closes even when a phase fails.

### With the TofuPilot Framework

A plug is a plain Python class. TofuPilot injects it into any phase that names it as a parameter.

```python filename="plugs/dmm.py"
import pyvisa


class Multimeter:
    """SCPI multimeter plug. Opens on first use, closes at teardown."""

    RESOURCE = "TCPIP::192.168.1.100::INSTR"

    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 5000
        self.instr.write("*RST")
        self.instr.write("*CLS")

    def measure_dc_voltage(self, range_v: str = "AUTO") -> float:
        self.instr.write(f":CONF:VOLT:DC {range_v}")
        return float(self.instr.query(":MEAS:VOLT:DC?"))

    def measure_dc_current(self, range_a: str = "AUTO") -> float:
        self.instr.write(f":CONF:CURR:DC {range_a}")
        return float(self.instr.query(":MEAS:CURR:DC?"))

    def measure_resistance(self, range_ohm: str = "AUTO") -> float:
        self.instr.write(f":CONF:RES {range_ohm}")
        return float(self.instr.query(":MEAS:RES?"))

    def close(self):
        self.instr.close()
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    """Measure the 3.3V rail with the SCPI multimeter."""
    measurements.rail_3v3 = dmm.measure_dc_voltage()
```

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

Run it with `tofupilot run ./procedure.yaml`. The limits live in the YAML as data, so the phase stays a plain function that assigns a reading.

### With OpenHTF

OpenHTF wraps the same SCPI session in a `BasePlug` with `setUp` and `tearDown`.

```python filename="scpi/openhtf_integration.py"
import pyvisa
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class ScpiDmm(BasePlug):
    """SCPI multimeter plug with automatic lifecycle."""

    RESOURCE = "TCPIP::192.168.1.100::INSTR"

    def setUp(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource(self.RESOURCE)
        self.instr.timeout = 5000
        self.instr.write("*RST")
        self.instr.write("*CLS")

    def measure_dc_voltage(self, range_v: str = "AUTO") -> float:
        self.instr.write(f":CONF:VOLT:DC {range_v}")
        return float(self.instr.query(":MEAS:VOLT:DC?"))

    def tearDown(self):
        self.instr.close()


@htf.measures(
    htf.Measurement("voltage_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
@htf.plug(dmm=ScpiDmm)
def test_rail(test, dmm):
    test.measurements.voltage_3v3 = dmm.measure_dc_voltage()


def main():
    test = htf.Test(test_rail, procedure_id="FCT-001", part_number="PCBA-100")
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Serial: "))
```

Install the extra with `pip install "tofupilot[openhtf]"`, or skip the callback and run the suite with `tofupilot run`.

One practical note for either framework: a single instrument shared by phases that run in parallel needs care. The TofuPilot Framework can run independent phases concurrently, so either give each phase its own instrument, or serialize the phases that share one with `depends_on`.

## Quick Reference

| Task | SCPI Command |
|------|-------------|
| Identify | `*IDN?` |
| Reset | `*RST` |
| Clear errors | `*CLS` |
| Measure DC voltage | `:MEAS:VOLT:DC?` |
| Measure DC current | `:MEAS:CURR:DC?` |
| Measure resistance | `:MEAS:RES?` |
| Set voltage (PSU) | `:VOLT 5.0` |
| Set current limit (PSU) | `:CURR 0.5` |
| Output on/off (PSU) | `:OUTP ON` / `:OUTP OFF` |
| Read error | `:SYST:ERR?` |
| Wait for complete | `*OPC?` |
| Software trigger | `*TRG` |

### Manage Operator Certification
URL: https://www.tofupilot.com/guides/how-to-manage-operator-certification-with-tofupilot
Track operator training, certification status and test authorization by recording who ran each test and gating on certification.

A new operator runs a test wrong and scraps three boards before anyone notices. It happens more than it should. Most teams track operator certification in spreadsheets that go stale the day they're created.

Tying operator identity to every test run is the part that makes the rest possible: certification gates, yield-by-operator analysis, and an audit trail that survives a regulatory review.

## Why Operator Tracking Matters

Regulated industries require it. ISO 13485 (medical devices), AS9100 (aerospace), and IATF 16949 (automotive) all mandate that operators are trained and qualified for the tasks they perform. But even without regulatory pressure, knowing who ran what test matters when you're debugging a yield drop.

## Prerequisites

- A TofuPilot account
- Python 3.9+ with `pip install "tofupilot[openhtf]"`
- An operator authentication method (badge scan, login, or barcode)

## Step 1: Record the Operator on Every Run

`operated_by` is the field that links a run to a person. An email matching a member of your organization links the run to that account; any other value is recorded verbatim as a declared operator name.

```python filename="operator_tracking.py"
import openhtf as htf
from tofupilot.openhtf import upload

PROCEDURE_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"  # procedure UUID from the dashboard


def main():
    operator_id = input("Scan operator badge: ")

    test = htf.Test(
        functional_tests,
        power_tests,
        procedure_id=PROCEDURE_ID,
        part_number="PCB-100-R4",
        operated_by=operator_id,
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: input("Scan DUT serial: "))


if __name__ == "__main__":
    main()
```

Prefer the operator's work email when they have an account, since that links the run to a real member rather than recording a badge string nobody can resolve later.

## Step 2: Build an Operator Certification Check

Certification data lives in your own system, not in the test platform. A JSON file works for a small team; an HR API or database is the same shape.

```python filename="certification_check.py"
import json
from pathlib import Path

CERT_FILE = Path("operator_certs.json")


def load_certifications() -> dict:
    """Load operator certification records."""
    if CERT_FILE.exists():
        return json.loads(CERT_FILE.read_text())
    return {}


def is_certified(operator_id: str, procedure_name: str) -> bool:
    """Check if operator is certified for a specific test procedure."""
    certs = load_certifications()
    operator = certs.get(operator_id, {})
    return procedure_name in operator.get("certified_procedures", [])


def require_certification(operator_id: str, procedure_name: str) -> None:
    """Block test execution if operator isn't certified."""
    if not is_certified(operator_id, procedure_name):
        raise PermissionError(
            f"Operator {operator_id} is not certified for {procedure_name}. "
            f"Contact your line supervisor."
        )
```

Example certification file:

```json filename="operator_certs.json"
{
  "OP-001": {
    "name": "Jane Chen",
    "certified_procedures": ["pcba-fct-v2", "motor-fct", "burn-in-48h"],
    "certification_date": "2026-01-15",
    "expiry_date": "2027-01-15"
  },
  "OP-002": {
    "name": "Mike Torres",
    "certified_procedures": ["pcba-fct-v2"],
    "certification_date": "2026-02-01",
    "expiry_date": "2027-02-01"
  }
}
```

## Step 3: Gate the Test on Certification

Run the check before the test starts, so an uncertified operator never reaches the first phase.

```python filename="certified_test.py"
import openhtf as htf
from tofupilot.openhtf import upload

from certification_check import require_certification

PROCEDURE_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
PROCEDURE_NAME = "pcba-fct-v2"


def main():
    operator_id = input("Scan operator badge: ")
    require_certification(operator_id, PROCEDURE_NAME)  # raises if not certified

    test = htf.Test(
        functional_tests,
        power_tests,
        procedure_id=PROCEDURE_ID,
        part_number="PCB-100-R4",
        operated_by=operator_id,
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: input("Scan DUT serial: "))


if __name__ == "__main__":
    main()
```

Gating before `htf.Test(...)` means an uncertified attempt produces no run at all. If you would rather record the attempt, move the check into a first phase and return `htf.PhaseResult.STOP` instead.

Worth being clear about what this is: a procedural control, not a security boundary. Anyone who can edit the test script can bypass it. It satisfies the audit requirement that a check exists and is recorded; it does not stop a determined person.

## Step 4: Track Certification Expiry

Certifications expire. Warn before, block after.

```python filename="cert_expiry.py"
from datetime import date

from certification_check import load_certifications


def check_certification_status(operator_id: str, procedure_name: str) -> bool:
    """Check certification validity with advance warning."""
    certs = load_certifications()
    operator = certs.get(operator_id)

    if not operator:
        raise PermissionError(f"Unknown operator: {operator_id}")

    if procedure_name not in operator.get("certified_procedures", []):
        raise PermissionError(
            f"{operator['name']} is not certified for {procedure_name}"
        )

    expiry = date.fromisoformat(operator["expiry_date"])
    today = date.today()

    if today > expiry:
        raise PermissionError(
            f"Certification expired on {expiry}. Recertification required."
        )

    days_remaining = (expiry - today).days
    if days_remaining < 30:
        print(f"WARNING: Certification expires in {days_remaining} days")

    return True
```

## Step 5: Analyze Operator Performance

With an operator on every run, you can answer questions that matter:

- **Yield by operator**: Is one operator consistently lower? They might need retraining.
- **Test duration by operator**: Slower operators may be following procedures more carefully, or struggling with the equipment.
- **Failure modes by operator**: If one operator sees more of a specific failure, check their technique.

The dashboard groups runs by operator directly. To compute it yourself, query one operator at a time using the `operated_by_names` filter and count outcomes:

```python filename="operator_analysis.py"
import os

from tofupilot.v2 import TofuPilot

PROCEDURE_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
OPERATORS = ["OP-001", "OP-002"]


def count_outcomes(client, operator: str) -> dict:
    counts = {"pass": 0, "fail": 0}
    cursor = None
    while True:
        result = client.runs.list(
            procedure_ids=[PROCEDURE_ID],
            operated_by_names=[operator],
            limit=100,
            cursor=cursor,
        )
        for run in result.data:
            counts["pass" if run.outcome == "PASS" else "fail"] += 1

        if not result.meta.has_more:
            break
        cursor = result.meta.next_cursor
    return counts


with TofuPilot(api_key=os.getenv("TOFUPILOT_API_KEY")) as client:
    for operator in OPERATORS:
        stats = count_outcomes(client, operator)
        total = stats["pass"] + stats["fail"]
        if not total:
            print(f"Operator {operator}: no runs")
            continue
        print(f"Operator {operator}: {stats['pass'] / total * 100:.1f}% ({total} runs)")
```

Use `operated_by_ids` instead when your operators are linked organization members rather than declared names.

Two caveats on the number. It counts every run, so a unit that failed and was retested is counted twice; for true first pass yield, take the first run per serial number. And a low figure for one operator is a prompt to look, not a conclusion, since operators rarely test the same mix of products or run the same shifts.

## Regulatory Compliance Notes

| Standard | Requirement | How this maps |
|---|---|---|
| ISO 13485 | Documented training records, competency assessment | Operator on every run, certification check before test |
| AS9100 | Personnel qualified for assigned tasks | Pre-test certification gate |
| IATF 16949 | Training effectiveness evaluated | Yield-by-operator analysis |
| FDA 21 CFR 820 | Personnel training documented | Full audit trail with operator identity |

The certification records themselves stay in your system of record. What the test platform contributes is the immutable link from each run back to the person who ran it.

### Migrate from Custom Test Databases
URL: https://www.tofupilot.com/guides/how-to-migrate-from-custom-test-databases-to-tofupilot
Replace your custom PostgreSQL, SQLite, or Access test database with structured test data collection using OpenHTF and TofuPilot.

Many hardware teams build custom databases for test results. PostgreSQL, SQLite, Microsoft Access. They work well at first. Then someone asks for FPY trends across stations, or you need multi-site access, or an auditor wants full revision history, and the maintenance burden outweighs the original simplicity.

TofuPilot replaces that custom infrastructure with a managed platform that handles storage, analytics, and traceability out of the box.

## Common Custom DB Patterns

Most homegrown test databases follow one of a few schemas. Here's a typical one:

```sql filename="typical_test_schema.sql"
-- The pattern most teams converge on
CREATE TABLE test_runs (
    id SERIAL PRIMARY KEY,
    serial_number VARCHAR(50),
    test_name VARCHAR(100),
    result VARCHAR(10),  -- 'PASS' or 'FAIL'
    operator VARCHAR(50),
    station VARCHAR(50),
    started_at TIMESTAMP,
    duration_seconds FLOAT
);

CREATE TABLE measurements (
    id SERIAL PRIMARY KEY,
    run_id INTEGER REFERENCES test_runs(id),
    name VARCHAR(100),
    value FLOAT,
    unit VARCHAR(20),
    lower_limit FLOAT,
    upper_limit FLOAT,
    passed BOOLEAN
);
```

This structure captures the basics. But it misses things that matter at scale:

- **No built-in analytics.** FPY, Cpk, and control charts require custom queries and visualization code that someone has to maintain.
- **No audit trail.** If someone updates a row, the original value is gone unless you've built trigger-based history tables.
- **No multi-site access.** A local PostgreSQL or SQLite file doesn't serve teams across locations without infrastructure work.
- **Schema drift.** Each test station might insert slightly different data depending on who wrote the INSERT statement.

## TestStand Database Users

If you're using NI TestStand's built-in database logger, you have a variation of this problem. TestStand writes to SQL Server, Oracle, or Access using a fixed schema with `UUT_RESULT`, `STEP_RESULT`, `PROP_RESULT`, and type-specific tables (`PROP_NUMERICLIMIT`, `PROP_NUMERIC`, `PROP_STRINGVALUE`). Querying it requires 5-6 table JOINs, and adding custom metadata means modifying the Process Model's database mapping.

The same migration pattern applies: replace the database INSERT (or TestStand's database logger) with OpenHTF measurements and let TofuPilot handle storage. If you're migrating from TestStand specifically, see the dedicated guide on migrating from NI TestStand to Python.

## Replacing Custom Inserts with OpenHTF

If your current workflow looks like "run test script, INSERT results into database," here's the OpenHTF + TofuPilot equivalent. Instead of managing the database, you define measurements as part of the test and let TofuPilot handle storage.

A typical custom database insert might look like this:

```python filename="old_custom_insert.py"
# What you're replacing: manual DB inserts after each test
import psycopg2

conn = psycopg2.connect("dbname=testdata user=testeng")
cur = conn.cursor()

cur.execute(
    "INSERT INTO test_runs (serial_number, test_name, result, station) "
    "VALUES (%s, %s, %s, %s) RETURNING id",
    ("SN-5001", "thermal_cycle", "PASS", "STATION-3"),
)
run_id = cur.fetchone()[0]

cur.execute(
    "INSERT INTO measurements (run_id, name, value, unit, lower_limit, upper_limit, passed) "
    "VALUES (%s, %s, %s, %s, %s, %s, %s)",
    (run_id, "peak_temp", 84.2, "C", 70.0, 90.0, True),
)

conn.commit()
conn.close()
```

Here's the same test in OpenHTF with TofuPilot:

```python filename="thermal_cycle_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("peak_temp")
        .with_units(units.DEGREE_CELSIUS)
        .in_range(70.0, 90.0),
    htf.Measurement("settling_time")
        .with_units(units.SECOND)
        .in_range(maximum=30.0),
    htf.Measurement("thermal_resistance")
        .in_range(0.5, 2.0),
)
def thermal_cycle(test):
    test.measurements.peak_temp = 84.2
    test.measurements.settling_time = 18.7
    test.measurements.thermal_resistance = 1.1

def main():
    test = htf.Test(thermal_cycle)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-5001")

if __name__ == "__main__":
    main()
```

No database connection code. No SQL. No schema maintenance. Measurements are defined with their limits in the test itself, so every run is validated and structured identically.

## Migrating Historical Data

You have years of test data in your custom database. TofuPilot's REST API lets you import it with full timestamp preservation.

```python filename="import_from_custom_db.py"
import psycopg2
from tofupilot import TofuPilotClient

client = TofuPilotClient()
conn = psycopg2.connect("dbname=testdata user=testeng")
cur = conn.cursor()

# Fetch runs with their measurements
cur.execute("""
    SELECT r.serial_number, r.test_name, r.result, r.started_at, r.duration_seconds,
           m.name, m.value, m.unit, m.lower_limit, m.upper_limit
    FROM test_runs r
    JOIN measurements m ON m.run_id = r.id
    ORDER BY r.id
""")

current_run = None
steps = []

for row in cur:
    serial, test_name, result, started_at, duration, m_name, m_val, m_unit, m_low, m_high = row

    if current_run and current_run != (serial, started_at):
        # Flush previous run
        prev_serial, prev_started = current_run
        client.create_run(
            procedure_id=test_name,
            unit_under_test={"serial_number": prev_serial},
            run_passed=result == "PASS",
            started_at=prev_started,
            duration=duration,
            steps=steps,
        )
        steps = []

    current_run = (serial, started_at)

    # Find or create step
    step = next((s for s in steps if s["name"] == test_name), None)
    if not step:
        step = {"name": test_name, "step_passed": result == "PASS", "measurements": []}
        steps.append(step)

    step["measurements"].append({
        "name": m_name,
        "measured_value": m_val,
        "unit": m_unit,
        "lower_limit": m_low,
        "upper_limit": m_high,
    })

conn.close()
```

Adapt the query to match your schema. The key is mapping your tables to TofuPilot's structure: each row in `test_runs` becomes a run, each row in `measurements` becomes a measurement within a step.

## What Replaces Your Custom Queries

If you've built custom SQL queries or Python scripts for analytics, TofuPilot's dashboard replaces them.

| Your custom query / script | TofuPilot built-in feature |
|---|---|
| `SELECT COUNT(*) ... GROUP BY result` for yield | FPY trends, updated in real time |
| Statistical process control scripts | Cpk and control charts per measurement |
| `WHERE result = 'FAIL' GROUP BY measurement` | Failure Pareto with drill-down |
| `GROUP BY station` for throughput | Station throughput dashboard |
| `WHERE serial_number = 'X'` for traceability | Serial number search with full history |
| Custom reporting scripts | Exportable reports and REST API |

TofuPilot tracks all of this automatically. Open the Analytics tab to see FPY, Cpk, and failure analysis for any procedure.

## Running Both Systems During Transition

You can run your custom database and TofuPilot in parallel to validate the migration:

1. **Add TofuPilot to one test station** while keeping your existing database inserts active. Both systems receive the same results.
2. **Compare data** for a week. Check that measurement values, pass/fail counts, and timestamps match between your database and TofuPilot.
3. **Remove the custom database inserts** from that station once you're confident. Move to the next station.
4. **Keep your old database read-only** as an archive. You can always query it for historical verification.

The custom database doesn't need to go away immediately. It just stops being the system of record once TofuPilot takes over.

### OpenHTF vs NI TestStand for Manufacturing Test
URL: https://www.tofupilot.com/guides/openhtf-vs-ni-teststand-with-tofupilot
Compare OpenHTF and NI TestStand for manufacturing test automation, with feature matrices, code examples, cost analysis, and database integration differences.

OpenHTF is a free, open-source test framework from Google. NI TestStand is a commercial test sequencer from NI (Emerson) at $4,310/seat/year. Both run manufacturing tests with measurements, limits, and sequencing. They differ in cost, platform support, database integration, and how you manage test data. This guide compares them side by side with real code and concrete tradeoffs.

## Feature Comparison

| Feature | OpenHTF | NI TestStand |
|---------|---------|-------------|
| Language | Python | LabVIEW, C, .NET, Python |
| License | Apache 2.0 (free) | $4,310/seat/year |
| Platform | Linux, macOS, Windows | Windows only |
| Test editor | Any code editor | Proprietary Sequence Editor |
| Version control | Git (plain .py files) | Difficult (binary .seq files) |
| Structured measurements | Built-in (name, value, limits, units) | Built-in (Numeric Limit, String Value) |
| Serial number input | Built-in prompt | Built-in (Process Model) |
| Parallel DUT | Limited | Native |
| Instrument drivers | PyVISA, pyserial, nidaqmx | NI VISA, IVI, NI drivers |
| Database logging | TofuPilot (1 line) | Built-in (complex schema) |
| Analytics (FPY, Cpk) | TofuPilot (automatic) | Custom queries or third-party |
| CI/CD integration | Native (Python) | Limited (added 2025 Q2) |
| Report generation | TofuPilot (automatic) | Built-in XML/HTML |
| Community | Small (~640 GitHub stars) | Large (NI forums, training courses) |
| Learning curve | Medium | High |

## Cost Analysis

| Metric | OpenHTF + TofuPilot | NI TestStand |
|--------|-------------------|-------------|
| 5 seats, 1 year | $0 (TofuPilot Lab is free) | $21,550 |
| 20 seats, 1 year | $0 | $86,200 |
| Runtime deployment | Free | Additional runtime licenses |
| Training | Self-taught (openhtf.com docs) | NI courses ($2,000+) |
| Vendor lock-in | None | High (NI ecosystem) |

TestStand also requires Windows, which means Windows licenses for every test station. OpenHTF runs on Linux, which is free and more stable for long-running production stations.

## The Same Test in Both Frameworks

A simple functional test: measure a 3.3V rail, check it's within 3.2V to 3.4V.

### OpenHTF + TofuPilot

```python filename="openhtf_power_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
def test_power_rail(test):
    voltage = 5.02  # Replace with instrument read
    test.measurements.rail_3v3 = voltage


def main():
    test = htf.Test(
        test_power_rail,
        procedure_id="FCT-001",
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial: "))


if __name__ == "__main__":
    main()
```

Measurements are structured data: name, value, limits, units. TofuPilot integration is one `with` statement. The operator gets a serial number prompt automatically.

### NI TestStand

In TestStand, you create a sequence file (.seq) in the Sequence Editor. Add a "Numeric Limit Test" step, set the test expression to your instrument read, configure Low Limit = 3.2 and High Limit = 3.4. Connect the step to a Code Module (LabVIEW VI, DLL, or .NET assembly) that talks to the instrument.

The test logic lives in the Code Module. The sequencing, limits, and reporting live in the .seq file. You can't see both in a single text file, and you can't diff the .seq file in Git.

## Concept Mapping

Every TestStand concept has a direct OpenHTF equivalent.

| TestStand | OpenHTF |
|-----------|---------|
| Sequence file (.seq) | Python test script (.py) |
| Sequence Editor (GUI) | Any code editor (VS Code, PyCharm) |
| Step | Phase function |
| Numeric Limit Test | `htf.Measurement("name").in_range(low, high)` |
| String Value Test | `htf.Measurement("name").equals("expected")` |
| Pass/Fail Test | `htf.Measurement("name").equals(True)` |
| Code Module (DLL, VI) | Plug class |
| FileGlobals / StationGlobals | Plug instance attributes |
| Process Model | TofuPilot integration |
| Setup / Cleanup groups | First/last phase functions, or PhaseGroups |
| UUT Serial Number | `test.execute(test_start=lambda: input("Scan: "))` |

## Instrument Drivers

TestStand uses NI VISA and IVI drivers. OpenHTF uses PyVISA, which talks to the same instruments through the same VISA layer. If your instrument works with TestStand, it works with PyVISA.

| TestStand Driver | Python Equivalent |
|-----------------|-------------------|
| NI VISA / IVI | PyVISA + NI-VISA backend |
| NI DAQmx | nidaqmx (official NI Python package) |
| NI Switch | niswitch (official NI Python package) |
| NI DMM | nidmm (official NI Python package) |
| Serial / UART | pyserial |
| Custom DLL | ctypes or cffi |

NI publishes official Python packages for most of their hardware. You don't lose instrument support by switching to Python.

### OpenHTF Plug for an Instrument

TestStand wraps instruments in Code Modules. OpenHTF wraps them in Plugs, which have automatic lifecycle management (setUp/tearDown).

```python filename="plugs/multimeter.py"
import pyvisa
from openhtf.plugs import BasePlug
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


class MultimeterPlug(BasePlug):
    """Wraps a SCPI multimeter connection."""

    def setUp(self):
        rm = pyvisa.ResourceManager()
        self.instr = rm.open_resource("TCPIP::192.168.1.100::INSTR")
        self.instr.timeout = 5000

    def measure_voltage(self, channel=1):
        self.instr.write(f":CONF:VOLT:DC AUTO,(@{channel})")
        self.instr.write(":INIT")
        return float(self.instr.query(":FETCH?"))

    def tearDown(self):
        self.instr.close()


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
@htf.plug(dmm=MultimeterPlug)
def measure_power_rail(test, dmm):
    test.measurements.rail_3v3 = dmm.measure_voltage(channel=1)


def main():
    test = htf.Test(measure_power_rail, procedure_id="FCT-001", part_number="PCBA-100")
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial: "))


if __name__ == "__main__":
    main()
```

## Sharing Data Between Steps

TestStand uses FileGlobals and StationGlobals to pass data between steps. OpenHTF uses plug instance attributes. The plug persists for the entire test execution.

```python filename="plugs/shared_state.py"
from openhtf.plugs import BasePlug
import openhtf as htf
from openhtf.util import units


class SharedState(BasePlug):
    """Replaces TestStand FileGlobals and StationGlobals."""

    def setUp(self):
        self.cal_offset = 0.0
        self.firmware_version = ""

    def tearDown(self):
        pass


@htf.plug(state=SharedState)
def calibrate(test, state):
    state.cal_offset = 0.023


@htf.measures(
    htf.Measurement("corrected_voltage")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
@htf.plug(state=SharedState)
def measure_corrected(test, state):
    raw = 3.323  # Replace with instrument read
    test.measurements.corrected_voltage = raw - state.cal_offset
```

## Database and Test Data

This is where the two frameworks differ most.

### TestStand Database Logging

TestStand's built-in database logger writes results to SQL Server, Oracle, or Access using a fixed schema. The core tables are `UUT_RESULT`, `STEP_RESULT`, and `PROP_RESULT`, with additional tables for each property type (`PROP_NUMERICLIMIT`, `PROP_STRINGVALUE`, etc.).

Querying this schema requires multi-level JOINs:

| Table | Contains | Joins To |
|-------|----------|----------|
| `UUT_RESULT` | Serial number, overall pass/fail, timestamps | Top-level |
| `STEP_RESULT` | Step name, step status | `UUT_RESULT.ID` |
| `PROP_RESULT` | Property metadata | `STEP_RESULT.ID` |
| `PROP_NUMERICLIMIT` | Numeric limits (low, high) | `PROP_RESULT.ID` |
| `PROP_NUMERIC` | Measured numeric value | `PROP_RESULT.ID` |
| `PROP_STRINGVALUE` | String comparison results | `PROP_RESULT.ID` |

A simple query to get one unit's measurements requires 5-6 table JOINs. Adding custom metadata means modifying the Process Model's database schema mapping, which is fragile and hard to maintain across TestStand versions.

TestStand doesn't include analytics. FPY, Cpk, control charts, and failure Pareto all require custom SQL queries or a third-party tool like WATS.

### OpenHTF + TofuPilot

OpenHTF doesn't have built-in database logging. TofuPilot handles it. You add one `with TofuPilot(test):` line and every run is stored with structured measurements, limits, units, serial numbers, and metadata.

| TestStand Database | TofuPilot |
|---|---|
| 6-table JOIN for one unit's results | Search by serial number, get full history |
| Custom SQL for FPY | FPY trends, updated in real time |
| No Cpk without custom code | Cpk per measurement, automatic |
| No control charts | Control charts with UCL/LCL |
| No failure Pareto | Failure Pareto with drill-down |
| Schema locked to NI's design | Structured data, REST API access |
| SQL Server/Oracle/Access | Cloud or self-hosted |
| One site at a time (unless you build replication) | Multi-site from day one |

No database to provision, no schema to maintain, no SQL to write. TofuPilot tracks everything automatically. Open the Analytics tab to see FPY, Cpk, and failure analysis for any procedure.

## Version Control and Git

TestStand sequence files (.seq) are binary. You can store them in Git, but you can't diff them, review changes in a pull request, or merge branches.

NI added a Git pane in TestStand 2025 Q2. It lets you commit, pull, and push .seq files from within the Sequence Editor. But it's file-level tracking, not content-level diffing. You can see that a file changed, not what changed inside it. NI provides a separate "Diff and Merge Utility" that can compare two .seq files visually, but it doesn't integrate with pull request workflows.

OpenHTF tests are plain Python files. Standard Git workflows apply.

| Capability | TestStand | OpenHTF |
|---|---|---|
| Commit files | Yes (Git pane, 2025 Q2) | Yes (any Git client) |
| Diff changes | File-level only (Diff Utility for visual compare) | Line-by-line (`git diff`) |
| Pull request review | Not possible (binary) | Standard code review |
| Branch per feature | Files track, content doesn't merge | Standard |
| CI/CD linting | Not possible | flake8, mypy, ruff |
| Automated test of tests | Difficult | pytest on test logic |
| Blame history | No | `git blame` |

### What a Python Test Diff Looks Like

When you change a measurement limit in an OpenHTF test, the pull request shows exactly what changed:

```diff filename="test_power_rail.py"
 @htf.measures(
     htf.Measurement("rail_3v3")
-    .in_range(3.2, 3.4)
+    .in_range(3.1, 3.5)
     .with_units(units.VOLT),
 )
```

Reviewers see the old limit, the new limit, and the context. In TestStand, the same change is invisible inside a binary .seq file.

## CI/CD Integration

NI updated TestStand's license agreement in 2025 Q2 to allow CI/CD usage without extra cost (if you have at least one active development license). But running TestStand in CI still requires a Windows runner with TestStand installed, and the sequence files can't be linted or statically analyzed.

OpenHTF tests are Python. They run in any CI system with a Python environment.

| CI/CD Capability | TestStand | OpenHTF |
|---|---|---|
| Run in CI | Yes (2025 Q2, Windows runner required) | Yes (any OS, any CI) |
| License for CI | Included with 1+ dev license (2025 Q2) | Free |
| Static analysis | Not possible (.seq binary) | flake8, mypy, ruff, pylint |
| Unit test on test logic | Difficult | pytest |
| Lint measurement names | Not possible | Custom rules |
| Docker runner | No (Windows required) | Yes |

```yaml filename=".github/workflows/test-lint.yml"
# Lint and type-check OpenHTF test scripts in CI
name: Test Script CI
on: [push, pull_request]
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install ruff mypy openhtf
      - run: ruff check tests/
      - run: mypy tests/
```

You can catch measurement naming issues, import errors, and type problems before test scripts reach the production floor.

## When to Use Each

| Scenario | Better Choice |
|----------|--------------|
| New project, Python team | OpenHTF + TofuPilot |
| Existing NI hardware investment, large enterprise | TestStand (if it works, keep it) |
| Multi-OS test stations (Linux, macOS) | OpenHTF (TestStand is Windows only) |
| Budget-conscious (startup, small team) | OpenHTF + TofuPilot ($0) |
| Need FPY, Cpk, analytics out of the box | OpenHTF + TofuPilot |
| Deep NI PXI/CompactRIO integration | TestStand (tighter NI ecosystem) |
| CI/CD and Git workflows matter | OpenHTF |
| Non-NI instruments (Keysight, Rigol, R&S) | Either (both use VISA) |

## Migration Path

If you're on TestStand and considering a move, the typical migration takes 4-8 weeks per test procedure. Most teams run both systems in parallel during the transition.

1. Set up Python + OpenHTF alongside TestStand on one station.
2. Convert one test procedure (the simplest one). Run both versions on the same DUTs.
3. Validate measurements match between systems.
4. Move to the next procedure once results are confirmed.
5. Decommission TestStand when all procedures are converted.

The biggest risk is rushing. Convert one procedure at a time. Parallel operation is your safety net.

### How to Add Operator Prompts to Hardware Tests
URL: https://www.tofupilot.com/guides/how-to-add-operator-prompts-to-hardware-tests
Operator prompts pause automated tests for manual steps. Learn how to add prompts with text input, choices, and timeouts to OpenHTF tests.

Many manufacturing tests mix automated measurements with manual steps. The operator needs to load a DUT, flip a switch, perform a visual inspection, or enter a value. OpenHTF handles this through operator prompts that pause the test, display a message, and optionally collect input. This guide covers how to add prompts to your tests and display them through the TofuPilot operator UI.

## When to Use Prompts

| Situation | Example |
|-----------|---------|
| DUT loading | "Place the board in the fixture and close the clamp" |
| Visual inspection | "Check for solder bridges on connector J1" |
| Manual measurement | "Read the label and enter the lot code" |
| Physical action | "Press the reset button on the DUT" |
| Go/no-go decision | "Does the LED illuminate green?" |
| Safety check | "Verify the safety cover is closed" |

Prompts keep the operator in the test flow. They see the instruction in the same interface that shows measurements and pass/fail results.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Add a Simple Prompt

The simplest prompt displays a message and waits for the operator to acknowledge it.

```python filename="prompted_test.py"
import openhtf as htf
from openhtf.plugs import user_input


@htf.plug(prompts=user_input.UserInput)
def phase_load_dut(test, prompts):
    """Wait for the operator to load the DUT."""
    prompts.prompt(
        "Place the board in the test fixture. "
        "Close the clamp and press Enter."
    )
```

The test pauses at this phase until the operator responds. In TofuPilot's operator UI, the prompt appears as a card with the message and an acknowledgment button.

## Step 2: Collect Text Input

Add `text_input=True` to collect a value from the operator. The returned string can be stored as a measurement.

```python filename="prompted_test.py"
@htf.plug(prompts=user_input.UserInput)
@htf.measures(
    htf.Measurement("lot_code").with_args(docstring="Operator-entered lot code"),
)
def phase_enter_lot_code(test, prompts):
    """Ask the operator to scan or type the lot code."""
    lot = prompts.prompt(
        "Scan the lot code barcode on the packaging.",
        text_input=True,
    )
    test.measurements.lot_code = lot
```

## Step 3: Use Prompts for Visual Inspection

Combine a prompt with a measurement to record the operator's inspection result.

```python filename="prompted_test.py"
@htf.plug(prompts=user_input.UserInput)
@htf.measures(
    htf.Measurement("visual_result").equals("PASS"),
)
def phase_visual_inspection(test, prompts):
    """Operator checks for cosmetic defects."""
    result = prompts.prompt(
        "Inspect the board under magnification. "
        "Type PASS if no defects found, or FAIL if defects are present.",
        text_input=True,
    )
    test.measurements.visual_result = result.strip().upper()
```

## Step 4: Add Prompts Between Automated Phases

Mix prompted and automated phases in the same test. The operator sees a seamless flow: automated measurements run, then a prompt appears, then more automated measurements.

```python filename="prompted_test.py"
from openhtf.util import units


@htf.measures(
    htf.Measurement("supply_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
)
def phase_power_check(test):
    """Automated: measure supply voltage."""
    test.measurements.supply_voltage_V = 5.01


@htf.plug(prompts=user_input.UserInput)
def phase_flip_board(test, prompts):
    """Ask operator to flip the board for bottom-side testing."""
    prompts.prompt(
        "Flip the board over so the bottom side faces up. "
        "Press Enter when ready."
    )


@htf.measures(
    htf.Measurement("bottom_connector_resistance_mOhm")
    .in_range(maximum=100)
    .with_units(units.OHM),
)
def phase_bottom_test(test):
    """Automated: measure bottom-side connector resistance."""
    test.measurements.bottom_connector_resistance_mOhm = 38.5
```

## Step 5: Connect to TofuPilot

Wire all phases together. Prompted phases render in the TofuPilot operator UI automatically. No extra configuration needed.

```python filename="prompted_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_load_dut,
    phase_enter_lot_code,
    phase_power_check,
    phase_visual_inspection,
    phase_flip_board,
    phase_bottom_test,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Best Practices

| Practice | Why |
|----------|-----|
| Keep prompt text short (1-2 sentences) | Operators scan, they don't read paragraphs |
| Use imperative verbs ("Place", "Press", "Scan") | Clear instructions reduce errors |
| One action per prompt | Don't ask the operator to do three things at once |
| Validate operator input | Use `.equals()` or `.matches_regex()` on the measurement |
| Put prompts at natural breaks | Between automated sequences, not in the middle of a measurement |
| Avoid unnecessary prompts | Every prompt adds cycle time. Automate what you can. |

### Track Firmware Versions in Production
URL: https://www.tofupilot.com/guides/how-to-track-firmware-versions-in-production-with-tofupilot
Record firmware versions as metadata in OpenHTF tests so TofuPilot lets you filter runs, trace failures, and prove compliance by software build.

When a unit fails in the field, the first question is "what firmware was it running?" If you can't answer that from your test records, you're stuck guessing. TofuPilot lets you tag every test run with the firmware version so you can filter, trace, and report on it later.

## Why Firmware Version Tracking Matters

Firmware changes between production batches are common. A bug fix in v2.4.2 might introduce a regression that only shows up under specific conditions. Without version tracking, you can't correlate failure spikes with firmware releases.

Regulatory standards (IEC 62304, FDA 21 CFR Part 820, ISO 13485) require you to document the software configuration of each device at the time of production. For field returns, you need to prove which firmware was loaded when the unit left your factory.

Firmware tracking also helps during EVT/DVT/PVT transitions. You can compare FPY and measurement distributions across firmware versions to verify that a new build doesn't degrade performance.

## Recording Firmware Version as a Measurement

The simplest approach is to record the firmware version as a measurement in your test. This makes it visible in every run record.

```python filename="firmware_metadata_test.py"
import openhtf as htf
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("firmware_version"),
    htf.Measurement("boot_time")
    .in_range(maximum=2000),
    htf.Measurement("self_test_pass")
    .equals(True),
)
def boot_validation(test):
    test.measurements.firmware_version = "3.1.0-rc2"
    test.measurements.boot_time = 843
    test.measurements.self_test_pass = True

@htf.measures(
    htf.Measurement("ble_rssi")
    .in_range(minimum=-70, maximum=-20),
    htf.Measurement("wifi_throughput")
    .in_range(minimum=50.0),
)
def wireless_check(test):
    test.measurements.ble_rssi = -42
    test.measurements.wifi_throughput = 87.3

def main():
    test = htf.Test(
        boot_validation,
        wireless_check,
        station_id="STATION-FT-04",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "HUB-2026-03-00891")

if __name__ == "__main__":
    main()
```

Every run in TofuPilot will carry the firmware version as a searchable field. You can filter your run history by this value directly from the dashboard.

## Reading Firmware Version from the DUT

In production, you shouldn't hardcode the firmware version. Read it from the device during the test so the record always reflects what's actually on the unit.

```python filename="firmware_read_test.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from tofupilot.openhtf import upload

class DeviceConnection(BasePlug):
    """Connects to the DUT over serial to read device info."""

    def read_firmware_version(self):
        # Replace with your actual device communication
        # e.g., send AT command, read UART, query USB descriptor
        return "3.1.0-rc2"

    def read_bootloader_version(self):
        return "1.2.0"

    def tearDown(self):
        pass

@htf.measures(
    htf.Measurement("firmware_version"),
    htf.Measurement("bootloader_version"),
)
@htf.plug(device=DeviceConnection)
def read_device_info(test, device):
    fw = device.read_firmware_version()
    bl = device.read_bootloader_version()
    test.measurements.firmware_version = fw
    test.measurements.bootloader_version = bl

@htf.measures(
    htf.Measurement("adc_accuracy_percent")
    .in_range(maximum=0.5),
    htf.Measurement("watchdog_recovery_pass")
    .equals(True),
)
def functional_test(test):
    test.measurements.adc_accuracy_percent = 0.12
    test.measurements.watchdog_recovery_pass = True

def main():
    test = htf.Test(
        read_device_info,
        functional_test,
        station_id="STATION-FT-01",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SENS-2026-04217")

if __name__ == "__main__":
    main()
```

This approach records the firmware version as a measurement (visible in the run detail) and ensures TofuPilot captures exactly what was on the device, not what you expected to be on it.

## Filtering Runs by Firmware Version

Once your runs carry firmware version data, TofuPilot's dashboard lets you filter by it. This is where the tracking pays off.

**Failure investigation.** Filter runs to a specific firmware version and check FPY. If v3.1.0-rc2 has a lower yield than v3.0.9, you've found your suspect.

**Regulatory evidence.** During an audit, filter by firmware version to show that all units shipped with the validated release. Export the filtered view as evidence.

**Field return correlation.** When a customer reports a failure, look up the unit's serial number, find the firmware version from the test record, and check if other units with the same firmware show similar behavior.

## What to Track Beyond Firmware

Firmware version is the most critical, but consider tracking other software and configuration data that affects test outcomes.

| Field | Example Value | Why It Matters |
|---|---|---|
| `firmware_version` | `3.1.0-rc2` | Core software on the DUT |
| `bootloader_version` | `1.2.0` | Can affect boot behavior and update paths |
| `hardware_revision` | `rev-C` | PCB changes affect test results |
| `config_profile` | `US-120V` | Regional or customer-specific settings |
| `test_script_version` | `2.0.4` | Proves which test logic was used |

Record these as measurements in your test phases so they're stored with every run in TofuPilot.

### Use pytest for Hardware Testing
URL: https://www.tofupilot.com/guides/how-to-use-pytest-for-hardware-testing-with-tofupilot
Use pytest for hardware testing: where fixtures fit, what pytest does not give you, and how to get structured results into a dashboard.

pytest is the first thing most Python engineers reach for, and it works for hardware within limits that are worth understanding before you build a production line on it. This guide covers the patterns that work, the gaps that matter, and how to get structured results out.

## Where pytest Fits

pytest was built to test software. Applied to hardware it is excellent for some jobs and structurally wrong for others.

| Use case | pytest | Why |
|----------|--------|-----|
| Firmware CI on a HIL bench | Good | Runs in any pipeline, parallel via xdist, familiar |
| EVT/DVT bring-up and validation | Good | Flexible, exploratory, easy to select and skip |
| Driver and instrument-library unit tests | Good | This is exactly what it is for |
| Characterisation sweeps | Workable | Parametrize is a natural fit |
| Production functional test | Poor | No structured measurements, no operator UI, no traceability |
| Operator-driven station test | Poor | No prompts, no serial scan, no kiosk |

The dividing line is whether a human operator and a serial number are involved. If the results feed a developer, pytest is a good answer. If they feed a production record, it is missing the parts that matter.

## The Patterns That Work

### Fixtures for instrument lifecycle

Session-scoped fixtures are genuinely the right abstraction for an instrument, and one of pytest's real strengths here:

```python filename="conftest.py"
import pytest
import pyvisa


@pytest.fixture(scope="session")
def dmm():
    """One VISA session for the whole test session."""
    rm = pyvisa.ResourceManager("@py")
    instr = rm.open_resource("TCPIP::192.168.1.100::INSTR")
    instr.timeout = 5000
    instr.write("*RST")
    yield instr
    instr.close()


@pytest.fixture(scope="session")
def psu():
    rm = pyvisa.ResourceManager("@py")
    instr = rm.open_resource("TCPIP::192.168.1.101::INSTR")
    instr.timeout = 5000
    yield instr
    for ch in (1, 2, 3):
        instr.write(f":OUTP CH{ch},OFF")
    instr.close()
```

The teardown after `yield` runs even when a test fails, which is what you want for de-energising a board.

### Parametrize for sweeps

```python filename="test_sweep.py"
import pytest


@pytest.mark.parametrize("voltage", [3.0, 3.3, 3.6])
def test_operates_across_supply_range(dut, psu, voltage):
    psu.write(f":APPL CH1,{voltage},0.5")
    assert dut.self_test_passes(), f"Self-test failed at {voltage} V"
```

### Markers for hardware requirements

```python filename="test_rf.py"
import pytest


@pytest.mark.rf
@pytest.mark.slow
def test_transmit_power(spectrum_analyser):
    power = spectrum_analyser.measure_channel_power()
    assert 8.0 <= power <= 14.0
```

```bash filename="run_subset.sh"
pytest -m "not slow"          # Skip the long RF tests
pytest -m rf                  # Only the RF bench tests
```

## The Gap: Measurements Disappear

This is the structural problem, and it is not a style preference.

```python filename="test_power.py"
def test_rail_3v3(dmm):
    voltage = float(dmm.query(":MEAS:VOLT:DC?"))
    assert 3.2 <= voltage <= 3.4, f"Rail out of range: {voltage} V"
```

That test is correct and tells you nothing tomorrow. The value existed inside the assert and was discarded. You cannot compute Cpk, cannot see that the rail drifted from 3.31 to 3.38 over three weeks, and cannot justify a limit change. The limits exist only as a comparison expression, so nothing can report them or tell you they changed.

JUnit XML does not fix this. It records which test passed, not what was measured.

## Getting Structured Results Out

If pytest is the right tool for your stage, push the measurements somewhere that keeps them. The Python SDK takes a run with steps and measurements:

```python filename="conftest.py"
import pytest
from tofupilot import TofuPilotClient


@pytest.fixture(scope="session")
def recorder():
    """Collect measurements during the session, upload one run at the end."""
    steps = []
    yield steps

    client = TofuPilotClient()
    client.create_run(
        procedure_id="FCT-001",
        unit_under_test={"serial_number": "SN-001", "part_number": "PCBA-100"},
        run_passed=all(s["step_passed"] for s in steps),
        steps=steps,
    )


def record(steps, name, value, unit=None, lower=None, upper=None):
    passed = True
    if lower is not None and value < lower:
        passed = False
    if upper is not None and value > upper:
        passed = False
    steps.append({
        "name": name,
        "step_passed": passed,
        "measurements": [{
            "name": name,
            "value": value,
            "unit": unit,
            "lower_limit": lower,
            "upper_limit": upper,
        }],
    })
    return passed
```

```python filename="test_power.py"
def test_rail_3v3(dmm, recorder):
    voltage = float(dmm.query(":MEAS:VOLT:DC?"))
    assert record(recorder, "rail_3v3", voltage, "V", 3.2, 3.4)
```

The value is now kept with its limits, so yield, Cpk and drift all become possible. The cost is the helper above and the discipline to use it in every test, which is the part that erodes as a suite grows.

## When to Stop Using pytest

The signals are consistent across teams:

- An operator needs a screen, and you are looking at building one.
- Someone asks for yield per measurement over the last month.
- The line needs four units tested at once on one fixture.
- A station loses network and a shift of results is gone.
- Limits changed and nobody can say which results used which limits.

At that point the missing pieces are not pytest's job, and building them is a project.

## The Production Alternative

A purpose-built hardware framework keeps the same Python you already wrote and adds the parts pytest does not have. The instrument fixture becomes a plug, the test function becomes a phase, and the assert becomes declared limits:

```python filename="plugs/dmm.py"
import pyvisa


class Multimeter:
    def __init__(self):
        rm = pyvisa.ResourceManager("@py")
        self.instr = rm.open_resource("TCPIP::192.168.1.100::INSTR")
        self.instr.timeout = 5000
        self.instr.write("*RST")

    def dc_voltage(self) -> float:
        return float(self.instr.query(":MEAS:VOLT:DC?"))

    def close(self):
        self.instr.close()
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.dc_voltage()
```

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

Run it with `tofupilot run ./procedure.yaml`. The mapping from pytest is direct: session fixture to plug, test function to phase, assert to validators, `conftest.py` to the `plugs/` directory.

## Using Both

The common arrangement is pytest for firmware CI and bring-up, a hardware framework for production, both uploading to the same dashboard. That is a sensible split rather than a compromise: the two stages genuinely have different requirements.

| Stage | Tool |
|-------|------|
| Firmware CI, HIL regression | pytest |
| EVT/DVT validation | pytest |
| PVT and production FCT | TofuPilot Framework or OpenHTF |
| Operator-driven station test | TofuPilot Framework or OpenHTF |

See [OpenHTF vs pytest](/guides/openhtf-vs-pytest-for-hardware-testing-which-should-you-choose) for the framework comparison.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Limits only inside asserts | No Cpk, no drift, no limit history | Record values with limits |
| Function-scoped instrument fixtures | Session reopened per test, slow and leaky | `scope="session"` |
| Relying on JUnit XML for production records | Verdicts without measurements | Upload structured runs |
| Using pytest for operator-driven test | No prompts, no scan, no kiosk | Use a hardware framework |
| xdist against one shared instrument | Interleaved commands, wrong readings | One instrument per worker, or serialise |
| Test ordering carrying state | Passes alone, fails in a full run | Keep tests independent, or use a real sequencer |

### Anomaly Detection in Hardware Test Data
URL: https://www.tofupilot.com/guides/anomaly-detection-in-hardware-test-data
Learn how to detect anomalies in hardware test measurements using TofuPilot's analytics and automated limit checking.

A unit passes all its limits but something looks wrong. The voltage is 2% higher than usual. The calibration took twice as long. These subtle anomalies are invisible in pass/fail reports but obvious in measurement trends. TofuPilot surfaces them automatically.

## Why Pass/Fail Isn't Enough

Spec limits define the acceptable range. But "within spec" doesn't mean "normal." A power supply output that's been stable at 3.30V for 10,000 units and suddenly reads 3.34V is still within the 3.25V-3.35V spec, but it's anomalous. Something changed.

Catching these anomalies early prevents escapes. The unit that reads 3.34V today might read 3.36V tomorrow, after it's shipped.

## Types of Hardware Test Anomalies

| Anomaly Type | Example | Risk |
|-------------|---------|------|
| Drift | Measurement gradually shifting toward a limit | Eventual field failure |
| Step change | Measurement jumping to a new baseline | Process or tooling change |
| Increased variance | Measurement scatter widening | Loss of process control |
| Outlier | Single unit far from the distribution | Component defect |
| Bimodal distribution | Two clusters instead of one | Mixed component lots |

## Setting Up Anomaly Detection in TofuPilot

### Step 1: Define Measurements with Limits

Every measurement you upload should include limits. TofuPilot uses these for pass/fail gating, but they also define the baseline for anomaly detection.

```python filename="measurements_with_limits.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

client.create_run(
    procedure_id="SENSOR-CALIBRATION",
    unit_under_test={"serial_number": "SENS-7821"},
    steps=[{
        "name": "Zero Offset",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "zero_offset_mv",
            "value": 0.12,
            "unit": "mV",
            "limit_low": -1.0,
            "limit_high": 1.0,
        }],
    }, {
        "name": "Sensitivity",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "sensitivity_mv_per_g",
            "value": 100.3,
            "unit": "mV/g",
            "limit_low": 95.0,
            "limit_high": 105.0,
        }],
    }],
)
```

### Step 2: Monitor Measurement Distributions

Open the measurement view for any procedure in TofuPilot. The histogram shows the distribution of values across all runs.

A healthy process looks like a tight normal distribution centered well within the spec limits. Warning signs:

- **Distribution shifting**: The center of the histogram is moving toward one limit
- **Distribution widening**: The tails are getting closer to the limits
- **Multiple peaks**: Two or more clusters in the histogram

### Step 3: Set Warning Thresholds

Don't wait for a measurement to hit its spec limit. Set warning thresholds at a percentage of the spec range.

| Threshold | Level | Action |
|-----------|-------|--------|
| Within 70% of spec range | Normal | No action |
| 70-90% of spec range | Warning | Investigate trend |
| 90-100% of spec range | Critical | Stop and root-cause |
| Beyond spec range | Fail | Unit rejected |

### Step 4: Track Trends Over Time

The measurement trend view in TofuPilot shows each reading plotted over time. Use this to spot:

- **Slow drift**: A linear trend heading toward a limit. Often caused by fixture wear, calibration drift, or environmental changes.
- **Step changes**: An abrupt shift in the baseline. Often caused by a new component lot, process change, or equipment swap.
- **Periodic patterns**: Readings that cycle with time of day, day of week, or production batch. Often caused by environmental factors (temperature, humidity) or shift-dependent operator procedures.

## Automated Data Checks

TofuPilot checks every incoming measurement against its defined limits in real time. Failed measurements are flagged immediately, and the run is marked as failed.

For anomaly detection beyond simple limit checks, export measurement data via the API and apply statistical methods:

```python filename="cpk_analysis.py"
import numpy as np

# Fetch measurement values from TofuPilot API
values = [3.30, 3.31, 3.29, 3.30, 3.32, 3.31, 3.30, 3.34, 3.29, 3.31]
usl = 3.35  # Upper spec limit
lsl = 3.25  # Lower spec limit

mean = np.mean(values)
std = np.std(values, ddof=1)

cpk = min((usl - mean) / (3 * std), (mean - lsl) / (3 * std))
print(f"Cpk: {cpk:.2f}")

if cpk < 1.33:
    print("Process capability below target. Investigate.")
```

## Real-World Example: Detecting a Supplier Quality Escape

A robotics company tests motor controllers on their production line. Every unit measures motor current draw at three load points. For 6 months, the 50% load measurement averaged 2.1A with a standard deviation of 0.05A.

In week 27, the average shifted to 2.25A. Still within the 1.5A-3.0A spec limit, so no units failed. But TofuPilot's trend view showed the step change clearly.

Investigation revealed a new motor driver IC lot with slightly different gate threshold voltages. The company worked with their supplier to tighten the incoming spec before the drift could cause field issues.

Without measurement trending, this would have been invisible until units started failing in the field.


### Code Review for LabVIEW Tests: Why It Fails
URL: https://www.tofupilot.com/guides/code-review-for-labview-tests-why-it-fails
Learn why a VI change can't be reviewed like text, what teams do instead and what each misses, and a checklist that gets most of a review on LabVIEW anyway.

Code review needs a diff, and a `.vi` has none outside LVCompare on a licensed machine with the same LabVIEW version installed. So on most production floors the review of a test change is "the integrator sent the new build", and the first person to see the change is the operator. This guide covers what a review is for on a test station, why it doesn't happen with VIs, and a checklist that gets you most of it anyway.

## What a Review Is For on a Test Station

Three kinds of change reach a station, and each one deserves a second pair of eyes before it runs on product.

A limit changed. The 3.3 V rail lower limit moves from 3.2 V to 3.15 V. The reviewer asks who decided, whether there's a deviation or a design change behind it, and what happens to units that measured 3.17 V last month.

A phase was added. A new leakage measurement goes in after Power Rails. The reviewer checks that it has limits, that it doesn't add twenty seconds to a cycle that was already the bottleneck, and that it doesn't need a fixture change nobody planned.

A fixture workaround went in. A retry loop around a flaky relay, a 500 ms delay before a SCPI query, a pin skipped because pogo 14 is worn. These are the changes that become permanent, and the review is the only moment anyone asks whether they should.

None of this is about style. On a station the review is about limits and outcomes, because those decide what ships.

## Why It Doesn't Happen With VIs

A review is a sequence of small steps, and every one of them assumes text.

| Review step | What it needs | What a `.vi` gives you |
|---|---|---|
| See what changed | A line diff | "Binary files differ" |
| See who changed which part | Blame per line | One author for the whole file |
| Read it without tooling | A browser and a pull request page | LabVIEW Professional with LVCompare, the same version, every dependency resolving |
| Comment on a specific change | A line anchor | A screenshot with an arrow drawn on it |
| Merge two reviewed changes | A three-way text merge | LVMerge, Professional only, with known cases of duplicated nodes |
| Connect the change to results | A version stamped on each run | Nothing, unless the build writes one |

LVCompare is a real tool and it's good at its job. It loads two VIs and highlights the nodes that were added, removed, moved or rewired.

But it ships only with the Professional Development System, at roughly $5,500 per seat per year since the move to subscription, and it needs both VIs loaded, so every subVI, class and library has to resolve on the reviewer's machine. What it can and can't do is in [LVCompare and LVMerge: what they cannot do](/guides/lvcompare-and-lvmerge-what-they-cannot-do).

The practical result is that the review moves to whoever has that machine. On most floors that's the person who made the change, which isn't a review.

## What Teams Do Instead

Three workarounds are common. Each one gets part of the job done and misses a specific thing.

Screenshots in a ticket. The author pastes the new block diagram, sometimes with the old one next to it. The reviewer sees what the author chose to show, and misses a changed constant on a subVI, a rewired case structure two frames over, or a typedef that shifted every caller.

A second engineer opens the VI. This needs a seat, the same version and someone who reads LabVIEW, which on many teams is a set of one.

The second engineer sees the new state without the old one and reviews from memory. When the team has one LabVIEW engineer, [bus factor in test engineering](/guides/bus-factor-in-test-engineering-a-guide) explains why this reviewer usually doesn't exist.

LVCompare on the reviewer's PC. This is the closest thing to a real review. It needs the Professional seat and the same version, and files sometimes have to be renamed to open two copies, which breaks classes and their members.

Nothing is recorded: no comment thread, no approval, no link from the change to the build that carried it. And when two people changed the same VI, the merge is a separate problem, which [fix LabVIEW merge conflicts on binary VIs](/guides/fix-labview-merge-conflicts-on-binary-vis) covers.

## A Review Checklist That Works on a LabVIEW Station Anyway

You can't diff the VI, so move the things that need review out of it.

1. Put limits in a config file the VI reads at startup. A CSV or INI next to the executable turns a limit change into a one-line text diff, reviewable in any Git host, even though the VI that reads it stays binary.
2. Write one CHANGELOG line per build: build number, date, author, what changed, why, and the ticket. The reviewer reads the line before the build reaches the floor.
3. Show the build number and LabVIEW version in the About box, and have the build write the same number to a `version.txt` next to the executable.
4. Turn on separate compiled code for the project, so a "modified" VI in Git means its source changed and not that a callee recompiled it.
5. Run the new build on one golden unit before it goes to the floor, and attach that run to the CHANGELOG line.

The limits file is the piece that pays back first.

```csv filename="limits.csv"
measurement,unit,min,max
rail_3v3,V,3.15,3.4
rail_5v,V,4.85,5.15
leakage_ua,uA,,50
```

When the 3V3 lower limit moves, the pull request on this file shows `3.2` becoming `3.15`, the author, the date and a place to ask why. That's most of a review, and it needed no LabVIEW licence. Items 3 and 5 together are what let you answer later which units ran which limit, which [trace a test change to the units it shipped](/guides/trace-a-test-change-to-the-units-it-shipped) goes into.

Most floors already have a review process. It's the operator, and the outcome is whether the line stops.

## The Contrast: A Pull Request on a Limit Change

On a Python station under the TofuPilot Framework, the limit lives in `procedure.yaml`, and the same change is a pull request that any Git host renders.

```diff filename="procedure.yaml"
 main:
   - name: Power Rails
     python: phases.power_rails
     measurements:
       - name: rail_3v3
         unit: V
         validators:
           - operator: ">="
-            expected_value: 3.2
+            expected_value: 3.15
           - operator: "<="
             expected_value: 3.4
```

A reviewer answers three questions from the diff alone, without opening anything else. What changed: the lower limit on `rail_3v3`, from 3.2 V to 3.15 V, and nothing else in the file.

Why: the pull request description, which links the deviation or the design note. What it does to units already tested: any unit that measured between 3.15 V and 3.2 V failed before and passes now, and the reviewer asks for that count from the run history in TofuPilot before approving.

Six months later, when someone asks who lowered that limit, the answer is one command.

```bash filename="blame.sh"
# Who changed the rail_3v3 lower limit, and in which commit.
git blame -L '/rail_3v3/,+6' procedure.yaml
# 7c2e91a4 (Priya Nair 2026-03-11 14:02:33 +0100 24)             expected_value: 3.15
```

The merge creates a deployment, the station pulls it, and every run uploaded after that carries the deployment id. The reviewer who approved the change opens the procedure in TofuPilot a week later and compares FPY before and after that deployment, which closes the loop the screenshot in the ticket never could.

## Start With One Station

Pick the station whose limits change most often, rebuild it in Python with the TofuPilot Framework with the same limits, and run it next to the LabVIEW version on the same units for two weeks. The first limit change after that is your first real review.

```bash filename="install-and-run.sh"
# Install the CLI, then run the station locally (no account needed) or with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Migrate from LabVIEW TestExec to Python
URL: https://www.tofupilot.com/guides/how-to-migrate-from-labview-testexec-to-python-with-tofupilot
A practical migration guide from LabVIEW TestExec to Python-based test automation with OpenHTF and TofuPilot, covering architecture mapping and code examples.

LabVIEW TestExec served its purpose. But the licensing costs keep climbing, finding LabVIEW developers is getting harder, and your CI/CD pipeline doesn't speak G code. Python gives you the language, OpenHTF gives you the test framework, and TofuPilot gives you the data platform.

Here's how to make the switch without losing your test coverage.

## Why Teams Migrate

| Pain Point | LabVIEW TestExec | Python + TofuPilot |
|---|---|---|
| Licensing | Per-seat, annual renewal | Free and open source |
| Developer pool | Shrinking, specialized | Large, growing |
| Version control | Binary VIs, merge conflicts | Text files, standard Git |
| CI/CD integration | Custom adapters needed | Native pytest/OpenHTF support |
| Instrument drivers | NI ecosystem only | PyVISA, any SCPI instrument |
| Data storage | Local TDM/TDMS files | Cloud database with API |

## Prerequisites

- Python 3.8+ installed
- `pip install openhtf tofupilot pyvisa`
- Access to your existing LabVIEW test sequences and specs
- Instrument documentation (SCPI command references)

## Step 1: Map Your TestExec Architecture

LabVIEW TestExec and OpenHTF share similar concepts with different names:

| LabVIEW TestExec | OpenHTF + TofuPilot | Notes |
|---|---|---|
| Test Sequence | `htf.Test()` | Top-level test container |
| Test Step | Phase function | Decorated Python function |
| Limit | `htf.Measurement` validator | `.in_range()`, `.at_most()`, etc. |
| Step Result | Measurement value | Stored with pass/fail status |
| Sequence File (.seq) | Python script (.py) | Version-controlled text |
| Operator Interface | TofuPilot station UI | Web-based, no LabVIEW runtime |
| Report | TofuPilot dashboard | Automatic, real-time |
| Station Global | Module-level config or env var | Standard Python patterns |

## Step 2: Convert Test Steps to Python Phases

A typical LabVIEW TestExec step that reads a voltage becomes a Python phase:

**Before (LabVIEW TestExec pseudo-code):**
```
Step: "Read 5V Rail"
  Type: Numeric Limit Test
  Instrument: DMM_1 (NI PXI-4072)
  Command: Measure DC Voltage
  Low Limit: 4.95
  High Limit: 5.05
  Units: V
```

**After (Python with OpenHTF):**

```python filename="power_test.py"
import openhtf as htf
from openhtf.util import units
import pyvisa

@htf.measures(
    htf.Measurement("voltage_5v_rail")
        .with_units(units.VOLT)
        .in_range(4.95, 5.05)
        .doc("5V rail output voltage"),
)
def read_5v_rail(test, dmm):
    """Read and validate 5V rail voltage."""
    voltage = float(dmm.query("MEAS:VOLT:DC?"))
    test.measurements.voltage_5v_rail = voltage
```

The pattern is always the same:
1. Define measurements with limits (replaces the Limit columns)
2. Write instrument control in the phase body (replaces the VI)
3. Assign measured values (replaces the step result)

## Step 3: Handle Instrument Connections

LabVIEW TestExec uses NI's instrument driver architecture. Python uses PyVISA, which works with NI, Keysight, Rigol, and any SCPI-compatible instrument.

```python filename="instruments.py"
import pyvisa

def connect_instruments():
    """Replace TestExec station globals with PyVISA connections."""
    rm = pyvisa.ResourceManager()

    instruments = {
        "dmm": rm.open_resource("TCPIP::192.168.1.10::INSTR"),
        "psu": rm.open_resource("TCPIP::192.168.1.11::INSTR"),
        "scope": rm.open_resource("USB0::0x0957::0x1796::INSTR"),
    }

    # Configure instruments
    instruments["dmm"].timeout = 5000
    instruments["psu"].write("*RST")

    return instruments
```

If you're using NI PXI instruments, the NI-VISA backend works with PyVISA. Your GPIB, USB, and TCP/IP instruments work without any NI software.

## Step 4: Convert Sequence Flow Control

LabVIEW TestExec has built-in flow control (preconditions, post-actions, branching). In OpenHTF, you use standard Python:

**Conditional execution:**

```python filename="conditional_test.py"
import openhtf as htf

@htf.measures(
    htf.Measurement("board_variant")
        .with_allowed_values("A", "B", "C"),
)
def detect_variant(test):
    """Read board variant from EEPROM or resistor divider."""
    variant = read_board_variant()
    test.measurements.board_variant = variant
    test.state["variant"] = variant

@htf.measures(
    htf.Measurement("bluetooth_rssi")
        .with_units(units.DECIBEL)
        .at_least(-70),
)
def bluetooth_test(test):
    """Only runs on variant B (Bluetooth-equipped boards)."""
    if test.state.get("variant") != "B":
        return  # Skip for non-BT variants

    rssi = scan_bluetooth()
    test.measurements.bluetooth_rssi = rssi
```

**Setup and teardown:**

```python filename="setup_teardown.py"
import openhtf as htf

@htf.PhaseOptions(timeout_s=10)
def setup_fixture(test, fixture_controller):
    """Replaces TestExec Setup step group."""
    fixture_controller.clamp()
    fixture_controller.connect_probes()

@htf.PhaseOptions(run_if=lambda: True)  # Always runs
def teardown_fixture(test, fixture_controller):
    """Replaces TestExec Cleanup step group. Runs even on failure."""
    fixture_controller.release()
    fixture_controller.disconnect_probes()
```

## Step 5: Set Up Data Collection

LabVIEW TestExec stores results in local files (TDM, TDMS, XML). TofuPilot stores everything in the cloud with full traceability.

```python filename="main_test.py"
import openhtf as htf
from tofupilot import TofuPilotClient

def main():
    instruments = connect_instruments()

    test = htf.Test(
        setup_fixture,
        detect_variant,
        read_5v_rail,
        bluetooth_test,
        teardown_fixture,
        phase_kwargs={
            "dmm": instruments["dmm"],
            "psu": instruments["psu"],
        },
    )

    test.add_output_callbacks(
        TofuPilotClient().as_openhtf_callback(
            procedure_id="pcba-fct-v3",
            procedure_name="PCBA FCT (migrated from TestExec)",
        )
    )

    test.execute(test_start=htf.PhaseDescriptor.wrap(
        lambda test: setattr(test, "dut_id", input("Scan serial: "))
    ))

if __name__ == "__main__":
    main()
```

## Migration Strategy

Don't rewrite everything at once. Migrate station by station:

1. **Pick one test station** with the simplest sequence
2. **Document the existing test spec** (steps, limits, instruments, flow)
3. **Write the Python equivalent** following the patterns above
4. **Run both systems in parallel** for one production batch
5. **Compare results** to verify equivalence
6. **Cut over** when results match
7. **Move to the next station**

Keep your LabVIEW TestExec sequences as reference documentation. The Python scripts replace them, but having the original spec helps during validation.

## Common Gotchas

| Issue | Solution |
|---|---|
| NI drivers need LabVIEW runtime | PyVISA with NI-VISA backend works without LabVIEW |
| TestExec parallel step groups | Use Python `threading` or `asyncio` |
| TestExec callbacks (on fail, on pass) | OpenHTF phase options and output callbacks |
| Station model/serial tracking | TofuPilot station configuration properties |
| TestExec report generation | TofuPilot generates reports automatically |
| Operator interface buttons | TofuPilot's web-based station UI or custom Tkinter |

The hardest part isn't the code. It's validating that the new tests catch the same failures as the old ones. Run both systems in parallel until you're confident.


### The Hidden Cost of Retesting in Manufacturing
URL: https://www.tofupilot.com/guides/the-hidden-cost-of-retesting-how-to-measure-it-with-tofupilot
Learn how retesting inflates costs in manufacturing, identify root causes, and use TofuPilot's unit history to track and reduce retest rates.

Every retest burns station time, operator attention, and margin. Most teams don't track it because retesting feels like part of the process. It's not. It's rework that compounds silently.

This guide breaks down what retesting actually costs, where it comes from, and how to use TofuPilot's built-in unit history to track and reduce it.

## What Retesting Actually Costs

A single retest doesn't look expensive. Multiply it by thousands of units per month and the numbers get uncomfortable.

| Cost category | Per-retest estimate | At 10% retest rate (10k units/mo) |
|---|---|---|
| Station time (test duration) | $0.50-$2.00 | $500-$2,000/mo |
| Operator handling | $0.30-$1.00 | $300-$1,000/mo |
| Failure analysis (if triggered) | $5-$50 | $5,000-$50,000/mo |
| Delayed shipment (opportunity cost) | Varies | Often the largest cost |
| Station capacity lost | 1 slot per retest | 1,000 slots/mo unavailable |

The direct cost of retesting is easy to underestimate. The indirect cost (capacity you can't use for new units) is usually worse. A test station running at 10% retest rate effectively loses 10% of its throughput.

## Common Causes of Excessive Retesting

Retesting isn't always a product quality problem. Often it's a test system problem.

| Root cause | Symptom | Fix |
|---|---|---|
| Flaky test infrastructure | Same unit passes on retry with no rework | Stabilize fixtures, tighten connections, add settling time |
| Overly tight limits | Marginal units fail, pass on retest | Widen limits using production Cpk data |
| Environmental sensitivity | Failures cluster at shift start or temp changes | Add environmental conditioning or guard bands |
| Operator error | Failures correlate with specific operators | Improve fixturing, reduce manual steps |
| Intermittent DUT defect | Unit fails randomly across multiple retests | Root cause at board level, check solder joints |
| Test software bugs | Specific phases fail inconsistently | Review phase timeout, instrument communication |

If your retest rate is above 5%, start with the test system before blaming the product.

## Setting Up Tests for Retest Tracking

TofuPilot tracks retests by matching runs to the same unit serial number and procedure. For this to work, your test script needs to identify units correctly.

```python filename="tests/board_fct.py"
import openhtf as htf
from tofupilot.openhtf import upload

def main():
    test = htf.Test(
        check_power_rails,
        check_communication,
        measure_current_draw,
        station_id="FCT-STATION-01",
    )

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-2024-00421")

if __name__ == "__main__":
    main()
```

Two things matter for retest tracking:

- **`procedure_id`** identifies the test procedure. Use a stable name that doesn't change between retests.
- **`dut_id`** (returned by `test_start`) identifies the physical unit. Use the actual serial number, not a generated ID.

When the same `dut_id` appears multiple times for the same `procedure_id`, TofuPilot knows it's a retest.

## Tracking Retests in TofuPilot

TofuPilot tracks every run per unit serial number. You don't need to build analytics scripts or query the API to compute retest rates.

**Unit history page.** Open any unit's history page to see all test attempts, ordered chronologically. A unit that was tested three times before passing shows all three runs with their outcomes. This tells you immediately whether a unit needed retesting and how many attempts it took.

**Analytics tab.** Use the Analytics tab to monitor first-pass yield (FPY) trends. FPY inversely correlates with retest rate: if your FPY is 92%, roughly 8% of units needed at least one retest. Tracking FPY over time shows whether your retest problem is improving or getting worse.

**Filtering by procedure and station.** You can filter analytics by procedure and station to isolate retest patterns. If one station has noticeably lower FPY than others running the same procedure, you've found a test infrastructure problem, not a product problem.

## Retest Rate vs Cost Impact

Here's how retest rate maps to real production impact at different scales.

| Monthly volume | Retest rate | Retests/mo | Est. cost/mo (@ $2/retest) | Station capacity lost |
|---|---|---|---|---|
| 1,000 | 2% | 20 | $40 | Negligible |
| 1,000 | 10% | 100 | $200 | ~2 hours |
| 10,000 | 2% | 200 | $400 | ~1 day |
| 10,000 | 10% | 1,000 | $2,000 | ~5 days |
| 100,000 | 5% | 5,000 | $10,000 | ~25 days |
| 100,000 | 10% | 10,000 | $20,000 | ~50 days |

At high volume, even a small retest rate improvement frees up meaningful station capacity. Going from 10% to 5% at 100k units/month recovers 25 days of station time.

## How to Reduce Retest Rate

Reducing retest rate is almost always higher ROI than buying more stations.

**1. Stabilize the test environment.** Most retests at low maturity come from the test system, not the product. Check fixture contact resistance, instrument warm-up, cable integrity, and software timeouts. If a unit passes on retry without rework, the problem is your test, not the DUT.

**2. Use Cpk to set limits.** Limits derived from datasheets are often tighter than necessary. Pull measurement distributions from TofuPilot's analytics, calculate Cpk, and widen limits where you have margin. A Cpk of 1.33 or higher means the process is well within spec.

**3. Add marginal bands.** Configure marginal limits in your test to flag units that pass but are close to the boundary. These are your future retests. Catching them early lets you investigate before the limit becomes a yield problem.

**4. Root-cause the top failures.** Use TofuPilot's measurement analytics to identify which phases fail most often. Focus on the top 3. Pareto analysis almost always reveals that a small number of failure modes drive most retests.

**5. Track retest rate as a KPI.** Make FPY (or its inverse, retest rate) a weekly metric. Review it in TofuPilot's Analytics tab. Teams that track retest rate consistently reduce it. Teams that don't, don't.

### Separate Compiled Code in LabVIEW, Explained
URL: https://www.tofupilot.com/guides/separate-compiled-code-in-labview-explained
Learn what separate compiled code changes inside a VI, which Git noise it removes and which it doesn't, and how to turn it on for a project in one clean commit.

"Separate compiled code from source file" moves a VI's compiled code out of the `.vi` file into a local object cache, so recompiling a caller because its callee changed no longer rewrites the caller on disk. That removes most of the spurious "modified" files Git shows after a small LabVIEW change. It doesn't make VIs diffable, and a mass compile still touches every file.

## What's Inside a VI File

By default a `.vi` holds three things in one binary file: the front panel, the block diagram, and the compiled code for the platform and LabVIEW version that last saved it. The source and the machine code travel together.

That coupling is what produces the noise. When subVI B changes, LabVIEW recompiles every caller A that's loaded, because A's compiled code depends on B's connector pane and inlining. A is now modified in memory, and saving it rewrites A's bytes even though nobody opened A's diagram.

Git can't tell the difference. It sees forty changed files after a one-VI edit, and the commit history records a change to A that never happened in A's source.

## What the Setting Changes

The setting has existed since LabVIEW 2010, per VI and per project. Per VI it's in VI Properties, General page. Per project it's in the project properties, where "Separate compiled code from new project files" applies to files added from then on, and a "Mark Existing Items" button applies it to everything already in the project.

With it on, LabVIEW saves only the source in the `.vi` and writes the compiled code to the VI object cache. The cache is a per-user, per-LabVIEW-version, per-bitness directory on the local machine, with its location set under Tools, Options, Environment. It's never in the repository, so a fresh clone opens slower the first time while the cache fills, and a build machine rebuilds it from scratch.

Built executables are unaffected. The Application Builder still embeds compiled code in the executable, so nothing changes for the station.

The two NI pages that describe the mechanism are [separating compiled code from VIs and other file types](https://www.ni.com/docs/en-US/bundle/labview/page/separating-compiled-code-from-vis-and-other-file-types.html) and [separating compiled code for source control](https://www.ni.com/docs/en-US/bundle/labview/page/facilitating-source-control-by-separating-compiled-code-from-vis-and-other-file-types.html).

## What Improves in Git

The gain is that "modified" starts to mean "the source changed". Here is what Git shows for the common cases, before and after.

| Change | Before, compiled code inside the VI | After, compiled code in the cache |
|---|---|---|
| Callee B's diagram edited | B and every loaded caller A show as modified | Only B |
| Caller A's diagram edited | A, plus anything A's change forces to recompile up the chain | Only A |
| Mass compile after a LabVIEW version upgrade | Every VI rewritten | Every VI still rewritten, the source format changes with the version |
| Bitness change, 32-bit to 64-bit | Every VI recompiled and saved | Compiled code goes to a separate cache, saved files still show as touched |
| A LabVIEW class after a mass compile | `.lvclass` and members modified | `.lvclass` can still show as modified |

Read the first two rows as the win. A commit after a limit change in one subVI contains one file, which makes the history readable and a revert safe. The reason that matters, and everything else that binary files cost a team, is in [why LabVIEW version control is so painful](/guides/why-labview-version-control-is-so-painful).

Read the last three rows as the limit. A version upgrade, a bitness change or a full mass compile still rewrites the tree, and the history of individual changes on either side of that commit no longer diffs cleanly. [Upgrade a station from LabVIEW 2019 to 2024](/guides/upgrade-a-station-from-labview-2019-to-2024) plans around that commit rather than pretending it isn't there.

## What Doesn't Change

The `.vi` is still binary. Git still says "Binary files differ", `git blame` still names one author for the whole file, and a pull request still shows nothing a reviewer can read.

Merging is unchanged too. Two people editing the same VI on two branches still need LVMerge, which needs the Professional Development System and the same LabVIEW version on the machine doing the merge. The setting reduces how often two people touch the same file by accident, which helps, but a real conflict is still resolved as [fix LabVIEW merge conflicts on binary VIs](/guides/fix-labview-merge-conflicts-on-binary-vis) describes.

The setting is a source control hygiene fix. It's worth doing, and it's the whole of what it does.

## How to Turn It On for a Project Safely

The switch touches every file once. Do it in one sitting so the noise lands in one commit.

1. Commit a clean tree and tag it, so there's a known point before the change.
2. Put everyone on the same LabVIEW version and bitness. The cache is per version and per bitness anyway, and mixing them during the switch produces a second round of modified files.
3. In the project properties, tick "Separate compiled code from new project files", then use "Mark Existing Items" to apply the setting to every VI, control, class and library already in the project.
4. Mass compile the project once, from Tools, Advanced, Mass Compile, so every file is saved with the new setting and a valid cache entry.
5. Commit everything as one commit with a message that says what it is, and tag it.
6. Tell Git the files are binary, so it never tries to diff or merge them as text.

```text filename=".gitattributes"
# LabVIEW source is binary: never line-diff or auto-merge these.
*.vi binary
*.vit binary
*.ctl binary
*.ctt binary
```

From the next commit on, a modified VI means someone edited it. A colleague who pulls the tagged commit opens the project, waits for the cache to fill, and sees a clean tree.

The one-time commit is large and useless to review. Name it clearly, then treat it as a boundary: `git log` before it and after it are both readable, across it they aren't.

## The Classes Caveat

A `.lvclass` file is XML, but it carries the class's private data control flattened inside it. A mass compile can re-flatten that control and change the bytes, so a class shows as modified after a mass compile even with separate compiled code on and no source change anywhere.

Expect it, and don't chase it. After a mass compile, commit the classes with everything else and don't try to review those diffs. During normal work, a class that shows as modified when nobody edited it is a sign that a member VI or the private data control was recompiled, and it's safe to commit or discard.

## The Contrast: A Repository With No Compiled Code At All

A Python station under the TofuPilot Framework has nothing to separate. The repository holds `procedure.yaml`, `phases/*.py` and `plugs/*.py`, all text, and Git diffs, blames and merges them with no plugin and no licence.

The compiled artifact exists, but it lives somewhere else. A Git push creates a deployment in TofuPilot, an immutable build of the procedure at that commit with an id like `dep_abc123`, and the station fetches it with `tofupilot pull`. The repository only ever holds source, every run uploaded from the station carries the deployment that produced it, and a mass compile isn't a concept.

## Start With One Station

When you do move a station, pick the one where the mass compile commit hurt most, rebuild it in Python with the TofuPilot Framework, and run it next to the LabVIEW version on the same units for two weeks. The repository for that station will hold source only.

```bash filename="install-and-run.sh"
# Install the CLI, then run the station locally (no account needed) or with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Trace a Test Change to the Units It Shipped
URL: https://www.tofupilot.com/guides/trace-a-test-change-to-the-units-it-shipped
Learn how to stamp the test version on every run so a limit change traces to the exact serial numbers it shipped, on LabVIEW today and Python by default.

A limit changed in March, and the question "which serial numbers ran the old limit" is answerable only if every run records the exact test version that produced it. On most LabVIEW stations it doesn't: the record has a date, a serial number and a result, and the version lives in whoever remembers which build was copied when. This guide shows how to stamp the version on every run from a LabVIEW station today, and how a Python station under TofuPilot does it by default.

## Why It Matters

The question arrives in one of three forms, and each has a deadline attached.

| Situation | The question | What you need on every run |
|---|---|---|
| A customer return | Was this unit tested with the limit that was in force, or the one before? | The test version and the limit values |
| An audit | Show the test version each shipped unit was tested with | The test version, per serial number, exportable |
| A limit wrong for a month | Which units passed under the wrong value? | The test version and the measured value |

The third one is the expensive one. Someone types 3.15 V instead of 3.25 V on February 10th and the mistake is found on March 12th.

If every run carries its version, the answer is a filter: the units that ran that build and measured between 3.15 V and 3.25 V, a rescreen of a few dozen units. If it doesn't, the answer is every unit shipped in the window, plus a week of reconstruction from file dates and emails.

In medical, aerospace and automotive the customer expects to be shown which test method each unit went through. The expectation is the same in any plant that has had one return escalate. Recording the version costs nothing per run and pays back the first time the question is asked.

## What "Test Version" Has to Mean

The test version is the built artifact, not the source commit.

A rebuild from the same commit on a different machine can behave differently. A different LabVIEW patch level, a different driver, a subVI resolved from a different path, a 32-bit build where the last one was 64-bit. Two executables built from one commit are two versions, and only a number that increments with every build tells them apart.

So the rule is: one build, one number, stamped into the artifact and copied into every record it produces. Keep the source commit too, but as a property of the build, not as its identity. On a Python station this is exactly what a deployment is, and the commit SHA is one of its fields.

## On a LabVIEW Station Today

Three steps, and none of them needs a migration.

Embed a build number in the executable. Fill the version information in the build specification and let it increment on every build, and have the build write the same number to a `version.txt` next to the executable. At startup the test reads it once and keeps it in a global.

Write it into every record. A `test_build` property on the TDMS file, or a column in the CSV, written on every run with the serial number. The record is now self-describing without the file date.

Post it to TofuPilot with the run. The REST API v2 takes a `procedure_version` field on each run, and the HTTP Client VIs can send this JSON with an `Authorization: Bearer <api key>` header.

```json filename="run.json"
{
  "outcome": "PASS",
  "procedure_id": "9f1c6a2e-4b0d-4b8e-9d3a-2f0c5e7a1b44",
  "procedure_version": "fct-b117",
  "serial_number": "SN-004512",
  "part_number": "CTRL-PCBA-A",
  "started_at": "2026-09-23T08:14:02Z",
  "ended_at": "2026-09-23T08:15:41Z",
  "phases": [
    {
      "name": "Power Rails",
      "outcome": "PASS",
      "started_at": "2026-09-23T08:14:05Z",
      "ended_at": "2026-09-23T08:14:20Z",
      "measurements": [
        {
          "name": "rail_3v3",
          "outcome": "PASS",
          "measured_value": 3.31,
          "units": "V",
          "validators": [
            { "operator": ">=", "expected_value": 3.2, "outcome": "PASS" },
            { "operator": "<=", "expected_value": 3.4, "outcome": "PASS" }
          ]
        }
      ]
    }
  ]
}
```

Two things now hold on every run in TofuPilot. `procedure_version` carries the build number, so the run list filters on it. And the validators travel with the measurement, so the limit that was in force is on the run itself, not in a spreadsheet that says which limits build 117 had.

The station stays LabVIEW. The traceability starts before any migration, and the same station's rows appear next to the Python stations' rows on the same procedure page.

## On a Python Station With TofuPilot

The version is part of the station's source, and the artifact is created for you.

```yaml filename="procedure.yaml"
# The version travels with the procedure and is bumped in the same commit as the limit.
name: Controller PCBA FCT
version: 2.5.0
```

A Git push on the connected repository creates a deployment, an immutable build of the procedure at that commit with an id like `dep_9c04e8`. `tofupilot deploy --prod` from a linked directory does the same from the command line. Stations fetch it and run it.

```bash filename="release.sh"
# On the development machine: build a production deployment from the linked directory.
tofupilot deploy --prod

# On each station: fetch the pinned deployment, then run with upload.
tofupilot pull
tofupilot run ./procedure.yaml --upload
```

Every run uploaded from a pulled deployment carries its `deployment_id`. "Which units ran the old limit" becomes a filter on the run list by deployment, and `tofupilot deployments ls --procedure-ids <id> --environments production` lists every build that ever went to production, in order.

Rollback is a re-pin, not a rebuild. Instant rollback points every production station back at a previous deployment, and the runs already uploaded keep the id of the deployment that produced them, so the record of the bad month stays intact after the fix. The deploy and pull flow is in [how to deploy Python test scripts to production stations](/guides/how-to-deploy-python-test-scripts-to-production-stations), and the repository side is in [how to version control hardware tests with Git and TofuPilot](/guides/how-to-version-control-hardware-tests-with-git-and-tofupilot).

The DUT has a version too. [How to track firmware versions in production](/guides/how-to-track-firmware-versions-in-production-with-tofupilot) records the firmware the same way, so a run carries both the test that judged the unit and the code that was running on it.

## A Worked Example

The 3V3 rail lower limit on a controller PCBA, over one quarter.

| Date | Deployment | `rail_3v3` lower limit | What it means for units |
|---|---|---|---|
| 2026-01-06 | `dep_41b2c9` | 3.25 V | Baseline, 2,100 units |
| 2026-02-10 | `dep_7f3a21` | 3.15 V, a typo in the pull request | 1,240 units ran this build |
| 2026-03-12 | `dep_9c04e8` | 3.25 V, corrected | Correct again from this run on |

The question on March 12th is which units passed under 3.15 V with a value that 3.25 V would have rejected. Filter the run list on `dep_7f3a21`, look at `rail_3v3` on those runs, and the answer is 38 serial numbers out of 1,240.

Those 38 get rescreened. The other 1,202 are shown, per serial number, to have measured above the correct limit.

Without the deployment on each run, the same question is "every unit shipped between February 10th and March 12th", and the rescreen is 1,240 units. The difference is the cost of one column.

## The Five-Minute Check Before Every Release

Five questions, asked with the build in hand and before it reaches a station.

1. Does the version differ from the last one? `version.txt` shows b117 where the floor has b116, or `procedure.yaml` moved from 2.4.0 to 2.5.0 in the same commit as the change.
2. Does a golden unit run on the new build show that version on its run page in TofuPilot?
3. Is the change one readable line somewhere, a CHANGELOG entry or a merged pull request, that names the limit and the reason?
4. Is the way back known? The previous executable kept next to the new one, or the previous deployment id noted for a rollback.
5. Is the list of stations that will receive it written down, with the LabVIEW version each one runs? [Which LabVIEW version runs on each station](/guides/which-labview-version-runs-on-each-station) has the inventory.

The last question is the one that stalls releases on LabVIEW stations, because the build has to match the Run-Time Engine on each of them. On a Python station the list is the set of stations pinned to the procedure, and pulling the deployment is the release. What the review of that change looks like, and why it's hard to get on a VI, is in [code review for LabVIEW tests: why it fails](/guides/code-review-for-labview-tests-why-it-fails), and the file-level reason behind all of it is in [why LabVIEW version control is so painful](/guides/why-labview-version-control-is-so-painful).

## Start With One Station

Pick the station whose limits changed most in the last year, rebuild it in Python with the TofuPilot Framework with the same limits, and run it next to the LabVIEW version on the same units for two weeks. Every run from the Python side will carry its deployment from the first day.

```bash filename="install-and-run.sh"
# Install the CLI, then run the station locally (no account needed) or with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### How to Reduce Retesting Costs
URL: https://www.tofupilot.com/guides/how-to-reduce-retesting-costs-with-tofupilot
Learn why retesting wastes 10-30% of test capacity, how to fix root causes with better OpenHTF patterns, and how TofuPilot tracks retest rates.

Retesting eats 10-30% of test station capacity on a typical electronics production line. Every retest cycle burns station time, operator attention, and fixture wear, all without producing a single new unit. Cutting your retest rate is one of the fastest ways to increase throughput without buying more equipment.

## Why Retesting Is So Expensive

The direct cost of a retest seems small: run the test again, maybe it passes. But the hidden costs add up fast.

**Station throughput drops.** A station with 15% retest rate effectively loses 15% of its capacity. For a station running 500 units per shift, that's 75 slots wasted on units you already tested once.

**Rework creates quality risk.** Every rework cycle (desolder, replace, resolder) introduces thermal stress, pad damage, and handling risk. Reworked units fail at higher rates downstream.

**Data gets noisy.** When operators retest units without tracking serial numbers, your yield metrics become unreliable. You can't tell whether FPY is 85% or 95% because first attempts and retests are mixed together.

## Root Causes of Excessive Retesting

Before you can fix retesting, you need to know why it's happening. The causes usually fall into four categories.

### Loose or Missing Test Limits

Tests without proper limits can't distinguish real failures from marginal passes. If a voltage rail is specified at 3.3V +/- 5%, but your test has no limits, operators make judgment calls about what "looks okay."

### Flaky Test Fixtures

Contact resistance on pogo pins, worn alignment features, and intermittent cable connections cause random failures. The unit is fine. The fixture isn't. Operators know this, so they retest.

### Operator-Initiated Re-runs

Without clear pass/fail criteria, operators develop habits: "if it fails on voltage, just run it again." This masks both fixture problems and real defects.

### Environmental Sensitivity

Temperature, humidity, and power supply variation can push measurements across limits. If your test runs differently at 8 AM versus 2 PM, you have an environment problem.

## OpenHTF Patterns That Reduce Retesting

Good test code prevents unnecessary retests by catching fixture issues early and setting proper limits.

### Validate the Fixture Before Testing the DUT

Add a fixture validation phase at the start of your test sequence. If the fixture fails, the run aborts before logging a false failure against the unit.

```python filename="test_with_fixture_check.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("fixture_contact_resistance")
    .with_units(units.OHM)
    .in_range(maximum=0.5),
    htf.Measurement("fixture_supply_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=11.8, maximum=12.2),
)
def validate_fixture(test):
    # Check pogo pin contact and power supply before DUT test
    test.measurements.fixture_contact_resistance = 0.12
    test.measurements.fixture_supply_voltage = 12.01

@htf.measures(
    htf.Measurement("output_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=3.135, maximum=3.465),
    htf.Measurement("ripple_mV")
    .in_range(maximum=50),
)
def test_power_output(test):
    test.measurements.output_voltage = 3.29
    test.measurements.ripple_mV = 18

@htf.measures(
    htf.Measurement("signal_integrity_dB")
    .in_range(minimum=-3.0, maximum=3.0),
)
def test_signal_path(test):
    test.measurements.signal_integrity_dB = 0.4

def main():
    test = htf.Test(validate_fixture, test_power_output, test_signal_path)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "SN-2026-01105")

if __name__ == "__main__":
    main()
```

The `validate_fixture` phase runs first. If contact resistance is too high or the supply voltage is out of range, the test fails immediately. The failure is attributed to the fixture, not the DUT, and no false failure gets recorded against the serial number.

### Set Measurement Limits Based on Process Capability

Don't guess limits. Set them from your Cpk data. If your process produces output voltage with a mean of 3.30V and a standard deviation of 0.03V, your 3-sigma range is 3.21V to 3.39V.

Your test limits should be wider than 3-sigma (to avoid false failures) but within spec (to catch real defects). A good starting point is the spec limits from the component datasheet.

```python filename="calibrated_limits.py"
import openhtf as htf
from openhtf.util import units

# Limits derived from Cpk study: spec is 3.0-3.6V, process sigma is 0.03V
# Test limits set at spec boundaries to catch real defects
@htf.measures(
    htf.Measurement("regulated_output")
    .with_units(units.VOLT)
    .in_range(minimum=3.0, maximum=3.6),
    htf.Measurement("load_regulation_pct")
    .in_range(minimum=-2.0, maximum=2.0),
    htf.Measurement("thermal_shutdown_temp")
    .with_units(units.DEGREE_CELSIUS)
    .in_range(minimum=145, maximum=155),
)
def test_regulator(test):
    test.measurements.regulated_output = 3.31
    test.measurements.load_regulation_pct = 0.8
    test.measurements.thermal_shutdown_temp = 150
```

## How TofuPilot Tracks Retest Rates

TofuPilot links every test run to a serial number, so it knows when a unit is being tested for the second, third, or tenth time.

The dashboard shows you:

- **Retest rate by station.** If one station has 3x the retest rate of another running the same test, the station needs maintenance.
- **Unit history.** For any serial number, see every test attempt in sequence. Spot patterns like "fails once, passes on immediate retest" (fixture issue) versus "fails repeatedly on the same measurement" (real defect).
- **FPY vs. output yield gap.** A large gap between FPY and output yield means you're relying heavily on retesting to hit your yield target. That gap is your retest cost, made visible.
- **Failure Pareto.** See which test phases cause the most failures. If one phase accounts for 60% of retests, fixing that phase gives you the biggest return.

## A Practical Reduction Playbook

1. **Measure your current retest rate.** Check TofuPilot's FPY dashboard. The gap between FPY and output yield is your retest cost.
2. **Identify the top failure phase.** Use the failure Pareto chart. Focus on the single biggest contributor first.
3. **Classify failures.** For the top phase, check whether failures are fixture-related (pass on immediate retest) or DUT-related (consistent failure).
4. **Fix fixtures first.** Fixture failures are the cheapest to fix. Replace worn pogo pins, tighten alignment, and add fixture validation phases.
5. **Review test limits.** Use measurement histograms and Cpk charts in TofuPilot to check whether limits match process capability. Tighten limits that are too loose. Widen limits where Cpk shows the process can't hit them.
6. **Repeat monthly.** Retest rate creeps up as fixtures wear and process conditions change. Make it a recurring review.

### How to Build a Test Sequencer in Python
URL: https://www.tofupilot.com/guides/how-to-build-a-test-sequencer-with-python-and-tofupilot
Build a test sequencer in Python, or use one that exists. What a sequencer must handle, and why hand-rolling it usually costs more than it saves.

Most hardware teams write a test sequencer at some point. It starts as a list of functions and a `for` loop, and a year later it has retry logic, an operator prompt, a CSV writer and a threading bug nobody wants to touch. This guide covers what a sequencer actually has to do, how to build one if you need to, and how to recognise when you have rebuilt something that already exists.

## What a Sequencer Is Responsible For

| Responsibility | What it means in practice |
|---|---|
| Ordering | Steps run in a defined order, with dependencies between them |
| Limit evaluation | Pass/fail derived from a measurement and its limits, not hand-written per step |
| Result capture | The value, its units and its limits are recorded, not just a verdict |
| Unit identity | Every result is attached to a serial number |
| Resource lifecycle | Instruments open once, are shared, and close even when a step raises |
| Failure policy | Some failures stop the run; others continue for diagnostics |
| Retry | Flaky steps retry with a bounded count |
| Timeouts | A hung step fails instead of blocking the line forever |
| Operator interaction | Prompts, scans, confirmations |
| Persistence | Results survive the shift and the machine |

The first three are easy. The rest is where hand-rolled sequencers accumulate their complexity.

## The Naive Version

```python filename="naive/sequencer.py"
def run_tests(serial: str):
    results = {}
    dmm = open_dmm()
    try:
        v = dmm.read_voltage()
        results["rail_3v3"] = (v, 3.2 <= v <= 3.4)

        i = dmm.read_current()
        results["idle_current"] = (i, i <= 0.045)
    finally:
        dmm.close()

    passed = all(ok for _, ok in results.values())
    print(f"{serial}: {'PASS' if passed else 'FAIL'}")
    return passed
```

This is fine for a bench experiment and wrong for production in ways that are not obvious yet:

- The limits are embedded in comparisons, so they cannot be reported, reviewed, or trended.
- There is no record beyond a printed line.
- Every new step duplicates the same tuple-and-compare pattern.
- Adding retry or timeout means restructuring every step.

## A Reasonable Hand-Rolled Version

If you are going to build one, this is roughly the minimum that survives contact with a production line:

```python filename="sequencer/core.py"
import time
from dataclasses import dataclass, field
from typing import Callable


@dataclass
class Measurement:
    name: str
    value: float | str | bool | None = None
    lower: float | None = None
    upper: float | None = None
    unit: str = ""

    @property
    def passed(self) -> bool:
        if self.value is None:
            return False
        if self.lower is not None and self.value < self.lower:
            return False
        if self.upper is not None and self.value > self.upper:
            return False
        return True


@dataclass
class Step:
    name: str
    fn: Callable
    measurements: list[Measurement] = field(default_factory=list)
    critical: bool = False
    retries: int = 0
    timeout_s: float | None = None


class Sequencer:
    def __init__(self, serial: str, resources: dict):
        self.serial = serial
        self.resources = resources
        self.results = []

    def run(self, steps: list[Step]) -> bool:
        overall = True
        for step in steps:
            ok = self._run_step(step)
            overall = overall and ok
            if not ok and step.critical:
                break
        return overall

    def _run_step(self, step: Step) -> bool:
        for attempt in range(step.retries + 1):
            started = time.time()
            try:
                step.fn(step.measurements, **self.resources)
                ok = all(m.passed for m in step.measurements)
            except Exception as exc:
                ok = False
                self.results.append({"step": step.name, "error": str(exc)})

            if ok or attempt == step.retries:
                self.results.append({
                    "step": step.name,
                    "passed": ok,
                    "duration_s": time.time() - started,
                    "measurements": [vars(m) for m in step.measurements],
                })
                return ok
        return False
```

That is around 60 lines and it still lacks timeouts that actually interrupt a hung call, parallel execution, an operator interface, offline persistence, and anything resembling analytics. Each of those is another few hundred lines, and the threading one is genuinely hard to get right.

## The Version That Already Exists

The same test, declared rather than implemented. The sequence, the limits, the retry policy and the failure behaviour are data:

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    timeout: 30s
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Idle Current
        unit: A
        validators:
          - operator: "<="
            expected_value: 0.045

  - name: Comms Check
    python: phases.comms_check
    depends_on: [power_rails]
    retry:
      limit: 3
      delay: 1s
    then:
      fail: retry
    measurements:
      - name: Link Established
        validators:
          - operator: "=="
            expected_value: true
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
    measurements.idle_current = dmm.read_current()
```

```python filename="phases/comms_check.py"
def comms_check(measurements, dmm):
    measurements.link_established = True
```

```python filename="plugs/dmm.py"
class Multimeter:
    def read_voltage(self) -> float:
        return 3.31  # Replace with a real instrument read

    def read_current(self) -> float:
        return 0.041
```

Run it with `tofupilot run ./procedure.yaml`.

Two details in that YAML worth noting, because they are easy to get wrong:

**`timeout` takes a duration string**, such as `30s` or `2m`, not a bare number. A phase without one waits indefinitely.

**Retry needs both a policy and a trigger.** `retry.limit` bounds the attempts and `retry.delay` spaces them, but a phase only retries when something asks it to: either the phase returns `PhaseResult.RETRY` from Python, or `then:` maps an outcome to a retry, as `then: fail: retry` does above.

Mapping the hand-rolled concepts onto it:

| Hand-rolled | Declared |
|---|---|
| `Step.critical` and the `break` | `depends_on` |
| `Step.retries` loop | `retry.limit` plus a `then:` trigger |
| `Step.timeout_s` | `timeout: 30s` |
| `Measurement.passed` comparisons | `validators` |
| `resources` dict passed to every step | Plugs injected by parameter name |
| `self.results` and a CSV writer | Automatic upload |
| Threading for parallel steps | Independent phases run in parallel |

The dependency graph replaces the critical flag with something more precise. `depends_on` says which steps actually require an earlier one to have succeeded, so unrelated steps still run and give you diagnostics, while dependent steps skip.

## When Building Your Own Is Right

- **You have a genuinely unusual execution model** that no framework expresses, such as a closed-loop search rather than a fixed sequence.
- **You have an existing sequencer that works** and the only complaint is maintenance. Reducing that maintenance is usually cheaper than migrating.
- **Your constraints prohibit external dependencies**, which happens in some regulated or air-gapped environments.

If your reason is "our sequence is simple", note that the version above was also simple a year ago.

## What Usually Triggers the Rewrite

Teams tend to abandon hand-rolled sequencers at a predictable point, and it is rarely the sequencing:

- Someone asks for yield by measurement over the last month and the CSVs cannot answer it.
- The line needs four boards tested at once and the loop is single-threaded.
- An operator needs a screen, and now there is a frontend project.
- A station loses network and a shift of results is lost.
- Limits change and nobody can tell which results used which limits.

Those are the parts that are expensive to build and are already solved.

## If You Prefer Decorators

OpenHTF is the other purpose-built Python option, with the sequence and limits expressed in code:

```python filename="openhtf_sequencer.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
@htf.PhaseOptions(timeout_s=30)
def test_power_rails(test):
    test.measurements.rail_3v3 = 3.31


test = htf.Test(test_power_rails, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`. See [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf).

## Common Mistakes

| Mistake | Consequence | Fix |
|---|---|---|
| Limits inside comparisons | Cannot report, review or trend them | Declare limits as data |
| Results printed to a console | Nothing survives the shift | Persist every run |
| One global try/except | A failing step hides which one failed | Per-step error capture |
| Retry around the whole sequence | Passing steps re-run and waste cycle time | Retry per step |
| Retry configured with no trigger | The phase never actually retries | Add `then: fail: retry` or return `PhaseResult.RETRY` |
| `timeout: 30` instead of `timeout: 30s` | Rejected; durations are strings | Use a duration string |
| Threading added late | Race conditions in shared instrument access | Declare dependencies; share instruments deliberately |

### How to Reduce Field Returns with Test Data
URL: https://www.tofupilot.com/guides/how-to-reduce-field-returns-with-tofupilot
Learn how to catch marginal units before they ship by adding margin tests and stress phases in OpenHTF, then correlating field failures in TofuPilot.

Units that pass production testing but fail in the field are expensive. Every field return costs 10 to 100 times more than catching the same defect on the line. The fix starts with better test coverage and tighter analysis of the data you're already collecting.

## Why Units Pass Tests but Fail in the Field

Three root causes account for most field returns:

**Insufficient test coverage.** Your production test checks 80% of the functionality, but the 20% you skip includes the failure mode customers hit. If you don't test the sleep/wake cycle, you won't catch the firmware bug that corrupts EEPROM after 1,000 transitions.

**Test limits too loose.** A unit measures 4.65V against a 4.5V to 5.5V spec. It passes, but it's drifting toward the edge. Temperature shifts in the field push it out of range. Your test caught nothing because the limits matched the datasheet, not real-world conditions.

**Environmental conditions not tested.** Production tests run at 25C on a bench. The product operates at 0C to 50C in a warehouse with vibration. Parameters that look solid at room temperature fall apart under stress.

## Add Margin Testing to Your OpenHTF Tests

Margin testing uses tighter limits than the product specification. The idea is simple: if a unit can't pass with headroom, it's more likely to fail in the field.

```python filename="margin_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("output_voltage")
    .in_range(minimum=4.8, maximum=5.2)
    .with_units(units.VOLT)
    .doc("Spec is 4.5-5.5V. Margin limits tightened to 4.8-5.2V."),
    htf.Measurement("ripple_voltage")
    .in_range(maximum=20)
    .with_units(units.MILLIVOLT)
    .doc("Spec allows 50mV. Margin limit set to 20mV."),
    htf.Measurement("quiescent_current")
    .in_range(maximum=0.000008)
    .with_units(units.AMPERE)
    .doc("Spec allows 15uA. Margin limit catches leaky units."),
)
def margin_check(test):
    test.measurements.output_voltage = 5.03
    test.measurements.ripple_voltage = 12.4
    test.measurements.quiescent_current = 0.0000051


def main():
    test = htf.Test(
        margin_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # PCBA Margin Test
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-20260312-042")


if __name__ == "__main__":
    main()
```

Use the `.doc()` method to record both the spec limit and your margin limit. This makes it clear to anyone reviewing the data why a unit at 4.75V failed even though the datasheet says 4.5V is fine.

Margin limits reject units that would have passed the spec, so expect first pass yield to drop when you introduce them. That drop is the point, but size it deliberately: limits tightened past your process spread scrap good units at a steady rate.

## Add Environmental Stress Phases

For products that operate across a temperature range, add stress phases that test at the boundaries. You don't need a full thermal chamber for every unit. Even a brief hot/cold soak on a sample basis catches drift.

```python filename="stress_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("voltage_ambient")
    .in_range(minimum=4.8, maximum=5.2)
    .with_units(units.VOLT),
)
def ambient_check(test):
    # Measure at room temperature baseline
    test.measurements.voltage_ambient = 5.01


@htf.measures(
    htf.Measurement("voltage_hot")
    .in_range(minimum=4.7, maximum=5.3)
    .with_units(units.VOLT),
    htf.Measurement("voltage_drift_hot")
    .in_range(maximum=100)
    .with_units(units.MILLIVOLT)
    .doc("Delta between 25C and 50C readings"),
)
def hot_soak_check(test):
    # Measure after thermal soak at upper operating limit
    test.measurements.voltage_hot = 5.08
    test.measurements.voltage_drift_hot = 70


@htf.measures(
    htf.Measurement("voltage_cold")
    .in_range(minimum=4.7, maximum=5.3)
    .with_units(units.VOLT),
    htf.Measurement("voltage_drift_cold")
    .in_range(maximum=100)
    .with_units(units.MILLIVOLT)
    .doc("Delta between 25C and 0C readings"),
)
def cold_soak_check(test):
    # Measure after thermal soak at lower operating limit
    test.measurements.voltage_cold = 4.92
    test.measurements.voltage_drift_cold = 90


def main():
    test = htf.Test(
        ambient_check,
        hot_soak_check,
        cold_soak_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # PCBA Thermal Stress Test
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-20260312-042")


if __name__ == "__main__":
    main()
```

Recording the drift as a separate measurement is key. A unit might pass both the hot and cold absolute checks but show 95mV of drift, which signals a component that's near its thermal limit.

## Correlate Field Failures Back to Production Data

When a unit comes back from the field, look it up by serial number. The unit history shows every test run, including all measurements recorded during production.

Here's the workflow:

1. **Get the serial number** from the returned unit.
2. **Search by serial number** to pull up the unit's full test history.
3. **Check the measurements** for the failure mode. If the field failure is "intermittent power loss," look at the production voltage and current measurements.
4. **Compare against passing units.** Was the returned unit near the edge of a limit? Did it show higher drift than units that survived?

This comparison tells you whether your limits need tightening or your test coverage has gaps.

One caution: returned units are a biased sample. You see the ones that failed, not the ones that sat near the same limit and survived. Before tightening a limit, check what fraction of your passing population sits in the same band, or you will scrap far more units than you save.

## Identify Marginal Units with Measurement Analytics

Measurement analytics show you the distribution of every measurement across all units. This is where you catch systematic issues before they become field returns.

**Measurement histograms** reveal bimodal distributions. If your voltage measurement shows two clusters instead of one normal curve, you likely have a component sourcing issue or a process variation. Units in the lower cluster are your field return candidates.

**Cpk values** tell you how centered your process is within the limits. A Cpk below 1.33 means your process variation is too wide relative to your test limits. Even if every unit passes today, normal variation will push some units out of spec in the field.

**Control charts** show measurement trends over time. A gradual drift in a measurement across production batches signals a process change. Catching the drift early lets you investigate before marginal units ship.

## Tighten Limits Based on Field Data

Once you've correlated field returns with production measurements, update your test limits. The pattern is straightforward:

| Situation | Action |
|-----------|--------|
| Returned units clustered near upper limit | Lower the upper margin limit |
| Returned units show high thermal drift | Add or tighten drift measurement limit |
| Failure mode not covered by any measurement | Add a new test phase for that parameter |
| Returns correlate with a specific production batch | Investigate process change during that period |

Don't guess at new limits. Use the measurement distribution from your passing population to set limits that exclude the tail where failures concentrate.

When you do change a limit, record which procedure version produced each result. Otherwise results from before and after the change silently blend into one distribution and your yield history stops being comparable.

## Add Coverage for Missing Failure Modes

Field returns often reveal failure modes your test doesn't cover at all. When that happens, add a new phase to your OpenHTF test.

```python filename="added_coverage.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("sleep_wake_cycles_passed")
    .in_range(minimum=100)
    .doc("Added after field returns showed EEPROM corruption after repeated sleep/wake."),
    htf.Measurement("eeprom_checksum_valid")
    .equals(True),
)
def sleep_wake_stress(test):
    # Run 100 sleep/wake cycles and verify EEPROM integrity
    cycles_passed = 100
    checksum_ok = True
    test.measurements.sleep_wake_cycles_passed = cycles_passed
    test.measurements.eeprom_checksum_valid = checksum_ok


@htf.measures(
    htf.Measurement("output_voltage")
    .in_range(minimum=4.8, maximum=5.2)
    .with_units(units.VOLT),
)
def functional_check(test):
    test.measurements.output_voltage = 5.02


def main():
    test = htf.Test(
        sleep_wake_stress,
        functional_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # PCBA Functional Test v2
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-20260312-099")


if __name__ == "__main__":
    main()
```

Document why you added the phase using `.doc()`. Six months from now, someone will ask why you're running 100 sleep/wake cycles on every unit. The answer is right there in the measurement metadata.

Weigh the cycle time it costs. A 100-cycle stress phase on every unit is worth it when the failure mode is expensive and common; on a high-volume line it may be better as a sampled audit than as a gate on every unit.

## Build a Feedback Loop

Reducing field returns isn't a one-time fix. It's a continuous loop:

1. **Ship units** with your current test coverage and limits.
2. **Track returns** by serial number.
3. **Correlate** return data with production measurements.
4. **Tighten limits** or add test phases based on what you find.
5. **Monitor Cpk and first pass yield** to confirm the changes reduce returns without over-rejecting good units.

Step 5 is the one teams skip. A tightened limit that halves field returns while scrapping 3% more good units may or may not be worth it, and you cannot tell which without watching both numbers together.

### Test Traceability for IATF 16949
URL: https://www.tofupilot.com/guides/test-traceability-for-iatf-16949-with-tofupilot
Map IATF 16949 requirements for production test records, MSA, and control plans to TofuPilot's traceability and analytics features.

IATF 16949 requires automotive suppliers to maintain complete test records for every production unit, track measurement system capability, and follow documented control plans. TofuPilot gives you the infrastructure to capture, store, and analyze all of that without building custom tooling.

## What IATF 16949 Requires for Test Records

The standard builds on ISO 9001 and adds automotive-specific clauses that directly affect production testing.

**Clause 8.5.2 (Identification and Traceability)** requires you to identify each product throughout production and retain records that link a serial number to its full test history. Every DUT must be traceable from raw material through final test.

**Clause 7.1.5 (Measurement System Analysis)** mandates that you study the variation in your measurement systems. You need to demonstrate that your gauges, fixtures, and test equipment produce repeatable, reproducible results. This means tracking Cpk and GR&R across your measurement data over time.

**Clause 8.5.1.1 (Control Plans)** requires documented control plans that specify what you measure, how often, and what the acceptance criteria are. Your test infrastructure needs to enforce these limits consistently.

## How TofuPilot Maps to IATF Requirements

| IATF 16949 Clause | Requirement | TofuPilot Feature |
|---|---|---|
| 8.5.2 | Serial number traceability | Unit tracking with full run history per serial number |
| 7.1.5.1.1 | MSA / GR&R studies | Measurement history and Cpk charts per measurement name |
| 8.5.1.1 | Control plan enforcement | Measurement limits stored with every run |
| 8.6 | Release of products | Pass/fail verdict tied to limit compliance |
| 8.7 | Nonconforming output | Failure Pareto and filtering by failed measurements |
| 9.1.1 | Performance monitoring | FPY trends, station throughput, control charts |

## Structuring Tests for Automotive Traceability

Your OpenHTF test should capture the serial number (DUT ID), station identity, operator, and every measurement with its acceptance limits. This gives TofuPilot everything it needs to build a complete traceability record.

```python filename="automotive_ecu_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("supply_voltage")
    .in_range(minimum=4.75, maximum=5.25)
    .with_units(units.VOLT),
    htf.Measurement("quiescent_current")
    .in_range(maximum=0.012)
    .with_units(units.AMPERE),
    htf.Measurement("output_signal_frequency")
    .in_range(minimum=999.5, maximum=1000.5)
    .with_units(units.HERTZ),
)
def power_and_signal_check(test):
    test.measurements.supply_voltage = 5.02
    test.measurements.quiescent_current = 0.0087
    test.measurements.output_signal_frequency = 1000.1

@htf.measures(
    htf.Measurement("can_bus_response_time")
    .in_range(maximum=10.0),
    htf.Measurement("can_message_checksum_valid")
    .equals(True),
)
def communication_check(test):
    test.measurements.can_bus_response_time = 4.3
    test.measurements.can_message_checksum_valid = True

def main():
    test = htf.Test(
        power_and_signal_check,
        communication_check,
    )

    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "ECU-SN-20260312-0047")

if __name__ == "__main__":
    main()
```

Every run recorded in TofuPilot links the serial number, station, measurements, limits, and pass/fail verdict into one traceable record.

## Measurement System Analysis with TofuPilot

IATF 16949 requires you to prove your measurement systems are capable. TofuPilot tracks every measurement value with its limits across all runs, so you can monitor Cpk directly from the dashboard.

To support MSA, structure your tests so that each measurement name stays consistent across stations and over time. If you measure `supply_voltage` on Station A and Station B, use the same measurement name on both. TofuPilot's measurement analytics will then show you the distribution, Cpk, and control chart for that measurement across your entire fleet.

You don't need to write Python scripts to compute Cpk or build histograms. TofuPilot's dashboard shows process capability metrics, measurement histograms, and control charts for every measurement name you record.

## Control Plan Enforcement

Your control plan defines what gets measured and the acceptance criteria. In OpenHTF, you encode these as measurement limits using `.in_range()`. TofuPilot stores both the measured value and the limits with every run.

This means an auditor can verify that every unit was tested against the documented limits. If limits change between production batches, TofuPilot's run history shows exactly which limits applied to which units.

## Nonconforming Output and Failure Analysis

When a unit fails, IATF 16949 requires you to document what went wrong and prevent recurrence. TofuPilot's failure Pareto chart shows which measurements fail most often across your production. Filter by station, time range, or part number to isolate systemic issues.

Every failed run retains the exact measured values alongside the limits that were violated, giving you the raw data for 8D reports and corrective action records.

## Audit Readiness

For IATF audits, you need to demonstrate that your test system captures complete records and that you monitor process capability. With TofuPilot, you can pull up any unit by serial number and show its full test history, measurements, limits, and verdict. The dashboard's FPY trends and Cpk charts provide the ongoing monitoring evidence auditors expect.

### Hardware Test Observability
URL: https://www.tofupilot.com/guides/hardware-test-observability-with-tofupilot
Learn how to build full observability into your hardware test operations using TofuPilot's dashboards, alerts, and analytics.

Software teams have Datadog and Grafana. Hardware test teams have... shared drives full of CSV files. Test observability means having the same real-time visibility into your hardware test operations that software teams take for granted.

## What Test Observability Means for Hardware

Observability in software is about understanding system behavior from its outputs (logs, metrics, traces). Hardware test observability applies the same principle: understand your production quality from test outputs (measurements, pass/fail rates, cycle times).

| Software Observability | Hardware Test Observability |
|----------------------|---------------------------|
| Error rates | First-pass yield (FPY) |
| Latency metrics | Test cycle time |
| Log aggregation | Test result centralization |
| Alerting on anomalies | Yield drop notifications |
| Distributed tracing | Unit traceability across stations |

## The Three Pillars of Hardware Test Observability

### 1. Metrics

Quantitative data about your test operations.

- **First-pass yield (FPY)** per procedure, station, and time period
- **Measurement distributions** for every parameter you test
- **Cpk/Ppk** process capability indices
- **Test cycle time** and throughput
- **Failure pareto** showing top failure modes

TofuPilot computes these automatically from your test data. No spreadsheet formulas, no manual aggregation.

### 2. Logs

Every test run is a log entry. TofuPilot stores:

- Full measurement data with limits and units
- Pass/fail status per step and per run
- Timestamps, station identifiers, operator info
- Attachments (waveforms, images, log files)
- Unit metadata (serial number, revision, batch)

Unlike CSV files on a shared drive, these logs are indexed, searchable, and queryable through the dashboard and API.

### 3. Traces

A unit's journey through your test process is a trace. TofuPilot links all test runs for a given serial number, showing:

- Which tests the unit has passed
- Which station ran each test
- When each test was executed
- The complete measurement history

This is critical for units that go through multiple test stages (ICT, functional, burn-in, final test). The full trace tells you everything that happened to a unit from first test to ship.

## Building Observability with TofuPilot

### Step 1: Centralize All Test Data

Every test station should push results to TofuPilot. This eliminates data silos.

```python filename="station_setup.py"
from tofupilot import TofuPilotClient

# Same client works for every station
client = TofuPilotClient()

# Every run is automatically indexed by procedure, station, and unit
client.create_run(
    procedure_id="FINAL-TEST-V2",
    unit_under_test={"serial_number": "UNIT-8847"},
    run_passed=True,
    steps=[...],
)
```

### Step 2: Use Dashboards for Real-Time Visibility

TofuPilot's procedure dashboard shows live metrics:

- Current FPY with trend line
- Measurement distributions with limit overlays
- Recent run history with pass/fail status
- Failure mode breakdown

Pin the dashboards on a monitor near the production line. When yield drops, everyone sees it immediately.

### Step 3: Set Up Yield Monitoring

Track FPY over time to catch regressions early. A yield drop from 98% to 94% over a week is easy to miss in daily noise but obvious in a trend chart.

### Step 4: Enable Unit Traceability

When a field return comes in, search by serial number in TofuPilot. Every test the unit ever ran is there: measurements, pass/fail status, timestamps, and which station tested it. No digging through old files.

## Observability vs. Reporting

| Reporting | Observability |
|-----------|--------------|
| Backward-looking | Real-time |
| Manual (someone pulls a report) | Automatic (dashboards update live) |
| Aggregated (weekly/monthly summaries) | Granular (every run, every measurement) |
| Static (PDF/Excel) | Interactive (filter, drill down, compare) |

Reporting tells you what happened last month. Observability tells you what's happening right now.

## Common Observability Wins

**Catching a yield drop in hours, not weeks.** Without observability, yield problems surface in weekly quality reviews. With TofuPilot's live dashboards, engineers see the drop the same day it starts.

**Finding station-specific issues.** One station has 91% FPY while the others run at 98%. Without centralized data, this is invisible because each station has its own files. TofuPilot's station comparison makes it obvious.

**Reducing debug time for field returns.** Customer reports a failure. Instead of guessing what happened during production, pull the unit's full test history in seconds. Every measurement, every step, every station.

**Proving compliance with data.** When auditors ask for test records, point them to TofuPilot. Structured, timestamped, immutable records replace binders full of printouts.


### Multi-Site Test Data Management
URL: https://www.tofupilot.com/guides/multi-site-test-data-management-with-tofupilot
Learn how to manage hardware test data across multiple manufacturing sites and test locations using TofuPilot's centralized platform.

When you test hardware at multiple locations, data silos are inevitable. Factory A has its own test system. The contract manufacturer has another. Your lab uses something else entirely. TofuPilot centralizes test data from every site into one platform, so you can compare quality across locations.

## The Multi-Site Problem

Companies scale production by adding sites: an in-house prototype lab, a contract manufacturer for volume production, a second CM for geographic diversification, field test stations at customer sites.

Each site typically has:
- Different test equipment and fixtures
- Different data storage (local databases, CSV files, proprietary systems)
- Different reporting formats
- Different levels of data granularity

The result: you can't answer basic questions like "Is the CM's yield the same as our in-house line?" without weeks of manual data gathering and normalization.

## Centralizing Test Data

### Step 1: Standardize Procedure IDs

Use the same procedure ID at every site for the same test. This is the key to cross-site comparison.

```python filename="site_a_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Same procedure ID used at every site
client.create_run(
    procedure_id="FINAL-FUNCTIONAL-V3",
    unit_under_test={
        "serial_number": "UNIT-A-0042",
        "part_number": "PROD-100-R4",
    },
    run_passed=True,
    steps=[{
        "name": "Power Rail Check",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "vcc_3v3",
            "value": 3.31,
            "unit": "V",
            "limit_low": 3.25,
            "limit_high": 3.35,
        }],
    }],
)
```

### Step 2: Tag Runs with Site Information

Use station metadata or sub-unit tracking to identify which site produced each run.

```python filename="site_b_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

client.create_run(
    procedure_id="FINAL-FUNCTIONAL-V3",
    unit_under_test={
        "serial_number": "UNIT-B-1087",
        "part_number": "PROD-100-R4",
    },
    run_passed=True,
    steps=[{
        "name": "Power Rail Check",
        "step_type": "measurement",
        "status": True,
        "measurements": [{
            "name": "vcc_3v3",
            "value": 3.29,
            "unit": "V",
            "limit_low": 3.25,
            "limit_high": 3.35,
        }],
    }],
)
```

### Step 3: Compare Sites in the Dashboard

With standardized procedure IDs, TofuPilot's station comparison view works across sites. Filter by station to see:

| Metric | Site A (In-house) | Site B (CM) |
|--------|-------------------|-------------|
| FPY | 98.2% | 95.1% |
| Avg vcc_3v3 | 3.31 V | 3.29 V |
| Std dev vcc_3v3 | 0.012 V | 0.028 V |
| Cycle time | 42 s | 55 s |

This table tells a story: Site B has lower yield, lower average voltage, higher variance, and longer cycle time. The higher variance suggests a fixture or process control issue at the CM.

## Cross-Site Quality Governance

### Set Consistent Limits

Use the same measurement limits at every site. TofuPilot enforces limits at the procedure level, so a unit that passes at Site A will also pass at Site B (assuming the measurements are accurate).

### Monitor Site-to-Site Differences

Check these metrics regularly across sites:

1. **FPY gap**: If one site's yield is consistently lower, investigate
2. **Measurement distribution overlap**: Measurement histograms from different sites should overlap. If they don't, the sites are producing or testing differently
3. **Failure mode differences**: Different top failure modes at different sites suggest different process issues
4. **Cycle time variation**: Significantly different cycle times may indicate different test coverage or equipment issues

### Handle CM Data Integration

Contract manufacturers often use their own test systems. Options for getting their data into TofuPilot:

1. **Direct integration**: CM installs TofuPilot client on their test stations. Cleanest solution, real-time data.
2. **Batch upload**: CM exports CSV/JSON files, you upload via API. Works when the CM can't install software on their stations.
3. **API relay**: CM pushes data to an intermediary that forwards to TofuPilot. Good for CMs with existing data systems.

```python filename="cm_batch_upload.py"
import csv
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Upload CM data from their export file
with open("cm_test_results.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        client.create_run(
            procedure_id="FINAL-FUNCTIONAL-V3",
            unit_under_test={"serial_number": row["serial"]},
            run_passed=row["result"] == "PASS",
            steps=[{
                "name": "Power Rail Check",
                "step_type": "measurement",
                "status": float(row["vcc_3v3"]) >= 3.25,
                "measurements": [{
                    "name": "vcc_3v3",
                    "value": float(row["vcc_3v3"]),
                    "unit": "V",
                    "limit_low": 3.25,
                    "limit_high": 3.35,
                }],
            }],
        )
```

## Fleet Telemetry Across Sites

For companies deploying products globally (robotics fleets, industrial equipment, medical devices), TofuPilot provides a single view of test data regardless of where the unit was tested.

Search by serial number to see a unit's full history: manufactured and tested at Site A, field-tested at the customer's location, returned and retested at Site B. Every measurement, every result, every timestamp in one place.

This is especially valuable for warranty analysis. When a unit comes back for repair, you can compare its original production test data with its current state to understand what degraded.


### Battery Cell Grading and Binning
URL: https://www.tofupilot.com/guides/cell-grading-and-binning-with-tofupilot
Learn how to grade and bin battery cells based on test data using TofuPilot for capacity matching, impedance sorting, and production traceability.

A battery pack is only as good as its weakest cell. Cell grading sorts production cells by measured performance so you can build packs from matched cells. TofuPilot stores every cell's test data and lets you query, filter, and grade cells based on actual measurements.

## Why Cell Grading Matters

Cells from the same production line have natural variation. Capacity varies by 2-5%. Impedance varies by 10-20%. If you build a pack from unmatched cells:

- The weakest cell limits pack capacity
- Cells age at different rates, causing imbalance
- BMS has to work harder to keep cells balanced
- Pack lifetime is shorter than it needs to be

Grading cells before assembly ensures packs perform consistently and last longer.

## Grading Criteria

| Parameter | How it's measured | Why it matters |
|-----------|------------------|---------------|
| Capacity (Ah) | Full charge/discharge cycle | Determines pack energy |
| Internal impedance (mohm) | AC impedance at 1kHz | Affects power delivery and heat |
| Open-circuit voltage (V) | Rest voltage after formation | Indicates state of charge consistency |
| Self-discharge rate (mV/day) | Voltage drop over 7-14 day rest | Catches internal micro-shorts |
| Weight (g) | Scale measurement | Catches underfilled or overfilled cells |

## Setting Up Cell Grading in TofuPilot

### Step 1: Define the Grading Test Procedure

```python filename="cell_grading_test.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

def grade_cell(serial, capacity_ah, impedance_mohm, ocv_v, self_discharge_mv_day, weight_g):
    # Determine grade
    if capacity_ah >= 3.1 and impedance_mohm <= 30 and self_discharge_mv_day <= 0.5:
        grade = "A"
        passed = True
    elif capacity_ah >= 2.9 and impedance_mohm <= 40 and self_discharge_mv_day <= 1.0:
        grade = "B"
        passed = True
    elif capacity_ah >= 2.7 and impedance_mohm <= 50:
        grade = "C"
        passed = True
    else:
        grade = "REJECT"
        passed = False

    client.create_run(
        procedure_id="CELL-GRADING",
        unit_under_test={
            "serial_number": serial,
            "part_number": "CELL-21700-NMC",
        },
        run_passed=passed,
        steps=[{
            "name": "Capacity Test",
            "step_type": "measurement",
            "status": capacity_ah >= 2.7,
            "measurements": [
                {"name": "capacity_ah", "value": capacity_ah, "unit": "Ah", "limit_low": 2.7},
            ],
        }, {
            "name": "Impedance Test",
            "step_type": "measurement",
            "status": impedance_mohm <= 50,
            "measurements": [
                {"name": "impedance_mohm", "value": impedance_mohm, "unit": "mohm", "limit_high": 50},
            ],
        }, {
            "name": "Self-Discharge",
            "step_type": "measurement",
            "status": self_discharge_mv_day <= 1.0,
            "measurements": [
                {"name": "self_discharge_mv_day", "value": self_discharge_mv_day, "unit": "mV/day", "limit_high": 1.0},
                {"name": "ocv_v", "value": ocv_v, "unit": "V", "limit_low": 3.5, "limit_high": 4.2},
            ],
        }, {
            "name": "Physical",
            "step_type": "measurement",
            "status": 68.0 <= weight_g <= 72.0,
            "measurements": [
                {"name": "weight_g", "value": weight_g, "unit": "g", "limit_low": 68.0, "limit_high": 72.0},
                {"name": "grade", "value": grade, "unit": ""},
            ],
        }],
    )
    return grade
```

### Step 2: Analyze Grade Distribution

After grading a production batch, TofuPilot's dashboard shows the distribution:

| Grade | Count | Percentage | Capacity range | Impedance range |
|-------|-------|-----------|----------------|-----------------|
| A | 850 | 85% | 3.10-3.25 Ah | 22-30 mohm |
| B | 120 | 12% | 2.90-3.09 Ah | 31-40 mohm |
| C | 20 | 2% | 2.70-2.89 Ah | 41-50 mohm |
| REJECT | 10 | 1% | < 2.70 Ah | > 50 mohm |

Track this distribution over time. If Grade A yield drops from 85% to 75%, the manufacturing process is drifting.

### Step 3: Pack Assembly with Matched Cells

Query TofuPilot for Grade A cells with capacity within a tight window for pack assembly.

```python filename="pack_matching.py"
from tofupilot import TofuPilotClient

client = TofuPilotClient()

# Get all Grade A cells from the latest batch
runs = client.get_runs(
    procedure_id="CELL-GRADING",
    run_passed=True,
    limit=1000,
)

# Filter for Grade A cells within a 50mAh capacity window
grade_a_cells = []
for run in runs:
    for step in run.get("steps", []):
        for m in step.get("measurements", []):
            if m["name"] == "capacity_ah" and m["value"] >= 3.1:
                grade_a_cells.append({
                    "serial": run["unit_under_test"]["serial_number"],
                    "capacity": m["value"],
                })

# Sort by capacity and group into matched sets
grade_a_cells.sort(key=lambda c: c["capacity"])

# Create packs of 12 cells with capacity spread < 50mAh
pack_size = 12
for i in range(0, len(grade_a_cells) - pack_size + 1, pack_size):
    pack = grade_a_cells[i:i + pack_size]
    spread = pack[-1]["capacity"] - pack[0]["capacity"]
    if spread <= 0.05:
        print(f"Pack: {[c['serial'] for c in pack]}, spread: {spread*1000:.0f} mAh")
```

## Self-Discharge Screening

Self-discharge is the most important safety screen. Cells with elevated self-discharge rates may have internal micro-shorts that can lead to thermal events.

The test requires a long rest period (7-14 days), which makes it the bottleneck in cell production. Track self-discharge data in TofuPilot to:

- Set data-driven pass/fail thresholds based on production distributions
- Identify cells that need extended screening
- Correlate self-discharge with other parameters (formation data, impedance)

## Traceability from Cell to Pack

When a pack fails in the field, trace back to the individual cells:

1. Look up the pack serial in TofuPilot
2. Find which cells were assembled into that pack
3. Pull each cell's grading data
4. Check if any cell was marginal at grading time

This traceability is required by most automotive OEMs and is becoming standard across the battery industry.


### How to Build a Test Panel GUI in Python
URL: https://www.tofupilot.com/guides/how-to-build-a-python-test-panel-gui-for-instrument-control
Learn how to build a tkinter-based test panel GUI that controls instruments and runs OpenHTF tests with TofuPilot logging.

Building a custom operator panel lets your test technicians run tests without touching the command line. This guide shows how to create a tkinter-based GUI that controls instruments, collects serial numbers, and runs OpenHTF tests with TofuPilot logging.

## Prerequisites

- Python 3.8+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)
- A TofuPilot API key configured

## Step 1: Create the Main Panel Layout

Start with a basic tkinter window that has a serial number entry, instrument controls, and a test execution area.

```python filename="panel.py"
import tkinter as tk
from tkinter import ttk
import threading


class TestPanel:
    def __init__(self, root):
        self.root = root
        self.root.title("Test Panel")
        self.root.geometry("480x520")
        self.root.resizable(False, False)

        self.instrument_connected = False

        self._build_instrument_frame()
        self._build_serial_frame()
        self._build_control_frame()
        self._build_status_frame()

    def _build_instrument_frame(self):
        frame = ttk.LabelFrame(self.root, text="Instrument", padding=10)
        frame.pack(fill="x", padx=10, pady=(10, 5))

        self.connect_btn = ttk.Button(
            frame, text="Connect", command=self._connect_instrument
        )
        self.connect_btn.pack(side="left", padx=(0, 5))

        self.disconnect_btn = ttk.Button(
            frame, text="Disconnect", command=self._disconnect_instrument, state="disabled"
        )
        self.disconnect_btn.pack(side="left")

        self.instr_status = ttk.Label(frame, text="Disconnected", foreground="gray")
        self.instr_status.pack(side="right")

    def _build_serial_frame(self):
        frame = ttk.LabelFrame(self.root, text="Unit Under Test", padding=10)
        frame.pack(fill="x", padx=10, pady=5)

        ttk.Label(frame, text="Serial Number:").pack(anchor="w")
        self.serial_var = tk.StringVar()
        self.serial_entry = ttk.Entry(frame, textvariable=self.serial_var, width=30)
        self.serial_entry.pack(fill="x", pady=(2, 0))

    def _build_control_frame(self):
        frame = ttk.Frame(self.root, padding=10)
        frame.pack(fill="x", padx=10)

        self.start_btn = ttk.Button(
            frame, text="Start Test", command=self._start_test, state="disabled"
        )
        self.start_btn.pack(fill="x")

    def _build_status_frame(self):
        frame = ttk.LabelFrame(self.root, text="Status", padding=10)
        frame.pack(fill="both", expand=True, padx=10, pady=(5, 10))

        self.result_label = ttk.Label(
            frame, text="Idle", font=("Helvetica", 18, "bold"), anchor="center"
        )
        self.result_label.pack(pady=(0, 10))

        self.log_text = tk.Text(frame, height=10, state="disabled", font=("Courier", 10))
        self.log_text.pack(fill="both", expand=True)

    def _log(self, message):
        self.log_text.configure(state="normal")
        self.log_text.insert("end", message + "\n")
        self.log_text.see("end")
        self.log_text.configure(state="disabled")

    def _set_result(self, text, color):
        self.result_label.configure(text=text, foreground=color)

    def _connect_instrument(self):
        self._log("Connecting to instrument...")
        self.instrument_connected = True
        self.instr_status.configure(text="Connected", foreground="green")
        self.connect_btn.configure(state="disabled")
        self.disconnect_btn.configure(state="normal")
        self.start_btn.configure(state="normal")
        self._log("Instrument connected.")

    def _disconnect_instrument(self):
        self._log("Disconnecting instrument...")
        self.instrument_connected = False
        self.instr_status.configure(text="Disconnected", foreground="gray")
        self.connect_btn.configure(state="normal")
        self.disconnect_btn.configure(state="disabled")
        self.start_btn.configure(state="disabled")
        self._log("Instrument disconnected.")

    def _start_test(self):
        serial = self.serial_var.get().strip()
        if not serial:
            self._log("Error: enter a serial number.")
            return

        self.start_btn.configure(state="disabled")
        self.serial_entry.configure(state="disabled")
        self._set_result("Running...", "orange")
        self._log(f"Starting test for {serial}...")

        thread = threading.Thread(target=self._run_test, args=(serial,), daemon=True)
        thread.start()

    def _run_test(self, serial):
        # Placeholder — see Step 4 for full implementation
        pass


if __name__ == "__main__":
    root = tk.Tk()
    app = TestPanel(root)
    root.mainloop()
```

This gives you a clean panel with instrument connect/disconnect buttons, a serial number field, and a status area that shows logs and pass/fail results.

## Step 2: Add an Instrument Plug

Create an OpenHTF plug that wraps your instrument communication. This example simulates a power supply, but you'd replace the internals with your actual SCPI or serial commands.

```python filename="instrument_plug.py"
import time
import openhtf as htf


class PowerSupplyPlug(htf.plugs.BasePlug):
    """Plug for controlling a bench power supply."""

    ADDRESS = "GPIB0::5::INSTR"

    def setUp(self):
        self._connected = False
        self._voltage = 0.0
        # Replace with real instrument connection (e.g., pyvisa)
        self._connected = True

    def set_voltage(self, voltage):
        if not self._connected:
            raise RuntimeError("Power supply not connected")
        self._voltage = voltage
        time.sleep(0.3)

    def measure_current(self):
        if not self._connected:
            raise RuntimeError("Power supply not connected")
        time.sleep(0.2)
        return 0.125

    def tearDown(self):
        if self._connected:
            self.set_voltage(0.0)
            self._connected = False
```

The `tearDown` method runs automatically when the test finishes, so the instrument always returns to a safe state.

## Step 3: Define OpenHTF Test Phases

Write test phases that use the instrument plug to take measurements. Note how plug injection uses the `@htf.plug` decorator.

```python filename="test_phases.py"
import openhtf as htf
from openhtf.util import units
from instrument_plug import PowerSupplyPlug


@htf.plug(psu=PowerSupplyPlug)
@htf.measures(
    htf.Measurement("supply_voltage_setpoint").with_units(units.VOLT),
    htf.Measurement("quiescent_current")
        .in_range(minimum=0.05, maximum=0.25)
        .with_units(units.AMPERE),
)
def phase_quiescent_current(test, psu):
    """Measure quiescent current at nominal voltage."""
    psu.set_voltage(3.3)
    test.measurements.supply_voltage_setpoint = 3.3

    current = psu.measure_current()
    test.measurements.quiescent_current = current


@htf.plug(psu=PowerSupplyPlug)
@htf.measures(
    htf.Measurement("overcurrent_detected").equals(True),
)
def phase_overcurrent_protection(test, psu):
    """Verify the DUT triggers overcurrent protection at high voltage."""
    psu.set_voltage(5.5)
    test.measurements.overcurrent_detected = True
```

Each phase focuses on one thing. The measurement validators (`.in_range`, `.equals`) tell OpenHTF what counts as a pass or fail.

## Step 4: Wire the GUI to OpenHTF Execution

Now connect the panel's `_run_test` method to actually execute the OpenHTF test with TofuPilot logging.

```python filename="panel_with_test.py"
import tkinter as tk
from tkinter import ttk
import threading
import openhtf as htf
from tofupilot.openhtf import upload
from test_phases import phase_quiescent_current, phase_overcurrent_protection


class TestPanel:
    def __init__(self, root):
        self.root = root
        self.root.title("Test Panel")
        self.root.geometry("480x520")
        self.root.resizable(False, False)

        self.instrument_connected = False

        self._build_instrument_frame()
        self._build_serial_frame()
        self._build_control_frame()
        self._build_status_frame()

    def _build_instrument_frame(self):
        frame = ttk.LabelFrame(self.root, text="Instrument", padding=10)
        frame.pack(fill="x", padx=10, pady=(10, 5))

        self.connect_btn = ttk.Button(
            frame, text="Connect", command=self._connect_instrument
        )
        self.connect_btn.pack(side="left", padx=(0, 5))

        self.disconnect_btn = ttk.Button(
            frame, text="Disconnect", command=self._disconnect_instrument, state="disabled"
        )
        self.disconnect_btn.pack(side="left")

        self.instr_status = ttk.Label(frame, text="Disconnected", foreground="gray")
        self.instr_status.pack(side="right")

    def _build_serial_frame(self):
        frame = ttk.LabelFrame(self.root, text="Unit Under Test", padding=10)
        frame.pack(fill="x", padx=10, pady=5)

        ttk.Label(frame, text="Serial Number:").pack(anchor="w")
        self.serial_var = tk.StringVar()
        self.serial_entry = ttk.Entry(frame, textvariable=self.serial_var, width=30)
        self.serial_entry.pack(fill="x", pady=(2, 0))

    def _build_control_frame(self):
        frame = ttk.Frame(self.root, padding=10)
        frame.pack(fill="x", padx=10)

        self.start_btn = ttk.Button(
            frame, text="Start Test", command=self._start_test, state="disabled"
        )
        self.start_btn.pack(fill="x")

    def _build_status_frame(self):
        frame = ttk.LabelFrame(self.root, text="Status", padding=10)
        frame.pack(fill="both", expand=True, padx=10, pady=(5, 10))

        self.result_label = ttk.Label(
            frame, text="Idle", font=("Helvetica", 18, "bold"), anchor="center"
        )
        self.result_label.pack(pady=(0, 10))

        self.log_text = tk.Text(frame, height=10, state="disabled", font=("Courier", 10))
        self.log_text.pack(fill="both", expand=True)

    def _log(self, message):
        self.log_text.configure(state="normal")
        self.log_text.insert("end", message + "\n")
        self.log_text.see("end")
        self.log_text.configure(state="disabled")

    def _set_result(self, text, color):
        self.result_label.configure(text=text, foreground=color)

    def _connect_instrument(self):
        self._log("Connecting to instrument...")
        self.instrument_connected = True
        self.instr_status.configure(text="Connected", foreground="green")
        self.connect_btn.configure(state="disabled")
        self.disconnect_btn.configure(state="normal")
        self.start_btn.configure(state="normal")
        self._log("Instrument connected.")

    def _disconnect_instrument(self):
        self._log("Disconnecting instrument...")
        self.instrument_connected = False
        self.instr_status.configure(text="Disconnected", foreground="gray")
        self.connect_btn.configure(state="normal")
        self.disconnect_btn.configure(state="disabled")
        self.start_btn.configure(state="disabled")
        self._log("Instrument disconnected.")

    def _start_test(self):
        serial = self.serial_var.get().strip()
        if not serial:
            self._log("Error: enter a serial number.")
            return

        self.start_btn.configure(state="disabled")
        self.serial_entry.configure(state="disabled")
        self._set_result("Running...", "orange")
        self._log(f"Starting test for {serial}...")

        thread = threading.Thread(target=self._run_test, args=(serial,), daemon=True)
        thread.start()

    def _run_test(self, serial):
        try:
            test = htf.Test(
                phase_quiescent_current,
                phase_overcurrent_protection,
                procedure_id="PCB-FUNC-001",
                part_number="PCB-2400",
            )

            test.add_output_callbacks(upload())
            test_record = test.execute(test_start=lambda: serial)

            outcome = test_record.outcome.name

            if outcome == "PASS":
                self.root.after(0, self._set_result, "PASS", "green")
                self.root.after(0, self._log, f"Test PASSED for {serial}.")
            else:
                self.root.after(0, self._set_result, "FAIL", "red")
                self.root.after(0, self._log, f"Test FAILED for {serial}.")

        except Exception as e:
            self.root.after(0, self._set_result, "ERROR", "red")
            self.root.after(0, self._log, f"Error: {e}")

        finally:
            self.root.after(0, self._reset_controls)

    def _reset_controls(self):
        self.serial_entry.configure(state="normal")
        self.serial_var.set("")
        if self.instrument_connected:
            self.start_btn.configure(state="normal")


if __name__ == "__main__":
    root = tk.Tk()
    app = TestPanel(root)
    root.mainloop()
```

A few things to note here:

- `test.execute` runs synchronously in the background thread, so the GUI stays responsive.
- `root.after(0, ...)` schedules GUI updates on the main thread. Tkinter isn't thread-safe, so you can't call widget methods directly from a worker thread.
- `TofuPilot(test)` wraps execution and automatically uploads the test record, including all measurements and outcomes.

### How to Improve First Pass Yield
URL: https://www.tofupilot.com/guides/how-to-improve-first-pass-yield-in-electronics-manufacturing-with-tofupilot
Actionable strategies to improve first pass yield (FPY) in electronics manufacturing, with root cause analysis, Pareto charts, limit refinement, and TofuPilot.

Low first pass yield costs more than scrap. It means retest cycles, debug time, and late shipments. This guide covers practical strategies to find what's failing, why it's failing, and how to fix it using production data from TofuPilot.

## Prerequisites

- A TofuPilot account with test runs already flowing in
- OpenHTF test scripts with measurements and limits defined
- At least a few hundred runs for meaningful analysis

## Step 1: Understand Where You're Losing Yield

Before fixing anything, you need to know what's actually failing. Most teams have a gut feeling about their worst tests, but the data usually tells a different story.

TofuPilot's Analytics tab shows your FPY over time, broken down by procedure. Start there. Look for:

- **Procedures with FPY below 95%**: these are your top targets
- **FPY drops that correlate with a date**: usually a process change, new component lot, or test script update
- **Stations with lower FPY than others running the same procedure**: points to fixturing or calibration issues

The Pareto chart on the Analytics tab ranks failure modes by frequency. This is where you focus. Fixing the top two or three failure modes typically recovers most of your lost yield.

## Step 2: Run Root Cause Analysis on Top Failures

Once you've identified your worst-performing measurements, dig into the data. There are three common root cause categories.

### Test limit issues

Limits that are too tight cause false failures. Limits that are too loose let bad units through. Both hurt yield, just in different ways.

TofuPilot's control charts show measurement trends with 3-sigma limits automatically. Use the Cpk view to identify measurements with poor process capability. A Cpk below 1.0 means your process spread is wider than your spec limits, and you'll keep failing units until you fix either the process or the limits.

| Cpk Range | Interpretation | Action |
|-----------|---------------|--------|
| < 1.0 | Process not capable | Widen limits or improve process |
| 1.0 - 1.33 | Barely capable | Monitor closely, plan improvement |
| 1.33 - 1.67 | Capable | Acceptable for most production |
| > 1.67 | Highly capable | Consider tightening limits |

### Process issues

If a measurement's distribution shifts over time or between stations, that's a process issue, not a limit issue. Common causes:

- Component lot variation (especially passives and connectors)
- Fixture wear or contamination
- Environmental changes (temperature, humidity in the test area)
- Operator-dependent steps with inconsistent execution

### Test script issues

Sometimes the test itself is the problem. Flaky measurements, missing settling time, or incorrect instrument configuration can all show up as yield loss. Look for measurements with bimodal distributions or high variance that doesn't correlate with the DUT.

## Step 3: Refine Your Measurement Limits

Good limits come from a combination of the component datasheet, your design margins, and actual production data. Here's the workflow.

### Start with datasheet limits

Your initial limits should reflect the design intent. Pull minimum and maximum values from the relevant datasheets and your design specs.

```python filename="tests/power_rail_test.py"
import openhtf as htf
from openhtf.util import units

@htf.measures(
    htf.Measurement("vdd_3v3")
        .with_units(units.VOLT)
        .in_range(minimum=3.135, maximum=3.465)  # 3.3V +/- 5% from regulator datasheet
        .doc("Main 3.3V rail, measured at C12"),
    htf.Measurement("vdd_1v8")
        .with_units(units.VOLT)
        .in_range(minimum=1.746, maximum=1.854)  # 1.8V +/- 3% from LDO datasheet
        .doc("Core 1.8V rail, measured at C45"),
)
def measure_power_rails(test):
    # Instrument reads go here
    test.measurements.vdd_3v3 = read_voltage("3V3_TP")
    test.measurements.vdd_1v8 = read_voltage("1V8_TP")
```

### Tighten with production data

After collecting a few hundred units, check the actual distribution in TofuPilot's control charts. If your measurements cluster tightly around the nominal with Cpk > 1.67, you've got room to tighten. Tighter limits catch marginal units before they become field returns.

### Add marginal limits for early warning

OpenHTF supports marginal limits that flag units as "marginal pass" without outright failing them. This gives you early warning when a measurement is drifting toward spec limits.

```python filename="tests/power_rail_test.py"
import openhtf as htf
from openhtf.util import units

@htf.measures(
    htf.Measurement("vdd_3v3")
        .with_units(units.VOLT)
        .in_range(
            minimum=3.135, maximum=3.465,
            marginal_minimum=3.168, marginal_maximum=3.432,  # Inner 2% band
        )
        .doc("Main 3.3V rail with marginal detection"),
)
def measure_power_rails(test):
    test.measurements.vdd_3v3 = read_voltage("3V3_TP")
```

Marginal units pass but get flagged in TofuPilot. When marginal rates climb, you know a process shift is underway before it starts causing hard failures.

## Step 4: Improve Test Structure for Reliable Results

Flaky tests kill yield numbers. A few structural practices make a big difference.

### Separate measurements cleanly

Each measurement should test one thing. If a single phase measures five different voltages, a failure in any one of them makes it harder to identify the root cause.

```python filename="tests/board_test.py"
import openhtf as htf
from openhtf.util import units
import time

# Good: one measurement per phase, clear naming
@htf.measures(
    htf.Measurement("supply_current_idle")
        .with_units(units.AMPERE)
        .in_range(minimum=0.010, maximum=0.025)
        .doc("Board idle current draw at 3.3V input"),
)
def measure_idle_current(test):
    set_load("idle")
    time.sleep(0.5)  # Settling time matters
    test.measurements.supply_current_idle = read_current("ISENSE")


@htf.measures(
    htf.Measurement("supply_current_active")
        .with_units(units.AMPERE)
        .in_range(minimum=0.080, maximum=0.150)
        .doc("Board current draw during active processing"),
)
def measure_active_current(test):
    set_load("active")
    time.sleep(1.0)  # Active mode needs longer settling
    test.measurements.supply_current_active = read_current("ISENSE")
```

### Add settling time and retries for noisy measurements

If a measurement is inherently noisy (RF power, current draw during transitions), average multiple readings or add a retry with a short delay. Don't just hope for the best.

```python filename="tests/rf_test.py"
import openhtf as htf
import time

@htf.measures(
    htf.Measurement("tx_power_dbm")
        .in_range(minimum=18.0, maximum=22.0)
        .doc("Transmit power at 2.4 GHz, averaged over 5 readings"),
)
def measure_tx_power(test):
    readings = []
    for _ in range(5):
        readings.append(read_rf_power("TX_OUT"))
        time.sleep(0.1)
    test.measurements.tx_power_dbm = sum(readings) / len(readings)
```

### Use descriptive measurement names

Names like `m1`, `test_3`, or `voltage` make it impossible to do root cause analysis at scale. Use names that describe what's being measured and where.

| Bad Name | Good Name |
|----------|-----------|
| `voltage_1` | `vdd_3v3_at_c12` |
| `test_pass` | `wifi_association_2g4` |
| `current` | `supply_current_idle` |
| `temp` | `pcb_temp_post_burn_in` |

## Step 5: Build a Continuous Improvement Workflow

Improving FPY isn't a one-time project. It's a weekly practice.

### Weekly review process

1. **Check FPY trend** in TofuPilot's Analytics tab. Is it improving, flat, or declining?
2. **Review the Pareto chart** for top failure modes. Have the top failures changed since last week?
3. **Inspect control charts** for any measurements showing drift or increased variance
4. **Update limits** if production data shows Cpk values that justify tightening or loosening
5. **Track marginal rates** as a leading indicator of future yield loss

### When to act vs. when to monitor

| Signal | Action |
|--------|--------|
| FPY drops > 2% in a week | Investigate immediately |
| New failure mode enters top 3 | Root cause within 48 hours |
| Cpk drops below 1.33 | Plan process improvement |
| Marginal rate increases > 50% | Investigate component lots or fixture |
| FPY stable above target | Monitor weekly, no action needed |

## Summary

Improving FPY follows a repeatable pattern: identify the top failures with Pareto analysis, run root cause analysis using TofuPilot's control charts and Cpk data, refine limits based on production distributions, and monitor weekly for regressions. The teams that sustain high yield are the ones that treat this as a continuous loop, not a one-time fix.

### How to Build Manufacturing Test KPIs
URL: https://www.tofupilot.com/guides/how-to-build-kpi-dashboards-with-tofupilot
Learn how to structure your OpenHTF tests so TofuPilot automatically generates manufacturing KPI dashboards with FPY, throughput, Cpk, and failure analysis.

TofuPilot generates manufacturing KPI dashboards from your test data automatically. You don't need to build reports in Python or connect BI tools for standard metrics. Write well-structured tests, and the dashboards populate themselves.

## Key Manufacturing KPIs

These are the metrics that matter on the production floor.

**First Pass Yield (FPY)** is the percentage of units that pass all tests on the first attempt. It's the single best indicator of process health. TofuPilot calculates FPY per procedure, per station, and across your entire line.

**Throughput** measures units tested per hour or per shift. It tells you whether your line is hitting capacity targets. Bottlenecks show up as throughput drops on specific stations.

**Cpk** (Process Capability Index) quantifies how centered your measurements are within spec limits. A Cpk above 1.33 means your process has comfortable margin. Below 1.0 means you're producing out-of-spec parts.

**Cycle time** is how long each test takes. It's the denominator in throughput and the first thing to optimize when you need more capacity.

**Failure rate by step** breaks down where failures occur. A Pareto chart of failure modes tells you exactly where to invest engineering effort.

## Structuring Tests for Complete KPIs

Every KPI above comes from data your tests already produce. The key is structuring tests so TofuPilot can extract all of it cleanly.

```python filename="kpi_ready_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("boot_time")
    .in_range(maximum=3.0),
    htf.Measurement("firmware_version")
    .with_allowed_values("2.4.1", "2.4.2"),
)
def firmware_validation(test):
    """Validate firmware boots correctly."""
    test.measurements.boot_time = 1.8
    test.measurements.firmware_version = "2.4.1"

@htf.measures(
    htf.Measurement("battery_voltage")
    .with_units(units.VOLT)
    .in_range(minimum=3.6, maximum=4.2),
    htf.Measurement("charge_current")
    .in_range(minimum=0.450, maximum=0.550)
    .with_units(units.AMPERE),
    htf.Measurement("discharge_capacity")
    .in_range(minimum=2800),
)
def battery_test(test):
    """Test battery charging and capacity."""
    test.measurements.battery_voltage = 3.95
    test.measurements.charge_current = 0.502
    test.measurements.discharge_capacity = 3050

@htf.measures(
    htf.Measurement("touch_sensitivity_pct")
    .in_range(minimum=85.0),
    htf.Measurement("display_brightness")
    .in_range(minimum=400, maximum=600),
    htf.Measurement("pixel_defect_count")
    .in_range(maximum=0),
)
def display_and_touch_test(test):
    """Verify display and touchscreen performance."""
    test.measurements.touch_sensitivity_pct = 94.2
    test.measurements.display_brightness = 520
    test.measurements.pixel_defect_count = 0

@htf.measures(
    htf.Measurement("speaker_thd_pct")
    .in_range(maximum=1.0),
    htf.Measurement("microphone_snr")
    .in_range(minimum=60.0),
)
def audio_test(test):
    """Test speaker and microphone quality."""
    test.measurements.speaker_thd_pct = 0.4
    test.measurements.microphone_snr = 68.5

def main():
    test = htf.Test(
        firmware_validation,
        battery_test,
        display_and_touch_test,
        audio_test,
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "DEVICE-007")

if __name__ == "__main__":
    main()
```

This test gives TofuPilot everything it needs. Numeric measurements with limits feed Cpk calculations. Phase structure feeds step-level failure analysis. Timestamps feed throughput and cycle time. Pass/fail outcomes feed FPY.

## What Makes Tests KPI-Ready

Three things make the difference between tests that produce useful dashboards and tests that don't.

**Numeric measurements with limits.** Every measurement that has `in_range()` limits gets Cpk tracking, histograms, and control charts. Boolean pass/fail checks are fine for go/no-go tests, but they don't feed the analytics.

**Consistent phase names.** If you rename a phase, TofuPilot treats it as a new step. Keep names stable so historical comparisons work. Use descriptive names that make sense in a dashboard context.

**Unique serial numbers.** The DUT ID connects a unit's full test history across stations. Use real serial numbers, not placeholder values. This enables traceability and retest tracking.

## Viewing Dashboards in TofuPilot

TofuPilot's dashboard shows your KPIs in real time as test results upload. You'll find:

- FPY trends over time, filterable by procedure, station, or time range
- Throughput charts showing units per hour by station
- Measurement histograms and Cpk values for every numeric measurement
- Failure Pareto charts ranking failure modes by frequency
- Yield gauges for at-a-glance status

Custom dashboards let you combine widgets for specific views. Build a line manager dashboard with FPY and throughput, or an engineering dashboard focused on Cpk and failure analysis.

## Keeping Dashboards Accurate

Dashboards are only as good as the data behind them. Run every unit through the test, including units you already know will fail. Skipping known-bad units inflates your FPY.

Use separate procedures for retest vs. first test. This keeps FPY calculations clean while still tracking retest outcomes. TofuPilot handles both and shows them independently.

### What Is Cpk (Process Capability Index)
URL: https://www.tofupilot.com/guides/what-is-cpk-process-capability-index
Cpk measures how much room a process has inside its specification limits. How to calculate it, what the values mean, and what invalidates it.

Cpk answers one question: given how much a measurement varies and where it sits between its limits, how much room does the process actually have? It is the standard way to say whether a test is likely to keep passing, or whether it is one small drift away from scrapping boards.

## The Formula

Cpk is the worse of the two one-sided distances from the mean to each specification limit, measured in units of 3 standard deviations:

```
CPU  = (USL - mean) / (3σ)
CPL  = (mean - LSL) / (3σ)
Cpk  = min(CPU, CPL)
```

Taking the minimum is what makes Cpk useful. A process centred perfectly between wide limits and a process hugging one limit can have the same spread, and only Cpk distinguishes them.

## Worked Example

A 3.3 V rail with limits of 3.2 V and 3.4 V, from 200 production boards:

| Statistic | Value |
|-----------|-------|
| Mean | 3.31 V |
| Standard deviation | 0.018 V |
| LSL | 3.2 V |
| USL | 3.4 V |

```
CPU = (3.40 - 3.31) / (3 × 0.018) = 0.09 / 0.054 = 1.67
CPL = (3.31 - 3.20) / (3 × 0.018) = 0.11 / 0.054 = 2.04
Cpk = min(1.67, 2.04) = 1.67
```

The process is slightly high of centre, so the upper limit is the binding one. Cpk of 1.67 is comfortable.

## Cp vs Cpk

**Cp** ignores where the process sits and measures only whether the spread could fit:

```
Cp = (USL - LSL) / (6σ)
```

In the example above, Cp is `0.2 / 0.108 = 1.85`. The gap between Cp 1.85 and Cpk 1.67 is the cost of being off-centre. When Cp is much larger than Cpk, the fix is usually recentring rather than reducing variation, which is almost always the cheaper intervention.

See [Cp, potential process capability](/guides/what-is-cp-potential-process-capability).

## Interpreting the Number

| Cpk | Defects (approx, centred) | Meaning |
|-----|---------------------------|---------|
| 0.67 | ~4.5 % | Failing constantly |
| 1.00 | ~0.27 % | Limits at exactly 3σ; marginal |
| 1.33 | ~63 ppm | Common minimum target |
| 1.67 | ~0.6 ppm | Comfortable |
| 2.00 | ~2 ppb | Six-sigma capable |

Most electronics manufacturers target Cpk of at least 1.33 on measurements that matter. Below 1.0, the test fails often enough that operators start treating failures as noise, which is its own problem.

A very high Cpk is worth a question rather than celebration: either the process is genuinely excellent, or the limits are so loose the test cannot fail, in which case it is spending cycle time without protecting anything.

## What Invalidates Cpk

Cpk is easy to compute and easy to compute meaninglessly. Four conditions have to hold.

**The process must be stable.** Capability on an out-of-control process averages two or more distributions into one number describing none of them. Check the [control chart](/guides/how-to-use-control-charts-for-tests-with-tofupilot) first; if there are shifts or trends, fix those before calculating capability.

**The data should be roughly normal.** The defect-rate column above assumes normality. Heavily skewed or bimodal data makes those ppm figures fiction. Bimodal usually means two populations mixed, such as two fixtures or two component lots, and the fix is to separate them.

**The measurement must have resolution.** If the instrument resolves to 0.01 V and the real spread is 0.005 V, the computed sigma is an artefact. A sigma of zero is a resolution problem, never perfection.

**The sample must be adequate.** Thirty units is a rough minimum; fewer makes the sigma estimate unstable enough that the Cpk is noise.

## Getting Data That Supports Cpk

Cpk requires the measured values. A pass/fail record cannot produce it at any sample size, which is the practical reason to store numbers rather than verdicts.

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.dc_voltage()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def dc_voltage(self) -> float:
        return 3.31  # Replace with a real instrument read
```

Each run stores the value with its limits, so Cpk per measurement is computed from the data rather than a separate study.

Two habits that keep the number meaningful:

**Keep measurement names stable.** Renaming `Rail 3V3` splits the distribution in two and the capability history with it.

**Version the procedure.** A limit change alters Cpk with no process change at all. Without the `version` on each run, that looks like a process event.

## Calculating It Yourself

```python filename="analysis/capability.py"
import statistics


def cpk(values: list[float], lsl: float, usl: float) -> float:
    """Process capability index for a two-sided specification."""
    if len(values) < 30:
        raise ValueError("Sample too small for a stable sigma estimate")

    mean = statistics.mean(values)
    sigma = statistics.stdev(values)
    if sigma == 0:
        raise ValueError("Zero variation; check measurement resolution")

    cpu = (usl - mean) / (3 * sigma)
    cpl = (mean - lsl) / (3 * sigma)
    return min(cpu, cpl)


def cp(values: list[float], lsl: float, usl: float) -> float:
    """Potential capability, ignoring centring."""
    sigma = statistics.stdev(values)
    return (usl - lsl) / (6 * sigma)
```

## Acting on a Low Cpk

| Symptom | Likely cause | Action |
|---------|--------------|--------|
| Cp good, Cpk poor | Process off-centre | Recentre; usually the cheapest fix |
| Cp and Cpk both poor | Too much variation | Reduce variation, or widen limits if the design allows |
| Cpk fell without a process change | Limits were tightened | Check the procedure version history |
| Bimodal distribution | Two populations mixed | Split by station, fixture or lot and recompute |
| Sigma near zero | Instrument resolution too coarse | Improve the measurement first |

Widening limits is a legitimate response only when the design genuinely tolerates it. Widening them because the process cannot hold them is how marginal product ships.

## One-Sided Measurements

Leakage current has a maximum and no minimum; insulation resistance the reverse. Cpk is undefined for those, and the correct indices are CPU and CPL alone. Do not invent a second limit to make a Cpk appear. See [CPL and CPU](/guides/what-is-cpl-and-cpu-one-sided-capability).

## Short Term vs Long Term

Cpk conventionally uses short-term, within-subgroup variation. Ppk uses the overall long-term standard deviation. Ppk is usually lower, because it includes the drift between subgroups that Cpk excludes. See [Cpk vs Ppk](/guides/cpk-vs-ppk-whats-the-difference).

## If You Already Run OpenHTF

```python filename="fct.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
def phase_power(test):
    test.measurements.rail_3v3 = 3.31


test = htf.Test(phase_power, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Computing Cpk on an unstable process | A number describing no real distribution | Check the control chart first |
| Recording pass/fail only | Cpk cannot be computed at all | Store the measured values |
| Ignoring bimodal data | Capability is an average of two populations | Split by station, fixture or lot |
| Treating sigma of zero as perfect | Resolution artefact read as excellence | Check instrument resolution |
| Reporting Cpk for a one-sided limit | Requires a fabricated second limit | Use CPU or CPL |
| Widening limits to fix a low Cpk | Marginal product ships | Recentre or reduce variation first |

### What Is Ppk (Process Performance Index)
URL: https://www.tofupilot.com/guides/what-is-ppk-process-performance-index
Ppk measures long-term process performance using overall standard deviation. Learn how it differs from Cpk and how TofuPilot tracks both.

Ppk is the process performance index. It measures the same thing as Cpk (how well your process fits within spec limits) but uses the overall standard deviation instead of the sample standard deviation. This makes Ppk a long-term metric that captures all sources of variation: batch-to-batch, shift-to-shift, operator-to-operator, and environmental.

## The Formula

**Ppk = min((USL - X̄) / 3σo, (X̄ - LSL) / 3σo)**

Where:
- USL is the upper specification limit
- LSL is the lower specification limit
- X̄ is the process mean
- σo is the overall standard deviation (n divisor)

The only difference from Cpk is the standard deviation. Cpk uses σ (sample, n-1 divisor). Ppk uses σo (overall, n divisor).

## Sample vs Overall Standard Deviation

This distinction matters. The sample σ (n-1 divisor) provides an unbiased estimate of process variation and is used for short-term capability. The overall σo (n divisor) captures the total observed variation across the full dataset, including all between-run sources.

| Statistic | Divisor | Captures |
|-----------|---------|----------|
| Sample σ | n-1 | Short-term variation: instrument noise, part-to-part within a batch |
| Overall σo | n | All variation: batch shifts, operator differences, temperature changes, lot variation |

Overall σo is always equal to or larger than sample σ. So Ppk is always equal to or lower than Cpk.

## What the Gap Between Cpk and Ppk Tells You

When Cpk and Ppk are close, your process is consistent over time. The variation you see in a single batch is representative of the whole production run.

When Cpk is significantly higher than Ppk, something is changing between batches. Your process looks capable in any given short window, but over days or weeks, extra variation appears.

| Cpk | Ppk | Diagnosis |
|-----|-----|-----------|
| 1.5 | 1.4 | Process is stable. Short-term and long-term variation are similar. |
| 1.5 | 0.9 | Hidden variation between batches. Investigate lot changes, operator differences, or environmental shifts. |
| 0.8 | 0.8 | Process isn't capable even short-term. Fix fundamental variation first. |

## When to Use Ppk vs Cpk

Use Cpk to assess inherent process capability under controlled conditions. Use Ppk to assess real-world performance over time.

PPAP (Production Part Approval Process) in automotive typically requires Ppk >= 1.67 for initial process studies. Ongoing production monitoring uses Cpk >= 1.33.

## Reading Ppk in TofuPilot

Open the Process Control page, select a numeric measurement, and switch to the Capability tab. The second KPI row shows the Pp family: Ppk, Pp, Ppl, and Ppu in teal. The first row shows the Cp family in purple.

Compare the two rows at a glance. If the teal values are noticeably lower than the purple ones, you have between-batch variation to investigate.

Click any index to toggle its line on the daily trend chart. Tracking Ppk alongside Cpk over time reveals when long-term variation starts growing.

## Common Pitfalls

**Small datasets blur the distinction.** With fewer than 50 data points from a single batch, σ and σo converge. The Cpk/Ppk gap only becomes meaningful with data spanning multiple batches, shifts, or days.

**Don't compare Ppk across different time windows without context.** Ppk over 7 days will differ from Ppk over 90 days because more sources of variation enter the calculation.

### What Is Pp (Overall Process Performance)
URL: https://www.tofupilot.com/guides/what-is-pp-overall-process-performance
Pp measures long-term process spread relative to spec width using overall standard deviation. Learn the formula, compare Pp to Cp, and track it in TofuPilot.

Pp is the overall process performance index. It compares the specification window to the total process variation measured using the overall (population) standard deviation. Pp is to Ppk what Cp is to Cpk: it measures spread without accounting for centering.

## The Formula

**Pp = (USL - LSL) / 6σo**

Where:
- USL is the upper specification limit
- LSL is the lower specification limit
- σo is the overall (population) standard deviation

The structure is identical to Cp. The only difference is σo (overall) instead of σ (within-subgroup). This means Pp captures all variation: within-batch, between-batch, between-shift, and between-operator.

## Pp vs Cp

Cp uses the within-subgroup σ and answers: "If we only look at short-term variation, does the spec window have room?" Pp uses the overall σo and answers: "Over the full production run, does the spec window have room?"

| Index | Standard Deviation | Time Horizon | Question |
|-------|-------------------|--------------|----------|
| Cp | Within-subgroup σ | Short-term | Can this process fit within specs under controlled conditions? |
| Pp | Overall σo | Long-term | Does this process fit within specs over real production? |

When Pp is close to Cp, the process is stable over time. When Pp is noticeably lower than Cp, there's hidden variation between production runs that inflates the overall spread.

## What Pp Values Mean

| Pp | Interpretation |
|----|---------------|
| < 1.0 | Long-term variation exceeds spec window. Process can't consistently meet specs over time. |
| 1.0 | Long-term variation exactly fills the spec window. |
| 1.33 | Process uses 75% of the spec window long-term. Standard target. |
| > 1.67 | Process has significant long-term margin. |

## Reading Pp in TofuPilot

Open the Process Control page, select a numeric measurement, and switch to the Capability tab. Pp appears in the second (teal) KPI row alongside Ppk, Ppl, and Ppu. The first (purple) row shows the Cp family.

If Pp is lower than Cp, your process has more variation over time than within individual batches. That points to batch-level or environmental sources of variation worth investigating.

Like all indices on the Capability tab, click Pp to toggle its trend line on the daily chart. Watching Pp over weeks helps you see whether long-term variation is growing or shrinking.

## Pp Requires Both Spec Limits

Like Cp, Pp needs both USL and LSL. For one-sided specifications, TofuPilot computes Ppu (upper only) or Ppl (lower only) instead.

## When Pp Matters Most

Pp is particularly useful during initial process validation (PPAP, process qualification). It tells you whether the process can sustain capability over a meaningful production window, not just during an optimized short run. Automotive PPAP studies typically require Pp >= 1.67 before granting production approval.

### What Is Ppl and Ppu (One-Sided Performance)
URL: https://www.tofupilot.com/guides/what-is-ppl-and-ppu-one-sided-performance
Ppl and Ppu measure long-term process performance against individual spec limits. Learn the formulas, how they relate to Cpl/Cpu, and track them in TofuPilot.

Ppl and Ppu are one-sided performance indices. They measure the distance from the process mean to each spec limit using the overall standard deviation. Ppl covers the lower limit, Ppu covers the upper. Together they form Ppk: Ppk = min(Ppl, Ppu).

## The Formulas

**Ppu = (USL - X̄) / 3σo**

**Ppl = (X̄ - LSL) / 3σo**

Where:
- USL is the upper specification limit
- LSL is the lower specification limit
- X̄ is the process mean
- σo is the overall standard deviation (n divisor)

These are identical to Cpu and Cpl, except they use σo (overall, n divisor) instead of σ (sample, n-1 divisor). This makes Ppl and Ppu long-term metrics that include all sources of variation.

## Ppl/Ppu vs Cpl/Cpu

| Index Pair | Standard Deviation | What It Captures |
|-----------|-------------------|-----------------|
| Cpl / Cpu | Sample σ (n-1 divisor) | Short-term margin to each spec limit |
| Ppl / Ppu | Overall σo (n divisor) | Long-term margin to each spec limit |

If Cpu is 1.5 but Ppu is 0.9, the process has enough short-term margin on the upper side but loses it over time. Something between batches is pushing the distribution toward USL. Conversely, if Cpl and Ppl are similar, the lower-side margin is stable.

## Directional Drift Detection

The real value of one-sided indices (both Cp and Pp families) is diagnosing which direction your process is drifting.

| Signal | Meaning | Action |
|--------|---------|--------|
| Ppu dropping, Ppl stable | Process mean drifting toward USL over time | Investigate upward drift: calibration, temperature, component lots |
| Ppl dropping, Ppu stable | Process mean drifting toward LSL over time | Investigate downward drift |
| Both dropping | Overall variation increasing | Reduce σ, don't adjust mean |
| Ppu < Cpu | Long-term upper margin worse than short-term | Between-batch variation biased upward |

## Reading Ppl and Ppu in TofuPilot

Open the Process Control page, select a numeric measurement, and switch to the Capability tab. The second (teal) KPI row shows Ppk, Pp, Ppl, and Ppu.

Compare the teal row to the purple row above it. If Ppu is significantly lower than Cpu, or Ppl is significantly lower than Cpl, the gap tells you where long-term variation is hurting you most.

Click any index to toggle it on the daily trend chart. Plotting Ppl and Ppu together over weeks reveals whether the process is drifting toward one spec limit over time.

## When One-Sided Specs Apply

For measurements with only an upper limit (leakage, noise, response time), TofuPilot shows Ppu and "--" for Ppl. For measurements with only a lower limit (signal strength, gain), TofuPilot shows Ppl and "--" for Ppu. This is the correct behavior. There's no opposing limit to measure against.

### Cpk vs Ppk: What's the Difference
URL: https://www.tofupilot.com/guides/cpk-vs-ppk-whats-the-difference
Cpk and Ppk both measure process capability, but use different standard deviations. Learn when to use each, what the gap reveals, and compare them in TofuPilot.

Cpk and Ppk answer the same question (can my process meet spec?) but over different time horizons. Cpk uses the sample standard deviation and reflects short-term capability. Ppk uses the overall standard deviation and reflects long-term performance. The gap between them tells you whether your process is stable over time.

## The Formulas Side by Side

| Index | Formula | Standard Deviation |
|-------|---------|-------------------|
| Cpk | min((USL - X̄) / 3σ, (X̄ - LSL) / 3σ) | Sample σ (n-1 divisor) |
| Ppk | min((USL - X̄) / 3σo, (X̄ - LSL) / 3σo) | Overall σo (n divisor) |

The formulas are identical except for σ. That one difference changes the meaning.

## Sample vs Overall Standard Deviation

The sample σ (n-1 divisor) provides an unbiased estimate of process variation. For individual measurements (one value per unit, which is the typical case in electronics testing), this captures short-term variation under controlled conditions.

The overall σo (n divisor) captures the total observed variation across the full dataset, including all between-run sources: batch-to-batch shifts, operator differences, temperature changes, component lot variation, fixture wear.

Overall σo is always >= sample σ. So Ppk is always <= Cpk. When they're equal, the process is perfectly stable over time. When Ppk is lower, there's hidden variation between production runs.

## What the Gap Tells You

| Cpk | Ppk | Gap | Diagnosis |
|-----|-----|-----|-----------|
| 1.5 | 1.45 | Small | Process is stable. Short-term and long-term variation are similar. No action needed. |
| 1.5 | 1.0 | Large | Process looks capable in any given batch but degrades over time. Investigate what changes between batches. |
| 1.5 | 0.7 | Very large | Significant between-batch variation. Common causes: component lot changes, operator training gaps, environmental shifts. |
| 0.8 | 0.8 | None | Process isn't capable even short-term. Fix fundamental variation before worrying about batch-to-batch stability. |

A large Cpk-Ppk gap is a signal to investigate sources of variation that don't show up within a single batch. Common culprits:

- **Component lot variation.** A new reel of resistors shifts the circuit's behavior slightly. Each lot is in spec, but the distribution center moves.
- **Operator differences.** Different fixture handling, different placement pressure, different ambient temperature from body heat near the DUT.
- **Environmental changes.** Morning vs afternoon temperature, seasonal humidity shifts, facility voltage fluctuations.
- **Fixture wear.** Pogo pin spring force degrading over thousands of insertions. Still passing, but adding variation.

## When to Use Each

| Context | Use | Why |
|---------|-----|-----|
| Ongoing production monitoring | Cpk | You want to know if the process is inherently capable under controlled conditions. |
| PPAP / initial process study | Ppk | Automotive standards require Ppk >= 1.67 to prove real-world performance. |
| Process qualification | Both | Compare them. A large gap means the process isn't ready for unsupervised production. |
| Root cause analysis | Both | If Cpk is fine but Ppk is low, the problem is between batches, not within them. |

## Industry Requirements

| Standard | Cpk Requirement | Ppk Requirement |
|----------|----------------|----------------|
| IATF 16949 (automotive) | >= 1.33 ongoing | >= 1.67 initial study |
| AS9100 (aerospace) | >= 1.33 production | >= 1.33 qualification |
| ISO 13485 (medical) | Per product risk | Per product risk |
| General electronics | >= 1.33 typical | >= 1.33 typical |

## Comparing Cpk and Ppk in TofuPilot

Open the Process Control page, select a numeric measurement, and switch to the Capability tab. The purple row shows Cpk (with Cp, Cpl, Cpu). The teal row shows Ppk (with Pp, Ppl, Ppu). You can compare both families at a glance.

Click any index to toggle it on the daily trend chart. Plotting Cpk and Ppk together over time is the clearest way to see whether the gap is growing, shrinking, or stable.

If you notice Ppk trending downward while Cpk stays flat, something external to the process is introducing variation. Filter by station, operator, or batch in the sidebar to isolate the source.

## Quick Decision Guide

**Cpk high, Ppk high:** Process is capable and stable. Monitor and maintain.

**Cpk high, Ppk low:** Process is capable within batches but unstable over time. Investigate between-batch variation.

**Cpk low, Ppk low:** Process isn't capable. Reduce variation first (σ), then worry about stability.

**Cpk low, Ppk high:** Unusual. Possible data issue or very small dataset. Verify your data filters.

### Data Sovereignty for Manufacturing Test Data
URL: https://www.tofupilot.com/guides/data-sovereignty-for-manufacturing-test-data
Learn why test data jurisdiction matters for manufacturing companies and how to protect process parameters and yield data from foreign surveillance laws.

Your test results, process parameters, and yield data are trade secrets. Where that data is hosted, and which government can legally compel access to it, determines your actual exposure. This guide breaks down the laws, the risks, and the practical options.

## What Manufacturing Test Data Contains

Test data isn't just pass/fail flags. A typical test database holds:

| Data Type | Examples | Why It's Sensitive |
|-----------|----------|--------------------|
| Process parameters | Voltage thresholds, torque values, calibration offsets | Reveals manufacturing know-how |
| Yield and quality metrics | FPY, Cpk, retest rates, failure Pareto | Exposes production maturity |
| Serial genealogy | Serial numbers, sub-assembly tracking, revision history | Full product traceability |
| Test sequences | Phase ordering, measurement limits, pass criteria | Core test IP |
| Station metadata | Station IDs, operator logs, throughput data | Factory operational intelligence |

For companies in defense, medtech, automotive, or aerospace, this data is regulated. For everyone else, it's still competitive intelligence you don't want a foreign government browsing.

## How US Surveillance Law Applies to SaaS

Three US laws matter for non-US companies using US-based SaaS:

| Law | What It Covers | Who It Compels | Key Detail |
|-----|---------------|----------------|------------|
| CLOUD Act (2018) | All data held by US providers, regardless of storage location | Any US company or company with sufficient US nexus | A US warrant reaches data in EU data centers if the provider is American |
| FISA Section 702 | Communications and data of non-US persons outside the US | US electronic communication service providers | No individual warrant required for non-US targets. Reauthorized April 2024 |
| National Security Letters | Subscriber metadata, transaction records | US companies | Issued by the FBI without a judge. Comes with a gag order by default |

The practical risk of your test data being targeted by US intelligence is low. These tools focus on counter-terrorism, espionage, and cyber threats. But the legal possibility exists, and your customers' security teams will ask about it.

## Why EU Data Centers Don't Solve This

A common misconception: "Our US vendor hosts in Frankfurt, so we're fine."

The CLOUD Act explicitly states that US legal process applies to data controlled by US companies regardless of where it's stored. A US company running servers in `eu-west-1` is still a US company.

The EU has tried to solve this with data transfer frameworks. All of them have been fragile:

| Framework | Years Active | What Happened |
|-----------|-------------|---------------|
| Safe Harbor | 2000-2015 | Invalidated by EU Court of Justice (Schrems I) |
| Privacy Shield | 2016-2020 | Invalidated by EU Court of Justice (Schrems II) |
| Data Privacy Framework | 2023-present | Active, but built on a US executive order that can be revoked. Challenge expected |

Relying on transfer frameworks means accepting the risk that your legal basis for data transfers could disappear overnight, as it did twice already.

## What Actually Protects Your Data

Protection comes in layers. No single measure is absolute.

| Layer | US SaaS on AWS | Non-US SaaS on US infra | Non-US SaaS, non-US infra | Self-Hosted |
|-------|---------------|------------------------|---------------------------|-------------|
| App provider compellable by US law | Yes | No | No | No |
| Infra provider compellable by US law | Yes | Yes | No | No |
| Encryption at rest | Varies | Yes | Yes | Yes |
| Full jurisdiction control | No | No | Depends on provider | Yes |

The key insight: using a non-US SaaS provider on US-owned infrastructure removes one attack vector (the app provider can't be compelled), but the infrastructure provider remains subject to US law. Only self-hosting or using non-US infrastructure removes both.

For most manufacturing companies, the combination of a non-US app provider with encryption at rest provides a strong practical posture. The infrastructure provider holds encrypted data but has no context about what it contains. For regulated industries or classified environments, self-hosting is the only option that provides full sovereignty.

## How TofuPilot Handles Data Sovereignty

TofuPilot SA is a Swiss-incorporated company with no US legal entity. US surveillance laws cannot compel TofuPilot to disclose customer data.

**Cloud deployment:**

- Database and file storage hosted in EU
- Data encrypted at rest and in transit
- TofuPilot operates under Swiss data protection law (nFADP), recognized as adequate by the EU
- No dependency on the EU-US Data Privacy Framework

**Self-hosted deployment:**

- Single Docker image, runs on your infrastructure
- Full air-gap support with no external dependencies
- Zero data leaves your network
- All features available, including analytics and traceability

| Concern | Cloud | Self-Hosted |
|---------|-------|-------------|
| TofuPilot compellable by US law | No | No |
| Infrastructure under US jurisdiction | Partially (US-owned infra providers) | No (your servers) |
| GDPR compliance | Yes, by corporate structure | Yes, fully on-premise |
| Air-gap support | No | Yes |

For companies that need to answer "where is our test data and who can access it" in a vendor security review, TofuPilot provides a clear answer at both the application and infrastructure level.

### What Is Cp (Potential Process Capability)
URL: https://www.tofupilot.com/guides/what-is-cp-potential-process-capability
Cp measures how much of the spec window your process variation uses, ignoring centering. Learn the formula, how Cp relates to Cpk, and read it in TofuPilot.

Cp is the potential process capability index. It compares the width of your specification window to the width of your process variation. Cp tells you the best Cpk you could achieve if your process were perfectly centered between the spec limits.

## The Formula

**Cp = (USL - LSL) / 6σ**

Where:
- USL is the upper specification limit
- LSL is the lower specification limit
- σ is the within-subgroup (sample) standard deviation

The numerator is the spec window. The denominator is 6σ, which covers 99.73% of a normal distribution. A Cp of 1.0 means your process variation exactly fills the spec window. Above 1.0, there's room to spare. Below 1.0, variation exceeds the spec range.

## What Cp Values Mean

| Cp | Interpretation |
|----|---------------|
| < 1.0 | Process variation exceeds spec window. Even if perfectly centered, some units fall outside limits. |
| 1.0 | Variation exactly matches spec width. No margin for drift. |
| 1.33 | Process uses 75% of the spec window. Standard production target. |
| 1.67 | Process uses 60% of the spec window. Safety-critical target. |
| 2.0 | Process uses 50% of the spec window. |

## Why Cp Alone Isn't Enough

Cp ignores where the process mean sits. A process with Cp = 1.5 could be perfectly centered (great) or shifted so far that units are failing on one side (bad). That's why Cp is always read alongside Cpk.

| Scenario | Cp | Cpk | What It Tells You |
|----------|-----|------|-------------------|
| Centered, capable | 1.5 | 1.5 | Ideal state. |
| Off-center, capable spread | 1.5 | 0.8 | Enough spread margin, but the mean drifted. Re-center it. |
| Centered, too much variation | 0.8 | 0.8 | Variation is the problem. Reduce σ. |

When Cp is good but Cpk is low, the fix is a centering adjustment (calibration, tool offset, recipe change). When Cp itself is low, you need to reduce fundamental process variation.

## Cp Requires Both Spec Limits

Cp needs USL and LSL. For one-sided specs (like maximum leakage current or maximum response time), Cp can't be calculated. TofuPilot shows "--" for Cp when only one limit is defined, and reports Cpu or Cpl instead.

## Reading Cp in TofuPilot

Open the Process Control page, select a numeric measurement, and switch to the Capability tab. Cp appears in the top KPI row alongside Cpk, Cpl, and Cpu. Click the Cp card to toggle its line on the daily trend chart.

Hover over the Cp value to see the formula and a plain-language explanation. If Cp is significantly higher than Cpk, the tooltip confirms that re-centering the process would improve capability.

```python filename="two_sided_limits.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("resistance")
        .in_range(minimum=95, maximum=105)
        .with_units(units.OHM),
)
def measure_resistance(test):
    test.measurements.resistance = 100.3

def main():
    test = htf.Test(measure_resistance)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "UNIT-0101")

if __name__ == "__main__":
    main()
```

Both `minimum` and `maximum` are set, so TofuPilot can compute Cp. If you only set one, you'll get a one-sided index instead.

### What Is SPC (Statistical Process Control)
URL: https://www.tofupilot.com/guides/what-is-spc-statistical-process-control
SPC uses production test data to separate normal variation from real problems. Learn the core tools, how they connect, and how TofuPilot automates SPC.

Statistical Process Control (SPC) is a method for monitoring and controlling a manufacturing process using data. Instead of inspecting quality at the end, SPC tracks measurements continuously to catch problems while they're still fixable. The core idea: every process has variation, and SPC helps you tell the difference between normal noise and something that actually changed.

## Common Cause vs Special Cause Variation

All manufacturing processes produce variation. SPC splits it into two types.

**Common cause variation** is inherent to the process. Instrument noise, slight material differences, ambient temperature fluctuations. It's random, stable, and predictable within a range. You can't eliminate it without fundamentally changing the process.

**Special cause variation** is something new. A calibration drifted, a component lot changed, a fixture pin bent. It's non-random and signals that the process changed. SPC tools are designed to detect this.

The goal isn't zero variation. It's knowing which type you're looking at so you take the right action. Adjusting a process in response to common cause variation (overreacting to noise) actually makes things worse.

## The SPC Toolkit

SPC has a small set of tools. Each answers a different question about your process.

| Tool | Question It Answers | What to Look For |
|------|-------------------|-----------------|
| Control chart | Is my process stable over time? | Points outside control limits, patterns, trends |
| Histogram | What does my distribution look like? | Shape, centering, spread relative to spec limits |
| Capability indices (Cp, Cpk) | Can my process meet spec short-term? | Values above 1.33 |
| Performance indices (Pp, Ppk) | Can my process meet spec long-term? | Values close to Cp/Cpk (stable process) |
| Cpk trend | Is capability improving or degrading? | Sustained direction changes |

These tools work together. A control chart tells you the process shifted. The histogram tells you how the distribution changed. Capability indices quantify whether you still meet spec.

## Control Charts

A control chart plots measurement values over time against three lines: a center line (process mean), an upper control limit (UCL), and a lower control limit (LCL). Control limits are calculated from your data at ±3σ from the mean.

Points inside the limits mean the process is behaving normally. Signals that something changed:

- A single point beyond UCL or LCL
- Seven consecutive points on one side of the center line
- Six consecutive points trending in one direction
- Alternating up-down patterns (possible measurement system issue)

Control limits are not spec limits. Spec limits (USL/LSL) come from your product requirements. Control limits come from your process data. A process can be in control but out of spec, or in spec but out of control. SPC catches the second case, which spec-only monitoring misses.

## Histograms

A histogram groups measurement values into bins and shows how many fall in each. Overlaid with a normal distribution curve and spec limits, it reveals the shape and position of your process.

| Shape | What It Means |
|-------|-------------|
| Normal (bell curve) | Process is predictable. Variation is random. |
| Skewed | One-sided drift. Check for a physical constraint or asymmetric tolerance. |
| Bimodal (two peaks) | Two populations mixed. Possible causes: two stations, two component lots, two operators. |
| Truncated (cut off at one end) | Parts at the limit are being screened out or the spec is acting as a wall. |
| Flat (uniform) | Process isn't controlled. Something is varying widely. |

TofuPilot renders histograms alongside control charts for each measurement. The normal curve overlay and spec limit lines appear automatically when limits are defined.

## Capability Indices

Capability indices reduce your process to a single number that says "can we meet spec?"

**Short-term (Cp family):** Uses sample σ (n-1 divisor).
- **Cp** = (USL - LSL) / 6σ. How much of the spec window the process uses.
- **Cpk** = min(Cpu, Cpl). How well the process fits within spec, accounting for centering.
- **Cpu** = (USL - X̄) / 3σ. Distance from mean to upper limit.
- **Cpl** = (X̄ - LSL) / 3σ. Distance from mean to lower limit.

**Long-term (Pp family):** Uses overall σo (n divisor). Same formulas, different σ.
- **Pp**, **Ppk**, **Ppu**, **Ppl**

When Cpk and Ppk are close, the process is stable. When Ppk is lower, there's hidden variation between batches. TofuPilot shows both families side by side in the Capability tab.

## Setting Up SPC in TofuPilot

SPC starts with well-defined measurements and limits. Write your tests with explicit spec limits so TofuPilot has the data it needs.

```python filename="spc_ready_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("supply_voltage")
        .in_range(minimum=4.75, maximum=5.25)
        .with_units(units.VOLT),
    htf.Measurement("current_draw")
        .in_range(minimum=0.090, maximum=0.110)
        .with_units(units.AMPERE),
    htf.Measurement("frequency")
        .in_range(minimum=999.5, maximum=1000.5)
        .with_units(units.HERTZ),
)
def functional_check(test):
    test.measurements.supply_voltage = 5.02
    test.measurements.current_draw = 0.098
    test.measurements.frequency = 1000.1

def main():
    test = htf.Test(functional_check)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "PCB-0001")

if __name__ == "__main__":
    main()
```

Once 30+ runs accumulate, open the Process Control page for your procedure. Select a measurement to see its control chart, histogram, and capability indices. The measurement overview ranks all measurements by failure count, fail rate, or Cpk so you can focus on the worst performers first.

## When to Act

| Signal | Meaning | Response |
|--------|---------|----------|
| Point beyond UCL/LCL | Special cause event | Investigate immediately. Check last lot change, calibration, operator. |
| 7 points same side of mean | Process has shifted | Check calibration, component lot, environmental change. |
| Cpk dropping over days | Capability degrading | Investigate drift source before it causes failures. |
| Cpk < 1.0 | Not capable | Reduce σ (process improvement) or widen spec (if justified). |
| Cpk >> Ppk | Short-term fine, long-term not | Between-batch variation. Investigate lot, operator, or environmental factors. |
| Cp >> Cpk | Capable but off-center | Re-center the process. Calibration or recipe adjustment. |
| Histogram bimodal | Two populations | Separate by station, operator, or lot. Fix the source. |

## What SPC Won't Tell You

SPC detects that something changed. It doesn't tell you why. When a control chart signals a special cause, you still need to investigate: check the operator log, review the component lot, inspect the fixture, look at environmental data. SPC narrows the search. Domain knowledge finishes the job.

### What Is Cpl and Cpu (One-Sided Capability)
URL: https://www.tofupilot.com/guides/what-is-cpl-and-cpu-one-sided-capability
CPL and CPU measure capability against a single specification limit. How to calculate them, when to use them, and how to record one-sided measurements.

Cpk assumes a measurement has both an upper and a lower specification limit. Plenty of real measurements have only one. Leakage current has a maximum and no minimum. Insulation resistance has a minimum and no maximum. CPL and CPU are the one-sided capability indices for exactly these cases.

## Definitions

**CPU** (capability, upper) measures how much room sits between the process and the upper specification limit:

```
CPU = (USL - mean) / (3σ)
```

**CPL** (capability, lower) measures the room above the lower specification limit:

```
CPL = (mean - LSL) / (3σ)
```

**Cpk** is simply the worse of the two when both limits exist:

```
Cpk = min(CPU, CPL)
```

So CPL and CPU are not a different tool from Cpk. They are its two halves, and for a one-sided measurement only one half is defined.

## Worked Example

A board's quiescent current has a maximum of 45 mA and no meaningful minimum. From 200 production units:

| Statistic | Value |
|-----------|-------|
| Mean | 38.0 mA |
| Standard deviation | 1.8 mA |
| USL | 45 mA |
| LSL | none |

```
CPU = (45 - 38.0) / (3 × 1.8) = 7.0 / 5.4 = 1.30
```

CPU of 1.30 means about 4.3 standard deviations of headroom to the limit. Reporting a Cpk here would be wrong, because there is no lower limit for the calculation to use.

## Interpreting the Number

| Value | Meaning | Typical response |
|-------|---------|------------------|
| < 1.00 | Process spread exceeds the available room | Escapes are routine; fix the process or the design |
| 1.00 - 1.33 | Marginal | Watch it; a small drift causes failures |
| 1.33 - 1.67 | Generally acceptable | Standard target for production |
| 1.67 - 2.00 | Comfortable | Consider whether the limit is too loose |
| > 2.00 | Very comfortable | The measurement may not be discriminating |

A very high CPU deserves a question rather than celebration: either the process is genuinely excellent, or the limit was set so loosely that the test cannot fail, in which case it is consuming cycle time without protecting anything.

## Where One-Sided Limits Show Up

| Measurement | Limit type | Index |
|-------------|-----------|-------|
| Leakage current | Maximum only | CPU |
| Quiescent current | Maximum only | CPU |
| Hipot leakage | Maximum only | CPU |
| Boot time | Maximum only | CPU |
| Harmonic distortion | Maximum only | CPU |
| BGA void percentage | Maximum only | CPU |
| Insulation resistance | Minimum only | CPL |
| Pull / bond strength | Minimum only | CPL |
| Battery capacity | Minimum only | CPL |
| Transmit power | Minimum only | CPL |
| RF sensitivity margin | Minimum only | CPL |

## Recording a One-Sided Measurement

The index can only be computed if the data says the limit is one-sided. In a procedure, that means declaring a single validator rather than inventing a second bound.

```yaml filename="procedure.yaml"
name: Board Power Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Quiescent Current
    python: phases.quiescent_current
    measurements:
      - name: Quiescent Current
        unit: mA
        description: Upper limit only; CPU applies
        validators:
          - operator: "<="
            expected_value: 45.0

  - name: Insulation Resistance
    python: phases.insulation
    measurements:
      - name: Insulation Resistance
        unit: "MΩ"
        description: Lower limit only; CPL applies
        validators:
          - operator: ">="
            expected_value: 100.0
```

```python filename="phases/quiescent_current.py"
def quiescent_current(measurements, dmm):
    measurements.quiescent_current = dmm.measure_dc_current_ma()
```

```python filename="phases/insulation.py"
def insulation(measurements, dmm):
    measurements.insulation_resistance = dmm.measure_insulation_mohm()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def measure_dc_current_ma(self) -> float:
        return 38.2  # Replace with a real instrument read

    def measure_insulation_mohm(self) -> float:
        return 480.0  # Replace with a real instrument read
```

This is the part teams most often get wrong. Adding a fabricated lower limit to a leakage measurement, so that a Cpk number appears, produces a figure that describes nothing. The honest record has one validator, and the analysis reports CPU.

## Calculating It From Your Data

```python filename="analysis/one_sided.py"
import statistics


def cpu(values: list[float], usl: float) -> float:
    """Capability against an upper specification limit."""
    sigma = statistics.stdev(values)
    if sigma == 0:
        raise ValueError("Zero variation; check the measurement resolution")
    return (usl - statistics.mean(values)) / (3 * sigma)


def cpl(values: list[float], lsl: float) -> float:
    """Capability against a lower specification limit."""
    sigma = statistics.stdev(values)
    if sigma == 0:
        raise ValueError("Zero variation; check the measurement resolution")
    return (statistics.mean(values) - lsl) / (3 * sigma)


currents = [38.1, 37.6, 39.0, 38.4, 37.9, 38.8, 38.2]
print(f"CPU = {cpu(currents, usl=45.0):.2f}")
```

Two cautions on the inputs:

**Zero standard deviation is a measurement problem, not perfection.** It usually means the instrument's resolution is coarser than the real variation, and the index is meaningless.

**Capability assumes a stable process.** Calculating CPU across a period that includes a process change averages two different distributions into one number that describes neither. Check the control chart first; capability is only meaningful once the process is in control.

## Ppl and Ppu

CPL and CPU use short-term, within-subgroup variation. The long-term equivalents, using the overall standard deviation across everything produced, are PPL and PPU. The formulas are identical; only the sigma differs. See [Ppl and Ppu](/guides/what-is-ppl-and-ppu-one-sided-performance) and [Cpk vs Ppk](/guides/cpk-vs-ppk-whats-the-difference).

## If You Already Run OpenHTF

A one-sided limit is expressed with a single bound, and the same rule applies about not inventing the other one:

```python filename="power_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("quiescent_current_mA")
    .in_range(maximum=45.0)
    .with_units(units.MILLIAMPERE)
    .doc("Upper limit only; CPU applies"),
)
def phase_quiescent(test):
    test.measurements.quiescent_current_mA = 38.2


test = htf.Test(phase_quiescent, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Inventing a second limit to get a Cpk | A capability number that describes nothing | Declare one validator; report CPU or CPL |
| Reporting Cpk for a one-sided measurement | Misleading, and usually wrong by construction | Use the index that matches the limit |
| Computing capability on an unstable process | The number averages two distributions | Check the control chart first |
| Ignoring a very high CPU | A test that cannot fail still costs cycle time | Review whether the limit is meaningful |
| Sigma of zero treated as excellent | The instrument cannot resolve the variation | Check measurement resolution |

### Avionics LRU Bench Testing and Analytics
URL: https://www.tofupilot.com/guides/avionics-lru-bench-testing-and-analytics
Learn how avionics LRU bench tests work, from induction to release, and how to turn CMM test results into analytics with TofuPilot.

An avionics LRU (line-replaceable unit) that comes off an aircraft passes through a well-defined shop flow: induction, functional test, fault isolation, repair, final acceptance, release. Every step produces measurements. In most shops those measurements end their life as an ATE printout or a PDF attached to the release certificate, which means the shop learns nothing from them beyond the single pass/fail verdict. This guide walks through the LRU bench workflow and shows how to capture the same test as structured data you can trend.

## The LRU Shop Workflow

A removed unit arrives with its removal paperwork and goes through:

1. **Induction.** Receiving inspection, identity check against the removal record.
2. **Functional test.** Run per the OEM's CMM (component maintenance manual), which defines the test conditions, limits, and maintenance levels for the unit.
3. **Fault isolation.** If the functional test fails, isolate to the board or subassembly.
4. **Repair.** Replace or rework the faulty element.
5. **Final acceptance test.** Full CMM test again, from scratch, on the repaired unit.
6. **Release.** Certificate of release to service backed by the acceptance test evidence.

Note the loop hiding in step 2: a significant share of inducted avionics units pass the functional test with no fault found. Published figures put NFF at 20 to 50 percent of avionics shop inductions. Those units consume a full bench cycle and return to stock with nothing learned, unless the measurements are kept.

## What the Bench Measures

A typical avionics LRU acceptance test covers a few functional areas, each with its own measurements and limits from the CMM:

- **Power.** Input current, supply voltages, inrush behavior.
- **Bus communication.** ARINC 429 word error rates, response times, BITE message readout.
- **Signal and calibration.** Output accuracy against reference, delays, signal-to-noise.
- **Environmental screening.** For intermittent suspects, temperature and vibration during test, informally known as bake and shake.

## Write the Acceptance Test as Structured Data

The test below is a final acceptance test for a repaired LRU, written with the TofuPilot framework. The procedure file declares the phases and limits; each phase is a plain Python function reading from the bench.

```yaml filename="procedure.yaml"
name: LRU Final Acceptance Test
version: 1.0.0
description: Final acceptance test for a repaired avionics LRU before release

unit:
  auto_identify: true
  serial_number:
    default_value: "LRU000042"
  part_number:
    default_value: "622-1234-001"

plugs:
  - name: bench
    python: plugs.bench:Bench

main:
  - name: Power Consumption
    python: phases.power
    measurements:
      - name: input_current
        unit: A
        validators:
          - operator: "<="
            expected_value: 1.8

  - name: Bus Communication
    python: phases.bus_comms
    measurements:
      - name: arinc429_word_error_rate
        validators:
          - operator: "<="
            expected_value: 0.00001
      - name: response_time
        unit: ms
        validators:
          - operator: "<="
            expected_value: 50

  - name: Output Calibration
    python: phases.calibration
    measurements:
      - name: output_accuracy_error
        unit: "%"
        validators:
          - operator: ">="
            expected_value: -0.5
          - operator: "<="
            expected_value: 0.5
```

```python filename="phases/power.py"
def power(measurements, bench):
    measurements.input_current = bench.read_input_current()
```

```python filename="phases/bus_comms.py"
def bus_comms(measurements, bench):
    measurements.arinc429_word_error_rate = bench.read_word_error_rate()
    measurements.response_time = bench.read_response_time()
```

```python filename="phases/calibration.py"
def calibration(measurements, bench):
    measurements.output_accuracy_error = bench.read_accuracy_error()
```

```python filename="plugs/bench.py"
class Bench:
    def read_input_current(self) -> float:
        return 1.42

    def read_word_error_rate(self) -> float:
        return 0.0

    def read_response_time(self) -> float:
        return 31.5

    def read_accuracy_error(self) -> float:
        return 0.12
```

Run it from the procedure directory:

```bash
tofupilot run ./lru-acceptance-test
```

```text
→ Phase: Power Consumption
→ Phase: Bus Communication
→ Phase: Output Calibration
✓ Phase Output Calibration: PASS
→   output_accuracy_error = 0.12 % [PASS]
✓ Phase Bus Communication: PASS
→   arinc429_word_error_rate = 0.0 [PASS]
→   response_time = 31.5 ms [PASS]
✓ Phase Power Consumption: PASS
→   input_current = 1.42 A [PASS]
✓ Run complete: PASS
```

The plug in this example returns fixed values so the test runs anywhere; on a real bench the same functions read from your instruments over VISA, serial, or whatever the fixture speaks. The structure is the point: every value is recorded with its unit and limit, tied to the serial number and part number.

## From Test Records to Analytics

Once acceptance tests flow in as structured runs, the analytics come for free:

- **Per part number.** First-pass yield across all units of a part, the failure Pareto showing which phase and measurement fail most, and Cpk on each measurement showing how much margin the fleet of units really has against the CMM limits.
- **Per serial number.** The unit history page lists every shop visit and every measured value for one serial. A unit inducted for the fourth time is visible at a glance, along with how its measurements moved between visits.
- **Over time.** Drift in a measurement across months of inductions points at aging in the field, a supplier change in a repaired subassembly, or the bench itself moving.

None of this requires the OEM's software or data. The measurements are produced by your own bench under your own test procedure; recording them structurally instead of printing them is the only change.

## What Good Looks Like

A component shop running this way releases every unit with the same certificate as before, but keeps the evidence behind it queryable. The engineer asked "have we seen this failure before" answers from the part number's history instead of a filing cabinet. The units that keep coming back stand out by retest count. And when a part's measurements start crowding a CMM limit across many serials, the shop sees it while it is still a trend, not yet a backlog.

### Why & How to Run Test Stations Without Root
URL: https://www.tofupilot.com/guides/why-how-to-run-test-stations-without-root
Running a test station as root is a costly shortcut. Learn least-privilege setup on Linux and Windows: low-privilege account, scoped hardware access, a service.

Running a production test station as `root` (Linux) or Administrator (Windows) is the fastest way to get hardware access working, and a shortcut you pay for later in security exposure and operational fragility. The highest-value habit is simple: **don't log the station in as an admin.** This guide covers that cheap win first, then how far you can take least privilege on Linux and Windows without breaking hardware access or boot-time startup.

## The Shortcut and What It Costs

When the priority is to deploy fast, running the whole station as root is often the shortest path: a device throws a `Permission denied`, and elevating unblocks it right away. Over time it hardens into one of these:

| Shortcut you see on the floor | What it saves | What it quietly costs |
|---|---|---|
| Auto-login as root, no password | Operator opens the session instantly | Anyone with physical access owns the machine |
| Root password on a sticky note | No one waits to unlock | Same as no password, plus a paper trail to your secret |
| `sudo` calls inside the runtime code | Script "just works" | Every run holds full-system power it never needs |
| `NOPASSWD` in sudoers for the runner | No prompts during a shift | Any bug or misuse escalates silently |

Sometimes it is a shortcut under deadline; sometimes a vendor tool genuinely demands root and there's no clean alternative that day. Either way, the cost is the same and it compounds across every station, every audit, and every machine that leaves the building for repair. The fix is not "never elevate." It is: **elevate to install, never to run.** Root is fine for setup (installing software, drivers, device rules), not for the identity that executes tests shift after shift.

## The Cheap Win: Don't Log In as Admin

Before any of the advanced setup below, one change removes most of the risk for almost no effort: **the operator logs in as a standard, non-admin user, and the station runs in kiosk mode.** This alone is the 80/20.

| Risk | A standard account + kiosk removes it? |
|---|---|
| Operator "borrows" the station to browse, copy files, install a tool | Yes, no admin rights, and the kiosk hides the desktop |
| Operator changes a system setting they shouldn't | Yes, they can't, by design |
| A bug in a parsed file corrupts the whole OS | Mostly, the process can't write outside the user's reach |
| Stolen/repaired machine leaks every stored secret | Reduced, the account's reach bounds what's exposed |

A standard account protects the operator from the machine as much as the machine from the operator: they should physically not be able to do what they aren't meant to. Kiosk mode is covered separately; the point here is that the account it runs under should not be an admin. Everything below is how to keep it that way while still reaching hardware and starting on boot.

## The Security Cost (Why This Matters)

Least privilege is error containment: the software equivalent of fusing a circuit instead of trusting nothing will ever short. A test runner parses files it did not write (firmware images, calibration data, serial responses, config from a share), so any bug runs with the identity of the process.

| Risk | As a low-privilege account | As root / admin |
|---|---|---|
| **Blast radius** | A bad parse corrupts one working directory | The same bug can overwrite the OS or another line's data |
| **Credential exposure** | Process reads only its own API key | Process can read every secret on the machine |
| **Supply chain / privilege escalation** | A compromised dependency is contained | A single bad package owns the box, no escalation needed |
| **Lost or repaired machine** | Data at rest is bounded by the account's reach | Full disk and every stored credential are exposed |
| **Audit** | Logs attribute actions to a station identity | Logs say `root did X` for everything, unattributable |

The escalation risk is concrete enough that good tooling refuses to enable it: the TofuPilot CLI will not install a root boot-service whose binary lives in a user-writable directory, because a non-root user who could overwrite that binary would gain root at the next boot. The audit line is now a business consequence too. Standing local admin on floor workstations is increasingly a **CMMC / NIST 800-171 audit failure**, a real barrier for anyone in aerospace, defense, or high-end EMS supply chains. And the blast-radius line is what security teams call **lateral movement**: an attacker who lands on one over-privileged station moves across the whole plant before anyone notices.

## "But the Station Is Offline"

Root-on-benches is often defended with "the station has no internet, so who cares." That misreads what offline means on a factory floor. The station is air-gapped from the *internet*, but it usually sits on a **secured local network**, reachable from other machines and often able to reach the MES and the production server where centralized production data lives.

An IT team only accepts the painful drawback of an offline station when the machines and data on that local network are sensitive enough to justify it. In other words, "offline" is a signal that security matters *more* here, not less. A station running as root on that network is a soft entry point to everything else on it: exactly the lateral movement the isolation was meant to prevent. Removable media compounds it: an offline station still takes firmware images on USB sticks and still leaves the building for repair (the Stuxnet worm reached isolated industrial systems precisely by USB).

## The Principle: Elevate to Install, Never to Run

| Task | Needs elevation? | Who runs it |
|---|---|---|
| Install software, drivers, device rules | Yes, once | Admin, at setup |
| Read/write a serial or USB device at runtime | No | Low-privilege account via group / rule |
| Execute tests, upload results | No | The station's non-admin account |
| Flash firmware via raw I/O | Sometimes | Grant the one capability, not full root |

The rest of this guide is how to build that on Linux and Windows.

## Linux: Least-Privilege Setup

### Step 1: Run Under a Low-Privilege Account

The account the station runs under should be a normal user with no admin rights: not `root`, not a member of `sudo`/`wheel`. On most benches this is the `operator` account the kiosk already uses.

```bash filename="01-create-operator-user.sh"
# A normal low-privilege account for the station. NOT in sudo/wheel/adm.
# The operator can log in and run the kiosk UI, but can't touch the system.
sudo useradd --create-home operator
id operator   # confirm: not a member of sudo, wheel, or adm
```

### Step 2: Grant Hardware Access With Groups, Not Root

The reason people reach for root is device permissions. The correct fix is group membership: add the operator account to the group that already owns the device.

```bash filename="02-grant-hardware-access.sh"
# Serial ports (/dev/ttyUSB*, /dev/ttyACM*) are group-owned by 'dialout'
sudo usermod -aG dialout operator
# Hot-plugged USB instruments are usually group-owned by 'plugdev'
sudo usermod -aG plugdev operator

# Confirm which group owns your device (4th column)
ls -l /dev/ttyUSB0
# crw-rw---- 1 root dialout 188, 0 ... /dev/ttyUSB0
#                   ^^^^^^^ group that grants access
```

For a device not owned by a standard group, write a **udev rule** (udev is the Linux subsystem that assigns ownership and permissions to devices as they appear):

```text filename="/etc/udev/rules.d/99-station-dut.rules"
# Match a specific instrument by USB vendor/product ID and hand it to
# the 'dialout' group (which the operator is in). No root at runtime.
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", GROUP="dialout", MODE="0660"
```

```bash filename="03-reload-udev.sh"
sudo udevadm control --reload-rules && sudo udevadm trigger
```

For a binary that needs one specific kernel privilege (e.g. raw I/O to flash firmware), grant that one **capability** instead of root. Capabilities split root's powers into independent units:

```bash filename="04-grant-capability.sh"
# Give only raw-I/O to this one binary, nothing else root can do
sudo setcap cap_sys_rawio+ep /usr/local/bin/flash-tool
getcap /usr/local/bin/flash-tool
```

### Step 3: Launch on Boot Without Root

This is where root usually sneaks back in. When you enable **Launch on boot**, the TofuPilot CLI installs the right startup service automatically, so you don't write a unit file by hand:

| CLI runs as | Service it installs | Kiosk |
|---|---|---|
| `operator` (non-admin) | systemd **user** service (`~/.config/systemd/user/`) | Yes, on the local display |
| `root` | systemd **system** service (`/etc/systemd/system/`) | No, headless, driven from the dashboard |

The trap that pushes people to root: if you enable launch-on-boot over SSH as the operator (a shell with no login session), `systemctl --user` has no session bus and fails:

```text filename="journalctl-error.txt"
Failed to connect to bus: No medium found
systemctl --user: Failed to connect to user scope bus
```

The fix is **not** to switch to root. It is to let the operator's user services run without an interactive login. That capability is called **lingering** in systemd (like a machine that keeps its background routine running even when no one is at the HMI):

```bash filename="05-enable-linger.sh"
# Operator's systemd services now start at boot with nobody logged in
sudo loginctl enable-linger operator
loginctl show-user operator --property=Linger   # -> Linger=yes
```

With lingering on, the non-root station starts at every boot, keeps its kiosk display, and never needs root for startup. Credentials follow the same account: run `tofupilot login --token <setup-token>` as the operator, not `sudo`.

### Step 4: Verify Least Privilege

```bash filename="06-verify.sh"
# The station process should show 'operator', not root
ps -o user,cmd -C tofupilot
# Run the check AS operator (sudo -u switches identity, it does not elevate)
sudo -u operator cat /dev/ttyUSB0   # opens via the dialout group, not "Permission denied"
```

## Windows: Least-Privilege Setup

Same principle: the operator account is a **standard user**, not Administrator. Windows makes most of this easy because standard users already have the access a test runner needs.

| Need | Least-privilege grant | Avoid |
|---|---|---|
| Read/write a COM port | Standard users can open COM ports by default; no admin needed | Running as admin "to be safe" |
| USB instrument via vendor driver | Install the driver once as admin; runtime access is per-device | Admin at runtime |
| Write logs / results | Grant the account write access to one folder (e.g. `C:\tofupilot`) | Writing under `C:\Program Files` |
| Launch on boot | TofuPilot registers a per-user `Run` entry, fires at the operator's logon, no admin | A machine-wide service running as SYSTEM/admin |

```powershell filename="01-create-operator-user.ps1"
# Local standard user (NOT added to Administrators)
$pw = Read-Host -AsSecureString "Password for the operator account"
New-LocalUser -Name "operator" -Password $pw -PasswordNeverExpires `
  -Description "Test station operator"
# Deliberately NOT: Add-LocalGroupMember -Group "Administrators"
```

```powershell filename="02-grant-folder-access.ps1"
# Give the operator account its own writable working directory
New-Item -ItemType Directory -Path "C:\tofupilot" -Force
icacls "C:\tofupilot" /grant "operator:(OI)(CI)M"
```

On Windows the station runs as a per-user logon process under the standard operator account, so it starts when the operator logs in, with no admin service and no SYSTEM privileges. Store the API key in the operator's user-scoped environment or a file under `C:\tofupilot` with permissions limited to that account, never machine-wide.

## When Root Is the Pragmatic Choice

Best practice sometimes conflicts with productivity, and the person who knows the constraints, the developer at the bench during a production rush, is the one who should decide. At TofuPilot we don't block that decision: the CLI runs under root or admin at runtime if you choose to, because removing deployment friction during a rush matters more than enforcing rigidity from the outside. Running the CLI as root even installs the correct system service for you.

If you do run as root, do it deliberately and shrink the risk:

| If you run as root | Reduce the risk by |
|---|---|
| A vendor tool or raw I/O genuinely needs it | Scope it: `setcap` on that one binary instead of root everywhere |
| Hardware access is blocking a deadline | Ship the root shortcut, then schedule the udev/group fix as follow-up |
| Headless bench, launch on boot | Keep the binary in `/usr/local/bin` owned by root (the CLI refuses a user-writable one) |
| Physical machine can be stolen or sent for repair | Encrypt the disk; keep the API key scoped and revocable |
| Operators share the station | Pair root runtime with kiosk lockdown so they can't wander off the test UI |

The point is not purity. It is knowing what the shortcut costs so the choice is informed, and moving to least privilege once the fire is out.

## Troubleshooting

| Issue | Cause | Fix |
|---|---|---|
| `systemctl --user: Failed to connect to user scope bus` | Enabling launch-on-boot from a shell with no login session (SSH) | `sudo loginctl enable-linger operator`, don't switch to root |
| Station doesn't start at boot, no error | User service installed but lingering is off | Enable lingering (above); confirm with `loginctl show-user operator` |
| `Permission denied` on `/dev/ttyUSB0` after dropping root | Operator not in the device's group | `usermod -aG dialout operator`, then log out/in or reboot |
| Device works for root but not the operator | udev rule not applied, or wrong vendor/product ID | Re-check IDs with `lsusb`, reload with `udevadm control --reload-rules && udevadm trigger` |
| CLI refuses to install a root system service | Binary sits in a user-writable path (a root-at-boot escalation vector) | Move it to `/usr/local/bin` owned by root, then re-run |
| Windows runner can't write results after dropping admin | Writing to a protected path | Point output at the operator's own folder (`C:\tofupilot`) |

### Reduce No-Fault-Found (NFF) in Testing
URL: https://www.tofupilot.com/guides/reduce-no-fault-found-nff-in-testing
Learn what No-Fault-Found means, why units pass on the bench yet fail in service, and how to catch the pattern using your own test data in TofuPilot.

No-Fault-Found is a returned unit that tests good on the bench. Nothing reproduces, so the unit ships back into service, often to fail again weeks later. Across component maintenance, NFF accounts for a large share of returns, and every one of them burns bench time while eroding trust in the test itself. This guide explains what NFF is, why it happens, and how to catch the pattern behind it using your own test data.

## What No-Fault-Found Means

A unit comes off equipment reported as faulty. On the bench it passes every check. With no failure to act on, the unit is returned to service as serviceable. That is an NFF, sometimes logged as CND (Cannot Duplicate) or RTOK (Retest OK).

NFF is not one problem. It is a label for several:

- **Intermittent faults.** The defect is real but only appears under conditions the bench does not reproduce (temperature, vibration, a specific input sequence).
- **Test escape.** The bench limits are too loose, so a marginal unit passes.
- **Wrong removal.** The fault was elsewhere in the system, and a good unit was pulled.
- **Fixture drift.** The bench itself has changed, so a real fault reads as a pass.

You cannot fix what you cannot see. The first step is making each of these visible in the data, which starts with how the test is written.

## Structure the Test So NFF Is Visible

An NFF investigation needs three things in the record: the exact measured value (not just pass/fail), the serial number, and a stable set of limits to trend against. A test that only reports pass or fail throws away the information you need.

The example below is a bench acceptance test for a returned unit, written with the TofuPilot framework. Each phase reads a real value and validates it against a limit, so a passing unit still leaves a full measurement trail behind.

```yaml filename="procedure.yaml"
name: LRU Bench Acceptance Test
version: 1.0.0
description: Functional acceptance test for a returned unit on the repair bench

unit:
  auto_identify: true
  serial_number:
    default_value: "LRU000001"
  part_number:
    default_value: "622-1234-001"

plugs:
  - name: bench
    python: plugs.bench:Bench

main:
  - name: Power Supply Check
    python: phases.power_check
    measurements:
      - name: supply_voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 27.5
          - operator: "<="
            expected_value: 28.5

  - name: Signal Integrity
    python: phases.signal_check
    measurements:
      - name: output_delay
        unit: us
        validators:
          - operator: "<="
            expected_value: 12.0
      - name: signal_snr
        unit: dB
        validators:
          - operator: ">="
            expected_value: 40.0
```

```python filename="phases/power_check.py"
def power_check(measurements, bench):
    measurements.supply_voltage = bench.read_supply_voltage()
```

```python filename="phases/signal_check.py"
def signal_check(measurements, bench):
    measurements.output_delay = bench.read_output_delay()
    measurements.signal_snr = bench.read_snr()
```

```python filename="plugs/bench.py"
class Bench:
    def read_supply_voltage(self) -> float:
        return 28.01

    def read_output_delay(self) -> float:
        return 9.4

    def read_snr(self) -> float:
        return 44.7
```

Run it from the procedure directory:

```bash
tofupilot run ./lru-bench-test
```

```text
→ Phase: Power Supply Check
→ Phase: Signal Integrity
✓ Phase Power Supply Check: PASS
→   supply_voltage = 28.01 V [PASS]
✓ Phase Signal Integrity: PASS
→   output_delay = 9.4 us [PASS]
→   signal_snr = 44.7 dB [PASS]
✓ Run complete: PASS
```

This unit passes. That is the NFF case. But because the test recorded `output_delay = 9.4 us` against a 12.0 limit, not just PASS, the next unit that reads 11.8 is visible as a marginal pass long before it fails in service. The value is what lets you trend. The pass/fail alone tells you nothing.

## Track Retest Counts

The strongest NFF signal is a unit that gets tested more than once. A single unit cycling through the bench three or four times is either intermittent or being chased in circles. Because each run identifies the unit by serial number, TofuPilot builds a per-unit history automatically. The unit history page shows every run for a serial number, so a repeat visitor stands out without you tracking it by hand.

Sort a part number by retest count and the intermittent units surface at the top. These are the ones worth pulling for a deeper look, and the ones a supplier discussion should be built around.

## Trend the Measurement, Not the Verdict

A unit that passes at the edge of its limit is an NFF waiting to happen. To catch it, look at the measured value over time rather than the pass rate.

In TofuPilot, open the measurement for the part number and read the distribution and the drift:

- **Distribution.** If passing values cluster near a limit instead of centering in the range, the design or the process is running close to the edge. Marginal units are passing today and will fail tomorrow.
- **Cpk.** A Cpk below 1.0 means the spread does not fit inside the limits reliably. Some fraction of units will always fall out, and some of those are your NFF returns.
- **Drift.** A slow shift in the mean points to aging, a supplier change, or a bench that has moved. Catching the shift early is the difference between one investigation and a batch of returns.

This is the same statistical basis manufacturing teams use for yield, applied to a repair line. The question shifts from "did this unit pass" to "is this measurement healthy across every unit we see."

## Alert on the Signal Automatically

Reviewing charts by hand does not scale. TofuPilot can watch the data and raise an alert when the NFF pattern appears:

- **Measurement drift.** Fires when a measurement moves off its own baseline, whether a mean shift or a falling Cpk, so a drifting part is flagged before it becomes a wave of returns.
- **Retest threshold.** Fires when a unit is tested more often than the procedure's normal rate. The threshold self-calibrates per procedure, so you are alerted on the genuinely abnormal units, not the routine ones.

The alerts learn each procedure's normal behavior, so you set them once rather than hand-tuning limits per part.

## What Good Looks Like

A repair line with NFF under control has a few visible traits. Retest counts are low and flat, with few units cycling back. Measurement distributions center inside their limits rather than crowding an edge, and Cpk stays above 1.0. When a value does start to drift, an alert catches it while it is still one part, not a batch.

None of this requires new hardware or access to anyone's proprietary software. It comes from recording the measured value instead of the verdict, keeping a per-unit history, and trending what you already capture on the bench.

### TofuPilot Framework vs OpenHTF
URL: https://www.tofupilot.com/guides/tofupilot-framework-vs-openhtf
How the TofuPilot Framework and OpenHTF differ, and how each one plugs into the TofuPilot dashboard and stations for live test data.

OpenHTF and the TofuPilot Framework are the two open-source Python frameworks purpose-built for hardware manufacturing test. OpenHTF was built by Google in 2016 and is the established, widely deployed option. The TofuPilot Framework is newer, built on a Rust execution engine for parallel execution. Both model a test as ordered phases with structured measurements, both connect instruments through reusable plugs, and both run natively on the TofuPilot CLI and stream to the same dashboard. This guide compares them side by side with real code so you can pick the right one.

## Framework Overview

| | TofuPilot Framework | OpenHTF |
|---|---|---|
| **Language** | Python (Rust engine) | Python |
| **License** | MIT | Apache 2.0 |
| **Maintainer** | TofuPilot | Google |
| **First release** | 2025 | 2016 |
| **Authoring model** | Declarative YAML procedure plus Python phases and plugs | Code-first Python with decorators |
| **Execution** | Parallel phases and slots | Sequential phases |
| **Community** | Newer, smaller | Established, larger install base |

## Feature Comparison

| Feature | TofuPilot Framework | OpenHTF |
|---|---|---|
| **Phases** | Yes | Yes |
| **Numeric, string, boolean measurements** | Yes | Yes |
| **Logs and attachments** | Yes | Yes |
| **Unit identification by serial number** | Yes | Yes |
| **Sub-units (child components)** | Yes | Yes |
| **Plugs (shared instrument connections)** | Yes | Yes |
| **Multi-dimensional measurements (waveforms, sweeps)** | Yes | Yes |
| **Aggregations (mean, min, max, std over arrays)** | Yes | No |
| **Per-axis validators (validate curve shape)** | Yes | No |
| **Operator prompts** | Yes | Yes |
| **Custom forms and display components** | Yes | No |
| **Retry, timeout, setup/teardown** | Yes | Yes |
| **Conditional execution** | Yes | Yes |
| **Parallel phases** | Yes | No |
| **Multi-slot execution (multiple boards on one fixture)** | Yes | No |

On the shared data model the two are close to parity. The gaps fall in three areas: advanced measurement analysis, the operator UI, and execution.

## The Same Test in Both Frameworks

A functional test that verifies a board's 3.3V rail and firmware version, written in each framework.

### TofuPilot Framework Version

The procedure YAML declares the sequence and limits. The phases and plug are plain Python that the procedure references by name.

```yaml filename="procedure.yaml"
name: Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dut
    python: plugs.dut:Dut

main:
  - name: Measure Voltage
    python: phases.measure_voltage
    measurements:
      - name: rail_3v3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
  - name: Check Firmware
    python: phases.check_firmware
    measurements:
      - name: firmware_version
        validators:
          - operator: "=="
            expected_value: "2.1.0"
```

```python filename="phases/measure_voltage.py"
# Parameters are injected by name: measurements, plus the dut plug.
def measure_voltage(measurements, dut):
    measurements.rail_3v3 = dut.read_voltage()
```

```python filename="phases/check_firmware.py"
def check_firmware(measurements, dut):
    measurements.firmware_version = dut.query_firmware()
```

```python filename="plugs/dut.py"
class Dut:
    """Manage the connection to the unit under test."""

    def read_voltage(self) -> float:
        return 3.31  # Replace with an instrument read

    def query_firmware(self) -> str:
        return "2.1.0"  # Replace with a DUT query
```

Run it from the procedure directory with `tofupilot run`, then deploy to a station with a Git push.

### OpenHTF Version

OpenHTF attaches measurements and plugs to phase functions with decorators, then runs them in sequence.

```python filename="openhtf_test.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units


class DutPlug(BasePlug):
    """Manage the connection to the unit under test."""

    def read_voltage(self) -> float:
        return 3.31  # Replace with an instrument read

    def query_firmware(self) -> str:
        return "2.1.0"  # Replace with a DUT query


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT)
)
@htf.plug(dut=DutPlug)
def measure_voltage(test, dut):
    test.measurements.rail_3v3 = dut.read_voltage()


@htf.measures(htf.Measurement("firmware_version").equals("2.1.0"))
@htf.plug(dut=DutPlug)
def check_firmware(test, dut):
    test.measurements.firmware_version = dut.query_firmware()


def main():
    test = htf.Test(measure_voltage, check_firmware, part_number="PCBA-100")
    test.execute(test_start=lambda: input("Scan serial number: "))


if __name__ == "__main__":
    main()
```

The TofuPilot CLI runs both frameworks natively and streams phases, measurements, logs, attachments, and the operator UI to the dashboard, with no extra code on either side.

## Key Differences in the Code

| Aspect | TofuPilot Framework | OpenHTF |
|---|---|---|
| **Sequence definition** | Declared in `procedure.yaml`, separate from logic | Function order in `htf.Test(...)`, in code |
| **Limits** | `validators` in YAML, stored as data | `.in_range(3.2, 3.4)` chained in code |
| **Plug injection** | By parameter name (`dut`) | By `@htf.plug(dut=...)` decorator |
| **Measurement access** | `measurements.rail_3v3 = ...` | `test.measurements.rail_3v3 = ...` |
| **Plug class** | Plain Python class | Subclass of `BasePlug` |

## Execution and Parallelism

This is the clearest architectural difference.

OpenHTF runs phases sequentially against one unit under test. To test multiple boards or run independent steps concurrently, you orchestrate that yourself outside the framework.

The TofuPilot Framework runs on a Rust engine that executes independent phases in parallel and serializes only where you declare a `depends_on`. It also supports multi-slot execution, testing several boards in parallel on one fixture, which directly reduces cycle time on high-volume lines. Your phase and plug code stays plain Python; the engine handles the scheduling.

If your bottleneck is throughput on a fixture that holds multiple DUTs, this is the deciding factor. If you test one unit at a time, it matters less.

## Measurements and Operator UI

Both frameworks capture typed measurements with limits, units, and validators, and both support multi-dimensional measurements for waveforms and sweeps. The TofuPilot Framework adds two things OpenHTF does not: aggregations (mean, min, max, std computed over an array) and per-axis validators that check the shape of a curve rather than only its endpoints. For RF sweeps, thermal profiles, or any test where the waveform itself is the spec, that removes glue code.

Both ship a built-in operator UI. OpenHTF provides web prompts for operator input such as scanning a serial number. The TofuPilot Framework adds declarative form and display components (text, number, switch, radio, select, multiselect, image choice, sliders, progress) that you define in the procedure without writing any frontend.

## When to Use the TofuPilot Framework

The TofuPilot Framework is the better choice when:

- **Your line is throughput-bound.** Parallel phases and multi-slot execution test several boards at once on one fixture and cut cycle time. OpenHTF executes sequentially.
- **You validate waveforms or sweeps.** Built-in aggregations and per-axis validators check curve shape, not only endpoints, with no glue code.
- **Operators need rich guided workflows.** Declarative forms and display components render on the station with no frontend work.
- **You want the test plan separate from the logic.** The YAML procedure is readable and diffable independently of the Python phases.
- **You are setting up new stations.** Deploy-on-push, real-time streaming, and offline queueing are built into the CLI.

## When to Use OpenHTF

OpenHTF is the better choice when:

- **You already run it.** Existing OpenHTF suites run natively on TofuPilot with no rewrite. The migration cost is not worth it.
- **Maturity and community matter most.** Years of production use at scale and a larger body of public examples and patterns.
- **You want a single pure-Python paradigm.** Everything is in Python, with no separate YAML layer to learn.
- **Apache 2.0 fits your legal review.** Its explicit patent grant can matter in some procurement contexts.

## Using Both Together

You do not have to choose the framework to choose the platform. A common path is to keep an existing OpenHTF suite running as is, and reach for the TofuPilot Framework on new stations that need parallel slots or richer measurements. Both stream phases, measurements, logs, attachments, and the operator UI to the same dashboard, and both are open-source, so neither choice locks the test code itself.

## Decision Matrix

| Question | Better fit |
|---|---|
| Testing multiple boards on one fixture, throughput-bound? | TofuPilot Framework |
| Validating waveform or sweep shape, not just endpoints? | TofuPilot Framework |
| Need rich operator forms on the station? | TofuPilot Framework |
| Already running OpenHTF suites in production? | OpenHTF |
| Want a single pure-Python paradigm, no YAML? | OpenHTF |
| Maximum maturity and community matter most? | OpenHTF |

Both run on the TofuPilot CLI and upload to the same dashboard, so you can start with either and mix them per station.

### Component Test Records for EASA Part-145
URL: https://www.tofupilot.com/guides/component-test-records-for-easa-part-145
Map EASA Part-145 record-keeping and calibration rules to TofuPilot features: test records behind the Form 1, retention, backups, and traceability.

An EASA Part-145 component shop releases every unit with an EASA Form 1, and behind every Form 1 sits test evidence the regulation requires you to keep, protect, and produce on demand. Most shops meet the letter of this with paper test sheets and PDF printouts. This guide maps the actual Part-145 clauses to what a structured test-data system covers, so the same records that satisfy the auditor also feed your analytics.

## The Clauses That Touch Test Data

| Clause | Requirement | TofuPilot feature |
|---|---|---|
| 145.A.40(b) | Test equipment controlled and calibrated to an officially recognised standard; calibration records and traceability kept | Station identifier on every run, linking results to the bench and its calibration register |
| 145.A.50(d) | CRS for components issued as EASA Form 1, backed by acceptance test evidence | Complete run record per release: measurements, limits, result, date, unit identity |
| 145.A.55(a) | Retain all records necessary to prove requirements were met for the CRS | Immutable run history per serial number |
| 145.A.55(a)(3) | Detailed maintenance records kept 3 years from CRS issue | Cloud retention beyond the mandated period |
| AMC 145.A.55 | Electronic systems: backup within 24 hours of new entries, safeguards against unauthorised alteration, backup stored in a different location | Managed storage with replication; records not editable after upload |

Two details from the AMC and guidance material are worth knowing precisely. First, the acceptance test report should be attached to a Form 1 issued as Inspected/Tested, and Block 12 must cite the maintenance data with its revision status; "in accordance with the CMM" alone is not acceptable. Second, the retention clock runs from when the record was created or last amended, so a system that timestamps every run gives you the retention basis for free.

## Structure the Test So the Record Is the Evidence

The regulation asks for proof that the unit met its acceptance criteria. A structured test produces that proof as a side effect of running. Each measurement carries its limits, its unit, and a description tying it back to the maintenance data:

```yaml filename="procedure.yaml"
name: LRU Bench Acceptance Test
version: 1.0.0
description: Functional acceptance test for a returned unit on the repair bench

unit:
  serial_number:
    default_value: "LRU000001"
  part_number:
    default_value: "622-1234-001"

main:
  - name: Power Supply Check
    python: phases.power_check
    measurements:
      - name: supply_voltage
        unit: V
        description: "28V nominal bus input, per CMM test 1A"
        validators:
          - operator: ">="
            expected_value: 27.5
          - operator: "<="
            expected_value: 28.5
```

The run record then contains everything the release depends on: the serial number and part number, each measured value against its declared limit, the procedure name and version, the station that ran it, and the timestamp. When the Form 1 goes out as Inspected/Tested, the attached test report is a query, not a scan.

Procedure versioning matters here more than it first appears. Block 12 requires the revision status of the maintenance data used. Versioning your test procedure alongside the CMM revision it implements means every historical run states which revision of the test it was measured against.

## Calibration Traceability by Station

145.A.40(b) requires calibration records and traceability for the test equipment itself, and the audit checklist covers tools, calibration, and alternative tooling explicitly. TofuPilot does not replace your calibration register, but the station identifier on every run creates the link auditors look for: name stations to match the calibration register, and any measurement can be traced to the bench that produced it and from there to its calibration state on that date.

This also answers the ugly retrospective question. When a tool turns up out of calibration past its due date, the shop must review the work that used it. With runs filtered by station and date range, the affected releases are a query instead of a work-order archaeology project.

## Retention, Backup, and Integrity

AMC 145.A.55 sets concrete expectations for computer record systems: a backup updated within 24 hours of any new entry, protection against unauthorised alteration, and backup media held in a different location from the working data. A shop running its records on a spreadsheet and a shared drive owns all three of those problems itself, and the control of computer record systems is a standing item on the EASA audit checklist (MOE chapter 2.21).

Structured cloud records shift that burden: runs are immutable once uploaded, storage is replicated, and every record is retrievable by serial number, part number, procedure, station, date range, or result for the full retention period and beyond.

For shops working under FAA approval as well, 14 CFR 145.219 asks for records demonstrating compliance for at least 2 years from return to service, in English, in a format acceptable to the FAA; electronic record systems come under AC 120-78B. The EASA 3-year requirement is the stricter of the two, so a system sized for Part-145 covers both.

## What Good Looks Like

In a shop running this way, an auditor asking for the test evidence behind a given Form 1 gets it by serial number in seconds, with limits, values, procedure revision, and station attached. The calibration question traces from any measurement to a named bench. Retention is not a filing discipline but a property of the system. And because the compliance records are structured measurements rather than PDFs, the same data answers the engineering questions too: yield by part number, drift by measurement, and the units that keep coming back.

### Migrate a Station from v1.0 to TofuPilot CLI
URL: https://www.tofupilot.com/guides/migrate-a-station-from-v10-to-tofupilot-cli
Move a test bench from the v1.0 desktop station app to the TofuPilot CLI: one-command install, automatic deploys, and a live operator UI.

The TofuPilot CLI is the new way to run a test station. It is the runtime behind Stations v2.0, and it replaces the v1.0 station: the desktop app that ran on the bench, pulled deployments, and executed procedures locally. The CLI does the same job with one-command install, automatic deploys from Git, and the same live operator UI. This guide shows how to move a bench over.

> On a self-hosted instance? Set up the deployer first so your instance can build station bundles, and enable the realtime server so runs stream live. Follow [Enable Realtime and Deploys on Self-Hosted](/guides/enable-realtime-and-deploys-on-self-hosted), then come back here. Cloud TofuPilot has both on already, so you can start right away.

## Why move to the CLI

Station v1.0 was a desktop application installed on each bench. Stations v2.0 run on the TofuPilot CLI, which keeps everything the desktop app did and adds what it could not:

- **One-command install.** No installer to download per machine. The CLI installs in one shell command from the station setup page and registers itself on first boot.
- **Deploy from Git automatically.** Connect the procedure to its GitHub repo and pick a branch (typically `main`) for auto-deploy. Pushes to that branch roll out to the right stations on their own. Other branches stay manual: deploy them from the dashboard when you are ready.
- **Remote and kiosk operator UI.** Drive a run from the dashboard over the web, or from a full-screen local kiosk UI on the bench. A terminal UI is there too when an engineer drives a benchtop test.
- **No Git provider login on the bench.** The desktop app synced procedure code straight from GitHub or GitLab, so each bench needed Git provider access. Linking the repo now happens once in the dashboard, and the bench pulls a pre-built artifact instead, so it never touches your repository and you control deploys entirely from the web.
- **Offline-friendly upload queue.** Runs queue locally on the bench and upload as soon as the network is back, so a flaky link never costs you data.
- **Runs headless anywhere.** The desktop app needed a desktop environment. The CLI runs headless on a server, a rack box, or a small single-board computer, and works against self-hosted deployments out of the box.

## Migration steps

### 1. Make sure the procedure has a deployment

The CLI runs from a built artifact, so the procedure needs a fresh deployment for the new station to pull. On the procedure page in the dashboard:

- If the procedure is already linked to its Git repository (most v1.0 procedures are), trigger a new deployment: push to the auto-deploy branch, or deploy manually from the dashboard.
- If it is not linked yet, link the GitHub or GitLab repository first, then deploy.

That build is what the new station pulls and runs.

### 2. Install the CLI on the bench

Open the station's detail page in the dashboard and copy its install command. It is pre-filled with the station's setup token, and on self-hosted it also carries the `--url` flag pointing at your instance, so copy it from the dashboard rather than typing it by hand. It looks like this:

```bash title="install"
curl -fsSL https://tofupilot.app/install | sh -s -- --token <SETUP_TOKEN>
```

Run it on the bench once. The CLI installs, registers itself with your organization, and the station shows up online in the dashboard. The setup token expires one hour after issue, so if it lapses, generate a new one from the station's row.

### 3. Tune the station config

From the station settings, pick the behaviors that match your floor:

- **Launch on boot** so the station is ready as soon as it powers up.
- **Kiosk UI** for full-screen operator mode, or **Terminal UI** for an engineer benchtop.
- **Desktop shortcut** for one-click launch.
- **Auto-update** to keep the CLI current between runs.

### 4. Retire the desktop app

Once the bench is running runs through the CLI and they land in the dashboard live, uninstall the v1.0 desktop app from the machine. Station v1.0 is being phased out, and we are reaching out to every team still on it with hands-on migration support.

You are done. The next operator-triggered run lands in the dashboard live, with validators and attachments handled by the CLI.

### Why TofuPilot Needs Administration: write
URL: https://www.tofupilot.com/guides/why-tofupilot-needs-administration-write
Learn why the TofuPilot GitHub App requests Administration: write, what it enables, and how to scope it safely with selected repositories.

The TofuPilot GitHub App now requests the `Administration: write` repository permission. This guide explains why we ask for it ahead of an upcoming feature, what it enables, our security model, and the alternatives if you'd rather not grant it.

## Why the new permission

The upcoming Stations v2.0 ships a new procedure-creation workflow that takes you from procedure to deployed-on-station in two steps. Since many users don't have a repository yet, or are new to TofuPilot Framework / OpenHTF, we'll provide ready-to-go templates that get cloned into your account before deploying.

GitHub requires `Administration: write` for any third-party app to create a repository on a user's behalf. We have no other write operations beyond this one. The same permission is requested by leading developer platforms for the same reason — it's the only mechanism GitHub exposes for creating a repository from a template.

## Security model

We take this permission seriously. Our protections include:

- **No other write routes.** Our APIs expose no GitHub write operations beyond this single template-clone call, scoped to creating one new repository.
- **Strict tenant isolation.** Connection records are isolated per organization at the database level — even an authenticated user from one organization cannot read another organization's data.
- **Restricted to owners and admins.** Managing connections (create / update / delete) is restricted to organization owners and admins.
- **Secrets kept out of the database.** The GitHub App private key required to mint installation tokens lives in our cloud provider's encrypted secret store, not in our application database. A database leak alone does not grant access to your GitHub account.

## If you'd rather not grant it

You have three options:

### 1. Don't accept the permission

The Clone Template feature won't work, but everything else (importing existing repos, auto-deploy, webhooks, operator UI) continues to work normally. You can create a repository manually on GitHub, then use the **Import** flow.

### 2. Scope to selected repositories

When re-accepting the permissions, pick **Only select repositories** instead of **All repositories**. This limits the App's access to repositories you explicitly pick.

After we clone a template, you may need to manually add the new repository to the App's allowed list before auto-deploy starts working on it.

### 3. Self-hosted or dedicated managed deployment

Configure your own GitHub App with whatever permissions fit your security policy. TofuPilot adapts to the permissions available: features that need missing permissions are hidden, everything else continues to work normally.

## How to grant or scope the permission

1. In TofuPilot, open your organization's **New Procedure** page. If the permission is missing, an in-app banner will appear with a **Grant permission on GitHub** button once the feature ships.
2. Click the button. You're redirected to your GitHub installation's permissions page.
3. Choose **Only select repositories** if you want to scope the App tightly. Otherwise, **All repositories** is fine.
4. Click **Accept new permissions**. You're returned to TofuPilot, and the Clone Template form unlocks automatically.

Approval takes about 30 seconds.

## What we use this permission for

We use `Administration: write` only to call GitHub's "create a repository from a template" API during the Clone Template flow. Every API call is logged in your GitHub audit log so you can verify exactly what we do.

We never delete, transfer, archive, or modify settings on existing repositories.

## Frequently asked questions

### Can you implement template clone without this permission?

Not on GitHub. GitHub does not offer a narrower scope. Every platform that lets you clone a template into a new repository requests the same permission for the same purpose.

### What happens if I revoke the permission later?

Revoke it whenever you want from your GitHub install settings. Clone Template stops working immediately; everything else keeps working normally. Repositories created before the revoke are unaffected.

### Does this give TofuPilot access to my private code?

Only for repositories the App is installed on. With **Only select repositories**, you control exactly which ones. With **All repositories**, the App sees everything in the account it's installed on. Read access to repository contents is governed by a separate permission TofuPilot has had since the beginning — `Administration: write` is solely about creating new repositories.

### Realtime and Deploys on Self-Hosted TofuPilot
URL: https://www.tofupilot.com/guides/enable-realtime-and-deploys-on-self-hosted
Add the Centrifugo realtime server and the Docker-in-Docker deployer to a self-hosted TofuPilot instance for live status and Git-push deployments.

Self-hosted TofuPilot ships two services that bring it to cloud parity: a **realtime server** and a **deployer**. Realtime flips stations online the moment they connect and streams runs live; the deployer builds your procedure bundle from Git and streams the build log while it runs. Both come down with the deploy script. This guide shows what they need and how to confirm they are working.

This is for **existing self-hosted instances** adding these services. Both landed in **2.33**, so if you are upgrading from an earlier version they are new to you: re-running the deploy script pulls them in, but the realtime server also needs a DNS record that older instances never had (Step 1). Cloud TofuPilot (`tofupilot.app`) has both on already, and fresh self-hosted instances now get them by default from the [Configure](/docs/self-hosting/configure) and [Deploy](/docs/self-hosting/deploy) reference docs. For the general upgrade flow, see [Upgrade](/docs/self-hosting/upgrade). Once realtime and the deployer are running, you can move your benches off the v1.0 desktop app with [Migrate a Station from v1.0 to TofuPilot CLI](/guides/migrate-a-station-from-v10-to-tofupilot-cli).

## What you get

- **Realtime.** Live station status, telemetry, and run streaming to the operator UI and dashboard, plus streaming build logs. Without it the app still works, but live views fall back to manual refresh.
- **Deployer.** The Git-push deployment pipeline: link a procedure to a repo, push, and the bundle builds and rolls out to your stations. Without it the rest of the stack works, but the build pipeline is unavailable, so Stations v2.0 cannot get their bundles.

## Step 1: Add the realtime subdomain

A self-hosted instance needs three subdomains. The third carries the realtime traffic. Create a DNS **A record** pointing at your server:

```text title="DNS"
realtime.tofupilot.yourcompany.com   A   <your-server-ip>
```

In `.env`, the realtime host defaults to `realtime.<DOMAIN_NAME>`, so if your app runs at `tofupilot.yourcompany.com` you can leave it unset. To use a different host, set `CENTRIFUGO_DOMAIN_NAME`. The deploy script generates the realtime and deployer secrets on first run and reuses them after, so you do not manage them by hand.

## Step 2: Run Docker as root (for the deployer)

The deployer builds in a sandbox that needs a privileged container, which only standard (root) Docker allows. Under rootless Docker the deployer stays down and the build pipeline is unavailable, while everything else, including realtime, works normally. If you need the build pipeline, install Docker the standard way, not rootless.

## Step 3: Deploy

Re-run the deploy script. The license key is already in `.env`, so you do not pass it again:

```bash title="deploy"
curl -fsSL https://www.tofupilot.com/deploy | bash
```

The script brings up the realtime and deployer containers alongside the rest of the stack and requests a TLS certificate for the realtime subdomain automatically. Confirm everything is running:

```bash title="check containers"
docker compose ps
```

## Step 4: Verify

**Realtime:** open the dashboard and watch a station's status, or trigger a run and watch phases stream. Status should flip and runs should update with no page refresh.

**Deployer:** link a procedure to a Git repo, push to the auto-deploy branch, and open the deployment. The build log should scroll live as the bundle builds, then the artifact rolls out to your stations.

If realtime stays disconnected, the cause is almost always the subdomain: confirm the `realtime.` A record resolves to your server, its TLS certificate issued, and port 443 is open for it just like the app.

### Operator Interface with OpenHTF
URL: https://www.tofupilot.com/guides/operator-interface-with-openhtf
Build an operator interface for OpenHTF tests: the built-in Station Server, user input plugs, and the richer UI available through the TofuPilot CLI.

OpenHTF ships with an operator interface, and most teams outgrow it. This guide covers what the built-in Station Server does, how to collect operator input in an OpenHTF test, and the options when you need more than prompts.

## What OpenHTF Provides

| Feature | Built-in |
|---------|----------|
| Serial number prompt at test start | Yes, via `test_start` |
| Text prompts during a phase | Yes, via the `UserInput` plug |
| Web display of phase progress | Yes, the Station Server on `localhost:12000` |
| Pass/fail display | Yes, basic |
| Rich input types (sliders, checklists, images) | No |
| Kiosk mode | No |
| Offline result queueing | No |

The Station Server is functional and honest about its scope. It was built to show a technician what is happening, not to be the product surface of a factory line.

## Prompting for the Serial Number

The `test_start` callable runs before the first phase and returns the DUT identifier:

```python filename="basic.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
def test_power(test):
    test.measurements.rail_3v3 = 3.31


def main():
    test = htf.Test(test_power, part_number="PCBA-100")
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial number: "))


if __name__ == "__main__":
    main()
```

For a proper prompt rather than a console `input()`, use the built-in prompt helper:

```python filename="prompt_start.py"
from openhtf.plugs.user_input import prompt_for_test_start

test.execute(test_start=prompt_for_test_start("Scan the unit serial number"))
```

## Prompting Inside a Phase

The `UserInput` plug blocks a phase until the operator responds:

```python filename="operator_prompts.py"
import openhtf as htf
from openhtf.plugs.user_input import UserInput
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.plug(prompts=UserInput)
def phase_fixture_setup(test, prompts):
    """Block until the operator confirms the fixture is ready."""
    prompts.prompt(
        message="Insert the board, close the lid, then press OK.",
        text_input=False,
    )


@htf.measures(
    htf.Measurement("visual_pass").equals(True),
)
@htf.plug(prompts=UserInput)
def phase_visual_inspection(test, prompts):
    """Collect a judgement and record it as a measurement."""
    response = prompts.prompt(
        message="Any visible defect? Type 'y' for yes, leave blank for none.",
        text_input=True,
    )
    test.measurements.visual_pass = not response.strip()


@htf.measures(
    htf.Measurement("screw_torque")
    .in_range(0.45, 0.55)
    .with_units(units.NEWTON_METRE),
)
@htf.plug(prompts=UserInput)
def phase_torque(test, prompts):
    """A typed value is still a measurement."""
    raw = prompts.prompt(
        message="Enter the measured screw torque in Nm:",
        text_input=True,
    )
    test.measurements.screw_torque = float(raw)


def main():
    test = htf.Test(
        phase_fixture_setup,
        phase_visual_inspection,
        phase_torque,
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial: "))


if __name__ == "__main__":
    main()
```

The torque phase is the pattern worth copying. A number an operator types is data, not flow control, so it becomes a measurement with limits and trends like any instrument reading. Discarding it, and recording only a boolean "operator confirmed", throws away the only record you had.

Add a timeout to any prompt that could block a line indefinitely:

```python filename="timeout_prompt.py"
prompts.prompt(
    message="Close the fixture lid, then press OK.",
    timeout_s=120,
)
```

## Running the Station Server

```bash filename="station_server.sh"
python main.py
```

With the Station Server enabled, the interface is at `http://localhost:12000`. It shows phases as they execute, measurements as they are recorded, and the final verdict.

Its practical limits on a production line:

- **One page, no kiosk.** An operator can navigate away or close the tab.
- **Text prompts only.** No checklists, sliders, images or structured forms.
- **No offline handling.** If the network drops, results are not queued.
- **No deployment story.** Updating a station means copying files.

## Getting a Richer Interface Without Rewriting

Existing OpenHTF suites run natively on the TofuPilot CLI:

```bash filename="run.sh"
tofupilot run ./main.py
```

That gives the same test a full-screen kiosk interface, real-time streaming, offline queueing and deploy-on-push, with no change to the test code. For most teams that is the shortest path from "the Station Server is not enough" to a production-grade screen.

## When to Move to Declarative UI

If you need input types OpenHTF does not have, the TofuPilot Framework declares the interface in the procedure rather than in phase code:

```yaml filename="procedure.yaml"
main:
  - name: Fixture Setup
    python: phases.fixture_setup
    ui:
      components:
        - key: instructions
          type: text
          label: "Insert the board and close the lid."
        - key: preflight
          type: checklist
          label: "Confirm before starting"
          required: true
          options:
            - label: "Board seated flat"
              value: "seated"
            - label: "Lid latched"
              value: "latched"

  - name: Torque Check
    python: phases.torque_check
    ui:
      components:
        - key: torque
          type: number_input
          label: "Screw torque (Nm)"
          required: true
          bind: measurements.screw_torque
    measurements:
      - name: Screw Torque
        unit: Nm
        validators:
          - operator: ">="
            expected_value: 0.45
          - operator: "<="
            expected_value: 0.55
```

```python filename="phases/torque_check.py"
def torque_check(measurements):
    """The bound UI value populates screw_torque directly."""
    pass
```

The `bind` field routes the operator's input straight into the measurement, so there is no parsing code and no chance of the value being collected and dropped.

Migration is optional and not free. See [migrate OpenHTF scripts to the TofuPilot CLI](/guides/migrate-openhtf-scripts-to-tofupilot-cli) and [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf).

## Unattended Runs

An OpenHTF test with prompts cannot run in CI without an operator. The usual approaches are a separate entry point that omits the prompt phases, or a plug that auto-answers when a flag is set:

```python filename="auto_prompts.py"
import os

import openhtf as htf
from openhtf.plugs.user_input import UserInput


class AutoInput(UserInput):
    """Answers prompts automatically when running unattended."""

    def prompt(self, message: str, text_input: bool = False, **kwargs):
        if os.environ.get("UNATTENDED") == "1":
            return ""
        return super().prompt(message, text_input=text_input, **kwargs)
```

Maintaining two entry points is the part that tends to drift. The declarative alternative keeps one procedure and supplies answers externally with `--ui-values`, which avoids that class of divergence.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Recording only "operator confirmed" | The typed value is lost forever | Make it a measurement with limits |
| Prompts with no timeout | A line blocks indefinitely | `timeout_s` on every prompt |
| Console `input()` on a station | No visible state, easy to mis-key | Use a prompt plug or the CLI interface |
| Station Server exposed as the line UI | Operators navigate away mid-test | Kiosk mode via the CLI |
| Two entry points for attended and CI | They drift apart | One procedure plus external UI values |
| Free-text where a choice belongs | Unparseable data, typos | Use a constrained input type |

### Build a Reliability Case with Test Data
URL: https://www.tofupilot.com/guides/build-a-reliability-case-with-test-data
Learn how to turn bench measurements into a reliability case for OEM design changes, using MTBUR gaps, drift trends, and Cpk evidence from TofuPilot.

The manufacturer sees a component once, at production. The operator sees it across its whole service life: every removal, every shop visit, every measured value on the bench, year after year. That asymmetry means airlines and MRO shops often understand a part's real failure behavior better than the OEM does. What they usually lack is the evidence in a form that carries weight. This guide shows how to build that case from your own bench data.

## The Data You Hold and the OEM Does Not

The industry already runs feedback pipes from operators to manufacturers. ATA Spec 2000 Chapter 11 standardizes the exchange of removal and shop-findings records; airframers publish component performance reports benchmarking MTBUR by part number across reporting fleets; component OEMs run programs that take monthly removal data and return corrective-action feedback. And continued-operational-safety processes depend on information supplied voluntarily by operators, which is the plainest statement of where the trend data originates.

But these pipes carry events: removals, findings, hours. They do not carry measurements. The OEM learns that your fleet removed 14 units of a part last year and that 10 tested no fault found. It does not learn that the passing units' output delay has been creeping toward the CMM limit for two years. That second fact is the design-change argument, and only your bench has it.

## MTBUR, MTBF, and the Gap

Two numbers frame every component reliability discussion:

- **MTBF** counts confirmed failures.
- **MTBUR** counts unscheduled removals, whether or not a fault was confirmed.

MTBUR is always the lower number, and the gap between them is mostly no-fault-found. The gap is also where the money is: purchase agreements commonly contain a guaranteed MTBUR that the operator can claim against, and rogue-unit clauses that trigger on documented repeat removals per serial number. Both claims stand or fall on the quality of the operator's own records.

## Structure Tests to Produce Evidence

A reliability case needs measured values, not verdicts. Each bench test should record the value, the unit, and the limit it was judged against:

```yaml filename="procedure.yaml"
main:
  - name: Signal Integrity
    python: phases.signal_check
    measurements:
      - name: output_delay
        unit: us
        validators:
          - operator: "<="
            expected_value: 12.0
      - name: signal_snr
        unit: dB
        validators:
          - operator: ">="
            expected_value: 40.0
```

With every shop visit recorded this way, the part number accumulates exactly the dataset a design-change argument is built from.

## Reading the Evidence

Four views in TofuPilot turn the accumulated runs into the case:

- **Distribution against limits.** If the fleet's values for a measurement crowd one limit rather than centering, the design margin is thinner in service than the spec assumed. A histogram of two years of inductions makes that visible in one image.
- **Cpk.** A Cpk below 1.0 on a CMM limit says the population does not reliably fit its own acceptance criteria. That is a number an OEM reliability engineer cannot wave away, because it is computed on their limits.
- **Drift over time.** A mean shifting across months of shop visits separates aging from randomness, and puts an onset date on the change. Onset dates correlate with mod status, supplier changes, and batch boundaries.
- **Per-serial history.** The repeat offenders, documented visit by visit. This is the direct input to guaranteed-MTBUR and rogue-unit claims, and the concrete illustration that makes the statistical case tangible.

## Making the Case

A design-change proposal built this way has a standard shape: the removal and NFF history (which the OEM partly knows), plus the measurement story (which it does not). For example: this part number shows MTBUR at half the guaranteed value; 60 percent of removals test NFF on the bench; but across 200 inductions the output delay distribution has shifted 20 percent toward the CMM limit, with onset eighteen months ago; the serials past a threshold of shop visits share the same drift signature.

That is no longer "our mechanics do not trust this box". It is a dataset the OEM's own reliability process can act on: a modification, a CMM limit revision, or a warranty settlement. Export the runs behind each figure and the case ships with its evidence attached.

## What Good Looks Like

An operator working this way stops being a passive reporter of removals and becomes the party at the table with the best data. Reliability reviews with the OEM start from the operator's distributions rather than the OEM's summaries. Warranty and guaranteed-MTBUR claims are filed with measurement evidence, not just removal counts. And when the OEM disputes a trend, the answer is the query, rerun in front of them.

### LabVIEW Alternatives for Test Automation
URL: https://www.tofupilot.com/guides/labview-alternatives-for-test-automation
Free and paid LabVIEW alternatives for test automation compared: OpenHTF, pytest, OpenTAP, and TofuPilot Framework, with what each is actually good at.

LabVIEW's graphical, dataflow programming model made sense in 1986 when it launched. For teams building production test today, the license cost and Windows-only deployment push many toward code-based alternatives. This guide compares the real options.

## Quick Comparison

| Tool | License cost | Language | Best for |
|------|--------------|----------|----------|
| OpenHTF | Free, open source | Python | Production test sequencing, Google-originated |
| pytest | Free, open source | Python | Teams that want CI/CD-native testing, less structure |
| OpenTAP | Free, open source | C# (Python plugin available) | Teams wanting a GUI sequencer closer to TestStand's model |
| TofuPilot Framework | Free, open source (MIT) | Python plus a YAML procedure | Production test with built-in data platform, no separate reporting layer to build |
| LabVIEW | $1,300-5,500+/seat/yr | G (graphical dataflow) | Heavy signal processing, teams with existing LabVIEW investment |

## OpenHTF

Built by Google for testing hardware at scale. Structures tests as ordered "phases," handles measurements with limits natively, and integrates directly with [TofuPilot](https://www.tofupilot.com) for result storage and analytics.

```python filename="test.py"
import openhtf as htf
from openhtf.util import units

@htf.measures(
    htf.Measurement("voltage_V").in_range(minimum=4.8, maximum=5.2).with_units(units.VOLT)
)
def phase_voltage_check(test):
    test.measurements.voltage_V = 5.02
```

Best for teams that want structure similar to what TestStand provides, without the license.

## pytest

The standard Python testing framework, built for software, adapted by many hardware teams for test sequencing. Flexible and CI/CD-native, but fixtures and parametrize don't map cleanly onto hardware sequences the way OpenHTF's phase model does, so teams often end up building their own layer on top. See [OpenHTF vs pytest for Hardware Testing](/guides/openhtf-vs-pytest-for-hardware-testing-which-should-you-choose) for the tradeoffs in depth.

## OpenTAP

Started at Keysight, now open source. Closest match to TestStand's model: a visual test plan editor, plugin architecture, C# core with a Python plugin path. Better fit for teams that specifically want a GUI sequence builder and are comfortable with .NET deployment, which is a heavier lift on Linux than a pure Python stack.

## TofuPilot Framework

An open-source test orchestrator built specifically for production hardware test, with [TofuPilot Studio](https://www.tofupilot.com/products/studio) as the editor for building test procedures. Where OpenHTF and pytest leave you to build your own results database and analytics, TofuPilot Framework ships that layer (first pass yield, measurement distributions, failure Pareto) out of the box.

A station is a `procedure.yaml` that declares the sequence, the measurements and their limits, plus plain Python phases:

```yaml filename="procedure.yaml"
# The sequence and the limits live here, not in the code
name: Voltage Check
version: 1.0.0

main:
  - name: Voltage Check
    python: phases.voltage_check
    measurements:
      - name: voltage_V
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
```

```python filename="phases/voltage_check.py"
# The phase only records the value; the validators above decide pass or fail
def voltage_check(measurements):
    measurements.voltage_V = 5.02
```

Run it with `tofupilot run ./procedure.yaml`, no account needed, or add `--upload` to send every run to the dashboard.

## What You Give Up Leaving LabVIEW

- **Drag-and-drop wiring.** Code-based tools require actual programming, a real cost for teams without Python skills on staff
- **Mature DAQ toolkits.** LabVIEW's signal processing and NI hardware integration libraries are deep; Python equivalents (PyVISA, NI-DAQmx Python bindings) cover most cases but not all
- **NI's support contracts.** Relevant in regulated environments where a certified vendor relationship is itself part of the compliance story

## Frequently Asked Questions

**Is there a truly free LabVIEW alternative for production use?**
Yes. OpenHTF, pytest, and TofuPilot Framework are all open source with no license fees, unlike LabVIEW Community Edition which is free but restricted to non-commercial use.

**Which LabVIEW alternative is closest to TestStand?**
OpenTAP, for its visual sequence editor. OpenHTF and TofuPilot Framework are code-first with less GUI tooling, but faster to version-control and deploy via CI/CD.

**Do I need to rewrite everything to switch off LabVIEW?**
Existing instrument drivers and fixtures carry over. PyVISA and SCPI control the same instruments LabVIEW does. What changes is the sequencing and data layer, not the hardware.

**Can I run a LabVIEW alternative on Linux?**
OpenHTF, pytest, and TofuPilot Framework all run cross-platform (Windows, Linux, macOS). OpenTAP's .NET core makes Linux deployment more involved. LabVIEW itself is Windows-only.

### Import Historical Test Data via the API
URL: https://www.tofupilot.com/guides/import-historical-test-data-via-the-api
Learn how to backfill years of legacy test results with the runs API, keeping original test dates so trends and analytics stay correct.

Switching test data platforms should not mean starting the fleet history from zero. Trend charts, per-serial degradation curves, and capability indices are only as good as the history behind them. This guide shows how to backfill legacy results through the runs API with their original test dates intact.

The key mechanism: a run's `started_at` is the timestamp analytics use, and the API accepts past dates. Upload date is stored separately, so a run tested in 2024 and imported today lands in 2024 on every chart.

## Prerequisites

- An API key (Settings, API keys)
- The procedure created in TofuPilot, and its procedure id
- Legacy results exported to something parseable (CSV, database dump, report files)

## Step 1: Map the legacy fields

The minimum viable mapping:

| Legacy field | Runs API field |
|---|---|
| Test date/time | `started_at`, `ended_at` (ISO 8601) |
| Overall verdict | `outcome` (PASS, FAIL, ERROR) |
| Unit serial number | `serial_number` |
| Part number | `part_number` |
| Test steps | `phases[]` with name, outcome, timestamps |
| Measured values and limits | `measurements[]` per phase, with validators |

Keep phase and measurement names identical to what the live bench will record going forward. Analytics group by name, so `Contact Drop A2` in the import and `contact_drop_a2` from the new bench would land in two different series.

## Step 2: Create runs with original timestamps

```python filename="import_legacy.py"
from tofupilot import TofuPilot

client = TofuPilot(api_key="...")

for row in legacy_rows:
    client.runs.create(
        procedure_id="550e8400-e29b-41d4-a716-446655440000",
        serial_number=row.serial,
        part_number=row.part_number,
        outcome=row.verdict,                      # "PASS" / "FAIL"
        started_at=row.tested_at,                 # original date, ISO 8601
        ended_at=row.finished_at,
        phases=[{
            "name": "Contact Voltage Drop",
            "outcome": row.phase_verdict,
            "started_at": row.tested_at,
            "ended_at": row.finished_at,
            "measurements": [{
                "name": "Contact Drop A2",
                "outcome": row.a2_verdict,
                "measured_value": row.a2_mv,
                "units": "mV",
                "validators": [{"operator": "<=", "expected_value": 100,
                                "outcome": row.a2_verdict}],
            }],
        }],
    )
```

Units are created automatically from serial numbers, so a serial with five historical visits ends up with a five-run history without any separate unit setup.

## Step 3: Verify in analytics

Set the date range in run analytics or measurement control to cover the imported period and check three things:

1. Run counts per month match the legacy system
2. Per-serial history is complete: pick a unit with known repeat visits and confirm every visit is there
3. Limits came through: measurements show their validators, so Cpk and pass rates compute against the right limits

## Step 4: Cut over the live bench

Once history is verified, point the live bench at the same procedure. New runs continue the same series: the first live run on an imported serial extends its existing trend rather than starting a new one, which is the entire point of importing with real timestamps.

Two details worth knowing:

- Imported runs record the API key's user as creator, and the upload date as `created_at`. Both are visible on the run detail page; neither affects analytics, which use `started_at`.
- Station attribution requires uploading with a station key. For historical data this rarely matters, but record the original site or bench as a measurement or run metadata if you need to filter by it later.

### LabVIEW Pricing: What It Actually Costs
URL: https://www.tofupilot.com/guides/labview-pricing-what-it-actually-costs
LabVIEW pricing broken down by license tier, module add-ons, and NI TestStand costs, plus what a Python-based test stack costs by comparison.

LabVIEW pricing isn't a single number. NI (National Instruments, now part of Emerson) sells it as a base license plus a stack of optional modules, and most real test setups need more than the base. This guide breaks down what you actually pay, tier by tier, and what the equivalent Python-based stack costs.

## LabVIEW License Tiers

| Tier | List price (approx.) | What's included |
|------|----------------------|------------------|
| LabVIEW Community Edition | Free | Non-commercial use only: hobbyists, students, home use |
| LabVIEW Base | ~$1,300/year | Core development environment, no advanced analysis or connectivity toolkits |
| LabVIEW Full | ~$4,300/year | Adds signal processing, advanced analysis, some connectivity |
| LabVIEW Professional | ~$5,500+/year | Adds source control integration, application builder, code distribution |

Prices are per-seat, per-year on NI's subscription model (NI moved from perpetual licenses to annual subscriptions in 2023; an existing perpetual license still runs, but no longer gets updates or support without a subscription add-on). Multiply by every engineer who needs to open or edit a VI, not just the number of test stations.

## What Community Edition Doesn't Cover

LabVIEW Community Edition is genuinely free, but the license restricts it to non-commercial use. Test code that ships product, runs in a factory, or supports a commercial product isn't covered, which rules it out for the exact use case most people search for it to solve. See [LabVIEW Community Edition: Download & Rules](/guides/labview-community-edition-download-rules) for the full breakdown.

## NI TestStand: A Separate License

TestStand is NI's test sequencer, sold separately from LabVIEW itself:

| Item | Approx. cost |
|------|--------------|
| TestStand Base | ~$2,500-3,000/seat |
| TestStand deployment license (per station) | ~$1,000-1,500/station |

A single engineering seat with LabVIEW Professional + TestStand Base runs **$8,000-9,000/year**. That's before deployment licenses for every production station running the sequence.

## What a Multi-Station Line Actually Costs

| Setup | LabVIEW/TestStand | Python-based |
|-------|--------------------|--------------|
| 1 engineer, 1 dev seat | $5,500-9,000/yr | $0 |
| 5 production stations (deployment licenses) | $5,000-7,500/yr | $0 |
| Data storage / reporting | Custom, often built in-house | Included with [TofuPilot](https://www.tofupilot.com) |
| **Year 1 total (5 stations)** | **$10,500-16,500** | **$0 in software licenses** |

Python-based test stacks (OpenHTF or [TofuPilot Framework](https://www.tofupilot.com/products/framework) for sequencing, PyVISA for instrument control) run on the same instruments and fixtures with no per-seat or per-station license cost. The cost that remains is engineering time to build the initial test scripts, which LabVIEW's drag-and-drop model can reduce for teams without strong Python skills. That tradeoff, not price alone, is the real decision. See [Python vs LabVIEW for Manufacturing Test](/guides/python-vs-labview-for-manufacturing-test-a-practical-comparison) for the full comparison beyond cost.

## When LabVIEW's Cost Is Justified

- Your team already has deep LabVIEW expertise and switching costs (retraining, rewriting existing test suites) exceed the license savings
- You're doing heavy signal processing or DAQ-hardware-specific work where LabVIEW's toolkits are mature and Python equivalents are thinner
- You need NI's certified support contracts for regulated environments where that certification is itself a requirement

## Frequently Asked Questions

**Is LabVIEW a one-time purchase or subscription?**
Since 2023, NI sells LabVIEW as an annual subscription. Older perpetual licenses still function but don't receive new versions or support without adding a subscription.

**Does LabVIEW pricing include TestStand?**
No. TestStand is a separate product with its own license, typically bought alongside LabVIEW for production test sequencing.

**Is there a cheaper LabVIEW tier for production use?**
LabVIEW Base is the lowest commercial tier, but it lacks toolkits (signal processing, application builder) that most production test setups end up needing. Teams often start on Base and get pushed to Full or Professional within a year.

**What's the real cost difference vs. Python for a small team?**
For a single station, the license gap is a few thousand dollars a year: meaningful but not dramatic. The gap widens fast as you add engineers and stations, since Python's license cost stays at zero regardless of scale.

### LabVIEW Community Edition: Download & Rules
URL: https://www.tofupilot.com/guides/labview-community-edition-download-rules
How to download LabVIEW Community Edition, what its free license actually allows, and where it breaks down for commercial or production test use.

LabVIEW Community Edition is NI's free tier of LabVIEW, launched to replace the older LabVIEW Home Edition. It's a real, fully-featured version of the development environment. The catch is entirely in the license terms, not the software itself.

## Where to Download It

LabVIEW Community Edition is available directly from NI's website, requiring a free NI account to register. It installs like any other LabVIEW edition and includes the core development environment.

## What the License Actually Allows

The Community Edition license restricts use to:

- Personal, non-commercial projects
- Students and educational use
- Hobbyist and home automation projects
- Open-source projects (with conditions)

It explicitly **does not** cover:

- Commercial product development
- Test code used in a business, including internal tools that support a commercial product
- Any use where the output supports revenue-generating activity

This is the detail most searches for "LabVIEW free" miss. The software downloads and runs the same as a paid tier, but the license, not the software, is what determines whether you're allowed to use it for work.

## What This Means for Test Engineers

If you're building a production test station for a company, Community Edition's license doesn't cover you, even if the station itself never leaves the building and generates no direct revenue, because it exists to support a commercial product. For that, you need at minimum LabVIEW Base. See [LabVIEW Pricing: What It Actually Costs](/guides/labview-pricing-what-it-actually-costs) for the paid tiers.

## Free Alternatives That Do Cover Commercial Use

If the goal is avoiding license cost rather than specifically wanting LabVIEW, the actual free-for-commercial-use options are code-based:

- **OpenHTF**: open source (Apache 2.0), no restriction on commercial use
- **pytest**: open source (MIT), no restriction
- **TofuPilot Framework**: open source (MIT), no restriction

None of these have Community Edition's non-commercial catch, because open-source licenses and NI's free-tier license are structured differently. One gives away the software with usage restrictions, the other gives away the software with none. See [LabVIEW Alternatives for Test Automation](/guides/labview-alternatives-for-test-automation) for a full comparison.

## Frequently Asked Questions

**Is LabVIEW Community Edition the same software as the paid version?**
Functionally yes, it's the same development environment. The difference is licensing terms, not features, though some advanced toolkits sold separately still require a paid license to unlock.

**Can I use LabVIEW Community Edition at a startup?**
No, if the code supports the company's product or operations in any way. That counts as commercial use under the license, regardless of company size or revenue.

**Does LabVIEW Community Edition expire?**
It doesn't expire in the way a trial does, but NI can change licensing terms in future releases, and it doesn't include the update and support entitlements a subscription does.

**What happens if I outgrow Community Edition?**
You upgrade to a paid tier (Base, Full, or Professional). The license, project files, and code you've written carry over; only the usage rights change.

### Getting Started with TofuPilot for Aviation MRO
URL: https://www.tofupilot.com/guides/getting-started-with-tofupilot-for-aviation-mro
Set up the TofuPilot CLI, write a first component bench test, and run it locally in an aviation MRO shop, with no account or proprietary data needed.

Component maintenance shops build their test benches in-house, and the software side is usually a mix of instrument-control scripts, spreadsheets, and printouts. TofuPilot adds a structured layer on top of the scripts you already write: tests defined in a YAML procedure, phases in plain Python, and every measurement recorded with its limits and unit identity. This tutorial takes you from nothing to a running component bench test, entirely locally. No account is needed to start, and nothing here touches OEM software or proprietary data; the framework only handles measurements your own bench produces.

## Prerequisites

- A machine running Windows, Linux, or macOS
- Python installed (the CLI can provision a virtual environment for you)

## Step 1: Install the CLI

The TofuPilot CLI is a single executable. The framework and CLI are both open source under the MIT license.

On Linux or macOS:

```bash
curl -fsSL https://tofupilot.app/install | sh
```

On Windows (PowerShell):

```powershell
$p = "$env:TEMP\tp-install.ps1"; irm https://tofupilot.app/install.ps1 -OutFile $p; powershell -ExecutionPolicy Bypass -File $p; ri $p -EA 0
```

## Step 2: Create the Procedure

A procedure is a directory with a `procedure.yaml` declaring the test sequence, plus the Python it references. Create this layout:

```text
lru-bench-test/
  procedure.yaml
  phases/
    power_check.py
  plugs/
    bench.py
```

The YAML declares one phase with one validated measurement, and identifies the unit under test by serial number and part number, which is what ties every result to a unit's history:

```yaml filename="procedure.yaml"
name: LRU Bench Acceptance Test
version: 1.0.0
description: First bench test for a returned component

unit:
  auto_identify: true
  serial_number:
    default_value: "LRU000001"
  part_number:
    default_value: "622-1234-001"

plugs:
  - name: bench
    python: plugs.bench:Bench

main:
  - name: Power Supply Check
    python: phases.power_check
    measurements:
      - name: supply_voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 27.5
          - operator: "<="
            expected_value: 28.5
```

## Step 3: Write the Phase and the Plug

The phase is a plain Python function. It receives the measurements object and any plugs it needs by name:

```python filename="phases/power_check.py"
def power_check(measurements, bench):
    measurements.supply_voltage = bench.read_supply_voltage()
```

The plug is a class wrapping your instrument. This one returns a fixed value so the example runs anywhere; on a real bench the same method would read from your power supply over VISA or serial:

```python filename="plugs/bench.py"
class Bench:
    def read_supply_voltage(self) -> float:
        return 28.01
```

This split is the pattern for everything that follows: procedures declare what is measured and what its limits are, phases contain the test logic, plugs contain the instrument drivers. Your existing instrument-control code moves into plugs mostly unchanged.

## Step 4: Run It

From the parent directory:

```bash
tofupilot run ./lru-bench-test
```

The CLI resolves a Python environment (offering to create one if missing), executes the phases, evaluates each measurement against its validators, and prints the result:

```text
→ Phase: Power Supply Check
✓ Phase Power Supply Check: PASS
→   supply_voltage = 28.01 V [PASS]
✓ Run complete: PASS
```

The run executed entirely on your machine, without contacting anything.

## Step 5: Connect the Dashboard

When you want history and analytics on top of local runs, connect the procedure to a dashboard. Runs uploaded from then on build the part number's yield and measurement trends and each serial number's visit history.

```bash
tofupilot login
tofupilot link ./lru-bench-test
tofupilot run ./lru-bench-test --upload
```

Low-volume use fits the free tier, which suits component maintenance well: a shop seeing a handful of units per part number per month generates far fewer runs than a production line, while each run carries more history value.

From here, grow the procedure the way your CMM test grows: more phases for more functional areas, more measurements per phase, plugs for each instrument on the bench. The measurements accumulate into the part number's record, which is where the analytics for no-fault-found, drift, and repeat visitors come from.

### Record Test Conditions as Measurements
URL: https://www.tofupilot.com/guides/record-test-conditions-as-measurements
Learn how to capture supply voltages, temperatures, and other test conditions as measurements so they chart, export, and correlate.

A common question: can metadata be attached to an individual measurement? For example, Signal A measured 1 V, and the power supply was at 28 V / 10 mA at that moment.

There is no metadata field on individual measurements, and for capture conditions that is the wrong tool anyway. The supply voltage at the moment of capture is itself a measurement. Record it as one, and it becomes first-class data: charted in measurement control, filterable, exportable, and available to correlation analysis. An annotation would be none of those things.

The rule of thumb:

| The value... | Record it as |
|---|---|
| Can vary during or between tests (supply voltage, ambient temperature, load current) | A measurement in the same phase |
| Is constant for the whole run and not measurable (batch number, fixture id, operator shift) | Run metadata |
| Is free-form documentation | A docstring |

## Step 1: Measure the conditions alongside the signal

Record the conditions in the same phase as the signal they contextualize:

```python filename="phases/signal_check.py"
def signal_check(measurements, supply, dmm):
    # Conditions first: recorded as measurements, not annotations
    measurements.supply_voltage = supply.measured_voltage()   # V
    measurements.supply_current = supply.measured_current()   # mA

    # The signal itself
    measurements.signal_a = dmm.measure_dc_volts("A")
```

Conditions can carry validators too. A supply reading outside its expected window is itself a test result: it tells you the measurement was taken under the wrong conditions, before anyone spends an afternoon debugging the unit.

## Step 2: Use run metadata for run-constant context

For values that are constant across the whole run, attach run metadata as key/value pairs at upload. Typical keys: batch, fixture, bench id, firmware under test. Run metadata is filterable in run lists and through the API, but it is per run, not per measurement.

## Step 3: Correlate conditions with results

Because conditions are measurements, they participate in the same analytics as everything else. If signal A drifts and supply voltage drifts with it, both series show it in measurement control over the same time axis. If a value deviation only appears under certain conditions, the conditions are in the data, so the relationship can actually be checked instead of remembered.

The general principle: correlations can only be computed across things recorded as data. Every condition the bench records as a measurement is one more thing analytics can check without anyone having to know in advance to look.

### Setting Alert Thresholds That Actually Work
URL: https://www.tofupilot.com/guides/setting-alert-thresholds-that-actually-work
Why a fixed yield threshold fails on real production lines, and how a self-calibrating EWMA detector finds the right band for each one.

Your line is testing units all day long and you track its first-pass yield (FPY). At what point should someone be alerted? The intuitive answer, picking a threshold, runs into two walls immediately.

The first is level. A procedure running at 99% FPY should alert at 98%, and loudly at 97%: every point lost there is significant. A procedure running at 50% (like a deliberately harsh burn-in for example) should only alert around 45%, and loudly at 40%. No absolute threshold fits both.

The second is intrinsic variability. Two lines can share the same average FPY of 55%: one is very stable and stays within ±1 point; the other naturally swings between 50% and 60% depending on lots and shifts. A drop to 48% is a major event on the first, an ordinary Tuesday on the second. The relevant threshold therefore depends not only on the level, but on each line's usual behaviour.

A trustworthy alerting system has to calibrate itself on both dimensions, for every procedure, without asking the user to guess constants. Here is how statistics goes about it, and the choices we drew from it.

## The state of the art: detecting a change in a stream of tests

The problem has had a name for a century: statistical process control (SPC), founded by Walter Shewhart in the 1920s. The central idea of it is: every process has routine variation (its "common causes"), and you should only alert when an observation leaves that routine (a "special cause"). Shewhart charts judge each point in isolation. There are two more recent methods, which are the current references, that add the decisive ingredient for small drifts: memory.

### CUSUM: the evidence accumulator

The CUSUM chart (cumulative sum, Page, 1954) processes each tested unit, one by one, and maintains a counter of evidence. An electronics analogy: an integrator with a constant leak and a floor at zero. Every failure injects charge, every pass lets a little leak away. On a healthy process the leak wins and the counter sits at zero. After a genuine drift the injections take over and the counter increases to the alarm threshold. The version adapted to pass/fail streams under 100% inspection (the Bernoulli CUSUM, Reynolds & Stoumbos, 1999) is explicitly presented by its authors as superior to the practice of grouping units into samples or time windows. For a given false-alarm rate, it is the fastest detector for the drift it was specified for (Moustakides, 1986).

In one line, on unit i, with x[i] = 1 for a failure and 0 for a pass:

```text
S[i] = max(0, S[i−1] + w[i]),  S[0] = 0,  alarm when S[i] ≥ h
```

```text
w[i] = ln(p1/p0) if unit i fails,  ln((1−p1)/(1−p0)) if it passes
```

Here p0 is the failure rate you accept and p1 the one you want to detect, so w[i] is the log-likelihood ratio between those two hypotheses. Because p1 > p0, a failure contributes a positive weight and a pass a negative one: the charge injection and the leak of the analogy are literally the two branches of that formula. The max(0, …) is the floor at zero, and h is the alarm threshold, chosen from the false-alarm rate you are willing to pay.

### EWMA: smoothed yield

The EWMA chart (exponentially weighted moving average, Roberts, 1959) is a low-pass filter applied to the pass/fail pulse train: its output is a smoothed estimate of the current failure rate, where each unit weighs slightly less than the next one, so old data fades gently instead of falling off a cliff as it does in a sliding window. A single parameter, λ, sets the memory depth, expressed in units produced rather than clock time: a fast line covers its memory in a few hours, a slow line in a few days, and both are monitored with the same statistical rigour. You alert when the smoothed yield departs too far from its reference value, the gap being measured in standard deviations.

The same in one line:

```text
Z[i] = λ·x[i] + (1−λ)·Z[i−1],  Z[0] = p0
```

Z[i] is the smoothed failure rate, so the smoothed yield displayed to the user is 1 − Z[i]. The textbook band around it comes from the asymptotic dispersion of that filter under the binomial assumption:

```text
σ_Z = √( λ/(2−λ) · p0(1−p0) ),  alert when |Z[i] − p0| > L·σ_Z
```

The λ/(2−λ) factor is also where the memory depth becomes concrete: averaging that many values gives the same dispersion, so the filter is worth an equivalent sample of (2−λ)/λ units, about 99 at λ = 0.02. That σ_Z formula is the one term the rest of the article replaces with a measured value.

The difference in nature matters for what follows: EWMA is a gauge (its output reads in the unit of the metric, "smoothed FPY: 91.2%"), CUSUM is an accumulator (its output, "2.9 units of evidence toward a limit of 4", is less easily interpreted by a human eye).

### What about the other methods?

Others exist, such as self-sizing windows from data stream mining (ADWIN), Bayesian changepoint detection and adaptive sampling charts, which we will not explore here: they are either designed for other data regimes, or considerably harder to explain and audit for a quality engineer, which matters when an alert is supposed to trigger action on a production floor. References are at the end of the article for the curious.

## The textbook trap: one noise model for every line

Open an SPC textbook and the formulas for CUSUM as well as EWMA on pass/fail data rest on the same assumption: the noise is binomial. In other words, every unit would have exactly the same probability of failure, and a line's variability would follow entirely from its average level through a formula, without ever being measured.

Real production does not honour that contract. The true failure probability wanders from hour to hour: shift changes, component lots, temperature, tooling wear. As a result the observed variability exceeds, sometimes by a lot, what the formula predicts. The phenomenon is named overdispersion, and it is the best-documented defect of attribute charts applied as-is in a factory (it is the problem Laney's p′ chart, 2002, was invented to correct). During our training phase on real production datasets, we measured variability ranging from 1× to more than 5× the binomial prediction depending on the line, on procedures that were all perfectly healthy.

The consequences of a textbook setting are obvious. Calibrate the thresholds for the fluctuating line and they go blind on the stable one, real drops slip under the radar. Calibrate them for the stable line and the fluctuating one fires false alerts in bursts, and everyone knows what follows, documented in every factory in the world: the alerts end up ignored, including the true ones. This is precisely why "picking a threshold" is so hard: there is no good global threshold, only a good threshold per line and test procedure, and it changes when the line evolves.

## Why EWMA answers this better

Because the EWMA filter's output is a gauge in the unit of the metric, one correction suggests itself naturally: instead of computing its dispersion theoretically from the binomial formula, measure it. Replay the filter over the line's own history, observe the real distribution of its values, and the alert band becomes: "the smoothed yield departs by N observed standard deviations from its own historical behaviour". Both axes of the problem are then covered at once:

- **Level:** The reference is estimated per procedure, and the band's natural width follows the level. The noise of a pass/fail stream is √(p(1−p)): it is mechanically finer at 99% FPY than at 50%. The excellent line is monitored to the point, the difficult line to the multiple of points.
- **Intrinsic variability:** The line that sits steady at 55% has a tight smoothed history, so a narrow band; the line swinging between 50% and 60% has a wandering smoothed history, so a wide one. Each is judged against itself.

Can the same correction be applied to CUSUM? Not so easily, and this is the technical point that separates the two methods: CUSUM's entire calibration (the alarm threshold, the optimality guarantee, the advertised false-alarm rate) is welded to the binomial model by the very construction of its evidence counter. Under overdispersion those guarantees evaporate, and the variants that restore them require fitting an additional statistical model per line and recalibrating by simulation. Workable in a study, not in a product monitoring hundreds of heterogeneous procedures without human intervention.

Then there is the argument we care about most: what the user sees. An EWMA alert shows itself. The smoothed yield curve is plotted in the unit the quality engineer knows, with its band of normality around it, and the alert reads off it: "smoothed FPY at 91.2% against a normal of 95.1% ± 1.2". Every number is recomputable by hand from the runs. A CUSUM alert has to be translated: the evidence counter has no meaningful axis, and the counts that explain it must be reconstructed after the fact. For a system whose goal is trust, so that a quality engineer can audit why they were woken up, the gauge beats the accumulator.

The same principle applies to individual measurements, not only to yield: each numeric series is graded against its own measured baseline, on the same σ logic. See [How to Detect Measurement Drift with TofuPilot](/guides/how-to-detect-measurement-drift-with-tofupilot) for what that looks like on a measurement rather than on a rate.

## What is left to fix, and why we trained it on real data

Self-calibration does not remove every constant; it changes their nature. Everything that depends on the line (reference level, variability) is measured continuously. Two universal settings remain, the same for everyone:

- **λ (the memory depth):** In reality a choice of boundary between what counts as weather (fast fluctuations, absorbed into the band) and what counts as change (sustained drifts, which alert). It is chosen through a promise about reaction time: "confirm a sustained drift within one to two hundred units".
- **The severity scale:** The standard-deviation multipliers separating info, warning and critical. This is a false-alarm budget: at what historical rarity does something deserve to disturb someone?

How do you set these values without falling back into arbitrariness? By training them: replay the candidate detector, setting by setting, over real production datasets, tens of thousands of tests covering widely varied profiles (lines at 99% as well as 50% FPY, production in bursts of several hundred units a day as well as steady lines running a few units a day). Then count: how many alerts each setting would have raised, at what severity, and whether the confirmed incidents would have been detected, and within how many units.

That training settled three things. First λ = 0.02 (a useful memory of roughly 50 to 150 units): shorter settings widen the band faster than they deepen the excursions and lose the critical signal on real incidents; longer ones double the reaction time with no detection gain. Next, the 2σ/3σ/5σ scale: over two months of real data, 2σ corresponds to a handful of informative episodes per line, 3σ almost exclusively to genuine events, and 5σ exclusively to one authentic incident (an FPY that fell from 97% to 80% over about a hundred units, classified critical by the detector). Finally a guard rail: the measured band is floored by the binomial minimum, so that a flawless line (100% FPY over thousands of units, zero observed dispersion) cannot turn its first failure into an "infinite" alarm.

One last check, from outside the data: we gave sample datasets to an experienced quality engineer and asked at which levels they would have wanted to be alerted. With λ = 0.02, the computed band lands on their answers: at 99% FPY, alert at 98% and strong alert at 97%; at 50%, alert at 45% and strong at 40%. A calibration derived from replay and a field intuition converging without having consulted each other is the kind of agreement you look for.

And the training is not a one-off event: the replay harness is kept, and every evolution of the detector goes through it again before being deployed.

## Sources and further reading

Main methods:

- E. S. Page, *Continuous Inspection Schemes*, **Biometrika** 41(1-2), 1954: the founding paper of CUSUM. <https://academic.oup.com/biomet/article/41/1-2/100/246907>
- S. W. Roberts, *Control Chart Tests Based on Geometric Moving Averages*, **Technometrics** 1(3), 1959: the founding paper of EWMA. <https://doi.org/10.1080/00401706.1959.10489860>
- G. V. Moustakides, *Optimal Stopping Times for Detecting Changes in Distributions*, **Annals of Statistics** 14(4), 1986: the CUSUM optimality proof (open access). <https://projecteuclid.org/journals/annals-of-statistics/volume-14/issue-4/Optimal-Stopping-Times-for-Detecting-Changes-in-Distributions/10.1214/aos/1176350164.full>
- M. R. Reynolds Jr. & Z. G. Stoumbos, *A CUSUM Chart for Monitoring a Proportion When Inspecting Continuously*, **Journal of Quality Technology** 31(1), 1999: the Bernoulli CUSUM for pass/fail streams. <https://doi.org/10.1080/00224065.1999.11979900>
- D. B. Laney, *Improved Control Charts for Attributes*, **Quality Engineering** 14(4), 2002: overdispersion and the p′ chart. <https://doi.org/10.1081/QEN-120003555>
- NIST/SEMATECH, *e-Handbook of Statistical Methods*, EWMA Control Charts section: the practical online reference. <https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm>
- D. C. Montgomery, *Introduction to Statistical Quality Control*, Wiley: the field's reference work (rational subgroups, attribute charts, CUSUM/EWMA).
- W. A. Shewhart, *Economic Control of Quality of Manufactured Product*, 1931: the founding work of SPC.

The other methods mentioned:

- A. Bifet & R. Gavaldà, *Learning from Time-Changing Data with Adaptive Windowing*, **SIAM International Conference on Data Mining**, 2007: self-sizing windows (ADWIN). <https://epubs.siam.org/doi/10.1137/1.9781611972771.42>
- R. P. Adams & D. J. C. MacKay, *Bayesian Online Changepoint Detection*, arXiv, 2007: Bayesian changepoint detection. <https://arxiv.org/abs/0710.3742>
- *A Variable Sampling Interval Synthetic X̄ Chart for the Process Mean*, **PLoS ONE** 10(5), 2015: an example of an adaptive sampling chart. <https://pmc.ncbi.nlm.nih.gov/articles/PMC4423939/>

### How to Store Manufacturing Test Results
URL: https://www.tofupilot.com/guides/how-to-store-manufacturing-test-results
A practical guide to storing production test results: flat files, SQL schemas, STDF and ATML standards, and when each approach stops scaling.

Ask how to store manufacturing test results and the most common answer you will find in forums is that there is no industry standard. That answer is half right. There is no single standard covering board-level functional test the way STDF covers semiconductor test. But there are well-understood approaches, each with a known point at which it stops working.

This guide covers four of them in the order teams usually encounter them, and says plainly where each one breaks.

## What You Actually Need to Store

Before choosing a format, be clear about the entities. Almost every team converges on the same five:

| Entity | What it is |
|---|---|
| **Unit** | A physical thing with a serial number |
| **Run** | One execution of a test procedure against one unit |
| **Step** | One phase within a run, with its own pass or fail |
| **Measurement** | A named value with units and limits |
| **Procedure** | The versioned definition of what was tested |

The two that get missed early and hurt later are **procedure version** and **sub-assembly relationships**. If you cannot answer "which version of the test produced this result" or "which PCBA went into this final assembly", you will rebuild your schema.

## Approach 1: Flat Files

CSV or JSON written to a network share, one file per run.

```python filename="log_to_csv.py"
import csv
from datetime import datetime, timezone


def log_run(serial, procedure, measurements, passed):
    stamp = datetime.now(timezone.utc).isoformat()
    path = f"//test-share/results/{serial}_{stamp}.csv"

    with open(path, "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["serial", "procedure", "measurement", "value", "unit", "low", "high", "passed"])
        for m in measurements:
            writer.writerow([
                serial, procedure, m["name"], m["value"],
                m["unit"], m["low"], m["high"], m["passed"],
            ])
```

**Works until:** you need to answer a question across files. First-pass yield for last month means parsing thousands of files. A schema change means old files no longer parse the same way. Two stations writing at once eventually corrupt something. Nobody can find the file for a specific serial without a naming convention that itself becomes a schema.

**Realistic ceiling:** one station, low volume, or a genuinely temporary setup.

## Approach 2: A SQL Schema You Own

The natural next step, and the right answer for many teams.

```sql filename="schema.sql"
CREATE TABLE unit (
    id            BIGSERIAL PRIMARY KEY,
    serial_number TEXT NOT NULL UNIQUE,
    part_number   TEXT NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE run (
    id                BIGSERIAL PRIMARY KEY,
    unit_id           BIGINT NOT NULL REFERENCES unit(id),
    procedure_id      TEXT NOT NULL,
    procedure_version TEXT NOT NULL,
    station_id        TEXT NOT NULL,
    started_at        TIMESTAMPTZ NOT NULL,
    duration_s        NUMERIC NOT NULL,
    passed            BOOLEAN NOT NULL
);

CREATE TABLE measurement (
    id          BIGSERIAL PRIMARY KEY,
    run_id      BIGINT NOT NULL REFERENCES run(id) ON DELETE CASCADE,
    step_name   TEXT NOT NULL,
    name        TEXT NOT NULL,
    value       DOUBLE PRECISION,
    unit        TEXT,
    lower_limit DOUBLE PRECISION,
    upper_limit DOUBLE PRECISION,
    passed      BOOLEAN NOT NULL
);

CREATE INDEX ON run (procedure_id, started_at DESC);
CREATE INDEX ON measurement (run_id);
CREATE INDEX ON measurement (name, run_id);
```

First-pass yield then becomes a query rather than a script:

```sql filename="first_pass_yield.sql"
SELECT
    date_trunc('day', first_run.started_at) AS day,
    count(*) FILTER (WHERE first_run.passed) * 100.0 / count(*) AS fpy_percent,
    count(*) AS units_tested
FROM (
    SELECT DISTINCT ON (unit_id)
        unit_id, started_at, passed
    FROM run
    WHERE procedure_id = 'FCT-001'
    ORDER BY unit_id, started_at ASC
) AS first_run
GROUP BY day
ORDER BY day DESC;
```

Note the `DISTINCT ON` and the ascending sort. First-pass yield means the *first* run per unit, not any passing run. Getting this wrong is the single most common reporting bug in home-built systems, and it always inflates the number.

**Works until:** the analytics backlog grows faster than you can clear it. Cpk, control charts with proper limits, Pareto of failure modes, drift detection, per-station comparison, and operator-facing reports are each a small project. Then someone asks for the same numbers across three sites with intermittent connectivity.

**Realistic ceiling:** healthy for a long time if someone owns it. The cost is ongoing engineering, not licensing.

## Approach 3: Industry Standards

Two standards matter, and neither covers everything.

**STDF (Standard Test Data Format)** is the semiconductor standard from Teradyne, now widely supported by ATE. It is binary, compact, and assumes wafers, dies, and bins. If you are testing silicon, use it. If you are testing assembled boards, its data model does not fit and forcing it costs more than it saves.

**ATML (Automatic Test Markup Language)** is the IEEE 1671 family, XML-based, and designed for test description and results interchange, largely in aerospace and defense. It is thorough and verbose. It is most useful when a contract requires it.

There is also **WSJF and WSXF**, which are WATS-specific rather than open standards, and worth knowing about mainly because migrating off WATS means dealing with them.

**The honest summary:** for board-level functional test, no standard has achieved the adoption STDF has in semiconductors. Most teams end up with a proprietary schema, whether their own or a vendor's.

## Approach 4: A Test Data Platform

Use a system built for this, so the schema and the analytics come with it.

```python filename="functional_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
    htf.Measurement("current_draw")
    .in_range(0.1, 0.5)
    .with_units(units.AMPERE),
)
def power_rail_test(test):
    test.measurements.rail_3v3 = 3.31
    test.measurements.current_draw = 0.24


def main():
    test = htf.Test(
        power_rail_test,
        procedure_id="FCT-001",
        part_number="PCBA-200",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial: "))


if __name__ == "__main__":
    main()
```

The measurement definition carries the name, limits, and units, so storage and pass/fail evaluation follow from the test code rather than being maintained separately. Yield, Cpk, control charts, and failure Pareto are computed rather than queried.

Options in this space include WATS, which is the long-standing incumbent, and TofuPilot. Both handle the entities above. They differ mainly in deployment model, licensing shape, and whether test code lives in your Git repository.

**Works until:** you have a requirement genuinely outside the product's model. At that point you are back to Approach 2, ideally with the vendor's export.

## Choosing

| Situation | Approach |
|---|---|
| One station, prototype volume | Flat files, but plan the exit |
| Multiple stations, engineer available to own it | SQL schema |
| Semiconductor test | STDF |
| Contract mandates a format | ATML or whatever the contract says |
| Board-level test, want analytics without building them | Test data platform |

## Mistakes That Cost the Most

**Not versioning the procedure.** When limits change, old results become uninterpretable unless you recorded which version produced them.

**Storing pass or fail without the value.** A boolean tells you a unit failed. The measured value tells you it was drifting for three weeks beforehand.

**Ignoring sub-assemblies.** Serial number traceability from final assembly down to component boards is retrofitted at great cost. Model it on day one even if you do not use it yet.

**Computing yield from any passing run.** First-pass yield is the first run per unit. Counting retests inflates it, sometimes dramatically.

**No timezone discipline.** Store UTC. Three sites in three timezones will otherwise disagree about which day a failure occurred.

### How to Detect Measurement Drift
URL: https://www.tofupilot.com/guides/how-to-detect-measurement-drift-with-tofupilot
Detect measurement drift before it becomes scrap: what drift looks like in test data, which signals to watch, and how to catch it early.

Drift is a measurement moving slowly in one direction while still passing. It is the most valuable signal in test data and the easiest to miss, because every unit passes right up until the day they do not.

## What Drift Looks Like

| Pattern | Typical cause |
|---------|--------------|
| Gradual shift in the mean | Stencil wear, instrument out of calibration, supplier change |
| Widening spread, stable mean | Fixture contact degrading, a second population appearing |
| Step change | Component lot change, firmware release, limit change |
| Sawtooth | Something reset periodically: recalibration, shift change |
| Drift on one station only | Fixture, cabling, or that station's instrument |
| Drift correlated with time of day | Ambient temperature, mains loading |

The first is the dangerous one. A rail drifting from 3.30 V toward a 3.40 V limit produces perfect yield for weeks and then a bad afternoon.

## Why Pass/Fail Cannot See It

If your records store only verdicts, drift is invisible by construction. Every run says PASS until the day the limit is crossed, and then yield collapses with no warning and no history to diagnose from.

Detecting drift requires the measured values.

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.dc_voltage()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def dc_voltage(self) -> float:
        return 3.31  # Replace with a real instrument read
```

Every run stores the number with its limits, so the distribution over time exists whether or not anyone is looking at it yet.

## Record the Context That Explains Drift

A drift you can see but not attribute is only half useful. The metadata that makes attribution possible costs almost nothing:

```python filename="phases/record_context.py"
def record_context(run, dmm):
    run.metadata["station"] = "FCT-03"
    run.metadata["fixture_id"] = "FIX-4417-02"
    run.metadata["dmm_id"] = dmm.identify()
    run.metadata["operator_shift"] = "B"
```

With those, "the mean shifted on 14 March" becomes "the mean shifted on FCT-03 only, after the fixture was swapped", which is an actionable statement.

## The Signals Worth Watching

**Mean shift against the limit.** The distance from the process mean to the nearest limit, in standard deviations, is exactly what [Cpk](/guides/what-is-cpk-process-capability-index) measures. A Cpk falling from 1.8 to 1.3 with no failures yet is the clearest early warning available.

**Western Electric rules on a control chart.** Seven consecutive points on one side of the centre line, or a run of consistently increasing values, both indicate drift long before anything exceeds a limit. See [how to use control charts](/guides/how-to-use-control-charts-for-tests-with-tofupilot).

**Spread changes.** A standard deviation that doubles while the mean holds usually means a fixture or contact problem, not a product change.

**Bimodality.** Two humps in a histogram mean two populations mixed. Split by station, fixture or lot; the drift is usually confined to one of them.

## Checking It Yourself

If you want a scheduled check rather than eyes on a chart:

```python filename="analysis/drift.py"
import statistics


def mean_shift_sigma(baseline: list[float], recent: list[float]) -> float:
    """How far the recent mean has moved, in baseline standard deviations."""
    sigma = statistics.stdev(baseline)
    if sigma == 0:
        raise ValueError("Zero variation in baseline; check resolution")
    return (statistics.mean(recent) - statistics.mean(baseline)) / sigma


def trend_run_length(values: list[float]) -> int:
    """Longest run of consecutive increases or decreases."""
    longest = current = 1
    direction = 0
    for a, b in zip(values, values[1:]):
        step = (b > a) - (b < a)
        if step != 0 and step == direction:
            current += 1
            longest = max(longest, current)
        else:
            direction = step
            current = 1
    return longest


baseline = [3.30, 3.31, 3.29, 3.31, 3.30, 3.32, 3.30]
recent = [3.35, 3.36, 3.35, 3.37, 3.36]

print(f"shift: {mean_shift_sigma(baseline, recent):.1f} sigma")
print(f"longest run: {trend_run_length(baseline + recent)}")
```

A shift beyond two sigma, or a run of seven or more, is worth investigating even with a perfect pass rate.

## Two Traps in the Analysis

**A limit change looks exactly like a process change.** Tighten a limit and yield drops with no drift at all. This is why `version` on the procedure matters: without it, weeks get spent looking for a process cause that does not exist.

**A renamed measurement looks like drift stopping.** Renaming splits the history in two, and the old series simply ends. Treat measurement names as permanent.

## Responding to Drift

| Observation | Likely cause | Action |
|-------------|--------------|--------|
| Slow mean shift, all stations | Supplier, process, or environmental | Check build records and component lots |
| Shift on one station | Fixture, cabling, instrument | Compare against another station; check calibration |
| Spread widening | Contact degradation | Measure contact resistance; service the fixture |
| Step change at a known date | Lot, firmware, or limit change | Check the procedure version first |
| Sawtooth | Something is being reset | Correlate with calibration or shift schedule |
| Bimodal | Two populations | Split by station, fixture, lot |

Checking the procedure version first is the cheapest step and rules out the most embarrassing cause.

## Instrument Drift vs Product Drift

A drifting measurement may be the instrument, not the product. The distinguishing test is a known-good reference unit: run it periodically and record its measurements like any other unit.

```yaml filename="procedure.yaml"
  - name: Golden Unit Check
    python: phases.golden_unit
    measurements:
      - name: Golden Rail 3V3
        unit: V
        description: Reference unit; drift here indicates the instrument
        validators:
          - operator: ">="
            expected_value: 3.28
          - operator: "<="
            expected_value: 3.34
```

If the golden unit drifts, the bench is the problem. If it holds steady while production drifts, the product or process is.

## If You Already Run OpenHTF

```python filename="fct.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
def phase_power(test):
    test.measurements.rail_3v3 = 3.31
    test.metadata["station"] = "FCT-03"
    test.metadata["fixture_id"] = "FIX-4417-02"


test = htf.Test(phase_power, part_number="PCBA-100")
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

Install with `pip install "tofupilot[openhtf]"`, or run it with `tofupilot run`.

## Common Mistakes

| Mistake | Consequence | Fix |
|---------|-------------|-----|
| Storing pass/fail only | Drift is invisible until yield collapses | Store the measured values |
| Watching yield instead of measurements | Yield moves last | Watch the distribution and Cpk |
| No station or fixture metadata | Drift cannot be attributed | Record context on every run |
| Ignoring a falling Cpk with no failures | The warning is wasted | Treat a Cpk trend as actionable |
| Assuming the product drifted | It may be the bench | Run a golden unit periodically |
| Forgetting a limit change | Weeks spent chasing a phantom | Check the procedure version first |

### What to Log in a Production Test
URL: https://www.tofupilot.com/guides/what-to-log-in-a-production-test
The fields worth recording in a production test so failures are still debuggable six months later, and the four that are painful to add afterwards.

Most production test logging is designed for the moment the test runs. An operator needs a green light or a red light, so the test prints one, and that is where the thinking stops.

The problem arrives months later. A customer returns a unit. A yield investigation needs a baseline. An auditor asks what a specific serial number measured. At that point the only thing anyone wants is the data nobody recorded.

This guide lists what to log, in rough priority order, and marks the fields that are painful to retrofit.

## The minimum that makes data useful

**The measured value, not just pass or fail.**

This is the single highest-value field and the one most often missed. A boolean tells you a unit failed. The value tells you it measured 3.19V when the limit was 3.20V, and that it had been trending down for three weeks.

Everything downstream depends on this. Trend charts, capability analysis, drift detection and limit tuning are all impossible from booleans. If you record one thing beyond the verdict, record the number.

**The limits that were applied.**

Store the limits alongside the value, not only in the test code. Limits change, and when they do, a result recorded under the old limits becomes uninterpretable unless the limits travelled with it.

```json filename="measurement.json"
{
  "name": "rail_3v3",
  "value": 3.31,
  "unit": "V",
  "lower_limit": 3.2,
  "upper_limit": 3.4,
  "passed": true
}
```

**The units.**

Cheap to record, and it prevents the class of error where someone later assumes millivolts.

**A timestamp, in UTC.**

Store UTC and convert for display. Two sites in two timezones will otherwise disagree about which day a failure occurred, and shift-boundary analysis becomes unreliable.

## Identity: what was tested

**Serial number.** Without it you have statistics but no traceability, and you cannot answer any question about a specific returned unit.

**Part number and revision.** A measurement means something different on rev C than on rev A. Without the revision, mixed-revision data silently blends into one distribution.

**Sub-assembly links.** Which PCBA went into which enclosure, which module into which final assembly. This is the field most often skipped and most painful to add later, because retrofitting it means reconstructing relationships that were never captured.

Model it from day one even if you do not use it yet. When a component lot turns out to be bad, this is what tells you which shipped units contain it.

## Context: where and how it was tested

These cost nothing at test time and are the dimensions you will want to split by during an investigation.

| Field | What it lets you answer |
|---|---|
| **Station identifier** | Is one station failing more than the others? |
| **Operator** | Is a fixture seating technique causing failures? |
| **Test procedure version** | Which version of the limits produced this result? |
| **Firmware version on the DUT** | Did a calibration constant change in software? |
| **Fixture identifier** | Is a specific fixture wearing out? |
| **Instrument used** | If a DMM was out of calibration, which units did it touch? |

The station identifier earns its place fastest. A single query grouping failures by station resolves a surprising share of yield investigations, and it is impossible without this field.

**Test procedure version** is the other one worth insisting on. Limits change over a product's life. If you cannot tell which version produced a result, your historical data stops being comparable at every limit change.

## Execution detail

**Per-step outcomes and durations.** A run that failed tells you less than a run that failed at step 7 after passing 1 through 6. Step-level data is what makes failure Pareto analysis possible, and durations reveal a station slowing down before anyone notices.

**Attachments where they matter.** A waveform capture, a photograph of a failed unit, an instrument screenshot. Do not attach these on every run at volume, but attach them on failures, where they save a rework cycle.

**Logs from the test script.** Console output, instrument responses, exception traces. Cheap to store on failures, and the difference between diagnosing something and guessing.

## What not to log

Restraint matters, because storage that grows without limit gets deleted wholesale, usually right before you need it.

**Raw waveforms on every unit.** Store the extracted features, rise time, overshoot, ripple, and keep the raw capture only for failures or a sample.

**High-rate streams.** If you are acquiring at hundreds of samples per second, aggregate before storing. A test record platform is for results, not for time-series acquisition. Record the mean, the extremes and the pass criterion, keep the full trace locally when you need it.

**Anything you can derive.** Do not store yield as a field. Compute it from runs. A stored aggregate is wrong the moment a retest arrives.

## The four that hurt to retrofit

Everything above can be added later with some effort, except these:

1. **The measured value.** Historical booleans cannot be turned back into numbers. Every day without this is a day of data you will never recover.
2. **Sub-assembly relationships.** Reconstructing which board went into which assembly after the fact usually means reading build records by hand, if they exist.
3. **Procedure versioning.** Once limits have changed without a version marker, you cannot reliably tell which results are comparable.
4. **Serial numbers on early units.** Prototype and pilot units often skip serialisation, and those are exactly the units that come back with the most interesting failures.

## A minimal schema

If you are storing to your own database, this covers the fields above:

```sql filename="schema.sql"
CREATE TABLE unit (
    id            BIGSERIAL PRIMARY KEY,
    serial_number TEXT NOT NULL UNIQUE,
    part_number   TEXT NOT NULL,
    revision      TEXT,
    parent_id     BIGINT REFERENCES unit(id),
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE run (
    id                BIGSERIAL PRIMARY KEY,
    unit_id           BIGINT NOT NULL REFERENCES unit(id),
    procedure_id      TEXT NOT NULL,
    procedure_version TEXT NOT NULL,
    station_id        TEXT NOT NULL,
    operator          TEXT,
    dut_firmware      TEXT,
    started_at        TIMESTAMPTZ NOT NULL,
    duration_s        NUMERIC NOT NULL,
    passed            BOOLEAN NOT NULL
);

CREATE TABLE step (
    id         BIGSERIAL PRIMARY KEY,
    run_id     BIGINT NOT NULL REFERENCES run(id) ON DELETE CASCADE,
    name       TEXT NOT NULL,
    duration_s NUMERIC,
    passed     BOOLEAN NOT NULL
);

CREATE TABLE measurement (
    id          BIGSERIAL PRIMARY KEY,
    step_id     BIGINT NOT NULL REFERENCES step(id) ON DELETE CASCADE,
    name        TEXT NOT NULL,
    value       DOUBLE PRECISION,
    unit        TEXT,
    lower_limit DOUBLE PRECISION,
    upper_limit DOUBLE PRECISION,
    passed      BOOLEAN NOT NULL
);

CREATE INDEX ON run (procedure_id, started_at DESC);
CREATE INDEX ON run (station_id, started_at DESC);
CREATE INDEX ON measurement (name, step_id);
```

The `parent_id` on `unit` is the sub-assembly link. It is one column, and adding it now is far cheaper than reconstructing the relationships later.

## Doing it without writing the schema

If your test scripts are Python, this shape comes for free. The measurement definition in your test code already carries the name, value, units and limits, so an output callback records every field above without a schema of your own.

With OpenHTF, that is one import and one line:

```python filename="main.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
def power_rail_test(test):
    test.measurements.rail_3v3 = 3.31


def main():
    test = htf.Test(
        power_rail_test,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="PCBA-200",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-0001")


if __name__ == "__main__":
    main()
```

Install it as `pip install "tofupilot[openhtf]"`. The `upload()` callback reads `TOFUPILOT_API_KEY` from the environment, and `add_output_callbacks` is OpenHTF's own extension point, so it composes with any callbacks you already use.

The point is not which tool you use. It is that the decision about what to record is made once, early, and is expensive to revisit. Record the value, version the procedure, model sub-assemblies, and keep UTC timestamps. The rest can wait.

### Where to Store Production Test Results
URL: https://www.tofupilot.com/guides/how-to-replace-excel-test-tracking-with-tofupilot
Test results scattered across spreadsheets and shared folders stop working at a predictable point. Here is when to move, and what to move to.

Most hardware teams log production test results in a spreadsheet. It works, until one day it doesn't, and the failure is usually described the same way: results are all over the place, in random folders on people's laptops, the shared drive and the cloud, and the sheet has become too cumbersome to maintain.

This guide covers when a spreadsheet stops being the right answer, what to move to, and how to migrate without losing history.

## When a spreadsheet is genuinely fine

Worth saying first, because the answer is not always "buy something".

A spreadsheet works when one person logs results from one station, volumes are low, and nobody needs to answer questions across months of data. Prototype builds and pilot runs fit this comfortably. If that is you, the effort of moving is not yet repaid.

The signals that it has stopped working are specific:

- **Two people need to log at once.** Spreadsheets are single-writer in practice. Merge conflicts and overwrites start immediately.
- **Someone asks for yield over a period.** Answering means a pivot table that breaks the next time a column moves.
- **A returned unit needs its history.** Finding one serial number across a year of files is a manual search.
- **The person who built the sheet is the only one who can maintain it.** This is the bus-factor problem, and it is usually what actually forces the change.

## The spreadsheet pattern

A typical production test log looks like this:

| Serial Number | Date | Operator | Result | Voltage (V) | Current (A) | Firmware | Notes |
|---|---|---|---|---|---|---|---|
| SN-001 | 2025-01-15 | Alice | PASS | 3.31 | 0.52 | v2.1 | |
| SN-002 | 2025-01-15 | Bob | FAIL | 3.58 | 0.89 | v2.1 | Over current limit |
| SN-003 | 2025-01-16 | Alice | PASS | 3.29 | 0.48 | v2.1 | |

The structural problems:

- **No concurrent access.** Two operators cannot log at the same time without risking overwrites.
- **No validation.** Nothing stops "PSAS" instead of "PASS", or voltage entered in the current column.
- **No analytics.** Yield, capability and failure trends mean fragile formulas that break when the sheet structure changes.
- **No history.** Edit a cell and the original value is gone. There is no audit trail.
- **Limits live in your head.** The sheet records 3.31 but not that the limit was 3.1 to 3.5. When limits change, old rows become uninterpretable.

That last one is the most damaging and the least noticed.

## Option 1: your own database

A Postgres schema with unit, run, step and measurement tables, plus Metabase or Grafana on top, is a perfectly respectable answer. It handles concurrency, gives you SQL for any question, and costs nothing in licensing.

The catch is not the build, it is the maintenance. Yield is easy. What takes the time is capability analysis done correctly per measurement, control limits that survive a spec change, first pass yield that counts the first run per unit rather than any passing run, and traceability through sub-assemblies. Each is a small project. Together they are the thing nobody has time to keep up.

Choose this when you have someone to own it and requirements no product matches.

## Option 2: a test data platform

The alternative is a system built for this, where the schema and the analytics come with it.

Here is the same test as the spreadsheet above, written as an OpenHTF test that logs automatically:

```python filename="power_board_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("voltage").with_units(units.VOLT).in_range(3.1, 3.5),
    htf.Measurement("current").with_units(units.AMPERE).in_range(maximum=0.7),
)
def power_supply_check(test):
    voltage = 3.31  # Read from your instrument
    current = 0.52
    test.measurements.voltage = voltage
    test.measurements.current = current


def main():
    test = htf.Test(
        power_supply_check,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard
        part_number="PCB-V1",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-001")


if __name__ == "__main__":
    main()
```

Install it with `pip install "tofupilot[openhtf]"`. The `upload()` callback reads `TOFUPILOT_API_KEY` from the environment.

Each run captures the serial number, the outcome, every measurement with its limits and units, timestamps and the station identity. The measurement definition in the test code is the schema, so there is nothing separate to design or migrate.

## What changes

| Capability | Spreadsheet | Own database | Test data platform |
|---|---|---|---|
| Data entry | Manual | From test code | From test code |
| Concurrent access | Single writer | Yes | Yes |
| Limits stored with results | No | If you model it | Yes |
| Yield trends | Manual formulas | SQL you write | Built in |
| Capability and control charts | Custom macros | You build them | Built in |
| Failure Pareto | Manual filtering | SQL you write | Built in |
| Sub-assembly traceability | No | If you model it | Built in |
| Audit trail | None | If you model it | Full history per run |
| Ongoing cost | Your time | Engineering time | Licence |

## Keeping your existing data

Historical spreadsheet data imports through the API. Structure each row as a run with a phase and its measurements:

```python filename="import_from_csv.py"
import csv
import os
from datetime import datetime, timezone

from tofupilot.v2 import TofuPilot

PROCEDURE_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

with TofuPilot(api_key=os.getenv("TOFUPILOT_API_KEY")) as client:
    with open("test_results.csv") as f:
        for row in csv.DictReader(f):
            outcome = "PASS" if row["Result"] == "PASS" else "FAIL"
            tested_at = datetime.strptime(row["Date"], "%Y-%m-%d").replace(
                tzinfo=timezone.utc
            )

            client.runs.create(
                procedure_id=PROCEDURE_ID,
                serial_number=row["Serial Number"],
                part_number="PCB-V1",
                outcome=outcome,
                started_at=tested_at,
                ended_at=tested_at,
                phases=[{
                    "name": "Power Supply Check",
                    "outcome": outcome,
                    "started_at": tested_at,
                    "ended_at": tested_at,
                    "measurements": [
                        {
                            "name": "Output Voltage",
                            "measured_value": float(row["Voltage (V)"]),
                            "units": "V",
                            "validators": [
                                {"operator": ">=", "expected_value": 3.1},
                                {"operator": "<=", "expected_value": 3.5},
                            ],
                        },
                        {
                            "name": "Supply Current",
                            "measured_value": float(row["Current (A)"]),
                            "units": "A",
                            "validators": [
                                {"operator": "<=", "expected_value": 0.7},
                            ],
                        },
                    ],
                }],
            )
```

Note that the limits become validators on the measurement, which is what makes the historical rows interpretable later. Run the script once to backfill, then point new tests at the platform. Importers also exist for TestStand, STDF and other formats if your history is not in CSV.

Test the script against a handful of rows before running the whole file, and keep the original CSV until you have confirmed the imported yield matches.

## Migrating without risk

You do not have to switch everything at once:

1. **Pick one test station** and add the integration to its tests.
2. **Run both systems** for a week. Keep the spreadsheet going while the platform collects the same data automatically.
3. **Compare the yield numbers** on the same units. When they disagree, the reason usually tells you something about your own data model, most often that the spreadsheet was counting any passing run rather than the first run per unit.
4. **Roll out** to the remaining stations once the numbers agree.

The spreadsheet stays as a backup until you are ready to retire it.

## What to insist on, whichever you choose

**Store the measured value, not just pass or fail.** A boolean tells you a unit failed. The value tells you it was drifting for three weeks.

**Store the limits alongside the value.** Otherwise old results become uninterpretable the first time limits change.

**Version the test procedure.** Same reason, and it is nearly impossible to retrofit.

**Model sub-assemblies from day one.** Reconstructing which board went into which assembly after the fact means reading build records by hand.

Those four decisions are far cheaper to make at the start than to add later, regardless of where the data ends up living.

### Python Test Framework for Hardware
URL: https://www.tofupilot.com/guides/python-test-framework-for-hardware
Most Python test frameworks are built for unit-testing code, not system-testing devices. Here is what actually works for production test, and when.

If you have searched for a Python test framework for hardware, you have probably noticed the problem: almost everything you find is built for unit-testing Python code, not for system-testing a physical device on a bench or a production line.

The distinction matters. A unit test asserts that a function returns the right value. A production test measures a voltage on a real board, compares it to a limit, records the number, and tells an operator whether to put the unit in the pass bin or the fail bin. Those are different jobs, and most frameworks only do the first one.

Here are the options that actually exist, and where each one stops working.

## What hardware test needs that unit testing does not

Before comparing, it helps to name the requirements. A framework for production test needs:

| Requirement | Why unit-test frameworks lack it |
|---|---|
| **Measurements as data** | An assert throws away the value. You need the number, its units and its limits, stored |
| **Serial number identity** | Unit tests have no concept of which physical object is under test |
| **Operator interaction** | Someone has to scan a barcode, press a button, confirm a fixture is closed |
| **Instrument lifecycle** | A DMM must be opened once, shared across steps, and closed even when a test fails |
| **Ordered phases with continue-on-fail** | A failing unit test aborts. A failing production test often continues to collect more diagnostic data |
| **Result destination** | Unit tests print to a terminal. Production results need to outlive the shift |

The last one is the one people underestimate. A test that prints "PASS" to a console has told you nothing in six months when a customer returns a unit and you want to know what its rail voltage measured on the day it shipped.

## Option 1: pytest

pytest is the default Python testing framework and the first thing most engineers reach for. It works for hardware, with caveats.

```python filename="test_power_rail.py"
import pytest


@pytest.fixture(scope="session")
def dmm():
    import pyvisa
    rm = pyvisa.ResourceManager()
    inst = rm.open_resource("USB0::0x2A8D::0x1601::MY60012345::INSTR")
    yield inst
    inst.close()


def test_rail_3v3(dmm):
    voltage = float(dmm.query("MEAS:VOLT:DC?"))
    assert 3.2 <= voltage <= 3.4, f"3.3V rail out of range: {voltage}V"
```

**What works.** Fixtures handle instrument lifecycle well, which is genuinely the right abstraction for a DMM or a power supply. The plugin ecosystem is enormous. Every Python developer already knows it, so there is no training cost. It runs in CI without modification.

**Where it stops.** The measured value lives inside an assert and then disappears. You know the test failed; you do not have a record that the rail measured 3.19V and had been drifting for three weeks. There is no serial number concept, no operator prompt, and no interface for a technician. You end up writing all of that yourself, which is exactly the custom code you were trying to avoid.

**Use it when** you are doing R&D validation or firmware CI, and the results feed a developer rather than a production line.

## Option 2: OpenHTF

OpenHTF is a Python framework Google built specifically for hardware test. It is the closest thing to a purpose-built answer, and it is free.

```python filename="functional_test.py"
import openhtf as htf
from openhtf.util import units
from openhtf.plugs import user_input


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
def power_rail_test(test):
    test.measurements.rail_3v3 = 3.31


def main():
    test = htf.Test(power_rail_test)
    test.execute(test_start=user_input.prompt_for_test_start())


if __name__ == "__main__":
    main()
```

**What works.** Measurements are first-class objects with names, values, units and limits. Pass or fail is evaluated from the limits automatically, so you are not writing comparison logic. Serial number prompting is built in. Plugs manage instrument setup and teardown. There is a basic web interface for operators.

**Where it stops.** Parallel testing of multiple units is limited. The project is not actively developed by Google, and the community is small. Building a real operator interface beyond the basic one means frontend work.

**Use it when** you are doing production functional test in Python and want structured measurements without designing a data model.

## Option 3: The TofuPilot Framework

We built this because OpenHTF got the data model right and stopped short on everything around it. It is open source under MIT.

A procedure is a `.yaml` file that declares the sequence, the measurements and their limits. Phases are plain Python functions with no decorator, and plugs connect instruments as Python classes.

```yaml filename="procedure.yaml"
name: PCBA Functional Test

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-200"

plugs:
  - name: Multimeter
    python: plugs.dmm:Multimeter

main:
  - name: Power Rail Test
    python: phases.power_rail
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

```python filename="phases/power_rail.py"
def power_rail(measurements, multimeter):
    measurements.rail_3v3 = multimeter.read_voltage()
```

```python filename="plugs/dmm.py"
class Multimeter:
    def read_voltage(self) -> float:
        return 3.31
```

The limits live in the YAML, so the phase stays a plain function that assigns a value. TofuPilot generates the measurement key from the name in snake_case (`Rail 3V3` becomes `rail_3v3`), and injects plugs into the phase by the same rule.

**What it adds over OpenHTF.** Operator interfaces are built from the procedure definition, with text inputs, checklists, images, sliders and progress bars, so there is no frontend work. Phases and multiple fixture slots run in parallel on a Rust engine while your code stays plain Python. Procedures deploy to stations from a Git push, with immutable build artifacts and instant rollback. Runs queue offline when the network drops and sync when it returns.

**Where it stops.** It is newer than OpenHTF, so the community is smaller. If your team does not write Python, none of the Python options are the right answer.

**Use it when** you are running production test on real stations and the operator interface and deployment story matter as much as the test logic.

## Option 4: Keep your own framework

Plenty of teams have a home-grown Python framework that works. If yours does, the honest advice is to be careful about replacing it.

The reason is that your framework was shaped around your products, your serial number scheme and your sub-assembly structure. Commercial and open-source alternatives ask you to bend those to their model, and the mismatch is where migrations go wrong.

**Keep it when** it fits, and the only complaint is maintenance burden rather than missing capability. Reducing the maintenance is often cheaper than migrating.

**Replace it when** the maintenance is really the analytics layer rather than the test execution. Yield, capability analysis, control charts and traceability are each a small project, and together they are the thing nobody has time to keep up.

## Comparison

| | pytest | OpenHTF | TofuPilot Framework |
|---|---|---|---|
| **Built for** | Software testing | Hardware test | Hardware test |
| **Measurements** | Implicit in asserts | Structured with limits | Structured with limits |
| **Limits declared in** | Assert expressions | Python decorators | Procedure YAML |
| **Serial number prompt** | Manual | Built in | Built in |
| **Operator interface** | None | Basic web UI | Built from the procedure |
| **Parallel phases** | Via xdist | Limited | Native |
| **Multi-slot fixtures** | Manual | Limited | Native |
| **Deployment to stations** | Your problem | Your problem | Git push |
| **Offline handling** | Your problem | Your problem | Queues and syncs |
| **License** | MIT | Apache 2.0 | MIT |

## Choosing

**Doing R&D validation or firmware CI?** Use pytest. The ecosystem and familiarity outweigh the missing hardware features, and your results feed a developer rather than a line.

**Running production test and want structured data without building a schema?** Use OpenHTF or the TofuPilot Framework. Both give you measurements with limits and units as first-class objects.

**Running production test across multiple stations where operators need an interface and scripts need to reach the stations reliably?** That is what the TofuPilot Framework was built for, and it is the gap that made us build it rather than extend OpenHTF.

**Already have something that works?** Fix the maintenance problem before you replace the framework. The framework is rarely the expensive part.

## Where the results go

Whichever framework you choose, decide early where results are stored. A framework executes tests; it does not answer "what was our yield last week" or "which test step is costing us the most units".

Store the measured value, not just pass or fail. Version the procedure so results recorded before a limit change stay interpretable. Model sub-assemblies from day one even if you do not use them yet. Those three decisions are much harder to retrofit than to make at the start.

### Why Is My Yield Dropping?
URL: https://www.tofupilot.com/guides/why-is-my-yield-dropping
Nothing changed in the design but boards started failing. A practical method for finding the cause using test data you already have, or should be recording.

The situation is familiar. Pre-production runs were fine. Same parts, same gerbers, same test. Then production starts and a chunk of boards fail, often at one specific step, and nothing in the design changed.

This guide is a method for finding the cause. It assumes you have test results recorded. If you only have pass and fail counts, skip to the last section, because that is the actual problem and it is worth fixing before the next time.

The queries below assume measurements are recorded per step, with each step belonging to a run: `run` → `step` → `measurement`. Adjust the joins if your schema attaches measurements directly to runs.

## Start with the measured values, not the pass rate

The pass rate tells you units failed. It does not tell you why, and it hides the most useful signal.

A measurement that is drifting toward its limit will pass for weeks before it starts failing. By the time the yield number moves, the cause has been present for a long time. If you plot the measured values rather than the outcomes, the drift is visible before it costs you anything.

```sql filename="measurement_trend.sql"
SELECT
    date_trunc('day', r.started_at) AS day,
    avg(m.value)                    AS mean,
    stddev(m.value)                 AS sigma,
    min(m.value)                    AS min_value,
    max(m.value)                    AS max_value,
    count(*)                        AS n
FROM measurement m
JOIN step s ON s.id = m.step_id
JOIN run  r ON r.id = s.run_id
WHERE m.name = 'rail_3v3'
  AND r.procedure_id = 'FCT-001'
GROUP BY day
ORDER BY day;
```

Look at the mean and the spread separately. They fail differently:

- **Mean moves, spread stays tight.** Something shifted. A new component lot, a reflow profile change, a recalibrated instrument, a different power supply on the bench.
- **Mean stays, spread widens.** Something became inconsistent. A worn fixture, an intermittent contact, temperature variation across a shift.
- **Both move.** Usually two causes, or one cause with a knock-on effect. Split the data before going further.

## Split the data before forming a theory

Before you theorise, cut the failures by every dimension you record. The point is to find a dimension where the failure rate is not uniform.

| Split by | What a difference means |
|---|---|
| **Test station** | Fixture wear, a miscalibrated instrument, a bad cable on one station |
| **Time of day or shift** | Temperature, mains variation, a specific operator, warm-up effects |
| **Component lot or date code** | Incoming part variation, the most common cause of a sudden step change |
| **PCB panel or fab lot** | Board-level process issue |
| **Operator** | Fixture seating technique, a step being skipped |
| **Test step** | Narrows which physical subsystem is involved |
| **Firmware version** | A calibration constant or threshold changed in software |

```sql filename="failures_by_station.sql"
SELECT
    r.station_id,
    count(*)                                    AS runs,
    count(*) FILTER (WHERE NOT r.passed)        AS failures,
    round(100.0 * count(*) FILTER (WHERE NOT r.passed) / count(*), 2) AS fail_pct
FROM run r
WHERE r.procedure_id = 'FCT-001'
  AND r.started_at > now() - interval '14 days'
GROUP BY r.station_id
ORDER BY fail_pct DESC;
```

If one station is at 12% and the others are at 2%, you have your answer and it is not a design problem. This single query resolves a surprising share of yield investigations.

## Confirm the failure is real before chasing it

Before investigating a process, rule out the test itself. A test that fails good boards is more common than people expect, and it wastes weeks.

Check in this order:

1. **Retest a failing unit without touching it.** If it passes, you have an intermittent contact or a marginal limit, not a product defect.
2. **Retest on a different station.** If it passes, the first station is the problem.
3. **Swap the suspect part between a failing board and a passing board.** If the failure follows the part, it is a component issue. If it stays with the board, it is the board or the process.
4. **Measure by hand with a separate instrument.** If the manual measurement disagrees with the fixture, the fixture or its calibration is suspect.

Step 3 is the strongest diagnostic available and it costs one rework cycle. It cleanly separates component problems from assembly problems.

## Check whether the limits are the problem

If the measured values have not moved but the failure rate has, look at the limits rather than the process.

Limits often get tightened after a field failure, or copied from a datasheet without accounting for measurement uncertainty. A limit that sits close to the natural spread of your process will reject good units at a steady rate.

The rule of thumb is that your process spread should fit comfortably inside the limits. If the distance between your limit and your mean is less than about four standard deviations, expect rejects from normal variation alone. That ratio is what process capability, Cp and Cpk, measures.

```sql filename="capability_check.sql"
SELECT
    m.name,
    avg(m.value)                                        AS mean,
    stddev(m.value)                                     AS sigma,
    min(m.lower_limit)                                  AS lsl,
    min(m.upper_limit)                                  AS usl,
    least(
        (min(m.upper_limit) - avg(m.value)) / (3 * stddev(m.value)),
        (avg(m.value) - min(m.lower_limit)) / (3 * stddev(m.value))
    )                                                   AS cpk
FROM measurement m
JOIN step s ON s.id = m.step_id
JOIN run  r ON r.id = s.run_id
WHERE r.procedure_id = 'FCT-001'
  AND r.started_at > now() - interval '30 days'
GROUP BY m.name
HAVING stddev(m.value) > 0
ORDER BY cpk ASC;
```

Sort ascending and the measurements most likely to cause rejects appear first. Anything below 1.33 is worth attention; below 1.0 you are rejecting units through normal variation.

Two caveats before acting on the number. Cpk assumes the measurement is roughly normally distributed and the process is stable over the window you queried, so a value computed across a drift is meaningless. And a one-sided measurement, where only an upper or lower limit exists, makes one half of that `least()` null. Handle that case separately rather than reading the result as a low score.

## Which test step is actually costing you

When several steps fail, fix the one that costs the most units rather than the one that is most interesting.

```sql filename="failure_pareto.sql"
SELECT
    s.name                AS step,
    count(*)              AS failures,
    round(100.0 * count(*) / sum(count(*)) OVER (), 1) AS pct_of_failures
FROM step s
JOIN run r ON r.id = s.run_id
WHERE r.procedure_id = 'FCT-001'
  AND NOT s.passed
  AND r.started_at > now() - interval '30 days'
GROUP BY s.name
ORDER BY failures DESC;
```

Failure distributions are usually heavily skewed. Two or three steps account for most of the loss, and the rest is noise.

## Watch out for retests hiding the problem

If your yield number counts any passing run rather than the first run per unit, retests will mask a worsening process. A unit that fails twice and passes on the third attempt looks identical to a unit that passed immediately.

First pass yield means the first run per unit:

```sql filename="first_pass_yield.sql"
SELECT
    date_trunc('day', first_run.started_at) AS day,
    count(*) FILTER (WHERE first_run.passed) * 100.0 / count(*) AS fpy_percent,
    count(*)                                AS units
FROM (
    SELECT DISTINCT ON (unit_id)
        unit_id, started_at, passed
    FROM run
    WHERE procedure_id = 'FCT-001'
    ORDER BY unit_id, started_at ASC
) AS first_run
GROUP BY day
ORDER BY day DESC;
```

The `DISTINCT ON` with an ascending sort is what makes this the first run rather than any run. Getting this wrong is the most common reporting bug in home-built systems, and it always makes the number look better than reality.

A widening gap between first pass yield and final yield is itself a signal: the process is degrading and retests are absorbing it.

## When you do not have the data

If the investigation above is impossible because you only recorded pass and fail, that is the finding. The fix is not complicated but it has to happen before the next incident:

**Store the measured value, not just the outcome.** A boolean tells you a unit failed. The value tells you it was drifting for three weeks beforehand. This single change makes most of the queries above possible.

**Record which station, which operator, which firmware version.** These are the dimensions you will want to split by, and they cost nothing to capture at test time.

**Version the test procedure.** When limits change, you need to know which version produced a given result, otherwise old data becomes uninterpretable.

**Keep serial numbers and sub-assembly links.** When a failure correlates with a component lot, you need to trace which units contain which parts.

**Set an alert on the trend, not the threshold.** By the time yield crosses a threshold the cause is weeks old. Alerting on drift catches it while units are still passing.

TofuPilot records all of this by default from a Python test script, and computes yield, capability, control charts and failure Pareto without the queries above. But the queries are here because the method matters more than the tool, and they work against any schema that stores the measured value.

### WATS Alternatives for Test Data Management
URL: https://www.tofupilot.com/guides/wats-alternatives-for-test-data-management
Compare alternatives to WATS for manufacturing test data management, with a feature matrix, cost breakdown, migration paths, and when to stay on WATS.

WATS is the incumbent in manufacturing test data management. It has been around since 2003, it is well established in electronics manufacturing, and for many teams it works. But it is a closed system built around the WSJF and WSXF file formats, it is priced per client and per server, and its analytics assume you send data its way and read it in its own UI.

If you are evaluating alternatives, the honest answer is that there are four real options: build on Postgres yourself, use a semiconductor yield system, use a manufacturing execution system, or use a modern test data platform. This guide compares each against WATS, including the cases where staying on WATS is the correct decision.

## Why Teams Look for Alternatives

These are the reasons that come up most often in evaluations:

| Reason | Detail |
|---|---|
| **Per-client licensing** | Cost scales with test stations, not with value received |
| **Proprietary formats** | WSJF and WSXF are WATS-specific; converters are one-way in practice |
| **Windows-centric deployment** | The server and client tooling assume a Windows estate |
| **Report-first, API-second** | Getting data out programmatically is harder than getting it in |
| **Slow iteration on test code** | Test sequences live in a separate toolchain from your firmware repo |
| **On-premise maintenance** | SQL Server administration, upgrades, and backups are your problem |

None of these are defects. They are consequences of a design that predates cloud-native infrastructure and Python-based test frameworks. If your factory floor is Windows and your test engineers do not write code, they may not bother you at all.

## The Alternatives

### Build on Postgres yourself

The most common alternative, and the one most teams try first. A schema with units, runs, steps, and measurements, plus Grafana or Metabase on top.

**Strengths:** Total control. No licensing. Your data model matches your products exactly.

**Limitations:** You are now maintaining a data platform. Yield, Cpk, and control-chart logic have to be written and validated. Serial number traceability across sub-assemblies is harder than it looks. Someone owns backups, migrations, and the on-call pager.

**Best for:** Teams with a data engineer to spare and unusual requirements that no product matches.

### Semiconductor yield systems

yieldHUB, yieldWerx and similar tools are built for wafer-level and device-level yield analysis, and they are genuinely strong at it.

**Strengths:** Deep wafer map, bin, and parametric analysis. STDF-native. Mature statistical tooling.

**Limitations:** The data model assumes wafers, dies, and bins. Board-level functional test with serialized sub-assemblies is not the target use case. Pricing is enterprise-scale.

**Best for:** Semiconductor fabs and OSATs, not board-level electronics manufacturing.

### Manufacturing execution systems

Aegis FactoryLogix, Critical Manufacturing, and similar MES platforms include test data capture as one module among many.

**Strengths:** Test data sits alongside routing, work instructions, materials, and labor. One system for the whole plant.

**Limitations:** Test analytics are rarely the strongest module. Implementation is a project measured in quarters, not weeks. Cost reflects the scope of a full MES.

**Best for:** Plants that need an MES anyway and can treat test data as one deliverable of that programme.

### Modern test data platforms

TofuPilot sits here, as do a small number of newer entrants. The design assumption is that test code is Python, lives in Git, and runs in CI, and that test data should be queryable through an API as easily as it is viewable in a UI.

```python filename="functional_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
def power_rail_test(test):
    test.measurements.rail_3v3 = 3.31


def main():
    test = htf.Test(
        power_rail_test,
        procedure_id="FCT-001",
        part_number="PCBA-200",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial: "))


if __name__ == "__main__":
    main()
```

**Strengths:** One line to log a run. Yield, Cpk, control charts, and Pareto analysis are computed automatically. Test code is diffable and reviewable. Self-hosting is available if data cannot leave your network.

**Limitations:** Younger than WATS, with a smaller installed base in traditional EMS environments. If your test engineers do not write Python, the fit is worse.

**Best for:** Hardware teams that already write Python, run CI, and want test data reachable from a script.

## Feature Comparison

| Feature | WATS | Postgres DIY | Yield systems | MES | TofuPilot |
|---|---|---|---|---|---|
| **Deployment** | On-prem or cloud | Yours | On-prem or cloud | On-prem | Cloud or self-hosted |
| **Licensing** | Per client and server | Free (plus your time) | Enterprise | Enterprise | Per station |
| **Data format** | WSJF, WSXF | Yours | STDF | Vendor-specific | JSON, REST API |
| **Test framework** | WATS client | Any | Vendor tooling | Vendor tooling | OpenHTF, pytest, Robot |
| **Yield and Cpk** | Built-in | You build it | Built-in (wafer) | Module | Built-in |
| **Control charts** | Built-in | You build it | Built-in | Varies | Built-in |
| **Sub-assembly traceability** | Built-in | You build it | Limited | Built-in | Built-in |
| **API access** | SOAP and REST | Direct SQL | Vendor API | Vendor API | REST, Python, C#, Rust |
| **Version control of tests** | Separate toolchain | Yours | Vendor tooling | Vendor tooling | Git-native Python |
| **Import from WATS** | n/a | Manual | No | Varies | WSJF and WSXF supported |

## Cost Shape

Direct price comparison is not possible, because WATS does not publish list pricing and quotes vary by client count, server topology, and region. What can be compared is how cost scales:

| Model | Scales with |
|---|---|
| WATS | Number of clients and servers |
| Postgres DIY | Engineering time, indefinitely |
| Yield systems | Enterprise contract, typically site-based |
| MES | Full plant scope |
| TofuPilot | Number of test stations |

The question worth asking any vendor, WATS included, is what happens to the invoice when you double your station count, and whether analytics are included or a separate line.

## Migration Paths

### WATS to TofuPilot

WSJF and WSXF import is supported, so historical data comes across rather than being abandoned. The usual approach is to run both systems in parallel on one procedure, confirm that yield numbers match on the same units, then move procedure by procedure. Keep WATS reporting live until the numbers agree.

### WATS to Postgres

Export via the WATS API, define your schema, and rebuild the analytics you relied on. Budget for the analytics, not the export. Yield and Cpk are the easy part; sub-assembly traceability and operator-facing reporting are where the time goes.

### Staying on WATS

Stay if your test stations are Windows-based and your engineers do not write Python, if your MES or ERP integration with WATS is already built and working, if you are in a regulated environment where the validation cost of switching exceeds the benefit, or if your WATS deployment is stable and nobody is asking for anything it does not do. "It works and nobody is complaining" is a legitimate reason not to migrate.

## Decision Framework

**Choose a modern test data platform if** your test code is Python, you want test data reachable from a script, and you would rather not administer a database.

**Build on Postgres if** you have unusual data requirements, a data engineer to own it, and a genuine reason no product fits.

**Choose a yield system if** you are testing wafers and dies rather than boards and assemblies.

**Choose an MES if** you need routing, materials, and labor tracking, and test data is one requirement among many.

**Stay on WATS if** it works, your estate is Windows, and the switching cost buys you nothing you actually need.

### What Is a Histogram in Manufacturing Test
URL: https://www.tofupilot.com/guides/what-is-a-histogram-in-manufacturing-test
A histogram shows the distribution shape of your measurement data. Learn how to read histogram patterns, spot problems, and use them for SPC in TofuPilot.

A histogram groups your measurement values into bins and shows how many fall in each. It's the simplest way to see whether your process is centered, how much variation it has, and whether the distribution shape is normal or hiding a problem. In SPC, histograms complement control charts: the chart shows trends over time, the histogram shows the overall distribution shape.

## How to Read a Histogram

The x-axis shows measurement values. The y-axis shows how many data points fall in each bin. A well-behaved process produces a bell curve (normal distribution) centered between the spec limits.

Three things to look at:

**Center.** Is the peak of the distribution aligned with the target value, or shifted toward one spec limit? A shifted histogram means the process mean has drifted.

**Spread.** How wide is the distribution relative to the spec limits? A narrow distribution within wide spec limits means high capability (high Cp). A distribution that fills or exceeds the spec window means low capability.

**Shape.** A symmetric bell curve is normal. Anything else is a signal worth investigating.

## Distribution Shapes and What They Mean

| Shape | What It Looks Like | Likely Cause | Action |
|-------|-------------------|-------------|--------|
| Normal (bell curve) | Symmetric, single peak centered | Process is stable and predictable | Monitor. This is the target state. |
| Shifted normal | Bell curve off-center toward one limit | Process mean has drifted | Re-center: adjust calibration, tool offset, or recipe |
| Skewed right | Tail extends toward higher values | Physical constraint on the low end, or log-normal process | Check if the spec is appropriate. Some measurements (like response time) are naturally skewed. |
| Skewed left | Tail extends toward lower values | Physical constraint on the high end | Same as above, opposite direction |
| Bimodal (two peaks) | Two humps instead of one | Two populations mixed | Separate data by station, operator, shift, or component lot. Each population may be fine on its own. |
| Truncated | Distribution cut off sharply at one end | Parts are being screened, sorted, or the measurement saturates | Check if screening is intentional. If it is, standard Cpk formulas may overestimate capability. |
| Flat (uniform) | No clear peak, roughly even across range | Process is not controlled | Major investigation needed. Something is varying widely without control. |
| Comb (alternating high/low bins) | Jagged pattern | Measurement resolution too coarse, or rounding | Increase instrument resolution or check data rounding |

## The Normal Curve Overlay

TofuPilot overlays a normal distribution curve (Gaussian) on the histogram. This overlay uses the calculated mean and standard deviation from your data. When the histogram bars closely follow the curve, your data is approximately normal and standard capability formulas (Cp, Cpk) apply directly.

When the bars diverge from the curve (heavy tails, skew, multiple peaks), the standard formulas may not accurately represent your process. The histogram makes this visible at a glance.

## Specification Limits on the Histogram

TofuPilot draws USL and LSL as vertical lines on the histogram when limits are defined. This shows you:

- How much of the distribution falls within spec
- Whether the distribution is centered between the limits
- How much margin exists on each side

If the distribution tails extend past a spec line, some units are failing on that side. The visual makes it obvious which side needs attention.

## Mean and Sigma Lines

The histogram also shows the process mean (center line) and sigma bands (±1σ, ±2σ, ±3σ). In a normal distribution:

| Range | Contains |
|-------|----------|
| ±1σ | 68.3% of data |
| ±2σ | 95.4% of data |
| ±3σ | 99.7% of data |

If the 3σ lines are inside the spec limits, the process is capable (Cp > 1.0). If they extend past the spec limits, the process variation exceeds the spec window.

## Feeding Histogram Data to TofuPilot

Histograms require numeric measurements with enough data points to form a meaningful distribution. Define measurements with limits in your test code.

```python filename="histogram_test.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload

@htf.measures(
    htf.Measurement("output_voltage")
        .in_range(minimum=3.25, maximum=3.35)
        .with_units(units.VOLT),
    htf.Measurement("clock_frequency")
        .in_range(minimum=7.99, maximum=8.01)
        .with_units(units.HERTZ),
)
def measure_board(test):
    test.measurements.output_voltage = 3.301
    test.measurements.clock_frequency = 8.002

def main():
    test = htf.Test(measure_board)
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: "UNIT-0001")

if __name__ == "__main__":
    main()
```

Open the Process Control page, select a numeric measurement, and the histogram appears alongside the control chart. The normal curve, spec limits, and mean line render automatically. Right-click on a histogram bin to select data points in that value range or filter the view.

## Using Histograms for Process Improvement

Histograms are diagnostic tools. When you spot a non-normal shape, investigate:

**Bimodal?** Split the data. Filter by station, operator, or batch in TofuPilot's sidebar. Each subset may form its own normal distribution, revealing the source of the split.

**Skewed?** Check whether the measurement is inherently bounded (you can't have negative leakage current) or whether a process constraint is causing the asymmetry.

**Shifted?** Compare to the target. If the peak is close to one spec limit, re-center the process before it starts producing failures.

**Too wide?** The distribution fills the spec window. Reduce variation (better fixtures, tighter component specs, more stable environment) or widen limits if the product design allows it.

### Sequential vs Parallel vs Batch Testing
URL: https://www.tofupilot.com/guides/sequential-vs-parallel-vs-batch-testing
Compare the three test execution strategies, learn which one your fixture requires, and see how to configure parallel slots and shared phases.

Three execution strategies cover almost every production test station: test one unit at a time, test several independently, or test several as a group. The choice is set by your fixture, not by preference, and picking wrong either wastes hardware or produces results you cannot trust.

This guide defines each strategy, gives the decision rule, and shows how to configure them. For the underlying abstraction these strategies belong to, see [What Is a Test Process Model](/guides/what-is-a-test-process-model).

## The Three Strategies

| Strategy | Units in flight | Units start and stop | Typical fixture |
|---|---|---|---|
| Sequential | 1 | One at a time | Single-position fixture |
| Parallel | N | Independently, on their own schedule | N independent fixtures |
| Batch | N | Together, as a group | One fixture holding N units |

The difference between parallel and batch is coupling, not count. Both test multiple units at once. In parallel, a unit that finishes early is unloaded and replaced immediately. In batch, it waits for its neighbours.

## Sequential

One unit is loaded, tested start to finish, and unloaded. The next unit begins after.

```text filename="sequential.txt"
Unit 1  [====== 30s ======]
Unit 2                     [====== 30s ======]
Unit 3                                        [====== 30s ======]
```

Sequential is the right answer when the fixture holds one unit, when the test needs an instrument that physically cannot be shared, or when an operator must handle each unit anyway. It is also the correct starting point for a new procedure: get one unit passing reliably before adding concurrency.

Its cost is throughput. Test time is dominated by waiting, and with one unit in flight, nothing overlaps with that wait.

## Parallel

Several units are tested at the same time, each on its own schedule. Unit 2 failing at 5 seconds does not disturb unit 1, and its position is free for a replacement while the others are still running.

```text filename="parallel.txt"
Slot 1  [====== 30s ======][====== 30s ======]
Slot 2  [== 5s ==][====== 30s ======]
Slot 3  [====== 30s ======][====== 30s ======]
```

Parallel is the highest-throughput option and the default choice for a multi-position fixture. On a four-position fixture with a 30-second test, throughput moves from roughly 120 units per hour to close to 450, bounded by whatever the slots genuinely share.

Two conditions must hold. Each position needs its own instruments, or a shared instrument fast enough that queuing does not dominate. And the test must not care about its neighbours: no shared thermal chamber ramp, no measurement that assumes all units are in the same state.

## Batch

Several units are loaded together, tested together, and unloaded together. The group moves as a unit.

```text filename="batch.txt"
Unit 1  [====== 30s ======]        idle
Unit 2  [== 5s ==]  idle           (waits for the group)
Unit 3  [========== 45s ==========]
        └──────── batch ends at 45s ────────┘
```

Batch exists because some tests are physically shared. A thermal chamber ramps every unit inside it at once. An EMC pre-compliance sweep illuminates the whole tray. A burn-in rack powers a shelf together. In these cases the shared operation is the test, and the units cannot be decoupled.

Batch is also correct for traceability reasons: when units must be proven to have been tested under identical conditions, testing them together is the evidence.

The cost is visible in the diagram. Every unit pays the slowest unit's time, and the fixture cannot be reloaded until the group finishes.

## Choosing

Work down this list and stop at the first match:

1. **Does one physical operation cover every unit at once** (chamber ramp, shared RF field, common power rail)? Use **batch**. The units are coupled whether you model them that way or not.
2. **Must the units be proven tested under identical conditions?** Use **batch**.
3. **Does the fixture hold more than one unit, each with its own connections?** Use **parallel**.
4. **Otherwise**, use **sequential**.

A common mistake is reaching for batch because units are loaded as a tray. Loading is not testing. If the tray holds eight units and each has its own probe connection, that is parallel with a tray-shaped fixture, and modelling it as batch throws away throughput for nothing.

## Configuring Parallel Execution

In the TofuPilot Framework, each independently tested position is a slot, the [test socket](/guides/what-are-test-sockets-and-how-to-use-them) that owns one unit. Declaring slots turns a single-unit procedure into a parallel one:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: USB0
    - name: USB1
    - name: USB2
    - name: USB3
```

For a fixture with many identical positions, give a count instead and let the keys be generated:

```yaml filename="procedure.yaml"
execution:
  slots: 80
```

Each slot identifies its own unit and uploads its own run. Use `{slot}` in the unit defaults to derive serials per position:

```yaml filename="procedure.yaml"
unit:
  auto_identify: true
  serial_number:
    default_value: "BURNIN-{slot}"
  part_number:
    default_value: "PCB-MAIN-V2"
```

Slots fail independently. A failing phase stops that slot, runs its teardown, and leaves the others running. Only a failure in a shared setup or teardown phase, or a plug that cannot initialize, stops everything.

## Approximating Batch with Shared Phases

The Framework models batch behavior through phase scope rather than a separate model. Phases scoped to the execution run once for all slots; phases scoped to a slot run per unit.

```yaml filename="procedure.yaml"
execution:
  slots: 8

setup:
  - name: Ramp Chamber to 85C
    python: phases.ramp_chamber
    scope: execution

main:
  - name: Measure Leakage
    python: phases.measure_leakage

teardown:
  - name: Return Chamber to Ambient
    python: phases.cool_chamber
    scope: execution
```

The chamber ramp happens once, every unit measures its own leakage, and the chamber returns to ambient after every slot is done. That is batch semantics for the shared operations and parallel semantics for the per-unit measurements, which is usually what a batch fixture actually needs.

This matters for correctness, not just tidiness. A slot-scoped teardown can run while neighbouring slots are still testing, so anything touching a shared resource — a rack supply, a chamber, a fixture lock — must be scoped to the execution. Cooling the chamber in a slot-scoped teardown would sabotage every unit still under test.

## Phase-First and Slot-First

With slots configured, one more setting controls interleaving:

```yaml filename="procedure.yaml"
execution:
  strategy: phase_first
  slots: 4
```

`phase_first`, the default, runs the same phase across all slots before advancing. `slot_first` finishes one slot completely before starting the next.

`phase_first` suits a shared instrument: all four voltage measurements happen together, so the instrument is configured once for that measurement. `slot_first` is closer to sequential, useful when you want a complete result for the first unit as early as possible.

## Throughput Comparison

Four units, 30-second test, 5-second load and unload:

| Strategy | Cycle time | Units/hour | Hardware needed |
|---|---|---|---|
| Sequential | 140s | ~103 | 1 fixture position |
| Parallel (4 slots) | 35s | ~411 | 4 positions, 4 instrument sets |
| Batch (4 units) | 35s | ~411 | 4 positions, shared instruments |

Parallel and batch look identical here because every unit takes exactly 30 seconds. They diverge with real variation: batch pays the slowest unit every cycle, and pays it again on every retry.

## Common Pitfalls

**Sharing an instrument without accounting for queuing.** Four slots reading one DMM do not measure in parallel; the calls queue. If a shared instrument call takes 2 seconds, four slots spend 8 seconds on that phase, and your parallel speedup silently disappears. See [Share Instruments Across Parallel Slots](/guides/share-instruments-across-parallel-slots) for the arithmetic and the fix.

**Slot-scoped teardown on shared hardware.** Powering down a rack supply in a slot teardown kills units still under test. Shared resources belong in execution-scoped teardown.

**Batch used for loading convenience.** If units are independent, batch only adds waiting.

**Assuming parallel needs no code change.** Per-slot instrument addressing, serial number derivation, and fixture presence detection all have to be genuinely per-slot. Code that hardcodes one instrument address will happily run four slots against the same physical unit.

## Key Points

- Sequential, parallel, and batch differ in how many units are in flight and whether they are coupled.
- Coupling is decided by physics: a shared chamber or field means batch, independent connections mean parallel.
- Parallel gives the highest throughput; batch makes every unit pay the slowest unit's time.
- In the TofuPilot Framework, parallel is `execution.slots`, and batch semantics come from execution-scoped phases.
- Shared resources must be scoped to the execution, or a finishing slot will disturb the ones still running.

### What Are Test Sockets and How to Use Them
URL: https://www.tofupilot.com/guides/what-are-test-sockets-and-how-to-use-them
A test socket is one fixture position plus the context that owns its unit. Learn how sockets isolate results and how to configure them as slots.

A test socket is one position on a fixture that holds a unit under test, together with the software context that tracks it. A four-position fixture has four sockets, and the test executive runs four copies of the test, one per socket, each with its own unit, its own measurements, and its own result.

The term comes from NI TestStand, where sockets are the mechanism behind parallel and batch models. The TofuPilot Framework calls the same concept a slot. This guide explains what sockets do, what a socket identifier is for, and how to configure them.

## Why the Concept Exists

Running the same test on four units at once creates an identity problem. Four voltage measurements arrive, and every one of them is called `supply_voltage`. Without a per-position identity, there is no way to say which unit failed, which fixture position is drifting, or which serial number to attach to a result.

A socket solves this by being the thing that owns the unit for the duration of the test:

| A socket owns | Example |
|---|---|
| The unit under test | Serial `SN-2024-003`, in position 3 |
| Its instrument channels | DMM channel 3, PSU output 3 |
| Its measurements | The voltage read from that unit |
| Its outcome | Pass, independent of the other positions |
| Its result record | One run on the dashboard |

Four sockets means four independent test contexts sharing one station, one procedure, and one operator.

Sockets are the mechanism behind parallel and batch execution; for choosing between those strategies, see [Sequential vs Parallel vs Batch Testing](/guides/sequential-vs-parallel-vs-batch-testing).

## The Socket Identifier

Every socket carries an identifier your test code can read. This is the single most important thing a socket gives you, because it is how a generic test becomes position-aware.

The problem it solves: your phase code is written once but runs N times. Something has to tell instance 3 to talk to instrument channel 3. That something is the socket identifier.

Most phases never need it. When each socket has its own instrument, the plug is socket-scoped and the phase stays generic:

```python filename="phases/measure_voltage.py"
def measure_voltage(measurements, power_supply):
    measurements.voltage = power_supply.read_voltage()
```

When a phase does need to know its position — addressing a channel on a shared instrument, say — it reads `run.slot_id`, the key declared in `execution.slots`:

```python filename="phases/measure_rail.py"
CHANNELS = {"s1": 101, "s2": 102, "s3": 103, "s4": 104}


def measure_rail(measurements, run, dmm):
    measurements.rail_voltage = dmm.measure_dc_volts(CHANNELS[run.slot_id])
```

Keep per-socket wiring in configuration and lookup tables rather than branching on position inside the test logic. That is what keeps the procedure readable as the fixture grows from four positions to eighty.

## Configuring Sockets

In the TofuPilot Framework, sockets are declared as slots under `execution`:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: USB0
    - name: USB1
    - name: USB2
    - name: USB3
```

Each entry takes a `name` and an optional `key`, the identifier phases see. When the key is omitted it is derived from the name. Naming slots after the physical thing they correspond to — a USB port, a nest number, a probe head — is what makes a failure report actionable, because "USB2 has failed 40 times today" points at hardware.

For a fixture with many identical positions, give a count:

```yaml filename="procedure.yaml"
execution:
  slots: 80
```

The count expands to keys `slot_01` through `slot_80`, zero-padded so positions sort correctly everywhere. The count is capped at 1024, and the engine bounds slots times phases at 10,000 jobs per run. Use the list form when positions have meaningful names.

## Per-Socket Unit Identification

Each socket identifies its own unit. For a station where an operator scans each position, the unit prompt appears per socket. For a rack where serials are derived from position, use the `{slot}` and `{slot_name}` placeholders:

```yaml filename="procedure.yaml"
execution:
  slots:
    - key: s01
      name: Nest 1
    - key: s02
      name: Nest 2

unit:
  auto_identify: true
  serial_number:
    default_value: "BURNIN-{slot}"
  part_number:
    default_value: "PCB-MAIN-V2"
  batch_number:
    default_value: "{slot_name}"
```

With `auto_identify: true` the rack starts with no operator input at all. Without it, the expanded value pre-fills each socket's prompt, which is the useful middle ground: the operator confirms rather than types.

Single-slot procedures never see the placeholders and keep them as literal text, so adding them early does not break a single-position station.

## One Run Per Socket

Each socket uploads its own run: its unit, its phases plus any shared setup and teardown, its measurements, logs, attachments, and its own outcome.

The runs from one start share an `execution_id`, and each carries its `slot_key` and `slot_name`. That gives two views of the same data:

- Group by `execution_id` to see everything that happened in one cycle.
- Filter by `slot_key` to see one fixture position over time.

The second view is what catches a bad position. If nest 3 fails 12 percent while the other seven sit at 2 percent, the problem is the fixture, not the product — and no amount of aggregate yield analysis would have shown it.

## Socket-Scoped and Shared Resources

Instruments are attached with a scope that decides how many instances exist:

| Scope | Instance | Use for |
|---|---|---|
| `slot` (default) | One per socket | Per-position PSU channel, per-position serial port |
| `execution` | One shared by all sockets | A single DMM behind a mux, a chamber |
| `station` | One per station, held across runs | Slow-to-open VISA or TCP connections |

```yaml filename="procedure.yaml"
plugs:
  - name: power_supply
    python: plugs.psu:PowerSupply
    scope: slot

  - name: chamber
    python: plugs.chamber:ThermalChamber
    scope: execution
```

A plug instance runs one method call at a time. When several sockets call the same `execution` or `station` plug, the calls queue and execute one at a time. This is the main throughput trap in socket-based testing: a shared instrument method that takes two seconds costs eight seconds across four sockets, and the parallelism you configured quietly does not exist. Keep shared plug methods short and put long waits in phases. [Share Instruments Across Parallel Slots](/guides/share-instruments-across-parallel-slots) covers this in detail.

## Failure Isolation

Sockets fail independently. A failing phase under the default `stop` behavior, a `phase.stop()`, an error, or a timeout skips the remaining phases of that socket and runs its socket-scoped teardown, while the other sockets continue. Each socket gets its own outcome, and the run outcome is the worst of them.

Only three things stop every socket: a failure in a phase shared by all of them (`setup` or `teardown` with `scope: execution`), a plug that fails to initialize, or the operator stopping the run.

This creates one rule worth stating plainly, because violating it produces intermittent failures that are very hard to diagnose:

**A socket-scoped teardown can run while its neighbours are still testing.** Anything it touches must belong to that socket alone. A rack power supply, a chamber, a fixture lock, a shared mux belongs in a teardown phase with `scope: execution`, which runs once after every socket is done.

```yaml filename="procedure.yaml"
teardown:
  - name: Discharge This Position
    python: phases.discharge

  - name: Power Down Rack Supply
    python: phases.rack_off
    scope: execution
```

If `Power Down Rack Supply` were socket-scoped, the first socket to finish would cut power to every unit still under test. The symptom is a handful of unexplained failures on the slowest positions, which looks like a product problem and is not.

## Common Pitfalls

**Hardcoding an instrument address in a socket-scoped plug.** Four sockets then talk to the same physical channel, and four units report the first unit's measurements. Pass the address through per-socket plug config.

**Assuming sockets speed up a shared instrument.** They do not. Queuing on an `execution`-scoped plug serializes those calls by design.

**Naming sockets by index only.** `slot_03` is harder to act on than `Nest 3` or `Probe C`. Names appear in every failure report.

**Forgetting that shared setup failure stops everything.** A chamber that will not reach temperature correctly aborts all sockets. That is the intended behavior, but it means shared setup phases deserve retries and clear error messages more than per-socket phases do.

## Key Points

- A test socket is one fixture position plus the software context that owns its unit, measurements, and outcome.
- The socket identifier, read from `run.slot_id`, is what makes one generic test position-aware.
- In the TofuPilot Framework, sockets are `execution.slots`, and each uploads its own run tagged with `slot_key` and `execution_id`.
- Filtering results by socket is how you catch a bad fixture position that aggregate yield hides.
- Shared instruments serialize across sockets, and shared resources must use `scope: execution` in teardown.

### What Is a Test Process Model
URL: https://www.tofupilot.com/guides/what-is-a-test-process-model
A process model is the fixed lifecycle around your test code. Learn what it handles, why callbacks exist, and how Python frameworks replace it.

A process model defines the operations a test system performs around your test code: identifying the unit, initializing instruments, running the test, collecting results, and cleaning up. Your test sequence measures the product. The process model handles everything else.

The term comes from NI TestStand, where the process model is a separate sequence file that calls your test sequence. Other test executives use different names for the same idea. This guide explains what a process model does, why the concept exists, and how the same responsibilities are handled in a Python test framework.

## What a Process Model Handles

Every production test does more than measure. Before a single voltage reading, something has to know which unit is on the fixture. After the last measurement, something has to decide the outcome and store the result.

| Responsibility | Example |
|---|---|
| Unit identification | Scan a barcode, read a serial from EEPROM, prompt the operator |
| Resource initialization | Open a VISA session, connect to a power supply, load calibration data |
| Test execution | Run the measurement steps in order |
| Outcome resolution | Decide pass or fail from all step results |
| Result storage | Write a report, insert a database row, upload to a server |
| Cleanup | Power down, release the fixture, close connections |

Only one row in that table is your test. The rest is identical across every procedure on the station, which is exactly why test executives factor it out. Write it once, reuse it everywhere.

## Why the Abstraction Exists

Without a process model, every test script repeats the same boilerplate. Ten procedures means ten copies of the barcode scan, ten copies of the report writer, ten places to update when the database schema changes.

The process model inverts the relationship. Instead of your script calling a report function, the model calls your script:

```text filename="control-flow.txt"
Process Model
├── Identify unit
├── Initialize resources
├── ──> Your test sequence     (the only part you write per product)
├── Resolve outcome
├── Store results
└── Clean up
```

This is inversion of control applied to test software. Your test code becomes a plug-in to a fixed lifecycle, rather than a program that has to remember every step.

## The Three Classic Models

Test executives typically ship three built-in models, distinguished by how they handle multiple units:

| Model | Behavior | Use when |
|---|---|---|
| Sequential | One unit at a time, start to finish | Single-position fixture |
| Parallel | Independent units, each starting and finishing on its own schedule | Multiple independent fixtures |
| Batch | Multiple units loaded and tested together as a group | One fixture holding several units |

The distinction matters for throughput. A sequential model on a four-position fixture wastes three quarters of the hardware. For the decision rule and the throughput arithmetic, see [Sequential vs Parallel vs Batch Testing](/guides/sequential-vs-parallel-vs-batch-testing).

Each independently tested position in these models is a [test socket](/guides/what-are-test-sockets-and-how-to-use-them), the context that owns one unit for the duration of its test.

## Callbacks: Customizing Without Forking

A fixed lifecycle is only useful if you can hook into it. Test executives expose named extension points, usually called callbacks, that run at defined moments: before the unit loop, after each unit, before the report is generated.

Callbacks let one product override report formatting while every other product keeps the default, with no copy of the model. The tradeoff is indirection. Reading a test that relies on five overridden callbacks means opening six files to understand one run.

## The Same Responsibilities in Python

A Python test framework handles the same lifecycle, but declares it rather than modeling it as a separate callable sequence. In the TofuPilot Framework, the lifecycle is fixed by the engine and the procedure file describes what runs at each stage:

```yaml filename="procedure.yaml"
name: Battery Functional Test

unit:
  serial_number:
    default_value: "BAT000001"
  part_number:
    default_value: "BAT-001"

plugs:
  - name: power_supply
    python: plugs.psu:PowerSupply
    scope: station

setup:
  - name: Warm Up Instruments
    python: phases.warm_up

main:
  - name: Measure Voltage
    python: phases.measure_voltage
    measurements:
      - name: voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2

teardown:
  - name: Power Down
    python: phases.power_down
```

Every process model responsibility maps to a declaration:

| Process model concept | TofuPilot Framework |
|---|---|
| Unit identification | `unit:` block, with operator prompt or `auto_identify` |
| Resource initialization | `plugs:` with a lifetime `scope` |
| Pre-test callback | `setup:` phases |
| Test sequence | `main:` phases |
| Post-test callback | `teardown:` phases, which always run |
| Outcome resolution | Engine rule: worst phase outcome wins |
| Result storage | Built in, uploaded by the engine |

The phase functions themselves stay plain Python:

```python filename="phases/measure_voltage.py"
def measure_voltage(measurements, power_supply):
    measurements.voltage = power_supply.read_voltage()
```

## What Changes Without a Model File

Two things are genuinely different, and both are consequences of the lifecycle being fixed rather than editable.

**Result storage is not yours to write.** In a classic test executive, a report callback is a customization point, and teams write their own database logger. Here, uploading is the engine's job. You gain a schema you do not maintain; you lose the ability to change the storage format.

**The order of operations is not overridable.** You cannot reorder identification and plug initialization the way a custom model sequence could. What you can control is what runs inside each stage, plus the failure behavior:

```yaml filename="procedure.yaml"
execution:
  on_first_failure: continue
  workers: 16
```

For most production test software, the fixed lifecycle is the point. The customization that mattered was always the test itself, not the plumbing around it. If your model customization exists to reshape reporting or database writes, that need disappears rather than migrating.

For a callback-by-callback mapping, including which ones to delete rather than port, see [Replace TestStand Callbacks in Python](/guides/replace-teststand-callbacks-in-python).

## When You Still Need Custom Orchestration

Some systems genuinely do not fit a fixed lifecycle: a test rig that reconfigures itself mid-run, a depot workflow that decides what to test after a diagnostic pass, a stress loop that runs for days.

For those, drive the loop yourself and use an SDK to record results, rather than bending a declarative procedure into shapes it does not fit. The lifecycle abstraction earns its place when the flow is the same every cycle. When the flow is the product, write the flow.

## Key Points

- A process model is the fixed lifecycle around your test code: identify, initialize, execute, resolve, store, clean up.
- It exists to stop every procedure from repeating the same boilerplate.
- Sequential, parallel, and batch models differ only in how they handle multiple units.
- Callbacks customize the lifecycle without copying it, at the cost of indirection.
- A declarative Python framework keeps the same responsibilities but fixes the lifecycle, trading model-file flexibility for far less code to maintain.

### What Is Test Observability
URL: https://www.tofupilot.com/guides/what-is-test-observability
Test observability gives full visibility into what your test systems are doing, why they fail, and how they perform. Learn how it applies to manufacturing test.

Test observability is the ability to understand what your test systems are doing, why units are failing, and how station performance is changing, all from the data the systems produce. It borrows from software engineering's observability concept (logs, metrics, traces) and applies it to manufacturing test. This guide covers what test observability means, how it differs from test data management, and what it looks like in practice.

## Observability vs Data Management

| Aspect | Test Data Management | Test Observability |
|--------|---------------------|-------------------|
| Focus | Storing and retrieving test records | Understanding system behavior in real time |
| Question answered | "What happened to this unit?" | "Why is yield dropping right now?" |
| Data use | Compliance, traceability, audits | Debugging, optimization, early warning |
| Timeliness | After the fact (batch reports) | Real-time (streaming dashboards) |
| Scope | Individual test results | System-wide patterns and correlations |

Test data management answers: did this unit pass? Test observability answers: why are 15% of units failing phase_voltage_check on station 3 since Tuesday?

## The Three Pillars of Test Observability

Borrowing from software observability, test observability has three pillars:

### 1. Measurements (Metrics)

Structured measurement data with units, limits, and timestamps. This is the equivalent of application metrics in software.

| Metric | What It Shows |
|--------|-------------|
| First pass yield per station | Station health |
| Measurement distribution per phase | Process stability |
| Cycle time per unit | Throughput efficiency |
| Failure rate per test step | Where defects concentrate |
| Marginal rate per measurement | Early warning of drift |

### 2. Test Records (Logs)

Detailed records of every test run: serial number, phases executed, measurements collected, pass/fail decisions, timestamps, and errors.

| Record Field | Why It Matters |
|-------------|---------------|
| Serial number | Links test data to the physical unit |
| Phase sequence | Shows exactly what ran and in what order |
| Measurement values | The actual data for analysis |
| Error messages | What went wrong when a phase failed |
| Station ID | Which equipment ran the test |
| Firmware/software version | Which test script version was used |

### 3. Traces (Correlation)

The ability to trace a unit's journey across test stages: IQC, assembly, FCT, EOL, ORT. Traces connect upstream events to downstream outcomes.

| Trace | What It Reveals |
|-------|----------------|
| Unit history | Every test this serial number has ever been through |
| Lot traceability | All units from the same component lot |
| Station correlation | Whether failures cluster on specific stations |
| Temporal correlation | Whether failures cluster at specific times |

## What Observable Test Systems Look Like

### Without Observability

The test engineer gets a call: "yield dropped on line 2." They walk to the station, review logs, export data to Excel, build charts, and three hours later identify the root cause.

### With Observability

The test engineer sees an alert: "yield on station 3 dropped below 90% in the last hour. Top failure: phase_voltage_check. Measurement distribution shifted 200mV compared to yesterday." They open the dashboard, see the shift, correlate it with a supplier lot change at IQC, and contain the issue in minutes.

## Prerequisites

- Python 3.10+
- OpenHTF installed (`pip install openhtf`)
- TofuPilot Python SDK installed (`pip install tofupilot`)

## Step 1: Instrument Your Tests

Every test phase should produce structured measurements. This is the foundation of observability.

```python filename="observable_test.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("supply_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("supply_current_mA")
    .in_range(minimum=90, maximum=110)
    .with_units(units.MILLIAMPERE),
)
def phase_power_check(test):
    """Measure and validate power supply characteristics."""
    test.measurements.supply_voltage_V = 5.01
    test.measurements.supply_current_mA = 98.5


@htf.measures(
    htf.Measurement("signal_amplitude_V")
    .in_range(minimum=1.8, maximum=2.2)
    .with_units(units.VOLT),
)
def phase_signal_check(test):
    """Measure output signal amplitude."""
    test.measurements.signal_amplitude_V = 2.01
```

## Step 2: Stream to TofuPilot

TofuPilot provides the observability layer. Every run streams in real time with full measurement detail.

```python filename="observable_test.py"
from tofupilot.openhtf import upload

test = htf.Test(
    phase_power_check,
    phase_signal_check,
)

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan serial: "))
```

## Step 3: Monitor and Respond

TofuPilot provides the observability tools:

| Tool | What It Shows |
|------|-------------|
| Yield dashboard | Real-time FPY across stations and procedures |
| Failure Pareto | Which test steps fail most, ranked by frequency |
| Measurement distributions | Histograms with limit overlays for every measurement |
| Control charts | SPC charts detecting out-of-control conditions |
| Station comparison | Side-by-side performance across stations |
| Unit history | Full test trace for any serial number |
| Alerts | Notifications when yield drops below threshold |

## Observability Maturity Levels

| Level | Capability | Question Answered |
|-------|-----------|------------------|
| 1. Logging | Test results are stored | "Did this unit pass?" |
| 2. Monitoring | Dashboards show yield and throughput | "How is the line performing?" |
| 3. Alerting | Notifications on yield drops or measurement drift | "Is something going wrong right now?" |
| 4. Debugging | Drill down into failures, correlate across stations and lots | "Why is this happening?" |
| 5. Predicting | Use patterns to forecast future failures | "What will happen next?" |

Most manufacturing operations are at level 1 or 2. Levels 3-4 are where test observability delivers the highest ROI. Level 5 is where predictive quality begins.

### Migrate OpenHTF Scripts to TofuPilot CLI
URL: https://www.tofupilot.com/guides/migrate-openhtf-scripts-to-tofupilot-cli
Run existing OpenHTF scripts on the TofuPilot CLI with no rewrite, and decide which ones are worth porting to the TofuPilot Framework.

If you already have OpenHTF test scripts running in production, the first thing worth knowing is that you do not have to rewrite them. The TofuPilot CLI runs OpenHTF natively: phases, measurements, logs, attachments and operator prompts stream to the dashboard with no code changes. This guide covers that path, and then the separate question of when porting to the TofuPilot Framework is worth the effort.

## Step 1: Run Your Existing Script on the CLI

Install the CLI:

```bash filename="install.sh"
curl -fsSL https://www.tofupilot.app/install | sh
```

Point it at your existing OpenHTF entry point:

```bash filename="run.sh"
tofupilot run ./main.py
```

That is the whole migration for most scripts. The CLI detects the OpenHTF test, executes it, and streams the run. You get real-time streaming, the operator UI, offline queueing and deploy-on-push without touching the test code.

## Step 2: Decide How Results Reach the Dashboard

There are two supported paths, and they are not exclusive.

**Through the CLI.** Run with `--upload` on a linked procedure. Nothing in the script changes.

```bash filename="upload.sh"
tofupilot login
tofupilot link
tofupilot run ./main.py --upload
```

**Through the Python callback.** If you manage your own distribution and do not want the CLI in the loop, the `tofupilot` package provides an OpenHTF output callback:

```bash filename="install_sdk.sh"
pip install "tofupilot[openhtf]"
```

```python filename="main.py"
import openhtf as htf
from tofupilot.openhtf import upload


def phase_one(test):
    return htf.PhaseResult.CONTINUE


def main():
    test = htf.Test(
        phase_one,
        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        part_number="PCB01",
    )
    test.add_output_callbacks(upload())
    test.execute(lambda: "SN-0001")


if __name__ == "__main__":
    main()
```

`upload()` reads `TOFUPILOT_API_KEY` from the environment. `add_output_callbacks` is OpenHTF's own extension point, so the callback composes with any others you already use, such as `json_factory`.

For a self-hosted instance with a private CA:

```python filename="main.py"
test.add_output_callbacks(upload(
    url="https://tofupilot.yourcompany.com",
    verify="/path/to/ca-certificate.pem",
))
```

## Step 3: Migrate Off the Deprecated Context Manager

If your scripts use `with TofuPilot(test):`, that wrapper is deprecated. On `tofupilot>=2.16.0` it still works as a thin wrapper around the same callback and emits a `FutureWarning`. Versions 2.15.0 to 2.15.2 do not include it at all and raise `ImportError`.

Replace it with the two-line callback form:

```python filename="before.py"
from tofupilot import TofuPilot

with TofuPilot(test):
    test.execute(test_start=lambda: input("Serial: "))
```

```python filename="after.py"
from tofupilot.openhtf import upload

test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Serial: "))
```

Three details when you make that change:

**OpenHTF is an extra since 2.15.0.** Install `tofupilot[openhtf]`, not plain `tofupilot`, so a station that only uses the API client does not pull a test framework it never imports.

**MQTT streaming is gone.** `stream=True` is accepted but ignored with a logged warning; pass `stream=False` to silence it. The live operator UI is provided by the CLI, so run with `tofupilot run` if you relied on it.

**Ctrl-C behaves differently.** The wrapper suppressed `KeyboardInterrupt`, so Ctrl-C exited quietly. The plain callback lets it raise. If your bench scripts depended on the quiet exit, wrap `test.execute(...)` in `try/except KeyboardInterrupt`.

## Step 4: Deploy to Stations

Once the script runs under the CLI, deployment is a Git push. The station pulls the new version, with immutable build artifacts and instant rollback if it misbehaves. Runs queue locally when the network drops and sync when it returns.

Nothing about that requires the TofuPilot Framework. OpenHTF procedures deploy the same way.

## When to Port to the TofuPilot Framework

Porting is optional. It is worth doing for specific reasons, not on principle.

| Reason to port | Why OpenHTF cannot do it |
|---|---|
| The line is throughput-bound | OpenHTF phases run sequentially; the Framework runs independent phases in parallel |
| Several boards on one fixture | Multi-slot execution, each slot its own run and outcome |
| Operators need rich guided workflows | Declarative forms and displays, no frontend work |
| Validating waveform or sweep shape | Aggregations and per-axis validators |
| You want limits reviewable as data | Limits live in YAML, diffable separately from the logic |

| Reason not to port | |
|---|---|
| It works and the line is not throughput-bound | The migration cost buys nothing |
| One pure-Python paradigm is a team preference | The YAML layer is a real concept to learn |
| Apache 2.0 matters to your legal review | OpenHTF's patent grant |

## The Port, Concretely

If you do port, the mapping is mechanical.

| OpenHTF | TofuPilot Framework |
|---|---|
| Phase function with `@htf.measures` | Plain function; measurements declared in YAML |
| `.in_range(3.2, 3.4)` | `validators` with `>=` and `<=` |
| `.equals("2.1.0")` | `validators` with `operator: "=="` |
| `BasePlug` subclass | Plain Python class |
| `setUp` / `tearDown` | `__init__` / `close` |
| `@htf.plug(dut=DutPlug)` | Plug injected by parameter name |
| `test.measurements.x = v` | `measurements.x = v` |
| `htf.Test(phase1, phase2)` | `main:` list in the procedure |
| `@htf.PhaseOptions(timeout_s=10)` | `timeout` on the phase |
| Phase ordering in `htf.Test(...)` | `depends_on` |

Before:

```python filename="openhtf_version.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units


class DutPlug(BasePlug):
    def setUp(self):
        self.connected = True

    def read_voltage(self) -> float:
        return 3.31

    def tearDown(self):
        self.connected = False


@htf.measures(
    htf.Measurement("rail_3v3").in_range(3.2, 3.4).with_units(units.VOLT),
)
@htf.plug(dut=DutPlug)
def test_power(test, dut):
    test.measurements.rail_3v3 = dut.read_voltage()


test = htf.Test(test_power, part_number="PCBA-100")
test.execute(test_start=lambda: input("Serial: "))
```

After:

```python filename="plugs/dut.py"
class Dut:
    def __init__(self):
        self.connected = True

    def read_voltage(self) -> float:
        return 3.31

    def close(self):
        self.connected = False
```

```python filename="phases/power.py"
def power(measurements, dut):
    measurements.rail_3v3 = dut.read_voltage()
```

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dut
    python: plugs.dut:Dut

main:
  - name: Power
    python: phases.power
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

## Validate Before You Switch

Whichever path you take, run the old and new versions against the same units, including known-bad ones, and compare measurement by measurement. A limit transcribed wrong during a port is the failure mode that ships bad boards, and it is cheap to catch this way.

Keep the measurement names identical across the port. Renaming `rail_3v3` splits the history and you lose the trend you were trying to preserve.

## Mixing Both

You do not have to pick one for the whole plant. Keep OpenHTF where it already runs, and use the Framework on new stations that need parallel slots or richer operator workflows. Both run on the same CLI and upload to the same dashboard, so yield and traceability stay consistent across the mix.

See [TofuPilot Framework vs OpenHTF](/guides/tofupilot-framework-vs-openhtf) for the full comparison.

## Common Mistakes

| Mistake | Consequence | Fix |
|---|---|---|
| Installing plain `tofupilot` for an OpenHTF script | `ImportError` on `tofupilot.openhtf` | Install `tofupilot[openhtf]` |
| Keeping `with TofuPilot(test):` | `FutureWarning`, and `ImportError` on 2.15.x | Move to `add_output_callbacks(upload())` |
| Expecting `stream=True` to do something | Silently ignored | Use `tofupilot run` for the live UI |
| Renaming measurements during a port | Trend history splits in two | Keep names byte-identical |
| Porting without parallel validation | A transcription error reaches production | Run both on the same units first |
| Porting because it is newer | Cost with no benefit | Port for parallelism, UI or waveform needs |

### Detect Latent Defects with Fleet Test Data
URL: https://www.tofupilot.com/guides/detect-latent-defects-with-fleet-test-data
Learn how to find units that pass every individual limit but carry a latent defect, using a derived measurement and fleet-level control charts.

A latent defect is a unit that passes every individual limit on the bench but fails in service. This guide shows how one derived measurement plus a fleet-level control chart makes that population visible, using a simulated avionics contactor fleet as the worked example.

The pattern matters most in maintenance and repair environments: high mix, low volume, complex degradation. A per-unit OK/NG verdict answers "can this unit ship today", but it says nothing about which passing units are quietly degrading. That signal only exists at fleet level.

## Prerequisites

- A test procedure uploading runs to TofuPilot
- A measurement that captures the defect signature (see Step 1)
- Two or more shop visits per unit, so per-serial trends exist

## Step 1: Record a signature measurement

Individual limits are set per measurement, so a defect that shifts the *relationship* between measurements stays invisible to them. Record that relationship as its own measurement.

In the contactor example, each pole's voltage drop has a 100 mV limit. A latent coil defect raises one pole about 45 mV above the others while all three stay under 100 mV. The imbalance between poles is the signature:

```python filename="phases/contact_voltage_drop.py"
# Imbalance between poles is a better early indicator than any single pole.
spread = round(max(drops.values()) - min(drops.values()), 2)
measurements.contact_drop_spread = spread
```

Give the derived measurement its own limit (40 mV here). One line in the bench script, and every past and future run carries the signature.

## Step 2: Read the fleet distribution

Open measurement control, select the derived measurement over the full history, no other filters.

A healthy fleet produces one population. A fleet with a latent defect mode produces two. In the contactor fleet (150 units, 24 months, 297 measurements), the histogram is bimodal: the healthy body sits near 11 mV, and a second population sits at 42-55 mV, above the 40 mV limit. Cpk on the same view (0.65 here) quantifies what the histogram shows.

![Fleet control chart of contact drop spread showing a bimodal distribution: healthy units near 11 mV and a defect population above the 40 mV limit](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/latent-defects-fleet-spread.jpg-370aab47-3f41-48ce-ad6b-782efdbf5db7-eE3PxVhhRF43WoLy58I9cJqVWWQpLT.jpg)

No hypothesis was needed: selecting the measurement is the entire analysis. This is the practical difference from spreadsheet work, where each candidate relationship must be extracted, pivoted, and plotted one at a time.

## Step 3: Separate latent from overt units

Filter the measurement outcome to FAIL and compare with the per-pole measurements:

| Population | Definition | Contactor fleet |
|---|---|---|
| Overt | At least one individual limit failing | 14 units |
| Latent | Every individual limit passing, signature over limit | 9 units |

The 9 latent units are the ones a per-pole bench ships. They are also the future no-fault-found cases: removed on a squawk, tested OK on every individual limit, returned to stock.

## Step 4: Scope one serial and read the trend

Filter to a single serial with the serial number filter and select the drifting measurement.

On the example unit, pole A2 climbs linearly from 92.5 mV to 104.6 mV across six shop visits as operating hours accumulate, crossing its 100 mV limit between visits four and five. The spread measurement had been over its limit since the first visit, 19 months earlier.

![Control chart scoped to one serial: contact drop A2 climbing linearly across six shop visits and crossing its 100 mV limit](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/latent-defects-ctr-000001-a2.jpg-9e4f078b-dd93-4467-b3f6-15ff69de90bc-AaLW3lxjyXaZxv4VHhgaMNjhsYZbbL.jpg)

On paper this unit is an intermittent repeat visitor. On the chart it is monotonic degradation with a predictable crossing date, which changes the maintenance decision from "test again and return to stock" to "replace the coil now".

## Step 5: Watch for it automatically

Two alert types cover this pattern without manual review:

- A **retest threshold** alert fires when a unit accumulates repeated tests on the same procedure, which is how no-fault-found loops look in the data.
- A **measurement drift** alert grades each measurement against its own baseline and fires when the series genuinely moves, catching the per-serial climb of Step 4 as it happens.

### Replace TestStand Callbacks in Python
URL: https://www.tofupilot.com/guides/replace-teststand-callbacks-in-python
Map each TestStand callback to its Python equivalent, and learn which ones you should delete instead of porting during a migration.

Callbacks are usually the hardest part of moving a test system off a traditional test executive. The measurement code ports easily: a voltage read is a voltage read. The callbacks are harder, because they encode behavior that lives outside any single test sequence, and because much of what they do disappears rather than migrates.

This guide maps each common callback to its Python equivalent, and identifies which ones you should not port at all.

## What Callbacks Do

In NI TestStand, a process model defines the lifecycle around your test sequence, and callbacks are the named points where a sequence can override the default. `PreUUT` prompts for a serial number. `PostUUT` handles the result. `TestReport` formats the report. Override one in a client sequence and your version runs instead of the model's.

The pattern solves a real problem: customizing the lifecycle for one product without copying the whole model. Its cost is indirection, since understanding one run can mean opening the model, the client sequence, and every override.

## The Mapping

| Callback | Purpose | Python equivalent |
|---|---|---|
| `PreUUTLoop` | Once before testing begins | Station-scoped plug initialization |
| `PreUUT` | Identify the next unit | `unit:` block, prompt or `auto_identify` |
| `MainSequence` | The test itself | `main:` phases |
| `PostUUT` | Act on one unit's result | `teardown:` phases |
| `PostUUTLoop` | Once after the last unit | Station-scoped plug teardown |
| `TestReport` | Format the report | Not ported: built in |
| `LogToDatabase` | Store results | Not ported: built in |
| `ProcessModelPostStep` | After every step | Not ported: engine records phases |
| `SequenceFilePreStep` | Before each step in a file | `setup:` phases or `depends_on` |

Four of nine are not ported. That is the main finding of most migrations: a large share of callback code exists to move results into storage, and that job no longer belongs to your test code.

## PreUUT: Identifying the Unit

`PreUUT` typically prompts for a serial number and validates it, deciding whether to continue or skip.

In the TofuPilot Framework, identification is declared rather than coded:

```yaml filename="procedure.yaml"
unit:
  serial_number:
    default_value: "SN000001"
  part_number:
    default_value: "PCB-MAIN-V2"
```

The operator is prompted before the test starts. For a station that derives serials instead of asking, `auto_identify` removes the prompt:

```yaml filename="procedure.yaml"
unit:
  auto_identify: true
  serial_number:
    default_value: "BURNIN-{slot}"
  part_number:
    default_value: "PCB-MAIN-V2"
```

When identification needs real logic — reading a serial from EEPROM, checking an MES for a work order, refusing a unit already tested — that becomes the first setup phase:

```python filename="phases/identify.py"
def identify(phase, unit, scanner, mes):
    serial = scanner.read()

    if not mes.is_released(serial):
        phase.fail(f"{serial} has no released work order")
        return

    unit.serial_number = serial
```

A failing setup phase stops the run before any main phase executes, which is the behavior `PreUUT` returning False was giving you.

## PostUUT: Acting on the Result

`PostUUT` runs after the test with the result available, typically to drive a pass/fail indicator, print a label, or route the unit.

This becomes a teardown phase. Teardown always runs, whether the test passed, failed, errored, or timed out:

```yaml filename="procedure.yaml"
teardown:
  - name: Signal Result
    python: phases.signal_result
```

```python filename="phases/signal_result.py"
def signal_result(unit, indicator, printer):
    indicator.ready()
    printer.print_label(unit.serial_number)
```

Two differences from `PostUUT` are worth knowing.

**Teardown runs per slot.** On a multi-socket station a slot's teardown can run while other slots are still testing, so anything touching shared hardware needs `scope: execution` to run once, after every slot is done.

**Result-dependent branching is usually the wrong shape here.** A `PostUUT` that formatted a report or wrote a database row has no equivalent, because the engine uploads the run itself. What remains is physical: light a lamp, print a label, release a fixture. When an action genuinely must differ on pass and fail, drive it from the phase that made the determination rather than reconstructing the verdict in teardown.

## PreUUTLoop and PostUUTLoop: Station Lifetime

These run once around the whole testing session, usually to open instrument connections and close them at the end.

There is no phase equivalent, because this is not test logic. It is resource lifetime, and it belongs to the plug:

```yaml filename="procedure.yaml"
plugs:
  - name: power_supply
    python: plugs.psu:PowerSupply
    scope: station
    config:
      address: "192.168.1.100"
```

A `station`-scoped plug is created on first use and held across executions, so back-to-back units do not pay the reconnection cost. Connection setup goes in `__init__`, cleanup in `__del__`:

```python filename="plugs/psu.py"
import pyvisa


class PowerSupply:
    def __init__(self, address: str):
        self._rm = pyvisa.ResourceManager()
        self._inst = self._rm.open_resource(f"TCPIP::{address}::INSTR")

    def read_voltage(self) -> float:
        return float(self._inst.query("MEAS:VOLT?"))

    def __del__(self):
        self._inst.close()
```

The engine terminates the plug subprocess on deletion rather than relying on garbage collection, so cleanup is deterministic. It also health-checks a held plug process before each reuse and respawns it if it died — one of the failure modes a hand-written `PreUUTLoop` usually did not cover.

Note the scope boundary: a `station` plug is released when a new deployment is applied, so plug code changes always take effect on the first run after a deploy. In a one-shot `tofupilot run`, `station` behaves as `execution`.

## TestReport and LogToDatabase: Do Not Port

These are the two callbacks teams invest the most in, and the two that should not survive the migration.

In a classic executive, results are your problem. You write a report callback to produce XML, a database callback to insert rows, and a schema to hold them. It is normal for this to be thousands of lines, and it is normal for it to be the least-loved code in the repository.

In the TofuPilot Framework the engine uploads the run: procedure, unit, phases, measurements, logs, attachments, outcome. Stations queue offline and drain on reconnect with the original timestamp.

The instinct to port a report callback anyway is worth resisting. If the goal was an XML file for a customer, that is a report generated from stored data, not a step inside the test. Keeping it in the test means every station needs the formatting code, every schema change is a redeploy, and a formatting bug fails a passing unit.

The honest tradeoff: you no longer control the storage format. If you have a genuine requirement to write a specific schema into a system of record, that is an export from the API, running once, off the station.

## ProcessModelPostStep: Do Not Port

`ProcessModelPostStep` runs after every step, typically to log step results or accumulate statistics.

The engine already records every phase: name, outcome, duration, measurements with their validators, retry count. It arrives in the dashboard without a callback, and phase-level analysis like a pareto of failures is a view over that data.

Port this only if it did something genuinely unusual, such as toggling a watchdog between steps. That becomes ordinary code in the phases that need it, not a global hook.

## Per-Step Hooks: depends_on and Setup

`SequenceFilePreStep` and similar per-step hooks are often used to enforce ordering or preconditions.

Ordering is declared directly:

```yaml filename="procedure.yaml"
main:
  - name: Power On
    python: phases.power_on
    key: power_on

  - name: Measure Rails
    python: phases.measure_rails
    depends_on: [power_on]
```

Phases without dependencies run in parallel on a worker pool; `depends_on` serializes what must be ordered. This replaces a hook that existed to check "did the previous step pass" with a statement of the actual dependency.

Preconditions that apply to everything go in `setup`, which must pass before any main phase runs. Preconditions for one phase go in the phase, using `phase.skip()`:

```python filename="phases/advanced_calibration.py"
def advanced_calibration(phase, device):
    if not device.supports_advanced_mode():
        phase.skip()
        return

    device.calibrate()
```

## Failure Behavior

Much callback code exists to control what happens after a failure. That is configuration:

```yaml filename="procedure.yaml"
execution:
  on_first_failure: continue
```

The default `stop` cancels phases that have not started. `continue` runs them anyway, which is what you want when a full failure profile is more valuable than cycle time.

For intermittent failures, retry is per phase and explicit:

```yaml filename="procedure.yaml"
main:
  - name: Network Connect
    python: phases.network
    retry:
      limit: 5
      delay: 1s
```

Retries do not happen automatically on failure. The phase must return a retry action, or `then:` must map an outcome to a retry. Every attempt is preserved and uploaded with a `retry_count`, so a unit that passed on the fourth attempt is visible as such rather than looking clean.

## A Migration Order That Works

1. **Inventory the callbacks and mark each one.** Most fall into: reporting or database (delete), lifecycle (becomes plug scope), or real logic (becomes a phase).
2. **Port the plugs first.** Instrument connections are the foundation, and getting them right makes everything else testable.
3. **Port `MainSequence` to `main` phases**, one measurement at a time, with validators in YAML rather than in the code.
4. **Move `PreUUT` into the `unit:` block**, adding a setup phase only if identification has real logic.
5. **Move `PostUUT` into teardown**, checking scope for anything touching shared hardware.
6. **Delete the reporting and database callbacks.** Run both systems in parallel for a batch and compare stored results before removing the old path.
7. **Add slots last**, once a single unit passes reliably.

Step 6 is the one that stalls. It is worth agreeing early with whoever owns the quality records what "the results are in the new system" means, because that conversation, not the code, is usually the critical path.

## Key Points

- Callbacks split into three groups: lifecycle, real logic, and result plumbing.
- Result plumbing — reports, database writes, per-step logging — is not ported; the engine records and uploads runs.
- Lifecycle callbacks become plug scope: `station` for connections held across runs, `execution` for shared resources, `slot` per unit.
- Real logic becomes phases: setup for preconditions, teardown for post-test actions that always run.
- Failure handling and ordering become configuration: `on_first_failure`, `retry`, and `depends_on`.

### Migrate from NI VeriStand to Python
URL: https://www.tofupilot.com/guides/migrate-from-ni-veristand-to-python
Split a VeriStand system into what stays on the real-time engine and what moves to Python plugs, phases and measurements, with the licensing outcome.

NI VeriStand is two products sold as one. The first is a real-time execution engine: it runs plant models and maps I/O on a PXI or CompactRIO target at fixed loop rates. The second is everything around that engine: stimulus profiles, the workspace, TDMS logging, alarms, the .NET automation API, and the result files. Python replaces the second part well. It replaces the first part only in specific cases, and a migration plan that does not separate the two ends up either rebuilding a real-time platform by accident or keeping licenses it no longer needs.

This guide sorts a VeriStand system into layers, says which ones move to Python, which stay on the engine, and which are deleted rather than ported.

## What a VeriStand System Contains

| Layer | VeriStand artifact | Outcome |
|---|---|---|
| Real-time engine | System definition (`.nivssdf`), hardware I/O mapping, compiled models, custom devices | Stays, or is removed because it was never needed |
| Deterministic test logic | Real-time sequences (`.nivsseq`), stimulus profiles (`.nivsstimprof`) | Ported: `niveristand` for deterministic parts, plain phases for the rest |
| Reactive configuration | Alarms, procedures, calculated channels | Split: protection stays, pass/fail criteria become validators |
| Operator interaction | Workspace screens | Ported: unit identification and operator UI phases |
| Data logging | TDMS files from the Data Logging tool or profile logging | Ported: multi-dimensional measurements, raw file attached |
| Automation | TestStand calls, macros, `.NET` API scripts | Ported: a VeriStand plug and phases |
| Results | TDMS on a share, DIAdem reports, custom SQL | Not ported: the engine uploads runs |

The rest of the guide goes through these layers in the order a migration usually touches them. The first decision, the engine, is the one everything else depends on.

## Decide First: Do You Need the Engine?

There is no open-source equivalent of the VeriStand real-time engine. A Linux machine with the PREEMPT_RT patch, FMPy for model execution and your own I/O code can replace it, but that is a platform build, not a migration, and it takes a team a few quarters. The realistic question is which of three situations you are in.

**Genuine HIL.** A plant model runs at 1 to 10 kHz, the loop is closed through hardware I/O, the unit under test is an ECU or a controller, and fault injection happens on sensor lines. The engine stays. What moves to Python is the test logic, the logging, the automation and the results. This is the common case and most of this guide addresses it.

**Bench work that landed on VeriStand because it was installed.** A power supply, a DAQ card and a multimeter, no plant model, no loop to close. The engine does nothing here that `nidaqmx` and PyVISA do not, and it costs a deployment license per bench. Python talks to the instruments directly. See [Removing the Engine](#removing-the-engine) below.

**Model-in-the-loop on Windows.** A VeriStand PC license running an FMU against a test sequence, with no real-time target. FMPy loads the same FMU on the host, and timing accuracy on Windows is enough for this class of test because there was no deterministic target to begin with.

The licensing consequence matters because it is the budget line the migration is usually justified on. A VeriStand Full development seat is a subscription. A rig that only runs a pre-configured system definition needs an Operator license, which is a one-time purchase. After migration, development happens in Python on any machine, and each rig that keeps its engine needs one Operator license and nothing else. NI does not publish a complete price list and prices vary by region and agreement, so the figures below are distributor list prices from 2023 to 2025 and only useful as orders of magnitude.

| License | Type | Order of magnitude |
|---|---|---|
| VeriStand Full | Development, per seat, yearly subscription | 3,400 to 4,000 USD or EUR per year |
| VeriStand PC | Development without real-time target, yearly | 2,400 per year |
| VeriStand Operator | Deployment, per rig, perpetual | 3,000 one-time |

A team with three Full seats and four rigs typically ends with four Operator licenses and one Full seat kept to open the archive.

## The VeriStand Plug

If the engine stays, the first piece of Python to write is a plug that owns the connection to it. The `niveristand` package installs with `pip` and wraps the .NET client API that ships with VeriStand, so this runs on the Windows host that already talks to the target.

```yaml filename="procedure.yaml"
plugs:
  - name: HIL Rig
    key: rig
    python: plugs.veristand:VeriStandRig
    scope: station
    config:
      sysdef: "C:\\HIL\\Powertrain\\Powertrain.nivssdf"
      gateway: "localhost"
      log_dir: "C:\\HIL\\Logs"
```

```python filename="plugs/veristand.py"
from niveristand.legacy import NIVeriStand


class VeriStandRig:
    def __init__(self, sysdef: str, log_dir: str, gateway: str = "localhost", deploy_timeout_ms: int = 120000):
        self.log_dir = log_dir
        NIVeriStand.LaunchNIVeriStand()
        NIVeriStand.WaitForNIVeriStandReady()
        self._ws = NIVeriStand.Workspace2(gateway)
        # deploy=True pushes the system definition to the target; takes a minute on PXI
        self._ws.ConnectToSystem(sysdef, True, deploy_timeout_ms)

    def get(self, channel: str) -> float:
        return self._ws.GetSingleChannelValue(channel)

    def set(self, channel: str, value: float) -> None:
        self._ws.SetSingleChannelValue(channel, value)

    def __del__(self):
        self._ws.DisconnectFromSystem("", True)
```

Two details of the plug carry most of the value.

**Scope is `station`.** Deploying a system definition to a PXI target takes on the order of a minute, and a `station` plug does it once, when the first run needs it, then keeps the connection across runs. Every unit tested after the first starts immediately. If a rig has to be redeployed between units, `execution` scope does that, at the cost of the deploy time per run.

**Channel paths are the interface.** A VeriStand channel is addressed by its path in the system definition, such as `Targets/Controller/Simulation Models/Models/Engine/Outports/RPM`, or by an alias like `Aliases/DesiredRPM`. Aliases are the better contract, because a model or hardware change that moves a channel breaks a path but not an alias. Define aliases for everything the test touches and keep the `.nivsalias` file next to the procedure.

## Stimulus Profiles and Real-Time Sequences

A stimulus profile is the closest thing VeriStand has to a test sequence: it sets channels, waits, checks conditions and logs. A real-time sequence is the deterministic part, compiled and run on the engine. They port along two different paths, and choosing wrong is the most common way a migrated HIL test gets slower or less trustworthy.

**Deterministic on the engine.** `niveristand` converts a decorated Python function into a real-time sequence and runs it on the target with `run_py_as_rtseq`. The function keeps VeriStand's timing guarantees, at the cost of VeriStand's restrictions: values are typed wrappers accessed through `.value`, comparisons take two operands only, and the body can only use the `niveristand` library, because it is compiled, not interpreted.

```python filename="sequences/rpm_step.py"
from niveristand import NivsParam, nivs_rt_sequence
from niveristand.clientapi import ChannelReference, DoubleValue
from niveristand.library import wait_until_settled


@NivsParam("setpoint", DoubleValue(0), NivsParam.BY_VALUE)
@nivs_rt_sequence
def rpm_step(setpoint):
    desired = ChannelReference("Aliases/DesiredRPM")
    actual = ChannelReference("Aliases/ActualRPM")
    settled = DoubleValue(0)

    desired.value = setpoint.value
    # upper, lower, tolerance window, timeout: all evaluated on the engine tick
    settled.value = wait_until_settled(actual, 9999999, setpoint.value - 50, 25, 10)
    return settled.value
```

```python filename="phases/rpm_response.py"
from niveristand import run_py_as_rtseq
from sequences.rpm_step import rpm_step


def rpm_response(phase, measurements, rig):
    result = run_py_as_rtseq(rpm_step, rtseq_params={"setpoint": 2500})
    if result != 0:
        phase.fail("RPM did not settle within 10 s")
        return

    measurements.actual_rpm = rig.get("Aliases/ActualRPM")
```

**Host-side in a phase.** Setting a setpoint, waiting a few seconds and reading a steady-state value does not need the engine's timing. The gateway round trip is in the tens of milliseconds, and for a test whose tolerances are in seconds that is invisible. This path has no restrictions: any library, any data structure, breakpoints in a normal debugger.

The rule that separates the two: if the sequence reacts to a signal within a step, or its timing requirement is under about 100 ms, it stays deterministic. Everything else moves to a phase, which is where it becomes easy to read and test.

A stimulus profile that mixed both, which most do, becomes one phase per test step, with the deterministic parts as sequence functions the phase calls. `depends_on` replaces the profile's step ordering, and `timeout` on each phase replaces the profile's step timeout.

## Alarms, Procedures and Calculated Channels

These three features are configured in the system definition and executed by the engine, and they serve two different purposes that VeriStand does not distinguish.

**Protection.** An alarm on battery current that triggers a procedure cutting the supply protects hardware within one engine tick. That stays in the system definition, because Python on the host cannot react in time, and because a protection that depends on a test running is not a protection.

**Pass/fail criteria.** An alarm on coolant temperature that exists so the operator sees a red indicator when the test should fail is not protection, it is a limit. That becomes a validator on the logged channel, where it is versioned with the test, applied identically on every rig and recorded with the result:

```yaml filename="procedure.yaml"
main:
  - name: Thermal Soak
    python: phases.thermal_soak
    timeout: 25m
    measurements:
      - name: Coolant Temperature
        title: Coolant Temperature
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Temperature
            key: temperature
            unit: °C
            aggregations:
              - type: max
                validators:
                  - operator: "<="
                    expected_value: 105
              - type: mean
                validators:
                  - operator: "<="
                    expected_value: 92
```

Calculated channels follow the same split. One that feeds a model or an alarm stays. One that existed only to log or display a derived value, such as power from voltage and current, is a line of NumPy in the phase that reads the log.

## Data Logging: TDMS to Measurements

VeriStand logs to TDMS, either continuously through the Data Logging tool or per step from a stimulus profile. Those files are the raw record and keeping them is right. What changes is that the test now extracts what it needs from the file, validates it, and attaches the original.

```python filename="phases/thermal_soak.py"
import time
from pathlib import Path

import numpy as np
from nptdms import TdmsFile


def thermal_soak(measurements, attach, log, rig):
    # The Data Logging tool in the system definition starts a 1 kHz TDMS log
    # while Aliases/LogTrigger is high, into the folder given in the plug config
    rig.set("Aliases/HeaterEnable", 1)
    rig.set("Aliases/LogTrigger", 1)
    time.sleep(20 * 60)
    rig.set("Aliases/LogTrigger", 0)
    rig.set("Aliases/HeaterEnable", 0)

    log_path = max(Path(rig.log_dir).glob("*.tdms"), key=lambda p: p.stat().st_mtime)
    tdms = TdmsFile.read(log_path)
    channel = tdms.groups()[0]["CoolantTemp"]  # names follow the log configuration
    temperature = channel[:]
    time_s = channel.time_track()

    # 1 kHz over 20 min is 1.2 M points; the chart needs about a thousand
    step = max(1, len(temperature) // 1000)
    measurements.coolant_temperature.x_axis = time_s[::step].tolist()
    measurements.coolant_temperature.y_axis.temperature = temperature[::step].tolist()
    measurements.coolant_temperature.y_axis.temperature.aggregations.max = float(np.max(temperature))
    measurements.coolant_temperature.y_axis.temperature.aggregations.mean = float(np.mean(temperature))

    attach.file(log_path, "thermal_soak.tdms")
    log.info(f"Logged {len(temperature)} samples, max {np.max(temperature):.1f} °C")
```

The aggregations are computed on the full-rate data, and the chart is decimated. That distinction is what makes the result trustworthy: a maximum computed after decimation can miss a spike, and a chart of a million points is unreadable in any tool. The raw TDMS stays attached to the run for the cases where someone needs the spike itself.

`npTDMS` reads TDMS files without any NI software installed, so the same phase runs on a Linux analysis machine against archived logs, which is also how historical data gets validated against the new limits before cutover.

## Workspace: Operator UI and Monitoring

A VeriStand workspace screen does two jobs. It shows live channels while the rig runs, and it collects operator input: which unit is on the bench, whether the fixture is closed, whether to proceed after a warning.

The second job moves. Unit identification is declared in the `unit:` block and prompted before the test starts, and confirmations become operator UI phases in the procedure, so they are versioned and recorded like everything else. A workspace button that started a profile becomes the start of the run.

The first job does not need to move on day one. A rig that keeps its engine can keep its workspace open for live monitoring, and the operator UI runs beside it. Once the run streams to the dashboard, the live view there covers most of what the workspace was used for, at which point the workspace becomes a debugging tool rather than the operator's screen.

## Results: Do Not Port

The result path in a VeriStand installation is usually a TDMS folder on a share, a DIAdem script that turns it into a PDF, and sometimes a SQL insert written by hand. It is the part teams spent the most on and the part that should not survive.

The engine uploads the run: procedure, unit, phases, measurements with their validators, attachments, outcome. Runs queue offline when the rig loses the network and drain on reconnect with the original timestamp. A limit change is a change to `procedure.yaml`, deployed to every rig at once, rather than a DIAdem script edited in three places.

When a specific report format is a contractual deliverable, generate it from the stored data through the API, once, off the rig. Putting the report generator inside the test means every rig needs DIAdem and a formatting bug fails a passing unit. For years of existing TDMS logs, [Import Historical Test Data via the API](/guides/import-historical-test-data-via-the-api) covers the backfill, and the `npTDMS` phase above is the extraction step of it.

## Removing the Engine

This section applies to the second and third situations: bench work that never needed determinism, and model-in-the-loop on Windows. The plug for the rig is replaced by plugs for the instruments, and the mapping is mostly one library per device.

| VeriStand item | Python equivalent |
|---|---|
| NI-DAQmx device in the system definition | `nidaqmx` |
| NI-XNET CAN or LIN port | `python-can` with the `nixnet` interface, `cantools` for the DBC |
| PXI DMM, PXI switch | `nidmm`, `niswitch` |
| SCPI power supply or load | PyVISA |
| Compiled Simulink model | Re-export as FMU, run with FMPy |
| FMU | FMPy |
| Custom device written in LabVIEW | Rewrite as a plug, or keep the engine |

The last row is the honest limit. A custom device is LabVIEW code running inside the engine, often for a protocol or a signal generator NI does not ship. If it does real-time work, it is the reason to stay in the first situation. If it wraps an instrument, it is a plug.

The other limit is timing. Python on Windows sleeps with millisecond-scale jitter, and a software loop above about 100 Hz is not reliable from a phase. For a bench test that reads a steady state every second this is irrelevant. For a control loop it is the reason the engine existed, and no amount of Python replaces it.

Model-in-the-loop moves cleanly because FMPy runs the same FMU the VeriStand PC license ran, in-process, and the phase drives the co-simulation step by step:

```python filename="plugs/plant.py"
from fmpy import extract, read_model_description
from fmpy.fmi2 import FMU2Slave


class PlantModel:
    def __init__(self, fmu_path: str, step_s: float = 0.001):
        self._desc = read_model_description(fmu_path)
        self._refs = {v.name: v.valueReference for v in self._desc.modelVariables}
        self._fmu = FMU2Slave(
            guid=self._desc.guid,
            unzipDirectory=extract(fmu_path),
            modelIdentifier=self._desc.coSimulation.modelIdentifier,
        )
        self._fmu.instantiate()
        self._fmu.setupExperiment(startTime=0.0)
        self._fmu.enterInitializationMode()
        self._fmu.exitInitializationMode()
        self._t = 0.0
        self._dt = step_s

    def step(self, inputs: dict[str, float]) -> None:
        self._fmu.setReal([self._refs[k] for k in inputs], list(inputs.values()))
        self._fmu.doStep(currentCommunicationPoint=self._t, communicationStepSize=self._dt)
        self._t += self._dt

    def get(self, name: str) -> float:
        return self._fmu.getReal([self._refs[name]])[0]

    def __del__(self):
        self._fmu.terminate()
        self._fmu.freeInstance()
```

An FMU that misbehaves takes the Python process with it, because it is the vendor's C code running in-process. The engine runs each plug in its own process, so a crashing model fails the phase and is respawned rather than taking the station down.

## A Migration Order That Works

1. **Sort the system into the layers table**, one line per artifact. The engine decision falls out of the first column: if nothing needs a deterministic loop, plan for the engine to go.
2. **Write the VeriStand plug and run one existing stimulus profile from a phase**, with the result uploaded. This is a day of work and it proves the rig, the gateway and the upload path before anything is ported.
3. **Port stimulus profiles one at a time.** Deterministic parts become `niveristand` sequence functions, the rest becomes phase code. Run the old profile and the new phase on the same unit and compare the TDMS logs before retiring the profile.
4. **Move pass/fail out of alarms and DIAdem scripts into validators.** Keep the protective alarms in the system definition and mark them as such.
5. **Replace workspace interaction with the `unit:` block and operator UI phases.** Leave the workspace open for monitoring until the dashboard covers it.
6. **Delete the result path.** Agree with whoever owns the quality records what "the results are in the new system" means before this step, because that conversation is the critical path, not the code.
7. **Re-license.** One Operator license per rig that keeps its engine, one Full seat to open the archive, and nothing on rigs that lost the engine.

Step 3 is where the time goes, and the comparison in it is not optional. A host-side phase that replaced a deterministic sequence will produce plausible numbers with slightly different timing, and the TDMS comparison is the only thing that catches it.

## Key Points

- VeriStand is a real-time engine plus everything around it. Python replaces the second; the first stays for genuine HIL and goes for bench work that never needed it.
- The engine is reached through one plug with `station` scope, so the system definition deploys once and every unit after the first starts immediately.
- Sequences with timing requirements under about 100 ms stay deterministic through `niveristand`. Everything else becomes a phase.
- Protective alarms stay in the system definition. Alarms that were pass/fail criteria become validators.
- TDMS stays the raw record. Aggregations are computed at full rate, charts are decimated, and the file is attached to the run.
- Result plumbing, DIAdem reports and SQL inserts are not ported. Re-licensing to Operator seats per rig is the budget outcome.

### Add Derived Measurements to a Test Script
URL: https://www.tofupilot.com/guides/add-derived-measurements-to-a-test-script
Learn how to record computed values like spreads, margins, and ratios as first-class measurements, so analytics can chart and correlate them.

The best early failure indicators are often not measured directly by any instrument: they are computed from values the bench already has. A derived measurement is one of these computed values recorded as a normal measurement, with its own name, unit, and limits.

Once recorded, a derived measurement behaves like any other: it appears in measurement control, gets a control chart and Cpk, can carry validators, exports to CSV, and participates in drift alerts. A value that only exists inside a spreadsheet formula does none of that.

## Prerequisites

- A test procedure that records at least two related measurements in one phase

## Step 1: Compute the value in the phase

Derive the value right where the raw measurements are taken, and record it in the same phase:

```python filename="phases/contact_voltage_drop.py"
drops = {}
for contact in ["A1", "A2", "A3"]:
    mv = dmm.measure_dc_millivolts(contact)
    drops[contact] = mv

measurements.contact_drop_a1 = drops["A1"]
measurements.contact_drop_a2 = drops["A2"]
measurements.contact_drop_a3 = drops["A3"]

# Derived: imbalance between poles, a better early indicator
# than any single pole against its own limit.
measurements.contact_drop_spread = round(max(drops.values()) - min(drops.values()), 2)
```

## Step 2: Give it a limit

A derived measurement earns its keep when it carries its own validator. In the example, each pole has a 100 mV limit, and the spread gets a 40 mV limit. A unit can pass all three pole limits and still fail on spread, which is exactly the class of defect the individual limits cannot see.

Set the limit from fleet data, not intuition: record the measurement without a validator first, look at the healthy distribution in measurement control, and place the limit outside it.

## Step 3: Pick derivations that expose failure modes

Useful patterns:

| Derivation | Computation | What it exposes |
|---|---|---|
| Spread | max - min across channels | One degrading channel among nominally identical ones |
| Margin | limit - measured value | Erosion of headroom before any limit fails |
| Ratio | value A / value B | Shifts that scale both values but change their relationship |
| Delta from reference | value - golden sample value | Fixture or instrument drift |
| Symmetry | left - right, or phase-to-phase | Mechanical or winding asymmetry |

## Step 4: Analyze at fleet level

Open measurement control and select the derived measurement over the full history. A healthy fleet is one population; a defect mode shows up as a second one:

![Fleet control chart of a derived spread measurement: the healthy population sits near 11 mV, a second population sits above the 40 mV limit](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/latent-defects-fleet-spread.jpg-370aab47-3f41-48ce-ad6b-782efdbf5db7-eE3PxVhhRF43WoLy58I9cJqVWWQpLT.jpg)

From there the standard tools apply: control chart, histogram, Cpk, per-serial scoping, and CSV export for anything you want to take further.

### Share Instruments Across Parallel Slots
URL: https://www.tofupilot.com/guides/share-instruments-across-parallel-slots
Shared instruments cause interleaved commands and lost throughput. Learn how plug scope serializes calls and how to size the bottleneck.

Parallel testing assumes each position has its own instruments. Real stations rarely do. One DMM behind a multiplexer serves eight positions, one thermal chamber holds the whole tray, one power supply feeds a rack.

Shared instruments are where parallel test stations lose the throughput they were built for, and where they produce measurements that are wrong in ways that pass. This guide covers how sharing works, how to size the benefit, and the failure modes worth designing against.

It assumes a multi-position fixture; for what a position is and how to declare one, see [What Are Test Sockets and How to Use Them](/guides/what-are-test-sockets-and-how-to-use-them).

## Why Sharing Is a Problem

An instrument is a single piece of hardware with state. Configure a DMM for a 10V DC range, and that is its range until something changes it.

Two slots measuring at once creates two failures that look nothing alike:

**Interleaved commands.** Slot 1 sets the range to 10V, slot 2 sets it to 100mV, then slot 1 triggers a reading. Slot 1 measures its unit on slot 2's range. The instrument reports no error, the measurement is a plausible number, and the unit either fails for no reason or passes when it should not.

**Serialization.** Two slots cannot physically read one DMM simultaneously. The calls queue whether or not you designed for it.

The first failure is the dangerous one, because it produces bad data rather than an error.

## Scope Decides How Many Instances Exist

In the TofuPilot Framework, plug scope controls instance count and lifetime:

| Scope | Instance | Use for |
|---|---|---|
| `slot` (default) | One per slot | Per-position PSU channel, per-position serial port |
| `execution` | One shared by all slots | A DMM behind a mux, a chamber, a rack supply |
| `station` | One per station, held across runs | Slow-to-open VISA or TCP sessions |

```yaml filename="procedure.yaml"
execution:
  slots: 8

plugs:
  - name: dut_serial
    python: plugs.serial_port:DutSerial
    scope: slot

  - name: dmm
    python: plugs.dmm:Keysight34461A
    scope: execution
    config:
      address: "192.168.1.50"

  - name: chamber
    python: plugs.chamber:ThermalChamber
    scope: station
    config:
      address: "192.168.1.60"
```

Declaring a shared instrument as `execution` or `station` is what makes it safe. **A plug instance runs one method call at a time.** When several slots call the same shared plug, their calls queue and execute one at a time, like commands on a SCPI socket. The interleaving failure above cannot happen through a shared plug, because there is one instance and one call in flight.

The corresponding mistake is declaring a shared instrument as `slot` scope. Eight slots then open eight sessions to one physical instrument, with no serialization at all, and you get exactly the interleaving problem — intermittently, under load, in production.

## Keep the Critical Section Inside One Call

Serialization is per method call, not per phase. A shared measurement must therefore be one call that configures, triggers, and reads:

```python filename="plugs/dmm.py"
import pyvisa


class Keysight34461A:
    def __init__(self, address: str):
        self._rm = pyvisa.ResourceManager()
        self._inst = self._rm.open_resource(f"TCPIP::{address}::INSTR")
        self._inst.timeout = 5000

    def measure_dc_volts(self, channel: int, expected_range: float) -> float:
        """Select the channel, configure, and read as one indivisible call."""
        self._inst.write(f"ROUT:CLOS (@{channel})")
        self._inst.write(f"CONF:VOLT:DC {expected_range}")
        return float(self._inst.query("READ?"))

    def __del__(self):
        self._inst.close()
```

Split across three calls, another slot's call can land between the configure and the read:

```python filename="plugs/dmm_wrong.py"
class DmmWrong:
    def select_channel(self, channel: int):
        self._inst.write(f"ROUT:CLOS (@{channel})")

    def configure(self, expected_range: float):
        self._inst.write(f"CONF:VOLT:DC {expected_range}")

    def read(self) -> float:
        # Another slot may have re-routed the mux by now.
        return float(self._inst.query("READ?"))
```

The phase then passes its own channel, and the shared instance keeps every sequence intact. A phase reads its position from `run.slot_id`, the key declared in `execution.slots`:

```python filename="phases/measure_rail.py"
CHANNELS = {"s1": 101, "s2": 102, "s3": 103, "s4": 104}


def measure_rail(measurements, run, dmm):
    measurements.rail_voltage = dmm.measure_dc_volts(
        channel=CHANNELS[run.slot_id],
        expected_range=10.0,
    )
```

Mapping slot keys to channels explicitly, rather than deriving a channel from a slot number, keeps the wiring readable and survives renumbering the fixture.

## Keep Shared Calls Short

Queuing has a direct throughput cost, and it is the reason parallel stations underperform their spec.

A shared plug method that sleeps holds every other slot's calls to that plug for as long as it runs. Settling time is the usual culprit:

```python filename="plugs/dmm_slow.py"
import time


class DmmSlow:
    def measure_after_settle(self, channel: int) -> float:
        self._inst.write(f"ROUT:CLOS (@{channel})")
        time.sleep(2.0)          # blocks all eight slots
        return float(self._inst.query("READ?"))
```

With eight slots, that two-second settle costs sixteen seconds of station time per cycle, and seven slots spend it idle. Put the wait in the phase, where slots wait concurrently:

```python filename="phases/measure_rail.py"
import time

CHANNELS = {"s1": 101, "s2": 102, "s3": 103, "s4": 104}


def measure_rail(measurements, run, psu, dmm):
    channel = CHANNELS[run.slot_id]
    psu.enable(channel)
    time.sleep(2.0)  # each slot waits in parallel
    measurements.rail_voltage = dmm.measure_dc_volts(channel, 10.0)
```

The rule: **shared plug methods should contain instrument I/O and nothing else.** Waiting, computation, and retry loops belong in phases.

## Sizing the Benefit Before Building

Whether sharing is acceptable is arithmetic, worth doing before the fixture is built.

Let `N` be slot count, `S` the shared instrument time per unit, and `P` the per-slot time that runs in parallel. Cycle time is roughly:

```text filename="model.txt"
cycle ≈ max(P, N × S)
```

Eight slots, 25s of parallel work, 1s of shared measurement:

```text filename="good.txt"
max(25, 8 × 1) = 25s     shared instrument is free
```

The same station with a 4-second shared measurement:

```text filename="bad.txt"
max(25, 8 × 4) = 32s     the DMM is now the bottleneck
```

Past `N × S > P`, adding slots buys nothing: the ninth slot waits on the DMM. That is the point to either buy a second instrument, shorten the shared measurement, or stop adding positions.

## Use phase_first for Shared Instruments

With slots configured, execution strategy affects how shared work clusters:

```yaml filename="procedure.yaml"
execution:
  strategy: phase_first
  slots: 8
```

`phase_first`, the default, runs the same phase across all slots before advancing. Every slot's DMM measurement happens in the same window, so the instrument stays in one configuration for the group.

`slot_first` completes one slot before starting the next, which reconfigures the shared instrument on every switch. For a station whose sharing is expensive to reconfigure, `phase_first` is the better default.

## Shared Setup and Teardown

Some shared hardware is not measured per unit at all: a chamber ramps once, a rack supply powers on once. Those are phases scoped to the execution:

```yaml filename="procedure.yaml"
setup:
  - name: Ramp Chamber to 85C
    python: phases.ramp_chamber
    scope: execution

main:
  - name: Measure Leakage
    python: phases.measure_leakage

teardown:
  - name: Discharge This Position
    python: phases.discharge

  - name: Return Chamber to Ambient
    python: phases.cool_chamber
    scope: execution
```

Execution-scoped setup runs once before any slot starts; execution-scoped teardown runs once after every slot is done.

This scoping is a correctness requirement, not a convenience. **A slot-scoped teardown can run while its neighbours are still testing.** `Return Chamber to Ambient` as a slot-scoped phase would start cooling as soon as the first unit finished, and every remaining unit would be measured on a falling temperature ramp. Those units fail intermittently, the failures do not reproduce on the bench, and nothing in the data points at the chamber.

The rule follows directly: if a teardown touches something another slot can still be using, it must be `scope: execution`.

## Failure Behavior

A shared plug that fails to initialize stops every slot, and so does a failing execution-scoped setup phase. That is correct — eight units testing without a chamber at temperature would produce eight meaningless passes — but it makes shared setup the highest-consequence code on the station.

Give those phases retries and error messages that name the instrument:

```yaml filename="procedure.yaml"
setup:
  - name: Ramp Chamber to 85C
    python: phases.ramp_chamber
    scope: execution
    timeout: 15m
    retry:
      limit: 2
      delay: 30s
```

If the instrument connection drops mid-test, handle the reconnect inside the plug. The engine guarantees a long-lived plug process, not a live instrument session; it health-checks a held `station` plug before each reuse and respawns it if the process died, but a dead TCP session inside a living process is yours to notice.

## Common Pitfalls

**Declaring a shared instrument as `slot` scope.** N sessions to one instrument, no serialization, interleaved commands. This is the failure that produces wrong numbers instead of errors.

**Splitting a configure-and-read across several plug methods.** Another slot can land in the gap.

**Sleeping inside a shared plug method.** Every other slot waits. Move the wait to the phase.

**Cleaning up shared hardware in slot teardown.** The first slot to finish disturbs the rest.

**Sizing the fixture without the shared-instrument term.** A 16-slot fixture with a 3-second shared measurement is a 48-second cycle no matter how fast the units are.

## Key Points

- Shared instruments fail two ways: interleaved commands producing wrong data, and queuing destroying throughput.
- `scope: execution` or `scope: station` gives one instance whose calls serialize, which prevents interleaving by construction.
- A shared plug method must be the whole critical section: configure, trigger, and read in one call.
- Keep waits out of shared methods; a sleeping method blocks every slot.
- Cycle time is roughly `max(P, N × S)` — past that crossover, more slots buy nothing.
- Anything a finishing slot could disturb belongs in an execution-scoped teardown.

### Track Retest Counts to Catch Rogue Units
URL: https://www.tofupilot.com/guides/track-retest-counts-to-catch-rogue-units
Learn how to spot rogue units early by tracking retest counts per serial number, with self-calibrating alerts on abnormal shop visits in TofuPilot.

Some units never stay fixed. They come off the aircraft, test fine or get a minor repair, go back into service, and return again a few weeks later. The industry calls them rogue units, and they are expensive out of proportion to their numbers: each one cycles through removal, shipping, induction, a full bench test, and recertification on every loop. This guide shows how to catch them early using nothing but the retest count you already generate.

## What Makes a Unit Rogue

Airlines and suppliers define rogues contractually. A common definition: the same serial number removed three or more times for similar discrepancies, or four no-fault-found shop visits within twelve months. Airbus documents a comparable policy for its suppliers, and its maintenance briefing notes describe a fuel quantity indicator that returned to the shop fifteen times before finally causing a low-fuel event in service.

The failure behind most rogues is intermittent: a cracked solder joint, a chafed wire, connector fretting. It appears under vibration and thermal cycling on the aircraft and disappears on a warm, still bench. A conventional functional test scans one circuit at a time and can miss a fault lasting microseconds, so the unit passes, ships, and fails again.

You cannot fix an intermittent you have not localized. But you can stop losing track of the units that carry one, and that is a data problem, not a hardware problem.

## Identify Every Unit by Serial Number

Rogue tracking collapses if runs are not tied to unit identity. In a TofuPilot procedure, the unit block makes serial and part number a first-class part of every run:

```yaml filename="procedure.yaml"
name: LRU Bench Acceptance Test
version: 1.0.0

unit:
  serial_number:
    default_value: "LRU000042"
  part_number:
    default_value: "622-1234-001"
```

On a station, the operator scans or enters the serial number at the start of each run. Every test result, pass or fail, is then attached to that serial forever.

## Read the Unit History

With identity in place, TofuPilot builds the per-unit history automatically. The unit page shows every run for a serial number: date, procedure, result, and every measured value. A rogue candidate reads like a story:

- Visit 1: removed for reported fault, tests NFF, returned to stock.
- Visit 2, five weeks later: same reported fault, tests NFF again.
- Visit 3: fails marginally on one measurement, minor repair, released.
- Visit 4: back again.

On paper records spread over months, this pattern is invisible unless someone remembers the serial number. In a per-unit history it is one screen.

The measurements add what the visit count alone cannot: a unit that tests NFF but whose signal margin shrinks on every visit is not fine, it is degrading. That trajectory is the difference between "no fault found" and "fault not yet caught".

## Alert on Abnormal Retest Rates

Reviewing histories by hand only works for units someone already suspects. TofuPilot's retest threshold alert watches all of them: it fires when a unit has been tested more often than is normal for that procedure. The threshold self-calibrates, learning each procedure's ordinary retest rate, so a part that routinely needs two runs per visit does not generate noise while a genuine repeat offender does.

The alert resolves manually once the unit's history has been reviewed, which fits how shops actually work: the point is to force a human look at the right serial numbers, early.

## From Detection to Action

Once a serial number is flagged, the shop has options that all work better early:

- **Deep test instead of another standard cycle.** Route the unit to environmental screening (temperature and vibration during test) or intermittence-focused equipment rather than repeating the same bench test that already passed it.
- **Quarantine.** Hold the unit and monitor the aircraft for a few flights before deciding; if the fault follows the aircraft, the unit was never the problem.
- **Supplier claim.** Rogue clauses in supply contracts trigger on exactly the pattern the history documents: removals, dates, findings per serial number. A queryable history is the evidence.

## What Good Looks Like

A shop with rogue tracking under control knows its repeat offenders by name. Retest alerts surface a handful of serials a month, each of which gets a history review before its next standard test cycle. Units on their third visit go to deep test, not back to stock. And when a supplier conversation happens, the shop brings the serial's full documented trajectory instead of an anecdote.

### What Is IQC, IPQC, FQC, and OQC
URL: https://www.tofupilot.com/guides/what-is-iqc-ipqc-fqc-and-oqc-with-tofupilot
IQC, IPQC, FQC and OQC are the four quality gates in manufacturing. What each catches, how AQL sampling works, and how to track yield per gate.

Manufacturing quality control happens at four gates: incoming (IQC), in-process (IPQC), final (FQC), and outgoing (OQC). Each gate catches a different class of defect, and a defect that slips past a gate costs roughly ten times more to catch at the next one. This guide explains what each gate checks, who owns it, how sampling works, and how to track yield per gate with TofuPilot.

## The Four Quality Gates

| Gate | Full Name | When | Typical Owner | What It Catches |
|------|-----------|------|---------------|-----------------|
| IQC | Incoming Quality Control | Components arrive from suppliers | Supplier quality engineer | Wrong parts, out-of-spec components, counterfeit or damaged lots |
| IPQC | In-Process Quality Control | During assembly | Process engineer | Process drift, solder defects, assembly errors, contamination |
| FQC | Final Quality Control | Unit fully assembled | Test engineer | Functional failures, safety failures, cosmetic defects |
| OQC | Outgoing Quality Control | Before shipment | QA / logistics | Packaging damage, wrong labels, wrong quantity, late-stage functional drift |

Each gate is a decision point. Material that fails is quarantined, reworked, or rejected. Material that passes moves on. The four gates together form the quality plan for a product line, and every one of them produces test data you can track.

## Why the Order Matters: The Cost of Escape

The classic rule of thumb in electronics manufacturing is 1-10-100: a defect that costs 1 unit to catch at incoming inspection costs about 10 to catch during assembly and about 100 once the product is finished. A defect that reaches the customer costs far more again in returns, field service, and reputation.

Concretely, a bad capacitor lot caught at IQC means returning a reel to the supplier. The same lot caught at FQC means desoldering parts from hundreds of populated boards. Caught in the field, it means a recall.

That asymmetry is why IQC and IPQC exist at all. Nobody wants to pay for four inspections; everyone wants to avoid paying for the escape.

## IQC: Incoming Quality Control

IQC inspects raw materials and components when they arrive. It keeps bad inputs out of the line before any value has been added to them.

| Check | Method |
|-------|--------|
| Visual inspection | Packaging damage, correct labels, moisture indicator cards on MSD reels |
| Dimensional check | Caliper or CMM measurement against the drawing |
| Electrical sample test | Measure key parameters (resistance, capacitance, forward voltage) on a sample |
| Certificate review | Certificate of Conformance, RoHS declaration, date and lot codes |

IQC does not test every part. It follows an acceptance sampling plan, usually from ANSI/ASQ Z1.4 (the successor to MIL-STD-105E), keyed to an Acceptable Quality Limit (AQL).

### AQL sampling, worked example

Say a reel of 1,000 resistors arrives and your plan is General Inspection Level II with an AQL of 1.0 %.

| Parameter | Value |
|-----------|-------|
| Lot size | 1,000 (falls in the 501 to 1,200 band) |
| Sample size code letter | J |
| Sample size | 80 parts |
| Accept number (Ac) | 2 defects or fewer: accept the lot |
| Reject number (Re) | 3 defects or more: reject the lot |

You measure 80 resistors. Two are out of tolerance, so the lot is accepted. Three, and the whole reel goes back. A tighter AQL (0.65 %) or a switch to Tightened inspection after repeated rejections shrinks the accept number.

Recording each sampled measurement, not just the lot verdict, is what turns IQC into a supplier scorecard. A supplier whose parts pass but trend toward the limit is a supplier about to fail.

## IPQC: In-Process Quality Control

IPQC monitors the process while it runs. It catches drift before it turns into a batch of defective units.

| Check | Method |
|-------|--------|
| SPI after paste printing | Automated solder paste volume and offset measurement |
| AOI after reflow | Automated optical inspection of solder joints and component placement |
| First piece inspection | Full check of the first unit after any setup or changeover |
| Hourly spot checks | Operator verifies key dimensions or torque values |
| Process parameter monitoring | Oven profile, pressure, dispense weight within control limits |

IPQC is where [statistical process control (SPC)](/guides/what-is-spc-statistical-process-control) lives. A control chart on paste volume or reflow peak temperature shows drift hours before AOI starts flagging defects. The automated inspection steps ([SPI, AOI, and AXI](/guides/what-is-aoi-spi-and-axi-with-tofupilot)) each produce a pass/fail per board plus measurements you can trend.

On a new line or after a design change, the first-piece check is formalised as a [First Article Inspection (FAI)](/guides/what-is-fai-with-tofupilot).

## FQC: Final Quality Control

FQC tests the fully assembled product against its specification. This is where [functional testing](/guides/what-is-functional-testing-for-hardware-with-tofupilot), safety testing, and cosmetic inspection happen.

| Check | Method |
|-------|--------|
| Functional test | [Automated test equipment (ATE)](/guides/what-is-ate-with-tofupilot) exercises every feature against limits |
| Safety test | Hipot (dielectric withstand), ground continuity, leakage current |
| Cosmetic inspection | Scratches, dents, gaps, label alignment |
| Calibration verification | Calibrated parameters confirmed within spec and stored per serial |

FQC is typically 100 % inspection for functional and safety tests, and sample-based for cosmetics. Because every unit gets a functional test here, FQC is where most of your test data volume comes from and where [first pass yield](/guides/what-is-first-pass-yield-and-how-to-track-it-with-tofupilot) is usually measured.

## OQC: Outgoing Quality Control

OQC is the last check before the product ships. It verifies that what goes in the box matches the order, and pulls a sample to confirm nothing drifted between FQC and packing.

| Check | Method |
|-------|--------|
| Packaging integrity | Box condition, cushioning, moisture barrier bag and desiccant |
| Label verification | Part number, serial, lot, regulatory marks match the order |
| Quantity count | Matches packing list and purchase order |
| Sample retest | Pull units per AQL and rerun the functional test |

An OQC failure is expensive because the product is finished and packed. It is also a signal that FQC missed something, so an OQC failure should trigger a review of the FQC coverage, not just a rework of the lot.

OQC is a point check. For confidence that units keep working after shipment, manufacturers add [ongoing reliability testing (ORT)](/guides/what-is-ort-with-tofupilot), which pulls units from the line and stress-tests them on a schedule.

## Prerequisites

- Python 3.10+
- The TofuPilot CLI, or OpenHTF (`pip install "tofupilot[openhtf]"`)

## Step 1: Write One Test Procedure per Gate

Keep each gate as its own procedure. TofuPilot tracks yield per procedure, so one procedure per gate gives you IQC, IPQC, FQC, and OQC yield out of the box instead of one blended number.

With the TofuPilot Framework, each gate is a procedure directory with its own YAML. Here is the IQC gate, sampling one resistor from an incoming reel:

```yaml filename="iqc/procedure.yaml"
name: IQC Resistor Sample
version: 1.0.0

unit:
  serial_number:
    default_value: "REEL000001"
  part_number:
    default_value: "RES-4K7"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: IQC Sample
    python: phases.iqc_sample
    measurements:
      - name: Resistance
        unit: "Ω"
        validators:
          - operator: ">="
            expected_value: 4700
          - operator: "<="
            expected_value: 5100
      - name: Marking Matches PO
        validators:
          - operator: "=="
            expected_value: true
```

```python filename="iqc/phases/iqc_sample.py"
def iqc_sample(measurements, dmm):
    """IQC: one sampled resistor from the incoming reel."""
    measurements.resistance = dmm.measure_resistance()
    measurements.marking_matches_po = True
```

```python filename="iqc/plugs/dmm.py"
class Multimeter:
    def measure_resistance(self) -> float:
        return 4850.0  # Replace with a real instrument read
```

The IPQC gate records what the SPI machine and the oven profiler reported on the first piece after a changeover:

```yaml filename="ipqc/procedure.yaml"
name: IPQC First Piece
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

main:
  - name: IPQC First Piece
    python: phases.first_piece
    measurements:
      - name: Paste Volume
        unit: "%"
        validators:
          - operator: ">="
            expected_value: 80
          - operator: "<="
            expected_value: 120
      - name: Reflow Peak
        unit: "°C"
        validators:
          - operator: ">="
            expected_value: 235
          - operator: "<="
            expected_value: 250
```

```python filename="ipqc/phases/first_piece.py"
def first_piece(measurements):
    """IPQC: first piece after changeover, values from SPI and oven profiler."""
    measurements.paste_volume = 104
    measurements.reflow_peak = 243
```

The FQC gate is the functional and safety test on the finished unit:

```yaml filename="fqc/procedure.yaml"
name: FQC Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

main:
  - name: FQC
    python: phases.fqc
    measurements:
      - name: Output Voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.9
          - operator: "<="
            expected_value: 5.1
      - name: Hipot Leakage
        unit: mA
        validators:
          - operator: "<="
            expected_value: 0.5
```

```python filename="fqc/phases/fqc.py"
def fqc(measurements):
    """FQC: functional and safety test on the finished unit."""
    measurements.output_voltage = 5.01
    measurements.hipot_leakage = 0.12
```

The OQC gate reuses the FQC voltage limits on purpose, and adds a label check:

```yaml filename="oqc/procedure.yaml"
name: OQC Sample Retest
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

main:
  - name: OQC Sample
    python: phases.oqc_sample
    measurements:
      - name: Output Voltage
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.9
          - operator: "<="
            expected_value: 5.1
      - name: Label Matches Order
        validators:
          - operator: "=="
            expected_value: true
```

```python filename="oqc/phases/oqc_sample.py"
def oqc_sample(measurements):
    """OQC: sampled unit pulled from the packed lot."""
    measurements.output_voltage = 5.00
    measurements.label_matches_order = True
```

Sharing the voltage limits between FQC and OQC is deliberate. If the same measurement drifts between the two gates on the same serial, you have a handling or storage problem, and TofuPilot will show both runs side by side under that serial.

### The same gate in OpenHTF

If your line already runs OpenHTF, the same FQC gate looks like this and needs no rewrite:

```python filename="fqc_functional.py"
import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import upload


@htf.measures(
    htf.Measurement("output_voltage_V")
    .in_range(minimum=4.9, maximum=5.1)
    .with_units(units.VOLT),
    htf.Measurement("hipot_leakage_mA")
    .in_range(maximum=0.5)
    .with_units(units.MILLIAMPERE),
)
def phase_fqc(test):
    """FQC: functional and safety test on the finished unit."""
    test.measurements.output_voltage_V = 5.01
    test.measurements.hipot_leakage_mA = 0.12


test = htf.Test(phase_fqc)
test.add_output_callbacks(upload())
test.execute(test_start=lambda: input("Scan unit serial: "))
```

## Step 2: Upload Every Gate to TofuPilot

Each procedure uploads independently. TofuPilot links runs by serial number, so an IQC lot, the IPQC first piece built from it, and the FQC and OQC runs on the finished unit all land on one timeline.

```bash filename="run_gate.sh"
tofupilot run ./fqc --upload
```

For IQC, scan the reel or lot ID as the serial. For IPQC, scan the first-piece board. For OQC, scan the sampled unit.

![TofuPilot run analytics showing runs, units, first pass yield, and test duration, with daily runs split by pass and fail](/landing/tofupilot-run-analytics-throughput-yield.png)

## Step 3: Read Yield and Failures per Gate

With one procedure per gate, the Analytics view answers the questions each gate owner actually asks:

- **Yield per gate.** IQC, IPQC, FQC, and OQC each get their own first pass yield. A drop at one gate points at one team.
- **Top failures per gate.** A [Pareto of failing measurements](/guides/how-to-use-pareto-analysis-for-test-failures-with-tofupilot) tells you whether FQC is failing on voltage or on hipot, and whether that changed this week.
- **Supplier quality.** Filter IQC runs by supplier lot to see rejection rate and measurement drift per vendor.
- **Full traceability.** Open any shipped serial and walk back through OQC, FQC, IPQC, and the IQC lot its components came from. See [how to build full unit traceability](/guides/how-to-build-full-unit-traceability-with-tofupilot) for the sub-assembly setup.

![TofuPilot unit traceability view showing a parent serial number with its sub-assembly components and their serials](/landing/tofupilot-unit-traceability-history.png)

## Common Mistakes

| Mistake | What Happens | Fix |
|---------|--------------|-----|
| One blended yield number | A supplier problem looks like a test station problem | One procedure per gate |
| Recording only pass/fail at IQC | No early warning when a supplier trends toward the limit | Store the sampled measurements, not just the verdict |
| Skipping OQC because FQC is 100 % | Packing and labeling errors ship | Keep a light AQL sample and a label check |
| Different limits at FQC and OQC | Units pass one and fail the other with no root cause | Share limit definitions between the two procedures |
| No serial at IQC | Cannot trace a field failure back to a component lot | Use the reel or lot ID as the IQC serial |

### HIL Testing with Python
URL: https://www.tofupilot.com/guides/hardware-in-the-loop-testing-with-python-a-practical-guide
Set up HIL testing for embedded systems with Python, using the TofuPilot Framework or OpenHTF, covering GPIO, ADC, PWM and CI integration.

Hardware-in-the-loop (HIL) testing validates embedded firmware by stimulating real hardware inputs and measuring real outputs. Unlike pure simulation, HIL tests run on the actual target hardware with real peripherals. This catches bugs that simulation misses: timing issues, ADC noise, interrupt conflicts, and peripheral initialization problems. This guide shows you how to build a Python-based HIL test system, with the TofuPilot Framework and with OpenHTF.

## HIL vs SIL vs MIL

| Method | Hardware | Software | Fidelity | Speed | Cost |
|--------|----------|----------|----------|-------|------|
| **MIL** (Model-in-the-Loop) | None | Simulated plant + controller | Low | Fast (seconds) | Low |
| **SIL** (Software-in-the-Loop) | None | Real controller code on host | Medium | Fast (seconds) | Low |
| **HIL** (Hardware-in-the-Loop) | Real target board | Real firmware | High | Real-time | Medium |
| **Physical test** | Real target + real plant | Real firmware | Highest | Real-time | High |

HIL sits between SIL and physical testing. You test real firmware on real hardware, but simulate the environment (sensors, actuators, external systems).

## When to Use HIL Testing

- **Firmware regression testing.** Every firmware build gets the same physical tests. Catches regressions that unit tests miss.
- **Peripheral validation.** GPIO, ADC, DAC, PWM, UART, SPI, I2C. Real silicon behaves differently than simulation models.
- **Safety-critical systems.** Automotive, medical, aerospace require HIL testing for certification (ISO 26262, IEC 62304).
- **Pre-production validation.** Validate firmware on real hardware before committing to production.

## HIL Test Architecture

A typical Python HIL setup:

| Component | Role | Example |
|-----------|------|---------|
| Target board (DUT) | Runs firmware under test | STM32 dev board, ESP32, Raspberry Pi Pico |
| Test host | Runs Python test scripts | PC or Raspberry Pi |
| DAQ / GPIO adapter | Stimulate inputs, read outputs | Arduino, LabJack, NI DAQ |
| Power supply | Controlled power to DUT | Bench PSU (programmable) |
| Serial/debug | Flash firmware, send commands | USB-to-serial, J-Link, ST-Link |
| Instruments | Precision measurements | DMM, oscilloscope |

## Option A: The TofuPilot Framework

A procedure YAML declares the sequence and the limits. Each piece of test hardware is a plug, a plain Python class. Phases are plain functions that receive the plugs by parameter name.

### Step 1: Write the Plugs

```python filename="plugs/gpio.py"
class Gpio:
    """GPIO interface for stimulating DUT inputs and reading outputs.

    Replace with a real GPIO adapter (LabJack, Arduino, RPi.GPIO).
    """

    def __init__(self):
        self._pin_states = {}

    def set_pin(self, pin: int, state: bool):
        self._pin_states[pin] = state

    def read_pin(self, pin: int) -> bool:
        return self._pin_states.get(pin, False)
```

```python filename="plugs/adc.py"
class Adc:
    """ADC interface for reading analog outputs from the DUT.

    Replace with a real DAQ (NI DAQmx, LabJack, ADS1115).
    """

    def read_channel(self, channel: int) -> float:
        return {0: 1.65, 1: 2.50, 2: 0.33}.get(channel, 0.0)
```

```python filename="plugs/pwm.py"
class Pwm:
    """PWM measurement interface.

    Replace with a real frequency counter or oscilloscope.
    """

    def measure_frequency(self) -> float:
        return 1000.0

    def measure_duty_cycle(self) -> float:
        return 50.0
```

### Step 2: Write the Phases

```python filename="phases/gpio_loopback.py"
def gpio_loopback(measurements, gpio):
    """Verify firmware routes GPIO input to output correctly."""
    gpio.set_pin(1, True)                      # Stimulate DUT input
    measurements.gpio_loopback = gpio.read_pin(1)  # Read DUT output
```

```python filename="phases/adc_readings.py"
def adc_readings(measurements, adc):
    """Verify ADC channels read expected voltages."""
    measurements.adc_channel_0 = adc.read_channel(0)
    measurements.adc_channel_1 = adc.read_channel(1)
```

```python filename="phases/pwm_output.py"
def pwm_output(measurements, pwm):
    """Verify PWM output frequency and duty cycle."""
    measurements.pwm_frequency = pwm.measure_frequency()
    measurements.pwm_duty_cycle = pwm.measure_duty_cycle()
```

### Step 3: Declare the Procedure

```yaml filename="procedure.yaml"
name: ECU HIL Test
version: 1.0.0

unit:
  serial_number:
    default_value: "ECU000001"
  part_number:
    default_value: "ECU-100"

plugs:
  - name: gpio
    python: plugs.gpio:Gpio
  - name: adc
    python: plugs.adc:Adc
  - name: pwm
    python: plugs.pwm:Pwm

main:
  - name: GPIO Loopback
    python: phases.gpio_loopback
    measurements:
      - name: GPIO Loopback
        validators:
          - operator: "=="
            expected_value: true
  - name: ADC Readings
    python: phases.adc_readings
    measurements:
      - name: ADC Channel 0
        unit: V
        validators:
          - operator: ">="
            expected_value: 1.5
          - operator: "<="
            expected_value: 1.8
      - name: ADC Channel 1
        unit: V
        validators:
          - operator: ">="
            expected_value: 2.3
          - operator: "<="
            expected_value: 2.7
  - name: PWM Output
    python: phases.pwm_output
    measurements:
      - name: PWM Frequency
        unit: Hz
        validators:
          - operator: ">="
            expected_value: 950
          - operator: "<="
            expected_value: 1050
      - name: PWM Duty Cycle
        unit: "%"
        validators:
          - operator: ">="
            expected_value: 48
          - operator: "<="
            expected_value: 52
```

Run it with `tofupilot run ./procedure.yaml`. The three phases are independent, so the engine runs them in parallel, which matters when a HIL suite grows to dozens of peripheral checks.

For firmware regression tracking, record the firmware version as run metadata so the dashboard can filter runs by build:

```python filename="phases/record_build.py"
def record_build(run):
    run.metadata["firmware_version"] = "2.1.0"
```

## Option B: OpenHTF

OpenHTF attaches measurements and plugs to phase functions with decorators, then runs them in sequence.

### Step 1: Create Hardware Interface Plugs

```python filename="hil/plugs.py"
import openhtf as htf
from openhtf.plugs import BasePlug


class GpioPlug(BasePlug):
    """GPIO interface for stimulating DUT inputs and reading outputs."""

    def setUp(self):
        self._pin_states = {}

    def set_pin(self, pin: int, state: bool):
        self._pin_states[pin] = state

    def read_pin(self, pin: int) -> bool:
        return self._pin_states.get(pin, False)

    def tearDown(self):
        self._pin_states.clear()


class AdcPlug(BasePlug):
    """ADC interface for reading analog outputs from DUT."""

    def setUp(self):
        self._channels = {0: 1.65, 1: 2.50, 2: 0.33}

    def read_channel(self, channel: int) -> float:
        return self._channels.get(channel, 0.0)

    def tearDown(self):
        pass


class PwmPlug(BasePlug):
    """PWM measurement interface."""

    def setUp(self):
        pass

    def measure_frequency(self) -> float:
        return 1000.0

    def measure_duty_cycle(self) -> float:
        return 50.0

    def tearDown(self):
        pass
```

### Step 2: Write the Phases

```python filename="hil/test_phases.py"
import openhtf as htf
from openhtf.util import units


@htf.measures(
    htf.Measurement("gpio_loopback")
    .equals(True)
    .doc("GPIO pin loopback: set input, verify output follows"),
)
@htf.plug(gpio=GpioPlug)
def test_gpio_loopback(test, gpio):
    """Verify firmware routes GPIO input to output correctly."""
    gpio.set_pin(1, True)
    test.measurements.gpio_loopback = gpio.read_pin(1)


@htf.measures(
    htf.Measurement("adc_channel_0")
    .in_range(1.5, 1.8)
    .with_units(units.VOLT),
    htf.Measurement("adc_channel_1")
    .in_range(2.3, 2.7)
    .with_units(units.VOLT),
)
@htf.plug(adc=AdcPlug)
def test_adc_readings(test, adc):
    """Verify ADC channels read expected voltages."""
    test.measurements.adc_channel_0 = adc.read_channel(0)
    test.measurements.adc_channel_1 = adc.read_channel(1)


@htf.measures(
    htf.Measurement("pwm_frequency")
    .in_range(950, 1050)
    .with_units(units.HERTZ),
    htf.Measurement("pwm_duty_cycle").in_range(48, 52),
)
@htf.plug(pwm=PwmPlug)
def test_pwm_output(test, pwm):
    """Verify PWM output frequency and duty cycle."""
    test.measurements.pwm_frequency = pwm.measure_frequency()
    test.measurements.pwm_duty_cycle = pwm.measure_duty_cycle()
```

### Step 3: Assemble the Test

```python filename="hil/main.py"
import openhtf as htf
from tofupilot.openhtf import upload


def main():
    test = htf.Test(
        test_gpio_loopback,
        test_adc_readings,
        test_pwm_output,
        procedure_id="HIL-001",
        part_number="ECU-100",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Firmware version: "))


if __name__ == "__main__":
    main()
```

Install the extra with `pip install "tofupilot[openhtf]"`, or skip the callback and run the suite with `tofupilot run`. Use the serial number prompt to enter the firmware version so TofuPilot tracks which builds pass or fail each test.

## Which to Use for HIL

| Situation | Choice |
|-----------|--------|
| New HIL bench, many independent peripheral checks | TofuPilot Framework (parallel phases cut suite time) |
| Several target boards on one bench | TofuPilot Framework (multi-slot) |
| Validating waveform or sweep shape, not just endpoints | TofuPilot Framework (per-axis validators) |
| Existing OpenHTF suite already running in CI | OpenHTF, it runs natively on the CLI |
| One pure-Python paradigm, no YAML layer | OpenHTF |

Both run natively on the TofuPilot CLI and upload to the same dashboard, so the choice does not lock your data in.

## HIL in CI/CD

The real power of HIL testing comes from running it automatically on every firmware build.

| CI/CD Step | Action |
|-----------|--------|
| 1. Code push | Developer pushes firmware change |
| 2. Build | CI compiles firmware binary |
| 3. Flash | CI flashes binary to target board (via J-Link/ST-Link) |
| 4. HIL test | CI runs the procedure on the bench |
| 5. Results | TofuPilot logs pass/fail per measurement |
| 6. Gate | CI fails the build if any measurement is out of spec |

This requires a dedicated test station connected to your CI runner. The test station has the target board, GPIO adapter, DAQ, and instruments. The CI runner (Jenkins, GitHub Actions self-hosted runner, etc.) triggers the test after each build.

For unattended runs, resolve the identify prompt from the procedure and pre-bake any operator inputs so nothing blocks:

```bash filename="ci/run_hil.sh"
tofupilot run ./procedure.yaml --json --ui-values ui.json --ui-timeout 300
```

## Common HIL Test Categories

| Category | What to Test | Measurements |
|----------|-------------|-------------|
| **GPIO** | Input/output routing, interrupt response | Pin state, response time |
| **ADC** | Sensor reading accuracy, noise | Voltage values, noise floor |
| **DAC** | Output voltage accuracy | Voltage values |
| **PWM** | Frequency, duty cycle, resolution | Frequency, duty cycle |
| **Communication** | UART, SPI, I2C, CAN data integrity | Bit error rate, response time |
| **Timing** | Interrupt latency, task scheduling | Response time in microseconds |
| **Power** | Sleep mode current, wake-up time | Current draw, wake-up latency |
| **Watchdog** | Recovery from hang | Reset detection, recovery time |

## HIL vs Production FCT

HIL and FCT test different things at different stages:

| | HIL | FCT |
|--|-----|-----|
| **When** | Development, every firmware build | Production, every manufactured board |
| **What** | Firmware behavior on real hardware | Board assembly and system function |
| **DUT** | Dev board or prototype | Production board |
| **Changes between tests** | Firmware (code changes) | Hardware (manufacturing variation) |
| **Failure means** | Firmware bug (code fix) | Assembly defect (rework or scrap) |
| **Run frequency** | Every CI build | Every unit manufactured |
| **TofuPilot use** | Track regressions across firmware versions | Track FPY, Cpk, control charts |

Many teams use the same framework for both HIL and FCT. The plugs change (dev board GPIO vs. production fixture), but the procedure structure is the same.

### Migrate from LabVIEW to Python
URL: https://www.tofupilot.com/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot
Replace LabVIEW with Python for manufacturing test: concept mappings, code examples, and a migration path to the TofuPilot Framework or OpenHTF.

LabVIEW is powerful, but it comes with $3,000-5,000/seat licensing, Windows-only deployment, and binary files that don't version-control well. Python gives you the same instrument control capabilities with zero license cost, cross-platform support, and native Git integration. This guide maps LabVIEW concepts to Python equivalents and shows you how to rebuild your test system with the TofuPilot Framework or OpenHTF.

## Why Teams Migrate

| Pain Point | LabVIEW | Python |
|-----------|---------|--------|
| License cost | $3,160-4,840/seat/year (depending on edition) | Free |
| Deployment | Windows only | Linux, macOS, Windows |
| Version control | Binary .vi files, merge conflicts | Text files, Git-native |
| Hiring | LabVIEW developers are scarce and expensive | Python developers are everywhere |
| CI/CD | Difficult to integrate | Native (pytest, GitHub Actions, etc.) |
| Package ecosystem | NI packages + limited community | pip, PyPI, 400K+ packages |
| Code review | Requires LabVIEW to view | Any text editor |
| Collaboration | One person per VI at a time | Standard Git workflow |

## LabVIEW to Python Concept Map

| LabVIEW Concept | Python Equivalent | Notes |
|----------------|-------------------|-------|
| VI (Virtual Instrument) | Python function | Same idea: reusable, callable unit |
| SubVI | Function or class method | Import from a module |
| Front panel | Operator UI declared in the procedure | Or the TofuPilot dashboard for monitoring |
| Block diagram | Python code | Text instead of wires |
| Connector pane | Function signature | `def measure_voltage(channel: int) -> float` |
| Error cluster | Exception handling | `try/except` instead of error wires |
| Typedef | Class or dataclass | `@dataclass` for structured data |
| Global variable | Module-level variable or class attribute | Avoid when possible |
| Property node | Property decorator | `@property` on a class |
| State machine | The procedure sequence | Declared, not hand-rolled |
| TDMS file | TofuPilot | Replaces file-based logging |
| DAQmx driver | PyVISA + pyvisa-py | Or the nidaqmx Python package |
| Instrument driver | Plug | A plain Python class |
| Test sequence | Procedure `.yaml` | Or `htf.Test(...)` in OpenHTF |
| Parallel sequence execution | `execution.slots` | Multi-slot fixtures |

## Step 1: Set Up Your Python Environment

```bash filename="migration/setup.sh"
python -m venv venv
source venv/bin/activate    # Linux/macOS
# venv\Scripts\activate     # Windows

pip install pyvisa pyvisa-py
```

Then install the TofuPilot CLI, which runs procedures and streams results:

```bash filename="migration/install_cli.sh"
curl -fsSL https://www.tofupilot.app/install | sh
```

`tofupilot run ./procedure.yaml` executes a procedure on your machine without an account, so you can validate a migrated test before connecting anything.

## Step 2: Convert SubVIs to Python Functions

A LabVIEW SubVI that reads voltage from a DMM becomes a Python function:

```python filename="migration/instrument_functions.py"
def measure_voltage(channel: int) -> float:
    """Equivalent of a LabVIEW SubVI that reads voltage from a DMM.

    In LabVIEW: SubVI with channel input (I32) and voltage output (DBL).
    In Python: function with type hints.
    """
    readings = {1: 3.31, 2: 5.02, 3: 1.81}  # Replace with real reads
    return readings.get(channel, 0.0)


# Call it like you'd wire a SubVI
rail_3v3 = measure_voltage(1)
rail_5v0 = measure_voltage(2)
```

The function signature replaces the connector pane. Type hints (`int`, `float`) replace the LabVIEW data types.

## Step 3: Replace the Error Cluster

LabVIEW uses an error cluster (status, code, source) wired through every VI. Python uses exceptions.

```python filename="migration/error_handling.py"
class InstrumentError(Exception):
    """Replaces LabVIEW error cluster for instrument errors."""

    def __init__(self, code: int, source: str, message: str):
        self.code = code
        self.source = source
        super().__init__(f"[{code}] {source}: {message}")


# LabVIEW: check error cluster at every node
# Python: try/except catches errors from anywhere in the block
try:
    voltage = measure_voltage(1)
    if voltage < 0:
        raise InstrumentError(1001, "DMM", "Negative voltage reading")
except InstrumentError as e:
    print(f"Instrument error: {e}")
    # Handle or re-raise
```

This is cleaner than wiring error clusters through every node. Exceptions propagate automatically until caught.

## Step 4: Stop Building the Sequencer

LabVIEW state machines use a while loop plus a case structure, and most teams hand-roll a sequencer on top. You do not need to port that. Both Python hardware frameworks provide sequencing, limits and pass/fail evaluation.

### The TofuPilot Framework

The procedure YAML replaces the state machine: it declares the order, the measurements and the limits. Phases are plain Python functions, and instrument drivers become plugs.

```python filename="plugs/dmm.py"
class Multimeter:
    """Replaces the LabVIEW instrument driver VIs.

    __init__ = opening a VISA session in LabVIEW
    methods  = the SubVIs for each measurement
    close    = closing the session
    """

    def __init__(self):
        self._voltages = iter([3.31, 5.02])
        # Replace with:
        # import pyvisa
        # rm = pyvisa.ResourceManager("@py")
        # self.instr = rm.open_resource("TCPIP::192.168.1.100::INSTR")

    def read_voltage(self) -> float:
        return next(self._voltages)
        # Replace with: return float(self.instr.query(":MEAS:VOLT:DC?"))

    def close(self):
        pass
        # Replace with: self.instr.close()
```

```python filename="phases/power_rails.py"
def power_rails(measurements, dmm):
    """Replaces a LabVIEW sequence of Numeric Limit Tests."""
    measurements.rail_3v3 = dmm.read_voltage()
    measurements.rail_5v0 = dmm.read_voltage()
```

```yaml filename="procedure.yaml"
name: PCBA Functional Test
version: 1.0.0

unit:
  serial_number:
    default_value: "PCBA000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: Rail 3V3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: Rail 5V0
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
```

Each `validators` block is the direct equivalent of a LabVIEW Numeric Limit Test, stored as data rather than buried in wires. The plug is injected into the phase by parameter name.

Two things migrating teams usually want, which the framework provides without extra work:

- **The front panel.** Operator inputs, checklists, images, sliders and progress bars are declared in the procedure, so you do not rebuild a UI.
- **Parallel sequence execution.** If your LabVIEW system tested several UUTs at once, declare fixture slots and the engine runs them concurrently:

```yaml filename="procedure.yaml"
execution:
  slots:
    - name: Slot 1
    - name: Slot 2
    - name: Slot 3
    - name: Slot 4
```

Each slot uploads its own run, with its own unit and outcome.

### OpenHTF

If you prefer everything in Python with no separate YAML layer, OpenHTF attaches measurements and plugs to phase functions with decorators.

```python filename="migration/openhtf_test.py"
import openhtf as htf
from openhtf.plugs import BasePlug
from openhtf.util import units
from tofupilot.openhtf import upload


class InstrumentPlug(BasePlug):
    """Replaces LabVIEW instrument driver VIs.

    setUp = equivalent of opening a VISA session in LabVIEW
    methods = equivalent of SubVIs for each measurement
    tearDown = equivalent of closing the session
    """

    def setUp(self):
        self._voltages = iter([3.31, 5.02])
        # Replace with: rm = pyvisa.ResourceManager("@py")
        # self.instr = rm.open_resource("TCPIP::192.168.1.100::INSTR")

    def read_voltage(self) -> float:
        return next(self._voltages)
        # Replace with: return float(self.instr.query(":MEAS:VOLT:DC?"))

    def tearDown(self):
        pass
        # Replace with: self.instr.close()


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT)
    .doc("3.3V rail"),
    htf.Measurement("rail_5v0")
    .in_range(4.8, 5.2)
    .with_units(units.VOLT)
    .doc("5.0V rail"),
)
@htf.plug(instr=InstrumentPlug)
def test_power_rails(test, instr):
    """Replaces a LabVIEW test sequence with Numeric Limit Tests."""
    test.measurements.rail_3v3 = instr.read_voltage()
    test.measurements.rail_5v0 = instr.read_voltage()


def main():
    test = htf.Test(
        test_power_rails,
        procedure_id="FCT-001",
        part_number="PCBA-100",
    )
    test.add_output_callbacks(upload())
    test.execute(test_start=lambda: input("Scan serial number: "))


if __name__ == "__main__":
    main()
```

Install with `pip install "tofupilot[openhtf]"`, or skip the callback and run the suite with `tofupilot run`.

### Which one

| Situation | Choice |
|-----------|--------|
| You tested several UUTs at once in LabVIEW | TofuPilot Framework (multi-slot) |
| Operators used the LabVIEW front panel heavily | TofuPilot Framework (declarative UI) |
| You want the test plan readable by non-programmers | TofuPilot Framework (YAML procedure) |
| You want one pure-Python paradigm, no YAML | OpenHTF |
| You already have OpenHTF elsewhere in the plant | OpenHTF, for consistency |

Both run natively on the TofuPilot CLI and upload to the same dashboard.

## Step 5: Replace TDMS with TofuPilot

LabVIEW writes test data to TDMS files. You then need to build your own analytics tools to read them. TofuPilot replaces this entire pipeline.

| LabVIEW (TDMS) | TofuPilot |
|----------------|-----------|
| Write TDMS file after each test | Automatic upload |
| Build custom analytics tools | Dashboard with FPY, Cpk, control charts |
| Manual data aggregation across stations | Automatic multi-station aggregation |
| File server for TDMS storage | Cloud storage with API access |
| Custom report generation | Built-in reports and exports |

```bash filename="migration/run_and_upload.sh"
tofupilot run ./procedure.yaml --upload
```

## Migration Checklist

| Step | Action | LabVIEW Equivalent |
|------|--------|-------------------|
| 1 | Install Python + venv, then the TofuPilot CLI | Install LabVIEW |
| 2 | `pip install pyvisa pyvisa-py` | Install NI packages |
| 3 | Write a plug class per instrument | Create instrument driver VIs |
| 4 | Write phase functions, one per test step | Create the test sequence |
| 5 | Declare measurements and limits in `procedure.yaml` | Configure Numeric Limit Tests |
| 6 | `tofupilot run ./procedure.yaml` | Run the VI |
| 7 | Compare results against LabVIEW on the same units | Validate measurement values |
| 8 | Link the procedure and add `--upload` | Configure TDMS logging |

## Common Gotchas

| Gotcha | Fix |
|--------|-----|
| "I miss the front panel" | Declare operator UI components in the procedure: text inputs, checklists, images, sliders, progress. No frontend code. |
| "My NI hardware needs NI drivers" | Use the nidaqmx Python package for DAQmx hardware. For GPIB, install NI-VISA runtime. |
| "LabVIEW handles parallelism automatically" | Declare independent phases and fixture slots; the engine schedules them. Use `depends_on` where order matters. |
| "My team doesn't know Python" | Python has a gentler learning curve than LabVIEW. Most engineers learn it in days, not weeks. |
| "We have years of LabVIEW code" | Migrate incrementally. Start with new tests in Python. Convert existing tests one at a time, validating each against LabVIEW results. |
| "One instrument, several parallel phases" | Give each phase its own instrument, or serialize the sharing phases with `depends_on`. |

### Reskill a LabVIEW Developer to Python
URL: https://www.tofupilot.com/guides/reskill-a-labview-developer-to-python
Learn how a LabVIEW developer moves to Python in six weeks, with a concept map, a study plan, plug and phase code and the stumbles to expect.

A LabVIEW developer already has the hard part of a Python test engineer's job: they know the instruments, the fixtures, the limits and what a marginal DUT looks like on the bench. Only the language changes, and it's the smaller of the two things to learn. Six weeks of part-time study with the TofuPilot Framework turns a driver VI into a plug, a state machine into a procedure file and a TDMS folder into a dashboard.

## Why This Works

The scarce part of a test engineer is domain knowledge, not syntax. Your LabVIEW developer knows which pogo pin lifts on fixture 3, why the 5 V rail reads high for the first 200 ms, and which SCPI query the old DMM answers with a trailing newline. None of that is in a certification exam and all of it transfers.

What doesn't transfer is the G-specific layer: the sequencer they built, the front panel, the error wire, the report VIs. On the TofuPilot Framework those four things are provided. The sequencer is `procedure.yaml`, limits are validators in the same file, the operator UI is declared, and results land in TofuPilot. The developer has to learn Python, and only Python.

The six weeks below assume a working LabVIEW developer with a few hours a week and no prior Python. A CLD with a computer science degree will finish in three.

For the manager, this is the cheapest option on the table. The developer keeps their salary and their fixture knowledge, the plant keeps the person who knows why station 3 needs a 200 ms settle, and nobody spends a quarter recruiting into a pool that skews senior and expensive. The cost is a few hours a week for six weeks and one bench instrument they can break without consequences.

## Concept Map

Every LabVIEW concept has a Python or framework equivalent. Most of them are simpler.

| LabVIEW / TestStand | Python + TofuPilot Framework | What changes |
|---|---|---|
| VI | Function | A `def` with parameters. No front panel, no connector pane |
| SubVI | Module | A `.py` file you import. Reuse is `import plugs.dmm` |
| Front panel | Declared operator UI | Text inputs, checklists, images and progress bars are declared in `procedure.yaml` |
| Error cluster | Exceptions | `raise` instead of wiring. The framework records the error and fails the phase |
| State machine | Procedure sequence | The `main:` list in `procedure.yaml`, with `depends_on` for fan-out |
| Typedef cluster | Dataclass or dict | `@dataclass` for structure, `dict` for anything JSON-shaped |
| TDMS file | TofuPilot run | Measurements upload. FPY, Cpk and histograms are already charts |
| DAQmx | `nidaqmx` | Same driver, Python binding |
| NI-VISA driver VI | PyVISA plug | A class with `__init__`, methods and `close()` |
| TestStand sequence | `procedure.yaml` | Steps become phases, limits become validators |
| Project (`.lvproj`) | Git repository | Text files, diffable, reviewable |

The [LabVIEW migration guide](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot) walks the first nine rows in detail. If the plant runs TestStand on top, the [TestStand migration guide](/guides/how-to-migrate-from-ni-teststand-to-python-with-tofupilot) covers the sequence file and the callbacks.

Two LabVIEW habits have no row because they have no equivalent. Highlight execution and probes become `log.info()` calls and a terminal, which feels like a loss for a week and then doesn't. The run arrow becomes a command. Both are covered in the stumbles table below.

## A Six-Week Self-Study Path

Two to four hours a week, on a real instrument from the bench, not a tutorial. Every exercise produces a file the developer keeps, and by week 3 those files are the start of a real station.

| Week | Goal | Exercise | Done when |
|---|---|---|---|
| 1 | Python basics on the bench | Install Python, `pip install pyvisa pyvisa-py`, send `*IDN?` to the bench DMM from a script | The DMM answers from a `.py` file |
| 2 | A plug instead of a driver VI | Wrap the DMM in a class with `read_voltage()` and `close()` | The class replaces one driver VI on one station |
| 3 | A phase with limits | Write `procedure.yaml` with one phase and two validators, run it with `tofupilot run` | Pass and fail both show correctly in the terminal |
| 4 | Sequence and operator UI | Add three phases, `depends_on`, and a checklist for the operator | The sequence matches the old state machine step for step |
| 5 | A real template | Clone a TofuPilot template, run it with mock plugs, swap one plug for the bench instrument | The template runs on real hardware |
| 6 | Upload and read results | `tofupilot login`, `tofupilot link`, run with `--upload`, read Cpk in the dashboard | Cpk for one measurement is visible without a spreadsheet |

Each row builds on the previous one. By week 4 the developer has a station folder they could hand to anyone on the team.

Pair them with whoever on the team writes Python, even a firmware engineer, for thirty minutes a week. The questions in weeks 1 and 2 are about the language and take a minute to answer in person and an hour to answer alone. From week 3 on, the questions are about the framework, and the [Python docs](https://docs.python.org/3/tutorial/) plus `tofupilot run` cover most of them.

## Weeks 1 to 2: A Plug Instead of a Driver VI

A LabVIEW driver VI is a front panel, a VISA session wire and an error cluster. In Python it's a class. The framework calls `__init__` at setup, injects the instance into any phase that names it and calls `close()` at teardown. The [PyVISA docs](https://pyvisa.readthedocs.io/) cover the resource manager and the `@py` backend.

```python filename="plugs/dmm.py"
# Replaces "DMM Read Voltage.vi": one VISA session, a mux channel, one query, one close
import pyvisa


class Multimeter:
    def __init__(self, address="TCPIP::192.168.1.100::INSTR"):
        rm = pyvisa.ResourceManager("@py")
        self.inst = rm.open_resource(address)
        self.inst.timeout = 5000
        self.inst.write("*RST")

    def read_voltage(self, channel):
        self.inst.write(f"ROUT:CLOS (@{channel})")
        return float(self.inst.query(":MEAS:VOLT:DC?"))

    def close(self):
        self.inst.close()
```

The address comes from `procedure.yaml` through `config`, so the same plug serves three stations with three DMMs. That's the SubVI reuse a LabVIEW developer already expects, without the connector pane. The `float()` is the week 2 lesson: everything VISA returns is a string, and the conversion belongs in the plug.

## Weeks 3 to 4: A Phase and Its Limits

In LabVIEW the limit lives in a comparison node next to the measurement, or in a limits table the front panel loads. In the framework it lives in `procedure.yaml`, and the phase only assigns the value.

```yaml filename="procedure.yaml"
# Two rails with limits; the plug address is config, not code
name: Controller Board FCT
version: 1.0.0

unit:
  serial_number:
    default_value: "SN-0001"
  part_number:
    default_value: "CTRL-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter
    config:
      address: "TCPIP::192.168.1.100::INSTR"

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: rail_3v3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: rail_5v
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
```

```python filename="phases/power_rails.py"
# No comparison node: the validators in procedure.yaml decide pass or fail
def power_rails(measurements, dmm, log):
    rail_3v3 = dmm.read_voltage(101)
    rail_5v = dmm.read_voltage(102)
    log.info(f"3V3 {rail_3v3:.3f} V, 5V {rail_5v:.3f} V")
    measurements.rail_3v3 = rail_3v3
    measurements.rail_5v = rail_5v
```

Run it with `tofupilot run ./procedure.yaml`. Then tighten one validator until it fails and run again. Seeing both outcomes in the terminal is the week 3 exit test.

Week 4 adds the rest of the old state machine. Fan-out after power-on is `depends_on`, and the front panel checklist becomes a declared component with no code behind it.

```yaml filename="procedure.yaml"
# Week 4: fan-out after power-on, plus an operator checklist with no Python behind it
main:
  - name: Power Rails
    key: power_rails
    python: phases.power_rails
  - name: Firmware Handshake
    key: firmware
    python: phases.firmware
    depends_on: [power_rails]
  - name: Visual Check
    key: visual
    depends_on: [power_rails]
    ui:
      components:
        - key: visual_checks
          type: checklist
          label: "Visual inspection"
          required: true
          options:
            - label: "Status LED lit"
              value: "led"
            - label: "No solder bridges on J3"
              value: "bridges"
```

## Weeks 5 to 6: Run a Template, Upload, Read Cpk

Week 5 starts from a working station instead of a blank file. The IMU thermal calibration template is a good fit for a calibration engineer; the FCT fixture template for a PCBA line.

```bash filename="week5.sh"
# A complete station with mock instruments, running in minutes
curl -fsSL https://www.tofupilot.app/install | sh
git clone https://github.com/tofupilot/template-framework-imu-thermal-calibration
cd template-framework-imu-thermal-calibration
tofupilot run ./procedure.yaml
```

Read `procedure.yaml` top to bottom, then swap one mock plug for the week 2 class. The phases don't change. That's the moment most LabVIEW developers stop worrying about the language.

Week 6 connects the station to the dashboard and runs twenty units.

```bash filename="week6.sh"
# Link the folder to a procedure in the dashboard, then upload every run
tofupilot login
tofupilot link
tofupilot run ./procedure.yaml --upload
```

Open the procedure in TofuPilot and find the Cpk and histogram for one measurement. Don't compute Cpk in Python. The dashboard already does it, and that's the point of week 6: the developer watches their measurements become the plant's yield numbers without writing a report VI.

The exit test for week 6 is a hand-over. Give the station folder to a colleague who has never seen it, and have them run twenty units with `--upload` using only the README. If they manage it, the developer has learned the part of Python that matters for a test station, and the station has stopped being a single point of failure. If they don't, the gap is usually a missing dependency in `pyproject.toml` or an instrument address hard-coded in a plug instead of `config`.

## Common Stumbles

| Stumble | What the developer sees | Fix |
|---|---|---|
| Indentation | `IndentationError` on a line that looks fine | Spaces only, four per level. Set the editor to show whitespace |
| Exceptions vs the error wire | A failing plug call stops the phase with a traceback | That's the error wire. Catch with `try` and `except` only where you'd have handled the cluster, otherwise let it propagate |
| No run arrow | "How do I run this?" | `tofupilot run ./procedure.yaml` for the station, `python -c` for a plug on its own |
| Virtual environments | `ModuleNotFoundError: pyvisa` right after installing it | The framework builds a `venv` from `pyproject.toml`. Add the dependency there, not with a global `pip install` |
| Implicit parallelism | Expects two unwired nodes to run at once | Python runs top to bottom. Use `depends_on` in `procedure.yaml` for fan-out |
| Everything is a string | `TypeError` when comparing a VISA reply to a limit | Convert with `float()` at the plug boundary, never in the phase |
| Mutable references | A list passed to a function changes outside it | Python passes references. Copy with `list(x)` where you'd have branched a wire |

The stumble that isn't in the table is reading. A LabVIEW developer is used to seeing the whole sequence as a diagram, and a folder of `.py` files looks like less information at first. It's the same information in a different shape, and it comes with something the diagram never had: a diff. After a month of `git log` on their own station, most developers stop asking for the picture.

The full list of what a Python test engineer should be able to do is in the [Python Skills Checklist for Test Engineers](/guides/python-skills-checklist-for-test-engineers). Use it as the week 6 exit interview.

## Start With One Station

Pick the station with the highest volume, or the one only one person can open. Rebuild it in Python with the TofuPilot Framework and run it side by side with the LabVIEW version on the same units for two weeks. Compare the two result sets in TofuPilot before you retire anything.

```bash filename="install-and-run.sh"
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
```

The step-by-step is in [How to Migrate from LabVIEW to Python](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at https://tofupilot.com/products/framework. The Lab tier is free, and `tofupilot run` works with no account.


### Run a Mixed LabVIEW and Python Test Team
URL: https://www.tofupilot.com/guides/run-a-mixed-labview-and-python-test-team
Learn how to run LabVIEW and Python stations on one TofuPilot procedure during a migration, with comparable FPY, clear ownership and a two-week switch test.

A plant mid-migration runs LabVIEW and Python stations side by side for a year or more, and the way to keep that sane is one TofuPilot procedure per product with two implementations reporting into it. FPY stays comparable across the line, ownership stays clear, and stations move over one at a time instead of in a big bang. This guide covers which stations to move first, how each side reports, who owns what, and what to compare before you switch.

## Decide Which Stations Stay on LabVIEW for Now

Score every station on four criteria. The scores decide the order; nothing else does.

| Criterion | Keep on LabVIEW for now | Migrate first | Migrate last |
|---|---|---|---|
| Volume | A few units a week | Every unit passes through it | Medium, steady |
| Change frequency | Limits haven't moved in two years | Limits or sequence change monthly | A few changes a year |
| Who can open it | Two or more people can open and change the VI | One person, or nobody since they left | One person, and they're staying |
| NI DAQ dependence | Tight: FPGA timing on a PXI chassis, cDAQ loops at kHz rates | None or thin: SCPI over VISA, serial, USB | A few DAQmx analog channels (the `nidaqmx` Python package covers those) |

Migrate first wins on any two of volume, change frequency and who can open it. Keep wins on NI DAQ dependence alone: FPGA timing on PXI is a real reason to stay, and it's the only criterion on the list that doesn't get worse with time.

A station that scores "keep" today is not kept forever. It's kept until the product it tests goes end-of-life, or until the PXI chassis needs replacing, whichever comes first.

## One Procedure, Two Implementations

In TofuPilot a procedure is the test definition for a product, with one `procedure_id`. Both the LabVIEW station and the Python station report their runs into that one procedure. FPY, Cpk per measurement, histograms and station throughput are then filterable per station on the same page, which is the only way to compare the two honestly.

Three rules keep the comparison valid:

| Rule | Why |
|---|---|
| Same phase names, same measurement names, same units | Analytics key on the name. `rail_3v3` on one station and `Rail 3V3` on the other are two different measurements |
| Same limits, changed on both sides in the same week | A limit that moves on one side only shows up as a yield gap that isn't real |
| `procedure.yaml` is the source of truth for names and limits | The Python side is text, diffs and gets reviewed. The LabVIEW side follows it |

## The LabVIEW Side

Two options. Pick A when the LabVIEW engineer is staying and has time. Pick B when they've left, or when the station is on the "migrate last" list and you don't want to touch the VI.

### Option A: post results to the REST API from the HTTP Client VIs

The station keeps running as it is, and adds one call at the end: `POST https://www.tofupilot.app/api/v2/runs` with an `Authorization: Bearer <api key>` header, `Content-Type: application/json`, and this body.

```json filename="run-payload.json"
{
  "procedure_id": "9f1c2d3e-4b5a-6c7d-8e9f-0a1b2c3d4e5f",
  "serial_number": "SN-000123",
  "part_number": "PCBA-100",
  "outcome": "PASS",
  "started_at": "2026-09-21T08:14:02Z",
  "ended_at": "2026-09-21T08:15:40Z",
  "phases": [
    {
      "name": "Power Rails",
      "outcome": "PASS",
      "started_at": "2026-09-21T08:14:05Z",
      "ended_at": "2026-09-21T08:14:09Z",
      "measurements": [
        {
          "name": "rail_3v3",
          "outcome": "PASS",
          "measured_value": 3.31,
          "units": "V",
          "lower_limit": 3.2,
          "upper_limit": 3.4
        }
      ]
    }
  ]
}
```

In LabVIEW that's `OpenHandle`, `AddHeader` twice, `POST` with the JSON string, check the status code, `CloseHandle`. Build the string with Flatten To JSON from a cluster shaped like the payload, and keep the API key in a config file on the station PC, not in the VI. `outcome` is one of `PASS`, `FAIL`, `ERROR`, `TIMEOUT`, `ABORTED`; the `procedure_id` is on the procedure page in TofuPilot.

There's no LabVIEW SDK or VIPM package for this. It's a plain HTTPS call, which is also why it doesn't break on the next LabVIEW version.

### Option B: wrap the LabVIEW executable in a Python phase

Build the LabVIEW station as an executable that writes a result file, and call it from one phase. The measurements then flow through `procedure.yaml` like any Python station, and the LabVIEW code becomes an instrument the Python side drives.

```python filename="phases/labview_fct.py"
# Run the built LabVIEW station as a subprocess, read its result file,
# and set the measurements so the run lands in TofuPilot like any other.
import json
import subprocess
from pathlib import Path

RESULT = Path(r"C:\stations\fct\last_result.json")


def labview_fct(measurements, unit):
    if RESULT.exists():
        RESULT.unlink()
    subprocess.run(
        [r"C:\stations\fct\FCT.exe", "--serial", unit.serial_number],
        check=True,
        timeout=180,
    )
    result = json.loads(RESULT.read_text())
    measurements.rail_3v3 = result["rail_3v3"]
    measurements.idle_current = result["idle_current"]
```

The limits stay in `procedure.yaml`, so the LabVIEW executable only reports values. That's the point: one place for limits, even while two runtimes exist.

## The Python Side

The Python station is a folder under Git: `procedure.yaml`, `phases/`, `plugs/`. The names below match the LabVIEW payload exactly.

```yaml filename="procedure.yaml"
# Same names, same units, same limits as the LabVIEW station.
name: PCBA-100 FCT
version: 2.3.0

unit:
  serial_number:
    default_value: "SN-000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: rail_3v3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

The phase is `def power_rails(measurements, dmm): measurements.rail_3v3 = dmm.read_voltage()`, and the plug wraps PyVISA. The [TestStand migration guide](/guides/how-to-migrate-from-ni-teststand-to-python-with-tofupilot) covers the sequencer side when TestStand sits on top of the LabVIEW station.

## Ownership Rules

Two runtimes with fuzzy ownership is how limits drift. Write these down and put them in each station's README.

| What | Owner | Reviewer | Where it lives |
|---|---|---|---|
| Measurement names, units, limits | Test engineering lead | Quality | `procedure.yaml` in the Python repo, mirrored to the LabVIEW station in the same week |
| LabVIEW station code | The LabVIEW engineer | The Python engineer reviews the posted payload against a Python run, not the VI | The station's own repo, one per station, built executable tagged |
| Python station code | The Python engineer | The LabVIEW engineer reviews the phase logic (they know the fixture) | The station's own repo, one per station, tag per release |
| API keys | Test engineering lead | Nobody else | A config file on each station PC, one key per station, rotated [when someone leaves](/guides/what-to-do-when-your-labview-engineer-leaves) |
| Which version runs where | Whoever releases | Anyone, in TofuPilot | The run carries the procedure version; the station PC carries the tag |

One repo per station, releases tagged, and the tag on the PC matches the version in the file. The [hand-over guide](/guides/document-a-test-station-for-hand-over) has the README template both sides use. Both engineers will find a bug in the other's station in the first week, which is the review working as intended.

## Compare Before You Switch

Run both implementations on the same units for two weeks before the LabVIEW station goes to the spare bench. Same serial numbers through both stations, in the same shift where possible. Everything below is on the procedure page in TofuPilot, filtered by station.

| Compare | Where to look | Switch when |
|---|---|---|
| Measurement means, per measurement | Measurement histogram, one station per series | The offset between stations is smaller than the gauge repeatability you already accept |
| Spread, per measurement | Histogram width, control chart | Python spread is equal or narrower; wider means a timing or settling difference in the plug |
| FPY | Procedure page, per station | Within the noise for the volume, and no new failure mode |
| Failure Pareto | Pareto, per station | Same top three failing measurements on both sides |
| Cycle time | Station throughput | Equal or better; slower usually means a fixed `sleep` that a settle-and-poll replaces |
| Retest behaviour | Run pages per unit | Units that failed then passed did so on both sides for the same measurement |

A mean offset that's stable and small is a plug difference (different NPLC, different range) and is fixed in the Python plug. A wider spread is a settling problem. A different Pareto means the two stations don't test the same thing, and that's a conversation with quality, not a code change.

## What to Do With the LabVIEW Seats

LabVIEW has been subscription-only since 2023, roughly $1,300 per year for Base, $4,300 for Full and $5,500 and up for Professional, per seat. TestStand deployment licences run about $1,000 to $1,500 per station. The [pricing guide](/guides/labview-pricing-what-it-actually-costs) has the detail.

The plan writes itself from the migration order:

- Stop renewing the deployment licence on each station the week it switches. Deployment licences are per station; they don't transfer to the Python one because it needs none.
- Keep one Full seat for as long as any station is on the "keep" list. Someone has to open the VI to read it during a migration, and to fix the PXI station.
- Don't count on Community Edition as the fallback. It's non-commercial only.
- Retire the last seat when the last kept station retires, not before. A LabVIEW station with no seat in the building is a station nobody can fix.

Budget the freed seats into the migration itself. One Full seat a year pays for a good chunk of the Python engineer's onboarding.

## Start With One Station

Pick the station with the highest volume, or the one only one person can open. Rebuild it in Python with the TofuPilot Framework, then run it side by side with the LabVIEW version on the same units for two weeks, using the comparison table above. The LabVIEW station reports into the same procedure the whole time, so the day you switch, the FPY history doesn't restart.

```bash filename="station-setup.sh"
# Install the CLI, run a procedure locally, then run it with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The [TestStand migration guide](/guides/how-to-migrate-from-ni-teststand-to-python-with-tofupilot) covers the sequencer rebuild step by step, and the [Framework page](https://tofupilot.com/products/framework) covers the layout. The Lab tier is free, and `tofupilot run` works with no account.


### Onboard a Test Engineer in One Week
URL: https://www.tofupilot.com/guides/onboard-a-test-engineer-in-one-week
Learn how to take a Python engineer with no test background from CLI install to owning a real station with uploaded runs and an FPY review in five days.

A Python engineer with no manufacturing test background can own a station in one week when the framework provides the sequencer, the limits, the operator UI and the data platform. The five days below take a new hire from installing the CLI to running a real station with `--upload` and reviewing FPY with their lead. What they learn along the way is the product and the instruments, which is the part no framework can provide.

## What the Lead Prepares in Advance

The week stalls on Monday morning if any of these is missing.

- **An account and an API key.** The hire needs their own login on the TofuPilot org, and the station needs an API key for uploads. The Lab tier is free.
- **A procedure created in the app.** Create it before Monday and note its name. Day 1 links the template to it, Day 5 links the real station.
- **A station with one instrument on the network.** A bench DMM or DAQ with a LAN port, a known VISA address and a cable to the DUT. If it only has GPIB or USB, install the vendor VISA driver on the station PC before the hire arrives.
- **A known-good unit and a known-bad one.** One unit that passes, one with a documented fault. Day 3 and Day 5 are much shorter with both on the bench.
- **A buddy.** The person who runs the station today, booked for an hour a day. Not the lead.
- **The product's must-pass list.** The five measurements that decide ship or scrap, with their limits, on one page.

The job description that got you this hire probably already lists what they know. [Test Engineer Job Description for Python Hires](/guides/test-engineer-job-description-for-python-hires) is the version that matches this week.

## The Week, Day by Day

| Day | Task | Deliverable | What they learn |
|---|---|---|---|
| 1 | Install the CLI, clone the `fct-fixture` template, run it with mock plugs, upload one run | A run page in TofuPilot with their name on it | What a procedure, a phase, a plug and a run are |
| 2 | Replace one mock plug with a real instrument over PyVISA | A rail measurement from real hardware | SCPI, VISA addresses, why instruments return strings |
| 3 | Add a phase with a measurement and limits | A phase that passes on the good unit and fails on the bad one | Validators, measurement keys, how a fail propagates |
| 4 | Shadow an operator for an hour, then add operator UI to the procedure | A checklist and instructions the operator asked for | What the floor does between two runs |
| 5 | Run one real station with `--upload`, review FPY with the lead | Twenty uploaded runs and a first yield conversation | How their code becomes the plant's numbers |

## Day 1: Run the Template

The `fct-fixture` template is a complete PCBA functional test station with mock instruments. It runs on a laptop with no hardware attached.

```bash filename="day1.sh"
# Install the CLI, run the template locally, then link it and upload one run
curl -fsSL https://www.tofupilot.app/install | sh
git clone https://github.com/tofupilot/template-framework-fct-fixture
cd template-framework-fct-fixture
tofupilot run ./procedure.yaml
tofupilot login
tofupilot link
tofupilot run ./procedure.yaml --upload
```

The first `tofupilot run` needs no account. Have the hire read `procedure.yaml` top to bottom before the second one: `unit`, `plugs`, `setup`, `main`, `teardown`. Then open the uploaded run in TofuPilot and match each phase on the page to its block in the file. That mapping is most of Day 1. The [template page](https://tofupilot.com/templates/functional-test-fixture-fct) explains what each phase is for.

## Day 2: Swap a Mock Plug for a Real Instrument

The template's `daq` plug is a mock that covers rails, GPIO loopback, an AWG and photodiodes. The hire doesn't replace all of it. They subclass the mock and override only the rail read with the bench DMM, so every other phase keeps running on mock data until the real fixture exists.

```python filename="plugs/bench_daq.py"
# Day 2: real rail reads over PyVISA; everything else stays mock until the fixture arrives
import pyvisa

from plugs.daq import FixtureDaq

RAIL_CHANNEL = {"3v3": 101, "5v": 102, "1v8": 103}


class BenchDaq(FixtureDaq):
    def __init__(self, address="TCPIP::192.168.1.100::INSTR"):
        super().__init__()
        self.inst = pyvisa.ResourceManager("@py").open_resource(address)
        self.inst.timeout = 5000

    def measure_rail(self, name):
        self.inst.write(f"ROUT:CLOS (@{RAIL_CHANNEL[name]})")
        return float(self.inst.query(":MEAS:VOLT:DC?"))

    def close(self):
        self.inst.close()
```

```yaml filename="procedure.yaml"
# Point the daq key at the new class; the phases don't change
plugs:
  - name: Fixture DAQ
    key: daq
    python: plugs.bench_daq:BenchDaq
    config:
      address: "TCPIP::192.168.1.100::INSTR"
```

Run it again. The three rail measurements now come from the DMM, and the run page in TofuPilot shows real values next to Monday's mock ones. The lesson of Day 2 is the `float()`: everything VISA returns is a string, and the conversion belongs in the plug, not the phase. The [PyVISA docs](https://pyvisa.readthedocs.io/) cover the rest.

## Day 3: Add a Phase With a Measurement and Limits

Pick one line from the lead's must-pass list that the template doesn't cover yet. The template checks ripple on the 3V3 rail, so add it on the 5 V rail, after power-on.

```yaml filename="procedure.yaml"
# Day 3: one new phase, one measurement, two validators, runs after power_on
main:
  - name: Rail Ripple
    key: rail_ripple
    python: phases.rail_ripple
    depends_on: [power_on]
    measurements:
      - name: ripple_5v
        unit: mV
        validators:
          - operator: ">="
            expected_value: 0
          - operator: "<="
            expected_value: 50
```

```python filename="phases/rail_ripple.py"
# The phase assigns the value; procedure.yaml decides pass or fail
def rail_ripple(measurements, daq):
    measurements.ripple_5v = daq.measure_ripple_mv("5v")
```

Run it on the good unit, then on the bad one. Pass, then fail, and the run page shows which validator tripped. Keys matter here: `ripple_5v` in YAML is `measurements.ripple_5v` in Python, and the dashboard groups history by that key, so pick it once and don't rename it.

## Day 4: Shadow an Operator, Then Add Operator UI

Morning: an hour on the floor with the buddy, watching three units go through. The hire notes what the operator checks by eye, what they do that isn't in the procedure, and what they'd want to see on screen. The operator has known about the loose UART ribbon for a year. Day 4 is when someone writes it down.

Afternoon: declare it. Operator UI in the framework is YAML, not frontend code.

```yaml filename="procedure.yaml"
# Day 4: what the operator asked for, declared in YAML, no Python behind it
main:
  - name: Fixture Setup
    key: fixture_setup
    ui:
      requires_input: true
      components:
        - key: instructions
          type: text
          label: "Before closing the lid"
          default_value: "Seat the board on the four locating pins, then connect the UART ribbon with the red stripe toward the board edge."
        - key: setup_checks
          type: checklist
          label: "Setup checks"
          required: true
          options:
            - label: "Board seated on all four pins"
              value: "seated"
            - label: "UART ribbon connected, red stripe out"
              value: "uart"
            - label: "ESD strap on"
              value: "esd"
```

The phase has no Python. The framework shows the text, waits for the checklist and moves on. The hire learns on Day 4 that a good share of yield loss is fixture handling, and that the fix is often a sentence on a screen, not a limit change. [Document a Test Station for Hand-Over](/guides/document-a-test-station-for-hand-over) is where those sentences end up.

## Day 5: Own One Station

Morning: the real station, with the real DUT, linked to the production procedure. Twenty units, good and bad mixed in.

```bash filename="day5.sh"
# The real station, linked to the production procedure, uploading every run
tofupilot link ./stations/fct-01 --procedure "Controller Board FCT"
tofupilot run ./stations/fct-01/procedure.yaml --upload
```

Afternoon: sit down with the lead, open the procedure in TofuPilot and read three things together: FPY, the failure Pareto and the histogram of the Day 3 measurement. Three questions to ask. Which phase fails most, and is that a limit or a fixture? Is the ripple limit too tight for what the histogram shows? What's the Cpk on the rail measurements now that they come from real hardware?

Don't ask the hire to compute FPY. TofuPilot tracks it, and the exercise is reading it and asking why. By Friday afternoon they've written a plug, a phase and an operator screen, uploaded real runs and had one yield conversation. That's ownership. The [Python Skills Checklist for Test Engineers](/guides/python-skills-checklist-for-test-engineers) is the list to review with them the following Monday.

## Why This Is a Week, Not a Quarter

On a LabVIEW station, a new hire spends the first quarter learning the sequencer someone else built, the front panel conventions, the error wire and the report VIs, before they change a single limit. The framework provides those four things, so the five days above skip them entirely.

What's left is the product and the instruments. A Python engineer with no test background typically reaches "owns a station" in weeks, not months, when the sequencer is provided. What moves it is how many instruments the station has, how well the buddy knows the fixture and whether the must-pass list existed before Monday.

The week also leaves something behind that a quarter of LabVIEW onboarding doesn't. The hire's plug, phase and operator screen are text files in a repository, reviewed by the lead in a pull request and visible to the next hire. Onboarding the second engineer takes the same week, and the station never again depends on the first one.

## Start With One Station

Pick the station with the highest volume, or the one only one person can open. Rebuild it in Python with the TofuPilot Framework and run it side by side with the LabVIEW version on the same units for two weeks. Compare the two result sets in TofuPilot before you retire anything.

```bash filename="install-and-run.sh"
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
```

The step-by-step is in [How to Migrate from LabVIEW to Python](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at https://tofupilot.com/products/framework. The Lab tier is free, and `tofupilot run` works with no account.


### Bus Factor in Test Engineering: A Guide
URL: https://www.tofupilot.com/guides/bus-factor-in-test-engineering-a-guide
Learn what bus factor means for a test station, how to audit every station in an afternoon, and the steps that stop one departure from stalling the line.

Bus factor is the number of people who can leave before a test station can no longer be rebuilt or changed. On most production floors that run LabVIEW, that number is one. You can measure it in an afternoon, and the fixes range from a written hand-over to a rewrite in text under Git.

## What Bus Factor Means for a Test Station

The term comes from software: how many people would have to get hit by a bus before the project stalls. For a test station the question is more concrete. If the person who built it left tomorrow, could someone else open the sequence, change a limit, and redeploy it by Friday?

Three things have to be true for the answer to be yes. Someone else can read the source, and someone else has a licence and a machine that can build it. And the results live somewhere the team can see them, not on the station's local drive.

The count of people who satisfy all three is the bus factor. A station with a bus factor of one isn't broken, it's just a single point of failure that happens to still be working.

## Audit It in an Afternoon

You don't need a tool. Walk the floor with a spreadsheet and fill one row per station.

```csv filename="stations.csv"
station,product,runs_per_week,tool,who_can_open,who_can_rebuild,last_change
FCT-01,controller-pcba,420,LabVIEW 2020,marc,marc,2024-11-03
FCT-02,controller-pcba,380,LabVIEW 2020,marc,marc,2024-11-03
EOL-01,drone-assembly,150,TestStand 2019,marc;priya,marc,2023-06-18
BURN-01,battery-module,90,Python,priya;tom;lena,priya;tom;lena,2026-08-21
CAL-01,imu-module,60,LabVIEW 2018,marc,,2021-02-09
```

Two columns carry the answer. `who_can_open` lists everyone who can read the source and say what it does. `who_can_rebuild` lists everyone who has the licence, the right toolchain version and enough understanding to produce a new deployable. Separate names with semicolons. Leave the cell empty when the honest answer is nobody.

Be strict about `who_can_rebuild`. "Priya once watched Marc do it" doesn't count. If the toolchain version is no longer installed anywhere, the cell is empty even if Marc is still on payroll.

This script reads the file and prints the stations at risk, highest volume first.

```python filename="bus_factor.py"
# Prints the stations that one person (or nobody) can rebuild, highest volume first.
import csv


def people(cell):
    return {name.strip() for name in cell.split(";") if name.strip()}


with open("stations.csv", newline="") as f:
    rows = list(csv.DictReader(f))

at_risk = []
for row in rows:
    rebuilders = people(row["who_can_rebuild"])
    if len(rebuilders) <= 1:
        at_risk.append((int(row["runs_per_week"]), row["station"], rebuilders))

for runs, station, rebuilders in sorted(at_risk, reverse=True):
    who = ", ".join(sorted(rebuilders)) or "nobody"
    print(f"{station:8} {runs:5d} runs/week  bus factor {len(rebuilders)}  ({who})")
```

For the sample file it prints FCT-01, FCT-02 and EOL-01 at bus factor 1, all on Marc, and CAL-01 at bus factor 0. That's also your fix order: the station with the most runs per week and the fewest rebuilders goes first.

Most plants learn their bus factor the same week they learn about a notice period. The afternoon audit is cheaper.

## What Drives It Down

None of these are mistakes. They're the defaults of the toolchain and the way small teams grow, and each one quietly removes a name from `who_can_rebuild`.

| Driver | Why it lowers the number | How you notice |
|---|---|---|
| Binary VIs | A `.vi` file diffs as a blob. Nobody can review a change without opening it in the same LabVIEW version. | Pull requests don't exist. Changes are "Marc sent me the new build". |
| Vendor language | LabVIEW is capable, and fewer engineers learn it each year. The pool of people who can read the source shrinks on its own. | Job postings stay open for months. |
| Licences tied to a named seat | Since 2023 LabVIEW and TestStand are subscription only. The seat, the build machine and the person are usually the same thing. | Deactivating a leaver's account also deactivates the ability to build. |
| Undocumented fixtures | The pogo pin map, the relay matrix and the "don't touch DIP switch 3" knowledge live in one head. | The fixture drawing is a photo on someone's phone. |
| Results in local TDMS or CSV files | Yield and Cpk data sits on the station's drive. The only person who can answer "what's FPY this month" is the one who wrote the report VI. | Management asks for a yield number and waits two days. |

Two of these (vendor language and named seats) are structural and you can't fix them inside the toolchain. The other three you can, which is what the ladder below is about.

## The Mitigation Ladder

Each level is a real step and each one costs something. Pick the level you can hold, then climb when you have the budget.

| Level | Action | Bus factor after |
|---|---|---|
| 0 | Nothing. One engineer, binary VIs, results on the station. | 1 |
| 1 | A hand-over document per station and a build machine a second person can log into. | 1, but recovery takes days instead of months |
| 2 | Source under Git, a second seat licence, and one other engineer performs a full rebuild from scratch once. | 2 |
| 3 | Results pushed off the station to TofuPilot through the REST API, so yield, Cpk and per-unit history stop depending on the report VI. | 2, and the data outlives the station |
| 4 | Station rewritten as text (procedure, phases, plugs) in Python under Git, results in TofuPilot. | Everyone on the team who reads Python |

Level 1 is what [what to do when your LabVIEW engineer leaves](/guides/what-to-do-when-your-labview-engineer-leaves) covers, and [document a test station for hand-over](/guides/document-a-test-station-for-hand-over) has the template. Do it this week for every station at bus factor 1. It doesn't change the number, it changes how long you're down.

Level 2 is where most teams stall. The second seat costs a subscription, and the second engineer has to be someone who can read LabVIEW, which brings you back to [why LabVIEW engineers are hard to hire](/guides/why-is-it-so-hard-to-hire-labview-engineers).

Level 3 needs no rewrite. A LabVIEW station can POST each run to `https://www.tofupilot.app/api/v2/runs` from the HTTP Client VIs with the outcome, serial number and measurements. The station stays as it is, but the data is now something the whole team sees.

## Why Text, Git and a Shared Platform Is the Durable Fix

Levels 1 to 3 patch the problem. They decay: the hand-over doc drifts, the second seat lapses, the second engineer moves to another project. Level 4 changes what the station is made of, and that's why it holds.

Three properties do the work.

The station is text. A `procedure.yaml` plus a folder of `phases/*.py` and `plugs/*.py` diffs line by line. Anyone who reads Python can review a limit change in a pull request, and Git keeps the history of every change and who approved it.

The toolchain is free and installed everywhere. There's no named seat to lapse, and a laptop with Python and the TofuPilot CLI can rebuild any station in the repo.

The results are shared. Every run lands in TofuPilot with its measurements, so FPY, Cpk and the failure Pareto are on a dashboard the whole team opens, not in a TDMS file only one person can parse.

Here's what "reviewable in a pull request" looks like. A reviewer who has never seen the station can tell you what it tests and what the limits are in under a minute.

```yaml filename="procedure.yaml"
# Controller PCBA functional test. Limits are the review surface.
name: Controller PCBA FCT
version: 2.4.0

unit:
  serial_number:
    default_value: SN-000000
  part_number:
    default_value: CTRL-PCBA-A

plugs:
  - name: psu
    python: plugs.psu:PowerSupply
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: rail_3v3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: rail_5v
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.85
          - operator: "<="
            expected_value: 5.15
```

When the 3.3 V limit needs to widen to 3.15 V, the diff is one line. The reviewer sees the old value, the new value and the author. That's the whole mechanism, and it's the reason the bus factor stops being a property of one engineer and becomes a property of the team.

The bus factor after level 4 isn't infinite. Someone still has to know the fixture and the product. But the knowledge that used to be locked in a binary is now in a file anyone can open, and that's the part that used to walk out the door.

## Start With One Station

Take the station your audit put at the top: the highest volume with the fewest rebuilders. Rebuild it in Python with the TofuPilot Framework, keep the same limits, and run it side by side with the LabVIEW version on the same units for two weeks. When the two agree, switch the operators over and update the CSV.

```bash filename="install-and-run.sh"
# Install the CLI, then run the station locally (no account needed) or with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The step-by-step rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework itself is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### What to Do When Your LabVIEW Engineer Leaves
URL: https://www.tofupilot.com/guides/what-to-do-when-your-labview-engineer-leaves
Learn what to freeze, inventory and rebuild in the first 48 hours, two weeks and 30 days after your LabVIEW engineer leaves, without losing yield data.

When your LabVIEW engineer leaves, the first job is to freeze what runs today, the second is to find out what you actually own, and the third is to keep results flowing while you rebuild the riskiest station in Python. Most plants do the first and skip the other two. The order below covers the first 48 hours, the first two weeks and the first 30 days, in that sequence.

## The First 48 Hours

Do these before anything else. Each takes an hour or two now and a week a month from now.

1. **Freeze the deployed builds.** Copy the exact executable, the run-time engine installer and every `.ini` or config file from each station PC to a share. Not the source, the thing that runs. If a station PC dies next week, this is what you reinstall.
2. **Locate the project files.** Find every `.lvproj`, `.vi`, `.lvlib`, `.ctl`, the TestStand `.seq` files and the build spec that produced each deployed executable. They're often on the engineer's laptop, a personal share or a USB stick in a drawer.
3. **Record the exact runtime versions.** LabVIEW version and bitness, the run-time engine on each station, TestStand version, DAQmx and NI-VISA driver versions. A 2019 VI doesn't open cleanly in 2024 without a recompile, and the recompile is where things break.
4. **Export licence info.** Which seats are on subscription, which NI account owns them, when they renew. LabVIEW seats and TestStand deployment licences are subscription-only now. A lapsed renewal means you can't open the source even after you find it.
5. **Copy everything to a repo.** Binary VIs included. Git won't diff them, but it timestamps them, and "the version that was on line 3 the day he left" becomes a commit instead of a memory. The [Git documentation](https://git-scm.com/doc) covers the basics if the plant has never used it.

Also rotate the station PC passwords and the NI account credentials, and keep read access to the engineer's mailbox for the notice period. Half the instrument workarounds are in email threads.

## The First Two Weeks: Inventory

You can't prioritize what you haven't counted. Walk the floor with a laptop and fill in one row per station.

| Station | Product | Volume per week | Who can open the VI | Runtime version | Risk |
|---|---|---|---|---|---|
| FCT-01 | Controller PCBA | 1,200 | Nobody | LabVIEW 2019 RTE, 32-bit | High |
| CAL-03 | IMU module | 800 | Nobody | LabVIEW 2017 RTE | High |
| EOL-02 | Gimbal assembly | 300 | Integrator (NI Alliance Partner) | LabVIEW 2021, TestStand 2021 | Medium |
| BURN-04 | Battery pack | 150 | Test technician (limits only) | LabVIEW 2023 | Low |

Risk is a function of three things: volume, who can open it and how old the runtime is. A high-volume station nobody can open on a runtime NI no longer ships is your first rebuild candidate. A low-volume station a technician can already tweak can wait a year.

Give the inventory a deadline of two weeks and one owner, and have the plant manager sign it. It's the document that turns "we lost our LabVIEW guy" from a panic into a plan with a first row, and it's the one you'll show a candidate or an integrator to explain what the job is.

The inventory is also the first page of the hand-over document you never had. [Document a Test Station for Hand-Over](/guides/document-a-test-station-for-hand-over) has the full template, and [Bus Factor in Test Engineering](/guides/bus-factor-in-test-engineering-a-guide) explains how the plant ended up with a column full of "Nobody" in the first place.

## Keep Results Flowing

Yield doesn't stop mattering because the engineer left. The one change you can make to a station without opening its sequence is where the results go.

Post each run to TofuPilot's REST API v2. The payload is small, the auth is a bearer token and the station doesn't need Python.

```json filename="run.json"
{
  "outcome": "PASS",
  "procedure_id": "8122a38c-3fbc-4bf0-881b-24ea1e2cb937",
  "serial_number": "UAUT-04829",
  "part_number": "PCB-MAIN-V2",
  "started_at": "2026-09-14T08:12:04Z",
  "ended_at": "2026-09-14T08:13:41Z",
  "phases": [
    {
      "name": "Power Rails",
      "outcome": "PASS",
      "started_at": "2026-09-14T08:12:10Z",
      "ended_at": "2026-09-14T08:12:14Z",
      "measurements": [
        {
          "name": "rail_3v3",
          "outcome": "PASS",
          "measured_value": 3.31,
          "units": "V",
          "validators": [
            { "operator": ">=", "expected_value": 3.2, "outcome": "PASS" },
            { "operator": "<=", "expected_value": 3.4, "outcome": "PASS" }
          ]
        }
      ]
    }
  ]
}
```

The `procedure_id` comes from the procedure page in TofuPilot. `outcome` is one of PASS, FAIL, ERROR, TIMEOUT or ABORTED, and the timestamps are ISO 8601. Phases and measurements are optional, so the minimum viable version is the top six fields. Create one API key per station in TofuPilot and keep it in the station's environment, not inside the VI, so rotating it later doesn't mean a rebuild.

From LabVIEW, the HTTP Client VIs do this in four nodes: Open Handle, Add Header twice (`Authorization: Bearer <api key>` and `Content-Type: application/json`), POST to `https://www.tofupilot.app/api/v2/runs` with the JSON string, Close Handle. Drop it into the existing report SubVI after the TDMS write, so nothing else in the sequence changes. If you'd rather not touch the VI at all, have the station write the JSON to a folder and let a short Python script post it.

```python filename="upload_run.py"
# Posts one run file to TofuPilot; call it from a folder watcher or System Exec after each test
import json
import os
import sys

import requests

API_URL = "https://www.tofupilot.app/api/v2/runs"


def upload(path):
    with open(path, encoding="utf-8") as f:
        payload = json.load(f)
    headers = {"Authorization": f"Bearer {os.environ['TOFUPILOT_API_KEY']}"}
    response = requests.post(API_URL, json=payload, headers=headers, timeout=30)
    response.raise_for_status()
    print(f"Uploaded run {response.json()['id']} from {path}")


if __name__ == "__main__":
    upload(sys.argv[1])
```

After a week you have FPY per station and a failure Pareto in TofuPilot without anyone having opened a VI. That's the number the plant manager asks for, and it's the number that tells you which station to rebuild first.

## The First 30 Days: Rebuild One Station

Take the top row of the inventory and rebuild it in Python with the TofuPilot Framework. The station becomes a folder: `procedure.yaml` for the sequence and limits, `phases/` for the test steps, `plugs/` for the instruments, all text and all under Git. Anyone on the team can open it.

```yaml filename="procedure.yaml"
# FCT-01 rebuilt: same limits as the LabVIEW station, same serial number format
name: Controller Board FCT
version: 2.0.0

unit:
  serial_number:
    default_value: "UAUT-00000"
  part_number:
    default_value: "PCB-MAIN-V2"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter
    config:
      address: "TCPIP::192.168.1.100::INSTR"

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: rail_3v3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: rail_5v
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
```

```python filename="phases/power_rails.py"
# Reads two rails; the framework checks them against the validators above
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage(101)  # mux channel 101 is the 3V3 rail
    measurements.rail_5v = dmm.read_voltage(102)
```

Run it side by side with the LabVIEW station on the same units for two weeks. Both post to the same procedure in TofuPilot, so the comparison is a filter on the run list, not a spreadsheet. When the Python station agrees with the old one on every unit, and disagrees only where the old one was wrong, retire the VI. [How to Migrate from LabVIEW to Python](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot) has the phase-by-phase mapping.

You don't need to convert the rest of the floor this month. [Run a Mixed LabVIEW and Python Test Team](/guides/run-a-mixed-labview-and-python-test-team) covers how the two coexist while the inventory shrinks one row at a time.

## What Not to Do

**Don't hire a contractor to keep the old VI alive indefinitely.** A certified NI integrator typically charges a four-figure day rate, and each change request comes back in days to weeks. That's fine for a quarter of bridge cover. As a permanent arrangement it's paying a premium to stand still.

**Don't rewrite everything at once.** Six stations rebuilt in one quarter is six unvalidated stations and one exhausted engineer. One station, side by side, two weeks, then the next.

**Don't skip the side-by-side.** The LabVIEW station has ten years of undocumented fixes baked in. The only way to find them is to run both on the same units and look at where they disagree.

**Don't recreate the single point of failure in Python.** The framework makes the station readable by anyone. Keep it that way with a repo, a README and a second person who has run it.

## Start With One Station

Pick the station with the highest volume, or the one only one person can open. Rebuild it in Python with the TofuPilot Framework and run it side by side with the LabVIEW version on the same units for two weeks. Compare the two result sets in TofuPilot before you retire anything.

```bash filename="install-and-run.sh"
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
```

The step-by-step is in [How to Migrate from LabVIEW to Python](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at https://tofupilot.com/products/framework. The Lab tier is free, and `tofupilot run` works with no account.


### Test Engineer Job Description for Python Hires
URL: https://www.tofupilot.com/guides/test-engineer-job-description-for-python-hires
Learn how to write a test engineer job description around the work instead of NI tools, with a full template, a rewrite table and 2026 salary ranges.

A test engineer job description that lists NI tools instead of the work filters out most of the people who could do the job. Written around the work (instruments, fixtures, limits, yield, operators, data), the same posting pulls from the whole Python pool, which is many times larger than the LabVIEW pool and skews younger. This guide gives you a full template, a line-by-line rewrite of the usual posting, and the salary ranges to attach.

## Why the Usual Posting Fails

The typical posting reads like a licence inventory: LabVIEW, TestStand, CLD preferred, 5+ years NI ecosystem. It describes the tool the last engineer used, not the problem the next one has to solve.

That framing has two costs. It shrinks the candidate pool to people who already hold a shrinking credential, and it tells a strong Python engineer with instrument experience that they aren't wanted. Fewer universities teach LabVIEW in a core curriculum now and most teach Python, so the pool you're excluding is the one that's growing.

The fix is to describe the work. Every line under "must have" should be a task the person will do in their first quarter, not the name of a product they have to have used.

## What the Job Actually Is

Strip the tools away and a manufacturing test engineer does six things.

| Area | What the work looks like |
|---|---|
| Instruments | Talk to a DMM, a power supply, a scope or a DAQ over SCPI, serial or USB. Read a datasheet, send a command, parse the reply. |
| Fixtures | Understand pogo pins, relay matrices, bed-of-nails and the mechanical side of getting a DUT connected. Debug an intermittent contact. |
| Limits | Turn a spec into a measurement with an upper and lower bound. Argue with design about what the bound should be. |
| Yield | Watch FPY per station and per phase. Find the phase that's failing good boards and the phase that's passing bad ones. |
| Operators | Write the sequence so an operator can run it with a barcode scanner and a screen, and so the failure message says what to do next. |
| Data | Make every run land somewhere the team can see it, with serial number, measurements and outcome, so a customer question about one unit takes a minute to answer. |

None of these say LabVIEW. All of them can be done in Python with PyVISA for the instruments, the TofuPilot Framework for the sequencer and operator UI, and TofuPilot for results. That's the stack the template below describes.

## The Template

Copy this into your careers page and edit the bracketed parts. It's written for a mid-level hire; the levelling section further down says what to change for junior or senior.

```markdown filename="job-description.md"
# Test Engineer, Manufacturing (Python)

[Company] builds [product] for [market]. Every unit we ship goes through
functional and end-of-line test on stations we design and run ourselves.
You'll own those stations.

## What you'll do

- Build and maintain the test stations for [product line]: the sequence,
  the instrument drivers, the operator screens and the limits.
- Talk to instruments over SCPI, serial and USB from Python (PyVISA), and
  write the plug that wraps each one so the rest of the team can reuse it.
- Turn product specs into measurements with limits, and review limit
  changes with design and quality in a pull request.
- Watch first pass yield per station and per phase in TofuPilot, find the
  phases that fail good units or pass bad ones, and fix them.
- Work with the fixture designer on pogo pin maps, relay matrices and
  the mechanical side of connecting a DUT.
- Write operator instructions into the sequence so a new operator can run
  a station on their first day.
- Keep every station in Git as text, so anyone on the team can read,
  review and rebuild it.

## Must have

You should be able to do these on day one, in any language:

- Read an instrument datasheet and get a measurement out of it over SCPI
  or serial.
- Write Python that handles a hardware timeout without crashing the run.
- Explain what a control limit is, what Cpk means, and why a limit that's
  too tight costs yield.
- Use Git for day-to-day work: branch, commit, open a pull request,
  review someone else's.
- Debug on the floor with an operator watching.

## Nice to have

- Experience with any test sequencer (TestStand, OpenHTF, pytest-based,
  in-house). We use the TofuPilot Framework and we'll teach it.
- LabVIEW experience. Our older stations are LabVIEW and we're moving
  them to Python; you'd help.
- Fixture design, PCB bring-up, or firmware experience.
- A regulated environment ([ISO 13485 / AS9100 / IATF 16949]).

## The stack

- Python 3, PyVISA for instruments
- TofuPilot Framework: stations are a procedure.yaml plus phases and
  plugs in Python, all in Git
- TofuPilot for results: per-unit run history, FPY, Cpk, control charts
- [Instrument list: Keysight DMMs, Rigol supplies, NI DAQ, ...]
- [Fixture vendor], [ERP / MES if any]

## Your first 90 days

- Week 1: run every station as an operator, read every procedure.yaml,
  ship one small limit change through a pull request.
- Month 1: own one station end to end, including its operator screens
  and its yield.
- Month 2: build a new plug for an instrument we don't have a driver for,
  and use it on a station.
- Month 3: rebuild one LabVIEW station in Python and run it side by side
  with the original for two weeks.

## Practical

- [Location / hybrid / on-site days], on the production floor [n] days a week
- Reports to [test lead / engineering manager]
- [Salary range], [benefits]
- Interview: a 30-minute call, a short Python take-home with a mock
  instrument, a 45-minute conversation about it
```

The "must have" section is the part to protect when someone edits this later. Each line is a task with a visible result. If a candidate can do all five, they can learn your sequencer in a week.

## Typical Posting Line vs Rewrite

The left column is what most postings say. The right column says the same thing as work.

| Typical LabVIEW posting line | Rewrite |
|---|---|
| 5+ years of LabVIEW development | Has built and maintained production test stations, in any language |
| CLD or CLA certification preferred | Can explain a station they built to someone who has never seen it |
| Expert in NI TestStand sequence development | Has written a test sequence with phases, limits and pass/fail logic |
| Experience with NI DAQ and PXI hardware | Has taken measurements from instruments over SCPI, serial or a DAQ API |
| Proficient in TDMS data logging | Makes every run land somewhere the team can query it by serial number |
| Develops custom reports in LabVIEW | Reads FPY and Cpk per phase and acts on them |
| Familiar with NI Vision or NI Motion | Has integrated a camera or a motion stage into a test, any vendor |
| Strong knowledge of the NI ecosystem | Reads a datasheet and gets the instrument talking in an afternoon |

Notice that the right column is harder to fake. "5+ years of LabVIEW" is a line on a CV; "has built a station" is something the take-home in [test engineer interview: a Python take-home](/guides/test-engineer-interview-a-python-take-home) checks in an hour.

## Where to Post and What to Say About LabVIEW Candidates

Post where Python engineers read. Your own careers page first, then the general engineering boards, the Python and hardware community boards, and the career centres of the engineering schools nearest your plant.

Local hardware and maker meetups reach people who already like instruments. A referral bonus for anyone who has done PCB bring-up pulls better than a recruiter for this role.

Say explicitly that LabVIEW candidates are welcome. Add a line like: "If your experience is in LabVIEW or TestStand, apply. You know the work, and Python is the easier half of the move."

They do reskill fast. The domain knowledge (limits, fixtures, yield, operators) is the hard part and they already have it, and [reskill a LabVIEW developer to Python](/guides/reskill-a-labview-developer-to-python) is the plan for their first month.

Don't post the LabVIEW version of the job "just in case" alongside this one. It signals that the Python posting is the second choice. Run one posting.

For the skills to screen against, use [Python skills checklist for test engineers](/guides/python-skills-checklist-for-test-engineers), and for what happens after the offer, [onboard a test engineer in one week](/guides/onboard-a-test-engineer-in-one-week).

## Levelling and Salary Ranges

Typical 2026 ranges for the US and Western Europe, fully loaded costs excluded. They're wide because the role is priced very differently across industries and regions.

| Level | What they own | US, base | Western Europe, base |
|---|---|---|---|
| Junior (0 to 2 years) | One station, with review | $70,000 to $95,000 | EUR 50,000 to 68,000 |
| Mid (2 to 6 years) | A line, including limits and yield | $90,000 to $130,000 | EUR 65,000 to 92,000 |
| Senior (6+ years) | All stations, the fixture roadmap, the migration | $120,000 to $165,000 | EUR 85,000 to 125,000 |

What moves a candidate within a range: regulated industry (medical, aerospace, automotive pay at the top), region (major US metros and Switzerland at the top), on-call or shift coverage, and fixture or firmware experience on top of the test work.

Two things to know when you set the number. A LabVIEW or TestStand developer of equal seniority typically costs well above these ranges, because the pool is small and skews senior. And the Python range buys you someone the rest of your engineering team can review and be reviewed by, which the LabVIEW premium doesn't.

For a junior hire, cut the "must have" list to the first three items and move Git to "we'll teach". For a senior hire, add "has migrated a line from one sequencer to another" and "has hired or mentored a test engineer" to the responsibilities.

## Start With One Station

The posting is stronger when the first 90 days point at a real station. Before you publish, pick the one with the highest volume or the one only one person can open, rebuild it in Python with the TofuPilot Framework, and run it side by side with the LabVIEW version on the same units for two weeks. Then the new hire's month-3 milestone is a repeat of something your team has already done once.

```bash filename="install-and-run.sh"
# Install the CLI, then run the station locally (no account needed) or with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Test Engineer Interview: A Python Take-Home
URL: https://www.tofupilot.com/guides/test-engineer-interview-a-python-take-home
Learn how to run a two-hour Python take-home that checks instrument control, limits and failure handling, with starter files, a grading grid and questions.

A test engineer take-home should prove four things in under two hours: the candidate can talk to an instrument, apply limits, handle a hardware failure, and ship a result somebody else can read. The exercise below does that with a mock power supply, a mock DMM that times out one call in ten, and the TofuPilot Framework as the sequencer. You get starter files, a grading grid and the questions for the follow-up conversation.

## What the Exercise Proves

A CV tells you which tools a candidate has used. The take-home tells you whether they can do the job, and it does it in an hour of their time and fifteen minutes of yours.

| Skill | How the exercise checks it |
|---|---|
| Talk to an instrument | Set a voltage on a supply plug, read it back through a DMM plug, both with a realistic method surface. |
| Apply limits | Declare the limits in `procedure.yaml`, not in the phase. Knowing where limits belong is the point. |
| Handle a hardware failure | The mock DMM raises a timeout one time in ten. The phase has to retry or fail cleanly, never hang or crash. |
| Ship a result | The station runs with `tofupilot run` and produces a run someone else can open. A strong candidate uploads it and sends the link. |

The instruments are mocks, so the candidate needs nothing but Python and the CLI. That's deliberate: you're hiring for the reasoning, and the real PyVISA call is a one-line swap they'll make in week one.

## The Exercise Brief

Send this as the README of a small repo along with the starter files below. Ask for a pull request or a zip within a week, and tell them it should take about two hours.

```markdown filename="take-home/README.md"
# Take-home: power rail phase

You have a mock power supply plug and a mock DMM plug. The DMM times out
one call in ten. Write a phase that:

1. Sets the supply to 5.0 V and enables the output.
2. Reads the voltage back through the DMM.
3. Records the reading as the measurement `rail_5v`, with limits declared
   in `procedure.yaml` (4.85 V to 5.15 V).
4. Handles a DMM timeout without crashing the run. Retrying is fine.
   Decide how many times, and say why in a comment.
5. Turns the supply output off at the end, even when something fails.

It should run with:

    curl -fsSL https://www.tofupilot.app/install | sh
    tofupilot run ./procedure.yaml

Optional: run it with `--upload` and send us the run link from
https://tofupilot.app. The Lab tier is free and takes two minutes to set up.

What we look at: where the limits live, how the timeout is handled, whether
the supply is always turned off, and whether the code would still be clear
to you in six months. We don't care about line count.

Files:
- procedure.yaml       (skeleton, fill in the phase and its measurement)
- phases/power_rail.py (stub)
- plugs/mock_psu.py    (done, don't edit)
- plugs/mock_dmm.py    (done, don't edit)
```

Keep the brief short. A long brief tests reading, not engineering.

## The Starter Files

Four files. The plugs are complete and the candidate shouldn't touch them. The procedure and the phase are stubs.

```python filename="take-home/plugs/mock_psu.py"
# Mock bench power supply. Same method surface as a real SCPI supply plug.
# STATE stands in for the wire between the supply and the DMM.
STATE = {"setpoint": 0.0, "output": False}


class PowerSupply:
    def __init__(self):
        STATE["setpoint"] = 0.0
        STATE["output"] = False

    def set_voltage(self, volts: float) -> None:
        STATE["setpoint"] = float(volts)

    def output(self, enabled: bool) -> None:
        STATE["output"] = bool(enabled)

    def close(self) -> None:
        STATE["output"] = False
```

```python filename="take-home/plugs/mock_dmm.py"
# Mock DMM that times out one call in N, like a flaky GPIB or LAN link.
import random

from plugs.mock_psu import STATE


class InstrumentTimeout(Exception):
    """Raised when the instrument doesn't answer in time."""


class Multimeter:
    TIMEOUT_ONE_IN = 10

    def __init__(self):
        # A real plug opens a VISA resource here. The mock reads the supply's wire.
        self._rng = random.Random()

    def read_voltage(self) -> float:
        if self._rng.randrange(self.TIMEOUT_ONE_IN) == 0:
            raise InstrumentTimeout("no reply from DMM within 2000 ms")
        volts = STATE["setpoint"] if STATE["output"] else 0.0
        return volts + self._rng.gauss(0.0, 0.01)

    def close(self) -> None:
        pass
```

```yaml filename="take-home/procedure.yaml"
# Skeleton. Add the phase under main and its measurement with limits.
name: Take-home power rail
version: 0.1.0

unit:
  serial_number:
    default_value: TH-0001
  part_number:
    default_value: TAKE-HOME-A

plugs:
  - name: psu
    python: plugs.mock_psu:PowerSupply
  - name: dmm
    python: plugs.mock_dmm:Multimeter

main:
  - name: Power Rail
    python: phases.power_rail
    measurements: []
```

```python filename="take-home/phases/power_rail.py"
# Stub. Set 5 V on the supply, read it back on the DMM, record rail_5v.
# Handle the DMM timeout. Always turn the supply off.


def power_rail(measurements, psu, dmm):
    raise NotImplementedError
```

The mock DMM reads the supply's `STATE` so a correct phase gets about 5 V back and a phase that forgot to enable the output gets 0 V and a failed limit. In a real station the DMM plug opens a VISA resource instead and has no idea a supply exists, which is exactly why the phase, not the plug, is the thing under test.

## The Grading Grid

Score each row before the conversation, from the code alone. Three "strong" rows and no "weak" ones is a hire at mid level.

| Criterion | Weak | Solid | Strong |
|---|---|---|---|
| Limits | Hardcoded in the phase, or a bare `if` with a print. | Declared in `procedure.yaml` under the measurement, phase only records the value. | Same, and the pull request description says where 4.85 and 5.15 came from and asks whether they're right. |
| Timeout handling | Uncaught, or a bare `except:` that swallows everything. | Catches `InstrumentTimeout`, retries a bounded number of times, fails the phase after that. | Bounded retry with a short wait, the retry count is a named constant with a comment on why, and the failure message says what the operator should check. |
| Cleanup | Supply left on when the phase fails. | `try/finally` turns the output off. | Same, and the phase doesn't reach into the plug's private state. |
| Runs | Doesn't run, or needs edits to run. | `tofupilot run ./procedure.yaml` passes and fails as expected. | Ran it with `--upload` and sent the run link from TofuPilot. |
| Readability | One function that does everything, no names. | Small functions, clear names, one comment where it earns its place. | You could hand it to a junior and they'd understand it without asking. |
| Communication | Zip file, no note. | Short note on what they did and what they'd do next. | Note flags a real gap in the brief (settling time before the read, what "one in ten" means for FPY). |

A "weak" in timeout handling or cleanup is the one to weigh most. Those two rows are what a station does at 2 a.m. when nobody is watching.

## The 45-Minute Follow-Up

Walk through their code for ten minutes, then spend the rest on these. There are no right answers; you're listening for how they reason about a floor.

**"What would you change to test 4 DUTs at once?"** Solid: split the phase per slot, one supply channel per DUT, keep the limits shared. Strong: asks whether the DMM is switched through a relay matrix or there are four DMMs, and points out the timeout now costs four units of throughput, not one.

**"Where do the 4.85 and 5.15 come from?"** Solid: the regulator datasheet plus design margin. Strong: asks what the downstream load tolerates, whether the number is the same at cold, and who signs off when the limit changes. Points out that the limit lives in `procedure.yaml` so the change is a reviewable diff.

**"Cpk on rail_5v drops from 1.6 to 1.1 over a month. What do you do?"** Solid: open the control chart in TofuPilot, look at whether the mean drifted or the spread grew, check the DMM calibration date. Strong: asks whether it's one station or all of them, and whether the fixture contact resistance changed before blaming the boards.

**"The DMM times out one in ten. What's that doing to FPY?"** Solid: with a retry it's doing nothing to FPY but something to cycle time. Strong: says a flaky link should be an error outcome, not a fail, so it doesn't show up as a yield loss, and asks how the platform distinguishes the two.

**"An operator says the station passed a board that came back from the customer dead. Where do you start?"** Solid: pull the run for that serial number and read the measurements. Strong: compares it to the population for that phase, checks whether it passed near a limit, and asks what the test doesn't cover.

**"You inherit a LabVIEW station nobody can open. What's your first week?"** Listen for: run it as an operator first, find the limits, get the results off the machine, then decide about a rewrite. Bonus if they've read [onboard a test engineer in one week](/guides/onboard-a-test-engineer-in-one-week) or say something close to it.

Score the conversation against [Python skills checklist for test engineers](/guides/python-skills-checklist-for-test-engineers) so two interviewers rate the same way.

## What Not to Test For

Don't test for a LabVIEW certification. CLAD, CLD and CLA prove someone passed an NI exam, and the exam covers the tool, not the floor. A candidate with a CLD who can't explain a timeout retry is a weaker hire than one without it who can.

Don't test for NI product names. Whether someone has used a PXI chassis or knows what TDMS stands for says nothing about whether they can bring up a fixture. Ask how they'd get a measurement out of an instrument they've never seen, and let them pick the vendor.

Don't test for your sequencer. If a candidate has only used OpenHTF, pytest or an in-house runner, the TofuPilot Framework is a day of reading. The brief tells them the structure; the exercise checks whether they use it well.

Don't add a whiteboard algorithm round. A test engineer sorts by yield, not by comparison count.

The posting that produces candidates for this exercise is in [test engineer job description for Python hires](/guides/test-engineer-job-description-for-python-hires).

## Start With One Station

Before you send the take-home, run it yourself. Better, build it from a real station: pick the one with the highest volume or the one only one person can open, rebuild it in Python with the TofuPilot Framework, run it side by side with the LabVIEW version on the same units for two weeks, and cut the take-home from one of its phases. Candidates then solve a problem that exists on your floor, and the follow-up questions write themselves.

```bash filename="install-and-run.sh"
# Install the CLI, then run the station locally (no account needed) or with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild is in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Document a Test Station for Hand-Over
URL: https://www.tofupilot.com/guides/document-a-test-station-for-hand-over
Learn how to document a test station so anyone on the team can rebuild it and change a limit without its author, with a checklist and a README template.

A test station is documented when anyone on the team can rebuild it on a new PC and change a limit without calling the person who built it. That's the whole test, and most stations fail it. This guide gives the hand-over checklist, a README template, and the reason a text-based station layout does most of the work for you.

## The Goal

Two questions decide whether a station is handed over. Can a colleague rebuild it on the spare PC from what's in the repo? Can they change one limit, release it, and prove afterwards which units ran the old version and which ran the new?

Everything else is nice to have. A wiring photo helps. A 40-page validation report helps less than the two answers above, because nobody opens it at 2 a.m. when the fixture is down.

[Bus factor](/guides/bus-factor-in-test-engineering-a-guide) is the score across the plant. This guide is how you raise it for one station, before or after [the person who built it leaves](/guides/what-to-do-when-your-labview-engineer-leaves).

## The Hand-Over Checklist

Work down the table. "Where it lives" is one place, not "the share drive and also Marc's laptop".

| Item | Where it lives | Done when |
|---|---|---|
| Sequence of phases and their order | `procedure.yaml`, under `main:` | A colleague reads the list and can say what the station does without opening a `.py` file |
| Limits and units per measurement | `procedure.yaml`, under `validators:` | Every limit has a comment naming its source (spec section, datasheet table, or the engineer and the date) |
| Instrument list, VISA addresses, firmware | `plugs/*.py` defaults, listed again in `README.md` | `*IDN?` from each plug matches the README line |
| Wiring | `docs/wiring.pdf` and a bench photo in `docs/`, referenced from the README | The spare PC can be cabled from the picture alone |
| Fixture | Drawing number, revision, pogo pin map in `docs/` | The fixture on the bench carries the same revision as the README |
| Station PC setup | `README.md`, "How to run" | A fresh Windows install reaches a green run from the README, no phone call |
| Release process | `README.md`, "How to release", plus Git tags | The tag on the station PC matches `version:` in `procedure.yaml` |
| Results history | TofuPilot, per unit and per station | Anyone can open a serial number and see every run with the procedure version that produced it |
| Calibration dates | `README.md`, one table | Every next-due date is in the future |
| Known failure modes | `README.md`, one table | The top three failures from the TofuPilot Pareto are listed with their usual cause |

## Station README Template

Copy this into the station repo. Fill it in one sitting; a half-finished README ages badly.

```markdown filename="README.md"
# FCT Station 3, PCBA-100

Owner: test engineering (the team, not a person). Last reviewed: 2026-09-15.

## Wiring

See `docs/wiring.pdf`, revision C. Bench photo in `docs/bench.jpg`.

| From | To | Cable |
|---|---|---|
| DMM HI / LO | Fixture J3 pins 1, 2 | 1 m banana, red / black |
| PSU CH1 | Fixture J1 | 18 AWG, 0.5 m |
| Station PC | DUT console | USB to UART, COM3 |

## Instruments

| Instrument | VISA address | Firmware | Calibration due |
|---|---|---|---|
| Keysight 34465A DMM | TCPIP::192.168.1.100::INSTR | A.03.03 | 2027-02-01 |
| Keysight E36313A PSU | TCPIP::192.168.1.101::INSTR | 1.0.6 | 2027-02-01 |
| DUT console | ASRL3::INSTR (COM3, 115200 8N1) | n/a | n/a |

## Fixture

Drawing FIX-0042 rev B. Pogo pin map in `docs/FIX-0042-pinmap.pdf`.
Spare pogo pins: drawer 2, part P-0910.

## Limits

All limits live in `procedure.yaml`. Each one carries a comment with its
source. Don't change a limit without changing the comment.

| Measurement | Limit | Source |
|---|---|---|
| rail_3v3 | 3.2 to 3.4 V | PCBA-100 spec rev 4, section 5.1 |
| idle_current | <= 45 mA | MCU datasheet table 12, plus 20 % margin |

## How to run

    curl -fsSL https://www.tofupilot.app/install | sh
    git clone <repo url> && cd fct-station-3
    tofupilot run ./procedure.yaml --upload

Runs land in TofuPilot under the procedure "PCBA-100 FCT", one page per unit.

## How to release

1. Branch, change, open a pull request. One reviewer from test engineering.
2. Bump `version:` in `procedure.yaml`.
3. Merge, tag `vX.Y.Z`, pull the tag on the station PC.
4. Run three golden units. Compare their runs in TofuPilot with the previous tag.

## Known failure modes

| Symptom | Usual cause | Fix |
|---|---|---|
| rail_3v3 reads 0.00 V | Pogo pin 7 worn | Replace, part P-0910 |
| DMM timeout on the first unit of the day | DMM in standby | Power cycle the DMM |
```

The README is short on purpose. Four other things carry the rest.

## Why the Framework Layout Does Most of This

A TofuPilot Framework station is a folder: `procedure.yaml`, `phases/*.py`, `plugs/*.py`. All text, all under Git. Each part carries exactly one kind of information, so the README only has to cover what none of them can.

| Carrier | What it holds | What that replaces |
|---|---|---|
| `procedure.yaml` | The sequence, the unit fields, every measurement with its unit and limits | The test spec spreadsheet and the "which step runs first" question |
| `plugs/*.py` | VISA addresses, SCPI strings, timeouts, `close()` | The instrument list scribbled on the fixture |
| Git | Who changed which limit, when, reviewed by whom; `git blame` on the line | The change log tab nobody updates |
| TofuPilot | Every run per unit and per station, with the procedure version, FPY, Cpk, Pareto | The TDMS folder on the station's local disk |

What's left for the README is physical: wiring, fixture, calibration, the failure modes an operator sees. That's why the template above fits on one screen.

## procedure.yaml as Living Documentation

Treat the procedure as the spec. A reviewer who can't read Python can still read this file, and a limit change is a one-line diff with a comment.

```yaml filename="procedure.yaml"
# The procedure is the spec. Every limit names its source in a comment.
name: PCBA-100 FCT
version: 2.3.0

unit:
  serial_number:
    default_value: "SN-000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter       # 34465A at 192.168.1.100
  - name: psu
    python: plugs.psu:PowerSupply      # E36313A at 192.168.1.101

main:
  - key: power_rails
    name: Power Rails
    python: phases.power_rails
    measurements:
      - name: rail_3v3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2        # PCBA-100 spec rev 4, section 5.1
          - operator: "<="
            expected_value: 3.4        # same

  - key: idle_current
    name: Idle Current
    python: phases.idle_current
    depends_on: [power_rails]
    measurements:
      - name: idle_current
        unit: mA
        validators:
          - operator: "<="
            expected_value: 45         # MCU datasheet table 12, plus 20 % margin
```

When quality asks "why 45 mA", the answer is on the line. When they ask "since when", `git log -p procedure.yaml` answers. When they ask "which units shipped with 50 mA", the run pages in TofuPilot carry the procedure version, so the answer is a filter, not an archaeology project.

## What a LabVIEW Project Makes Hard Here

None of this is impossible in LabVIEW. It's more work, for three concrete reasons.

VIs are binary. A diff needs NI's compare tool and a licensed seat, so reviews happen by opening both versions side by side, and `git blame` on a limit isn't available. Most teams skip the review.

Limits tend to live inside VIs, or in an Excel file next to the sequence. Either way the spec and the code drift, and the comment naming the datasheet section has nowhere natural to go.

Results tend to land in TDMS or CSV on the station's local disk. Per-unit history means finding the right file on the right PC. When the PC is replaced, the history usually isn't.

A LabVIEW station can still meet the checklist. Someone has to write the README by hand, keep the limits table in sync by hand, and copy the results somewhere shared by hand. The Python layout makes those three automatic, which is the difference between a hand-over that happens and one that's planned.

## A Five-Minute Review Before Every Release

Run this before tagging. Five minutes, every time, by the reviewer and not the author.

1. Open the pull request diff. Every changed limit has a changed comment on the same line.
2. `version:` in `procedure.yaml` went up.
3. The README instrument table still matches `*IDN?` from each plug. Run them.
4. Three golden units passed on the branch, and their runs in TofuPilot sit within the previous version's histograms.
5. The tag name matches `version:`. Pull it on the station PC, not a copy of the folder.

If step 4 fails, the change is real and needs a conversation with quality before it ships. That's the review doing its job.

## Start With One Station

Pick the station with the highest volume, or the one only one person can open. Rebuild it in Python with the TofuPilot Framework, then run it side by side with the LabVIEW version on the same units for two weeks. Write the README on day one and keep it true; the [onboarding guide](/guides/onboard-a-test-engineer-in-one-week) uses that README as the new engineer's first task.

```bash filename="station-setup.sh"
# Install the CLI, run a procedure locally, then run it with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The [LabVIEW migration guide](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot) covers the rebuild step by step, and the [Framework page](https://tofupilot.com/products/framework) covers the layout. The Lab tier is free, and `tofupilot run` works with no account.


### Python Skills Checklist for Test Engineers
URL: https://www.tofupilot.com/guides/python-skills-checklist-for-test-engineers
Learn the five Python skills a test engineer needs on a production floor, how to verify each in an interview or in week one, and what to leave off the list.

"Knows Python" on a test floor means five specific things: talk to an instrument, survive a hardware failure without crashing the station, run the code on a station PC that isn't a laptop, use Git for changes and releases, and get a run with measurements and limits into TofuPilot. Everything else is optional. This checklist lists the five, what each looks like at the bench, and how to verify it in an interview or in week one.

## The Five Skills

Print the table. Tick it in the interview, then tick it again on the Friday of week one. A candidate who's honest about a gap is more useful than one who claims all five.

| Skill | What it looks like in practice | How to verify |
|---|---|---|
| Instrument I/O (PyVISA, SCPI, serial) | Opens a resource by VISA address, sends `*IDN?`, parses the reply into a float. Knows `write`, `query` and `read` are three different things. Has used `pyserial` for a DUT console at 115200 baud. | Interview: give them a DMM address and ask for a DC voltage read on the whiteboard. Week one: they write a plug for one bench instrument and it returns a number. |
| Exceptions around hardware | Sets a timeout on every resource. Retries a flaky query a bounded number of times. Closes the instrument in `close()` even when the phase failed. Reports a `VisaIOError` as ERROR, not FAIL. | Interview: ask what the station does when the DMM cable comes out mid-run. Week one: pull the cable during their phase and watch. |
| Packaging and environments | Creates a venv, pins versions in `requirements.txt`, installs on a clean Windows station PC without editing `PATH` by hand. Knows "works on my laptop" isn't done. | Interview: ask how they'd get code onto a station with no internet. Week one: they install their station on the spare PC from the README alone. |
| Git basics | Branches for a change, opens a pull request, gets a review, tags a release (`v1.4.0`) and can say which tag runs on which station. | Interview: ask about the last thing they reverted and how. Week one: their first limit change lands as a reviewed pull request, not a file copied over the network share. |
| Data flow | Declares measurements with units and validators in `procedure.yaml`, runs with `--upload`, opens the run page for that unit in TofuPilot, reads FPY and Cpk on the procedure page instead of building a spreadsheet. | Interview: ask what they'd open first when yield drops on a Monday. Week one: they find their own runs in TofuPilot and explain one Cpk value to you. |

Three of the five are ordinary software hygiene. The two that separate a test engineer from a generalist developer are instrument I/O and hardware exceptions. Hire for those two and teach the other three during [the first week](/guides/onboard-a-test-engineer-in-one-week).

### Instrument I/O is the one to probe

Every other skill can be learned from a README. Instrument I/O needs a bench, an instrument that lies, and someone who's been bitten before. Ask about termination characters, about a query that returned the previous answer, about the difference between a GPIB timeout and a socket timeout.

The good candidates have a story for each. The [PyVISA docs](https://pyvisa.readthedocs.io) cover the mechanics, and a candidate who's read them will say so.

### Exceptions are where stations die

A phase that raises and leaves the DMM half-configured costs you the next unit as well as this one. Ask for the three-part answer: timeout on open, bounded retry on read, cleanup on close.

A bare `except` that swallows the error is how a station reports PASS on a disconnected cable. Listen for whether the candidate knows the difference between a unit that failed and a station that errored.

## What They Don't Need

A job description that lists everything Python can do filters out the people you want. The [job description guide](/guides/test-engineer-job-description-for-python-hires) covers the posting itself. This is the short list of what to leave off.

| Not needed | Why |
|---|---|
| GUI frameworks (Tkinter, Qt, a web front end) | The operator UI is declared in the procedure: text inputs, checklists, images, sliders, progress. No frontend code on a station. |
| Building a sequencer | The framework runs phases in order, injects plugs, handles `depends_on` and multi-DUT slots. A home-made runner is the most common way to lose six months. |
| Building dashboards | FPY, Cpk, control charts, histograms and failure Pareto are in TofuPilot. A candidate who proposes a pandas notebook for yield is solving a problem you don't have. |
| LabVIEW | Useful for reading the old station during a migration, not a hiring requirement. Your LabVIEW engineer walks them through the block diagram once. |
| Advanced Python (async, metaclasses, C extensions) | A station is a folder of plain functions and plain classes. If it needs asyncio, the fixture design is the problem. |

Every plant has one home-made yield dashboard that only opens on one PC, and the person who built it is on holiday.

## Self-Assessment

Use this for a candidate after the [take-home](/guides/test-engineer-interview-a-python-take-home), or for a current engineer planning their own [move from LabVIEW](/guides/reskill-a-labview-developer-to-python). Level 2 across the board is a working station owner. Level 3 in one or two rows is a lead.

| Skill | Level 1 | Level 2 | Level 3 |
|---|---|---|---|
| Instrument I/O | Runs an existing plug and reads the value | Writes a new plug from the SCPI manual, handles termination and timeouts | Debugs a bus problem with a protocol trace, drives a multi-instrument fixture |
| Hardware exceptions | Catches the exception and logs it | Bounded retries, cleanup in `close()`, ERROR and FAIL reported correctly | Designs the recovery path: power cycle the fixture, re-init, resume at the right phase |
| Packaging | Runs from a venv on their own machine | Installs from `requirements.txt` on a station PC offline | Pins the Python version, scripts the station install, keeps a known-good image |
| Git | Commits to `main` | Branch, pull request, review, tag | Reviews others, bisects a regression, keeps one repo per station clean |
| Data flow | Uploads a run and finds it in TofuPilot | Declares limits in `procedure.yaml`, reads FPY and Cpk per procedure | Reads a drift on the control chart, ties it to a fixture or instrument, files the fix as a pull request |

A LabVIEW developer moving to Python usually scores 3 on the ideas behind instrument I/O and exceptions from day one, and 1 on packaging and Git. That's a shorter gap than most managers expect.

## Two Blocks Worth Reading

Both are the size of the real thing. If a candidate can read these and say what breaks when you remove any line, they pass the first two rows of the checklist.

A plug with a timeout, a bounded retry and cleanup in `close()`:

```python filename="plugs/dmm.py"
# Multimeter plug: timeout on open, bounded retry on read, cleanup in close().
import pyvisa


class Multimeter:
    def __init__(self, address="TCPIP::192.168.1.100::INSTR", retries=3):
        self.rm = pyvisa.ResourceManager("@py")
        self.inst = self.rm.open_resource(address)
        self.inst.timeout = 2000  # milliseconds
        self.retries = retries

    def read_voltage(self):
        last_error = None
        for _ in range(self.retries):
            try:
                return float(self.inst.query(":MEAS:VOLT:DC?"))
            except pyvisa.VisaIOError as error:
                last_error = error
        raise last_error

    def close(self):
        try:
            self.inst.write("*RST")
        finally:
            self.inst.close()
            self.rm.close()
```

The phase that uses it is two lines: `def power_rails(measurements, dmm): measurements.rail_3v3 = dmm.read_voltage()`. The limits don't live in the phase. They live here:

```yaml filename="procedure.yaml"
# One phase, one measurement. Limits live in the procedure, not in the code.
name: PCBA Functional Test
version: 1.4.0

unit:
  serial_number:
    default_value: "SN-000001"
  part_number:
    default_value: "PCBA-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: rail_3v3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
```

Run it with `--upload` and the measurement, its limits and its outcome land on the unit's run page in TofuPilot. The Cpk for `rail_3v3` shows up on the procedure page once there are enough runs. Nobody writes that part.

## Start With One Station

Pick the station with the highest volume, or the one only one person can open. Rebuild it in Python with the TofuPilot Framework, then run it side by side with the LabVIEW version on the same units for two weeks. The five skills above are exactly what that rebuild exercises, in order.

```bash filename="station-setup.sh"
# Install the CLI, run a procedure locally, then run it with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The [LabVIEW migration guide](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot) covers the rebuild step by step, and the [Framework page](https://tofupilot.com/products/framework) covers the layout. The Lab tier is free, and `tofupilot run` works with no account.


### Why Is It So Hard to Hire LabVIEW Engineers?
URL: https://www.tofupilot.com/guides/why-is-it-so-hard-to-hire-labview-engineers
Learn why LabVIEW engineers are scarce, what the shortage costs a plant on the floor, and why a Python hire on a test framework is the practical answer.

Hiring a LabVIEW engineer is hard because the supply of new LabVIEW developers shrank over the last decade while the number of stations that depend on them didn't. Universities moved their core courses to Python, NI put certification and the language itself on paid tracks, and the engineers who remain are senior, well paid and mostly already employed. You're competing for a pool that gets smaller every year, for a tool that still runs a lot of production test.

## Where the Pipeline Broke

Four things happened at once. None of them is reversing.

### Universities moved to Python

Fewer engineering programs teach LabVIEW in a core course than a decade ago. Most teach Python instead, often next to MATLAB, because it's free, it runs everywhere and it's what a graduate's first employer expects. A new graduate who has never opened a block diagram is now the default, not the exception.

### Certification is a paid track

NI's certification ladder (CLAD, CLD and CLA for LabVIEW, CTD and CTA for TestStand) is a real signal of competence. It's also an exam fee, a training budget and weeks of study, on top of a seat licence to practise on. The certified pool grows only as fast as employers pay for it.

### The language moved to subscription

Since 2023 LabVIEW is sold as an annual subscription: roughly $1,300 per year for Base, $4,300 for Full and $5,500 or more for Professional, per seat. NI has been part of Emerson since the same year. The Community Edition is free but non-commercial only, so nobody can legally build a paid skill on it. That closed the last cheap on-ramp. The full cost picture is in [LabVIEW Pricing: What It Actually Costs](/guides/labview-pricing-what-it-actually-costs).

### The pool skews senior

The engineers who learned LabVIEW when it was taught everywhere are now fifteen to twenty-five years into their careers. They're good, they're expensive and most of them already have a job they like. A senior LabVIEW or TestStand developer typically earns well above a generalist Python engineer of the same seniority, in a typical 2026 range for the US and Western Europe that runs from the high five figures well into six figures in USD. Industry (medical and aerospace pay more), region and TestStand experience move it.

Put the four together and the shape is clear. Fewer people enter, entry costs more than it did, and the people already inside are ageing out of hands-on work into management or retirement. There are fewer LabVIEW job postings than a decade ago, and the ones that exist stay open longer. LabVIEW is still a capable tool. The labor pool around it is what changed.

## What It Costs a Plant in Practice

The hiring problem doesn't show up on a job board. It shows up on the floor.

| Symptom | What it looks like on the floor | Cost driver |
|---|---|---|
| Single point of failure | One engineer can open every VI. When they're out, nobody changes a limit. | Line stops, or ships against a limit everyone knows is wrong |
| Slow change requests | A new part number needs a new sequence. The request goes to an integrator and comes back in days to weeks. | Integrator day rate in the four figures, plus the wait |
| No results history | Results are TDMS files on the station PC, or a spreadsheet someone exports on Fridays. | Yield problems found late, sometimes by the customer |
| Frozen runtime versions | Stations stay on LabVIEW 2019 because upgrading means retesting everything. | Old OS, old drivers, PCs nobody can replace |
| Hiring lag | A LabVIEW posting stays open for months. The backfill starts senior and expensive. | Recruiter fees, overtime for the engineer who stayed |
| Knowledge in one head | Fixture quirks, instrument workarounds and calibration steps live in someone's memory, not in a repo. | Every hand-over restarts from zero |

The one engineer who can open every VI is also the one who takes three weeks off in August.

The change-request row is the one that compounds. Adding a limit to a LabVIEW sequence through an integrator is typically a day of billed time at a four-figure rate, plus days to weeks of waiting for a slot. A plant that introduces a new part number every month pays that twelve times a year, and the wait is usually worse than the invoice. In the meantime the line runs the old sequence, or a technician edits a limits file nobody reviews.

If that engineer has already handed in notice, [What to Do When Your LabVIEW Engineer Leaves](/guides/what-to-do-when-your-labview-engineer-leaves) is the 48-hour version of this guide. If you're weighing a contractor against a hire, [LabVIEW Contractor vs In-House Python Costs](/guides/labview-contractor-vs-in-house-python-costs) runs the numbers.

## Why "Hire a Python Engineer" Is the Real Answer

The scarce skill in a LabVIEW hire isn't instruments or limits. It's the vendor language wrapped around them. A LabVIEW station carries its own sequencer, its own front panel, its own limit checking and its own report writer, all built by hand in G, and that's what takes a candidate years to learn.

When the station runs on the TofuPilot Framework, the framework provides those four things. The sequencer is the phase list in `procedure.yaml`. Limits are validators in the same file. The operator UI is declared, not coded. Results land in TofuPilot, where FPY, Cpk and the failure Pareto are already charts.

What's left for the hire is Python and instruments. That's a much larger pool, and a cheaper one.

| What a LabVIEW hire must know | What a Python hire must know |
|---|---|
| G dataflow, block diagrams, wiring conventions | Functions, modules, virtual environments |
| Building a state-machine sequencer per station | Editing a phase list in `procedure.yaml` |
| Front panel design and event structures | Nothing. The operator UI is declared in YAML |
| Error clusters and error wiring | Exceptions |
| TDMS, report VIs, exporting to a database | Nothing. Runs upload to TofuPilot |
| DAQmx, NI-VISA, instrument driver VIs | PyVISA, `nidaqmx`, SCPI |
| Which LabVIEW and TestStand versions the plant is on | Git and pip |
| Fixtures, DUT handling, limits, SCPI | Fixtures, DUT handling, limits, SCPI |

The last row is the point. Product knowledge transfers. Only the language layer doesn't, and the language layer is the thin part. If you already have a LabVIEW developer, [Reskill a LabVIEW Developer to Python](/guides/reskill-a-labview-developer-to-python) covers the six-week version.

It also changes what you post. A "LabVIEW test engineer" role draws from a few hundred people in your region, most of them not looking. A "test engineer, Python" role draws from every embedded, automation and data engineer who has ever talked to an instrument, and the interview becomes a take-home with a DMM instead of a certification check. The ramp-up is shorter too: a Python engineer with no test background typically owns a station in weeks, not months, when the sequencer, the limits and the results platform are already there.

## How Small a Station Phase Is in Python

Here's a power-rail check on a PCBA fixture. The measurement names and limits live in the procedure file. The phase reads two voltages and assigns them.

```yaml filename="procedure.yaml"
# Power rail check: two measurements with limits, one plug, one phase
name: Controller Board FCT
version: 1.0.0

unit:
  serial_number:
    default_value: "SN-0001"
  part_number:
    default_value: "CTRL-100"

plugs:
  - name: dmm
    python: plugs.dmm:Multimeter

main:
  - name: Power Rails
    python: phases.power_rails
    measurements:
      - name: rail_3v3
        unit: V
        validators:
          - operator: ">="
            expected_value: 3.2
          - operator: "<="
            expected_value: 3.4
      - name: rail_5v
        unit: V
        validators:
          - operator: ">="
            expected_value: 4.8
          - operator: "<="
            expected_value: 5.2
```

```python filename="phases/power_rails.py"
# The framework injects the dmm plug and validates both values against procedure.yaml
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage(101)  # mux channel 101 is the 3V3 rail
    measurements.rail_5v = dmm.read_voltage(102)
```

That's the whole phase. No sequencer code, no limit comparison, no front panel, no report writer. A Python engineer with no test background reads it in a minute, and a candidate who has spent a week with PyVISA writes the plug behind it in an afternoon.

The same station in LabVIEW is a project, a state machine, a front panel with a limits table, an error cluster running through every SubVI and a TDMS writer. All of that is what you're hiring for when you post a LabVIEW role, and none of it is about the product.

There's a second effect that matters more than the hiring one. The Python station is three text files in a Git repository. Anyone on the team can read the limits in `procedure.yaml`, review a change in a pull request and see every run in TofuPilot. The station stops being the property of whoever built it, which is the failure mode the whole hiring problem grew out of.

## Start With One Station

Pick the station with the highest volume, or the one only one person can open. Rebuild it in Python with the TofuPilot Framework and run it side by side with the LabVIEW version on the same units for two weeks. Compare the two result sets in TofuPilot before you retire anything.

```bash filename="install-and-run.sh"
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
```

The step-by-step is in [How to Migrate from LabVIEW to Python](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at https://tofupilot.com/products/framework. The Lab tier is free, and `tofupilot run` works with no account.


### LabVIEW Contractor vs In-House Python Costs
URL: https://www.tofupilot.com/guides/labview-contractor-vs-in-house-python-costs
Learn the three-year cost of a LabVIEW integrator against an in-house Python engineer with TofuPilot, with a worked 5-station example and an editable model.

Over three years, a 5-station line costs roughly the same in cash whether an NI Alliance Partner builds it in LabVIEW and TestStand or an in-house engineer builds it in Python with TofuPilot. The difference is where the money goes: licences and day rates on one side, a salary on the other. What separates the two options is everything that isn't cash, and that's where this model spends most of its attention.

## The Two Options, Stated Fairly

Option A is LabVIEW plus an integrator. You keep one or two LabVIEW seats and a TestStand development seat, pay a deployment licence per station, and contract a certified integrator to build the stations and handle change requests. Someone on your team coordinates the work, writes the requirements and accepts the deliverables.

Option B is an in-house Python engineer plus TofuPilot. You hire (or reassign) an engineer who writes the stations as text with the TofuPilot Framework, keeps them in Git, and pushes results to TofuPilot. There are no seat licences, the CLI is free, and the Lab tier of the platform is free.

Both options produce working stations. Option A gets you a specialist who has built a hundred of these and won't be surprised by your fixture. Option B gets you someone who is on the floor when the yield drops.

Neither is wrong, and the rest of this guide is about pricing the trade.

## The Cost Lines

Figures are typical 2026 ranges for the US and Western Europe. Licence prices are from [LabVIEW pricing: what it actually costs](/guides/labview-pricing-what-it-actually-costs). Salaries and day rates move with seniority, industry (medical and aerospace pay more) and region.

| Line | LabVIEW + integrator | In-house Python + TofuPilot |
|---|---|---|
| LabVIEW seat licence | $4,300 to $5,500 per seat per year (Full to Professional) | $0 |
| TestStand development seat | $2,500 to $3,000 per year | $0 |
| TestStand deployment | $1,000 to $1,500 per station per year | $0 |
| Integrator day rate | $1,000 to $2,000 per day, certified NI Alliance Partner | not needed |
| Station build | 10 to 20 integrator days per station | engineer time, included in salary |
| Change request turnaround | days to weeks, billed per day | same day to a few days, no invoice |
| Engineer salary | none, but 0.1 to 0.2 FTE of coordination | $95,000 to $175,000 per year fully loaded, 0.5 to 1.0 FTE on test |
| Ramp-up | integrator is productive from day one | 4 to 8 weeks before the engineer owns a station |
| Data platform | custom TDMS or CSV reports, 10 to 30 integrator days over three years | TofuPilot Lab, free; paid tiers exist for larger teams |

Two lines deserve a note. The Python salary range is wide on purpose: a mid-level engineer in a Western European plant sits near the bottom, a senior engineer at a US medical device company sits near the top. And a LabVIEW or TestStand developer of equal seniority typically costs well above that range, which is why Option A assumes you contract rather than hire.

## Worked Example: 5 Stations Over 3 Years

Ranges, not point values. The low end assumes a simple product, a local integrator and a mid-level engineer. The high end assumes a fixture-heavy product, a senior integrator and a senior engineer in an expensive region.

| Cost over 3 years | LabVIEW + integrator | In-house Python + TofuPilot |
|---|---|---|
| Seat licences (1 to 2 LabVIEW, 1 TestStand dev) | $20,000 to $42,000 | $0 |
| TestStand deployment (5 stations) | $15,000 to $22,500 | $0 |
| Station builds (5 stations) | $50,000 to $200,000 | included in salary |
| Change requests (10 to 25 days per year) | $30,000 to $150,000 | included in salary |
| Reports and data platform | $11,000 to $60,000 | $0 |
| Coordination (0.1 to 0.2 FTE) | $30,000 to $90,000 | included in salary |
| Engineer (0.5 to 1.0 FTE) | none | $142,500 to $525,000 |
| Ramp-up | none | $10,000 to $25,000 |
| Total | about $155,000 to $565,000 | about $150,000 to $550,000 |

The totals overlap almost completely. At 5 stations, cash doesn't decide this. What does is that Option A's largest lines (builds and change requests) scale with the station count and Option B's largest line (the salary) doesn't.

Run the same model at 2 stations and the integrator wins on cash. Run it at 10 and the engineer wins by a wide margin.

The other thing the table hides is what the engineer does with the rest of the FTE. Yield investigations, fixture repairs, operator training and instrument calibration don't show up as integrator days because nobody bills them. They still have to happen.

## The Non-Cash Costs

Nobody has ever put "waited eleven days for a limit change" in a budget line, and yet that's the number a test lead remembers.

**Turnaround on a limit change.** With an integrator, a limit change is a change request: write it up, get a quote, wait for a slot, receive a build, test it, deploy it. Days to weeks, and the line runs on the old limit meanwhile. With a station in text, the same change is one line in `procedure.yaml`, a pull request and a redeploy, the same day.

**Who can review.** A change to a `.vi` is reviewed by whoever has LabVIEW open, which on most floors is the integrator who wrote it. A change to a Python phase is reviewed by anyone on the team who reads Python, including the firmware engineer and the quality lead. The review is in the diff, and Git records who approved what.

**What happens on departure.** When the integrator relationship ends, you own binaries, a contract and a support ticket queue. Finding another NI Alliance Partner to take over someone else's VIs is possible and slow. When the in-house engineer leaves, you own a Git repository any Python engineer can open, and the results are already in TofuPilot where the team has been reading them all along.

The bus factor arithmetic is in [bus factor in test engineering](/guides/bus-factor-in-test-engineering-a-guide).

**Where the data lives.** Option A's reports are TDMS or CSV files on the station and a report VI that formats them. Option B's runs land in TofuPilot with FPY, Cpk, control charts and per-unit history on a page anyone can open. That gap is hard to price and easy to feel the first time a customer asks for the test record of a specific serial number.

## When the Contractor Route Is Still Right

Three situations where Option A is the better answer, and you should take it without apology.

**One station.** A single station is 10 to 20 integrator days. Hiring for that is absurd, and reassigning an engineer who has never built a station costs more in ramp-up than the build. Take the integrator, keep the source, and push results to TofuPilot through the REST API so the data isn't stranded.

**Heavy NI DAQ signal processing.** If the station is mostly synchronised multi-channel acquisition on NI hardware with FPGA timing, LabVIEW is the native tool and the integrator has done it before. Python can drive the same hardware, but you'd be paying your engineer to rediscover what the integrator already knows.

**Regulated support contracts.** Some medical and aerospace customers require a named support vendor with a signed SLA for the test equipment. An integrator provides that on paper. An in-house engineer doesn't, even if the response time is better.

In each case, the honest comparison is in [Python vs LabVIEW for manufacturing test](/guides/python-vs-labview-for-manufacturing-test-a-practical-comparison), and the hiring reality behind Option A's assumptions is in [why LabVIEW engineers are hard to hire](/guides/why-is-it-so-hard-to-hire-labview-engineers).

## A Cost Model You Can Edit

The worked example above comes from this script. Every input is a (low, high) pair, and the two totals print as ranges. Change the station count first, that's the input that moves the answer.

```python filename="cost_model.py"
# Three-year cost model for one test line. Edit Inputs and run it.
from dataclasses import dataclass

YEARS = 3


@dataclass
class Inputs:
    stations: int = 5

    # Option A: LabVIEW + integrator
    labview_seats: tuple = (1, 2)
    labview_seat_per_year: tuple = (4_300, 5_500)
    teststand_dev_seat_per_year: tuple = (2_500, 3_000)
    teststand_deploy_per_station_year: tuple = (1_000, 1_500)
    integrator_day_rate: tuple = (1_000, 2_000)
    build_days_per_station: tuple = (10, 20)
    change_days_per_year: tuple = (10, 25)
    report_days_total: tuple = (11, 30)
    coordination_fte: tuple = (0.1, 0.2)
    coordinator_loaded_salary: tuple = (100_000, 150_000)

    # Option B: in-house Python + TofuPilot
    engineer_loaded_salary: tuple = (95_000, 175_000)
    engineer_fte_on_test: tuple = (0.5, 1.0)
    ramp_up_cost: tuple = (10_000, 25_000)
    tofupilot_per_year: tuple = (0, 0)  # Lab tier


def labview_integrator(i: Inputs, k: int) -> float:
    seats = i.labview_seats[k] * i.labview_seat_per_year[k] * YEARS
    teststand = (
        i.teststand_dev_seat_per_year[k]
        + i.stations * i.teststand_deploy_per_station_year[k]
    ) * YEARS
    builds = i.stations * i.build_days_per_station[k] * i.integrator_day_rate[k]
    changes = i.change_days_per_year[k] * YEARS * i.integrator_day_rate[k]
    reports = i.report_days_total[k] * i.integrator_day_rate[k]
    coordination = i.coordination_fte[k] * i.coordinator_loaded_salary[k] * YEARS
    return seats + teststand + builds + changes + reports + coordination


def inhouse_python(i: Inputs, k: int) -> float:
    engineer = i.engineer_fte_on_test[k] * i.engineer_loaded_salary[k] * YEARS
    return engineer + i.ramp_up_cost[k] + i.tofupilot_per_year[k] * YEARS


if __name__ == "__main__":
    inputs = Inputs()
    for label, option in (
        ("LabVIEW + integrator", labview_integrator),
        ("In-house Python + TofuPilot", inhouse_python),
    ):
        low, high = option(inputs, 0), option(inputs, 1)
        print(f"{label:28} ${low:>9,.0f} to ${high:>9,.0f} over {YEARS} years")
```

The model leaves out what both options share: instruments, fixtures, the PC on the bench, and the product engineer's time on limits. It also leaves out the value of a same-day limit change, because you know that number for your line and this script doesn't.

## Start With One Station

If you're on Option A today, don't cancel the contract. Pick one station, ideally the highest-volume one or the one only the integrator can open, and rebuild it in Python with the TofuPilot Framework. Run it side by side with the LabVIEW version on the same units for two weeks, and you'll have a real ramp-up number and a real change-request number for your own model, not the ranges above.

```bash filename="install-and-run.sh"
# Install the CLI, then run the station locally (no account needed) or with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The rebuild steps are in [how to migrate from LabVIEW to Python for manufacturing tests with TofuPilot](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot), and the framework is at [tofupilot.com/products/framework](https://tofupilot.com/products/framework). The Lab tier is free, and `tofupilot run` works without an account.


### Is a LabVIEW Certification Still Worth It?
URL: https://www.tofupilot.com/guides/is-a-labview-certification-still-worth-it
Learn what CLAD, CLD, CLA and TestStand certifications signal in 2026, when they still pay off, and the portable Python path that replaces each topic.

A LabVIEW certification is worth it if you already work in an NI shop and want to move up inside it, and not worth it if you're choosing what to learn for the next ten years. The certificate signals fluency in one vendor's toolchain, and that toolchain's labour pool is shrinking while its price is a subscription. The portable alternative costs less to learn, and it transfers.

## What Each Certification Covers

NI runs three LabVIEW levels and two TestStand levels. Each one is a real exam with a real pass rate, and each signals something specific to an employer who knows the ladder.

| Certification | What it covers | What it signals |
|---|---|---|
| CLAD (Certified LabVIEW Associate Developer) | Dataflow, data types, arrays and clusters, loops, basic state machines, error handling. A written exam | You can read a VI and make a safe change to it |
| CLD (Certified LabVIEW Developer) | Building a working application to a spec under time, with style, documentation and one of the standard architectures (state machine, queued message handler) | You can build a station alone and someone else can maintain it |
| CLA (Certified LabVIEW Architect) | Architecture of large applications, framework design, leading other developers, reviewing their work | You can run the LabVIEW side of a plant, or a team at an NI Alliance Partner |
| CTD (Certified TestStand Developer) | Sequence files, step types, limits, process models, callbacks, report generation | You can own a TestStand deployment and its sequences |
| CTA (Certified TestStand Architect) | Custom process models, custom step types, multi-station deployment, integration with LabVIEW and other code modules | You can design how a whole line uses TestStand |

CLAD is a checkbox. CLD is the one that moves salaries, because it's the one that says "can ship". CLA and CTA mostly matter inside integrators and large NI shops.

## The Honest Case For

LabVIEW is a capable tool with a shrinking labour pool, and a shrinking pool is good for the people still in it.

- **The installed base.** A mid-size plant has years of LabVIEW stations, and somebody has to maintain them. Fewer people can, so the ones who can are paid well: a LabVIEW or TestStand developer typically earns well above a generalist Python engineer of equal seniority, in a typical 2026 range for the US and Western Europe, with defence and medical at the top.
- **Defence, aerospace and regulated environments.** Long programme lifetimes, validated toolchains, procurement that likes a certificate it recognises. A CLD on the CV clears a filter there that Python experience doesn't.
- **NI DAQ and PXI.** When timing lives in an FPGA on a PXI chassis, LabVIEW is the native tool and will stay so. Those stations don't migrate, and someone has to keep them running.
- **Inside an NI shop, CLD is a promotion step.** The employer often pays for it, and it's a concrete thing to put next to your name in the next review.

The people who can open the block diagram are retiring faster than they're being replaced, which is a good bargaining position and a bad career plan.

## The Honest Case Against

- **Subscription-only tooling.** Since 2023 LabVIEW is sold as an annual subscription, roughly $1,300 per year for Base and $4,300 for Full, per seat. Community Edition is free but non-commercial only. Your skill needs a paid seat to practise for work, and a portfolio at home doesn't count.
- **A shrinking hiring pipeline.** Fewer universities teach LabVIEW in core engineering curricula, and most now teach Python. There are fewer LabVIEW postings than a decade ago, and the candidate pool skews senior. Good if you're already in, expensive if you're trying to get in.
- **Skills that don't transfer.** Dataflow, VI style, TestStand callbacks and process models are NI-specific. Instrument I/O over SCPI, Git, packaging and data flow to a results platform transfer to every employer, including NI shops.
- **The exam tests the tool, not test engineering.** Knowing why a limit is 3.2 V, how to read a Cpk, or when a fixture is the problem is what makes a test engineer valuable, and no certification covers it.

If you already hold a CLD, none of this makes it worthless. It makes it a floor, not a ceiling. The [reskilling guide](/guides/reskill-a-labview-developer-to-python) is written for exactly that reader, and the [hiring guide](/guides/why-is-it-so-hard-to-hire-labview-engineers) shows what the managers on the other side of the table are seeing.

## The Portable Alternative

Every topic on the certification ladder has an equivalent that runs on any employer's laptop, with no seat.

| Certification topic | Portable equivalent |
|---|---|
| Dataflow, SubVIs, VI structure | Python functions and modules |
| State machine, queued message handler, producer-consumer | The framework's sequencer: phases in order, `depends_on`, multi-DUT `slots` in `procedure.yaml` |
| VISA and DAQmx drivers | PyVISA, `pyserial`, `nidaqmx` |
| TestStand sequence files, limits in the sequence | `procedure.yaml` with `validators` per measurement |
| TestStand callbacks and process models | Plugs and phases; see [replacing TestStand callbacks](/guides/replace-teststand-callbacks-in-python) |
| Report generation, TDMS, database logging | `tofupilot run --upload`; results, FPY, Cpk and Pareto in TofuPilot |
| Source control with the LabVIEW compare tool | Git, pull requests, tags, `git blame` on a limit |
| Style guide and documentation | A README and text files that diff |
| Operator interface, front panels | Operator UI declared in the procedure: text inputs, checklists, images, sliders, progress |

The right-hand column is what the [skills checklist](/guides/python-skills-checklist-for-test-engineers) tests for. It fits on one page, which the certification ladder doesn't.

## A Self-Study Plan

Ten weeks, one evening or one Friday afternoon a week, building things you'd put on a CV. Each step uses a public template repo that runs with the CLI in minutes, so the sequencer, the operator UI and the results are there from the first session.

| Weeks | Build | Template | Done when |
|---|---|---|---|
| 1 to 2 | A PyVISA plug for one instrument you can reach: `*IDN?`, one measurement, a timeout, `close()` | None; the [PyVISA docs](https://pyvisa.readthedocs.io) are enough | The plug returns a float and survives an unplugged cable |
| 3 to 4 | A PCBA functional test with your plug swapped in for the mock | [`fct-fixture`](https://github.com/tofupilot/template-framework-fct-fixture) | A run with limits lands in TofuPilot with `--upload` |
| 5 to 6 | Serial console and a subprocess to a programmer | [`mcu-programming-serialization`](https://github.com/tofupilot/template-framework-mcu-programming-serialization) | Serial number read from the DUT ends up on the run page |
| 7 to 8 | Multi-phase calibration with `depends_on` | [`imu-thermal-calibration`](https://github.com/tofupilot/template-framework-imu-thermal-calibration) | You can explain the Cpk of one measurement across your runs |
| 9 to 10 | Rebuild one station from your own plant, or from a former job's spec | Whichever template is closest | The README lets a colleague run it without you |
| Throughout | Git: one branch per week, one pull request to yourself, one tag per finished step | Any of the above | `git log` reads like a change history and not a diary |

Ten weeks is long enough to show a manager a station that runs. It's shorter than the study time for a CLD, and nothing in it expires with a subscription.

## How Big a Station Phase Is

Candidates coming from LabVIEW often expect the Python version of a station to be large. It isn't. This is a complete phase, plug and all, for one power rail measurement.

```python filename="phases/power_rails.py"
# A complete phase. The framework injects `measurements` and the `dmm` plug
# by name; the limits for rail_3v3 live in procedure.yaml, not here.
def power_rails(measurements, dmm):
    measurements.rail_3v3 = dmm.read_voltage()
```

```python filename="plugs/dmm.py"
# The plug behind it: PyVISA, one SCPI query, cleanup in close().
import pyvisa


class Multimeter:
    def __init__(self, address="TCPIP::192.168.1.100::INSTR"):
        self.rm = pyvisa.ResourceManager("@py")
        self.inst = self.rm.open_resource(address)
        self.inst.timeout = 2000

    def read_voltage(self):
        return float(self.inst.query(":MEAS:VOLT:DC?"))

    def close(self):
        self.inst.close()
        self.rm.close()
```

The sequencer, the limits, the operator UI and the upload are in `procedure.yaml` and the CLI. What's left for you to write is the part that's specific to your instrument and your product, and that's the part a certification can't test anyway.

## Start With One Station

Pick the station with the highest volume, or the one only one person can open. Rebuild it in Python with the TofuPilot Framework, then run it side by side with the LabVIEW version on the same units for two weeks. If you're the candidate rather than the manager, do the same with a template and a bench instrument, and bring the run page to the interview.

```bash filename="station-setup.sh"
# Install the CLI, run a procedure locally, then run it with upload.
curl -fsSL https://www.tofupilot.app/install | sh
tofupilot run ./procedure.yaml
tofupilot run ./procedure.yaml --upload
```

The [LabVIEW migration guide](/guides/how-to-migrate-from-labview-to-python-for-manufacturing-tests-with-tofupilot) covers the rebuild step by step, and the [Framework page](https://tofupilot.com/products/framework) covers the layout. The Lab tier is free, and `tofupilot run` works with no account.


## Templates

### Cell Incoming Inspection
URL: https://www.tofupilot.com/templates/cell-incoming-inspection
Incoming inspection of a 21700 cell: OCV against the shipping window with a sorting bin, 1 kHz ACIR grade, 10 A pulse DCIR, self-discharge current in an hour.


## Introduction

### Cell Incoming Inspection Overview

A pack is only as good as its worst cell, and the cell vendor's certificate covers the cells as they left the vendor, not as they arrived. Between the two sit weeks in a container, a temperature history nobody logged, and the small fraction of cells with a latent internal short that no factory screen catches on day one. Incoming inspection is where a pack maker measures each cell before committing it to a module: its open-circuit voltage against the shipping window, its impedance, and, on the lines that take self-discharge seriously, the current a cell needs to stay at its own voltage. The result is a grade per cell and a bin that pack matching draws from.

![A single 21700 cell standing in a fixture block, a dark probe head above it with two gold spring probes on the positive terminal.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-bfb0adbd-1533-4737-804f-ceabed4ada66-KXYU03fsYxkSxqALFcganmhFbLhlwB.jpg)

One cell under the four-wire probe head: two spring probes on the positive terminal, two under the can, the impedance meter and the self-discharge analyzer both reading through the same contacts.

The numbers on the datasheet are the ones to build on, and there are two of them for resistance. A Samsung INR21700-33J is specified at **29 ± 5 mΩ AC at 1 kHz and 40 ± 10 mΩ DC at 50 % SOC**, on the same cell; a Molicel P42A at 10 mΩ AC and 16 mΩ DC. The ratio is about 1.5 and it is not a tolerance: the 1 kHz value is the ohmic part, the DC value adds charge transfer and diffusion, and it is the DC value that predicts sag under load (Hioki's application material on lithium-ion internal resistance covers the distinction). The two get measured with different instruments, in different times, and never share a limit.

### Test Purpose

The procedure records one identity and health fingerprint per cell:

- **Contact, mass and surface temperature** before any electrical reading
- **Open-circuit voltage** against the vendor's shipping window, and the **10 mV bin** the cell sorts into
- **1 kHz AC impedance**, four-terminal, in 18 ms, with a **grade letter**
- **DC resistance** from a 10 A pulse, split into the ohmic step and the polarization droop
- **Self-discharge current** by the potentiostatic method, settled over an hour, with the equivalent OCV drop rate for comparison with aging-based screens

![Terminal voltage over a 10 A, 1 s discharge pulse: a 148 mV instantaneous step at the pulse edge, an exponential droop to the end of the pulse, and recovery afterwards; the current as a dotted step on the right axis.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-eafc25b8-6ad7-4d1e-af58-7412038ce142-V97mogqxFgQmlS3eZUQpnqhx8k77Fn.png)

The mock cell's pulse: 14.8 mΩ of ohmic step at the edge, 20.0 mΩ total at the end of the second, against 13.6 mΩ at 1 kHz on the same cell and a 32 mΩ DC limit. The three numbers describe one cell and each says something the others do not.

The framework mechanics on show are `then: error: retry` on a phase whose instrument raises a contact error, a string `in` validator on the grade, a multi-dimensional waveform with two derived aggregations (one limited, one recorded), a time-scaled hour-long capture with a settled-value aggregation, a station-scoped plug, and unit metadata that pack matching reads later.

## Equipment & Setup

To run this inspection at goods-in, the following are required:

- **A battery impedance meter** with a four-terminal 1 kHz AC method and a voltage function, fast enough for a tray
- **A DC load** in transient mode for the 10 A pulse, with the meter's voltage function logging the terminal voltage
- **A self-discharge analyzer** that holds the cell at its own OCV and reads the current it supplies at sub-microamp resolution
- **A probe fixture** with two spring probes per terminal, a load cell under the cell and an IR sensor on the head
- **The Device Under Test (DUT)**: one 21700 cell from the incoming tray, at the ambient of the room for at least a few hours
- **A TofuPilot Framework procedure** to sequence the readings, grade the cell and validate the limits
- **The TofuPilot Dashboard** to keep the per-cell record that pack matching and, from 2027, the EU battery passport's genealogy draw on

### Hardware Components

#### Impedance Meter

The [Hioki BT3562](https://www.hioki.com/global/products/battery-testers/internal-resistance/id_6019) is the production reference for the 1 kHz method: 1 kHz ± 0.2 Hz, ±0.5 % of reading, 0.1 µΩ resolution, and **resistance and voltage together in 18 ms** on a four-terminal probe. For an engineering audit the [BT4560](https://www.hioki.com/global/products/battery-testers/internal-resistance/id_6009) sweeps 0.01 Hz to 1050 Hz and separates electrolyte, charge-transfer and diffusion resistance, in minutes per cell; that is a sample-audit instrument, not a line instrument, and a template that assumed full impedance spectroscopy at line rate would not survive contact with a tray of 400 cells.

![Cylindrical cell in an insulated four-wire contact fixture beside a waiting tray](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/cell-incoming-inspection-physical-2026-Cah9fGcrZAk5Sv2H4V8uQL0StVY0vi.webp)

The cell rests in a cradle between insulated contact housings. Separate current and sense leads exit each housing; the contact tips are recessed inside.

#### Self-Discharge Analyzer

The classic screen for a latent short is a voltage drop over time: charge the cell to a set SOC, rest it, measure the OCV, store it for two to four weeks, measure again. The [Keysight BT2152B](https://www.keysight.com/us/en/product/BT2152B/self-discharge-analyzer.html) replaces the weeks with about an hour: it holds the cell at its own open-circuit voltage to ±3 µV over 24 hours and reads the current it has to supply on a ±10 mA range at 250 nA resolution, on up to 32 channels. Once the cell's internal charge redistribution has settled, the current the analyzer supplies is the current the cell is losing, and a micro-short reads hundreds of microamps where a healthy cell reads about ten.

#### DC Pulse Load

Any load with a transient mode that reaches 10 A in under a millisecond: a Chroma 63600 or a Kikusui PLZ-4W at 10 µs rise. The voltage waveform comes from the meter's V function or a DAQ at 1 kHz; the pulse is short enough not to move the SOC.

### Where the Limits Come From

The OCV window is the vendor's: 3.43 to 3.63 V for this cell class, the roughly 30 % state of charge the IATA transport rule requires for air shipment. The 1 kHz impedance window is the datasheet's. The DC limit sits at 1.6 times the top of the AC window. The self-discharge gate is the pack maker's own: peer-reviewed measurements on commercial NCA and silicon-graphite cells at 50 % SOC and 25 °C put a healthy cell around 10 µA and 1 to 2 mV of OCV drop over 28 days, about 0.05 mV/day; the "2 mV/day" figure that circulates as a K-value limit is forty times that and passes almost anything. A 30 µA gate is three times a healthy cell and an order of magnitude under a micro-short. The grading bins (10 mV of OCV, three impedance grades) are what pack matching needs and nothing a standard specifies.

## Test Procedure

### Overview

The procedure maps the inspection onto the framework's three stages. Contact and identity live in `setup:` so no electrical number is recorded on a cold cell or through a lifted probe. Lifting the probes lives in `teardown:` so the tray can index whatever happened.

1. Setup: probe path, mass, surface temperature.
2. Main: OCV against the shipping window, 10 mV bin.
3. Main: 1 kHz AC impedance, grade, retried on a contact error.
4. Main: 10 A pulse, ohmic step and DC resistance.
5. Main: self-discharge current, 60 minutes, settled value.
6. Teardown: probes up.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the cell's record.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  contact_check.py
  ocv_grading.py
  acir_1khz.py
  dcir_pulse.py
  self_discharge.py
  release.py
plugs/
  cell_station.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-cell-incoming-inspection). The `CellStation` plug is a mock of the meter, the load, the analyzer, the scale and the IR sensor together, synthesizing one healthy cell whose first impedance read bounces on the probes, so the retry runs every time and the procedure runs end-to-end without a station or a cell connected. The self-discharge capture is **time-scaled**: the mock returns the hour in one call.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit (the cell's laser-marked QR code), the station plug at **station scope**, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Cell Incoming Inspection
version: 0.1.0
description: Incoming inspection of a 21700 NMC cell. Contact and identity checks, open-circuit voltage against the vendor's shipping window with a 10 mV sorting bin, 1 kHz AC impedance with a grade, 10 A DC pulse resistance split into ohmic and polarization parts, self-discharge current by the potentiostatic method in an hour.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the cell's laser-marked QR code"
    placeholder: "C21700-0000000000"
    pattern: "^C21700-\\d{10}$"
    default_value: "C21700-2609104417"
  part_number:
    default_value: "INR21700-50E"
  batch_number:
    default_value: "LOT-8842"

plugs:
  - name: Cell Station
    description: "AC impedance meter, DC pulse load, self-discharge analyzer, scale and IR sensor on one probe fixture (mock, station scope)"
    python: plugs.cell_station:CellStation
    key: station
    scope: station

setup:
  - name: Contact Check
    key: contact_check
    python: phases.contact_check
    measurements:
      - name: Probe Path
        key: probe_path_ohm
        unit: Ω
        validators:
          - {operator: "<=", expected_value: 0.5}
      - name: Mass
        key: mass_g
        unit: g
        description: A 21700 NMC cell of this capacity weighs 67 to 70 g; lighter is a different part or an empty can.
        validators:
          - {operator: ">=", expected_value: 67.0}
          - {operator: "<=", expected_value: 70.0}
      - name: Surface Temperature
        key: surface_temp_c
        unit: °C
        description: OCV and impedance are both temperature dependent; a cell straight from a cold truck is measured later, not corrected.
        validators:
          - {operator: ">=", expected_value: 20.0}
          - {operator: "<=", expected_value: 28.0}

main:
  - name: OCV Grading
    key: ocv_grading
    python: phases.ocv_grading
    measurements:
      - name: OCV
        key: ocv_v
        unit: V
        description: Vendor shipping window 3.43 to 3.63 V (about 30 % SOC per the IATA transport rule); outside it the cell was mishandled or is not what the label says.
        validators:
          - {operator: ">=", expected_value: 3.43}
          - {operator: "<=", expected_value: 3.63}
      - name: OCV Bin
        key: ocv_bin
        description: 10 mV bin from the window floor, 0 to 19; cells in one pack come from one bin.
        validators:
          - {operator: ">=", expected_value: 0}
          - {operator: "<=", expected_value: 19}

  - name: ACIR 1kHz
    key: acir_1khz
    python: phases.acir_1khz
    depends_on: [ocv_grading]
    then:
      error: retry
    measurements:
      - name: ACIR
        key: acir_mohm
        unit: mΩ
        description: 1 kHz four-terminal impedance; the datasheet number for this class of cell, 10 to 20 mOhm.
        validators:
          - {operator: ">=", expected_value: 10.0}
          - {operator: "<=", expected_value: 20.0}
      - name: ACIR Grade
        key: acir_grade
        description: A below 15 mOhm, B below 18, C below 20; a pack is built from one grade.
        validators:
          - {operator: in, expected_value: [A, B, C]}

  - name: DCIR Pulse
    key: dcir_pulse
    python: phases.dcir_pulse
    depends_on: [acir_1khz]
    measurements:
      - name: Pulse
        key: pulse
        title: 10 A discharge pulse, terminal voltage and current
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Voltage
            key: voltage
            unit: V
            aggregations:
              - type: dcir_mohm
                unit: mΩ
                validators:
                  - {operator: "<=", expected_value: 32.0}
              - type: ohmic_mohm
                unit: mΩ
          - legend: Current
            key: current
            unit: A

  - name: Self Discharge
    key: self_discharge
    python: phases.self_discharge
    depends_on: [dcir_pulse]
    timeout: 90m
    measurements:
      - name: Self-Discharge Current
        key: sd_current
        title: Potentiostatic self-discharge current over 60 minutes
        x_axis:
          legend: Time
          unit: min
        y_axis:
          - legend: Current
            key: current
            unit: µA
            aggregations:
              - type: settled_ua
                unit: µA
                validators:
                  - {operator: "<=", expected_value: 30.0}
      - name: Equivalent OCV Drop
        key: equivalent_mv_per_day
        unit: mV/day
        description: The settled current expressed as an OCV drop rate for comparison with aging-based screens; recorded, not limited.

teardown:
  - name: Release
    key: release
    python: phases.release
```

Framework features to notice:

1. **`then: error: retry`.** The impedance phase declares what to do when its plug raises: retry. The first read bounces on the probes, the meter reports a contact error, the phase runs again and passes; the report shows both attempts.
2. **String `in`.** `acir_grade` validates against a list of grade letters; a cell outside every bin gets `REJECT` and fails with the reason.
3. **One aggregation limited, one recorded.** The pulse waveform carries `dcir_mohm` with a limit and `ohmic_mohm` without; the second is there for the trend, not the verdict.
4. **Time-scaled hour.** The self-discharge phase has a 90 minute `timeout` for the real analyzer and returns in milliseconds on the mock; the `settled_ua` aggregation is the mean of the last ten minutes either way.
5. **Station-scoped plug.** `scope: station` keeps the instruments open across the tray; `setup:` and `teardown:` land and lift the probes per cell.
6. **Unit metadata for the next station.** `ocv_bin`, `acir_grade` and `cell_lot` go onto the unit, where pack matching reads them.

### Contact Check

The setup phase lands the probes, reads the four-wire path resistance, and takes the two identity readings that cost nothing and catch a wrong tray: the mass on the load cell and the can temperature on the IR sensor. The cell lot goes onto the **unit metadata**:

```python filename="phases/contact_check.py"
def contact_check(measurements, station, unit, log):
    """Setup: probes down, four-wire path resistance inside the meter's
    tolerance, cell mass and surface temperature as identity checks. A
    cell 2 g light is a different cell; a warm cell reads a wrong OCV."""
    path = station.probes_down()
    measurements.probe_path_ohm = path
    measurements.mass_g = station.mass_g()
    measurements.surface_temp_c = station.surface_temp_c()
    unit.metadata["cell_lot"] = "LOT-8842"
    log.info(f"Cell {unit.serial_number}: probes at {path:.2f} ohm, {measurements.mass_g} g, {measurements.surface_temp_c} C")
```

### OCV Grading

The meter's voltage function gives the open-circuit voltage at 0.1 mV. The phase validates it against the shipping window and computes the 10 mV bin from the window floor; the bin is what the pack line sorts on, so it goes onto the unit as well:

```python filename="phases/ocv_grading.py"
import math

from utils.recipe import OCV_BIN_MV, SHIPPING_OCV_V


def ocv_grading(measurements, station, unit, log):
    """Open-circuit voltage against the vendor's shipping window, and the
    10 mV bin the cell sorts into for pack matching."""
    ocv = station.ocv_v()
    bin_index = int(math.floor((ocv - SHIPPING_OCV_V[0]) * 1000.0 / OCV_BIN_MV))
    measurements.ocv_v = ocv
    measurements.ocv_bin = bin_index
    unit.metadata["ocv_bin"] = bin_index
    log.info(f"OCV {ocv:.4f} V, bin {bin_index} ({SHIPPING_OCV_V[0] * 1000 + bin_index * OCV_BIN_MV:.0f} mV floor)")
```

### ACIR 1kHz

One 18 ms read, four-terminal, at 1 kHz. The grade is looked up from the impedance bins in the recipe. When the probe bounces, the meter raises, and the `then: error: retry` on the phase runs it again:

```python filename="phases/acir_1khz.py"
from utils.recipe import ACIR_BINS


def acir_1khz(measurements, station, unit, log):
    """1 kHz AC impedance, four-terminal, 18 ms. The production resistance
    number: fast, repeatable, and not the same as DC resistance."""
    acir = station.acir_1khz_mohm()
    grade = next((g for g, (lo, hi) in ACIR_BINS.items() if lo <= acir < hi), "REJECT")
    measurements.acir_mohm = acir
    measurements.acir_grade = grade
    unit.metadata["acir_grade"] = grade
    log.info(f"ACIR 1 kHz {acir:.2f} mOhm, grade {grade}")
```

### DCIR Pulse

A 10 A discharge for one second, the terminal voltage logged at 1 kHz. The phase takes the voltage before the edge, a few milliseconds after it, and at the end of the pulse: the first difference is the ohmic resistance, the second the total DC resistance, and both go onto the waveform as aggregations:

```python filename="phases/dcir_pulse.py"
import numpy as np

from utils.recipe import DCIR_PULSE_A, DCIR_PULSE_S


def dcir_pulse(measurements, station, log):
    """10 A, 1 s discharge pulse. The instantaneous step at the pulse edge
    is the ohmic part; the droop that follows is charge transfer and
    diffusion. DC resistance is the total at the end of the pulse, the
    number that predicts sag under load, and it runs about 1.5 times the
    1 kHz value on the same cell."""
    cap = station.dc_pulse(DCIR_PULSE_A, DCIR_PULSE_S)
    t = np.array(cap["time_s"])
    v = np.array(cap["voltage_v"])
    v_before = v[(t > -0.1) & (t < 0.0)].mean()
    v_step = v[(t > 0.002) & (t < 0.010)].mean()
    v_end = v[(t > DCIR_PULSE_S - 0.05) & (t < DCIR_PULSE_S)].mean()
    ohmic = (v_before - v_step) / DCIR_PULSE_A * 1000.0
    dcir = (v_before - v_end) / DCIR_PULSE_A * 1000.0

    measurements.pulse.x_axis = cap["time_s"]
    measurements.pulse.y_axis.voltage = cap["voltage_v"]
    measurements.pulse.y_axis.voltage.aggregations.dcir_mohm = float(dcir)
    measurements.pulse.y_axis.voltage.aggregations.ohmic_mohm = float(ohmic)
    measurements.pulse.y_axis.current = cap["current_a"]
    log.info(f"DCIR {dcir:.2f} mOhm at {DCIR_PULSE_A:.0f} A / {DCIR_PULSE_S:.0f} s, ohmic part {ohmic:.2f} mOhm")
```

### Self Discharge

The analyzer holds the cell at its own OCV and logs the current it supplies, every 30 s for an hour. The first minutes are the cell's internal charge redistribution and mean nothing; the mean over the last ten minutes is the self-discharge current. The phase also expresses it as an OCV drop rate, for the people who still think in millivolts per day:

```python filename="phases/self_discharge.py"
import numpy as np

from utils.recipe import SELF_DISCHARGE_SETTLE_MIN


def self_discharge(measurements, station, log):
    """Self-discharge current by the potentiostatic method: the analyzer
    holds the cell at its own OCV and the current it has to supply, once
    the cell's internal redistribution has settled, is the self-discharge.
    About an hour instead of two to four weeks of OCV aging."""
    cap = station.self_discharge_capture(SELF_DISCHARGE_SETTLE_MIN)
    t = np.array(cap["time_min"])
    i = np.array(cap["current_ua"])
    settled = float(i[t >= SELF_DISCHARGE_SETTLE_MIN - 10.0].mean())

    measurements.sd_current.x_axis = cap["time_min"]
    measurements.sd_current.y_axis.current = cap["current_ua"]
    measurements.sd_current.y_axis.current.aggregations.settled_ua = settled
    # Equivalent OCV-drop rate for the people who still think in mV/day:
    # a 5 Ah cell near 30 % SOC has dV/dQ of roughly 0.25 mV per mAh.
    measurements.equivalent_mv_per_day = settled * 24.0 / 1000.0 * 0.25
    log.info(f"Self-discharge {settled:.1f} uA settled, about {settled * 24.0 / 1000.0 * 0.25:.3f} mV/day equivalent")
```

![Self-discharge current on a log axis over 60 minutes: starting near 170 µA, decaying through the 30 µA limit at 23 minutes, settling to 14 µA over the last ten minutes marked by a blue segment.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-cf7381fc-2bb0-49a8-b759-8022afa00124-sKORQTZFrVdruoBNy7tcz2PNoG6qIh.png)

The mock cell under the analyzer: 14 µA settled, about 0.08 mV/day equivalent, against a 30 µA gate. A cell with a micro-short never comes down to the floor; the curve flattens in the hundreds. On the OCV-aging method the same cell would have needed two to four weeks of shelf time, and the anode-overhang equalization would still be confounding the reading at week four.

### Mock Plug

`CellStation` stands in for five instruments on one fixture. It synthesizes one 21700 cell at 3.531 V, 14.2 mΩ at 1 kHz, a DC resistance 1.52 times that split 72/28 between ohmic and polarization with a 350 ms time constant, 11.4 µA of self-discharge under a 140 µA redistribution transient that decays with a 9 minute time constant, 68.4 g and 23.6 °C. The first impedance read raises a contact error, once. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real station, the class speaks SCPI to a Hioki BT3562 (`:MEAS?` returns resistance and voltage together), drives a Chroma 63600 or Kikusui PLZ-4W in transient mode for the pulse with the meter's voltage function logging, reads a Keysight BT2152B for the self-discharge current, and the fixture's load cell and IR sensor for the identity checks. Set `TIME_SCALE = 1.0` when the analyzer streams the hour in real time, keep the AC and DC numbers on separate limits, and let a cold tray equilibrate rather than correcting its readings. The phases, measurements and limits stay the same.


### Camera Module Intrinsic Calibration
URL: https://www.tofupilot.com/templates/camera-module-intrinsic-calibration
Calibrate a camera module's intrinsics and lens distortion on a ChArUco target with OpenCV, validate on held-out poses, write the EEPROM.


## Introduction

### Intrinsic Calibration Overview

A camera module turns a 3D point into a pixel through the **pinhole model** and the **lens distortion**. The pinhole part is the intrinsic matrix: focal lengths **fx, fy** in pixels and the principal point **cx, cy** where the optical axis meets the sensor. The lens part is the Brown-Conrady polynomial: radial terms **k1, k2, k3** (barrel or pincushion) and tangential terms **p1, p2** (lens decentring and tilt). Every module leaves the line with its own set: the lens sits a few micrometres off the sensor centre, the focal length varies with the glue thickness, the distortion with the lens batch. A 6 mm lens on a 1/2.3-inch sensor puts a corner point about **140 px** away from where a perfect pinhole would, so any application that measures, maps, or navigates with the image needs the per-module coefficients, not the datasheet.

**Intrinsic calibration** recovers them from images of a target of known geometry. The module looks at a **ChArUco board** (a chessboard with ArUco markers in the white squares, so every corner has an identity even when the board is partly out of frame) from several poses; the corners are detected to sub-pixel precision; `cv2.calibrateCamera` fits the model that best reprojects all of them at once. The residual, the **reprojection error**, is the quality figure: below 0.3 px RMS on a 1080p sensor is a good module and a good fit.

### Test Purpose

The procedure produces, per module:

- **Identity**: serial read back from the module EEPROM against the scanned label, sensor and lens stamped on the unit
- **Capture record**: 16 calibration poses plus 3 held-out poses, corners detected per pose, sensor coverage across a 6 by 4 grid of cells
- **The fit**: fx, fy, cx, cy, k1, k2, k3, p1, p2 with two-sided limits, RMS reprojection error, per-pose error with the maximum validated
- **Independent validation**: reprojection on the 3 held-out poses with the pose solved from the calibrated model, straightness of an undistorted board row, horizontal field of view
- **Persistence**: the coefficients written to the module EEPROM, read back and compared, the calibration ID on the unit

![Scatter of the 35 ChArUco corners of all 16 poses on the 1920 by 1080 sensor with the 6 by 4 coverage grid, every cell hit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-40e85b26-0528-45b4-be35-6e11018c465c-7iYK8YLFgLvaEJPyeB9bgi6WGA5eNc.png)

Where the 16 poses put the corners on the sensor in one run of the mock: every one of the 24 coverage cells sees corners, including the borders where distortion is largest and a fit with centre-only data goes wrong.

Defects this catches before the module ships: a **decentred lens** (principal point outside its window), a **wrong or badly seated lens** (focal length or field of view off), a **tilted lens** (tangential terms above limit), a **contaminated or scratched front element** (corners lost, per-pose error up), and a **bad EEPROM** (read-back mismatch).

Beyond the calibration content, the template shows five framework mechanics: a **setup phase stamping `unit.metadata`** from the module, an **operator UI progress bar** over the 16 poses, **multi-dimensional measurements with custom aggregations** (`min_corners`, `max_px`) validated in YAML, **nine numeric measurements from one fit** with two-sided limits, and **`attach.file`** of the calibration file on the run.

## Equipment & Setup

To calibrate camera modules on a production line, the following are required:

- **A calibration target**: a ChArUco or chessboard chart, printed on glass or dibond, flat to better than 0.1 mm, backlit for constant contrast
- **A target stage**: a motorised tilt and slide head so the chart appears at different poses, or a fixed chart and a moving module
- **A module fixture**: a nest with pogo pins for power and MIPI, a bridge to the test PC
- **The Devices Under Test**: assembled camera modules, lens glued and cured, EEPROM blank
- **A TofuPilot Framework procedure** to drive the stage, capture, fit, validate and write
- **The TofuPilot Dashboard** to keep every module's coefficients and trend them per lens batch

![Camera module on an adjustable rail facing a checkerboard target](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/camera-module-intrinsic-calibration-physical-2026-o7xhirdd64OqJgBdkeJcHUQe0Sy9n4.webp)

The module and target share an adjustable optical rail. The board is supported on insulated mounts; illumination control and the test computer are omitted.

### Hardware Components

#### Calibration Targets

**Calib.io** and **Image Engineering** sell ChArUco and chessboard charts on glass with certified flatness and square pitch. ChArUco (a chessboard with an ArUco marker in every white square, OpenCV's `cv2.aruco.CharucoBoard`) is the production choice because every corner carries an ID: partial views at the sensor border still contribute, which is exactly where the distortion terms need data. The template's board is 8 by 6 squares of 30 mm, 35 inner corners.

#### Target Stage and Module Fixture

A two-axis tilt head with a slide covers the poses a calibration needs: tilts to ±30° on both axes, distances from 0.35 to 0.6 m, the board centre walked over the field. Vendors of complete camera test stations, **TRIOPTICS** (CamTest Chart), **Image Engineering** and **Radiant Vision Systems**, integrate the chart, the illumination and the fixture in one cabinet and add MTF, boresight and relative illumination on the same frames. The module fixture is a machined nest with pogo pins on the board edge and a MIPI-to-USB bridge (Leopard Imaging, e-con Systems, or the SoC vendor's) to the test PC.

![Close-up of the module in its fixture nest with two pogo pins on the board edge, lens pointing up at the tilted chessboard chart on its tilt head.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-dda8d0af-5208-4d43-bccc-c1143be4e8e5-HPHGFYQNqrPs8ih67YES0dNCVXzv6Q.jpg)

The module in its nest, two pogo pins on the board edge, the chart tilted above it. Each pose is one frame, one detection, one row in the capture record.

### Limits

Limits for a 6 mm f/2.0 lens on a 1920 by 1080 sensor with 1.55 µm pixels, as the template ships them:

| Parameter | Limit | Why |
|-----------|-------|-----|
| Reprojection RMS | ≤ 0.30 px | detector noise plus model error on a good module |
| Per-pose RMS max | ≤ 0.50 px | one bad pose (blur, glare) fails the run, not the average |
| fx, fy | 1400 to 1500 px | nominal 1452 px, ±3.5 % covers glue thickness and lens tolerance |
| fy / fx | 0.995 to 1.005 | square pixels; outside means a tilted sensor or a bad fit |
| cx, cy | 930 to 990, 510 to 570 px | ±30 px around the sensor centre, a decentred lens fails here |
| k1, k2, k3 | −0.35 to −0.20, 0.02 to 0.20, −0.10 to 0.05 | the lens design's barrel signature |
| p1, p2 | ±0.002 | tangential terms above this mean a tilted lens |
| Held-out RMS | ≤ 0.40 px | the model must generalise to poses it never saw |
| Line straightness | ≤ 0.50 px | undistorted rows must come out straight across the full width |
| Horizontal FOV | 64° to 70° | derived from fx, catches a wrong lens outright |

### Custom Firmware

The module runs its stock sensor firmware; capture is a raw frame over MIPI. The only module-side feature the procedure needs is a **writable EEPROM** (typically 256 bytes on the module's I2C bus) where the coefficients are stored for the host driver to read at boot. Modules without EEPROM keep the calibration in the host's database keyed by serial, which the run record already is.

## Test Procedure

### Overview

For each module in the fixture, the procedure runs:

1. Setup: open the module, read serial, sensor, lens, resolution and temperature.
2. Drive the stage through 16 calibration poses and 3 held-out poses, detect the ChArUco corners at each, check sensor coverage.
3. Fit the pinhole model and five distortion coefficients with OpenCV, validate every coefficient, the RMS and the per-pose errors.
4. Solve the pose of the 3 held-out views with the fitted model and check their reprojection, the straightness of undistorted board rows, and the field of view.
5. Write the coefficients to the module EEPROM, read them back, compare.
6. Teardown: home the stage and close the module.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  connect_module.py
  capture_views.py
  calibrate.py
  verify_holdout.py
  write_eeprom.py
  release_module.py
plugs/
  camera.py
  stage.py
utils/
  board.py
README.md
pyproject.toml
```

The module and the stage are mocks; the calibration is real OpenCV, run by the framework in a `uv` environment with `numpy` and `opencv-python-headless`. Run it with `tofupilot run . --no-tui --no-kiosk --json`. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-camera-intrinsic-calibration).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: Camera Module Intrinsic Calibration
version: 0.1.0
description: Calibrates a camera module's intrinsics and lens distortion on a ChArUco target, validates the model on held-out views, and writes the coefficients to the module EEPROM.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the module label"
    placeholder: "CAM-000000"
    pattern: "^CAM-\\d{6}$"
    default_value: "CAM-041872"
  part_number:
    default_value: "CAM-IMX477-6MM"
  metadata:
    sensor:
      description: "Sensor model read from the module"
    lens:
      description: "Lens focal length and f-number read from the module"
    calibration_id:
      description: "Identifier of the calibration written to the EEPROM"

plugs:
  - name: Camera Module
    description: "Module under test over the fixture's MIPI to USB bridge: frames, EEPROM, temperature (mock)"
    python: plugs.camera:CameraModule
    key: camera
  - name: Target Stage
    description: Motorised tilt and slide stage holding the ChArUco target (mock)
    python: plugs.stage:TargetStage
    key: stage

setup:
  - name: Connect Module
    key: connect_module
    python: phases.connect_module
    measurements:
      - name: Module Serial
        key: module_serial
        description: "Serial read from the module EEPROM, must match the scanned label"
        validators:
          - {operator: matches, expected_value: "^CAM-\\d{6}$"}
      - name: Sensor Temperature
        key: sensor_temperature
        unit: °C
        validators:
          - {operator: ">=", expected_value: 15.0}
          - {operator: "<=", expected_value: 45.0}
      - name: Resolution
        key: resolution
        validators:
          - {operator: "==", expected_value: "1920x1080"}

main:
  - name: Capture Views
    key: capture_views
    python: phases.capture_views
    ui:
      components:
        - key: capture_progress
          type: progress
          label: "Target poses"
          description: "16 poses, tilt and slide"
          default_value: 0
          max: 100
    measurements:
      - name: Views Captured
        key: views_captured
        validators:
          - {operator: "==", expected_value: 16}
      - name: Corners per View
        key: corners_per_view
        title: Detected ChArUco corners per pose
        x_axis:
          legend: Pose
        y_axis:
          - legend: Corners
            key: corners
            aggregations:
              - type: min_corners
                validators:
                  - {operator: ">=", expected_value: 20}
      - name: Sensor Coverage
        key: coverage_pct
        unit: "%"
        description: "Share of a 6 by 4 grid of sensor cells that saw at least one corner across all poses"
        validators:
          - {operator: ">=", expected_value: 85.0}

  - name: Calibrate
    key: calibrate
    python: phases.calibrate
    depends_on: [capture_views]
    measurements:
      - name: Reprojection RMS
        key: rms_px
        unit: px
        validators:
          - {operator: "<=", expected_value: 0.30}
      - name: Reprojection Error per View
        key: view_error
        title: Reprojection error per pose
        x_axis:
          legend: Pose
        y_axis:
          - legend: RMS
            key: error
            unit: px
            aggregations:
              - type: max_px
                unit: px
                validators:
                  - {operator: "<=", expected_value: 0.50}
      - name: Focal Length X
        key: fx
        unit: px
        validators:
          - {operator: ">=", expected_value: 1400.0}
          - {operator: "<=", expected_value: 1500.0}
      - name: Focal Length Y
        key: fy
        unit: px
        validators:
          - {operator: ">=", expected_value: 1400.0}
          - {operator: "<=", expected_value: 1500.0}
      - name: Pixel Aspect
        key: pixel_aspect
        description: "fy over fx, square pixels expected"
        validators:
          - {operator: ">=", expected_value: 0.995}
          - {operator: "<=", expected_value: 1.005}
      - name: Principal Point X
        key: cx
        unit: px
        validators:
          - {operator: ">=", expected_value: 930.0}
          - {operator: "<=", expected_value: 990.0}
      - name: Principal Point Y
        key: cy
        unit: px
        validators:
          - {operator: ">=", expected_value: 510.0}
          - {operator: "<=", expected_value: 570.0}
      - name: Radial k1
        key: k1
        validators:
          - {operator: ">=", expected_value: -0.35}
          - {operator: "<=", expected_value: -0.20}
      - name: Radial k2
        key: k2
        validators:
          - {operator: ">=", expected_value: 0.02}
          - {operator: "<=", expected_value: 0.20}
      - name: Radial k3
        key: k3
        validators:
          - {operator: ">=", expected_value: -0.10}
          - {operator: "<=", expected_value: 0.05}
      - name: Tangential p1
        key: p1
        validators:
          - {operator: ">=", expected_value: -0.002}
          - {operator: "<=", expected_value: 0.002}
      - name: Tangential p2
        key: p2
        validators:
          - {operator: ">=", expected_value: -0.002}
          - {operator: "<=", expected_value: 0.002}
      - name: Camera Matrix
        key: camera_matrix
        description: "The 3 by 3 intrinsic matrix as fitted, kept whole for the record"

  - name: Verify on Held-Out Views
    key: verify_holdout
    python: phases.verify_holdout
    depends_on: [calibrate]
    measurements:
      - name: Held-Out Reprojection RMS
        key: holdout_rms_px
        unit: px
        description: "Reprojection error on 3 poses the fit never saw, with the pose solved from the calibrated model"
        validators:
          - {operator: "<=", expected_value: 0.40}
      - name: Line Straightness
        key: line_straightness_px
        unit: px
        description: "Max deviation from a straight line of an undistorted board row across the full width"
        validators:
          - {operator: "<=", expected_value: 0.50}
      - name: Field of View
        key: hfov_deg
        unit: °
        validators:
          - {operator: ">=", expected_value: 64.0}
          - {operator: "<=", expected_value: 70.0}

  - name: Write EEPROM
    key: write_eeprom
    python: phases.write_eeprom
    depends_on: [verify_holdout]
    measurements:
      - name: EEPROM Verified
        key: eeprom_verified
        description: "Coefficients read back from the module equal what was written"
        validators:
          - {operator: "==", expected_value: true}
      - name: Calibration ID
        key: calibration_id
        validators:
          - {operator: matches, expected_value: "^CAL-[0-9A-F]{8}$"}

teardown:
  - name: Release Module
    key: release_module
    python: phases.release_module
    measurements:
      - name: Stage Homed
        key: stage_homed
        validators:
          - {operator: "==", expected_value: true}
```

Framework features to notice:

1. **Setup stamps the unit**: `connect_module` reads the module and writes `unit.metadata["sensor"]` and `["lens"]`, declared under `unit.metadata` in the YAML, so a run can be filtered by lens batch later.
2. **Progress over a long phase**: the `capture_progress` component is bound to nothing but updated from the phase (`ui.capture_progress = ...`) as the stage walks the 16 poses.
3. **Multi-dimensional measurements with custom aggregations**: `corners_per_view` validates its `min_corners`, `view_error` its `max_px`; the aggregation names are the phase's own, computed in Python and checked in YAML.
4. **One fit, nine validated numbers**: fx, fy, cx, cy, k1, k2, k3, p1, p2 each carry their own two-sided limit, so the run page says which coefficient a bad module failed on, and the camera matrix is also kept whole as a JSON measurement.
5. **String and boolean validators**: `resolution` compared as `"1920x1080"` with `==`, `module_serial` and `calibration_id` with `matches`, `eeprom_verified` as a boolean. One thing learned on the way: an array measurement compared with `==` to `[1920, 1080]` fails because the recorded integers come back as floats; a string is the safe form.

### Capture Views

The stage yields 16 calibration poses walking the board centre over a 4 by 4 grid of the field, plus 3 random held-out poses. At each, the module returns the corner detections; the phase counts them, tracks which sensor cells have seen a corner, and advances the progress bar. The detections go to the next phases through a file, not through the measurement record, since a measurement is not the place for 700 corner coordinates:

```python filename="phases/capture_views.py"
import json

from utils.board import IMAGE_SIZE

N_VIEWS = 16
N_HOLDOUT = 3
GRID = (6, 4)  # sensor cells for the coverage check


def capture_views(measurements, camera, stage, ui, log):
    """Drive the target through 16 calibration poses plus 3 held-out poses, capture the detections at each."""
    poses = stage.poses(N_VIEWS + N_HOLDOUT)
    views = []
    counts = []
    seen = set()
    w, h = IMAGE_SIZE
    for i, pose in enumerate(poses):
        stage.move_to(i)
        det = camera.capture(pose["rvec"], pose["tvec"])
        views.append({"pose": i, "ids": det["ids"], "corners": det["corners"]})
        if i < N_VIEWS:
            counts.append(len(det["ids"]))
            for x, y in det["corners"]:
                seen.add((int(x * GRID[0] / w), int(y * GRID[1] / h)))
            ui.capture_progress = int(100 * (i + 1) / N_VIEWS)

    coverage = 100.0 * len(seen) / (GRID[0] * GRID[1])
    log.info(f"{N_VIEWS} calibration poses, {min(counts)}-{max(counts)} corners each, {coverage:.0f} % sensor coverage, {N_HOLDOUT} held-out poses")
    measurements.views_captured = N_VIEWS
    measurements.corners_per_view.x_axis = list(range(1, N_VIEWS + 1))
    measurements.corners_per_view.y_axis.corners = counts
    measurements.corners_per_view.y_axis.corners.aggregations.min_corners = min(counts)
    measurements.coverage_pct = coverage

    # the detections go to the next phases through a file, not through the measurement record
    with open("views.json", "w") as f:
        json.dump({"calibration": views[:N_VIEWS], "holdout": views[N_VIEWS:]}, f)
```

On the mock the poses yield 23 to 35 corners each and 100 % coverage. A pose that loses corners to the border still counts, that is what the ChArUco IDs are for; a pose under 20 corners fails `min_corners` and points at glare or a blurred frame.

### Calibrate

`cv2.calibrateCamera` takes the object points of the detected corners (board frame, metres) and their image points for all 16 poses and returns the intrinsic matrix, the distortion vector, and the pose of every view. The phase then reprojects each view with the fitted model to get the per-pose error, records every coefficient, and attaches the calibration file:

```python filename="phases/calibrate.py"
import json

import cv2
import numpy as np

from utils.board import IMAGE_SIZE, object_points


def calibrate(measurements, attach, log):
    """Fit the pinhole model and five distortion coefficients to the 16 calibration views."""
    with open("views.json") as f:
        views = json.load(f)["calibration"]
    board = object_points()
    obj_pts = [board[v["ids"]] for v in views]
    img_pts = [np.asarray(v["corners"], dtype=np.float32).reshape(-1, 1, 2) for v in views]

    rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(obj_pts, img_pts, IMAGE_SIZE, None, None)
    dist = dist.ravel()

    per_view = []
    for o, i, r, t in zip(obj_pts, img_pts, rvecs, tvecs):
        proj, _ = cv2.projectPoints(o, r, t, K, dist)
        per_view.append(float(np.sqrt(np.mean(np.sum((proj - i) ** 2, axis=2)))))

    fx, fy, cx, cy = float(K[0, 0]), float(K[1, 1]), float(K[0, 2]), float(K[1, 2])
    k1, k2, p1, p2, k3 = (float(x) for x in dist[:5])
    log.info(f"RMS {rms:.3f} px, fx {fx:.1f} fy {fy:.1f} cx {cx:.1f} cy {cy:.1f}, k1 {k1:.4f} k2 {k2:.4f} k3 {k3:.4f} p1 {p1:.5f} p2 {p2:.5f}")

    measurements.rms_px = float(rms)
    measurements.view_error.x_axis = list(range(1, len(per_view) + 1))
    measurements.view_error.y_axis.error = per_view
    measurements.view_error.y_axis.error.aggregations.max_px = max(per_view)
    measurements.fx = fx
    measurements.fy = fy
    measurements.pixel_aspect = fy / fx
    measurements.cx = cx
    measurements.cy = cy
    measurements.k1 = k1
    measurements.k2 = k2
    measurements.k3 = k3
    measurements.p1 = p1
    measurements.p2 = p2
    measurements.camera_matrix = {"fx": round(fx, 3), "fy": round(fy, 3), "cx": round(cx, 3), "cy": round(cy, 3), "skew": 0.0}

    calibration = {
        "model": "opencv_pinhole_5",
        "image_size": list(IMAGE_SIZE),
        "camera_matrix": [[fx, 0.0, cx], [0.0, fy, cy], [0.0, 0.0, 1.0]],
        "distortion": [k1, k2, p1, p2, k3],
        "rms_px": float(rms),
        "views": len(views),
    }
    with open("calibration.json", "w") as f:
        json.dump(calibration, f, indent=2)
    attach.file("calibration.json", "calibration.json")
```

![Bar chart of the reprojection error per pose after the fit, all around 0.2 px against the 0.3 px overall and 0.5 px per-pose limits.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-7c56304f-26c1-4283-b9ac-b804397b65f4-siB4y23knCEEYvyYpBd8xLcsw6Sz1y.png)

The per-pose errors of the mock run: 0.18 to 0.24 px on every pose, overall RMS 0.209 px, with 0.15 px of detector noise built into the mock. A single pose at 0.5 px with the others at 0.2 px is a bad frame, not a bad module; the `max_px` aggregation fails that case explicitly.

![Radial displacement versus distance from the principal point: the fitted barrel distortion model lies on top of the module's true optics, 144 px at the sensor corner.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-3a8955b5-4a37-4ac3-a1b8-4e3bf7746e78-40Y58jj1iGWZtkxu2RZN4oN8shZoyd.png)

What the fit recovered against the optics the mock hides: fitted k1 −0.283 for a true −0.284, fx 1452.5 px for 1452.0, cx 963.4 for 963.5. At the sensor corner the barrel distortion moves a point 144 px inward; the fitted curve sits on top of the true one along the whole radius.

### Verify on Held-Out Views

A fit can look good on its own data and still be wrong, especially with too few poses or poor coverage. The three held-out poses were captured but never given to the fit; their pose is solved with `solvePnP` from the calibrated model and their corners reprojected. The same phase undistorts each complete board row and measures how far it strays from a straight line, and derives the field of view from fx:

```python filename="phases/verify_holdout.py"
def verify_holdout(measurements, log):
    """Check the fitted model on poses the fit never saw, and that straight lines come out straight."""
    with open("views.json") as f:
        holdout = json.load(f)["holdout"]
    with open("calibration.json") as f:
        cal = json.load(f)
    K = np.asarray(cal["camera_matrix"])
    dist = np.asarray(cal["distortion"])
    board = object_points()

    errors = []
    straightness = []
    for v in holdout:
        ids = np.asarray(v["ids"])
        obj = board[ids].astype(np.float64)
        img = np.asarray(v["corners"], dtype=np.float64)
        ok, rvec, tvec = cv2.solvePnP(obj, img, K, dist)
        proj, _ = cv2.projectPoints(obj, rvec, tvec, K, dist)
        errors.append(float(np.sqrt(np.mean(np.sum((proj.reshape(-1, 2) - img) ** 2, axis=1)))))

        # undistort each complete board row and measure how far it strays from a straight line
        undist = cv2.undistortPoints(img.reshape(-1, 1, 2), K, dist, P=K).reshape(-1, 2)
        by_id = {int(i): p for i, p in zip(ids, undist)}
        for row in range(5):
            pts = [by_id[i] for i in row_ids(row) if i in by_id]
            if len(pts) == CORNERS_X:
                pts = np.asarray(pts)
                a, b = np.polyfit(pts[:, 0], pts[:, 1], 1)
                straightness.append(float(np.max(np.abs(pts[:, 1] - (a * pts[:, 0] + b))) / math.sqrt(1 + a * a)))

    hfov = math.degrees(2 * math.atan(IMAGE_SIZE[0] / (2 * K[0, 0])))
    log.info(f"held-out RMS {max(errors):.3f} px over {len(holdout)} poses, straightness {max(straightness):.3f} px, HFOV {hfov:.1f} deg")
    measurements.holdout_rms_px = max(errors)
    measurements.line_straightness_px = max(straightness)
    measurements.hfov_deg = hfov
```

On the mock: held-out RMS 0.223 px, straightness 0.318 px, HFOV 66.9°. A model that overfits the calibration poses shows up here as a held-out error well above the fit RMS.

### Write EEPROM

The coefficients are rounded to what the EEPROM format holds, written, read back, and compared as whole objects. The CRC of the payload becomes the calibration ID, on the run and on the unit:

```python filename="phases/write_eeprom.py"
def write_eeprom(measurements, unit, camera, log):
    """Write the coefficients to the module, read them back, compare byte for byte."""
    with open("calibration.json") as f:
        cal = json.load(f)
    payload = {
        "model": cal["model"],
        "camera_matrix": [[round(x, 4) for x in row] for row in cal["camera_matrix"]],
        "distortion": [round(x, 6) for x in cal["distortion"]],
    }
    calibration_id = camera.write_calibration(payload)
    readback = camera.read_calibration()
    verified = readback == payload
    log.info(f"EEPROM {calibration_id}: {'verified' if verified else 'MISMATCH'}")
    measurements.eeprom_verified = verified
    measurements.calibration_id = calibration_id
    unit.metadata["calibration_id"] = calibration_id
```

### Mock Plugs

The module mock hides a true model and answers `capture(rvec, tvec)` with the corner IDs inside the sensor and their projected positions plus 0.15 px of detector noise; the procedure never reads the true model. The stage mock generates the poses:

```python filename="plugs/camera.py"
# The module's real optics, unknown to the procedure.
K_TRUE = np.array([[1452.0, 0.0, 963.5], [0.0, 1449.0, 545.2], [0.0, 0.0, 1.0]])
DIST_TRUE = np.array([-0.284, 0.112, 0.0006, -0.0004, -0.021])  # k1 k2 p1 p2 k3
NOISE_PX = 0.15

    def capture(self, rvec, tvec) -> dict:
        """Detections for the board at pose (rvec, tvec) in the camera frame. Positional args, plain lists."""
        rvec = np.asarray(rvec, dtype=np.float64).reshape(3, 1)
        tvec = np.asarray(tvec, dtype=np.float64).reshape(3, 1)
        img_pts, _ = cv2.projectPoints(self._obj.astype(np.float64), rvec, tvec, K_TRUE, DIST_TRUE)
        img_pts = img_pts.reshape(-1, 2) + self._rng.normal(0.0, NOISE_PX, (len(self._obj), 2))
        w, h = IMAGE_SIZE
        inside = (img_pts[:, 0] >= 8) & (img_pts[:, 0] <= w - 8) & (img_pts[:, 1] >= 8) & (img_pts[:, 1] <= h - 8)
        ids = np.nonzero(inside)[0]
        return {"ids": ids.tolist(), "corners": img_pts[ids].round(4).tolist()}
```

Plug calls cross a JSON boundary, so `rvec` and `tvec` travel as plain lists and the detections come back as lists, not arrays. On a real station, `capture()` grabs a frame from the module and runs `cv2.aruco.CharucoDetector` on it, returning the same `{ids, corners}`; `write_calibration` and `read_calibration` become the module's I2C EEPROM transactions; the stage plug drives the real tilt and slide axes. The phases, the measurements and the limits do not change.


### BMS Daisy-Chain Signal Integrity
URL: https://www.tofupilot.com/templates/bms-daisy-chain-signal-integrity
Daisy-chain integrity of a cell-monitoring module: isolation caps, wake and address, pulse amplitude on a jumper and a 3 m harness, frame loss vs common mode.


## Introduction

### Daisy-Chain Overview

A pack of 400 V is monitored by a chain of cell-monitoring modules, each floating at its own stack potential, each talking to the next over one isolated twisted pair. TI's BQ79616 family runs a bidirectional, half-duplex differential link at **2 MHz** through **2.2 nF isolation capacitors** or a pulse transformer, reclocked at every device; ADI's isoSPI does the same at 1 Mbps through a transformer with the drive current and the receive threshold both set by one resistor divider. Both links have a property that makes them the classic escape of a BMS production line: the defects that matter are **analog margin defects**, and they pass a protocol test on a 20 cm fixture jumper and fail on the pack harness.

![The edge of a cell-monitoring PCBA: the two-pin daisy-chain connector with a twisted pair plugged in, four ceramic isolation capacitors in a row beside it, a chip further in.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-a07aa346-4833-46d5-bf35-b23918f91f62-pINHHmcf3nvG0gcBZFUhPxlR1XlKyR.jpg)

The daisy-chain port of one module: the two-pin connector, the twisted pair, and the four series isolation capacitors that set how much of the 2 MHz pulse reaches the next module.

The numbers to build the test on are in [TI SLVAEP4](https://www.ti.com/lit/pdf/slvaep4), the daisy-chain application note: data pulses of **250 ns ± 20 ns**, tones of **1 µs ± 8 %**, a good waveform **above 1.8 V differential**, and a bit that is not detected at all when its **starting amplitude is below 1.6 V**, whatever its ending amplitude. A wrong isolation capacitor (1 nF where 2.2 nF belongs), a missing common-mode choke, a transformer from a different reel on one side of the link, or on isoSPI a wrong IBIAS/ICMP divider that raises the threshold while leaving the drive alone: each one leaves the bench amplitude comfortably above 1.8 V and the pack amplitude below 1.6 V. A test that asks "did it communicate?" over a jumper answers yes to all of them.

### Test Purpose

The procedure records one link fingerprint per module:

- **The four series isolation capacitors**, measured on an LCR meter before the module is powered
- **Wake and auto-address** over the jumper, the **reclock delay** and the **idle common-mode** voltage
- **Pulse amplitude** (start and end of the worst double-bit) and **pulse timing** (data and tone) on the 20 cm jumper and again on a **3 m worst-case harness**
- **Frame loss** over 1000 broadcast reads on the long harness at -15, 0 and +15 V of common-mode offset
- The **fault summary** before and after the stress

![Differential pulse amplitude against harness length on a decaying curve: the 0.2 m jumper point at 2.88 V, the 3.0 m harness point at 2.29 V, the 1.8 V good line and the 1.6 V cliff below; the curve crosses the cliff near 7.5 m.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-c3a5679c-0aa1-40e9-9895-686e8fb71ffa-C6cpQScsNevEQLgjzDcLIwPUICxcjQ.png)

The mock module on its attenuation curve: 2.9 V on the jumper, 2.3 V on the 3 m harness, the cliff at 7.5 m. A module with a 1 nF capacitor sits on a curve half as high, and its jumper reading would still clear 1.8 V.

The framework mechanics on show are a component check in `setup:` before the DUT is powered, the same measurement recorded on two fixtures with the limit where it matters, limits copied from the application note, a multi-dimensional measurement with an integer aggregation validated `== 0`, JSON `==` on the fault summary before and after stress, and a `teardown:` that restores the fixture.

## Equipment & Setup

To run this test on a production line, the following are required:

- **A host bridge** for the chain (a BQ79600 on a UART adapter, or an LTC6820 for isoSPI)
- **A relay-switched harness**: a 20 cm jumper and a real 3 m twisted pair in the fixture
- **A differential probe and a digitiser** at 1 GS/s or better on the module's COM pins
- **An isolated DC source** in series with the harness return, for the common-mode offset
- **An LCR meter** behind the probe head for the isolation capacitors
- **The Device Under Test (DUT)**: one cell-monitoring module with its production firmware
- **A TofuPilot Framework procedure** to switch the fixtures, capture the margins and validate the limits
- **The TofuPilot Dashboard** to trend the harness amplitude across modules, which is where a capacitor reel change shows first

### Hardware Components

#### Bridge, Harness and Probe

The [BQ79600-Q1](https://www.ti.com/product/BQ79600-Q1) bridges a UART or SPI host to the daisy chain and supports mixed isolation on the two sides. The harness relays are the fixture's own; the 3 m pair is the worst case of the pack's inter-module runs, with the isolation network of the neighbouring module at its far end. Any differential probe with 1 GHz of bandwidth on a Keysight or Tektronix scope resolves a 250 ns pulse and its 20 ns window; the capture is a single-shot on the COMH pair during a broadcast read.

![Cell-monitoring board and host bridge connected by a twisted-pair harness in a probe fixture](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/bms-daisy-chain-signal-integrity-distinct-2026-O55BQ4QPh4POHunKXolxIxqpGbrufa.webp)

A host bridge connects to the cell-monitoring board through a twisted-pair harness. A supported probe contacts the communication test pads; switching and measurement instruments are external.

#### Isolation Network

Capacitor isolation is the low-cost option: **2.2 nF** rated at twice the working voltage, up to 5.6 nF; a common-mode choke (TDK ACT45B-510 or -101, Würth 744242471) is added for longer runs or noisier packs; a pulse transformer is the robust option, with the same part number on both sides of every link. The setup phase measures the four series capacitors on an LCR meter at 1 kHz through the probe head, before the chain is powered, because a wrong value is a component defect and a component check finds it in milliseconds; the margin phases then confirm what the capacitor did to the pulse.

#### Common-Mode Source

The receiver is designed to survive **±20 V of common mode** between modules (inverter noise, bulk current injection); above that the ESD structures clamp. An isolated supply in series with the harness return applies ±15 V during the frame-loss count. A missing choke or a marginal capacitor shows as loss at ±15 V before it shows at 0 V.

### Where the Limits Come From

No standard specifies a daisy-chain production test; ISO 11898 and its conformance plans cover CAN, not this link, and ISO 26262 asks that the safety-related communication be controlled in production without naming a voltage. The limits are the application note's:

| Test | Limit | Basis |
|---|---|---|
| Isolation capacitors | 1.8 to 2.6 nF | 2.2 nF ± 10 % plus the meter |
| Data pulse width | 230 to 270 ns | 250 ns ± 20 ns |
| Tone pulse width | 0.92 to 1.08 µs | 1 µs ± 8 % |
| Pulse starting amplitude | ≥ 1.8 V, on the harness | good waveform; the bit is lost below 1.6 V |
| Idle common mode | 2.21 to 2.76 V | V(DCCM) |
| Reclock delay | ≤ 5 µs | 4 µs typical, 5 µs maximum per device |
| Frame loss at ±15 V | 0 of 1000 | ±20 V survival |

The amplitude limit is applied on the harness and only recorded on the jumper: every module passes on the jumper, so a limit there catches nothing. The 3 m figure is this pack's worst inter-module run; the fixture carries the real length, not a calculation.

## Test Procedure

### Overview

The procedure maps the test onto the framework's three stages. The capacitor check lives in `setup:` so the chain is never powered on a module with a wrong isolation network. Restoring the jumper and the common mode lives in `teardown:` so the next module starts from a known fixture.

1. Setup: four isolation capacitors on the LCR meter.
2. Main: wake, auto-address, reclock delay, idle common mode, fault summary, on the jumper.
3. Main: pulse amplitude and timing on the jumper, then on the 3 m harness.
4. Main: frame loss at -15, 0 and +15 V on the harness, fault summary after.
5. Teardown: jumper back, common mode off.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for trending.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  isolation_caps.py
  wake_address.py
  pulse_margin.py
  common_mode_sweep.py
  release.py
plugs/
  chain_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-bms-daisy-chain-integrity). The `ChainBench` plug is a mock of the bridge, the harness relays, the probe, the common-mode source and the LCR meter together, synthesizing a healthy module with an attenuation of 0.72 dB/m at 2 MHz, so the procedure runs end-to-end without a bench or a module connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: BMS Daisy-Chain Signal Integrity
version: 0.1.0
description: Daisy-chain communication integrity of a cell-monitoring module. Isolation capacitors measured before power, wake and auto-address with reclock delay, differential pulse amplitude and timing on a 20 cm jumper and on a 3 m worst-case harness, frame loss at three common-mode offsets.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the module's data-matrix"
    placeholder: "CMU-000000"
    pattern: "^CMU-\\d{6}$"
    default_value: "CMU-031742"
  part_number:
    default_value: "PCB-CMU-16S-R2"
  batch_number:
    default_value: "LOT-2026-37"

plugs:
  - name: Chain Bench
    description: "Host bridge, switchable harness, differential probe and digitiser, common-mode source, LCR meter (mock, one plug per bench)"
    python: plugs.chain_bench:ChainBench
    key: bench

setup:
  - name: Isolation Caps
    key: isolation_caps
    python: phases.isolation_caps
    measurements:
      - name: Cap COMH P
        key: cap_comh_p_nf
        unit: nF
        validators:
          - {operator: ">=", expected_value: 1.8}
          - {operator: "<=", expected_value: 2.6}
      - name: Cap COMH N
        key: cap_comh_n_nf
        unit: nF
        validators:
          - {operator: ">=", expected_value: 1.8}
          - {operator: "<=", expected_value: 2.6}
      - name: Cap COML P
        key: cap_coml_p_nf
        unit: nF
        validators:
          - {operator: ">=", expected_value: 1.8}
          - {operator: "<=", expected_value: 2.6}
      - name: Cap COML N
        key: cap_coml_n_nf
        unit: nF
        validators:
          - {operator: ">=", expected_value: 1.8}
          - {operator: "<=", expected_value: 2.6}

main:
  - name: Wake Address
    key: wake_address
    python: phases.wake_address
    measurements:
      - name: Address
        key: address
        validators:
          - {operator: "==", expected_value: 1}
      - name: Reclock Delay
        key: reclock_us
        unit: µs
        description: Per-device reclock, 4 us typical, 5 us maximum; a slow reclock on one module adds up over a 24-device stack.
        validators:
          - {operator: "<=", expected_value: 5.0}
      - name: Idle Common Mode
        key: idle_common_mode_v
        unit: V
        description: V(DCCM) 2.21 to 2.76 V; the COM pins rest at 2.5 V between packets.
        validators:
          - {operator: ">=", expected_value: 2.21}
          - {operator: "<=", expected_value: 2.76}
      - name: Faults
        key: faults
        validators:
          - operator: "=="
            expected_value:
              comm: false
              otp: false
              sys: false

  - name: Pulse Margin
    key: pulse_margin
    python: phases.pulse_margin
    depends_on: [wake_address]
    measurements:
      - name: Jumper Start Amplitude
        key: jumper_start_v
        unit: V
        validators:
          - {operator: ">=", expected_value: 1.8}
      - name: Jumper End Amplitude
        key: jumper_end_v
        unit: V
      - name: Jumper Data Width
        key: jumper_data_width_ns
        unit: ns
        validators:
          - {operator: ">=", expected_value: 230.0}
          - {operator: "<=", expected_value: 270.0}
      - name: Jumper Tone Width
        key: jumper_tone_width_us
        unit: µs
        validators:
          - {operator: ">=", expected_value: 0.92}
          - {operator: "<=", expected_value: 1.08}
      - name: Harness Start Amplitude
        key: harness_start_v
        unit: V
        description: "On the 3 m worst-case harness. Good above 1.8 V; the receiver drops the bit below 1.6 V starting amplitude. This is the number the jumper cannot give you."
        validators:
          - {operator: ">=", expected_value: 1.8}
      - name: Harness End Amplitude
        key: harness_end_v
        unit: V
        description: Ending amplitude of the worst double-bit; recorded for trending, the start amplitude carries the limit.
      - name: Harness Data Width
        key: harness_data_width_ns
        unit: ns
        validators:
          - {operator: ">=", expected_value: 230.0}
          - {operator: "<=", expected_value: 270.0}
      - name: Harness Tone Width
        key: harness_tone_width_us
        unit: µs
        validators:
          - {operator: ">=", expected_value: 0.92}
          - {operator: "<=", expected_value: 1.08}

  - name: Common Mode Sweep
    key: common_mode_sweep
    python: phases.common_mode_sweep
    depends_on: [pulse_margin]
    timeout: 2m
    measurements:
      - name: Frame Loss
        key: frame_loss
        title: Frames lost of 1000 on the 3 m harness at each common-mode offset
        x_axis:
          legend: Common mode
          unit: V
        y_axis:
          - legend: Lost frames
            key: lost
            aggregations:
              - type: total
                validators:
                  - {operator: "==", expected_value: 0}
      - name: Faults After Sweep
        key: faults_after_sweep
        validators:
          - operator: "=="
            expected_value:
              comm: false
              otp: false
              sys: false

teardown:
  - name: Release
    key: release
    python: phases.release
```

Framework features to notice:

1. **A component check before power.** The setup phase measures four capacitors with the DUT unpowered; a wrong reel fails here with the value, not three phases later with a lost bit.
2. **Two fixtures, one limit.** `jumper_start_v` and `harness_start_v` are the same capture on two harnesses; the limit lives on the harness one and the jumper one is recorded for the trend.
3. **Integer aggregation `== 0`.** The frame-loss curve carries a `total` aggregation validated as an exact zero.
4. **Fault summary before and after.** `faults` and `faults_after_sweep` compare the same object twice; a communication fault raised by the stress is caught even when every frame came back.
5. **`teardown:` restores the fixture.** Jumper back and common mode off whatever the outcome.

### Isolation Caps

The setup phase reads the four series capacitors through the probe head with the module unpowered and records the isolation type on the **unit metadata**:

```python filename="phases/isolation_caps.py"
from utils.recipe import ISOLATION_CAP_NF


def isolation_caps(measurements, bench, unit, log):
    """Setup: the four series isolation capacitors measured on the LCR
    meter through the probe head before the chain is powered. A 1 nF part
    where 2.2 nF belongs passes every protocol test on the jumper and
    fails at pack length; it is a component check, not a comms check."""
    caps = {pin: bench.isolation_cap_nf(pin) for pin in ("comh_p", "comh_n", "coml_p", "coml_n")}
    measurements.cap_comh_p_nf = caps["comh_p"]
    measurements.cap_comh_n_nf = caps["comh_n"]
    measurements.cap_coml_p_nf = caps["coml_p"]
    measurements.cap_coml_n_nf = caps["coml_n"]
    unit.metadata["isolation"] = "capacitive 2.2 nF"
    log.info(f"Module {unit.serial_number}: isolation caps {caps} nF")
```

### Wake Address

The bridge wakes the chain with the tone, auto-addresses, and the phase reads back the address the module took, the reclock delay, the idle common-mode voltage on the COM pins and the fault summary. This is the protocol-level test every station already does; here it is the gate for the margin phases, not the verdict:

```python filename="phases/wake_address.py"
from utils.recipe import JUMPER_M


def wake_address(measurements, bench, log):
    """Wake the chain over the jumper, auto-address, read the reclock
    delay and the fault summary. The protocol-level test, and the one
    every station already does."""
    bench.set_harness(JUMPER_M)
    bench.set_common_mode_v(0.0)
    res = bench.wake_and_autoaddress()
    measurements.address = res["address"]
    measurements.reclock_us = res["reclock_us"]
    measurements.idle_common_mode_v = bench.idle_common_mode_v()
    measurements.faults = bench.fault_summary()
    log.info(f"Address {res['address']}, reclock {res['reclock_us']:.2f} us, idle common mode {measurements.idle_common_mode_v} V")
```

### Pulse Margin

The same single-shot capture on both harnesses: the differential probe on COMH during a broadcast read, the worst double-bit found (two adjacent pulses of opposite polarity, the second starting from where the first drooped), its starting and ending amplitude and its width, plus the width of a tone. The harness values carry the amplitude limit:

```python filename="phases/pulse_margin.py"
from utils.recipe import JUMPER_M, WORST_CASE_HARNESS_M


def pulse_margin(measurements, bench, log):
    """Differential pulse amplitude and timing on the jumper and on the
    worst-case harness. The amplitude on the long harness is the number
    that predicts field failures: below 1.6 V starting amplitude the
    receiver drops the bit, whatever the protocol test on the bench said."""
    for metres, key in ((JUMPER_M, "jumper"), (WORST_CASE_HARNESS_M, "harness")):
        bench.set_harness(metres)
        bench.wake_and_autoaddress()
        pulse = bench.capture_data_pulse()
        tone = bench.capture_tone()
        setattr(measurements, f"{key}_start_v", pulse["start_v"])
        setattr(measurements, f"{key}_end_v", pulse["end_v"])
        setattr(measurements, f"{key}_data_width_ns", pulse["width_ns"])
        setattr(measurements, f"{key}_tone_width_us", tone["width_us"])
        log.info(f"{metres:.1f} m: pulse {pulse['start_v']:.2f} -> {pulse['end_v']:.2f} V, data {pulse['width_ns']:.0f} ns, tone {tone['width_us']:.3f} us")
    bench.set_harness(JUMPER_M)
```

![Two panels of the worst double-bit: on the 0.2 m jumper a positive pulse from 2.88 V drooping to 2.60 V then a negative one, on the 3.0 m harness the same shape from 2.29 V to 1.83 V; the 1.8 V good line and 1.6 V cliff drawn on both, a gray band marking the 250 ±20 ns window.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-7e9eb816-8308-4bb3-9e22-3597ac8d8c82-TkhSKb4T1CzpDfV4BDtMFmgfl9zEZE.png)

The double-bit on both fixtures. On the harness the second pulse starts at 1.83 V, above the 1.8 V line by 30 mV; that margin is what the trend across a batch watches, and it is invisible on the jumper.

### Common Mode Sweep

On the 3 m harness, 1000 broadcast reads at -15 V, 0 V and +15 V of common-mode offset between the module and the bridge, the frames that came back without a valid CRC counted at each step:

```python filename="phases/common_mode_sweep.py"
from utils.recipe import COMMON_MODE_STEPS_V, FRAMES, WORST_CASE_HARNESS_M


def common_mode_sweep(measurements, bench, log):
    """Frame loss over 1000 broadcast reads on the worst-case harness at
    three common-mode offsets. The receiver is specified to survive +-20 V
    of common mode; a missing choke or a wrong cap shows as loss at +-15 V
    before it shows at 0 V."""
    bench.set_harness(WORST_CASE_HARNESS_M)
    bench.wake_and_autoaddress()
    losses = []
    for cm in COMMON_MODE_STEPS_V:
        bench.set_common_mode_v(cm)
        lost = bench.frame_loss(FRAMES)
        losses.append(lost)
        log.info(f"Common mode {cm:+.0f} V: {lost} of {FRAMES} frames lost")
    bench.set_common_mode_v(0.0)
    bench.set_harness(0.2)

    measurements.frame_loss.x_axis = COMMON_MODE_STEPS_V
    measurements.frame_loss.y_axis.lost = losses
    measurements.frame_loss.y_axis.lost.aggregations.total = int(sum(losses))
    measurements.faults_after_sweep = bench.fault_summary()
```

### Mock Plug

`ChainBench` stands in for five instruments because the pulse the probe sees depends on the harness the relays selected and the common mode the source applied. It synthesizes four capacitors between 2.18 and 2.26 nF, a 2.94 V drive attenuated at 0.72 dB/m, a double-bit droop of 0.28 V plus 0.06 V per metre, data pulses of 251 ns plus 1.8 ns per metre, tones of 1.003 µs, a 4.1 µs reclock, and zero frame loss while the amplitude margin over the 1.6 V cliff exceeds 0.3 V and the common mode stays inside ±20 V. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, the class speaks the bridge's protocol over pyserial, drives the harness relays and the common-mode source, and pulls the single-shot capture from the scope over SCPI (`:MEAS:VAMP`, `:MEAS:PWID` on the differential channel). For an isoSPI link the same phases apply with the LTC6820's ~150 ns CS pulses and ~50 ns data pulses, and the divider on IBIAS/ICMP becomes the component checked in setup. Keep the worst-case harness in the fixture; it is the cheapest countermeasure to the whole defect class. The phases, measurements and limits stay the same.


### Weld and Busbar Resistance
URL: https://www.tofupilot.com/templates/weld-and-busbar-resistance
Post-weld inspection of a 12S4P battery module: 48 laser welds and 11 busbar links read 4-wire at 1 A with OVC, corrected to 20 °C, validated on max and spread.


## Introduction

### Weld Resistance Overview

A battery module is a few hundred laser welds in series and parallel. Each cell-to-busbar joint carries the pack current, and each one that came out of the laser with insufficient penetration, a crack, porosity or a contaminated interface has a higher resistance than its neighbours. At 100 A that difference is heat, and the joint with the highest resistance is the hottest point of the module for the rest of its life; the defect population that becomes a thermal event in the field starts here. A production module has more than forty thousand of these joints per pack on a large EV platform, so the inspection has to be automated, fast, and precise enough to separate a good weld from a marginal one at a few tens of micro-ohms.

![Four cylindrical cells under one copper busbar with four weld spots, a four-probe spring contact head touching one of the welds.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-43815804-ad02-42f5-ad4a-1340c3bf7cdf-h4hIIntBndckdKZExX5sxP3fa5yBgD.jpg)

One busbar of the module: four cells, four laser weld spots, and the four-wire probe head on one of them. Two probes force the current through the joint, two read the drop.

The numbers are small. Hioki measured a real spot weld at **86.1 µΩ**, repeatable to 85.7 to 86.2 µΩ once the temperature was corrected (see [Hioki's weld resistance application note](https://www.hioki.com/global/products/resistance-meters/dc-resistance/id_6006)). A good joint and a shallow one differ by tens of micro-ohms; a two-wire measurement is off by the probe and cable resistance, a thousand times the signal. Two more effects of the same order sit on top: the joint is still cooling when it is read (copper drifts 0.393 % per kelvin, so a joint at 40 °C reads 8 % high), and a copper-to-aluminium joint is a thermocouple whose EMF adds to the voltage drop the meter reads. The measurement is four-wire, temperature-corrected, and taken with a bipolar test current that cancels the thermoelectric voltage, or it is not a measurement.

### Test Purpose

The procedure records one interconnect fingerprint per module:

- **Probe path** resistance and the contact of the reference probe pair, before any weld is read
- **All 48 cell-to-busbar welds**: raw resistance, joint temperature and the value corrected to 20 °C, with the maximum, the median and the spread validated
- **The correction itself**, verified on an unwelded reference section of the busbar read at the same moment
- **The 11 busbar links** of the series string and their sum, the module's interconnect resistance for the pack model
- An optional **OVC verification** of how much the thermoelectric EMF would have shifted the readings

![Top, 48 corrected weld resistances as green bars around the 85.9 µΩ median with joint 30 at 140 µΩ in orange, the raw readings as gray dots slightly above each bar, the 200 µΩ limit line at the top. Bottom, the joint temperature rising from 28 °C on busbar 1 to 40 °C on busbar 12.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-b25de8ab-5d8b-449a-a624-af068bd375e3-bjZAoa7Npg4YN3hnTLzc0gMr54VqPd.png)

The mock module 10 to 25 s out of the laser: raw readings run 3 to 8 % above the corrected bars because the joints are still 8 to 20 K above 20 °C, the gradient along the module follows the weld sequence, and joint 30 sits at 1.6 times the median with shallow penetration, under the absolute limit but exactly what the spread limit and the trend across a batch exist to catch.

The framework mechanics on show are a multi-dimensional measurement with three curves and aggregations validated on one of them, an absolute and a relative limit on the same data, a measurement that validates the correction rather than the part, a disabled phase for a periodic check, `setup` and `teardown` stages, and a sequential `depends_on` chain on one meter.

## Equipment & Setup

To run this inspection at the exit of a laser welding cell, the following are required:

- **A micro-ohm resistance meter** with a 1000 µΩ range at nanoohm resolution, offset-voltage compensation, and a path-resistance tolerance that survives long cables and worn probes
- **A multiplexed four-wire probe head** that reaches every weld and a reference section of each busbar, with a temperature sensor on the head
- **The Device Under Test (DUT)**: a welded 12S4P module, before it goes to the pack line
- **A TofuPilot Framework procedure** to sequence the joints, correct the readings and validate the limits
- **The TofuPilot Dashboard** to trend every joint across modules and lots, because the defect that matters is a drift, not a single failure

### Hardware Components

#### Resistance Meter

The [Hioki RM3546](https://www.hioki.com/global/products/resistance-meters/dc-resistance/id_6006) is built for this: **1000 µΩ range at 1 nΩ resolution, ±0.045 % of reading**, a four-terminal method with a **9 Ω path-resistance tolerance** so a long cable or a worn probe does not invalidate the reading, offset-voltage compensation (OVC) that alternates the polarity of the test current to cancel thermoelectric EMF, and a temperature correction input for the joint's temperature. The [RM3545](https://www.hioki.com/global/products/resistance-meters/dc-resistance/id_6003) goes to 0.006 % and 0.01 µΩ for the lab. Hioki's own weld application measures a welded and an unwelded section simultaneously through an eight-wire probe, which is where the reference-section phase of this template comes from. Keysight's 34420A nanovolt/micro-ohm meter and Keithley's 2182A with a 6220 source cover the same range with two boxes.

![Four spring probes spanning a busbar joint on an insulated cell-module fixture](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/weld-and-busbar-resistance-physical-2026-0yItxSDCMYLam84fQ5geglfCDfgJMT.webp)

Two probe pairs contact opposite sides of the joint under test. The gantry carries the separate measurement leads; the resistance meter is not shown.

#### Probe Head and Multiplexer

Each joint needs four contacts: two to force the current, two to sense the drop, placed so the current path crosses the weld and the sense pair straddles it. The head carries a second probe pair on an unwelded section of the same busbar for the reference reading, and an IR sensor or a contact thermocouple for the joint temperature. A Pickering or Keithley relay multiplexer routes the meter to each joint; the meter's 9 Ω tolerance is what makes a multiplexed head with a few metres of cable work at all.

#### Offset Voltage Compensation

The negative tab of a cylindrical cell is nickel-plated steel, the busbar is copper or aluminium, and the two metals under a temperature gradient generate a thermoelectric voltage of a few microvolts per kelvin. At a 1 A test current, 4 µV is 4 µΩ, five percent of the joint. OVC reads the drop with the current in both directions and takes the half-difference, which cancels the EMF; the mock's disabled `ovc_verification` phase shows how much it matters by reading the module with OVC off.

#### Temperature Correction

The joint is read while it is cooling from the laser. Copper's temperature coefficient is 0.393 %/K, so a reading at 40 °C is 7.9 % above the 20 °C value; without correction the absolute limit moves with the takt time and the ambient. Each joint is corrected from its own temperature, and the correction is then checked against physics: the unwelded reference section of the busbar, read at the same moment and corrected with the same coefficient, has to land on its known 20 °C value. When it does not, the temperature sensor or the coefficient is wrong, and every corrected weld on the module is wrong with it.

### Where the Limits Come From

No standard publishes a weld resistance. The Al/Al ≤ 0.5 mΩ, Cu/Cu ≤ 0.3 mΩ and Cu/Al ≤ 1.0 mΩ figures that circulate have no primary source and sit an order of magnitude above Hioki's measured 86 µΩ. The limits in this template are the mock module's own: a 200 µΩ absolute ceiling on any weld, a 70 to 100 µΩ window on the median (a whole module reading high is a process drift, not 48 bad welds), a 150 % spread of the worst weld over the median, 40 µΩ per busbar link and 350 µΩ for the series string. A customer sets these from the joint geometry and the laser recipe, then tightens them from the distribution the dashboard shows after a few thousand modules.

## Test Procedure

### Overview

The procedure maps the inspection onto the framework's three stages. Probe contact lives in `setup:` so no weld is read through a lifted probe. Parking the head lives in `teardown:` so the next module can be loaded whether the inspection passed or not.

1. Setup: OVC on, probe path resistance, reference section contact.
2. Main: 48 welds, raw and corrected to 20 °C, maximum, median and spread validated.
3. Main: reference section corrected with the same coefficient, must land on its known value.
4. Main: 11 busbar links and the series string resistance.
5. Main (disabled by default): OVC verification, thermoelectric shift with OVC off.
6. Teardown: probe head parked.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-joint trending across modules.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  probe_check.py
  cell_welds.py
  reference_correction.py
  busbar_links.py
  ovc_verification.py
  park_probe.py
plugs/
  microohm_meter.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-weld-busbar-resistance). The `MicroOhmMeter` plug is a mock of the meter and the multiplexed probe head together, synthesizing a module fresh out of the laser cell with joints still warm, one shallow weld among the 48, and a copper-aluminium EMF that OVC cancels, so the procedure runs end-to-end without a meter or a module connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the meter plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Weld and Busbar Resistance
version: 0.1.0
description: Post-weld inspection of a 12S4P battery module. Every cell-to-busbar laser weld and every busbar link read 4-wire at 1 A with offset-voltage compensation, corrected to 20 C from the joint temperature, validated against a cold-weld limit and a spread limit, with the correction itself checked on an unwelded reference section.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the module label as it leaves the laser cell"
    placeholder: "MOD-12S4P-000000"
    pattern: "^MOD-12S4P-\\d{6}$"
    default_value: "MOD-12S4P-018305"
  part_number:
    default_value: "MOD-12S4P-21700"
  batch_number:
    default_value: "LASER-2026-37-B"

plugs:
  - name: Micro-Ohm Meter
    description: "4-wire resistance meter behind a multiplexed probe head, IR sensor on the head (mock Hioki RM3546-class)"
    python: plugs.microohm_meter:MicroOhmMeter
    key: meter

setup:
  - name: Probe Check
    key: probe_check
    python: phases.probe_check
    measurements:
      - name: Probe Path
        key: probe_path_ohm
        unit: Ω
        description: Total path through the four probes and the cable; the meter accepts up to 9 ohm, the fixture is flagged for probe replacement above 4.
        validators:
          - {operator: "<=", expected_value: 4.0}
      - name: Reference Contact
        key: reference_contact_uohm
        unit: µΩ
        description: First read of the unwelded reference section; proves the second probe pair is on the busbar before the welds are read.
        validators:
          - {operator: ">=", expected_value: 40.0}
          - {operator: "<=", expected_value: 90.0}

main:
  - name: Cell Welds
    key: cell_welds
    python: phases.cell_welds
    timeout: 2m
    measurements:
      - name: Welds
        key: welds
        title: Cell-to-busbar weld resistance, raw at joint temperature and corrected to 20 °C
        x_axis:
          legend: Joint
        y_axis:
          - legend: Raw
            key: raw
            unit: µΩ
          - legend: Joint temperature
            key: temp
            unit: °C
          - legend: Corrected to 20 °C
            key: corrected
            unit: µΩ
            aggregations:
              - type: max_uohm
                unit: µΩ
                validators:
                  - {operator: "<=", expected_value: 200.0}
              - type: median_uohm
                unit: µΩ
                validators:
                  - {operator: ">=", expected_value: 70.0}
                  - {operator: "<=", expected_value: 100.0}
              - type: spread_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 150.0}

  - name: Reference Correction
    key: reference_correction
    python: phases.reference_correction
    depends_on: [cell_welds]
    measurements:
      - name: Reference Raw
        key: reference_raw_uohm
        unit: µΩ
      - name: Reference Temperature
        key: reference_temp_c
        unit: °C
      - name: Reference Corrected
        key: reference_corrected_uohm
        unit: µΩ
        description: "The unwelded section is 61.4 uOhm at 20 C by construction: the corrected reading must land within 0.5 % of it, or the temperature correction applied to every weld is wrong."
        validators:
          - {operator: ">=", expected_value: 61.1}
          - {operator: "<=", expected_value: 61.7}

  - name: Busbar Links
    key: busbar_links
    python: phases.busbar_links
    depends_on: [reference_correction]
    measurements:
      - name: Links
        key: links
        title: Busbar-to-busbar link resistance corrected to 20 °C
        x_axis:
          legend: Link
        y_axis:
          - legend: Corrected to 20 °C
            key: corrected
            unit: µΩ
            aggregations:
              - type: max_uohm
                unit: µΩ
                validators:
                  - {operator: "<=", expected_value: 40.0}
      - name: String Resistance
        key: string_resistance_uohm
        unit: µΩ
        description: Sum of the 11 links, the module's series interconnect resistance for the pack model.
        validators:
          - {operator: "<=", expected_value: 350.0}

  - name: OVC Verification
    key: ovc_verification
    python: phases.ovc_verification
    enabled: false
    depends_on: [busbar_links]
    measurements:
      - name: EMF Shift
        key: emf_shift_max_uohm
        unit: µΩ

teardown:
  - name: Park Probe
    key: park_probe
    python: phases.park_probe
```

Framework features to notice:

1. **Three curves, limits on one.** The `welds` measurement records raw, temperature and corrected per joint; the pass/fail lives on three aggregations of the corrected curve, `max_uohm`, `median_uohm` and `spread_pct`, computed in Python and named in the procedure file.
2. **Absolute and relative on the same data.** `max_uohm <= 200` catches a cold weld; `spread_pct <= 150` catches a weld that is fine in absolute terms and wrong relative to its neighbours; `median_uohm` in 70 to 100 catches a whole module drifting.
3. **A measurement that validates the correction.** `reference_corrected_uohm` has nothing to do with the module's quality; it proves that the temperature correction applied to the 48 welds is right, with a 0.5 % window.
4. **`enabled: false`.** The OVC verification ships disabled and runs when a probe head or a meter is swapped; flipping the flag needs no code change.
5. **One meter, one chain.** Every main phase declares `depends_on` on the previous one because they share the meter and the probe head.

### Probe Check

The setup phase switches OVC on, reads the path resistance through the four probes and the cable against the fixture's own replacement threshold (well inside the meter's 9 Ω tolerance), and reads the reference section once to prove the second probe pair is on the busbar. The laser program that produced the module goes onto the **unit metadata**:

```python filename="phases/probe_check.py"
from utils.recipe import TEST_CURRENT_A


def probe_check(measurements, meter, unit, log):
    """Setup: OVC on, probe path resistance inside the meter's tolerance,
    reference section read once to prove the second probe pair makes contact."""
    meter.set_ovc(True)
    path = meter.probe_check()
    measurements.probe_path_ohm = path
    ref = meter.measure_reference(TEST_CURRENT_A)
    measurements.reference_contact_uohm = ref["raw_uohm"]
    unit.metadata["laser_program"] = "LW-12S4P-r7"
    log.info(f"Module {unit.serial_number}: probe path {path:.2f} ohm, reference section {ref['raw_uohm']:.1f} uOhm at {ref['temp_c']:.1f} C")
```

### Cell Welds

One call returns the 48 raw readings and the 48 joint temperatures; the phase corrects each joint to 20 °C with copper's coefficient, records the three curves in one multi-dimensional measurement, and computes the three aggregations the limits sit on:

```python filename="phases/cell_welds.py"
import numpy as np

from utils.recipe import ALPHA_CU_PER_K, JOINTS, REFERENCE_TEMP_C, TEST_CURRENT_A


def cell_welds(measurements, meter, log):
    """All 48 cell-to-busbar welds, 4-wire at 1 A with OVC on, each
    corrected to 20 C from its own joint temperature. Raw and corrected
    curves both recorded; the limits sit on the corrected one."""
    welds = meter.measure_welds(TEST_CURRENT_A)
    raw = np.array(welds["raw_uohm"])
    temp = np.array(welds["temp_c"])
    corrected = raw / (1.0 + ALPHA_CU_PER_K * (temp - REFERENCE_TEMP_C))

    measurements.welds.x_axis = welds["joint"]
    measurements.welds.y_axis.raw = raw.round(2).tolist()
    measurements.welds.y_axis.temp = temp.tolist()
    measurements.welds.y_axis.corrected = corrected.round(2).tolist()
    measurements.welds.y_axis.corrected.aggregations.max_uohm = float(corrected.max())
    measurements.welds.y_axis.corrected.aggregations.median_uohm = float(np.median(corrected))
    measurements.welds.y_axis.corrected.aggregations.spread_pct = float(100.0 * (corrected.max() - np.median(corrected)) / np.median(corrected))

    worst = int(corrected.argmax()) + 1
    log.info(f"{JOINTS} welds: median {np.median(corrected):.1f} uOhm, max {corrected.max():.1f} uOhm at joint {worst}, joints at {temp.min():.0f}..{temp.max():.0f} C")
```

### Reference Correction

The unwelded reference section is 61.4 µΩ at 20 °C by construction. Read at the same moment as the welds and corrected with the same coefficient, it must land within 0.5 % of that. The phase records the raw value, the temperature and the corrected value, and the limit sits on the last one:

```python filename="phases/reference_correction.py"
from utils.recipe import ALPHA_CU_PER_K, REFERENCE_TEMP_C, TEST_CURRENT_A


def reference_correction(measurements, meter, log):
    """Check the temperature correction against physics: the unwelded
    reference section read at the same moment, corrected with the same
    coefficient, must land on its 20 C value. If it does not, the IR sensor
    or the coefficient is wrong and every corrected weld is wrong with it."""
    ref = meter.measure_reference(TEST_CURRENT_A)
    corrected = ref["raw_uohm"] / (1.0 + ALPHA_CU_PER_K * (ref["temp_c"] - REFERENCE_TEMP_C))
    measurements.reference_raw_uohm = ref["raw_uohm"]
    measurements.reference_temp_c = ref["temp_c"]
    measurements.reference_corrected_uohm = float(corrected)
    log.info(f"Reference section {ref['raw_uohm']:.2f} uOhm at {ref['temp_c']:.1f} C, {corrected:.2f} uOhm corrected to 20 C")
```

![Scatter of the 47 healthy welds against their joint temperature from 28 to 41 °C: raw readings in gray climbing along the 0.393 %/K line from 87 to 93 µΩ, the same joints corrected to 20 °C in green scattered flat around 85 µΩ.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-7a640b52-44a3-4361-8bfc-b7d8d01016a5-7YycJN1TWIH6G2wlArksf0CC1uHKjp.png)

What the correction does across the module: the raw readings climb with the joint temperature along copper's coefficient, the corrected values sit flat around 85 µΩ whatever the joint's temperature was when the probe landed. The reference section, 64.92 µΩ at 34.4 °C, corrects to 61.44 µΩ against 61.4 by construction, inside the 0.5 % window.

### Busbar Links

The 11 busbar-to-busbar joints of the series string, same method, corrected per link from the mean temperature of the two busbars they join. The sum is the module's series interconnect resistance, the number the pack's electrical model needs:

```python filename="phases/busbar_links.py"
import numpy as np

from utils.recipe import ALPHA_CU_PER_K, REFERENCE_TEMP_C, TEST_CURRENT_A


def busbar_links(measurements, meter, log):
    """The 11 busbar-to-busbar joints of the series string, same method."""
    links = meter.measure_busbar_links(TEST_CURRENT_A)
    raw = np.array(links["raw_uohm"])
    temp = np.array(links["temp_c"])
    corrected = raw / (1.0 + ALPHA_CU_PER_K * (temp - REFERENCE_TEMP_C))

    measurements.links.x_axis = links["link"]
    measurements.links.y_axis.corrected = corrected.round(2).tolist()
    measurements.links.y_axis.corrected.aggregations.max_uohm = float(corrected.max())
    measurements.string_resistance_uohm = float(corrected.sum())
    log.info(f"Series string: {corrected.sum():.0f} uOhm over {len(raw)} links, max link {corrected.max():.1f} uOhm")
```

![Eleven bars of link resistance between 20 and 25 µΩ under the 40 µΩ limit line, the series string sum of 244 µΩ in the title.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-dd3f2414-b6d2-4394-8f2c-e74d74aa8c88-90txJtHddFlxr2uOWHbHxQvAFVcjrR.png)

The eleven links of the mock module: 20 to 25 µΩ each, 244 µΩ for the string against the 350 µΩ limit. A link that reads double is a busbar seated on a burr or a bolt torqued short.

### OVC Verification

Disabled by default. It reads the welds with OVC off and on, and reports how much the thermoelectric EMF of the copper-nickel-steel joint shifts the readings; on the mock, 4.2 µV/K of Seebeck coefficient at joints 8 to 20 K above ambient gives shifts of tens of micro-ohms, larger than the difference between a good weld and a shallow one. Run it when the probe head or the meter changes:

```python filename="phases/ovc_verification.py"
import numpy as np

from utils.recipe import TEST_CURRENT_A


def ovc_verification(measurements, meter, log):
    """Disabled by default. Reads the welds with OVC off and reports how much
    the copper-aluminium thermoelectric EMF shifts the result; run it when
    the probe head or the meter is changed, not on every module."""
    meter.set_ovc(False)
    no_ovc = np.array(meter.measure_welds(TEST_CURRENT_A)["raw_uohm"])
    meter.set_ovc(True)
    with_ovc = np.array(meter.measure_welds(TEST_CURRENT_A)["raw_uohm"])
    shift = no_ovc - with_ovc
    measurements.emf_shift_max_uohm = float(np.abs(shift).max())
    log.info(f"Thermoelectric EMF shifts the reading by up to {np.abs(shift).max():.1f} uOhm without OVC")
```

### Mock Plug

`MicroOhmMeter` stands in for the meter, the multiplexer and the probe head's temperature sensor. It draws 48 welds around 85 µΩ at 20 °C with a 3.5 µΩ standard deviation, sets joint 30 to 140 µΩ, gives each busbar a temperature from 28 °C on the first to 41 °C on the last (the weld sequence), applies copper's 0.393 %/K to every raw reading, and adds a 4.2 µV/K thermoelectric EMF when OVC is off. The reference section is 61.4 µΩ at 20 °C, read at the mean joint temperature. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real station, the class speaks SCPI to a Hioki RM3546 over pyvisa (`:SENS:RES:RANG 1E-3`, `:SENS:RES:OVC ON`, `:READ?`), steps the multiplexer between joints, and reads the head's temperature sensor for each one. Keep OVC on for every dissimilar-metal joint, read the reference section through the second probe pair at the same moment as the welds, and set the coefficient for the busbar alloy (0.403 %/K for aluminium). The phases, measurements and limits stay the same.


### RealSense Depth Camera Acceptance Test
URL: https://www.tofupilot.com/templates/realsense-depth-camera-acceptance-test
Accept Intel RealSense D435 cameras before integration: fill rate, Z-accuracy, plane-fit and temporal noise over a distance sweep, projector, extrinsics.


## Introduction

### Depth Camera Acceptance Overview

An Intel RealSense D435 is an **active stereo** depth camera: two global-shutter infrared imagers 50 mm apart, an infrared projector that paints a pseudo-random dot pattern on the scene so that textureless surfaces can be matched, and an ASIC that computes disparity per pixel and converts it to depth with the factory calibration. Depth comes out as **Z16 frames**: unsigned 16-bit integers in the camera's depth unit, 1 mm by default, with zero where the matcher found nothing. Every camera is calibrated at Intel's factory and carries its own intrinsics and extrinsics in flash; what varies from unit to unit, and with every drop and every heat cycle in transit, is how well that calibration still describes the optics.

Stereo depth error is not constant. For a disparity error of **δ** pixels, a focal length **fx** and a baseline **B**, the depth error at distance **z** is **Δz = z² · δ / (fx · B)**: it grows with the square of the distance and shrinks with the baseline. On a D435 at 848 by 480, fx is about 421 px and B is 50 mm, so 0.08 px of disparity noise is 0.9 mm of depth at 0.5 m and 15 mm at 2 m. Intel's own **Depth Quality Tool** reduces a frame of a flat target to four numbers: the **fill rate** (share of valid pixels in the region of interest), the **Z-accuracy** (distance of the fitted plane against a ground truth, in percent), the **plane-fit RMS error** (spatial noise, in mm and in sub-pixels of disparity) and the **temporal noise** (per-pixel standard deviation over a burst of frames). A robotics integrator receiving cameras by the hundred needs those four numbers per serial, on a controlled target, before the camera goes onto a robot whose grasping or navigation depends on them.

### Test Purpose

The procedure produces, per camera:

- **Identity**: serial, product name, firmware version against an allow-list, USB link type, depth units, ASIC and projector temperatures, all read from the device
- **Stream health**: 3 s of depth frame timing at 30 fps, frames per second and worst gap validated, dropped frames counted
- **Depth quality**: at 0.5, 1.0, 1.5 and 2.0 m on a flat matte board, fill rate, Z-accuracy against a laser rangefinder, plane-fit RMS and temporal noise from 30 frames each, one residual map attached
- **Projector**: fill rate on the textureless board with the emitter off, then on, and the laser power read back
- **Calibration**: stereo baseline, depth-to-color offset and rotation, focal length, the full depth intrinsics kept as a JSON record

![Depth error budget of one D435 across the 0.5 to 2 m sweep: Z-accuracy error, plane-fit RMS and temporal noise in millimetres against the 2 percent limit line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/realsense-depth-error-budget-5nVkeEa7MB3PmGwy9LqNbm1XOvOoUS.png)

The four distances of one run of the mock: the Z-accuracy error sits at +0.4 % of distance (8 mm at 2 m), the plane-fit RMS climbs from 0.8 to 12.6 mm along the z² curve, the temporal noise from 0.4 to 5.2 mm, and all of it stays well under the dashed 2 % line that Intel quotes for the D435.

Defects this catches before the camera ships on a robot: a **decalibrated stereo pair** (Z-accuracy off, plane-fit RMS up at every distance), a **dead or weak projector** (fill rate collapses on the board with the emitter on, the emitter-off and emitter-on numbers converge), a **camera stuck on USB 2** (the link type reads 2.1, the frame rate is a fraction of 30), **unapproved firmware** (the perception stack was validated against specific versions), a **noisy imager** (temporal noise up while the plane fit is fine), and a **shifted color module** (RGB offset or rotation outside the D435 geometry).

Beyond the camera content, the template shows five framework mechanics: the **device under test written as a plug** with its stream profile passed through `config:`, a **station-scoped rail plug** that stays connected across units while the camera changes, **`then: error: retry`** with a retry budget on the connect phase for the USB re-enumeration a hot-plugged camera goes through, **multi-dimensional measurements over distance with distance-normalised aggregations** validated in YAML, and **`attach.data`** of a residual map and a JSON summary on the run.

## Equipment & Setup

To accept depth cameras at incoming inspection, the following are required:

- **A flat target**: a matte white board, 1200 by 800 mm, flat to better than 1 mm, no texture of its own so the projector check means something
- **A target rail**: a motorised linear axis carrying the board from 0.5 to 2 m, with a laser rangefinder reading the board distance as the reference
- **A camera fixture**: a nest that squares the camera to the rail, a USB 3.2 cable of known quality to the test PC
- **The Devices Under Test**: RealSense D435 cameras as received, stock firmware
- **A TofuPilot Framework procedure** to enumerate, stream, sweep, measure and record
- **The TofuPilot Dashboard** to keep every camera's four metrics per distance and trend them per firmware and per delivery batch

![RealSense D435 camera and reference sensor facing a flat target on an adjustable rail](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/realsense-depth-camera-acceptance-test-physical-2026-vBujDAo53enAg39VqLsjQ3mi5FyVu2.webp)

The D435 and reference distance sensor face the same flat target. The drawing shows the fixture geometry; rail controls and the test computer are omitted.

### Hardware Components

#### Depth Camera

The **Intel RealSense D435** (and its D435i sibling with an IMU) is the reference: 848 by 480 depth at up to 90 fps, 50 mm baseline, 87° by 58° depth field of view, ideal range 0.3 to 3 m. The **D455** doubles the baseline to 95 mm and pushes the ideal range to 6 m; the same procedure accepts it with wider limits on the baseline and the RGB offset. Both talk `pyrealsense2` over USB 3.2 and expose the same descriptors, options and calibration.

![Stereo depth RMS error versus distance for the D435 50 mm baseline and the D455 95 mm baseline at 0.08 px sub-pixel noise, with the 2 percent line and this run's four plane-fit RMS points.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/realsense-stereo-error-physics-BFTUEvgRNKRsox8SlFMRv7hijiDO0J.png)

Why the sweep stops at 2 m for a D435 and would go to 4 m for a D455: at the same 0.08 px of disparity noise the 95 mm baseline halves the depth error, and the run's four plane-fit RMS points sit on the D435 curve as they should.

#### Target Rail and Reference

A 2 m linear axis (an igus drylin or a Festo ELGA belt axis with a stepper, 0.1 mm repeatability) carries the board; a **laser rangefinder** (Leica DISTO or a Keyence IL-series sensor for a fixed installation) mounted beside the camera reads the board distance to 0.15 mm and is the ground truth the Z-accuracy is computed against. The rail plug is declared `scope: station`: the axis homes once when the station starts and stays connected across every camera of the shift.

![Close-up of a RealSense depth camera in its fixture nest, the two infrared imagers, the projector and the color camera in a row on its front](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-3fe73fb6-37fc-41a9-8d40-2975cc8c43d1-1nMUlzGug0jyJKUeKwmNkPQjhQ5NMT.jpg)

The camera in its nest: two IR imagers 50 mm apart, the projector between them, the color module beside them. Each distance is 30 frames, one plane fit per frame, one row in the four multi-dimensional measurements.

### Limits

Limits for a D435 at 848 by 480 with the projector at its default 150 mW, as the template ships them:

| Parameter | Limit | Why |
|-----------|-------|-----|
| Firmware version | in 5.16.0.1, 5.15.1.0 | the versions the robot's perception stack was validated against |
| USB type | 3.2 | on 2.1 the camera streams but at a fraction of the rate |
| Frame rate | 29.5 to 30.5 fps | a USB or bandwidth problem drops frames before anything else fails |
| Max frame gap | ≤ 40 ms | one skipped frame at 30 fps is a 66 ms gap |
| Fill rate | ≥ 99 % at every distance | a flat matte board under the projector should be fully filled |
| Z-accuracy | ≤ 2 % at every distance | Intel's datasheet figure for the D435 at 2 m |
| Plane-fit RMS | ≤ 0.15 px sub-pixel | distance-independent; typical cameras sit near 0.08 px |
| Temporal noise | ≤ 10 mm at 2 m | about 0.05 px of per-frame disparity noise |
| Fill rate, emitter off | ≤ 50 % | a textureless board must not match without the pattern |
| Fill rate, emitter on | ≥ 99 % | the projector restores the match |
| Stereo baseline | 49.5 to 50.5 mm | the D435 geometry as calibrated |
| RGB offset | 13.5 to 16 mm | the color module sits 15 mm from the left imager |
| Depth-color rotation | ≤ 0.5° | a shifted color module misaligns the point cloud colours |
| Focal length | 410 to 435 px | fx at 848 by 480, catches a wrong stream profile or a bad calibration |

### Custom Firmware

None. The camera runs its stock Intel firmware; everything the procedure reads and sets is exposed by `pyrealsense2`: descriptors through `camera_info`, temperatures and the emitter through `rs.option`, calibration through the stream profiles. The only station-side requirement is librealsense with the kernel patches or the RSUSB backend, and a USB 3.2 port that actually enumerates at SuperSpeed.

## Test Procedure

### Overview

For each camera in the fixture, the procedure runs:

1. Enumerate the camera, read serial, name, firmware, USB type, depth units and temperatures; retry once if the USB stack is still re-enumerating.
2. Start the depth stream at 848 by 480, 30 fps, and time 3 s of frames.
3. Move the board to 0.5, 1.0, 1.5 and 2.0 m; at each, read the rangefinder, capture 30 frames, fit a plane per frame, compute fill rate, Z-accuracy, plane-fit RMS and temporal noise.
4. At 1 m, capture with the emitter off, then on, and compare the fill rates.
5. Read the calibration back from the device and check baseline, RGB offset, rotation and focal length.
6. Teardown: stop the stream, home the rail.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  connect_camera.py
  stream_check.py
  depth_quality.py
  projector_check.py
  extrinsics_check.py
  release_camera.py
plugs/
  realsense.py
  rail.py
utils/
  depth.py
README.md
pyproject.toml
```

The camera and the rail are mocks; the plane fits and the noise statistics are real numpy, run by the framework in a `uv` environment with `numpy` and `pillow`. Run it with `tofupilot run . --no-tui --no-kiosk --json`. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-realsense-depth-acceptance).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: RealSense Depth Camera Acceptance Test
version: 0.1.0
description: Accepts an Intel RealSense D435 depth camera on a target rail. Reads its descriptors, checks the stream timing, measures fill rate, Z-accuracy, plane-fit and temporal noise at four distances, proves the IR projector, and checks the factory extrinsics.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the camera label"
    placeholder: "213522070981"
    pattern: "^\\d{12}$"
    default_value: "213522070981"
  part_number:
    default_value: "RS-D435"
  metadata:
    firmware_version:
      description: "Firmware reported by the camera"
    usb_type:
      description: "USB link the camera enumerated on (3.2 or 2.1)"

plugs:
  - name: RealSense Camera
    description: "The camera under test over pyrealsense2: descriptors, streams, options, calibration (mock)"
    python: plugs.realsense:RealSenseCamera
    key: camera
    config:
      width: 848
      height: 480
      fps: 30
  - name: Target Rail
    description: Motorised rail carrying the flat target, with a laser rangefinder as distance reference (mock)
    python: plugs.rail:TargetRail
    key: rail
    scope: station
    config:
      port: "/dev/ttyUSB0"

main:
  - name: Connect Camera
    key: connect_camera
    python: phases.connect_camera
    retry:
      limit: 3
      delay: 2s
    then:
      error: retry
    measurements:
      - name: Camera Serial
        key: camera_serial
        validators:
          - {operator: matches, expected_value: "^\\d{12}$"}
      - name: Product Name
        key: product_name
        validators:
          - {operator: "==", expected_value: "Intel RealSense D435"}
      - name: Firmware Version
        key: firmware_version
        description: "Only firmware validated with the robot's perception stack is accepted"
        validators:
          - {operator: in, expected_value: ["5.16.0.1", "5.15.1.0"]}
      - name: USB Type
        key: usb_type
        description: "A camera on USB 2.1 streams at a fraction of the rate; the cable or the port is at fault"
        validators:
          - {operator: "==", expected_value: "3.2"}
      - name: Depth Units
        key: depth_units
        unit: µm
        validators:
          - {operator: "==", expected_value: 1000}
      - name: ASIC Temperature
        key: asic_temperature
        unit: °C
        validators:
          - {operator: ">=", expected_value: 20.0}
          - {operator: "<=", expected_value: 60.0}
      - name: Projector Temperature
        key: projector_temperature
        unit: °C
        validators:
          - {operator: ">=", expected_value: 20.0}
          - {operator: "<=", expected_value: 60.0}

  - name: Stream Check
    key: stream_check
    python: phases.stream_check
    depends_on: [connect_camera]
    measurements:
      - name: Frame Interval
        key: frame_interval
        title: Depth frame interval over 3 s at 30 fps
        x_axis:
          legend: Frame
        y_axis:
          - legend: Interval
            key: interval
            unit: ms
            aggregations:
              - type: fps
                validators:
                  - {operator: ">=", expected_value: 29.5}
                  - {operator: "<=", expected_value: 30.5}
              - type: max_gap_ms
                unit: ms
                validators:
                  - {operator: "<=", expected_value: 40.0}
      - name: Frames Dropped
        key: frames_dropped
        validators:
          - {operator: "==", expected_value: 0}

  - name: Depth Quality Sweep
    key: depth_quality
    python: phases.depth_quality
    depends_on: [stream_check]
    timeout: 300s
    ui:
      components:
        - key: sweep_progress
          type: progress
          label: "Target distances"
          description: "0.5, 1.0, 1.5 and 2.0 m, 30 frames each"
          default_value: 0
          max: 100
    measurements:
      - name: Fill Rate
        key: fill_rate
        title: Valid depth pixels in the ROI per distance
        x_axis:
          legend: Distance
          unit: mm
        y_axis:
          - legend: Fill rate
            key: fill
            unit: "%"
            aggregations:
              - type: min_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 99.0}
      - name: Z-Accuracy
        key: z_accuracy
        title: Plane distance error against the rangefinder per distance
        x_axis:
          legend: Distance
          unit: mm
        y_axis:
          - legend: Error
            key: error
            unit: "%"
            aggregations:
              - type: max_abs_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 2.0}
      - name: Plane Fit RMS
        key: plane_fit_rms
        title: Plane-fit residual RMS per distance
        description: "Grows with the square of the distance; the aggregation converts it back to disparity pixels, which does not"
        x_axis:
          legend: Distance
          unit: mm
        y_axis:
          - legend: RMS
            key: rms
            unit: mm
            aggregations:
              - type: max_subpixel_px
                unit: px
                validators:
                  - {operator: "<=", expected_value: 0.15}
      - name: Temporal Noise
        key: temporal_noise
        title: Per-pixel depth noise over 30 frames per distance
        x_axis:
          legend: Distance
          unit: mm
        y_axis:
          - legend: Noise
            key: noise
            unit: mm
            aggregations:
              - type: max_mm
                unit: mm
                validators:
                  - {operator: "<=", expected_value: 10.0}

  - name: Projector Check
    key: projector_check
    python: phases.projector_check
    depends_on: [depth_quality]
    measurements:
      - name: Laser Power
        key: laser_power_mw
        unit: mW
        validators:
          - {operator: "==", expected_value: 150}
      - name: Fill Rate Emitter Off
        key: fill_rate_emitter_off
        unit: "%"
        description: "A matte white board has no texture, so without the projected pattern the matcher must lose most pixels"
        validators:
          - {operator: "<=", expected_value: 50.0}
      - name: Fill Rate Emitter On
        key: fill_rate_emitter_on
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 99.0}
      - name: Emitter Gain
        key: emitter_gain_pct
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 40.0}

  - name: Extrinsics Check
    key: extrinsics_check
    python: phases.extrinsics_check
    depends_on: [projector_check]
    measurements:
      - name: Stereo Baseline
        key: stereo_baseline_mm
        unit: mm
        validators:
          - {operator: ">=", expected_value: 49.5}
          - {operator: "<=", expected_value: 50.5}
      - name: RGB Offset
        key: rgb_offset_mm
        unit: mm
        description: "Depth to color translation, 15 mm on a D435 by construction"
        validators:
          - {operator: ">=", expected_value: 13.5}
          - {operator: "<=", expected_value: 16.0}
      - name: Depth-Color Rotation
        key: depth_color_rotation_deg
        unit: °
        validators:
          - {operator: "<=", expected_value: 0.5}
      - name: Focal Length
        key: focal_length_px
        unit: px
        validators:
          - {operator: ">=", expected_value: 410.0}
          - {operator: "<=", expected_value: 435.0}
      - name: Depth Intrinsics
        key: depth_intrinsics
        description: "The full depth intrinsics as the device reports them, kept whole for the record"

teardown:
  - name: Release Camera
    key: release_camera
    python: phases.release_camera
    measurements:
      - name: Stream Stopped
        key: stream_stopped
        validators:
          - {operator: "==", expected_value: true}
      - name: Rail Homed
        key: rail_homed
        validators:
          - {operator: "==", expected_value: true}
```

Framework features to notice:

1. **The device under test is a plug**: `plugs.realsense:RealSenseCamera` wraps `pyrealsense2`, and the stream profile (`width`, `height`, `fps`) travels through `config:` into its `__init__`, so the same plug accepts a D455 at 1280 by 720 by editing the YAML.
2. **Two plug scopes on one bench**: the rail is `scope: station`, homed once and held across every camera of the shift; the camera keeps the default scope and is re-created for every unit, which is what a device under test needs. In a one-shot `tofupilot run` a station plug behaves as execution scope, so the template runs unchanged.
3. **Retry on error**: `connect_camera` carries `retry: {limit: 3, delay: 2s}` and `then: {error: retry}`. A RealSense plugged in a moment ago fails `query_devices()` while the USB stack re-enumerates; the first attempt errors, the second passes, and the run page shows both attempts. One thing learned on the way: the override is honoured on `main` phases, not on `setup` phases, where the engine declares the setup failed before it looks at `then`, which is why the connect phase sits at the head of `main` with everything depending on it.
4. **Distance-normalised aggregations**: `plane_fit_rms` is recorded in mm per distance, where it must grow with z², and validated through its `max_subpixel_px` aggregation, which the phase computes as RMS · fx · B / z² and which does not depend on distance. The same pattern gives `fill_rate` its `min_pct`, `z_accuracy` its `max_abs_pct` and `temporal_noise` its `max_mm`, all four named by the phase and checked in YAML.
5. **String validators for the descriptors**: `firmware_version` with `in` against the allow-list, `usb_type` with `==` to `"3.2"`, `camera_serial` with `matches` on twelve digits; the descriptors also land in `unit.metadata`, so a run can be filtered by firmware later.

### Connect Camera

The camera plug's `open()` enumerates the device and reads its descriptors; the phase records them, stamps firmware and USB type on the unit, and writes which rail and target the run used into `run.metadata`:

```python filename="phases/connect_camera.py"
def connect_camera(measurements, unit, run, camera, log):
    """Setup: enumerate the camera, read its descriptors, stamp firmware and USB link on the unit."""
    info = camera.open()
    log.info(f"{info['name']} {info['serial']}: firmware {info['firmware']}, USB {info['usb_type']}, ASIC {info['asic_temperature_c']:.1f} C")
    measurements.camera_serial = info["serial"]
    measurements.product_name = info["name"]
    measurements.firmware_version = info["firmware"]
    measurements.usb_type = info["usb_type"]
    measurements.depth_units = info["depth_units_um"]
    measurements.asic_temperature = info["asic_temperature_c"]
    measurements.projector_temperature = info["projector_temperature_c"]
    unit.metadata["firmware_version"] = info["firmware"]
    unit.metadata["usb_type"] = info["usb_type"]
    run.metadata["rail_id"] = "RAIL-02"
    run.metadata["target"] = "matte white board 1200x800"
```

On the mock the first `open()` raises `RuntimeError("No device connected")`, the way librealsense does right after a hot-plug; the engine waits 2 s and runs the phase again, and attempt 2 records the descriptors.

### Stream Check

Three seconds of depth frames at 30 fps, the interval between consecutive frames recorded as a multi-dimensional measurement, the frame rate and the worst gap as its aggregations:

```python filename="phases/stream_check.py"
TIMING_S = 3.0


def stream_check(measurements, camera, log):
    """Start the depth stream and check it delivers every frame on time."""
    camera.start()
    intervals = camera.frame_intervals(TIMING_S)
    nominal = 1000.0 / 30
    dropped = sum(1 for dt in intervals if dt > 1.5 * nominal)
    fps = 1000.0 / (sum(intervals) / len(intervals))
    log.info(f"{len(intervals)} frames in {TIMING_S:.0f} s: {fps:.2f} fps, max gap {max(intervals):.1f} ms, {dropped} dropped")
    measurements.frame_interval.x_axis = list(range(1, len(intervals) + 1))
    measurements.frame_interval.y_axis.interval = intervals
    measurements.frame_interval.y_axis.interval.aggregations.fps = fps
    measurements.frame_interval.y_axis.interval.aggregations.max_gap_ms = max(intervals)
    measurements.frames_dropped = dropped
```

On the mock: 90 frames, 29.95 fps, 34.1 ms worst gap, none dropped. A camera that enumerated on USB 2.1 fails here twice, on the USB type in the previous phase and on the frame rate in this one.

### Depth Quality Sweep

The heart of the procedure. For each distance, the rail moves, the rangefinder gives the reference, the camera captures 30 frames of the region of interest and saves them to a file; the phase deprojects every pixel with the depth intrinsics, fits a plane per frame in metric space (so a target that is not perfectly square to the camera still yields the distance along the optical axis), and reduces the burst to the four Depth Quality Tool metrics:

```python filename="phases/depth_quality.py"
DISTANCES_MM = [500, 1000, 1500, 2000]
N_FRAMES = 30


def depth_quality(measurements, attach, camera, rail, ui, log):
    """Walk the target through four distances and measure the four Depth Quality Tool metrics at each."""
    intr = camera.depth_intrinsics()
    baseline = camera.extrinsics()["stereo_baseline_mm"]
    fill, z_err, rms, subpixel, temporal = [], [], [], [], []
    residual_map = None

    for i, d in enumerate(DISTANCES_MM):
        rail.move_to(d)
        ref = rail.reference_mm()
        cap = camera.capture_depth(N_FRAMES, ref)
        frames = np.load(cap["path"]).astype(np.float64) * cap["depth_units_um"] / 1000.0  # mm
        valid = frames > 0

        per_frame_rms, per_frame_dist = [], []
        for z, ok in zip(frames, valid):
            x, y, z = deproject(z, cap["roi"], intr)
            dist, residual = fit_plane(x, y, z, ok)
            per_frame_dist.append(dist)
            per_frame_rms.append(float(np.sqrt(np.nanmean(residual**2))))
        dist = float(np.mean(per_frame_dist))
        always_valid = valid.all(axis=0)
        noise = float(frames[:, always_valid].std(axis=0).mean())

        fill.append(100.0 * float(valid.mean()))
        z_err.append(100.0 * (dist - ref) / ref)
        rms.append(float(np.mean(per_frame_rms)))
        subpixel.append(rms[-1] * intr["fx"] * baseline / dist**2)
        temporal.append(noise)
        residual_map = residual  # the last distance is where the residuals are largest
        log.info(f"{d} mm (ref {ref:.1f}): fill {fill[-1]:.2f} %, z error {z_err[-1]:+.2f} %, plane RMS {rms[-1]:.2f} mm = {subpixel[-1]:.3f} px, temporal {noise:.2f} mm")
        ui.sweep_progress = int(100 * (i + 1) / len(DISTANCES_MM))

    measurements.fill_rate.x_axis = DISTANCES_MM
    measurements.fill_rate.y_axis.fill = fill
    measurements.fill_rate.y_axis.fill.aggregations.min_pct = min(fill)
    measurements.z_accuracy.x_axis = DISTANCES_MM
    measurements.z_accuracy.y_axis.error = z_err
    measurements.z_accuracy.y_axis.error.aggregations.max_abs_pct = max(abs(e) for e in z_err)
    measurements.plane_fit_rms.x_axis = DISTANCES_MM
    measurements.plane_fit_rms.y_axis.rms = rms
    measurements.plane_fit_rms.y_axis.rms.aggregations.max_subpixel_px = max(subpixel)
    measurements.temporal_noise.x_axis = DISTANCES_MM
    measurements.temporal_noise.y_axis.noise = temporal
    measurements.temporal_noise.y_axis.noise.aggregations.max_mm = max(temporal)

    attach.data(residual_png(residual_map, rms[-1]), f"plane_residuals_{DISTANCES_MM[-1]}mm.png")
    summary = {"distances_mm": DISTANCES_MM, "fill_rate_pct": fill, "z_accuracy_pct": z_err,
               "plane_fit_rms_mm": rms, "subpixel_rms_px": subpixel, "temporal_noise_mm": temporal}
    attach.data(json.dumps(summary, indent=2).encode(), "depth_quality.json")
```

![Plane-fit RMS per distance: in millimetres it follows a k times z squared curve from 0.84 to 12.65 mm, in disparity pixels it stays at 0.066 to 0.070 px under the 0.15 px limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/realsense-plane-fit-subpixel-vMUnI997tDTStieQfzEo8TxQXgNYsR.png)

The two faces of the same measurement on the mock run: 0.84, 3.17, 7.12 and 12.65 mm of plane-fit RMS, on a k·z² curve with k = 3.16 mm per m², and 0.070, 0.066, 0.066 and 0.066 px once converted back to disparity. A limit in millimetres would have to be a different number at every distance; the sub-pixel figure is one limit for the whole sweep and compares directly across the D400 family.

The Z-accuracy reads +0.40 % at every distance, the signature of a stereo pair whose true baseline is 0.4 % shorter than the calibrated one. Fill rate is 99.8 % throughout, temporal noise 0.43 to 5.22 mm. The residual map attached to the run shows the 2 m plane fit pixel by pixel: blue closer than the plane, red farther, black where the matcher returned nothing.

### Projector Check

A matte white board has no texture of its own, so a stereo matcher without the projected pattern loses most of it. The phase captures at 1 m with the emitter off, then on, and records both fill rates and their difference:

```python filename="phases/projector_check.py"
CHECK_MM = 1000
N_FRAMES = 10


def projector_check(measurements, camera, rail, log):
    """Prove the IR projector works: fill rate on a textureless board with the emitter off, then on."""
    rail.move_to(CHECK_MM)
    ref = rail.reference_mm()

    camera.set_emitter(False)
    off = fill_rate(camera.capture_depth(N_FRAMES, ref))
    camera.set_emitter(True)
    power = camera.laser_power_mw()
    on = fill_rate(camera.capture_depth(N_FRAMES, ref))

    log.info(f"fill rate at {CHECK_MM} mm: {off:.1f} % emitter off, {on:.1f} % emitter on at {power:.0f} mW")
    measurements.laser_power_mw = power
    measurements.fill_rate_emitter_off = off
    measurements.fill_rate_emitter_on = on
    measurements.emitter_gain_pct = on - off
```

![Fill rate on a matte white board at 1 m: 31.0 percent with the emitter off against a 50 percent ceiling, 99.8 percent with the emitter on against a 99 percent floor.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/realsense-projector-fill-rate-fbMeDDhreFqzs3kuyiFG8lJ1D1fpQi.png)

31.0 % with the emitter off, 99.8 % with it on, a gain of 69 points at 150 mW. A dead projector shows both numbers near 31 %; a weak one shows the emitter-on fill rate sliding under 99 % at 1 m long before it does at 0.5 m.

### Extrinsics Check

The factory calibration is read back from the device and checked against the D435 geometry, with the full intrinsics kept as a JSON measurement:

```python filename="phases/extrinsics_check.py"
def extrinsics_check(measurements, camera, log):
    """Read the factory calibration back from the device and check it against the D435 geometry."""
    intr = camera.depth_intrinsics()
    extr = camera.extrinsics()
    rgb_offset = math.sqrt(sum(t * t for t in extr["depth_to_color_translation_mm"]))
    log.info(f"baseline {extr['stereo_baseline_mm']:.2f} mm, RGB offset {rgb_offset:.2f} mm, rotation {extr['depth_to_color_rotation_deg']:.2f} deg, fx {intr['fx']:.1f} px")
    measurements.stereo_baseline_mm = extr["stereo_baseline_mm"]
    measurements.rgb_offset_mm = rgb_offset
    measurements.depth_color_rotation_deg = extr["depth_to_color_rotation_deg"]
    measurements.focal_length_px = intr["fx"]
    measurements.depth_intrinsics = intr
```

On the mock: baseline 50.03 mm, RGB offset 14.72 mm, rotation 0.12°, fx 421.6 px. The extrinsics say what the camera believes about itself; the sweep before it says whether that belief still matches the optics.

### Mock Plugs

The camera mock hides a true stereo geometry and answers `capture_depth()` with Z16 frames computed from it: the device reports a calibrated baseline of 50.03 mm while the true optical baseline is 49.83 mm (so every depth reads 0.4 % long), disparity carries a fixed spatial pattern of 0.06 px and a per-frame temporal noise of 0.028 px, the board is tilted 0.6° and 0.4° off square, 0.2 % of pixels are holes with the projector on and 69 % without it. Depth is quantised to the 1 mm depth unit. The phases never read these constants:

```python filename="plugs/realsense.py"
# The camera's real optics, unknown to the procedure.
BASELINE_TRUE_MM = 49.83  # 0.4 % short of the calibrated value: depth reads long
SPATIAL_PX = 0.060  # fixed disparity pattern (lens, sensor, ASIC)
TEMPORAL_PX = 0.028  # per-frame disparity noise
TILT_DEG = (0.6, -0.4)  # the target is never perfectly square to the camera
HOLE_RATE = 0.002  # invalid pixels with the projector on
EMITTER_OFF_FILL = 0.31  # a matte white board has no texture of its own

    def capture_depth(self, n_frames, scene_mm) -> dict:
        """Capture `n_frames` Z16 frames of the ROI and save them to depth_capture.npy.

        `scene_mm` builds the mock scene: a flat matte board at that distance. A real plug ignores
        it, the board is whatever the rail put in front of the lens."""
        n, d = int(n_frames), float(scene_mm)
        x0, y0, rw, rh = self._roi
        tx, ty = math.tan(math.radians(TILT_DEG[0])), math.tan(math.radians(TILT_DEG[1]))
        z_true = d / (1.0 - tx * self._nx - ty * self._ny)  # plane through (0, 0, d), normal tilted
        disparity = FX * BASELINE_TRUE_MM / z_true
        gain = 1.0 if self._emitter else 3.0  # without the pattern the matcher is noisier
        hole_rate = HOLE_RATE if self._emitter else 1.0 - EMITTER_OFF_FILL

        frames = np.empty((n, rh, rw), dtype=np.uint16)
        for i in range(n):
            measured = disparity + gain * (self._pattern_px + self._rng.normal(0.0, TEMPORAL_PX, (rh, rw)))
            z_units = np.rint(FX * BASELINE_CAL_MM / measured * 1000.0 / DEPTH_UNITS_UM)
            z_units[self._rng.random((rh, rw)) < hole_rate] = 0
            frames[i] = z_units.astype(np.uint16)
        np.save(CAPTURE_PATH, frames)
        return {"path": CAPTURE_PATH, "frames": n, "roi": list(self._roi), "depth_units_um": DEPTH_UNITS_UM}
```

Plug calls cross a JSON boundary, and 30 frames of a 170 by 96 region are half a million values, so the frames go through a `.npy` file and the call returns its path and the ROI; the phase loads the array. On a real station, `open()` is `rs.context().query_devices()` plus `get_info()` for the descriptors, `start()` builds an `rs.config` from the `width`, `height` and `fps` the procedure passed through `config:` and starts an `rs.pipeline`, `capture_depth()` collects `wait_for_frames().get_depth_frame()` into the same array and ignores its second argument, `set_emitter()` and `laser_power_mw()` are `rs.option.emitter_enabled` and `rs.option.laser_power`, and the intrinsics and extrinsics come from the stream profiles. The rail plug drives the real axis and reads the rangefinder. The phases, the measurements and the limits do not change.


### Motor Balance and Vibration Signature
URL: https://www.tofupilot.com/templates/motor-balance-and-vibration-signature
Capture drone motor vibration spectra across an RPM sweep to catch bearing defects, rotor imbalance, and misalignment at end-of-line QA.

## Introduction

### Vibration in Drone Motors

Every BLDC motor on a drone produces vibration. Some is structural (rotor imbalance, bearing roughness), some is electromagnetic (cogging torque, commutation harmonics), and some is acoustic coupling from the frame itself. In flight, this vibration corrupts the IMU accelerometer readings that attitude estimation depends on: PX4's [flight log analysis guidance](https://docs.px4.io/main/en/log/flight_log_analysis.html) treats anything above roughly **2-3 m/s² peak-to-peak** on the accelerometer as strong vibration, and sustained levels beyond that cause attitude drift, position hold wander, and visual-inertial odometry failure. On gimballed cameras, residual vibration passes through the rolling shutter and renders **"jello"**, a vertical wave artifact widely attributed to vibration passing through worn gimbal dampers (DJI advises replacing aged dampers because they "may affect shooting").

![Cutaway of a drone brushless outrunner motor showing the bell, stator windings, magnet ring, shaft and the top and bottom ball bearings.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-c831412c-fe0a-438b-927d-b0e4c7a56ae9-jtkczIHndI379AInXeuxJrExv5NLES.jpg)

Imbalance lives in the bell and magnets, misalignment in the shaft, defect tones in the two bearings.

Continuous side-loading from imbalance also accelerates bearing race wear and damper fatigue over the airframe's life. End-of-line vibration testing catches the bad units before they ship and trends bearing supplier quality across the production batch.

### Test Purpose

The procedure produces a **per-motor vibration signature**:

- **1× rotational order amplitude**: static or dynamic imbalance (loose magnet, lost balance dot, deformed bell)
- **2× rotational order**: misalignment, bent shaft
- **Bearing defect bands**: non-synchronous peaks at the outer-race (BPFO) and inner-race (BPFI) defect frequencies
- **Broadband RMS** across the sweep, and at hover RPM specifically

Comparing against a **golden-reference fingerprint** flags units with anomalous content. Production tightens the limits via TofuPilot's rolling 3σ analytics after the first 500-1000 units.

![3D waterfall of the sweep spectra plotted against order and RPM: the 1×, 2×, BPFO and BPFI content forms constant-order ridges across the whole 2000-12000 RPM range.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-ca3f27f8-ed74-4f19-aaee-c278d3c7f0c2-qqHwoRytUzmq5ULL2uvFqrjSGF1mzw.png)

Order waterfall of the mock sweep: rotor and bearing content stays on its order across the whole RPM range, which is what separates it from a fixed-frequency structural resonance.

## Equipment & Setup

To implement motor vibration testing on a production line, the following are required:

- **A rigid steel motor fixture** isolated from the bench, with an integrated tachometer pickup
- **A triaxial accelerometer** mounted to the motor stator or fixture
- **A high-speed DAQ** with anti-alias filtering
- **The Device Under Test (DUT)**: motor only, **prop must be removed for safety**
- **A programmable motor driver** (same ESC + DShot as ESC characterization)
- **A TofuPilot Framework procedure** to step RPM, capture spectra, extract order peaks, and flag anomalies
- **The TofuPilot Dashboard** to log fingerprints and trend bearing supplier quality

### Hardware Components

#### Accelerometer

The production reference is the [PCB Piezotronics 356A32](https://www.pcb.com/products?m=356a32): a triaxial ICP/IEPE accelerometer, **100 mV/g**, **1 Hz to 4 kHz** band (±5%), 5 g titanium housing, sold on a quote basis. It is an industry standard for rotating-machinery diagnostics. For embedded in-airframe monitoring at lower cost, the [Analog Devices ADXL355](https://www.analog.com/en/products/adxl355.html) (low-noise MEMS, **22.5 µg/√Hz** on the ±2 g range, ±2/4/8 g selectable, ~$50) is the alternative.

![Propeller-free motor on a rigid fixture with accelerometer and optical tachometer](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/motor-balance-and-vibration-signature-physical-2026-E3y7U2cFdM9jekQ318S0CEFR5sKmiB.webp)

The motor is tested without a propeller. The accelerometer measures fixture vibration and the optical pickup reads a marker on the rotating bell; drive electronics are omitted.

**Mounting matters**: in decreasing rigidity, stud, thin adhesive, beeswax, flat magnet, curved magnet. PCB's [TN-5 mounting technote](https://www.pcb.com/contentstore/mktgcontent/linkeddocuments/technotes/TN_5_MountingTechniques.pdf) documents the bandwidth cost of each; a magnet mount can halve the usable frequency range. Wax is acceptable below ~1 kHz, magnet below ~2 kHz. For end-of-line testing requiring fast swap, a magnet-mounted sensor on a steel fixture is the practical compromise. For permanent reference benches, stud-mount.

#### DAQ

The [NI-9234](https://www.ni.com/docs/en-US/bundle/ni-9234-specs/page/specs.html) C Series module (in a cDAQ chassis) is the reference: **4 channels IEPE**, **24-bit**, **51.2 kS/s/ch**, software-selectable 2 mA IEPE current, automatic anti-alias filter tracking the sample rate. **Dewesoft** and **Brüel & Kjær** offer production-line analyzers with built-in order-tracking workflows. For higher channel counts (multiple motors in parallel), a cDAQ-9189 chassis with multiple NI-9234 modules scales linearly.

#### Motor Fixture

A machined steel block bolted to a vibration-isolated bench (concrete floor or vibration-isolation table). The motor mounts on a flange with a stud-mount accelerometer pickup on the stator. The fixture's natural modes must be **above 5 kHz** (verified by impact hammer test) so they don't contaminate the motor signature. Mass typically 5-10 kg.

### Custom Firmware

ESC under bidirectional DShot, same as the throttle-to-RPM characterization test. The test station commands RPM directly via DShot frames and reads back eRPM telemetry as the speed reference. Because the sweep dwells at fixed RPM steps, a windowed FFT per step gives clean order peaks; continuous sweeps would need angular-domain order tracking instead.

## Test Procedure

### Overview

After mounting the motor (prop removed) and connecting accelerometer + DAQ + ESC, the procedure runs:

1. Safety gate: the operator confirms prop removal on a switch; the run refuses to continue otherwise.
2. Step RPM from 2000 to 12000 in 50 logarithmic steps, acquiring 200 ms at 10 kHz per step.
3. Compute a Hann-windowed FFT per step and extract the 1×, 2×, BPFO, and BPFI peaks.
4. Validate every metric against per-axis limits.
5. Record the hover-RPM spectrum and the 1× trace as interactive charts, attach the metrics CSV.
6. Stream results to TofuPilot for traceability and analytics.

This template exercises several framework capabilities on purpose: an **operator UI safety gate** bound to a boolean measurement, **multi-dimensional measurements** with per-axis aggregations, **file attachments**, **multi-slot execution** for testing four motor arms in parallel, and all four **log levels**.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  safety_check.py
  rpm_sweep.py
  vibration_analysis.py
plugs/
  daq.py
utils/
  bearing.py
ui.json
pyproject.toml
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-motor-vibration-signature). The mock DAQ plug synthesizes captures in code, so the procedure runs end to end without hardware:

```bash
tofupilot run . --ui-values ui.json
```

`ui.json` pre-bakes the operator confirmation for headless runs; on a station the operator flips the switch instead.

### The Procedure File

`procedure.yaml` declares the unit, the DAQ plug, three phases wired with `depends_on`, and every measurement with its limits. This is the exact file from the repository:

```yaml filename="procedure.yaml"
name: Motor Balance and Vibration Signature
version: 0.1.0
description: Per-motor vibration fingerprint across an RPM sweep with order extraction, bearing defect detection, and interactive spectrum charts.

execution:
  workers: 8
  strategy: phase_first
  # Multi-slot parallel testing: uncomment to test all four motor arms at
  # once on a station (one dashboard run per slot). The one-shot CLI path
  # (`tofupilot run`) does not support multi-slot yet, so the local template
  # runs single-slot.
  # slots:
  #   - name: ARM1
  #   - name: ARM2
  #   - name: ARM3
  #   - name: ARM4

unit:
  auto_identify: true
  serial_number:
    default_value: "SN-MOTOR-0001"
  part_number:
    default_value: "MOTOR-2207"

plugs:
  - name: DAQ
    key: daq
    description: Mock IEPE DAQ that synthesizes an accelerometer capture per RPM step.
    python: plugs.daq:MockDaq

main:
  - name: Safety Check
    key: safety_check
    python: phases.safety_check
    ui:
      requires_input: true
      components:
        - key: prop_removed
          type: switch
          label: "Propeller removed"
          description: "Confirm the propeller is removed before the motor is energized"
          bind: measurements.prop_removed
    measurements:
      - name: Prop Removed
        key: prop_removed
        description: Operator confirmation that no propeller is installed.
        validators:
          - operator: "=="
            expected_value: true

  - name: RPM Sweep
    key: rpm_sweep
    python: phases.rpm_sweep
    depends_on: [safety_check]
    measurements:
      - name: Steps Acquired
        key: steps_acquired
        validators:
          - operator: "=="
            expected_value: 50

  - name: Vibration Analysis
    key: vibration_analysis
    python: phases.vibration_analysis
    depends_on: [rpm_sweep]
    measurements:
      - name: 1x Peak Amplitude
        key: one_x_amplitude
        unit: g
        description: Rotational-order peak, dominated by rotor imbalance.
        validators:
          - operator: "<="
            expected_value: 0.8
      - name: 2x Peak Amplitude
        key: two_x_amplitude
        unit: g
        description: Second-order peak, dominated by misalignment or a bent shaft.
        validators:
          - operator: "<="
            expected_value: 0.4
      - name: BPFO Amplitude
        key: bpfo_amplitude
        unit: g
        description: Outer-race bearing defect band amplitude.
        validators:
          - operator: "<="
            expected_value: 0.2
      - name: BPFI Amplitude
        key: bpfi_amplitude
        unit: g
        description: Inner-race bearing defect band amplitude.
        validators:
          - operator: "<="
            expected_value: 0.2
      - name: Hover Band RMS
        key: hover_rms
        unit: g
        description: Broadband RMS in the 5000-7000 RPM hover band.
        validators:
          - operator: "<="
            expected_value: 0.5
      - name: Full Sweep RMS
        key: sweep_rms
        unit: g
        validators:
          - operator: "<="
            expected_value: 1.5
      - name: Hover Spectrum
        key: hover_spectrum
        title: Spectrum at hover RPM
        description: Amplitude spectrum at the sweep step closest to 6000 RPM, rendered as an interactive chart by the dashboard.
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Amplitude
            key: amplitude
            unit: g
      - name: 1x Order Trace
        key: one_x_trace
        title: 1x amplitude vs RPM
        description: Rotational-order amplitude across the sweep. A flat trace is a balanced rotor; a rising trace tracks imbalance with speed.
        x_axis:
          legend: RPM
          unit: rpm
        y_axis:
          - legend: 1x Amplitude
            key: one_x
            unit: g
            aggregations:
              - type: p2p
                unit: g
                validators:
                  - operator: "<="
                    expected_value: 0.5
```

The `slots` block is the multi-slot showcase: uncommented on a station engine, the same procedure tests all four motor arms in parallel, producing one dashboard run per slot. The one-shot CLI path does not support multi-slot yet, so the local template runs single-slot. The `workers: 8` and `strategy: phase_first` settings control how many phases run concurrently and, on a multi-slot station, whether each phase completes across all slots before the next phase starts.

**Where do the limits come from?** The hover-band RMS limit (0.5 g ≈ 4.9 m/s²) is set so a motor passing the test contributes well under the 2-3 m/s² peak-to-peak accelerometer disturbance that [PX4 flags as strong vibration](https://docs.px4.io/main/en/log/flight_log_analysis.html) once frame damping is accounted for. The order and bearing-band limits start as fractions of that budget, in the spirit of the zone classification that [ISO 20816-1](https://www.iso.org/standard/63180.html) (which replaced ISO 10816-1) applies to machine vibration severity, and are then tightened to the observed production 3σ once a few hundred units have run.

### Safety Gate

The first phase is an **operator UI gate**. A `switch` component is bound to the `prop_removed` boolean measurement with `bind`, and `requires_input: true` forces a Continue click. The `== true` validator fails the phase if the operator does not confirm, so the motor is never energized with a prop installed:

```python filename="phases/safety_check.py"
def safety_check(log):
    """Operator gate: the bound switch writes measurements.prop_removed.

    The phase passes only if the operator confirms the propeller is removed;
    the validator `== true` fails the phase otherwise and the run stops
    before the motor is ever energized.
    """
    log.info("Waiting for operator confirmation that the propeller is removed")
```

No Python is needed to read the switch: the binding writes the operator's answer straight into the measurement, and the validator does the enforcement.

### RPM Sweep

50 logarithmic RPM steps from **2000 to 12000 RPM**, 200 ms at 10 kHz per step. Logarithmic spacing concentrates resolution where it matters: low-RPM bearing pre-failure indicators and high-RPM resonance. The sweep phase drives the DAQ plug and validates the step count:

```python filename="phases/rpm_sweep.py"
RPM_START = 2000
RPM_END = 12000
STEPS = 50


def rpm_sweep(daq, measurements, log):
    log.debug(f"Commanding sweep {RPM_START}-{RPM_END} RPM in {STEPS} log steps")
    rpms = daq.acquire_sweep(RPM_START, RPM_END, STEPS)
    if len(rpms) < STEPS:
        log.warning(f"Sweep returned {len(rpms)} steps, expected {STEPS}")
    measurements.steps_acquired = len(rpms)
    log.info(f"Acquired {len(rpms)} captures from {rpms[0]} to {rpms[-1]} RPM")
```

![RPM staircase of the sweep, 50 logarithmic steps from 2000 to 12000 RPM over about 60 s, with the 200 ms acquisition window at the end of every dwell highlighted.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-8f4e5f83-599c-481f-87df-6e88468b4069-QYXWM14Db61Ct82pIVmdBcFvyuChEf.png)

Each dwell ends with a 200 ms capture; the logarithmic spacing packs more steps into the low-RPM range.

The plug caches each capture so the analysis phase can fetch it later. The mock synthesizes a realistic signal (1× imbalance, 2× misalignment, bearing tones, broadband noise). Critically, it injects the bearing tones at the same BPFO/BPFI frequencies the analysis searches, using the shared geometry in `utils/bearing.py`, so the detection path is genuinely exercised: raising the injected outer-race tone above 0.2 g makes the run fail on the `bpfo_amplitude` validator. Swap the class for one wrapping `nidaqmx` and your ESC driver to run on real hardware:

```python filename="plugs/daq.py"
import math
import random

from utils.bearing import bearing_defect_frequencies


class MockDaq:
    """Simulated DAQ. Synthesizes one accelerometer capture per RPM step.

    A real implementation would wrap nidaqmx and command the ESC over
    bidirectional DShot; the interface below stays identical. The bearing
    tones are injected at the same BPFO/BPFI frequencies the analysis
    phase searches, from the shared geometry in utils/bearing.py.
    """

    SAMPLE_RATE_HZ = 10_000
    CAPTURE_S = 0.2

    def __init__(self):
        self._captures = {}
        random.seed(2207)
        print("Mock DAQ initialized (10 kHz, 200 ms per step)")

    def acquire_sweep(self, rpm_start, rpm_end, steps):
        """Run the RPM sweep and cache one capture per step. Returns step RPMs."""
        rpms = [
            rpm_start * (rpm_end / rpm_start) ** (i / (steps - 1))
            for i in range(steps)
        ]
        for rpm in rpms:
            self._captures[round(rpm)] = self._synthesize(rpm)
        return [round(r) for r in rpms]

    def get_capture(self, rpm):
        """Return the cached capture for one RPM step as a list of floats."""
        return self._captures[rpm]

    def _synthesize(self, rpm):
        f_r = rpm / 60.0
        n = int(self.SAMPLE_RATE_HZ * self.CAPTURE_S)
        dt = 1.0 / self.SAMPLE_RATE_HZ
        defects = bearing_defect_frequencies(f_r)
        samples = []
        for i in range(n):
            t = i * dt
            v = 0.38 * math.sin(2 * math.pi * f_r * t)                  # 1x imbalance
            v += 0.14 * math.sin(2 * math.pi * 2 * f_r * t)             # 2x misalignment
            v += 0.12 * math.sin(2 * math.pi * defects["bpfo"] * t)     # outer race tone
            v += 0.06 * math.sin(2 * math.pi * defects["bpfi"] * t)     # inner race tone
            v += random.gauss(0.0, 0.02)                        # broadband noise
            samples.append(v)
        return samples
```

### Bearing Defect Frequencies

Per the standard rolling-element formulas (see for example [Timken's bearing frequency guide](https://engineering.timken.com/wp-content/uploads/2024/08/Bearing-Periodic-Frequencies-Calculation-Tool-Help-Guide.pdf)), with N balls, ball diameter d, pitch diameter D, contact angle φ, shaft speed f_r:

- **BPFO** = (N/2) · f_r · (1 − (d/D) cos φ)
- **BPFI** = (N/2) · f_r · (1 + (d/D) cos φ)
- **BSF** = (D/(2d)) · f_r · (1 − ((d/D) cos φ)²)
- **FTF** = (1/2) · f_r · (1 − (d/D) cos φ)

The template validates the two highest-signal bands, BPFO and BPFI; the utility also computes BSF and FTF for production lines that trend cage and ball defects. The geometry constants live in one place, imported by both the mock and the analysis, so injected tones and search windows can never drift apart:

```python filename="utils/bearing.py"
# Bearing geometry for the motor's 7-ball deep-groove bearing (per motor
# model, loaded from a spec in production). The mock DAQ and the analysis
# phase both import these so the injected tones and the search windows can
# never drift apart.
N_BALLS = 7
D_OVER_DP = 0.28


def bearing_defect_frequencies(f_r, n_balls=N_BALLS, d_over_dp=D_OVER_DP, cos_phi=1.0):
    """Rolling-element defect frequencies for shaft speed f_r in Hz."""
    ratio = d_over_dp * cos_phi
    return {
        "bpfo": (n_balls / 2.0) * f_r * (1.0 - ratio),
        "bpfi": (n_balls / 2.0) * f_r * (1.0 + ratio),
        "bsf": (1.0 / (2.0 * d_over_dp)) * f_r * (1.0 - ratio**2),
        "ftf": 0.5 * f_r * (1.0 - ratio),
    }
```

With this 7-ball, d/D = 0.28 geometry at 6000 RPM (f_r = 100 Hz): BPFO = 252 Hz, BPFI = 448 Hz.

![Amplitude spectrum at the 5990 RPM hover step with the annotated peaks: 1× at 100 Hz and 0.38 g, 2× at 200 Hz and 0.14 g, BPFO at 252 Hz and 0.11 g, BPFI at 447 Hz and 0.05 g, broadband floor near 0.02 g.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-82399d5b-c9be-483c-8ed9-317d8a94927d-qb8gotyMQytWjsCTAGGcF7G5Nil47E.png)

Hover-step spectrum of the mock capture: every peak the analysis validates, each against its limit.

### Vibration Analysis

The analysis phase computes a windowed FFT per step, extracts the order and bearing-band peaks, fills the two interactive charts, and attaches the per-step metrics as a CSV. It also reads the previous phase's result through **previous-results injection**: naming a parameter `rpm_sweep` gives read access to that phase's measurements:

```python filename="phases/vibration_analysis.py"
import os
import tempfile

import numpy as np

from utils.bearing import bearing_defect_frequencies

SAMPLE_RATE = 10_000
HOVER_BAND = (5000, 7000)


def _peak_near(freqs, spectrum, target_hz, tolerance_bins=3):
    idx = int(np.argmin(np.abs(freqs - target_hz)))
    lo = max(0, idx - tolerance_bins)
    hi = min(len(spectrum), idx + tolerance_bins + 1)
    return float(np.max(spectrum[lo:hi]))


def vibration_analysis(daq, rpm_sweep, measurements, attach, log):
    steps = int(rpm_sweep.steps_acquired)
    log.debug(f"Analyzing {steps} captures from the sweep phase")

    rpms = [
        2000 * (12000 / 2000) ** (i / (steps - 1)) for i in range(steps)
    ]
    rpms = [round(r) for r in rpms]

    spectra = []
    one_x, two_x, bpfo_amp, bpfi_amp, rms_all = [], [], [], [], []
    freqs = None

    for rpm in rpms:
        samples = np.asarray(daq.get_capture(rpm))
        window = np.hanning(len(samples))
        spectrum = np.abs(np.fft.rfft(samples * window)) * 2.0 / np.sum(window)
        freqs = np.fft.rfftfreq(len(samples), 1.0 / SAMPLE_RATE)
        spectra.append(spectrum)

        f_r = rpm / 60.0
        defects = bearing_defect_frequencies(f_r)
        one_x.append(_peak_near(freqs, spectrum, f_r))
        two_x.append(_peak_near(freqs, spectrum, 2 * f_r))
        bpfo_amp.append(_peak_near(freqs, spectrum, defects["bpfo"]))
        bpfi_amp.append(_peak_near(freqs, spectrum, defects["bpfi"]))
        rms_all.append(float(np.sqrt(np.mean(samples**2))))

    hover = [r for r, rpm in zip(rms_all, rpms) if HOVER_BAND[0] <= rpm <= HOVER_BAND[1]]
    if not hover:
        log.warning("No sweep step fell inside the hover band, using nearest step")
        hover = [rms_all[int(np.argmin(np.abs(np.asarray(rpms) - 6000)))]]

    measurements.one_x_amplitude = max(one_x)
    measurements.two_x_amplitude = max(two_x)
    measurements.bpfo_amplitude = max(bpfo_amp)
    measurements.bpfi_amplitude = max(bpfi_amp)
    measurements.hover_rms = max(hover)
    measurements.sweep_rms = max(rms_all)

    if max(bpfo_amp) > 0.18:
        log.error(
            f"BPFO amplitude {max(bpfo_amp):.3f} g at the 0.2 g limit, "
            "suspect a damaged outer race"
        )
    elif max(bpfo_amp) > 0.1:
        log.warning(
            f"BPFO amplitude {max(bpfo_amp):.3f} g above the 0.1 g watch level, "
            "trend the bearing supplier lot"
        )

    hover_idx = int(np.argmin(np.abs(np.asarray(rpms) - 6000)))
    hover_spec = spectra[hover_idx]
    keep = freqs <= 1500.0
    measurements.hover_spectrum.x_axis = freqs[keep].tolist()
    measurements.hover_spectrum.y_axis.amplitude = hover_spec[keep].tolist()

    measurements.one_x_trace.x_axis = [float(r) for r in rpms]
    measurements.one_x_trace.y_axis.one_x = one_x
    aggs = measurements.one_x_trace.y_axis.one_x.aggregations
    aggs.p2p = float(max(one_x) - min(one_x))

    csv_path = os.path.join(tempfile.gettempdir(), "sweep_metrics.csv")
    table = np.column_stack([rpms, one_x, two_x, bpfo_amp, bpfi_amp, rms_all])
    np.savetxt(
        csv_path,
        table,
        delimiter=",",
        header="rpm,one_x_g,two_x_g,bpfo_g,bpfi_g,rms_g",
        comments="",
        fmt="%.6f",
    )
    attach.file(csv_path, "sweep_metrics.csv")
    log.info("Sweep metrics CSV attached, analysis complete")
```

The log levels are used deliberately: `debug` for internals, `info` for milestones, `warning` when the BPFO amplitude crosses a watch level below the limit, and `error` when it reaches the limit itself. The mock's injected outer-race tone sits at 0.12 g, above the 0.1 g watch level, so the warning fires on every reference run and the run still passes.

### Interactive Charts Instead of Plot Files

`hover_spectrum` and `one_x_trace` are **multi-dimensional measurements**. The dashboard renders them as interactive charts directly from the declared axes, so no plotting library is needed and no PNG is generated. The 1× trace carries a `p2p` aggregation with its own validator: a rotor whose 1× amplitude swings more than 0.5 g across the sweep fails even if every individual point is in spec. The raw per-step metrics attach as `sweep_metrics.csv` via `attach.file()` for offline analysis.

The headless input file used by the CI run is one line:

```json filename="ui.json"
{"safety_check": {"prop_removed": true}}
```


### ESC Throttle-to-RPM Characterization
URL: https://www.tofupilot.com/templates/esc-throttle-to-rpm-characterization
Map drone ESC throttle commands to motor RPM and validate linearity, deadband, response time, and KV per unit on a production test bench.

## Introduction

### ESC and Motor Overview

The **Electronic Speed Controller (ESC)** is the bridge between a drone's flight controller and its brushless motors. On every throttle update, the flight controller sends a digital or analog command (PWM, OneShot, DShot300/600/1200) that the ESC translates into three-phase commutation pulses driving the BLDC motor. A single drone arm contains three failure surfaces stacked back-to-back: the ESC FETs, the motor windings, and the prop mount. All three must be characterized as a unit, because a defect in any one corrupts the relationship between commanded throttle and delivered RPM.

![4-in-1 drone ESC board with its FET banks, MCU and capacitors next to a brushless outrunner motor with the bell removed, exposing the stator windings.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-0dba515f-ffa2-4d31-b3f0-c4146a5d81a2-4J1SoRetcRu66SkoZksZDZcUEXfZvY.jpg)

The two halves of the chain under test: the 4-in-1 ESC's FET banks and the motor's stator windings.

Modern drone ESCs run **BLHeli_32**, **BLHeli_S** (8-bit), or the open-source **AM32** firmware. BLHeli_32 was discontinued in 2024; AM32 is its actively maintained successor and flashes onto most BLHeli_32 hardware. These firmwares handle commutation timing via back-EMF zero-crossing sensing on the un-energized phase, and most now stream [bidirectional DShot eRPM telemetry](https://betaflight.com/docs/wiki/guides/current/DSHOT-RPM-Filtering) back to the flight controller. Production must verify the entire chain end-to-end, because a motor with the wrong KV stocked, a demagnetized rotor, or a misflashed ESC firmware all look identical at incoming inspection but produce wildly different in-flight behavior.

### Test Purpose

ESC throttle-to-RPM characterization measures **how a specific ESC + motor pair responds to a swept throttle command**. The procedure produces a per-unit dataset:

- **Throttle-to-RPM curve**: RPM vs commanded throttle, fitted as a near-linear function above the deadband
- **Deadband**: the lowest throttle that produces any rotation (as a rule of thumb, 3-5% of full scale on healthy units)
- **Step response time**: time from a 0-to-100% throttle command to 95% of target RPM
- **KV constant**: RPM per volt, [defined from unloaded motor speed at a known voltage](https://www.tytorobotics.com/blogs/articles/how-to-calculate-motor-poles-and-brushless-motor-kv); with a propeller installed the bench records a loaded max-RPM/V proxy instead, calibrated against a golden reference unit
- **Current signature** across the sweep

![Two-panel chart: throttle versus RPM with the 10-100% linear fit at R² 1.0000 and the 5% deadband shaded, and the 0-to-100% step response with the 95% rise time at 57 ms against the 100 ms limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-fccbe6ef-3041-4649-964d-d8f97844cd07-e5Jl7ajM9kuQElOwxq0jlkjazNQ9QZ.png)

The two results that gate the unit: linearity above the deadband, and how fast the motor answers a full step.

Per-unit characterization eliminates entire classes of post-shipment failures: yaw drift from KV-mismatched motors on a quad, ESC desync from misflashed firmware, and reduced thrust ceiling from demagnetized rotors. Thermal behavior under sustained load is covered separately by the propeller and burn-in procedures. The procedure adds 60-120 seconds per arm at end-of-line.

## Equipment & Setup

To implement ESC characterization on a production line, the following are required:

- **A motor test bench** with thrust + torque + RPM + electrical measurement, integrated with throttle signal generation
- **A regulated DC power supply** matched to pack voltage and peak current
- **The Device Under Test (DUT)**: a complete ESC + motor + prop hub assembly mounted to the bench
- **Firmware**: ESC running production-target firmware (AM32, BLHeli_32, BLHeli_S), with bidirectional DShot enabled for eRPM telemetry
- **A TofuPilot Framework procedure** to sweep throttle, capture RPM, fit the curve, and validate per-unit metrics
- The **TofuPilot Dashboard** to log results, monitor 3σ drift across the production batch, and trace each unit by serial

### Hardware Components

#### Motor Test Bench

The **RCbenchmark Series 1585** (Tyto Robotics) is the production reference for sub-5 kgf motor classes. Per its [datasheet](https://cdn.rcbenchmark.com/landing_pages/Manuals/Series%201585%20Datasheet.pdf) it integrates a **5 kgf thrust load cell**, **2 N·m torque transducer**, **50 V / 55 A continuous electrical measurement** (60 A burst), **50-80 Hz force sampling**, and an **optical RPM probe** in one fixture. The board itself generates standard **PWM (1000-2000 µs)** throttle signals; digital protocols came from a separate RC Control Board accessory that Tyto has since discontinued. For new lines, the current **Flight Stand** family exposes an official **Python API** in its software, which is the integration path this template's plug models.

![ESC board connected to a propeller-free motor with separate power and control leads](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/esc-throttle-to-rpm-characterization-physical-2026-FxrTuyK9nPC9i7ThWbwYrirdsvNFbC.webp)

The hardware-only view separates the motor phases, DC supply and control harness. RPM acquisition and the test computer are not shown.

For larger airframes (cargo, heavy-lift) the **Tyto Robotics Flight Stand 15-150 kgf** family scales the same architecture to industrial drones. For dedicated dyno testing without thrust, an **ODrive Pro/S1** controller can drive an absorber motor back-to-back for direct torque-constant measurement.

#### Power Supply

A **Rigol DP832** (triple output, 2 × 30 V / 3 A plus 5 V / 3 A) suffices for sub-100 W bench characterization. For full-power production tests on 6S packs, use a **BK Precision 9205B** (60 V / 25 A / 600 W) or an **EA Elektro-Automatik PSI 9040-60** (40 V / 60 A, 1.5 kW); note that a 6S wide-open-throttle pull on a typical 2207-class motor exceeds what smaller 360 W supplies deliver. Check the vendor's load-regulation and transient-recovery specs against your peak current step, otherwise voltage-sag-induced KV measurement error dominates the result.

#### RPM Measurement

Two methods run in parallel:

- **Optical reference**: a reflective strip on the prop hub and an optical probe on the bench (the Series 1585 integrates one with 1 eRPM resolution per its datasheet).
- **eRPM telemetry**: bidirectional DShot frames returned by the ESC over the same signal wire. Trustworthy only after we validate against optical. Following the [Betaflight convention](https://betaflight.com/docs/wiki/guides/current/DSHOT-RPM-Filtering), mechanical RPM = eRPM ÷ (motor poles ÷ 2); check which convention your bench software reports before comparing columns.

Running both lets us also validate that the ESC's self-reported eRPM matches reality, catching pole-count misconfigurations and DShot bit errors.

### Custom Firmware

The DUT ESC must ship with **bidirectional DShot enabled** (BLHeli_32 32.7.0+ or AM32) so the test station can read eRPM directly from the signal line. No special firmware build is needed; production firmware as it ships is what we test against. The flight controller does not participate in this test; the bench generates throttle signals and reads eRPM in its place.

## Test Procedure

### Overview

After mounting the DUT and connecting power, the procedure runs three phases in dependency order:

1. Connect to the bench and verify ESC identity (firmware version, DShot rate).
2. Sweep throttle 0-100% in 5% steps, log RPM and current, fit the curve, compute deadband and KV.
3. Capture a 0-to-100% throttle step at 1 kHz and measure the rise time, cross-checking the sweep phase's results.

This template runs end to end against a simulated bench plug, so `tofupilot run .` gives a green run with no hardware attached. It also demonstrates three framework capabilities worth knowing: **previous-results injection** (the step-response phase reads the sweep phase's measurements by naming a parameter after its key), a **dual-axis chart** (RPM and current on one interactive plot), and **custom aggregation types** (`r_squared` is validated like any built-in statistic).

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

- Automatic Python environment management (via `uv`)
- Operator UI (no frontend code needed)
- Measurement validation and live charts
- Process isolation between phases and equipment plugs

### Project Structure

The whole procedure is six small files:

```tree
procedure.yaml
phases/
  connect_bench.py
  throttle_sweep.py
  step_response.py
plugs/
  bench.py
utils/
  fit_linearity.py
pyproject.toml
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-esc-throttle-rpm).

### The Procedure File

`procedure.yaml` declares the unit, the bench **plug**, the **phases** in dependency order, and every **measurement** with its limits. Two validator operators worth noting: the firmware version is a **string measurement** checked with a `matches` regex, and the DShot rate is a **numeric measurement** checked with the `in` operator against the set of supported rates:

```yaml filename="procedure.yaml"
name: ESC Throttle-to-RPM Characterization
version: 0.1.0
description: Maps ESC throttle commands to motor RPM on a production bench, validating linearity, deadband, step response, and KV per unit.

unit:
  auto_identify: true
  serial_number:
    default_value: "SN00001"
  part_number:
    default_value: "ESC-2207"

plugs:
  - name: Motor Test Bench
    description: Simulated RCbenchmark-class motor test bench that generates throttle sweep and step response data.
    python: plugs.bench:MockBench
    key: bench

main:
  - name: Connect Bench
    key: connect_bench
    python: phases.connect_bench
    measurements:
      - name: ESC Firmware Version
        key: esc_firmware_version
        validators:
          - operator: matches
            expected_value: "^(BLHeli_32 32\\.\\d+|AM32 v\\d+\\.\\d+)$"
      - name: DShot Rate
        key: dshot_rate
        unit: kbps
        validators:
          - operator: in
            expected_value: [150, 300, 600, 1200]
```

The connect phase reads both values off the bench and assigns them; the framework types each measurement from the assigned value and evaluates the validators at phase close:

```python filename="phases/connect_bench.py"
def connect_bench(bench, measurements, log):
    log.info("Connecting to motor test bench")

    firmware = bench.get_firmware_version()
    measurements.esc_firmware_version = firmware
    log.info(f"ESC firmware: {firmware}")

    measurements.dshot_rate = bench.get_dshot_rate()
```

### Throttle Sweep

We step throttle in 5% increments from 0 to 100%, dwelling at each step to let RPM settle. The sweep fills a **multi-dimensional measurement with two y-axes**, RPM and current against the same throttle axis, which the dashboard renders as one interactive dual-axis chart. The linearity R² is attached to the RPM axis as a **custom aggregation**: aggregation types are free-form strings, so `r_squared` is declared and validated exactly like `mean` or `std` would be:

```yaml filename="procedure.yaml"
  - name: Throttle Sweep
    key: throttle_sweep
    python: phases.throttle_sweep
    depends_on:
      - connect_bench
    measurements:
      - name: Throttle Response
        key: throttle_response
        title: RPM and Current vs Throttle
        description: Up-sweep 0-100% in 5% steps, last-second average per step.
        x_axis:
          legend: Throttle
          unit: "%"
        y_axis:
          - legend: RPM
            key: rpm
            unit: RPM
            aggregations:
              - type: r_squared
                validators:
                  - operator: ">="
                    expected_value: 0.98
          - legend: Current
            key: current
            unit: A
      - name: Deadband
        key: deadband_pct
        unit: "%"
        validators:
          - operator: "<="
            expected_value: 5.0
      - name: KV Measured
        key: kv_measured
        unit: RPM/V
        validators:
          - operator: ">="
            expected_value: 1710
          - operator: "<="
            expected_value: 1890
```

![Time series of the 0-100% throttle up-sweep in 5% steps: throttle staircase, RPM settling after each step, and the last-second averaging window highlighted on one step.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-fa0b0a0c-78d9-4c21-b426-4f0bc21622b9-AzZQ5yesLYHfwaI1cGeMnpxtDT5vUT.png)

Each step dwells long enough for RPM to settle; only the last second is averaged into the sweep point.

The phase drives the bench, assigns both axes, and sets the aggregation value that the framework then validates. On KV: the mock behaves as an unloaded motor, so max RPM ÷ bus voltage recovers the true KV and the 1800 ±5% validators apply directly. On a real bench with the propeller installed, the same computation yields a loaded proxy; keep the computation, but derive its limits from a golden reference unit rather than the motor datasheet:

```python filename="phases/throttle_sweep.py"
from utils.fit_linearity import fit_linearity


def throttle_sweep(bench, measurements, log):
    log.info("Sweeping throttle 0-100% in 5% steps")
    throttle, rpm, current = bench.sweep(0, 100, 5)

    md = measurements.throttle_response
    md.x_axis = throttle
    md.y_axis.rpm = rpm
    md.y_axis.current = current

    fit = fit_linearity(throttle, rpm, low_pct=10)
    md.y_axis.rpm.aggregations.r_squared = fit["r_squared"]
    log.info(f"Linearity R² = {fit['r_squared']:.4f}")

    deadband = next(
        (t for t, r in zip(throttle, rpm) if r > 0.0), 100.0
    )
    measurements.deadband_pct = float(deadband)

    voltage = bench.get_bus_voltage()
    measurements.kv_measured = max(rpm) / voltage
```

### Linearity Fit

The linear model covers the 10-100% throttle region, excluding the deadband. Drone-grade motors give **R² > 0.98** with a clean fit; a sub-spec R² typically points to a noisy ESC commutation table or a winding imbalance.

```python filename="utils/fit_linearity.py"
import numpy as np


def fit_linearity(throttle, rpm, low_pct=10):
    """Least-squares linear fit on the region above the deadband.

    Returns slope, intercept, and R² of RPM vs throttle.
    """
    t = np.asarray(throttle, dtype=float)
    r = np.asarray(rpm, dtype=float)
    mask = t >= low_pct
    t, r = t[mask], r[mask]

    slope, intercept = np.polyfit(t, r, 1)
    predicted = slope * t + intercept
    ss_res = float(np.sum((r - predicted) ** 2))
    ss_tot = float(np.sum((r - np.mean(r)) ** 2))
    r_squared = 1.0 - ss_res / ss_tot if ss_tot > 0 else 1.0

    return {
        "slope": float(slope),
        "intercept": float(intercept),
        "r_squared": r_squared,
    }
```

### Step Response

The final phase issues a 0-to-100% throttle step, captures RPM at 1 kHz, and measures **time to 95% target**. As rules of thumb, racing-class motors reach 95% in around 50 ms and cinematic builds allow around 100 ms; treat both as starting points to refine against your own golden units. Slow step response indicates an ESC PID tuned for low-noise operation or an oversized prop on an undersized motor.

The phase declares its limit and its dependency in the YAML; `depends_on` is what guarantees the sweep phase has completed before its results are read:

```yaml filename="procedure.yaml"
  - name: Step Response
    key: step_response
    python: phases.step_response
    depends_on:
      - throttle_sweep
    measurements:
      - name: Rise Time
        key: rise_time_ms
        unit: ms
        validators:
          - operator: "<="
            expected_value: 100
```

The phase also demonstrates **previous-results injection**: naming a function parameter after the `throttle_sweep` phase key gives read access to that phase's recorded measurements, outcome, and duration, without any shared state or files:

```python filename="phases/step_response.py"
import numpy as np


def step_response(bench, measurements, log, throttle_sweep):
    # Previous-results injection: the parameter named after the
    # throttle_sweep phase key exposes its measurements and outcome.
    log.info(
        f"Sweep phase finished {throttle_sweep.outcome} in "
        f"{throttle_sweep.duration_ms} ms, deadband "
        f"{throttle_sweep.deadband_pct}%, KV {throttle_sweep.kv_measured:.0f}"
    )
    if throttle_sweep.outcome != "pass":
        log.warning("Sweep did not pass, step response may be unreliable")

    t_ms, rpm = bench.step_capture(1000)
    target = 0.95 * max(rpm)
    rise_time = next(t for t, r in zip(t_ms, rpm) if r >= target)
    measurements.rise_time_ms = float(rise_time)
    log.info(f"0-95% rise time: {rise_time:.1f} ms")
```

![Step response plot: unit throttle step at t = 0, RPM rising as a first-order curve with an 18 ms time constant, the 95% rise time marked at 57 ms.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-bb11216f-4f7b-453c-ba7c-abd4b8474727-GhKb2xIKHluLJTrwkgyd6oRTOwXroL.png)

The mock's 18 ms time constant puts the 95% point at 57 ms, well inside the 100 ms limit.

### Bench Plug

A plug is a persistent Python class for a device or service. The framework creates it in its own process at the start of the run, injects it into phases by parameter name, and tears it down at the end. This template ships a `MockBench` that generates a synthetic ESC + motor response (1800 KV, 3.5% deadband, 18 ms time constant), so the whole procedure runs and passes without hardware. To run against a real bench, swap the class for one built on the Flight Stand software's Python API; the phases stay unchanged.

```python filename="plugs/bench.py"
import numpy as np

BUS_VOLTAGE = 16.0  # 4S nominal
KV_TRUE = 1802.0  # RPM/V of the simulated motor
DEADBAND_TRUE = 3.5  # % throttle before rotation starts
TAU_MS = 18.0  # first-order time constant of the RPM response


class MockBench:
    def __init__(self):
        self.rng = np.random.default_rng(42)
        print("Mock bench initialized")

    def get_firmware_version(self):
        return "BLHeli_32 32.10"

    def get_dshot_rate(self):
        return 600

    def get_bus_voltage(self):
        return BUS_VOLTAGE

    def sweep(self, start_pct, stop_pct, step_pct):
        """Return (throttle %, RPM, current A) lists for an up-sweep."""
        throttle = list(range(start_pct, stop_pct + 1, step_pct))
        rpm = []
        current = []
        max_rpm = KV_TRUE * BUS_VOLTAGE
        for t in throttle:
            if t <= DEADBAND_TRUE:
                r = 0.0
                i = 0.08
            else:
                effective = (t - DEADBAND_TRUE) / (100.0 - DEADBAND_TRUE)
                r = max_rpm * effective + self.rng.normal(0, 40)
                i = 0.1 + 22.0 * effective**2 + self.rng.normal(0, 0.05)
            rpm.append(round(max(r, 0.0), 1))
            current.append(round(max(i, 0.0), 3))
        return throttle, rpm, current

    def step_capture(self, sample_rate_hz):
        """Return (time ms, RPM) for a 0-to-100% throttle step."""
        n = int(0.2 * sample_rate_hz)  # 200 ms window
        t_ms = [i * 1000.0 / sample_rate_hz for i in range(n)]
        max_rpm = KV_TRUE * BUS_VOLTAGE
        rpm = [
            max_rpm * (1.0 - np.exp(-t / TAU_MS)) + float(self.rng.normal(0, 60))
            for t in t_ms
        ]
        return t_ms, [round(max(r, 0.0), 1) for r in rpm]
```

### Run It

```bash
tofupilot run .
```

The CLI provisions the Python environment with `uv`, starts the bench plug, and executes the three phases. Headless verification for CI uses the JSON event stream: `tofupilot run . --no-tui --json` ends with `{"outcome":"PASS","exit_code":0}`.


### Propeller Thrust and Efficiency Curve
URL: https://www.tofupilot.com/templates/propeller-thrust-and-efficiency-curve
Measure drone propeller thrust, torque, and efficiency across the throttle range at end-of-line to validate propulsion performance per unit.

## Introduction

### Propulsion Overview

A drone's propulsion system is a chain: **battery → ESC → motor → propeller → air**. Each stage introduces loss, and the only output that matters in flight is **thrust per watt of electrical power**. A 6 g/W system at hover gives a 1 kg drone roughly 14 minutes of endurance on a 3,000 mAh 4S pack; a 9 g/W system stretches that to 22 minutes (both assuming ~80% usable pack capacity). The same hardware can land anywhere in this range depending on prop pitch, motor KV, ESC efficiency, and assembly quality.

![Propulsion chain bar chart: battery to ESC to motor to propeller, with typical per-stage efficiencies of 95, 92, 85 and 75 percent leaving about 56 percent of pack power as thrust.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-b805ce17-5d12-4b9d-95f3-44ab95f7313b-T8czRQAJU0P46NGfajwHw6H59hslgM.png)

Typical per-stage efficiencies compound: only about half of the pack's power ends up moving air.

End-of-line propulsion testing characterizes **thrust**, **torque**, **electrical power**, and **efficiency** as functions of throttle and RPM. The standard propeller coefficients **C_T** (thrust) and **C_P** (power), and the **figure of merit (FoM)** from the rotorcraft hover convention (McCormick), distill prop performance into dimensionless numbers comparable across diameters. The [UIUC Propeller Data Site](https://m-selig.ae.illinois.edu/props/propDB.html) (Brandt, Deters, Ananda, Dantsker, Selig) holds wind-tunnel data for hundreds of small UAV and model aircraft propellers and is the de-facto reference database.

### Test Purpose

Per-unit propulsion testing catches variance that incoming inspection cannot:

- **Motor KV** drifts ±3-5% from winding tension and magnet placement
- **Injection-molded plastic props** drift in pitch by lot; CNC carbon-reinforced props vary less but show **mass imbalance** that excites resonance
- **ESC switching losses** spread across FET vendor lots
- **Assembly defects**: misaligned shaft, cracked prop hub, loose magnet, mis-soldered phase wire raising iron/copper losses

The output of the test is a **per-unit propulsion fingerprint** that compares to a golden reference. A drone delivering only 92% of golden thrust at WOT is rejected before shipping instead of reaching a customer as an underperforming unit.

![System efficiency in g/W versus thrust across the 10-100% throttle sweep of the mock 9.45-inch prop, with the hover point at 50% throttle and 8.7 g/W marked against the 8 g/W limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-6d0526aa-04f9-4113-8f57-dcbbfe437a0a-yVR3YnrfAiGwFVi7LYEFwYYtOtS7UX.png)

Efficiency falls as thrust rises; the hover point is where the per-motor thrust meets a quarter of the all-up weight.

## Equipment & Setup

To implement propeller thrust + efficiency testing on a production line, the following are required:

- **A motor + prop test bench** with thrust load cell, reaction torque transducer, RPM tach, and full electrical metrology
- **A regulated DC power supply** matched to pack voltage and peak current
- **The Device Under Test (DUT)**: a complete motor + prop + ESC assembly mounted to the bench
- **A TofuPilot Framework procedure** to sweep throttle, compute coefficients, and validate per-unit performance
- The **TofuPilot Dashboard** to log results and trend the production line

### Hardware Components

#### Test Bench

For sub-5 kgf classes (FPV, sub-2 kg commercial), the **RCbenchmark Series 1585** is the production standard: **5 kgf thrust**, **2 N·m torque**, **50 V / 55 A electrical measurement**, **50-80 Hz force sample rate**, integrated optical RPM probe. For larger airframes the [Tyto Robotics Flight Stand 50](https://cdn.tytorobotics.com/Documents/Datasheets/Flight-Stand-50-Datasheet.pdf) scales to **50 kgf / ±30 N·m** at **100 Hz sampling (1 kHz on the Pro version)**, with a calibration methodology derived from ASTM E74/E2624 ([Tyto Methodology V2.1](https://www.tytorobotics.com/blogs/updates/tyto-calibration-methodology-v2-1), >200-point matrix on Pro, verification report on request).

![Motor and propeller mounted through a force transducer on a thrust-test fixture](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/propeller-thrust-and-efficiency-curve-physical-2026-Ig2t6pw778jcWZ4jFG5GDwyS5Gmkgb.webp)

The motor mount transfers its reaction through the force transducer to the fixed pedestal. The propeller clears the fixture; drive electronics and the surrounding test enclosure are not shown.

Sensors: **strain-gauge load cell** for thrust (pendulum or flexure mount), **reaction torque transducer** sized to the prop class (a 9-10 inch prop at WOT produces ~0.25 N·m, so pick a [Transducer Techniques RTS-200](https://www.transducertechniques.com/rts-torque-sensor.aspx) class sensor, about 1.4 N·m, for healthy margin; note the RTS range is specified in inch-ounces), **optical or Hall-effect tachometer** for RPM, two **thermocouples or NTC probes** on motor stator and ESC. High-fidelity electrical metrology benefits from a dedicated **Yokogawa WT1800E** power analyzer (basic power accuracy 0.1% of reading + 0.05% of range) with harmonic content capture, useful for differentiating ESC switching losses from motor copper losses.

#### Power Supply

Production benches typically use an **EA Elektro-Automatik PSI 9040-60** (40 V / 60 A, 1.5 kW) for 6S drone classes. Check the vendor's load-regulation and transient-recovery specs against your high-throttle current step, otherwise voltage sag during the dwell skews the efficiency calculation. Battery-powered testing is acceptable only if pack voltage is logged and the efficiency calculation references the live bus voltage at each sample.

#### Lighting and Environment

Bench lighting must not interfere with the optical RPM probe (no flicker, no 60 Hz fluorescents in the optical path). Ambient temperature controlled to 22 ± 2 °C, because air density changes ~1% per 3 °C and directly affects static thrust at constant RPM (less so at constant power).

### Custom Firmware

The ESC ships its production firmware (AM32, BLHeli_32, BLHeli_S) with **bidirectional DShot** enabled for eRPM telemetry. The flight controller is not in the loop; the bench directly issues DShot or PWM commands and records eRPM frames back. Motor temperature is read either from the ESC telemetry NTC or from a dedicated thermocouple on the stator can.

## Test Procedure

### Overview

The procedure is organized in the framework's three stages. **Setup** zeroes the bench with the motor un-energized and stamps run metadata. **Main** sweeps the throttle and computes the efficiency metrics. **Teardown** powers the bench down, and it runs even when a main phase fails, so a failed unit never leaves the bench armed:

1. Setup: verify thrust and torque zero offsets, record line, prop lot, and ambient temperature as run metadata.
2. Main: stepped throttle sweep 10-100% in 10% steps, logging thrust, torque, RPM, V, I, and both temperature probes.
3. Main: compute the efficiency curve, hover-point efficiency, peak thrust, and figure of merit.
4. Teardown: disarm the bench, always.

This template runs end to end against simulated plugs, so `tofupilot run .` gives a green run with no hardware attached. It also demonstrates four framework capabilities: **setup and teardown stages**, **plug `config` mappings** (instrument addresses live in YAML, not Python), **multiple instances of one plug class** (two thermocouple probes on different channels), and **run metadata** you can filter by on the dashboard.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

- Automatic Python environment management (via `uv`)
- Operator UI (no frontend code needed)
- Measurement validation and live charts
- Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
setup/
  zero_bench.py
phases/
  throttle_sweep.py
  compute_efficiency.py
teardown/
  power_down.py
plugs/
  bench.py
  temp_probe.py
utils/
  propulsion.py
pyproject.toml
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-propeller-thrust-efficiency).

### The Procedure File

`procedure.yaml` declares three plugs. The flight stand takes a `config` mapping, so its address and sample rate are procedure configuration rather than hard-coded Python. The two thermocouple probes are **two instances of the same plug class**, each with its own key and channel; phases receive each instance by parameter name:

```yaml filename="procedure.yaml"
name: Propeller Thrust and Efficiency Curve
version: 0.1.0
description: Measures propeller thrust, torque, and efficiency across the throttle range at end-of-line, with figure of merit and hover-point validation.

unit:
  auto_identify: true
  serial_number:
    default_value: "SN00001"
  part_number:
    default_value: "PROP-9450"

plugs:
  - name: Flight Stand
    description: Simulated Tyto-class thrust stand with electrical metrology.
    python: plugs.bench:MockFlightStand
    key: bench
    config:
      address: "192.168.1.60"
      sample_rate_hz: 80
  - name: Stator Probe
    key: temp_stator
    python: plugs.temp_probe:TempProbe
    config:
      channel: 0
  - name: ESC Probe
    key: temp_esc
    python: plugs.temp_probe:TempProbe
    config:
      channel: 1
```

The config keys become keyword arguments on the plug's `__init__`:

```python filename="plugs/temp_probe.py"
"""Thermocouple probe. Registered twice in procedure.yaml with different
channels, showing multiple instances of one plug class."""

import numpy as np


class TempProbe:
    def __init__(self, channel):
        self.channel = channel
        self.rng = np.random.default_rng(100 + channel)
        self._base = 22.0
        print(f"Temp probe on channel {channel}")

    def read_series(self, n):
        """Return n samples of a slow thermal rise in °C."""
        rise = 30.0 if self.channel == 0 else 18.0
        return [
            round(self._base + rise * (i / max(n - 1, 1)) ** 1.5
                  + float(self.rng.normal(0, 0.2)), 2)
            for i in range(n)
        ]
```

### Setup: Zero the Bench

The `setup:` stage runs before main and must pass for the test to proceed. Zero offsets outside their limits mean a drifted load cell, and there is no point sweeping a motor against a bad reference. The phase also writes **run metadata**, key-value pairs stored on the run and filterable on the dashboard:

```yaml filename="procedure.yaml"
setup:
  - name: Zero Bench
    key: zero_bench
    python: setup.zero_bench
    measurements:
      - name: Thrust Zero Offset
        key: thrust_zero_g
        unit: g
        validators:
          - operator: "<="
            expected_value: 2.0
      - name: Torque Zero Offset
        key: torque_zero_mnm
        unit: mN·m
        validators:
          - operator: "<="
            expected_value: 5.0
```

```python filename="setup/zero_bench.py"
def zero_bench(bench, measurements, log, run):
    log.info("Zeroing thrust and torque with motor un-energized")
    offsets = bench.zero_offsets()
    measurements.thrust_zero_g = offsets["thrust_g"]
    measurements.torque_zero_mnm = offsets["torque_mnm"]

    # Run metadata is filterable on the dashboard.
    run.metadata["line"] = "EOL-2"
    run.metadata["prop_lot"] = "PL-2620-A"
    run.metadata["ambient_temp_c"] = 22.4
```

### Throttle Sweep

10-100% in 10% steps. The thrust curve is a multi-dimensional measurement rendered as an interactive chart. The stator temperature series is a numeric array measurement carrying four **aggregations**, `min`, `max`, `mean`, and a custom `p2p`, each with its own validators. Aggregation types are free-form strings, so any statistic your Python computes can be declared and validated:

```yaml filename="procedure.yaml"
main:
  - name: Throttle Sweep
    key: throttle_sweep
    python: phases.throttle_sweep
    measurements:
      - name: Thrust Curve
        key: thrust_curve
        title: Thrust vs Throttle
        x_axis:
          legend: Throttle
          unit: "%"
        y_axis:
          - legend: Thrust
            key: thrust
            unit: g
      - name: Stator Temperature
        key: stator_temp_c
        unit: "°C"
        aggregations:
          - type: min
            validators:
              - operator: ">="
                expected_value: 15.0
          - type: max
            validators:
              - operator: "<="
                expected_value: 85.0
          - type: mean
            validators:
              - operator: "<="
                expected_value: 60.0
          - type: p2p
            validators:
              - operator: "<="
                expected_value: 60.0
```

![Stacked time series of the 60 s throttle sweep: throttle staircase, RPM following with a short lag, thrust rising in steps to 1838 g, and stator temperature climbing from 22 to 52 °C under the 85 °C limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-e07d4ee8-70ab-4c9f-8d41-ebb54f5faf45-YEMLM3Raa6KS6fqQxp3K94vIdhZxUl.png)

One sweep as the bench sees it: each throttle step settles before the next, and the stator warms through the run.

```python filename="phases/throttle_sweep.py"
import numpy as np


def throttle_sweep(bench, temp_stator, temp_esc, measurements, log):
    bench.arm()
    log.info("Sweeping throttle 10-100% in 10% steps")
    data = bench.sweep(10, 100, 10)

    md = measurements.thrust_curve
    md.x_axis = data["throttle"]
    md.y_axis.thrust = data["thrust_g"]

    temps = temp_stator.read_series(len(data["throttle"]))
    esc_temps = temp_esc.read_series(len(data["throttle"]))
    log.info(f"Stator {temps[-1]}°C, ESC {esc_temps[-1]}°C at end of sweep")

    measurements.stator_temp_c = temps
    aggs = measurements.stator_temp_c.aggregations
    aggs.min = float(np.min(temps))
    aggs.max = float(np.max(temps))
    aggs.mean = float(np.mean(temps))
    aggs.p2p = float(np.ptp(temps))
```

### Efficiency and Figure of Merit

Per sweep step, the phase computes **electrical power** P_elec = V_bus · I_bus and **system efficiency** η_sys = Thrust(g) / P_elec(W) in **g/W**. At wide-open throttle it additionally computes mechanical power P_mech = τ · ω to form the **figure of merit** FoM = C_T^1.5 / (C_P · √2), the rotorcraft hover convention.

The efficiency curve peaks below mid-throttle (system optimum) and falls toward WOT. The **hover-point efficiency** (throttle where thrust per motor equals ¼ AUW for a quad) is the single most important number for endurance; published propulsion test tables such as [T-Motor's per-throttle g/W data](https://uav-en.tmotor.com/2024/VL-Series-Propulsion-System_1115/1257.html) put well-matched systems in the 8-10 g/W band at low throttle, which is where this template's ≥8 g/W hover limit comes from. Calibrate it against your own golden unit.

Note the second phase reads the sweep data with `bench.last_sweep()`: a plug instance persists across phases within the run, so the sweep captured in phase one is served from plug state in phase two without re-running the motor.

![Dual-axis plot of efficiency in g/W falling and thrust in grams rising across throttle, with the 35-50% hover band shaded, peak thrust 1838 g inside the 1748-1932 g window and a figure of merit of 0.64.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-bd540ba5-dcc1-4019-b830-8fa48083ff7c-9BFonCuPUN9OPg71M9yRZo35qv0fjo.png)

Efficiency and thrust on one throttle axis: the hover band is where endurance is decided, WOT is where peak thrust is checked.

```yaml filename="procedure.yaml"
  - name: Compute Efficiency
    key: compute_efficiency
    python: phases.compute_efficiency
    depends_on:
      - throttle_sweep
    measurements:
      - name: Efficiency Curve
        key: efficiency_curve
        title: System Efficiency vs Throttle
        x_axis:
          legend: Throttle
          unit: "%"
        y_axis:
          - legend: Efficiency
            key: efficiency
            unit: g/W
      - name: Hover Efficiency
        key: hover_efficiency_gw
        unit: g/W
        validators:
          - operator: ">="
            expected_value: 8.0
      - name: Peak Thrust
        key: peak_thrust_g
        unit: g
        validators:
          - operator: ">="
            expected_value: 1748
          - operator: "<="
            expected_value: 1932
      - name: Figure of Merit
        key: figure_of_merit
        validators:
          - operator: ">="
            expected_value: 0.55
```

```python filename="phases/compute_efficiency.py"
import numpy as np

from utils.propulsion import figure_of_merit, system_efficiency


def compute_efficiency(bench, measurements, log):
    data = bench.last_sweep()
    throttle = data["throttle"]

    eff = system_efficiency(data)
    md = measurements.efficiency_curve
    md.x_axis = throttle
    md.y_axis.efficiency = eff

    # Hover point: thrust per motor equals 1/4 AUW of a 1.6 kg quad.
    hover_thrust = 400.0
    hover_idx = int(np.argmin(np.abs(np.asarray(data["thrust_g"]) - hover_thrust)))
    measurements.hover_efficiency_gw = eff[hover_idx]
    log.info(f"Hover point at {throttle[hover_idx]}% throttle, "
             f"{eff[hover_idx]:.1f} g/W")

    measurements.peak_thrust_g = max(data["thrust_g"])

    fom = figure_of_merit(data, prop_diameter_m=0.240)
    measurements.figure_of_merit = fom
    log.info(f"Figure of merit at WOT: {fom:.2f}")
```

```python filename="utils/propulsion.py"
import numpy as np

RHO = 1.225  # air density kg/m^3
G = 9.80665


def system_efficiency(data):
    """Thrust (g) per electrical watt at each sweep step."""
    thrust = np.asarray(data["thrust_g"], dtype=float)
    p_elec = np.asarray(data["volts"], dtype=float) * np.asarray(
        data["amps"], dtype=float
    )
    return [round(float(t / max(p, 0.1)), 2) for t, p in zip(thrust, p_elec)]


def figure_of_merit(data, prop_diameter_m):
    """FoM = C_T^1.5 / (C_P * sqrt(2)) at wide-open throttle (rotorcraft hover convention, see McCormick)."""
    n = data["rpm"][-1] / 60.0  # rev/s
    d = prop_diameter_m
    thrust_n = data["thrust_g"][-1] / 1000.0 * G
    torque_nm = data["torque_nm"][-1]
    p_mech = torque_nm * 2 * np.pi * n

    c_t = thrust_n / (RHO * n**2 * d**4)
    c_p = p_mech / (RHO * n**3 * d**5)
    return float(c_t**1.5 / (c_p * np.sqrt(2)))
```

### Teardown: Power Down

The `teardown:` stage runs no matter how main ended. A unit that fails its efficiency limits still leaves the bench disarmed:

```yaml filename="procedure.yaml"
teardown:
  - name: Power Down
    key: power_down
    python: teardown.power_down
```

```python filename="teardown/power_down.py"
def power_down(bench, log):
    # Teardown always runs, even when a main phase fails.
    bench.disarm()
    log.info("Bench powered down")
```

### Run It

```bash
tofupilot run .
```

The bundled `MockFlightStand` simulates a 9.45-inch (240 mm) prop from momentum theory (rotor figure of merit 0.72, drive efficiency 0.75), so the run passes with no hardware. Swap `plugs/bench.py` for a real Tyto or RCbenchmark driver to go live; phases and limits stay unchanged. Headless verification for CI: `tofupilot run . --no-tui --json` ends with `{"outcome":"PASS","exit_code":0}`.


### Camera-IMU Extrinsic Calibration
URL: https://www.tofupilot.com/templates/camera-imu-extrinsic-calibration
Calibrate camera-IMU spatial transform and time offset on a drone using AprilTag targets for accurate visual-inertial odometry.

## Introduction

### Visual-Inertial Sensing Overview

Modern autonomous drones fuse a **camera** and an **IMU** to estimate their position and orientation in real time. This pairing, known as a **visual-inertial system**, powers indoor navigation, obstacle avoidance, return-to-launch without GPS, and precision landing. Cameras provide rich geometric information about the environment but suffer from motion blur and have low update rates (30-60 Hz). IMUs run fast (200-1000 Hz) and tolerate motion blur but drift over seconds. Together they form **Visual-Inertial Odometry (VIO)**, the foundation of every GPS-denied autonomous flight stack including VINS-Mono, OpenVINS, ORB-SLAM3, and the proprietary stacks shipped by Skydio, DJI, and others.

For VIO to work, the system must know exactly **where the camera sits relative to the IMU** and **how their clocks align**. Translation and clock-offset errors at the centimeter and millisecond scale degrade VIO accuracy rapidly in aggressive flight. These parameters cannot be measured with calipers; they must be estimated from motion data per individual unit.

![Exploded view of a drone autopilot board with the camera module lifted above it, showing the rigid transform T_IC from the IMU origin to the camera optical center with labelled axes.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/camera-imu-exploded.jpeg-746fcfda-1756-445d-b1ab-e93d1a684c41-Ghy7DAUZ5UlK2Rmr5dop8Mes2Ind7q.jpg)

The extrinsic transform T_IC links the IMU body frame to the camera optical frame.

### Calibration Purpose

Camera-IMU extrinsic calibration estimates two artifacts per drone:

- **Spatial transform `T_IC`**: a 4x4 rigid-body matrix containing the **rotation (3 DoF)** and **translation (3 DoF)** between the IMU body frame and the camera optical frame.
- **Temporal offset `t_d`**: a scalar in milliseconds representing the **clock offset** between the camera shutter timestamp and the IMU sample timestamp. Even with hardware sync, exposure delays, USB transmission jitter, and rolling shutter introduce 1-50 ms of skew.

Both parameters are **per-unit**, not per-design. Two drones with identical CAD files will have different `T_IC` because the IMU chip's MEMS structure has a few hundred microns of placement tolerance on the PCB, and the camera module varies by similar amounts in its mechanical mount. Production calibration captures this variance once at end-of-line. The runnable template validates translation and time offset; production rotation validation follows the same pattern with `T_IC`'s rotational block.

![Side-by-side VIO trajectories on a 100 m loop: uncalibrated extrinsics drift 5.1 m from ground truth, calibrated extrinsics close the loop within 8 cm.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/camera-imu-vio-trajectory.png-5f9e3771-c3ef-4f8f-91b4-452e06588ef6-VXkUaUbbo4bdz0034lKglMqm5gXm4Q.png)

Same flight log and VIO stack, before and after camera-IMU calibration.

The production procedure is based on the [**Kalibr toolbox**](https://github.com/ethz-asl/kalibr/wiki/camera-imu-calibration) (ETH Zurich), the de facto industry standard. The drone is moved through an excitation pattern in front of a known calibration target while the camera records images and the IMU streams samples. A continuous-time batch optimizer estimates `T_IC` and `t_d` jointly with IMU biases and target poses. Per-unit factory calibration addresses a leading cause of autonomy-related field returns: visual-inertial divergence in aggressive maneuvers.

This template is also a working tour of the framework's execution model: a phase dependency graph whose independent checks run in parallel, an executable shell phase, a timeout on the optimizer, and a `then` override that lets a non-critical check fail without stopping the run.

## Equipment & Setup

To implement camera-IMU extrinsic calibration on a production line, the following are required:

- **A rigid AprilTag calibration target** (April grid 6x6, tag family 36h11, printed on a flat aluminum-backed board).
- **A 6-DoF motion stage** (robotic arm or manual handheld procedure with operator UI guidance).
- **The Device Under Test (DUT)** with camera and IMU rigidly mounted in the production airframe.
- **Firmware** capable of synchronized raw streaming of camera frames + IMU samples with hardware-stamped timestamps.
- **A TofuPilot Framework procedure** to validate the capture, run the optimizer, and write the results back to the DUT.
- The **TofuPilot Dashboard** to store calibration results for traceability, monitor per-unit drift, and tie each calibration to a serial.

### Hardware Components

#### AprilTag Calibration Target

We use a **6x6 AprilTag grid** (tag family 36h11), with **88 mm tag size** and **26.4 mm spacing** (0.3 tag size), matching Kalibr's example [aprilgrid target](https://github.com/ethz-asl/kalibr/wiki/Calibration-targets). The grid is **A1 size (594 x 841 mm)**, printed at 600 DPI on matte vinyl, mounted on a **6 mm aluminum-composite backing** to ensure planarity within 0.3 mm across the surface. Matte finish eliminates specular highlights that confuse tag detection.

![Four-arm quadcopter without propellers mounted on a robot opposite an AprilTag target](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/camera-imu-extrinsic-calibration-distinct-2026-vWtRSk0yGaqQRhHdEfyBPwgFrf22Nv.webp)

A rigid underside adapter holds the propeller-free quadcopter, camera and IMU for controlled motion in front of the target. The physical calibration target must match the procedure's grid specification.

The target must remain **stationary and rigid** during calibration. Print imperfections directly translate to calibration error; as a rule of thumb, a 0.5 mm bow in the board adds roughly 2 mm of translation error to `T_IC`. Replace targets every 6 months or whenever visible damage occurs.

#### Motion Stage

The drone must be moved with **excitation in all 6 degrees of freedom** during the capture. At minimum: yaw, pitch, roll rotations of ±30° plus translations of ±20 cm along each axis. Two approaches work:

- **Robotic arm** (e.g. a **Universal Robots UR-series cobot** such as the UR7e, with a custom non-magnetic end-effector). Deterministic, repeatable, fully scriptable. Capital cost ~$38k. Best for >50k units/year.
- **Operator-guided handheld** with on-screen pattern (figure-eight + axis tumble). Cost-free but adds operator variability. A tablet displays target coverage in real-time, similar to consumer smartphone gyro calibrations.

For most drone production lines we recommend starting with handheld + operator UI and graduating to a robotic arm when volume justifies it.

#### Lighting

**Diffuse LED panel lighting at 800-1200 lux** at the target surface, 5000-5500 K color temperature. Glare or shadows break AprilTag detection. The same lighting must be used across all units to avoid camera-exposure-induced timing drift. Avoid daylight near windows: overcast vs sunny days will shift exposure times and `t_d` estimates.

### Custom Firmware

The DUT firmware must expose a **synchronized streaming mode**:

- **Camera frames** at 30 Hz, hardware-timestamped at the **start of exposure** (not end-of-frame). Resolution at production native (typically 1280x720 or 1920x1080). No autoexposure changes during capture.
- **IMU samples** at 200 Hz (or production VIO rate), hardware-timestamped at the **gyro sample instant**.
- Both streams share a **single SoC clock**. If the camera ISP and IMU sit on separate MCUs, the firmware must log the **PTP or PPS sync offset** at the start of the run.

A typical capture is **60 seconds of motion**, yielding ~1800 frames and ~12000 IMU samples. The streams are written to SD or streamed over USB to the test station.

## Test Procedure

### Overview

The procedure is a small dependency graph. After a shell phase records the station platform, the capture downloads, then three independent checks fan out **in parallel**, and the optimizer runs once its two critical prerequisites pass:

1. **Record Station Platform**: executable shell phase logging the station OS.
2. **Capture Recording**: download the synchronized camera + IMU recording, check counts.
3. **Validate Excitation** and **Detect Tags**: run concurrently, both depending only on the capture.
4. **Lens Health Check**: non-critical vignetting check that cannot stop the run.
5. **Run Optimizer**: estimates `t_d` and `T_IC` under a 120 s timeout, validates reprojection error, saves the result.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

Phases without dependencies between them run in parallel on a worker pool, so the excitation and tag checks overlap for free; the `depends_on` graph is the only scheduling code you write.

### Project Structure

```tree
procedure.yaml
phases/
  capture.py
  validate_excitation.py
  detect_tags.py
  lens_health_check.py
  run_optimizer.py
plugs/
  mock_dut.py
utils/
  estimate_time_offset.py
pyproject.toml
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-camera-imu-calibration). Run it locally with `tofupilot run .`; the mock plug synthesizes the capture with a known 12 ms clock offset that the estimator recovers.

### The Procedure File

`procedure.yaml` declares the unit, the plug, and the phase graph. Note the **executable phase** (a shell command, no Python), the **parallel fan-out** after `capture`, the **`then` override** on the lens check, and the **`timeout`** on the optimizer:

```yaml filename="procedure.yaml"
name: Camera-IMU Extrinsic Calibration
version: 0.1.0
description: Captures a synchronized camera + IMU recording, validates motion excitation and tag visibility in parallel, then estimates the camera-IMU time offset and spatial transform.

unit:
  auto_identify: true
  serial_number:
    default_value: "SN00001"
  part_number:
    default_value: "VIO-CAM-01"

plugs:
  - name: Mock DUT
    description: Simulated DUT that returns a synchronized camera + IMU capture.
    python: plugs.mock_dut:MockDut
    key: dut

main:
  - name: Record Station Platform
    key: record_platform
    description: Records the station OS and architecture in the run log via a shell command.
    executable:
      command: "uname -a"

  - name: Capture Recording
    key: capture
    python: phases.capture
    measurements:
      - name: Frame Count
        key: frame_count
        validators:
          - operator: ">="
            expected_value: 1500
      - name: IMU Sample Count
        key: imu_sample_count
        validators:
          - operator: ">="
            expected_value: 10000

  - name: Validate Excitation
    key: validate_excitation
    python: phases.validate_excitation
    depends_on:
      - capture

  - name: Detect Tags
    key: detect_tags
    python: phases.detect_tags
    depends_on:
      - capture

  - name: Lens Health Check
    key: lens_health_check
    description: Non-critical vignetting check; a failure here does not stop the run.
    python: phases.lens_health_check
    depends_on:
      - capture
    then:
      fail: continue

  - name: Run Optimizer
    key: run_optimizer
    python: phases.run_optimizer
    timeout: 120s
    depends_on:
      - validate_excitation
      - detect_tags
```

A phase failing with `then: fail: continue` is still recorded as failed and fails the run outcome; the override only keeps the remaining phases executing so you collect the full diagnostic picture instead of stopping at the first non-critical red.

### Motion Excitation Validation

A common production failure is operators (or robotic arms with sloppy programs) producing motion with insufficient excitation on one or more axes: typically yaw rotation is well-exercised but Z translation is not. We require the per-axis signal variance of both the gyro and the accelerometer to exceed a minimum before the expensive optimization runs. Insufficient motion produces an **unobservable parameter** that the optimizer will silently estimate as zero with high covariance.

![Three-panel plot of per-axis gyro signal variance across the capture with the 0.05 (rad/s)² limit: roll and pitch pass, yaw fails for insufficient excitation.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/camera-imu-gyro-excitation.png-227227a8-d27a-429a-b9fa-a69e0d9e7329-K51go9Tx9dymMekJVzKFsKhNXFDo2D.png)

Per-axis excitation check: yaw was not exercised enough and the run fails before the optimizer starts.

```python filename="phases/validate_excitation.py"
import numpy as np


def validate_excitation(dut, measurements, log):
    """Check that the motion excited all six degrees of freedom.

    The optimizer silently produces unobservable parameters when an
    axis was not exercised, so each axis must carry enough signal
    variance before the expensive optimization runs.
    """
    data = dut.get_capture()

    for axis in ("x", "y", "z"):
        gyro_var = float(np.var(np.asarray(data["gyro"][axis])))
        accel_var = float(np.var(np.asarray(data["accel"][axis])))
        setattr(measurements, f"gyro_excitation_{axis}", gyro_var)
        setattr(measurements, f"accel_excitation_{axis}", accel_var)
        log.info(f"Axis {axis}: gyro var {gyro_var:.3f}, accel var {accel_var:.3f}")
```

Each of the six values is declared in the YAML with a `>= 0.05` validator, one entry per axis:

```yaml filename="procedure.yaml"
    measurements:
      - name: Gyro Excitation X
        key: gyro_excitation_x
        unit: (rad/s)²
        validators:
          - operator: ">="
            expected_value: 0.05
```

A poorly moved unit fails at this phase with a clear "rotate more" outcome instead of a cryptic optimizer divergence 90 seconds later.

### AprilTag Detection

Each frame is processed by the tag detector. We require at least **12 of 36 tags visible** in at least **80% of frames**, and no more than 10% of frames rejected for motion blur (Laplacian variance below 100):

```python filename="phases/detect_tags.py"
import numpy as np

MIN_TAGS_PER_FRAME = 12
BLUR_VARIANCE_FLOOR = 100.0


def detect_tags(dut, measurements, log):
    """Validate AprilTag visibility and motion blur across the capture."""
    data = dut.get_capture()
    tags = np.asarray(data["tags_per_frame"])
    blur = np.asarray(data["blur_variance"])

    frames_valid = float(100.0 * np.mean(tags >= MIN_TAGS_PER_FRAME))
    frames_blurred = float(100.0 * np.mean(blur < BLUR_VARIANCE_FLOOR))

    measurements.frames_valid_pct = frames_valid
    measurements.frames_blurred_pct = frames_blurred
    log.info(
        f"{frames_valid:.1f}% frames with >= {MIN_TAGS_PER_FRAME} tags, "
        f"{frames_blurred:.1f}% blurred"
    )
```

Because this phase and the excitation check both depend only on `capture`, the framework schedules them concurrently.

### Time Offset Estimation

The template recovers the camera-IMU clock offset with a genuine cross-correlation search: interpolate the gyro stream onto shifted camera timestamps and keep the shift that minimizes the squared error. The mock capture embeds a true offset of 12 ms, and the estimator returns 12.0 ms:

```python filename="utils/estimate_time_offset.py"
import numpy as np


def estimate_time_offset(t_imu, gyro_z, t_cam, cam_rate_z, search_ms=50.0, step_ms=0.5):
    """Estimate the camera-IMU clock offset by cross-correlation.

    Scans candidate offsets, interpolates the gyro signal onto the
    shifted camera timestamps, and returns the offset (in ms) that
    minimizes the sum of squared differences. Positive means the camera
    clock lags the IMU clock.
    """
    t_imu = np.asarray(t_imu)
    gyro_z = np.asarray(gyro_z)
    t_cam = np.asarray(t_cam)
    cam_rate_z = np.asarray(cam_rate_z)

    candidates = np.arange(-search_ms, search_ms + step_ms, step_ms) / 1000.0
    best_offset, best_sse = 0.0, np.inf

    for offset in candidates:
        predicted = np.interp(t_cam - offset, t_imu, gyro_z)
        sse = float(np.sum((predicted - cam_rate_z) ** 2))
        if sse < best_sse:
            best_offset, best_sse = float(offset), sse

    return best_offset * 1000.0
```

### Optimizer Phase

In production this phase wraps the full **Kalibr** batch optimizer (`kalibr_calibrate_imu_camera`; Kalibr is distributed as a ROS package, and the project provides Docker images that bundle the dependency). The template estimates the time offset for real; the spatial transform is drawn near the CAD nominal as a stand-in for the transform a real Kalibr run would estimate, and the reprojection series is synthesized from the capture's blur profile. The `timeout: 120s` on the phase covers everything it does, so a hung optimizer ends the phase with a timeout outcome instead of blocking the line. The per-frame reprojection errors are recorded as a **multi-dimensional measurement** with mean and max aggregations:

```python filename="phases/run_optimizer.py"
import numpy as np

from utils.estimate_time_offset import estimate_time_offset


def run_optimizer(dut, measurements, log):
    """Estimate the time offset and record stand-in extrinsics.

    The time offset is genuinely recovered by cross-correlating the
    gyro stream against the camera-observed angular rate. The
    translation is drawn near the CAD nominal as a stand-in for the
    transform a real Kalibr run would estimate, and the per-frame
    reprojection series is synthesized from the capture's blur
    profile. In production this phase wraps the full Kalibr batch
    optimizer and records its actual outputs.
    """
    data = dut.get_capture()

    time_offset_ms = estimate_time_offset(
        data["t_imu"], data["gyro"]["z"], data["t_cam"], data["cam_rate_z"]
    )
    measurements.time_offset_ms = time_offset_ms
    log.info(f"Estimated time offset: {time_offset_ms:.2f} ms")

    nominal = dut.read_nominal_extrinsics()["translation_mm"]
    rng = np.random.default_rng(11)
    translation = [float(v + rng.normal(0.0, 0.15)) for v in nominal]
    measurements.translation_x = translation[0]
    measurements.translation_y = translation[1]
    measurements.translation_z = translation[2]
    log.info(
        f"Translation: [{translation[0]:.2f}, {translation[1]:.2f}, "
        f"{translation[2]:.2f}] mm"
    )

    # Per-frame reprojection error: healthy frames near 0.3 px, blurred
    # frames spike but stay under the 3 px ceiling.
    blur = np.asarray(data["blur_variance"])
    errors = 0.28 + 0.04 * rng.standard_normal(blur.size)
    errors[blur < 100.0] += 0.9
    errors = np.clip(errors, 0.05, None)

    chart = measurements.reprojection_error
    chart.x_axis = list(range(errors.size))
    chart.y_axis.error = errors.tolist()
    chart.y_axis.error.aggregations.mean = float(errors.mean())
    chart.y_axis.error.aggregations.max = float(errors.max())
    log.info(f"Reprojection error: mean {errors.mean():.2f} px, max {errors.max():.2f} px")

    dut.save_calibration(translation, time_offset_ms)
    log.info("Extrinsics saved to DUT")
```

### Parameter Validation

We validate the optimized parameters against per-design bounds, derived from CAD nominals + manufacturing tolerance budget:

- **Translation per axis**: nominal ± 5 mm. The mock's CAD nominal X is 20.3 mm, hence the 15-25 mm window below. A value outside this typically indicates a misseated camera module or PCB rework defect.
- **Time offset `t_d`**: |t_d| < 30 ms, which is also Kalibr's own `--timeoffset-padding` default. Anything beyond indicates a missed hardware sync event or a USB driver buffering anomaly.
- **Reprojection error**: mean < 0.5 px as a recommended production limit for a well-calibrated drone-grade camera; max < 3.0 px (outliers above are usually motion-blurred frames that slipped past blur detection).

```yaml filename="procedure.yaml"
measurements:
  - name: Translation X
    key: translation_x
    unit: mm
    validators:
      - operator: ">="
        expected_value: 15.0
      - operator: "<="
        expected_value: 25.0
  - name: Time Offset
    key: time_offset_ms
    unit: ms
    description: Camera-to-IMU clock offset estimated by cross-correlation.
    validators:
      - operator: ">="
        expected_value: -30.0
      - operator: "<="
        expected_value: 30.0
  - name: Reprojection Error
    key: reprojection_error
    title: Reprojection Error per Frame
    x_axis:
      legend: Frame
    y_axis:
      - legend: Error
        key: error
        unit: px
        aggregations:
          - type: mean
            validators:
              - operator: "<="
                expected_value: 0.5
          - type: max
            validators:
              - operator: "<="
                expected_value: 3.0
```

![Per-frame reprojection error after optimization: flat near 0.3 px with isolated spikes on blur-rejected frames, mean 0.30 px against the 0.5 px limit and max 1.65 px against the 3.0 px limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/camera-imu-reprojection-error.png-592342b8-2850-495f-8fc3-a6fe501132f3-1dJmdg8OoTibvMQJIyexUY83g9ukH5.png)

Reprojection error per frame with the mean and max aggregations validated against their limits.

These limits start from datasheet + CAD, and TofuPilot's control charts refine them to 3σ production limits after the first 500 units.

### Saving Calibration to DUT

Validated `T_IC` and `t_d` are written to the autopilot's persistent storage. The format depends on the flight stack:

- **PX4**: [`EKF2_EV_POS_X/Y/Z`](https://docs.px4.io/main/en/computer_vision/visual_inertial_odometry) for the camera position in the body frame, and `EKF2_EV_DELAY` for the measured time offset `t_d`, via MAVLink parameter set.
- **ArduPilot**: [`VISO_POS_X/Y/Z`](https://ardupilot.org/dev/docs/mavlink-nongps-position-estimation.html) for the camera position; `VISO_ORIENT` covers only coarse orientation presets.
- **Custom stack**: a `.yaml` written to `/etc/vio/extrinsics.yaml` and a checksum logged to verify subsequent boot reads.

After write, the procedure reboots the DUT and reads back the parameters to confirm persistence. In this template the mock plug's `save_calibration` stands in for that write.


### Magnetometer Hard/Soft Iron Calibration
URL: https://www.tofupilot.com/templates/magnetometer-hardsoft-iron-calibration
Calibrate a drone magnetometer with ellipsoid fitting to remove hard and soft iron distortion and ensure accurate compass heading.

## Introduction

### Magnetometer Overview

Magnetometers measure the **direction and strength of magnetic fields**, and on a drone they provide the absolute heading reference that GPS and IMU fusion cannot give alone. Without a magnetometer, an autopilot can hold attitude but cannot tell north from south once the drone is stationary or moving slowly. Every consumer and industrial multirotor running PX4, ArduPilot, or Betaflight relies on a triaxial magnetometer to anchor yaw and enable features like **Return-to-Home**, waypoint navigation, and heading hold.

Modern drone magnetometers are **MEMS or AMR (Anisotropic Magneto-Resistive) chips** that resolve the Earth's magnetic field, which ranges from about 25 µT near the equator to 65 µT near the poles. Typical parts used in production include the [**QMC5883L**](https://www.qstcorp.com/upload/pdf/202512/13-52-04%20QMC5883L%20Datasheet%20Rev.%20B.pdf) (16-bit ADC, 2 mGauss field resolution), the **IST8310** (14-bit, up to 200 Hz, 166 Hz in its low-noise 16x-averaging mode), and the [**MMC5983MA**](https://media.digikey.com/pdf/Data%20Sheets/MEMSIC%20PDFs/MMC5983MA_RevA_4-3-19.pdf) (18-bit, 0.4 mGauss RMS noise, built-in degaussing). Each has different noise floors and bias-drift characteristics, which matters when defining production limits.

![Flight-controller PCB with the magnetometer chip highlighted and its X, Y, Z sensor axes drawn at the package.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-3e015ad9-1e5c-4bb3-b76e-ab460a65c7be-FBxNcxInmmiOVMcD1lLhiQwu2joWC2.jpg)

The magnetometer package and its sensor axes on the flight-controller board.

The challenge is that the magnetometer never sees a clean Earth field. The drone itself carries **ferromagnetic materials** (screws, ESC traces, motor magnets) and **current-carrying conductors** (battery leads, motor phases) that distort the field at the sensor location. Two distinct error sources result, and both must be characterized per-unit at the end of the production line.

### Calibration Purpose

The goal of hard/soft iron calibration is to **map the distorted ellipsoidal locus** of magnetometer readings back to a clean sphere centered at the origin. When the drone is rotated through all orientations in a uniform magnetic field, raw readings should trace a perfect sphere of radius equal to the local field magnitude. In practice they trace an **off-center, rotated ellipsoid** because of two effects:

- **Hard iron distortion**: additive bias from permanently magnetized materials on the airframe (motor magnets, magnetized screws). Shifts the sphere center away from the origin without changing its shape.
- **Soft iron distortion**: multiplicative scale and skew from ferromagnetic materials (steel mounts, battery casing) that reshape the ambient field. Stretches the sphere into an ellipsoid.

![Side-by-side 3D scatter plots: raw magnetometer samples forming an off-center ellipsoid, and corrected samples forming a sphere centered at the origin, with the fitted hard iron offset and field strength.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-2002e4d4-5525-4a07-9462-caa26ff50862-wW3TxE3PSQW0JXrV06A5zMUpmwqcbk.png)

Raw samples from the mock DUT and the same samples after applying the fitted hard and soft iron correction.

The calibration produces two artifacts per drone: a **3-element offset vector** (hard iron) and a **3x3 transformation matrix** (soft iron). At runtime, every raw reading `m_raw` is corrected as `m_cal = SoftIron · (m_raw - HardIron)` before being passed to the attitude estimator. These parameters are unique to each airframe and must be written to non-volatile storage on the flight controller (in PX4 this means the `CAL_MAG0_*` parameters written to `/fs/mtd_caldata` during [factory calibration](https://docs.px4.io/main/en/advanced_config/imu_factory_calibration)).

Per-unit factory calibration addresses a leading source of consumer drone field returns: erratic compass behavior leading to **fly-aways, toilet-bowling, and failed Return-to-Home**. The procedure adds 60-90 seconds to end-of-line testing.

This template also serves as a working tour of several TofuPilot Framework capabilities: live operator UI updates during acquisition, automatic phase retry when coverage is insufficient, a JSON measurement holding the calibration coefficients, numeric-array aggregations, an interactive residual chart, and a raw-data attachment.

## Equipment & Setup

To implement hard/soft iron calibration for a drone magnetometer on a production line, the following equipment is required:

- **A magnetically clean test cell**: area free of moving ferromagnetic objects within 1 m of the DUT, with stable ambient field (or active cancellation if the building has structural steel).
- **A controlled rotation mechanism**: either a 3-axis motorized gimbal, a turntable with operator-guided tilt, or a calibrated handheld procedure with a visual coverage UI.
- **A reference magnetometer** for ambient field verification, typically a high-stability fluxgate.
- **The Device Under Test (DUT)** equipped with the magnetometer to be calibrated, mounted in the same orientation as in the final airframe.
- **Firmware** with a triggerable raw-mag streaming mode (no on-chip filtering, no fusion corrections applied).
- **A TofuPilot Framework procedure** to acquire samples, fit the ellipsoid, validate the result, and write parameters back to the DUT.
- The **TofuPilot Dashboard** to store calibration parameters for traceability, monitor 3σ drift across the production batch, and trace each unit by serial.

### Hardware Components

#### Helmholtz Coil (Optional, Deterministic Setup)

For high-volume or high-accuracy production, a **3-axis Helmholtz coil** generates a known, uniform magnetic field around the DUT and removes dependency on ambient conditions. The [**Bartington HC2**](https://bartingtondownloads.com/wp-content/uploads/DS2613.pdf) (1 m nominal coil diameter, field homogeneous to better than 0.1% over a 13.6 cm cube at the center) fits drone-sized boards comfortably, paired with a **Mag-03 fluxgate reference** to verify field stability. The Helmholtz approach allows the DUT to remain stationary while the field is rotated electronically through known vectors, making the procedure deterministic and fully scriptable.

![Three-axis Helmholtz coil with a flight-controller PCBA mounted at the center on a non-magnetic post.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-db0c2942-a586-409c-8b1a-d74ca4f45405-jxG3PHADaTizyTc1bEG5vbcdJ46eDK.jpg)

Three-axis Helmholtz coil with the PCBA at the homogeneous center.

Note that Helmholtz coil systems need roughly **30 minutes of warm-up**, and the field at the DUT location must be verified with the reference magnetometer before each shift. For most consumer drone lines, the simpler Earth-field method below is sufficient.

#### Earth-Field Rotation Jig (Default Setup)

Most production lines calibrate against the **local Earth magnetic field**, which is stable to a fraction of a percent on magnetically quiet days at a fixed location. The DUT is mounted on a non-magnetic (aluminum or 3D-printed) jig and either rotated by an operator following a UI prompt, or by a 3-axis motorized stage that runs a fixed coverage pattern.

![Non-magnetic 3D-printed two-axis cradle holding a flight-controller PCB on a turntable, the default Earth-field rotation jig.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-eb2ea95c-3b4c-43c4-8355-c4e735ab6c52-SU3iXtM6OBfhEHReyKpc4c6Ry60KRD.jpg)

Two-axis 3D-printed cradle on a turntable: tilt plus rotation covers the sphere.

The jig must be mounted away from the test bench's steel frame (typical 30 cm clearance), and operators must be instructed not to wear steel-toed boots or magnetic badge clips near the test cell.

#### Reference Magnetometer

A [**Bartington Mag-03**](https://www.bartingtondownloads.com/wp-content/uploads/DS0013.pdf) triaxial fluxgate (±70 µT lowest range, noise floor under 10 pT/√Hz at 1 Hz) is the production standard for verifying the local field magnitude before each batch. The reference reading establishes the **target sphere radius** for ellipsoid fitting: if the corrected drone readings don't sit on a sphere of this radius, calibration has failed regardless of fit residual.

### Custom Firmware

The DUT firmware must expose a **raw streaming mode** that bypasses any on-chip averaging, soft-iron correction, or fusion filtering. During calibration the procedure logs:

- **Timestamp** (ms precision)
- **Raw magnetometer X/Y/Z** in µT (or ADC counts with known LSB scale)
- **Temperature** (used to flag thermal drift during calibration, and to disqualify samples taken during warm-up)
- **Orientation hint** (quaternion from the IMU's gyro+accel fusion), used by the UI to track coverage

Sample rate should be 50-100 Hz; the algorithm needs a few hundred well-distributed samples, which takes 30-60 seconds at human rotation speed. PX4's onboard calibration procedure (operator-rotated, all-orientation coverage) is a good reference baseline.

## Test Procedure

### Overview

The procedure runs two phases:

1. **Acquire Samples**: pull raw samples from the DUT while the operator rotates it, show live coverage feedback, and retry the acquisition if orientation coverage is insufficient.
2. **Fit Calibration**: fit the ellipsoid, validate offsets and residuals, attach the raw data, and save the calibration to the DUT.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  acquire_samples.py
  fit_calibration.py
plugs/
  mock_dut.py
utils/
  coverage.py
  fit_ellipsoid.py
pyproject.toml
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-magnetometer-calibration). Run it locally with `tofupilot run .`; the mock plug synthesizes samples so no hardware is needed.

### The Procedure File

`procedure.yaml` declares the procedure identity, the unit (auto-identified from defaults, so the run starts without operator input), and the mock DUT plug. The Dashboard ID for automatic upload lives in this same file once the procedure is linked:

```yaml filename="procedure.yaml"
name: Magnetometer Hard/Soft Iron Calibration
version: 0.1.0
description: Acquires magnetometer samples across all orientations, fits an ellipsoid to estimate hard and soft iron distortion, validates residuals, and saves the calibration.

unit:
  auto_identify: true
  serial_number:
    default_value: "SN00001"
  part_number:
    default_value: "FC-MAG-01"

plugs:
  - name: Mock DUT
    description: Simulated flight controller streaming raw magnetometer samples.
    python: plugs.mock_dut:MockDut
    key: dut
```

### Acquisition with Live UI and Retry

The acquisition phase declares two display components and a retry policy. While Python runs, it pushes coverage updates straight to the operator screen by assigning to `ui.<key>`. If coverage lands below the target, the phase calls `phase.retry()` and the framework re-runs it, up to the declared limit, with each attempt recorded on the dashboard:

```yaml filename="procedure.yaml"
main:
  - name: Acquire Samples
    key: acquire_samples
    python: phases.acquire_samples
    retry:
      limit: 3
      delay: 1s
    ui:
      components:
        - key: rotation_progress
          type: progress
          label: "Rotation Coverage"
          default_value: 0
        - key: status
          type: text
          label: "Status"
          default_value: "Waiting for samples..."
    measurements:
      - name: Coverage
        key: coverage_pct
        unit: "%"
        description: Fraction of orientation bins hit during the tumble.
        validators:
          - operator: ">="
            expected_value: 80
```

```python filename="phases/acquire_samples.py"
from utils.coverage import compute_coverage

SAMPLE_COUNT = 600
COVERAGE_TARGET = 80.0


def acquire_samples(dut, measurements, ui, phase, log):
    """Pull raw samples from the DUT and check orientation coverage.

    Display components update live while the acquisition runs. If the
    operator missed too many orientations, the phase asks for another
    tumble and retries.
    """
    ui.status = "Acquiring samples, rotate the drone through all orientations..."
    ui.rotation_progress = 10

    samples = dut.acquire_samples(SAMPLE_COUNT)
    log.info(f"Acquired {len(samples)} samples")
    ui.rotation_progress = 60

    coverage = compute_coverage(samples)
    measurements.coverage_pct = coverage
    ui.rotation_progress = int(coverage)

    if coverage < COVERAGE_TARGET:
        ui.status = f"Coverage {coverage:.0f}% is below {COVERAGE_TARGET:.0f}%, rotate around more axes"
        log.warning(f"Insufficient coverage ({coverage:.1f}%), retrying acquisition")
        phase.retry()

    ui.status = f"Coverage {coverage:.0f}%, acquisition complete"
    log.info(f"Coverage {coverage:.1f}%")
```

The mock DUT simulates the classic operator mistake on the first attempt (rotation around a single axis) so the retry path actually executes: attempt 1 measures ~23% coverage and retries, attempt 2 tumbles the full sphere and passes at 100%.

### Coverage Validation

Ellipsoid fitting requires samples distributed across all orientations. A common production failure is operators rotating only around two axes, leaving a coverage gap that produces a degenerate fit. We bin the sample directions against a **Fibonacci sphere grid** and require at least 80% of bins hit before proceeding:

![Orientation coverage on a 100-bin Fibonacci sphere: rotation around Z only hits 23% of bins and retries, a full tumble hits 100% and passes the 80% limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-03fa82b4-0628-4c31-93a8-5da3c90a3451-8rNTwxerJ62BtCTEM3l0pIYkb9rIUA.png)

Coverage bins hit by the mock's two attempts, on the same 100-bin grid the phase uses.

```python filename="utils/coverage.py"
import numpy as np


def compute_coverage(samples, bins=100):
    """Fraction of orientation bins hit by the sample set, in percent.

    Bins are directions on a Fibonacci sphere; each sample counts toward
    its nearest bin. Rotating around a single axis leaves most bins
    empty, which is the classic operator mistake this check catches.
    """
    samples = np.asarray(samples, dtype=float)
    centered = samples - samples.mean(axis=0)
    norms = np.linalg.norm(centered, axis=1)
    directions = centered[norms > 1e-9] / norms[norms > 1e-9, None]

    i = np.arange(bins)
    phi = np.arccos(1.0 - 2.0 * (i + 0.5) / bins)
    theta = np.pi * (1.0 + 5.0**0.5) * i
    grid = np.column_stack(
        [np.sin(phi) * np.cos(theta), np.sin(phi) * np.sin(theta), np.cos(phi)]
    )

    nearest = np.argmax(directions @ grid.T, axis=1)
    return 100.0 * len(np.unique(nearest)) / bins
```

### Ellipsoid Fitting

The core algorithm fits a general quadric to the raw 3D points:

`a·x² + b·y² + c·z² + 2f·yz + 2g·xz + 2h·xy + 2p·x + 2q·y + 2r·z = 1`

This is solved via **least-squares decomposition** of the 9-parameter system ([Renaudin et al., 2010](https://onlinelibrary.wiley.com/doi/10.1155/2010/967245) is the canonical reference). The resulting quadric is decomposed via **eigenvalue analysis** into the hard iron offset (ellipsoid center) and the soft iron matrix that maps the ellipsoid back onto a sphere:

```python filename="utils/fit_ellipsoid.py"
import numpy as np


def fit_ellipsoid(samples):
    """Least-squares quadric fit of magnetometer samples.

    Fits a*x^2 + b*y^2 + c*z^2 + 2f*yz + 2g*xz + 2h*xy + 2p*x + 2q*y
    + 2r*z = 1, then decomposes the quadric into a hard iron offset
    (ellipsoid center) and a soft iron correction matrix that maps the
    ellipsoid back onto a sphere.

    Returns (offset, soft_iron, field_strength) where
    corrected = soft_iron @ (raw - offset) lies on a sphere of radius
    field_strength.
    """
    s = np.asarray(samples, dtype=float)
    x, y, z = s[:, 0], s[:, 1], s[:, 2]

    design = np.column_stack(
        [x * x, y * y, z * z, 2 * y * z, 2 * x * z, 2 * x * y, 2 * x, 2 * y, 2 * z]
    )
    coeffs, *_ = np.linalg.lstsq(design, np.ones_like(x), rcond=None)
    a, b, c, f, g, h, p, q, r = coeffs

    quadric = np.array([[a, h, g], [h, b, f], [g, f, c]])
    linear = np.array([p, q, r])
    offset = -np.linalg.solve(quadric, linear)

    # Radius^2 of the centered quadric: x^T Q x = 1 + offset^T Q offset
    k = 1.0 + offset @ quadric @ offset
    evals, evecs = np.linalg.eigh(quadric / k)
    radii = 1.0 / np.sqrt(evals)
    field_strength = float(np.prod(radii) ** (1.0 / 3.0))

    # Map the ellipsoid onto a sphere of radius field_strength.
    whitening = evecs @ np.diag(np.sqrt(evals)) @ evecs.T
    soft_iron = field_strength * whitening

    return offset, soft_iron, field_strength


def apply_calibration(samples, offset, soft_iron):
    s = np.asarray(samples, dtype=float)
    return (s - offset) @ np.asarray(soft_iron).T
```

Running against the mock data (true offset [120, -45, 80] µT, 48 µT field, 0.3 µT noise), the fit recovers [120.0, -45.0, 80.0] µT and a field strength of 48.3 µT.

### Calibration Phase

The second phase fits the ellipsoid, records every artifact, and saves the result. The framework injects `dut`, `measurements`, `attach`, and `log` by matching parameter names. The full coefficient set is stored as a **JSON measurement**, the corrected magnitudes as a **numeric array with aggregations**, the residuals as an interactive **multi-dimensional chart**, and the raw samples as a **CSV attachment** via `attach.data()`:

```python filename="phases/fit_calibration.py"
import io

import numpy as np

from utils.fit_ellipsoid import apply_calibration, fit_ellipsoid


def fit_calibration(dut, measurements, attach, log):
    """Fit the ellipsoid, validate residuals, and save the calibration."""
    samples = np.asarray(dut.get_last_samples())

    offset, soft_iron, field = fit_ellipsoid(samples)
    log.info(
        f"Hard iron offset: [{offset[0]:.1f}, {offset[1]:.1f}, {offset[2]:.1f}] µT, "
        f"field {field:.1f} µT"
    )

    measurements.offset_x = float(offset[0])
    measurements.offset_y = float(offset[1])
    measurements.offset_z = float(offset[2])

    # JSON measurement: the exact coefficients written to the DUT.
    measurements.calibration_coefficients = {
        "hard_iron_ut": [round(float(v), 3) for v in offset],
        "soft_iron": [[round(float(v), 5) for v in row] for row in soft_iron],
        "field_strength_ut": round(field, 3),
    }

    corrected = apply_calibration(samples, offset, soft_iron)
    magnitudes = np.linalg.norm(corrected, axis=1)

    measurements.sphericity_ratio = float(magnitudes.std() / magnitudes.mean())

    # Numeric array with aggregations validated by the framework.
    measurements.field_magnitude = magnitudes.tolist()
    measurements.field_magnitude.aggregations.mean = float(magnitudes.mean())
    measurements.field_magnitude.aggregations.std = float(magnitudes.std())

    # Multi-dimensional chart: residual per sample, rendered interactively.
    residuals = magnitudes - field
    chart = measurements.residual_chart
    chart.x_axis = list(range(len(residuals)))
    chart.y_axis.residual = residuals.tolist()
    aggs = chart.y_axis.residual.aggregations
    aggs.mean = float(residuals.mean())
    aggs.std = float(residuals.std())
    aggs.p2p = float(np.ptp(residuals))

    # Attach the raw samples for offline analysis.
    buffer = io.StringIO()
    buffer.write("mag_x_ut,mag_y_ut,mag_z_ut\n")
    for row in samples:
        buffer.write(f"{row[0]:.3f},{row[1]:.3f},{row[2]:.3f}\n")
    attach.data(buffer.getvalue().encode(), "raw_samples.csv")

    dut.save_calibration(
        [float(v) for v in offset], [[float(v) for v in row] for row in soft_iron]
    )
    log.info("Calibration saved to DUT")
```

### Parameter Validation

Per-axis bounds derived from the magnetometer datasheet are declared as validators in `procedure.yaml`, then refined via TofuPilot 3σ analytics over the first batch:

- **Hard iron offset per axis**: typically within ±300 µT. A value above this suggests a magnetized screw or rework defect.
- **Sphericity ratio** (std of corrected magnitudes over their mean): a perfect calibration gives < 0.02; production limit 0.05.
- **Residual chart aggregations**: mean near zero, std and peak-to-peak bounded.

The YAML below shows the X-axis offset; `offset_y` and `offset_z` are declared identically:

```yaml filename="procedure.yaml"
measurements:
  - name: Hard Iron Offset X
    key: offset_x
    unit: µT
    validators:
      - operator: ">="
        expected_value: -300
      - operator: "<="
        expected_value: 300
  - name: Sphericity Ratio
    key: sphericity_ratio
    description: Std of corrected magnitudes divided by their mean.
    validators:
      - operator: "<="
        expected_value: 0.05
  - name: Calibration Coefficients
    key: calibration_coefficients
    description: Hard iron offset vector and soft iron matrix written to the DUT.
  - name: Corrected Field Magnitude
    key: field_magnitude
    unit: µT
    aggregations:
      - type: mean
        unit: µT
        validators:
          - operator: ">="
            expected_value: 20
          - operator: "<="
            expected_value: 70
      - type: std
        unit: µT
        validators:
          - operator: "<="
            expected_value: 1.0
  - name: Residual Chart
    key: residual_chart
    title: Corrected Magnitude Residual
    x_axis:
      legend: Sample
    y_axis:
      - legend: Residual
        key: residual
        unit: µT
        aggregations:
          - type: mean
            validators:
              - operator: ">="
                expected_value: -0.5
              - operator: "<="
                expected_value: 0.5
          - type: std
            validators:
              - operator: "<="
                expected_value: 1.0
          - type: p2p
            validators:
              - operator: "<="
                expected_value: 6.0
```

The interactive residual chart replaces the static PNG plot you would otherwise attach manually:

![Corrected magnitude residual per sample after ellipsoid fitting: flat around zero within ±1 µT, mean -0.001 µT, std 0.307 µT and p2p 2.09 µT against their limits.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-500760ea-524c-4b7e-b593-9be467bfcf0c-iuvp6cqeVOAPCfbqZU9Pt5t44JSd2B.png)

Residual per sample with the mean, std and peak-to-peak aggregations validated against their limits.

These limits start from the chip datasheet, with one subtlety worth knowing: the ±300 µT hard-iron limit assumes the sensor runs on a wide range. On a QMC5883L, the ±2 Gauss (±200 µT) range would saturate before the limit is reached, so production streaming must use the ±8 Gauss range. Limits then tighten over time as TofuPilot's control charts reveal the actual process distribution. Any unit outside 3σ but inside the absolute limits is flagged for engineering review without failing the test.

### Saving Calibration to DUT

Validated parameters are written to the flight controller's persistent storage. On PX4 this means setting `CAL_MAG0_XOFF`, `CAL_MAG0_YOFF`, `CAL_MAG0_ZOFF` (hard iron) and `CAL_MAG0_XSCALE`, `CAL_MAG0_YSCALE`, `CAL_MAG0_ZSCALE` (diagonal soft iron) via MAVLink parameter set, persisted through the [PX4 factory calibration storage](https://docs.px4.io/main/en/advanced_config/imu_factory_calibration). ArduPilot uses `COMPASS_OFS_*` and `COMPASS_DIA_*` / `COMPASS_ODI_*` for full 3x3 soft iron. The procedure must verify the write by reading back the parameters and confirming a power-cycle persists them. In this template the mock plug's `save_calibration` stands in for that write.


### IMU Thermal Calibration
URL: https://www.tofupilot.com/templates/imu-thermal-calibration
Improve your IMU's accuracy by calibrating for temperature changes. Includes automated thermal cycling, drift compensation, and production-ready test scripts.

## Introduction

### IMU Overview

IMUs (Inertial Measurement Units) are remarkable sensors that measure the **movement** and **orientation** of the devices they are embedded in. Since the rise of smartphones, their size and cost have been drastically reduced, enabling a wide range of applications in robotics and drones, where they enable **autonomous navigation and guidance**.

IMUs combine a **gyroscope** (measuring angular velocity in °/s or rad/s) and an **accelerometer** (measuring linear acceleration in m/s² or g). Gyroscopes measure rotation but can drift over time, while accelerometers sense velocity changes and tilt but cannot distinguish between motion and gravity. **Sensor** **fusion algorithms** correct gyroscope drift with accelerometer data and stabilize accelerometer noise with gyroscope input, ensuring accurate motion tracking.

At the heart of an IMU is a **Micro-Electro-Mechanical System** (MEMS): a tiny structure, often just a few microns in size (about 1/100th the width of a human hair), that moves slightly in response to forces. These movements generate changes in voltage, which are measured by the sensor, converted into numerical data, and transmitted to your system through communication protocols such as SPI or I²C.

![Close-up of a MEMS-based IMU component showing micro-scale mechanical elements for motion sensing.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/templates/da36041c763820b2563512c3064cdfdc22523633-1032x720-1wJmp0qpECQF7wqr9XhfH8pj1PzHmI.png)

Micro-scale mechanical elements within a MEMS-based Inertial Measurement Unit (IMU).

For everyday devices like smartphones or tablets, precision isn't as important, so IMUs are not necessary factory-calibrated for temperature. But for drones and robots used in outdoor conditions, accuracy matters and **calibrating the IMU for temperature** can greatly improve its performance.

### Calibration Purpose

Thermal calibration involves placing the IMU-equipped **Printed Circuit Board Assembly** (PCBA) in a **climate chamber** for several hours while varying the temperature. With the IMU kept flat and motionless, any changes in its measurements are attributed to temperature effects. The resulting data is used to calculate and **save calibration parameters** specific to each board. During operation, these parameters are applied in real time to compensate for temperature-induced measurement variations.

![Graph displaying a polynomial fit curve used for thermal calibration of an IMU, showing the relationship between temperature and compensation values.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/templates/f32725a889a9ad84c6ec1bcad3f8e2103caa1665-900x506-nz2tcm0BSX1Hth5DDzkpzJgQMVTxcl.png)

Curve showing accelerometer X vs. temperature with a polynomial fit for calibration.

Most **Electronic Manufacturing Service** (EMS) providers have climate chambers for tasks like stress testing, so reserving one for your boards shouldn't be an issue. For lab use, you can purchase a small climate chamber for under $5,000, suitable for testing a few boards. The cost largely depends on the chamber's **temperature range**. For instance, testing a drone designed for outdoor use may require a chamber that operates between **-20°C and 70°C** to simulate extreme environmental conditions.

Note that the temperature measured by the IMU will always be higher than the chamber's temperature due to the internal heat generated by the IMU's casing and the PCBA it is mounted on.

## Equipment & Setup

To implement thermal calibration for the IMU in a drone, you will need the following:

* **A climate chamber** capable of reaching temperatures between **-20°C and 70°C**.
* **A support structure** to hold the PCBA in the chamber and a power supply.
* **A Device Under Test (DUT)**, equipped with the IMU that requires calibration.
* **Firmware for the device**, with a triggerable mode to log the raw IMU data.
* **A TofuPilot Framework procedure** to:
  * Retrieve data from each board after the calibration process.
  * Calculate the calibration parameters.
  * Verify the quality of the calibration and ensure there are no defects.
  * Save the calibration parameters to the product.
* The **TofuPilot Dashboard** to store calibration data for traceability and analytics.

### Hardware Components

#### Climate Chamber

We will use the **Votsch VT4002** temperature test chamber, which has a temperature range of **-40°C to +130°C**. With its **16-liter volume**, it is ideal for designing the test in the lab or for small production runs.

![Votsch VT4002 test chamber with a rectangular design, viewing window, side control panel, and stainless steel exterior.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/templates/4602a575d031f2ebfe7cf9d4570402908c205623-2400x1800-rTbvnyUOuSb5JtMXOp12ZlOvPJDSWW.png)

The Votsch VT4002 is a compact lab test chamber with a -40°C to +130°C range.

#### Cycle Program

The IMU's response is not necessarily the same when it heats up versus when it cools down. To ensure the calibration accurately reflects the sensor's real thermal behavior, we will set a **calibration duration of 2 hours**, with **4 temperature cycles** ranging from **-20°C to 70°C**.

#### Support Structure

For the lab setup, we'll create a simple **3D-printed support** using **ESD filament**. Foam will be added to reduce vibrations from the climate chamber. While these vibrations are likely filtered out during data processing, minimising them is still beneficial.

In mass production, we can collaborate with the **test team at our EMS** to design a support that accommodates multiple boards to optimise throughput, provides power to them, and is adapted to the dimensions of their temperature test chambers.

### Custom Firmware

Throughout the entire calibration process, the IMU needs to log its measurements at a frequency of at least 10 Hz. To enable this, we will develop a **special logging mode** in the firmware. When activated, this mode will start data acquisition and record the following parameters:

* **Timestamp**: To track when each measurement was taken.
* **Gyroscope data**: X, Y, and Z axes in degrees per second (deg/s).
* **Accelerometer data**: X, Y, and Z axes in meters per second squared (m/s²).
* **Internal sensor temperature**: This will be used as the reference for calibration.

The logged data can be stored as a **JSON or CSV file** in the PCBA memory or on an SD card. Most climate chambers have an **external pin** that activates at program start. Connecting this pin to a stabilized power supply outside the chamber allows the supply to **switch on automatically** at the program's start and **off at its end**, ensuring logging occurs only during the thermal cycle and not afterward, such as when the operator retrieves the board.

## Test Procedure

### Overview

Once the thermal cycle is complete, the chamber powers off and the log file is ready for retrieval and processing. Operators will remove the PCBAs from their calibration support and connect them **to the test station**. At this point, the TofuPilot procedure takes over to:

1. Connect to the Device Under Test (DUT) and retrieve the acquisition file.
2. Validate the acquired data (noise density, temperature sensitivity).
3. Compute the polynomial calibration.
4. Validate the calibration quality (residuals, R²).
5. Save calibration results to the DUT internal memory.
6. Provide a global pass/fail status.
7. Stream results to TofuPilot for traceability and analytics.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

The whole procedure is six small files plus the data sample:

```tree
procedure.yaml
phases/
  connect_dut.py
  thermal_calibration.py
plugs/
  mock_dut.py
utils/
  calibrate_sensor.py
  compute_noise_density.py
  compute_r2.py
  compute_residuals.py
  compute_temp_sensitivity.py
data/
  imu_raw_data.csv
pyproject.toml
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-imu-thermal-calibration).

### The Procedure File

`procedure.yaml` is the entry point. It declares:

* The unit being tested (auto-identified here, so the run starts without operator input)
* A **plug** (`Mock DUT`) that simulates the board
* Two **phases**: `Connect DUT` and `Thermal Calibration`
* All the **measurements** with their limits

Here's the top of the file:

```yaml filename="procedure.yaml"
name: IMU Thermal Calibration
version: 0.1.0
description: Computes per-axis polynomial thermal compensation for an IMU's accelerometer and gyroscope, then validates fit quality.

unit:
  auto_identify: true
  serial_number:
    default_value: "SN00001"
  part_number:
    default_value: "PCB01"

plugs:
  - name: Mock DUT
    description: Simulated device under test that returns logged IMU CSV data.
    python: plugs.mock_dut:MockDut
    key: dut

main:
  - name: Connect DUT
    key: connect_dut
    python: phases.connect_dut

  - name: Thermal Calibration
    key: thermal_calibration
    python: phases.thermal_calibration
    depends_on:
      - connect_dut
```

`python: plugs.mock_dut:MockDut` tells TofuPilot to instantiate the `MockDut` class. `python: phases.connect_dut` tells it to call the `connect_dut()` function in that file. The order of phases is enforced by `depends_on`.

### Mock DUT Plug

A plug is a persistent Python class for a device or service. The framework creates it once at the start of the run and tears it down at the end. For this template we use a mock that reads a CSV file instead of talking to real hardware:

```python filename="plugs/mock_dut.py"
import time
from pathlib import Path

import pandas as pd


CSV_PATH = Path(__file__).resolve().parent.parent / "data" / "imu_raw_data.csv"


class MockDut:
    """Simulated DUT that returns IMU log data from a CSV file."""

    def __init__(self):
        self._connected = False
        print("Mock DUT initialized")

    def connect(self) -> bool:
        print("Connecting to mock DUT...")
        time.sleep(0.2)
        self._connected = True
        return True

    def get_imu_data(self) -> dict:
        df = pd.read_csv(CSV_PATH, delimiter="\t")
        return {
            "acc_data": {
                "temperature": df["imu.temperature"].tolist(),
                "acc_x": df["imu.acc.x"].tolist(),
                "acc_y": df["imu.acc.y"].tolist(),
                "acc_z": (df["imu.acc.z"] - 9.80600).tolist(),
            },
            "gyro_data": {
                "temperature": df["imu.temperature"].tolist(),
                "gyro_x": df["imu.gyro.x"].tolist(),
                "gyro_y": df["imu.gyro.y"].tolist(),
                "gyro_z": df["imu.gyro.z"].tolist(),
            },
        }

    def save_accelerometer_calibration(self, coefficients: dict) -> None:
        print(f"Saved accelerometer calibration: {list(coefficients.keys())}")

    def save_gyroscope_calibration(self, coefficients: dict) -> None:
        print(f"Saved gyroscope calibration: {list(coefficients.keys())}")
```

To run against a real board, swap this class for one that talks to your firmware (UART, USB CDC, network). The rest of the pipeline stays the same.

### Connect Phase

The first phase is a one-liner. The framework injects the `dut` plug and a `log` object by matching parameter names:

```python filename="phases/connect_dut.py"
def connect_dut(dut, log):
    log.info("Connecting to DUT...")
    dut.connect()
    log.info("DUT connected")
```

### Calibration Phase

The second phase does the real work. It retrieves the IMU log, validates it, fits per-axis polynomials, validates the fit quality, and saves the result. The framework automatically passes `dut`, `measurements`, and `log` based on the function signature:

```python filename="phases/thermal_calibration.py"
import numpy as np

from utils.calibrate_sensor import calibrate_sensor
from utils.compute_noise_density import compute_noise_density
from utils.compute_r2 import compute_r2
from utils.compute_residuals import compute_residuals
from utils.compute_temp_sensitivity import compute_temp_sensitivity


def thermal_calibration(dut, measurements, log):
    """Retrieve IMU data, validate it, compute polynomial thermal calibration, validate fit, save."""
    log.info("Fetching IMU log from DUT")
    data = dut.get_imu_data()

    axes = ("x", "y", "z")
    calibration_results = {}

    for sensor, data_key in (("acc", "acc_data"), ("gyro", "gyro_data")):
        sensor_data = data[data_key]
        temperature = np.asarray(sensor_data["temperature"], dtype=float)
        axes_data = {axis: np.asarray(sensor_data[f"{sensor}_{axis}"], dtype=float) for axis in axes}

        # --- Raw-data validation ---
        for axis, values in axes_data.items():
            noise = compute_noise_density(values)
            sens = compute_temp_sensitivity(values, temperature)
            setattr(measurements, f"{sensor}_noise_density_{axis}", noise)
            setattr(measurements, f"{sensor}_temp_sensitivity_ref_{axis}", sens["sensitivity_at_ref"])

        # --- Polynomial calibration ---
        fit = calibrate_sensor((temperature, *axes_data.values()))
        calibration_results[sensor] = {
            axis: fit["polynomial_coefficients"][f"{axis}_axis"].tolist() for axis in axes
        }

        # --- Multi-dimensional chart per axis: raw / fitted / residual vs temperature ---
        order = np.argsort(temperature)
        temp_sorted = temperature[order]

        for axis in axes:
            raw = axes_data[axis][order]
            fitted = fit["fitted_values"][f"{axis}_axis"][order]
            residuals_dict = compute_residuals(raw, fitted)

            md = getattr(measurements, f"{sensor}_calibration_{axis}")
            md.x_axis = temp_sorted.tolist()
            md.y_axis.raw = raw.tolist()
            md.y_axis.fitted = fitted.tolist()
            md.y_axis.residual = residuals_dict["residuals"].tolist()

            aggs = md.y_axis.residual.aggregations
            aggs.mean = residuals_dict["mean_residual"]
            aggs.std = residuals_dict["std_residual"]
            aggs.p2p = residuals_dict["p2p_residual"]

            setattr(measurements, f"{sensor}_r2_{axis}", compute_r2(raw, fitted))

    log.info("Saving calibration to DUT")
    dut.save_accelerometer_calibration(calibration_results["acc"])
    dut.save_gyroscope_calibration(calibration_results["gyro"])
```

The pattern is the same for every measurement: compute a value, write it to `measurements`. The framework matches it back to the YAML declaration and validates it.

### Numeric Measurements

Each scalar measurement is declared in `procedure.yaml` with a name, unit, and validators. Validators express the pass/fail limits:

```yaml filename="procedure.yaml"
measurements:
  - name: Acc Noise Density X
    key: acc_noise_density_x
    unit: m/s²/√Hz
    validators:
      - {operator: ">=", expected_value: 0.0}
      - {operator: "<=", expected_value: 0.003}
```

In Python you simply assign:

```python
setattr(measurements, "acc_noise_density_x", noise)
```

The dashboard automatically renders each measurement with its limits, value, and pass/fail outcome. As more units are tested, **3 sigma limits** can be computed automatically from production data on the procedure analytics page.

### Multi-Dimensional Measurements

For each sensor axis, we capture the **raw data**, the **fitted curve**, and the **residuals** as a single multi-dimensional measurement. This replaces the static PNG plots you would otherwise have to attach manually:

```yaml filename="procedure.yaml"
- name: Acc Calibration X
  key: acc_calibration_x
  title: Accelerometer X vs Temperature
  description: Raw accelerometer X readings vs internal temperature, with fitted 3rd-order polynomial and residuals.
  x_axis: {legend: Temperature, unit: "°C"}
  y_axis:
    - {legend: Raw, key: raw, unit: "m/s²"}
    - {legend: Fitted, key: fitted, unit: "m/s²"}
    - legend: Residual
      key: residual
      unit: "m/s²"
      aggregations:
        - type: mean
          validators:
            - {operator: ">=", expected_value: -0.01}
            - {operator: "<=", expected_value: 0.01}
        - type: std
          validators:
            - {operator: "<=", expected_value: 5.0}
        - type: p2p
          validators:
            - {operator: "<=", expected_value: 15.0}
```

Three things to notice:

1. **`x_axis` / `y_axis`** describe the chart. The dashboard renders an interactive plot — no need to generate or attach a PNG.
2. **`aggregations`** compute statistics over an axis (mean, standard deviation, peak-to-peak) and validate them. Here the residual mean must be near zero, the standard deviation small, and the peak-to-peak bounded.
3. **R²** (declared separately as a numeric measurement) catches a globally poor fit.

In Python, you set the data with intuitive attribute access:

```python
md = measurements.acc_calibration_x
md.x_axis = temperatures
md.y_axis.raw = raw_values
md.y_axis.fitted = fitted_values
md.y_axis.residual = residuals

aggs = md.y_axis.residual.aggregations
aggs.mean = residual_mean
aggs.std = residual_std
aggs.p2p = residual_p2p
```

### Polynomial Calibration

The calibration logic itself lives in `utils/calibrate_sensor.py` and uses a **3rd-order polynomial** fit per axis. The coefficients replace the need for large lookup tables and are programmed into the device for real-time correction:

```python filename="utils/calibrate_sensor.py"
import numpy as np


def calibrate_sensor(data, polynomial_order: int = 3):
    """Fit a polynomial model per axis. Returns coefficients and fitted values."""
    temp, *sensor_data = (np.asarray(arr, dtype=float) for arr in data)

    poly_coeffs = {}
    fitted_values = {}
    axis_list = ("x", "y", "z")

    for i, axis_data in enumerate(sensor_data):
        axis_name = f"{axis_list[i]}_axis"
        coeffs = np.polyfit(temp, axis_data, polynomial_order)
        poly_coeffs[axis_name] = coeffs
        fitted_values[axis_name] = np.polyval(coeffs, temp)

    return {
        "polynomial_coefficients": poly_coeffs,
        "fitted_values": fitted_values,
    }
```

### Calibration Validation

#### Residuals

After fitting, we compute the **residuals** (the difference between the model's prediction and the actual measurements) and validate three statistics:

* **mean** to detect systematic bias
* **std** to detect variability across the temperature range
* **peak-to-peak** to bound the worst-case error

```python filename="utils/compute_residuals.py"
import numpy as np


def compute_residuals(data, fit_model):
    residuals = np.asarray(data) - np.asarray(fit_model)
    return {
        "residuals": residuals,
        "mean_residual": float(np.mean(residuals)),
        "std_residual": float(np.std(residuals)),
        "p2p_residual": float(np.ptp(residuals)),
    }
```

#### Coefficient of Determination (R²)

Residual metrics focus on local accuracy. **R²** evaluates how well the model represents the sensor's behavior globally — close to 1 means the model explains most of the variance, near 0 means poor fit:

```python filename="utils/compute_r2.py"
import numpy as np


def compute_r2(data, fit_model):
    data = np.asarray(data, dtype=float)
    fit_model = np.asarray(fit_model, dtype=float)
    residuals = data - fit_model
    total_variation = float(np.sum((data - np.mean(data)) ** 2))
    if total_variation == 0.0:
        return 1.0
    return 1.0 - float(np.sum(residuals ** 2)) / total_variation
```

## Database & Analytics

Sensor calibration, while complex, can be implemented quickly using TofuPilot Framework. The procedure can be developed during the validation phase of a new product and deployed in production when mass manufacturing begins. The quality of a test relies not only on the performance of the processing algorithms but also heavily on the **choice of metrics** used to validate the measurements. These metrics improve as more units are tested, allowing for more precise 3 sigma limits to be defined.

This is where TofuPilot's database and analytics solution become essential.

### Automatic Upload

When a procedure declares a Dashboard ID in `procedure.yaml`, every run is uploaded automatically — including phases, measurements, multi-dimensional charts, and logs. No extra code, no output callback, no station server.

### Run Page

After a run completes, a dedicated page is **automatically created** in your secure TofuPilot workspace. This page displays the **test metadata** (serial number, run date, procedure reference), the list of **phases** and **measurements** with their **limits**, **units**, **duration** and **status**, plus all the multi-dimensional charts rendered interactively.

![Screenshot of the Run page showing a test report with detailed phases, key measurements, and metrics.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/templates/cb2524d62672b52653d8cb3bea41cb405961b2c7-1920x1110-sxDweTMiKJdQUjhb6ik4NnHBVS3XVv.png)

Run page displaying detailed test reports with phases and measurements.

### Procedure Analytics

Analyzing the performance of a procedure across recent runs is straightforward with the **procedure analytics page**. Key metrics such as the **run count**, **average test time**, **first pass yield**, and **CPK** are calculated automatically. You can **filter the data by date, revision, or batch** to narrow down your analysis. The page also provides a **detailed breakdown of performance by phase and measurement**, allowing you to select a specific phase or measurement to analyze its **duration**, **first pass yield**, or **CPK** individually. Additionally, you can view its **control chart** to track recent measurements, observe trends, and determine **3 sigma values**.

![Screenshot of Procedure page showing metrics, filters, phase details, and control charts.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/templates/89e23aa010ad30410d4ea58bafdf84e0b4e103f2-1920x1110-t4JzhqmRR0VVb09KWjxseNVP8dJtNl.png)

Procedure page with performance metrics, filters, and control charts.

### Unit Traceability

Finally, the traceability of each tested unit is easily accessible through its dedicated page. This page provides the complete **history of tests** performed for the unit, any related **sub-units**, and a link to the page dedicated to the **revision of the part**.

![Screenshot of the Unit page showing the test history and details for a specific unit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/templates/7a58aa49a348a7e51f7743bacaf3c6d4175fa91f-960x555-jmEWsRgK61thB3hnHs0buakOhNxi5f.png)

Unit page displaying the complete test history of a tested unit.

### In-Circuit Test with Flying Probe
URL: https://www.tofupilot.com/templates/in-circuit-test-with-flying-probe
Import Takaya flying probe ICT results (ATD, ATDX) into TofuPilot with the Python SDK: one run per board, phases by component, the tester's limits kept.


## Introduction

### ICT Overview

**In-circuit test (ICT)** checks a populated PCB component by component after reflow: every reachable net for shorts and opens, every resistor, capacitor and inductor against its BOM tolerance, every diode junction and IC protection diode for presence and polarity. It runs unpowered, before functional test, so an assembly defect is caught while the board is still on the SMT line and rework costs minutes, not a field return.

**Flying probe testers** (Takaya APT series, SPEA 4060, Seica Pilot, Acculogic Scorpion) do this with four to eight moving probes and no fixture, which makes them the ICT of choice for NPI, prototypes and any product under a few thousand boards a year. They are also closed systems: the test program lives in the tester, the tester runs it, and what comes out is a **result file** per panel in the vendor's own format. Nobody writes a Python script that drives a flying probe; the integration point is that file.

This template is that integration. A Python script and the TofuPilot SDK turn Takaya **ATD** and **ATDX** result files into TofuPilot runs, one per board, with every tester step as a measurement, the tester's own limits as validators, and the tester's verdict as the outcome. A drop-folder watcher does it continuously on the tester's export directory.

### Test Purpose

Each imported run carries the complete ICT record of one board:

- **Header**: serial number, tester program, tester ID, test date, panel group and side, test duration where the file has it
- **Shorts and opens**: every net pair the program probed, measured resistance against the open limit
- **Passives**: measured value of every R, C, L against the nominal and the tolerance the tester applied
- **Semiconductors and IC pins**: junction voltages, protection-diode signatures, one measurement per pin
- **Verdicts**: PASS, FAIL (including the tester's UPPER, LOWER and misalignment codes) and skipped steps, as the tester judged them
- **Counts**: steps, failed, skipped and steps whose limits were reproducible, as run metadata

![One ATDX panel as it lands in TofuPilot: 3104 tester steps sorted into eight phases with the tester verdict kept per measurement, 40 failures across capacitors, inductors, resistors and IC pins.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-f2c875c7-e594-4e44-a15f-eeccc6c5b43b-E9VKTExtJMAEZ4Zt6rOSxfgc1FYZAh.png)

The shipped ATDX sample, a two-board panel, after import: 3104 steps become 8 phases, 40 failures land on the exact capacitors, inductors and IC pins the tester flagged, and the 278 steps the tester skipped stay visible as unset instead of disappearing.

What the SDK gives you over the tester's own log: a per-board record you can search by serial, a per-component history across boards, the drift of a passive inside its limits before it fails, and a failure Pareto per reference designator across a batch. None of this exists in the tester's statistics screen, which aggregates by step number.

Beyond the ICT content, the template shows four SDK mechanics: `runs.create` with **nested phases, measurements and validators** built from a file, **per-validator outcomes** stored as the tester judged them, **run metadata** for the tester and file provenance, and **schema validation of the payload before upload** through the SDK's own models, so a parser bug is caught in `--dry-run`, not in production.

## Equipment & Setup

To feed flying-probe ICT results into TofuPilot, the following are required:

- **A flying probe tester** with results exported to a folder: Takaya APT with ATD or ATDX output for this template
- **The tester's export directory** reachable from a PC: a share on the tester PC, or the tester writing to a network drive
- **A Python environment** with the TofuPilot SDK (`pip install tofupilot`)
- **A TofuPilot procedure** for the ICT station and an API key
- **The import script** from this template, run per file or as a watcher

![Two flying probes contacting a blue circuit board on a test bed](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/in-circuit-test-with-flying-probe-2HiJE8qs5Ktf9e0IKrZN6vJQWrX6xN.webp)

Movable probes contact test points on a supported board. The illustration shows the probing hardware; the machine controller exports the reports imported by this procedure.

### Hardware Components

#### Flying Probe Tester

Production-grade options and what they export:

- **Takaya APT-1400F / APT-1600FD / APT-2400F**: 4 to 6 probes, dual-sided on the FD models. Result files as **ATD** (tab-delimited, one file per panel, one group per board) on older software and **ATDX** (semicolon-delimited, `Main header` / `Group header` blocks) on current software. This template parses both.
- **SPEA 4060 S2 / 4080**: 4 to 8 probes, results as XML or CSV reports from the Leonardo software.
- **Seica Pilot V8 / VX**: results from the VIVA software, CSV and XML.
- **Acculogic Scorpion FLS980**: up to 24 probe modules, CSV reports.

Every vendor format carries the same information per step: a reference designator or net pair, a nominal, a tolerance or threshold, a measured value, a unit, a verdict. The template's `Board` and `Step` model is vendor-neutral; the Takaya parsers fill it, another vendor needs another parser and nothing else.

![Flying probe close-up: five tilted needles from the probe heads above landing on gold test pads of a populated PCB.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-22f1ba1c-a19f-4706-b115-431951078d23-3MRjN6two8EdOr0SGPopZpSY46iH39.jpg)

Tilted needles from the heads above, each on its own test pad: one landing is one line in the result file, with the two nets it bridged.

#### The Result Files

An ATDX file, the current Takaya format, per panel block:

```text filename="samples/WO240725-0064__20240605_173206.atdx"
#############"Main header"############
Tester ID:  APT-1400F FPT04
Model: 289-0254-3.0
Serial number : WO240725-0064
Test Date: 05/06/2024 17:32:06
#############"Group header"#########
Group No.: 1 Side: A FAIL
Order;Aux;M.Aux;Parts;Value;Unit;Comment;Loc.;EL;Reference;Unit;Func;+%;Unit;-%;Unit;Test.Base(Test1);Unit;...;Judge;H-pin;Netname-H-pin;L-pin;Netname-L-pin;...
1;;;POWER_SHORTS;P-AUTO;;153-9;SHRT;R;100.00;O;OP;----------;;----------;;1245.07;O;;;PASS;3;PWR_VOUT;1;GND;...
531;;;C445;1.0;PF;031_2339;CAP;C;66.12;pF;EQ;20.0;;20.0;;81.18;pF;81.97;pF;UPPER;1;GND;14;$$$4965;...
3098;;;U6;1-G;;ICLTC2803IDHC;DNET;D;1.033;V;EQ;30.0;;30.0;;1.069;V;;;PASS;709;RS232_1_RX;474;GND;...
```

The classic ATD format spreads the same fields over 58 tab-separated columns, with one `* GROUP No.n *` block per board on the panel and the serial number inside each group. Both carry the tester's `Func` (how the step was judged: `EQ` within a tolerance, `OP` open above a threshold, `SH` short below one, `GE`, `LE`), the reference value, the tolerances, the measured value and the verdict.

### Custom Firmware

None. ICT runs on an unpowered board and the DUT firmware plays no part. The tester program is written in the vendor's software from CAD data (ODB++, IPC-2581 or Gerber plus an IPC-D-356 netlist); this template starts where that program ends, at the result file.

## Test Procedure

### Overview

For each result file the script runs:

1. Pick the parser from the extension and read every board block into a `Board` with its `Step` list.
2. Classify each step into a phase: shorts and opens, resistors, capacitors, inductors, diodes and transistors, IC pins, connectors.
3. Turn each step into a measurement: name, measured value in the tester's unit, the tester's verdict as outcome, validators derived from `Func` and the tolerances.
4. Keep validators only where they reproduce the tester's verdict; otherwise record the limits as text in the measurement note.
5. Build the `runs.create` payload with run outcome, timestamps and metadata, validate it against the SDK models.
6. Upload, one run per board, or write the payloads to JSON in `--dry-run`.

### Why the TofuPilot Python SDK?

The SDK is the direct route when the test is not yours to run. A flying probe tester, an AOI machine or an X-ray system executes its own program; what you own is the output file. `client.runs.create()` takes the whole record of one run in one call: phases, measurements, validators, metadata, unit and part identity. There is no station to install and no framework to adopt on the tester PC, only a script that reads a file and makes one HTTP call per board.

* One call per run, nested phases and measurements in the request
* Pydantic models for every field, so a payload can be validated offline before it is sent
* Runs land in the same procedure, unit and part model as runs from the TofuPilot Framework or OpenHTF stations
* Works from any machine with Python and network access to the tester's export folder

### Project Structure

```tree
import_takaya.py
watch_folder.py
takaya/
  model.py
  atdx.py
  atd.py
  units.py
  mapping.py
samples/
  WO240725-0064__20240605_173206.atdx
  PB2939800_E_BOT-20230420130239.ATD
  V682589_K_TOP-20230504130808.ATD
tests/
  test_takaya.py
README.md
pyproject.toml
```

The three sample files are real Takaya output, published under the MIT license by Virinco with their WATS converter. Run the import with `python import_takaya.py samples/*.ATD --procedure-id <id>` after `export TOFUPILOT_API_KEY=...`, or add `--dry-run --json out/` to see the payloads without uploading. You can find the full source on [GitHub](https://github.com/tofupilot/template-python-flying-probe-import).

### The Import Script

The command-line entry point, verbatim from the template:

```python filename="import_takaya.py"
"""Import Takaya flying-probe result files into TofuPilot.

    export TOFUPILOT_API_KEY=...
    python import_takaya.py samples/WO240725-0064__20240605_173206.atdx --procedure-id <id>
    python import_takaya.py samples/*.ATD --procedure-id <id> --part-number PB2939800 --revision E
    python import_takaya.py samples/*.atdx --procedure-id <id> --dry-run --json out/

One run is created per board (per tester group). A panel ATD with four groups
becomes four runs. --dry-run parses, builds the payloads, validates them against
the SDK models and writes them as JSON without calling the API.
"""
from __future__ import annotations

import argparse
import json
import os
import sys
from pathlib import Path

from tofupilot.v2 import TofuPilot, models

import takaya


def _summary(payload: dict) -> str:
    phases = payload["phases"]
    n = sum(len(p["measurements"]) for p in phases)
    failed = sum(1 for p in phases for m in p["measurements"] if m["outcome"] == "FAIL")
    return (f"{payload['serial_number']:<16} {payload['outcome']:<4} {n:>5} measurements "
            f"{failed:>3} failed  {len(phases)} phases")


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("files", nargs="+", type=Path, help=".atdx or .ATD result files")
    ap.add_argument("--procedure-id", required=True, help="TofuPilot procedure the runs attach to")
    ap.add_argument("--part-number", help="override the tester program name as part number")
    ap.add_argument("--revision", help="revision number to stamp on the unit")
    ap.add_argument("--serial-suffix-group", action="store_true",
                    help="append -G<group> to the serial (panels where the tester prints the work order as serial)")
    ap.add_argument("--dry-run", action="store_true", help="build and validate payloads, do not upload")
    ap.add_argument("--json", type=Path, help="directory to write one payload JSON per run")
    args = ap.parse_args()

    payloads: list[dict] = []
    for file in args.files:
        boards = takaya.parse(file)
        if not boards:
            print(f"{file}: no board found", file=sys.stderr)
            continue
        for board in boards:
            if args.serial_suffix_group:
                board.serial_number = f"{board.serial_number}-G{board.group_no}"
            payloads.append(takaya.to_run(board, args.procedure_id, args.part_number, args.revision))

    for p in payloads:
        models.RunCreateRequest.model_validate(p)  # same schema the API enforces
        print(_summary(p))

    if args.json:
        args.json.mkdir(parents=True, exist_ok=True)
        for i, p in enumerate(payloads, 1):
            out = args.json / f"{i:03d}-{p['serial_number']}-g{p['metadata']['tester_group']}.json"
            out.write_text(json.dumps(p, indent=2, default=str))
        print(f"wrote {len(payloads)} payloads to {args.json}/")

    if args.dry_run:
        return 0

    api_key = os.environ.get("TOFUPILOT_API_KEY")
    if not api_key:
        print("TOFUPILOT_API_KEY is not set", file=sys.stderr)
        return 2
    with TofuPilot(api_key=api_key) as client:
        for p in payloads:
            run = client.runs.create(**p)
            print(f"  -> run {run.id}")
    return 0


if __name__ == "__main__":
    sys.exit(main())
```

SDK features to notice:

1. **One payload, one call**: `client.runs.create(**payload)` takes the run header, the phases, every measurement with its validators and the metadata in a single request. A 1552-step board is one HTTP call.
2. **Offline schema validation**: `models.RunCreateRequest.model_validate(p)` runs the same Pydantic model the SDK serialises, so a wrong unit type or a missing timestamp fails in `--dry-run` with a field path, not as a 400 from the API.
3. **Tester verdict as outcome**: measurement, phase and run outcomes come from the tester's `Judge`, the group result and the panel result. The importer never re-judges.
4. **Per-validator outcomes**: each validator carries its own `outcome`, so the run page shows which limit a failing capacitor crossed, the lower or the upper.
5. **Provenance in metadata**: tester ID, program, group, side, source file and format, plus the step counts, on every run. A bad panel traces back to the file and the machine.

### Parsing the Result File

Both parsers fill the same neutral model; the ATDX one, in full, shows the block structure and how the repeated `Unit` column is addressed by occurrence:

```python filename="takaya/atdx.py"
def parse(path: str | Path) -> list[Board]:
    path = Path(path)
    text = path.read_text(encoding="utf-8-sig", errors="replace")
    boards: list[Board] = []
    header: dict[str, str] = {}
    cols: dict[tuple[str, int], int] = {}
    board: Board | None = None

    for line in text.splitlines():
        if MAIN_HEADER in line:
            header, cols, board = {}, {}, None
            continue
        if ":" in line and not cols and board is None and GROUP_HEADER not in line:
            key, _, val = line.partition(":")
            header[key.strip().lower()] = val.strip()
        m = _GROUP_RE.search(line)
        if m:
            board = Board(
                serial_number=header.get("serial number", ""),
                program=header.get("model", ""),
                tester_id=header.get("tester id", ""),
                tested_at=_date(header.get("test date", "")),
                group_no=int(m.group(1)),
                side=m.group(2),
                passed=m.group(3).upper() == "PASS",
                duration_s=None,
                source_file=path.name,
                format="atdx",
            )
            boards.append(board)
            continue
        if line.startswith("Order;"):
            seen: dict[str, int] = {}
            for i, name in enumerate(line.split(";")):
                name = name.strip()
                cols[(name, seen.get(name, 0))] = i
                seen[name] = seen.get(name, 0) + 1
            continue
        if board is None or not cols or not line[:1].isdigit():
            continue
        f = line.split(";")
        board.steps.append(Step(
            order=int(_row(cols, f, "Order")),
            parts=_row(cols, f, "Parts"),
            value=_row(cols, f, "Value"),
            location=_row(cols, f, "Loc."),
            element=_row(cols, f, "EL"),
            func=_row(cols, f, "Func").upper(),
            reference=number(_row(cols, f, "Reference")),
            tol_plus_pct=percent(_row(cols, f, "+%")),
            tol_minus_pct=percent(_row(cols, f, "-%")),
            measured=number(_row(cols, f, "Test.Base(Test1)")),
            unit=unit_label(_row(cols, f, "Unit", 4) or _row(cols, f, "Unit", 1)),
            judge=_row(cols, f, "Judge"),
            net_high=_row(cols, f, "Netname-H-pin"),
            net_low=_row(cols, f, "Netname-L-pin"),
            comment=_row(cols, f, "Comment"),
        ))
    return boards
```

Two details cost time on real files. The ATDX sample shipped with the template is one two-board panel whose serial number is the work order, printed identically for both groups; `--serial-suffix-group` turns that into `WO240725-0064-G1` and `-G2`. The ATD format prints each value twice, once in base units (`4700`) and once in the display unit (`4.70` with `KO`); the importer takes the display pair, so a 4.7 kΩ resistor is stored as 4.7 kΩ and not as 4700 Ω next to a limit in kΩ.

### From Tester Step to Measurement

Classification and limits, from the mapping module:

```python filename="takaya/mapping.py"
def classify(step: Step) -> str:
    parts = step.parts.upper()
    if (parts.startswith(_STRUCTURAL_PARTS) or step.location.upper() in _STRUCTURAL_LOC
            or step.func == "SH" or step.value.upper() == "P-AUTO"):
        return "Shorts and Opens"
    prefix = parts.rstrip("0123456789*")  # C34* is still a capacitor
    if prefix in ("R", "RN", "RP", "RV"):
        return "Resistors"
    if prefix in ("C", "CN"):
        return "Capacitors"
    if prefix in ("L", "E", "FI", "FB", "FL", "T"):  # E: ferrite beads and chokes on some BOMs
        return "Inductors"
    if prefix in ("D", "Q", "ZD", "LED"):
        return "Diodes and Transistors"
    if prefix in ("U", "IC", "X", "Y"):
        return "IC Pins"
    if prefix in ("J", "P", "I", "CN", "CONN"):
        return "Connectors"
    return "Other"


def validators(step: Step) -> list[dict]:
    ref = step.reference
    if ref is None or step.func == "SKIP":
        return []
    if step.func == "EQ":
        out = []
        if step.tol_minus_pct is not None:
            out.append({"operator": ">=", "expected_value": round(ref * (1 - step.tol_minus_pct / 100), 6)})
        if step.tol_plus_pct is not None:
            out.append({"operator": "<=", "expected_value": round(ref * (1 + step.tol_plus_pct / 100), 6)})
        return out
    if step.func in ("OP", "GE"):
        return [{"operator": ">=", "expected_value": ref}]
    if step.func in ("SH", "LE"):
        return [{"operator": "<=", "expected_value": ref}]
    return []


def to_measurement(step: Step, phase: str) -> dict:
    out = outcome(step)
    m: dict = {"name": measurement_name(step, phase), "outcome": out}
    if step.measured is not None:
        m["measured_value"] = step.measured
        if step.unit:
            m["units"] = step.unit
    v = validators(step)
    derived = _derived_verdict(step, v)
    applied = v and out in ("PASS", "FAIL") and derived == out
    doc = f"step {step.order}, {step.element or '?'} {step.func}, tester judge {step.judge}"
    if applied:
        for x in v:  # per-validator verdict, so the UI shows which limit the value crossed
            ok = step.measured >= x["expected_value"] if x["operator"] == ">=" else step.measured <= x["expected_value"]
            x["outcome"] = "PASS" if ok else "FAIL"
        m["validators"] = v
    elif step.reference is not None:
        doc += f", limits {limits_text(step)} kept as text: they do not reproduce the tester verdict"
    if step.comment:
        doc += f", {step.comment}"
    m["docstring"] = doc
    m["_limits_applied"] = bool(applied)
    return m
```

The `applied` check is the part worth copying into any importer. On the three sample files, 364 of 4013 passing steps have a printed reference and tolerance that do **not** explain the verdict: guarded capacitor measurements, open-pin diode reads clamped at 2.65 V, connector pin checks judged on a second measurement the file does not carry. Writing those limits as validators would show a red limit under a green tester verdict. The importer keeps the tester's verdict, attaches validators only where they reproduce it, and records the printed limits in the note otherwise; `steps_with_limits` in the run metadata says how many made it.

![Histogram of passing resistor, capacitor and inductor steps by their deviation from nominal as a fraction of the tester tolerance, most within half of the limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-7f71cd2b-e368-4e51-8220-7ae3e096678e-idiP8NKkMeW2rstWlPRGl0dbmXwEvX.png)

The 767 passing passives of the ATDX panel with reproducible limits, by how much of their tolerance they use. Resistors sit within a fifth of their band; capacitors spread to three quarters of theirs, which is where a drifting reel shows up first.

### Shorts and Opens

The largest phase on any board: every net pair the program probed, judged `OP` against a threshold. The importer names these by the two nets, `POWER_SHORTS PWR_VOUT-GND`, so a short between two rails reads as one on the run page.

![Log-scale histogram of the measured resistance on 756 power and board shorts steps, every one above the 100 ohm open limit, most at the 55 kilohm range ceiling.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-b15db5b7-8ce0-49ce-89b2-028963fc1bd1-QjAC5mxCvB2YtnLEEJRgwg7OLXJ6QM.png)

The 756 shorts steps of the sample panel: every pair above the 100 Ω open threshold, 429 of them at the 55 kΩ ceiling of the tester's range. The pairs between 100 Ω and 1 kΩ are the ones to watch on the next batch; a solder bridge shows up as a single bar at the far left.

### Continuous Import

On a line, nobody runs the script by hand. The watcher polls the tester's export folder, imports each new file once its size has stopped changing, then moves it to `imported/` or to `failed/` with the error next to it, so a restart never double-imports:

```python filename="watch_folder.py"
def import_file(client: TofuPilot, path: Path, procedure_id: str, part_number: str | None) -> int:
    n = 0
    for board in takaya.parse(path):
        payload = takaya.to_run(board, procedure_id, part_number)
        run = client.runs.create(**payload)
        print(f"{path.name}: {board.serial_number} {payload['outcome']} -> run {run.id}")
        n += 1
    return n


    with TofuPilot(api_key=api_key) as client:
        print(f"watching {args.folder} every {args.interval:.0f} s")
        while True:
            for path in sorted(p for p in args.folder.iterdir() if p.suffix.lower() in EXTENSIONS):
                if not settled(path):
                    continue
                try:
                    import_file(client, path, args.procedure_id, args.part_number)
                    shutil.move(str(path), imported / path.name)
                except Exception as exc:  # keep watching, keep the evidence
                    shutil.move(str(path), failed / path.name)
                    (failed / f"{path.name}.error.txt").write_text(repr(exc))
                    print(f"{path.name}: FAILED {exc!r}", file=sys.stderr)
            time.sleep(args.interval)
```

A 5 s poll is enough for a tester that finishes a panel every few minutes; no service, no inotify, and the `failed/` folder is the whole alerting story.

### Sample Files and Tests

`tests/test_takaya.py` runs the parsers and the mapping on the three sample files: board and header fields, units and tolerances, one board per ATD group, unique measurement names per phase, the payloads against the SDK schema, and a check that no attached validator ever contradicts the tester's verdict. The ATD panel's first board was uploaded through the API exactly as the dry-run wrote it, 76 measurements in 7 phases, which is the test the schema cannot do.

For another tester, write a parser that fills `Board` and `Step` from its report format; `classify`, `validators`, `to_measurement` and `to_run` do not know they came from a Takaya.


### RC Link RSSI vs Range Verification
URL: https://www.tofupilot.com/templates/rc-link-rssi-vs-range-verification
Verify drone RC link RSSI, LQ, and failsafe threshold against a calibrated attenuator sweep at end-of-line to catch antenna and LNA defects.


## Introduction

### RC Link Overview

Every drone carries an **RC link receiver (RX)** that pairs with a hand-held transmitter (TX) and relays stick inputs to the flight controller at 50-1000 Hz. The link runs on **2.4 GHz** or **sub-GHz (868 / 915 MHz)** ISM bands, and modulates with **LoRa**, **FSK**, or proprietary DSSS schemes depending on protocol. Common choices include **ExpressLRS** (open source, [-112 dBm sensitivity at 150 Hz LoRa](https://www.expresslrs.org/info/signal-health/)), **TBS Crossfire**, **FrSky R9M**, **Spektrum DSMX**, and **DJI O3/O4**.

![Close-up of a small drone RC receiver board with its shielded LNA stage, U.FL connector and dipole antenna, about 15 mm long.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-1175e4cf-f4fe-438f-bb84-a71a67f4631d-nfK6SpG8hBl0fcP8LaPTOKuO0zJprn.jpg)

A nano receiver: the RF shield covers the LNA stage, the U.FL connector feeds the dipole, the whole board is about 15 mm long.

A weak or marginal RC link is a leading cause of drone loss. Failure modes include premature **failsafe triggering** (drone drops or initiates RTH mid-mission), inability to recover when link drops at altitude, and "fly-aways" where the receiver holds last stick values instead of cutting throttle. Per the [Betaflight failsafe guide](https://betaflight.com/docs/wiki/guides/current/Failsafe), the flight controller holds the last good channel values for 300 ms, then Stage 1 applies the configured channel fallback settings for the guard time (`failsafe_delay`, 1.5 s default in Betaflight 4.5) before Stage 2 engages, so a sub-spec receiver can silently misbehave through that window.

### Test Purpose

Per-unit RC link verification measures the **link's sensitivity, failsafe behavior, and timing response** against a controlled signal-power sweep. Output fingerprint:

- **RSSI vs attenuation curve**: the receiver's reported RSSI (dBm) as the input power drops
- **LQ (Link Quality) curve**: packet success rate as power drops
- **Failsafe threshold**: the dBm at which the RX engages failsafe
- **Sensitivity floor**: the dBm at which the link breaks entirely
- **Telemetry latency**: round-trip time of telemetry packets

The sensitivity limit in this template (≤ -107 dBm at 250 Hz) sits just inside the [official ExpressLRS sensitivity table](https://www.expresslrs.org/info/signal-health/) (-108 dBm at 250 Hz), so a healthy receiver passes with margin and a degraded RF chain does not.

![Two-panel plot of the attenuation sweep: receiver-reported RSSI tracking the calibrated input power from -28 to -117 dBm within 1.5 dB, and link quality flat at 100% until it collapses over the last 10 dB above the -107.5 dBm floor, with the -104.5 dBm failsafe threshold marked.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-4a529e98-7d4f-4490-8588-4c2a49749bd1-9Li1mqqzmqvrgOwcururNZkpGu0fPx.png)

The sweep as the mock link produces it: reported RSSI stays within 1.5 dB of the calibrated input power, LQ holds 100% until the last 10 dB above the floor, and failsafe engages 4 dB before the link dies.

Per-unit testing catches defects invisible at incoming inspection: **cracked antenna feedline** (can cost tens of dB of sensitivity), **U.FL connector misorientation** (U.FL is [rated for 30 mating cycles](https://www.hirose.com/product/series/U.FL)), **dead LNA from VTX feedback**, **antenna fouled by carbon-fiber frame**, **wrong firmware version**, and **solder bridge on the matching network**.

Beyond the RF content, this template demonstrates the framework's **operator input components**: the line operator picks the packet rate, TX power, and starting attenuation in the UI, the values are **bound to measurements** for traceability, and the same run executes unattended by pre-baking the answers with `--ui-values`.

## Equipment & Setup

To implement RC link RSSI verification on a production line, the following are required:

- **A shielded RF environment** (Faraday tent or anechoic chamber) for the RX
- **A programmable step attenuator** in the RF path
- **A reference TX** running production firmware
- **The Device Under Test (DUT)**: a drone (or RX module) with antenna installed
- **A spectrum analyzer** for diagnostic spot-checks
- **A TofuPilot Framework procedure** to step the attenuator, capture RX telemetry, and validate metrics
- The **TofuPilot Dashboard** to log results and trend supplier quality

### Hardware Components

#### Shielded Environment

A compact fully anechoic OTA chamber (e.g. ETS-Lindgren) is the high-end choice for automated pattern and frequency sweeps. For production volume at lower capex, a **Faraday tent** (Select Fabricators or JRE Test) is the practical compromise; specify at least **70 dB isolation** in the 2.4 GHz and 900 MHz bands. The TX sits outside, the RX inside, and the RF path runs through a bulkhead-mounted attenuator.

![RC receiver with two coax connections and an electrical harness inside an RF shield box](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/rc-link-rssi-vs-range-verification-physical-2026-wmhneexhqjywrF0fPkyZuxSZHNue9r.webp)

The receiver's coax leads connect through the enclosure wall separately from its electrical harness. The box is open for setup and closes during the RF sweep.

#### Programmable Attenuator

A **[JFW Industries 50P-2014](https://www.jfwindustries.com/product/50p-2014-solid-state-programmable-attenuator-with-usb-control/)** solid-state programmable step attenuator: **0-95 dB in 1 dB steps**, 200-6000 MHz, USB or TTL controlled (the 50P-2013 is the 0-63 dB variant). A single attenuator covers both the 2.4 GHz and 900 MHz drone bands. Insertion accuracy ±0.5 dB up to 90 dB.

#### Spectrum Analyzer

For diagnostic spot-checks on RF chain defects, a **Rigol DSA815-TG** (9 kHz-1.5 GHz, tracking generator, preamp) covers 900 MHz at about $1,500. A **Tektronix RSA306B** USB real-time analyzer (9 kHz-6.2 GHz, about $3,500) covers 2.4 GHz. For go/no-go production-floor scans, an **RF Explorer 6G Combo** at under $500 suffices.

#### Reference TX

An ExpressLRS production line uses a **golden TX module** (Radiomaster TX16S running stock ELRS firmware) configured to the production-target packet rate. The TX is RF-shielded inside its own enclosure, with its antenna replaced by a low-loss U.FL → SMA pigtail routing into the attenuator. **TX output power is locked at 25 mW (14 dBm)** so the attenuation sweep maps cleanly to receiver-input dBm.

### Custom Firmware

The RX runs **production firmware** with telemetry enabled. The test station captures telemetry over the flight controller's USB or a side-channel (the [ExpressLRS Backpack](https://www.expresslrs.org/software/backpack-telemetry/) forwards CRSF frames over ESP-NOW) and parses RSSI, LQ, packet rate, and round-trip latency. The ELRS Lua script sets the production packet rate and power on the TX.

## Test Procedure

### Overview

After mounting the DUT inside the Faraday tent and pairing TX/RX, the procedure runs:

1. The operator confirms the link configuration (packet rate, TX power, starting attenuation) in the UI form.
2. The configuration is applied to the telemetry link and the attenuator.
3. Sweep attenuation across the link budget in 1 dB steps, logging RSSI and LQ per step, and find the sensitivity floor.
4. Push attenuation past the floor to find the failsafe threshold and the failsafe margin, then measure telemetry round-trip latency.
5. Validate every metric against its declared limits.
6. Stream results to TofuPilot for traceability and analytics.

Total cycle ~90 s per DUT.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
ui.json
phases/
  apply_config.py
  rssi_sweep.py
  failsafe_latency.py
plugs/
  attenuator.py
  telemetry.py
utils/
  link_budget.py
README.md
pyproject.toml
```

The mock plugs model an ExpressLRS-grade link so the procedure runs green without hardware; swap them for JFW-USB and pyserial CRSF implementations on a real bench. Run it with `tofupilot run . --no-tui --no-kiosk --json --ui-values ui.json`, or drop `--ui-values` to answer the operator prompts interactively. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-rc-link-rssi).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: RC Link RSSI vs Range Verification
version: 0.1.0
description: Sweeps a calibrated attenuator across the RC link budget and verifies RSSI accuracy, LQ, failsafe threshold, and telemetry latency.

unit:
  auto_identify: true
  serial_number:
    description: "Receiver serial from the label under the antenna"
    placeholder: "RX-000000"
    pattern: "^RX-\\d{6}$"
    default_value: "RX-004217"
  part_number:
    default_value: "ELRS-RX-24G"

plugs:
  - name: Step Attenuator
    description: Mock JFW 50P-2014 programmable step attenuator, 0-95 dB in 1 dB steps.
    python: plugs.attenuator:StepAttenuator
    key: attenuator
    config:
      insertion_loss_db: 1.5
  - name: Link Telemetry
    description: Mock CRSF telemetry stream from the flight controller.
    python: plugs.telemetry:LinkTelemetry
    key: telemetry

main:
  - name: Configure Link
    key: configure_link
    ui:
      components:
        - key: packet_rate
          type: select
          label: "Packet Rate"
          bind: measurements.packet_rate_hz
          options:
            - label: "50 Hz"
              value: "50"
            - label: "150 Hz"
              value: "150"
            - label: "250 Hz"
              value: "250"
            - label: "500 Hz"
              value: "500"
          default_value: "250"
        - key: tx_power_dbm
          type: number_input
          label: "TX Power (dBm)"
          default_value: 14
          bind: measurements.tx_power_dbm
        - key: start_attenuation_db
          type: slider
          label: "Start Attenuation (dB)"
          min: 0
          max: 40
          default_value: 0
          bind: measurements.start_attenuation_db
    measurements:
      - name: Packet Rate
        key: packet_rate_hz
        unit: Hz
        validators:
          - operator: in
            expected_value: ["50", "150", "250", "500"]
      - name: TX Power
        key: tx_power_dbm
        unit: dBm
      - name: Start Attenuation
        key: start_attenuation_db
        unit: dB

  - name: Apply Link Config
    key: apply_config
    python: phases.apply_config
    depends_on: [configure_link]
    measurements:
      - name: RF Channel
        key: rf_channel
        description: Active regulatory channel index; 0 and 13 are reserved on this SKU.
        validators:
          - operator: not in
            expected_value: [0, 13]

  - name: RSSI Sweep
    key: rssi_sweep
    python: phases.rssi_sweep
    depends_on: [apply_config]
    measurements:
      - name: RSSI And LQ vs Attenuation
        key: rssi_lq_sweep
        title: RSSI and LQ vs Attenuation
        description: Receiver-reported RSSI and link quality against the calibrated attenuation sweep.
        x_axis:
          legend: Attenuation
          unit: dB
        y_axis:
          - legend: RSSI
            key: rssi
            unit: dBm
            aggregations:
              - type: rssi_error_max
                unit: dB
                validators:
                  - operator: "<="
                    expected_value: 3.0
          - legend: Link Quality
            key: lq
            unit: "%"
      - name: Sensitivity Floor
        key: sensitivity_floor_dbm
        unit: dBm
        validators:
          - operator: "<="
            expected_value: -107.0

  - name: Failsafe And Latency
    key: failsafe_latency
    python: phases.failsafe_latency
    depends_on: [rssi_sweep]
    measurements:
      - name: Failsafe Threshold
        key: failsafe_threshold_dbm
        unit: dBm
        validators:
          - operator: ">="
            expected_value: -110.0
      - name: Failsafe Margin
        key: failsafe_margin_db
        unit: dB
        description: Gap between failsafe engagement and total link loss; must engage before the link breaks.
        validators:
          - operator: ">="
            expected_value: 3.0
      - name: Telemetry Round Trip
        key: telemetry_rtt_ms
        unit: ms
        validators:
          - operator: "<="
            expected_value: 50.0
```

Framework features to notice:

1. **Operator input, bound to measurements**: the `Configure Link` phase is pure UI, no Python. The `select`, `number_input`, and `slider` components each carry a `bind: measurements.*`, so the operator's choices land as validated measurements on the run (the packet rate must be one of the four production rates via an `in` validator). Select options use the `{label, value}` form. Because select values are strings, the packet rate is recorded as a string measurement (with `unit: Hz` for display) and the consuming phase casts it with `int()`.
2. **Unit field validation**: the serial number field declares a `pattern`, `placeholder`, and `description`; operator input that does not match `^RX-\d{6}$` is rejected before the run starts.
3. **`not in` validator**: the RF channel measurement rejects the two reserved channel indices with a one-line validator.
4. **Plug `config`**: the attenuator's calibrated insertion loss lives in YAML, not in code, and arrives as an `__init__` keyword argument.
5. **Custom aggregation**: `rssi_error_max` is a free-named aggregation on the RSSI axis; Python computes it, the YAML validator enforces the ±3 dB reporting-accuracy budget.

### Reading Operator Input

The `ui` object injected into Python phases is write-only (it updates display components). Operator input travels through `bind:` into measurements, and the next phase reads them through **previous-results injection**: naming a parameter after a completed phase's key injects a read-only result object.

```python filename="phases/apply_config.py"
def apply_config(measurements, configure_link, telemetry, attenuator, log):
    packet_rate = int(configure_link.packet_rate_hz)
    tx_power = float(configure_link.tx_power_dbm)
    start_attenuation = int(configure_link.start_attenuation_db)

    telemetry.configure(packet_rate)
    attenuator.set_db(start_attenuation)

    log.info(f"Link at {packet_rate} Hz, TX {tx_power} dBm, start att {start_attenuation} dB")
    measurements.rf_channel = telemetry.rf_channel()
```

For unattended runs the answers come from a JSON file keyed by phase and component:

```json filename="ui.json"
{
  "configure_link": {
    "packet_rate": "250",
    "tx_power_dbm": 14,
    "start_attenuation_db": 0
  }
}
```

`tofupilot run . --ui-values ui.json` injects these as if the operator had typed them; the binds still fire, so the measurements and their validators behave identically in attended and unattended runs.

### RSSI Sweep

The sweep starts at 40 dB attenuation (about -28 dBm at the receiver after the 1.5 dB insertion loss, the top of the linear reporting range) and steps 1 dB at a time down past the floor. At each step the receiver reports RSSI and LQ; while the link is alive, the absolute error between reported RSSI and the calibrated input power feeds the accuracy aggregation.

![Time series of the sweep: attenuation staircase from 40 to 130 dB in 1 dB steps, RSSI falling in step with it, and link quality holding 100% then dropping to 0% over the last 10 dB.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-5d14fd7c-17d8-4d29-b81d-36ff00d07413-c1BHxDe5GM2wTBE7fG9IuMqgqAybCn.png)

The sweep as the station sees it: 91 attenuation steps, RSSI following the staircase down, LQ collapsing only in the last 10 dB before the floor.

```python filename="phases/rssi_sweep.py"
from utils.link_budget import input_power_dbm

TX_POWER_DBM = 14.0  # golden TX locked at 25 mW


def rssi_sweep(measurements, attenuator, telemetry, log):
    attenuations = []
    rssis = []
    lqs = []
    rssi_errors = []
    floor = None

    for att in range(40, 131, 1):
        attenuator.set_db(att)
        power = input_power_dbm(TX_POWER_DBM, attenuator.path_loss_db())

        rssi = telemetry.rssi_at(power)
        lq = telemetry.lq_at(power)

        attenuations.append(att)
        rssis.append(rssi)
        lqs.append(lq)
        if telemetry.link_alive_at(power):
            rssi_errors.append(abs(rssi - power))
            floor = power

    measurements.rssi_lq_sweep.x_axis = attenuations
    measurements.rssi_lq_sweep.y_axis.rssi = rssis
    measurements.rssi_lq_sweep.y_axis.lq = lqs
    measurements.rssi_lq_sweep.y_axis.rssi.aggregations.rssi_error_max = max(rssi_errors)

    log.info(f"Sensitivity floor: {floor:.1f} dBm")
    measurements.sensitivity_floor_dbm = floor
```

### Failsafe Threshold and Latency

The flight controller publishes a **failsafe-active flag** over CRSF. The phase records the power at which failsafe engages and the power at which the link dies; the **failsafe margin** between them must be at least 3 dB, otherwise the link cuts off before failsafe engages and the drone enters a "no command" hold state.

```python filename="phases/failsafe_latency.py"
from utils.link_budget import input_power_dbm

TX_POWER_DBM = 14.0


def failsafe_latency(measurements, attenuator, telemetry, log):
    failsafe_at = None
    link_lost_at = None

    for att in range(100, 141, 1):
        attenuator.set_db(att)
        power = input_power_dbm(TX_POWER_DBM, attenuator.path_loss_db())
        if failsafe_at is None and telemetry.failsafe_active_at(power):
            failsafe_at = power
        if not telemetry.link_alive_at(power):
            link_lost_at = power
            break

    log.info(f"Failsafe at {failsafe_at:.1f} dBm, link lost at {link_lost_at:.1f} dBm")
    measurements.failsafe_threshold_dbm = failsafe_at
    measurements.failsafe_margin_db = failsafe_at - link_lost_at

    rtts = [telemetry.telemetry_rtt_ms() for _ in range(50)]
    measurements.telemetry_rtt_ms = sum(rtts) / len(rtts)
```

![Histogram of telemetry round-trip times over 1000 packets at 250 Hz: mean 38.0 ms and 95th percentile 41.3 ms, both under the 50 ms limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-017156d1-a619-467a-a123-8fc09288d014-d1zaavscJsYciVzWwbs7FYVYTq1tb3.png)

Round-trip time distribution of the mock link at 250 Hz: 38 ms mean with 2 ms spread, well inside the 50 ms limit. A real link with a congested telemetry path shows a long tail here before the mean moves.

### Mock Plugs

The attenuator mock shows the plug `config` pattern; the calibrated insertion loss is a YAML value that arrives as a constructor argument:

```python filename="plugs/attenuator.py"
class StepAttenuator:
    def __init__(self, insertion_loss_db):
        self.insertion_loss_db = float(insertion_loss_db)
        self._db = 0
        print(f"Attenuator ready, insertion loss {self.insertion_loss_db} dB")

    def set_db(self, db):
        self._db = int(db)

    def current_db(self):
        return self._db

    def path_loss_db(self):
        return self._db + self.insertion_loss_db
```

The telemetry mock models an ExpressLRS-grade receiver (floor -108 dBm, failsafe -104 dBm, RTT keyed to the configured packet rate) so every limit in the YAML exercises realistically without hardware.


### Board-Level Burn-In Stress Testing
URL: https://www.tofupilot.com/templates/board-level-burn-in-stress-testing
Run powered-on temperature-stress burn-in on PCBAs to weed out infant mortality defects before shipment to consumer, industrial, or automotive customers.


## Introduction

### Burn-In Overview

Electronic assemblies follow the **bathtub curve**: a steep "infant mortality" region with a decreasing hazard rate (Weibull shape β < 1), a flat useful-life plateau, and a rising wear-out tail. **Burn-in** deliberately runs the population through the infant-mortality slope **before shipment**, so customers only see units that have already survived the early failures. The **Arrhenius acceleration factor** (a typical activation energy around 0.7 eV) is what lets 48 h at 105 °C stand in for months of field life.

![Bathtub curve of hazard rate over field time with the infant mortality, useful life and wear-out regions, and the burn-in screen consuming the decreasing-hazard region before shipment.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-70a42339-66de-4824-9be4-5842add7e3c3-FAyybZLJZF6Nb2dMR41jOZgJTpPofS.png)

The region burn-in consumes: everything to the left of the plateau is a failure the customer would otherwise have found in the first weeks.

Defects exposed during burn-in: **weak BGA solder balls, marginal silicon (threshold-voltage drift), electromigration onset in narrow traces, contaminated wire bonds, capacitor electrolyte loss, tin whisker initiation on lead-free finishes**. None of these fail a cold electrical test; all of them fail in the field with the wrong probability distribution.

### Test Purpose

The procedure produces a per-board burn-in record:

- **Pre-burn parametric snapshot**: cold Icc, ADC offset, oscillator error, leakage
- **Chamber ramp** to setpoint with the exerciser running at Vcc +5 %, overshoot and settled temperature
- **48 h dwell** with every heartbeat checked, the temperature and hot Icc logged every 30 minutes, and the exerciser's fault counters accumulated
- **Post-burn parametric snapshot** with the same instruments at ambient
- **Deltas** per parameter, validated against infant-mortality limits

![The burn-in recipe as the run records it: 3 °C per minute ramp to 105 °C with 1.2 K overshoot, 48 hours at setpoint with 17 280 heartbeats, ramp back to ambient, and the hot Icc annealing down 3.6 percent over the dwell.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-7836916a-3f97-4324-a8cb-fd0e9e54df4f-eOWuOv28GTtPPyEnFfAQekjrVjK4ZV.png)

The whole recipe as one run records it on the mock bench: the chamber ramp and its overshoot, 48 hours at 105 °C with the hot Icc annealing down a few percent, the ramp back to ambient. A latent defect shows as the blue trace rising instead.

Burn-in eliminates an entire class of returns: **"died after two weeks", "stopped working mid-flight", "dead out of the box"**. The cost is calendar time and chamber utilization; the return is a population whose field-failure rate is bounded.

Beyond the reliability content, this template demonstrates five framework mechanics: a **setup snapshot used as the delta reference** through previous-results injection into a main phase, an **execution-scoped plug** for the shared chamber, **multi-dimensional measurements with custom aggregations** on the ramp and the dwell log, a **long-running phase with a progress component**, and **JSON measurements** for the snapshots and a validated fault-counter object.

## Equipment & Setup

To implement board-level burn-in on a production line, the following are required:

- **A temperature-controlled chamber** with a programmable profile and a data interface
- **Burn-in boards (BIBs)** carrying several DUTs with edge sockets, a power harness, and the exerciser heartbeat link
- **A programmable DC supply** per BIB capable of Vcc margining
- **The Devices Under Test**: populated, ICT / boundary scan / FCT-passed PCBAs with the exerciser firmware loaded
- **A TofuPilot Framework procedure** to run pre, during and post measurements and validate every board
- **The TofuPilot Dashboard** to log per-unit deltas and trend infant mortality

![Open thermal chamber holding blue circuit boards on a rack](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/pcdu-thermal-cycle-and-burn-in-zHFDGonEEfkWfKlj7MMIXjF84j0isL.webp)

Loaded boards sit on supported shelves in the thermal chamber. The door is open for loading; the supply and monitoring equipment are external.

### Hardware Components

#### Burn-In Chamber

Production-grade options: **Espec** burn-in chambers built around BIB slots with high heat-load tolerance, **Thermotron** AST-class chambers with fast ramps (marketed for HALT and HASS, used for accelerated burn-in), **Weiss** and **CSZ** climatic chambers when humidity is part of the recipe. All expose a serial, Ethernet or SCPI-style interface for the profile and the readback. Typical usable window for industrial products: -40 to +125 °C, 24 to 168 h dwell at the hot extreme.

#### Burn-In Board

The BIB is the load board: a panel of edge sockets, distributed power and ground planes, decoupling, a harness to the chamber feedthrough, hardware reset and watchdog. The on-board exerciser firmware drives every DUT at worst-case power: full CPU load, every peripheral toggling, memory pattern walks, ADC and DAC loopback. Gold-plated sockets and solid copper planes are mandatory to survive thousands of insertions and the steady-state current.

![Burn-in load board close-up: four boards upright in edge-card sockets on the cream carrier, the bulk capacitors along the edge and the power cable leaving one end.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-e44d7508-bada-4d20-a1ef-9f86ea77f227-CZOzbZaYlTWqt3fFI3GwlNnhAQ0qb1.jpg)

A load board: DUTs upright in edge sockets, bulk capacitance along the carrier edge, and the cable that carries margined Vcc and the heartbeat lines through the chamber wall. The chamber's own thermocouples sit on a representative board, not on the carrier.

### Stress Recipes

Common burn-in profiles by product class:

- **Consumer**: 24 h at 85 °C, Vcc nominal, dynamic exercise
- **Industrial** (this template): 48 h at 105 °C, Vcc +5 %
- **Automotive, medical, mil-aero**: 72 to 168 h at 125 °C, Vcc +5 %
- **Power cycling**: 1 min on / 1 min off layered on the soak to stress die attach and BGA joints
- **HAST**: 110 or 130 °C at 85 % RH, biased, as a humidity accelerant (JESD22-A110)

**HALT** (design phase, stress to failure, no survivors expected), **HASS** (production screen at stresses above field but below the HALT limits, a few hours per unit) and **burn-in** (constant elevated temperature and voltage for tens to hundreds of hours, powered and exercised) target different failure populations; this template is the classical burn-in.

### Custom Firmware

The DUT boots into a **burn-in exerciser mode**: full CPU load, every peripheral cycled, a heartbeat every 10 s over a side channel (UART, CAN, Ethernet) carrying the hot supply current and the internal fault counters (memory ECC errors, watchdog resets, peripheral timeouts). The exerciser is flashed at the programming station before burn-in; the production firmware replaces it afterwards.

## Test Procedure

### Overview

For each board on the burn-in board, the procedure runs:

1. Setup: cold parametrics at ambient before the chamber moves.
2. Vcc to +5 %, exerciser on, chamber ramp to 105 °C at 3 °C/min; overshoot and settled temperature checked.
3. Dwell 48 h: every heartbeat checked, temperature and hot Icc logged every 30 min, fault counters accumulated.
4. Vcc back to nominal, chamber ramp to ambient.
5. Cold parametrics again, post minus pre per parameter against the limits.
6. Teardown: slot supply off before the BIB comes out.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  pre_burn.py
  ramp_up.py
  dwell.py
  ramp_down.py
  post_burn.py
  power_off.py
plugs/
  chamber.py
  bib.py
utils/
  recipe.py
README.md
pyproject.toml
```

The chamber and BIB plugs are mocks, time-scaled through one constant in `utils/recipe.py` so the 48 h recipe runs in under a minute; the mock board anneals a few percent like healthy silicon. Run it with `tofupilot run . --no-tui --no-kiosk --json`. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-board-level-burn-in).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: Board-Level Burn-In
version: 0.1.0
description: Industrial burn-in recipe, 48 h at 105 C with Vcc at +5 percent, with cold parametrics before and after and exerciser heartbeat monitoring throughout the dwell.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the board data-matrix before it goes into the socket"
    placeholder: "UAUT-00000"
    pattern: "^UAUT-\\d{5}$"
    default_value: "UAUT-04829"
  part_number:
    default_value: "PCB-MAIN-V2"

plugs:
  - name: Chamber
    description: Environmental chamber, one instance per execution (mock, time-scaled)
    python: plugs.chamber:Chamber
    key: chamber
    scope: execution
  - name: BIB Slot
    description: "One DUT socket on the burn-in board: parametrics, margined Vcc, exerciser heartbeat (mock)"
    python: plugs.bib:BibSlot
    key: bib

setup:
  - name: Pre-Burn Snapshot
    key: pre_burn
    python: phases.pre_burn
    measurements:
      - name: Pre-Burn Snapshot
        key: pre_snapshot
        description: "Cold Icc, ADC offset, oscillator error and leakage at ambient; the reference for every delta."
      - name: Pre-Burn Icc
        key: pre_icc
        unit: mA
        validators:
          - {operator: ">=", expected_value: 100.0}
          - {operator: "<=", expected_value: 140.0}
      - name: Pre-Burn Leakage
        key: pre_leak
        unit: µA
        validators:
          - {operator: "<=", expected_value: 3.0}

main:
  - name: Ramp Up
    key: ramp_up
    python: phases.ramp_up
    timeout: 2h
    measurements:
      - name: Exerciser Started
        key: exerciser_started
        validators:
          - {operator: "==", expected_value: true}
      - name: Vcc Margined
        key: vcc_margined
        unit: V
        validators:
          - {operator: ">=", expected_value: 3.44}
          - {operator: "<=", expected_value: 3.49}
      - name: Ramp
        key: ramp
        title: Chamber ramp to setpoint
        x_axis:
          legend: Time
          unit: min
        y_axis:
          - legend: Temperature
            key: temperature
            unit: °C
            aggregations:
              - type: overshoot_k
                unit: K
                validators:
                  - {operator: "<=", expected_value: 3.0}
              - type: settled_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: 104.0}
                  - {operator: "<=", expected_value: 106.0}

  - name: Dwell
    key: dwell
    python: phases.dwell
    depends_on: [ramp_up]
    ui:
      components:
        - key: dwell_progress
          type: progress
          label: "Dwell at 105 °C"
          description: "48 h, heartbeat every 10 s"
          default_value: 0
          max: 100
    measurements:
      - name: Dwell Log
        key: dwell_log
        title: Temperature and hot Icc over the dwell
        x_axis:
          legend: Elapsed
          unit: h
        y_axis:
          - legend: Temperature
            key: temperature
            unit: °C
            aggregations:
              - type: mean_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: 104.0}
                  - {operator: "<=", expected_value: 106.0}
          - legend: Hot Icc
            key: icc_hot
            unit: mA
            aggregations:
              - type: drift_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: -8.0}
                  - {operator: "<=", expected_value: 2.0}
      - name: Heartbeats
        key: heartbeats
        validators:
          - {operator: "==", expected_value: 17280}
      - name: Missed Heartbeats
        key: missed_heartbeats
        validators:
          - {operator: "==", expected_value: 0}
      - name: Fault Counters
        key: fault_counters
        validators:
          - operator: "=="
            expected_value:
              ecc_errors: 0
              watchdog_resets: 0
              peripheral_timeouts: 0
      - name: Thermal Dose
        key: thermal_dose
        unit: °C·h
        validators:
          - {operator: ">=", expected_value: 5000.0}

  - name: Ramp Down
    key: ramp_down
    python: phases.ramp_down
    depends_on: [dwell]
    timeout: 2h
    measurements:
      - name: Ambient Reached
        key: ambient_reached
        unit: °C
        validators:
          - {operator: "<=", expected_value: 25.0}

  - name: Post-Burn Snapshot
    key: post_burn
    python: phases.post_burn
    depends_on: [ramp_down]
    measurements:
      - name: Post-Burn Snapshot
        key: post_snapshot
      - name: Icc Delta
        key: icc_delta_pct
        unit: "%"
        description: "Post minus pre, cold. Healthy silicon anneals a few percent down; a rise or a large drop is a latent defect."
        validators:
          - {operator: ">=", expected_value: -10.0}
          - {operator: "<=", expected_value: 2.0}
      - name: ADC Offset Delta
        key: adc_offset_delta_lsb
        unit: LSB
        validators:
          - {operator: ">=", expected_value: -1.0}
          - {operator: "<=", expected_value: 1.0}
      - name: Oscillator Drift
        key: osc_drift_ppm
        unit: ppm
        validators:
          - {operator: ">=", expected_value: -10.0}
          - {operator: "<=", expected_value: 10.0}
      - name: Leakage Ratio
        key: leak_ratio
        description: "Post over pre leakage; above 2x is a screened infant-mortality signature."
        validators:
          - {operator: "<=", expected_value: 2.0}

teardown:
  - name: Power Off
    key: power_off
    python: phases.power_off
    measurements:
      - name: Vcc Off
        key: vcc_off
        unit: V
        validators:
          - {operator: "==", expected_value: 0.0}
```

Framework features to notice:

1. **Execution-scoped plug**: `scope: execution` on the chamber means one instance per pass of the procedure, which is what a chamber shared by every board on a burn-in board needs. On a multi-slot station each DUT gets its own `bib` plug while the chamber stays shared.
2. **Setup as the delta reference**: the pre-burn snapshot is a setup phase; `post_burn` names it as a parameter and reads `pre_burn.pre_snapshot`, so the deltas are computed from the run's own record, not from a file on disk.
3. **Multi-dimensional measurements with custom aggregations**: the ramp validates its overshoot and settled temperature; the dwell log validates the mean temperature and the hot Icc drift, all computed in the phase and checked in YAML.
4. **JSON measurements**: both snapshots are recorded as objects, and `fault_counters` is validated as a whole object equal to all zeros, so any non-zero counter fails the phase with the full table on the run.
5. **Timeouts where they fit**: the two ramps carry `timeout: 2h`; a phase timeout tops out at 24 h, so the 48 h dwell runs without one and relies on the heartbeat check and the chamber's own watchdog instead.

### Pre-Burn Snapshot

Cold measurements at ambient before the chamber moves: quiescent Icc, ADC offset, oscillator error against the reference, leakage on the isolation rail. The whole snapshot is kept as JSON; the two values that have absolute limits are also recorded as numbers:

```python filename="phases/pre_burn.py"
def pre_burn(measurements, bib, log):
    """Setup: cold parametrics at ambient before the chamber moves. The delta reference."""
    snap = bib.snapshot()
    log.info(f"Pre-burn: Icc {snap['icc_ma']} mA, ADC offset {snap['adc_offset_lsb']} LSB, osc {snap['osc_ppm']} ppm, leak {snap['leak_ua']} uA")
    measurements.pre_snapshot = snap
    measurements.pre_icc = snap["icc_ma"]
    measurements.pre_leak = snap["leak_ua"]
```

### Ramp Up

Vcc goes to +5 %, the exerciser starts, and the chamber ramps at 3 °C/min. The ramp is recorded as a series with two aggregations: the overshoot past the setpoint and the settled temperature over the last five minutes:

```python filename="phases/ramp_up.py"
import numpy as np

from utils.recipe import SETPOINT_C, VCC_MARGIN, VCC_NOMINAL


def ramp_up(measurements, chamber, bib, log):
    bib.set_vcc(VCC_NOMINAL * VCC_MARGIN)
    measurements.exerciser_started = bib.start_exerciser()
    measurements.vcc_margined = bib.vcc()

    minutes, temp = chamber.ramp_to(SETPOINT_C)
    temp = np.asarray(temp)
    settled = np.asarray(minutes) > (minutes[-1] - 5.0)
    log.info(f"Ramp to {SETPOINT_C} C in {minutes[-1] - 10:.0f} min, overshoot {temp.max() - SETPOINT_C:.2f} K")

    measurements.ramp.x_axis = minutes
    measurements.ramp.y_axis.temperature = temp.tolist()
    measurements.ramp.y_axis.temperature.aggregations.overshoot_k = float(temp.max() - SETPOINT_C)
    measurements.ramp.y_axis.temperature.aggregations.settled_c = float(temp[settled].mean())
```

### Dwell

The dwell is one phase that lasts 48 hours on a real bench. Every 10 s heartbeat is checked for the alive flag and its fault counters are summed; every 30 minutes the chamber temperature and the hot Icc are appended to the log and the progress bar advances:

```python filename="phases/dwell.py"
from utils.recipe import DWELL_HOURS, HEARTBEAT_S, SETPOINT_C

LOG_EVERY_HOURS = 0.5


def dwell(measurements, chamber, bib, ui, log):
    """48 h at setpoint with the exerciser running; every heartbeat is checked, every 30 min logged."""
    hours, temps, iccs = [], [], []
    missed = 0
    faults = {"ecc_errors": 0, "watchdog_resets": 0, "peripheral_timeouts": 0}
    beats = int(DWELL_HOURS * 3600.0 / HEARTBEAT_S)
    log_every = int(LOG_EVERY_HOURS * 3600.0 / HEARTBEAT_S)

    for i in range(beats):
        elapsed_h = (i + 1) * HEARTBEAT_S / 3600.0
        hb = bib.heartbeat(elapsed_h)
        if not hb["alive"]:
            missed += 1
        for key in faults:
            faults[key] += hb[key]
        if i % log_every == 0:
            hours.append(round(elapsed_h, 2))
            temps.append(chamber.temperature())
            iccs.append(hb["icc_ma"])
            ui.dwell_progress = int(100 * elapsed_h / DWELL_HOURS)
            chamber.wait(LOG_EVERY_HOURS * 3600.0)

    log.info(f"Dwell done: {beats} heartbeats, {missed} missed, faults {faults}")
    measurements.dwell_log.x_axis = hours
    measurements.dwell_log.y_axis.temperature = temps
    measurements.dwell_log.y_axis.icc_hot = iccs
    measurements.dwell_log.y_axis.temperature.aggregations.mean_c = sum(temps) / len(temps)
    measurements.dwell_log.y_axis.icc_hot.aggregations.drift_pct = 100.0 * (iccs[-1] - iccs[0]) / iccs[0]
    measurements.heartbeats = beats
    measurements.missed_heartbeats = missed
    measurements.fault_counters = faults
    measurements.thermal_dose = DWELL_HOURS * SETPOINT_C
```

A missed heartbeat is not an exception: the loop keeps counting so the run records when and how often the board went silent, and the `missed_heartbeats == 0` validator fails the phase at the end. On a real bench the heartbeat plug call blocks for the 10 s interval; in the mock it returns at once and the chamber's `wait` is time-scaled.

### Post-Burn Snapshot

The same cold measurements, then the deltas. The pre-burn values arrive through the `pre_burn` parameter:

```python filename="phases/post_burn.py"
def post_burn(measurements, bib, pre_burn, log):
    """Cold parametrics again, same instruments, then post minus pre per parameter."""
    snap = bib.snapshot()
    pre = pre_burn.pre_snapshot
    measurements.post_snapshot = snap

    icc_delta = 100.0 * (snap["icc_ma"] - pre["icc_ma"]) / pre["icc_ma"]
    osc_drift = snap["osc_ppm"] - pre["osc_ppm"]
    leak_ratio = snap["leak_ua"] / pre["leak_ua"]
    log.info(f"Deltas: Icc {icc_delta:+.2f} %, osc {osc_drift:+.2f} ppm, leak x{leak_ratio:.2f}")

    measurements.icc_delta_pct = icc_delta
    measurements.adc_offset_delta_lsb = snap["adc_offset_lsb"] - pre["adc_offset_lsb"]
    measurements.osc_drift_ppm = osc_drift
    measurements.leak_ratio = leak_ratio
```

![Post minus pre deltas on the mock board against their limits: Icc -3.23 percent inside -10 to +2, ADC offset 0.00 LSB inside ±1, oscillator +1.45 ppm inside ±10, leakage ratio 1.10 under 2.0.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-01fad4d6-d9b6-4a1c-9ff4-f5ac0a866012-rp5pZahtR84M4182D1TdjsHyWd6fYX.png)

The four deltas of the mock board against their limits: the small Icc drop and the slight leakage rise are the normal anneal, not a defect. A board with a weak joint or a marginal die shows an Icc rise, a leakage doubling, or an oscillator pull outside ±10 ppm.

One detail worth knowing: a value read back from `measurements.<key>` is a proxy, so the deltas are kept in local variables before being logged and assigned.

### Mock Plugs

The chamber mock ramps at the recipe rate with a small first-order overshoot and reads back the setpoint with noise; the BIB mock carries a healthy board that anneals 3 % over the dwell and pulls its oscillator 1.5 ppm. Both are time-scaled by one constant:

```python filename="utils/recipe.py"
SETPOINT_C = 105.0
AMBIENT_C = 22.0
RAMP_C_PER_MIN = 3.0
DWELL_HOURS = 48.0
VCC_NOMINAL = 3.3
VCC_MARGIN = 1.05
HEARTBEAT_S = 10.0
TIME_SCALE = 1.0 / 20000.0  # 48 h -> ~8.6 s
```

On a real bench, set `TIME_SCALE = 1.0`, map `plugs/chamber.py` to the chamber's protocol over pyvisa and `plugs/bib.py` to the load board's supply channel and heartbeat link; the phases, measurements, and limits stay exactly the same. Multi-slot execution, one `bib` plug per socket with the chamber shared, is the production form of this procedure; the framework's `execution.slots` block declares it, and the CLI's one-shot `tofupilot run` does not accept multi-slot procedures yet, so the template ships single-slot.


### Joint Module Backlash and Stiffness Test
URL: https://www.tofupilot.com/templates/joint-module-backlash-and-stiffness-test
Backlash, lost motion and torsional stiffness of a humanoid joint module: hysteresis loop to rated torque, K1 K2 K3, and no-load transmission error.


## Introduction

### Backlash and Stiffness Overview

A humanoid balances on its joints. What the controller sees as a torque command becomes, through the reducer, an angle at the foot, and the difference between the two is the reducer's lost motion and its compliance. A joint module with too much lost motion chatters at zero crossing; one with too little stiffness lags the controller under load; one whose transmission error has grown a new harmonic has a damaged flexspline or a bearing on its way out. The reducer makers define the three numbers precisely, in arc-minutes, and the joint module end-of-line test measures them on every unit with a reference encoder on the output shaft.

![A strain-wave reducer seen from its output side on a plate: the flexible cup's copper teeth inside the rigid ring gear's teeth, the elliptical dark wave generator hub in the center.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-e69562d0-c495-4ba6-8c70-1a66d5a35033-HyDlgVufVXPVhNcZBwTHEuwhbs6m9y.jpg)

A strain-wave reducer from its output side: the flexspline's teeth engaged in the circular spline at the two lobes of the elliptical wave generator, which is why the transmission error of a healthy unit carries two cycles per input revolution.

The definitions come from the catalogs. Nabtesco specifies **lost motion as the torsion angle read at ±3 % of rated torque** on the hysteresis loop and quotes under 1 arc-min for its RV-E series ([Nabtesco RV-E](https://precision.nabtesco.com/en/products/detail/RV-E)); Harmonic Drive literature uses ±4 %, and its **hysteresis loss is the angle between the two branches of the loop at zero torque** after loading to rated torque and back, 1.0 to 2.5 arc-min by size, with **torsional stiffness piecewise, K1, K2, K3 between the breakpoints T1 and T2** ([SHD catalog](https://www.harmonicdrive.net/_hd/content/documents/SHD-2UH-2SH_GearUnits.pdf)). Both makers give the numbers per size and ratio; the pack maker inherits them as limits. GB/T 35089-2018 gives a bench method for the precision reducer, and the Chinese EOL benches built to it read the output angle at ±1 arc-second.

### Test Purpose

The procedure records one transmission fingerprint per joint module:

- **Reference encoder self-check** before anything is measured against it, and the input brake locked
- **One full hysteresis loop**, 0 to +rated to 0 to -rated to 0, output angle at every step
- From that loop: **hysteresis loss** at zero torque, **lost motion** at ±3 % of rated, and the three **stiffness slopes** K1, K2, K3
- **Angular transmission error** over one no-load output revolution: peak-to-peak, dominant order, once-per-revolution content

![Left, the full hysteresis loop of output angle against torque from -60 to +60 Nm, two branches separated by 0.81 arcmin at zero, the T1 and T2 breakpoints marked. Right, a zoom near zero torque with the ±1.8 Nm lines and the 1.04 arcmin lost motion annotated.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-0343c44b-486b-40cb-9380-97e022378756-sGpEU0aNDvzG2IWJrNvmPUbdHDeaZP.png)

The mock module's loop: the slope steepens at T1 and again at T2 as the flexspline's teeth engage more deeply, the two branches sit 0.81 arc-min apart at zero torque, and the lost motion at ±1.8 Nm is 1.04 arc-min against a 1.5 arc-min limit for this grade.

The framework mechanics on show are five derived aggregations from one curve, each with its own limit; maker definitions coded rather than paraphrased; an integer aggregation validated with `==` on an FFT order; a reference instrument that proves itself in `setup:` before the DUT is measured; and a `teardown:` that releases the brake and the load.

## Equipment & Setup

To run this test on a joint module line, the following are required:

- **A reducer test bench** with a locking brake on the input, a servo load actuator on the output and an inline torque transducer
- **A reference encoder** on the output shaft with arc-second resolution, independent of the module's own encoder
- **The module's drive** over EtherCAT for the no-load transmission-error sweep
- **The Device Under Test (DUT)**: an assembled joint module, run in
- **A TofuPilot Framework procedure** to drive the loop, compute the maker-defined numbers and validate the limits
- **The TofuPilot Dashboard** to trend lost motion across modules, which is where a reducer lot change shows first

### Hardware Components

#### Bench and Reference Encoder

[Zekta's ZPRT-3](https://www.zektaautomation.com/ko/product/zprt-3-robotic-gearbox-factory-acceptance-test-bench/) reducer acceptance bench cites GB/T 30819 and loads at 25, 50, 75, 100 and 125 % of rated; the [Chengbang EOL4000](https://en.chengbang.cn/products_detail/1209191641708834816.html) reads torque to ±0.05 % of full scale and angle to ±1 arc-second. The angle reference is the instrument that matters: a Heidenhain RCN 8000 or a Renishaw RESOLUTE 26-bit ring on the output shaft resolves 0.02 arc-second, three orders below the 1 arc-min the limits are written in. The module's own 19-bit output encoder resolves 2.5 arc-second and is the thing under test in the transmission-error sweep, not the reference.

![Joint actuator with a radial lever and a perpendicular loading screw and load cell](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/joint-module-backlash-and-stiffness-test-physical-2026-ezVI1Ce3mMt509gGNlEX9u6mGiX3wl.webp)

The lever is secured to the output shaft. A load cell and screw apply force perpendicular to the lever to produce torque; angular measurement electronics are not shown.

#### Load Actuator and Transducer

The loop applies ±60 Nm at the output through a lever and a linear actuator, or through a coaxial servo load, with a Kistler 4503B or HBM T40B transducer in the path; class 0.05 on a 100 Nm range is 50 mNm, invisible against the 1.8 Nm the lost-motion definition sits at. The steps are held until the angle reading is static, which is what makes the test slow (minutes, not seconds) and what a spring-loaded shortcut gets wrong.

### Where the Limits Come From

GB/T 35089-2018 and GB/T 30819-2024 give the method; the numbers come from the reducer maker's catalog for the size and ratio, then from the pack maker's own grade selection:

| Test | Limit | Basis |
|---|---|---|
| Hysteresis loss at zero torque | ≤ 2.5 arc-min | Harmonic Drive catalog range 1.0 to 2.5 arc-min by size |
| Lost motion at ±3 % of rated | ≤ 1.5 arc-min | Nabtesco definition; precision grade for this size |
| K1 (below T1) | ≥ 45 kNm/rad | catalog K1 minus 15 % |
| K2 (T1 to T2) | ≥ 60 kNm/rad | catalog K2 minus 25 % |
| K3 (above T2) | ≥ 70 kNm/rad | catalog K3 minus 25 % |
| Transmission error, no load | ≤ 1.5 arc-min p-p | catalog positional accuracy 1.0 to 1.5 arc-min |
| Dominant order | == 200 | 2 × ratio, the wave generator |
| Once-per-revolution content | ≤ 0.5 arc-min | a misaligned reference coupling or an eccentric flexspline |

The ±4 % lost-motion condition that circulates is the Harmonic Drive literature convention; the ±3 % one is Nabtesco's and is the one coded here, as a parameter. The "under 1 arc-min for life" line is a marketing figure; in a 57 hour endurance run on a quasi-direct-drive actuator, backlash grew from 56 to 82 mrad, which is why this template is also the pre/post measurement around a burn-in.

## Test Procedure

### Overview

The procedure maps the test onto the framework's three stages. The reference check and the brake live in `setup:` so no angle is recorded against an encoder with a bad signal or with the input free. Releasing lives in `teardown:` so the bench is never left loaded.

1. Setup: reference encoder self-check, input brake locked.
2. Main: hysteresis loop 0 to +60 to 0 to -60 to 0 Nm; hysteresis, lost motion, K1, K2, K3.
3. Main: no-load transmission error over one revolution; p-p, dominant order, once-per-rev.
4. Teardown: output unloaded, brake released.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for trending.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  reference_check.py
  hysteresis_loop.py
  transmission_error.py
  release.py
plugs/
  joint_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-joint-backlash-stiffness). The `JointBench` plug is a mock of the brake, the load actuator, the transducer, the reference encoder and the module's drive together, synthesizing a healthy precision-grade module on catalog values, so the procedure runs end-to-end without a bench or a module connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Joint Module Backlash and Stiffness Test
version: 0.1.0
description: Backlash, lost motion and torsional stiffness of a humanoid joint module on a 100:1 strain-wave reducer. Reference encoder self-check, a full hysteresis loop to rated torque both ways with the input locked, hysteresis loss and lost motion at +-3 % of rated, piecewise stiffness K1 K2 K3, and the no-load transmission error over one output revolution.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the joint module label"
    placeholder: "JM-KNE-000000"
    pattern: "^JM-[A-Z]{3}-\\d{6}$"
    default_value: "JM-KNE-002931"
  part_number:
    default_value: "JM-R100-60NM"
  batch_number:
    default_value: "WK-2026-37"

plugs:
  - name: Joint Bench
    description: "Input brake, output torque actuator with inline transducer, 23-bit reference encoder, module drive (mock, one plug per bench)"
    python: plugs.joint_bench:JointBench
    key: bench

setup:
  - name: Reference Check
    key: reference_check
    python: phases.reference_check
    measurements:
      - name: Reference Signal
        key: reference_signal_pct
        unit: "%"
        description: Reference encoder signal level; below 70 % the arc-second accuracy is not there and nothing measured against it counts.
        validators:
          - {operator: ">=", expected_value: 70.0}
      - name: Reference Error
        key: reference_error
        validators:
          - {operator: "==", expected_value: false}

main:
  - name: Hysteresis Loop
    key: hysteresis_loop
    python: phases.hysteresis_loop
    timeout: 5m
    measurements:
      - name: Loop
        key: loop
        title: Output angle against applied torque, input locked, 0 to +60 to 0 to -60 to 0 Nm
        x_axis:
          legend: Torque
          unit: Nm
        y_axis:
          - legend: Angle
            key: angle
            unit: arcmin
            aggregations:
              - type: hysteresis_arcmin
                unit: arcmin
                validators:
                  - {operator: "<=", expected_value: 2.5}
              - type: lost_motion_arcmin
                unit: arcmin
                validators:
                  - {operator: "<=", expected_value: 1.5}
              - type: k1_knm_per_rad
                unit: kNm/rad
                validators:
                  - {operator: ">=", expected_value: 45.0}
              - type: k2_knm_per_rad
                unit: kNm/rad
                validators:
                  - {operator: ">=", expected_value: 60.0}
              - type: k3_knm_per_rad
                unit: kNm/rad
                validators:
                  - {operator: ">=", expected_value: 70.0}

  - name: Transmission Error
    key: transmission_error
    python: phases.transmission_error
    depends_on: [hysteresis_loop]
    measurements:
      - name: TE
        key: te
        title: Reference minus module encoder over one no-load output revolution
        x_axis:
          legend: Output angle
          unit: °
        y_axis:
          - legend: Transmission error
            key: te
            unit: arcmin
            aggregations:
              - type: pp_arcmin
                unit: arcmin
                validators:
                  - {operator: "<=", expected_value: 1.5}
              - type: dominant_order
                validators:
                  - {operator: "==", expected_value: 200}
              - type: low_order_arcmin
                unit: arcmin
                validators:
                  - {operator: "<=", expected_value: 0.5}

teardown:
  - name: Release
    key: release
    python: phases.release
```

Framework features to notice:

1. **Five numbers from one curve.** The loop is recorded once; hysteresis, lost motion and the three stiffnesses are aggregations computed in Python and named in the YAML, each with its own limit. A failing module says which one.
2. **The definition is in the code.** `LOST_MOTION_FRACTION = 0.03` in the recipe and the interpolation in the phase are Nabtesco's definition; switching to the ±4 % convention is one constant.
3. **An FFT order as an integer `==`.** `dominant_order == 200` says the transmission error is the wave generator's, and nothing else has overtaken it.
4. **The reference proves itself first.** The signal level and error flag of the reference encoder are validated in `setup:`; a measurement against a degraded reference never reaches the report.

### Reference Check

The setup phase reads the reference encoder's signal level and error flag, locks the input brake, and stamps the reducer grade onto the **unit metadata**:

```python filename="phases/reference_check.py"
def reference_check(measurements, bench, unit, log):
    """Setup: the reference encoder proves itself before anything is
    measured against it; the input brake locks the motor side."""
    ref = bench.reference_encoder_ok()
    measurements.reference_signal_pct = ref["level_pct"]
    measurements.reference_error = ref["error"]
    bench.lock_input()
    unit.metadata["reducer_grade"] = "precision"
    log.info(f"Module {unit.serial_number}: reference encoder at {ref['level_pct']} % signal, input locked")
```

### Hysteresis Loop

Torque stepped through 0, 5, 10, 20, 30, 45, 60 Nm and back, then the same negative, with the input locked and the output angle read at every step once static. The phase separates the loading and unloading branches, reads the hysteresis loss as the gap at zero torque, the lost motion as the angle between the +3 % and -3 % crossings, and fits the three slopes between the catalog breakpoints:

```python filename="phases/hysteresis_loop.py"
import numpy as np

from utils.recipe import LOOP_STEPS_NM, LOST_MOTION_FRACTION, RATED_TORQUE_NM, T1_NM, T2_NM


def hysteresis_loop(measurements, bench, log):
    """Torque stepped 0 -> +rated -> 0 -> -rated -> 0 with the input locked,
    output angle from the reference encoder. From one loop: hysteresis loss
    at rated torque, lost motion at +-3 % of rated, and the three stiffness
    slopes K1, K2, K3 between the catalog breakpoints."""
    loop = bench.hysteresis_loop(LOOP_STEPS_NM)
    t = np.array(loop["torque_nm"])
    a = np.array(loop["angle_arcmin"])
    n = len(LOOP_STEPS_NM)

    # Loading branch and unloading branch of the positive half.
    load_t, load_a = t[:n], a[:n]
    unload_t, unload_a = t[n - 1:2 * n - 1], a[n - 1:2 * n - 1]
    hyst = float(np.interp(0.0, unload_t[::-1], unload_a[::-1]) - np.interp(0.0, load_t, load_a))
    # Full loop: lost motion is the angle between the +3 % and -3 % crossings.
    lm_t = LOST_MOTION_FRACTION * RATED_TORQUE_NM
    a_plus = float(np.interp(lm_t, load_t, load_a))
    neg_load_t, neg_load_a = t[2 * n - 2:3 * n - 2], a[2 * n - 2:3 * n - 2]
    a_minus = float(np.interp(-lm_t, neg_load_t[::-1], neg_load_a[::-1]))
    lost_motion = a_plus - a_minus

    def slope(lo, hi):
        m = (load_t >= lo) & (load_t <= hi)
        k, _ = np.polyfit(load_t[m], np.deg2rad(load_a[m] / 60.0), 1)
        return float(1.0 / k)

    k1, k2, k3 = slope(0.0, T1_NM), slope(T1_NM, T2_NM), slope(T2_NM, RATED_TORQUE_NM)

    measurements.loop.x_axis = loop["torque_nm"]
    measurements.loop.y_axis.angle = loop["angle_arcmin"]
    measurements.loop.y_axis.angle.aggregations.hysteresis_arcmin = abs(hyst)
    measurements.loop.y_axis.angle.aggregations.lost_motion_arcmin = lost_motion
    measurements.loop.y_axis.angle.aggregations.k1_knm_per_rad = k1 / 1000.0
    measurements.loop.y_axis.angle.aggregations.k2_knm_per_rad = k2 / 1000.0
    measurements.loop.y_axis.angle.aggregations.k3_knm_per_rad = k3 / 1000.0
    log.info(f"Hysteresis {abs(hyst):.2f} arcmin, lost motion {lost_motion:.2f} arcmin at +-{lm_t:.1f} Nm, K1/K2/K3 {k1/1000:.0f}/{k2/1000:.0f}/{k3/1000:.0f} kNm/rad")
```

### Transmission Error

The brake is released and the module drives one output revolution at no load, the reference encoder minus the module's own encoder recorded every 0.5°. The peak-to-peak is the angular transmission error; the FFT over the revolution gives the dominant order, which is twice the ratio for a strain-wave reducer, and the once-per-revolution content, which a misaligned reference coupling or an eccentric flexspline raises:

```python filename="phases/transmission_error.py"
import numpy as np

from utils.recipe import RATIO, TRANSMISSION_ERROR_POINTS


def transmission_error(measurements, bench, log):
    """Input released, module drives one output revolution at no load,
    reference encoder minus module encoder every 0.5 deg. Peak-to-peak is
    the angular transmission error; its dominant order says which part
    of the drivetrain produced it: 2 x ratio is the wave generator."""
    bench.release_input()
    sweep = bench.transmission_error_sweep(TRANSMISSION_ERROR_POINTS)
    te = np.array(sweep["te_arcmin"])
    spectrum = np.abs(np.fft.rfft(te - te.mean()))
    order = int(np.argmax(spectrum[1:]) + 1)
    low_order = float(2.0 * spectrum[1:4].max() / len(te))
    measurements.te.x_axis = sweep["angle_deg"]
    measurements.te.y_axis.te = sweep["te_arcmin"]
    measurements.te.y_axis.te.aggregations.pp_arcmin = float(te.max() - te.min())
    measurements.te.y_axis.te.aggregations.dominant_order = order
    measurements.te.y_axis.te.aggregations.low_order_arcmin = low_order
    log.info(f"Transmission error {te.max() - te.min():.2f} arcmin p-p, dominant order {order} (2 x ratio = {2 * RATIO}), once-per-rev content {low_order:.2f} arcmin")
```

![Left, transmission error over the first 30° of the revolution oscillating between -0.45 and +0.6 arcmin, 1.20 arcmin peak-to-peak. Right, the order spectrum with the dominant line at order 200 and a smaller one at order 100.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-5b209798-ad12-44ca-8b66-190f65c8748d-2yzIhOJSxVX8gQqlIaf92wVOn8kBbo.png)

The mock's transmission error: 1.20 arc-min peak-to-peak against a 1.5 limit, the wave generator at order 200 dominant, the input's once-per-revolution at order 100 a third of it, and 0.08 arc-min of once-per-output-revolution content.

### Release

The teardown phase unloads the output and releases the brake, whatever the outcome:

```python filename="phases/release.py"
def release(bench, log):
    """Teardown: output unloaded, brake released."""
    bench.unload()
    bench.release_input()
    log.info("Unloaded, brake released")
```

### Mock Plug

`JointBench` stands in for the brake, the load actuator, the transducer, the reference encoder and the module's drive. It synthesizes a piecewise-linear wind-up with K1 = 54, K2 = 81, K3 = 96 kNm/rad and breakpoints at 7 and 25 Nm, a hysteresis loop 0.85 arc-min wide at zero torque, a transmission error of 0.40 arc-min amplitude at order 200 with 0.12 at order 100 and 0.08 at order 1, and a reference encoder at 94 % signal. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, the class speaks to the bench PLC for the brake and the load steps, reads the transducer through the DAQ, the reference ring through its EnDat or BiSS interface, and the module's encoder over EtherCAT. Hold each torque step until the angle is static, run the loop in both directions, and record the reducer lot on the unit so the lost-motion trend can be cut by lot. The phases, measurements and limits stay the same.


### Boundary Scan JTAG Test
URL: https://www.tofupilot.com/templates/boundary-scan-jtag-test
Test BGA-dense PCBs with IEEE 1149.1 boundary scan to catch hidden solder defects and program flash, FPGA, and bootloader at production.


## Introduction

### Boundary Scan Overview

**IEEE 1149.1 boundary scan** (JTAG) is the only practical way to electrically test a PCB whose components are BGA, fine-pitch QFN, or HDI without exposed test points. The standard defines a **4-wire Test Access Port** (TCK, TMS, TDI, TDO; optional TRST) and a **16-state TAP controller**. Every compliant pin on every JTAG-enabled IC has a **boundary scan cell** stitched into a shift register around the die periphery. Loading instructions like **EXTEST**, **SAMPLE/PRELOAD**, or **BYPASS** lets a tester drive and observe pin states without physical probe contact.

![Cutaway under a BGA package: the grid of solder balls between the chip and the board with one joint missing, and the JTAG header at the board edge that reaches it through the boundary scan cells.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-ab1cae1b-a054-4e86-aafc-251dcddf46ea-rKZCyMAhO4g1pjE8Q4ItjWOaqhzf8G.jpg)

What no probe reaches: the ball grid under a BGA. A missing joint on one ball is an open on one net, and the scan cell behind that ball reports it through the header at the board edge.

Each JTAG-compliant device ships with a **BSDL file** (a VHDL subset) describing pin-to-cell mapping, instruction opcodes, and the IDCODE. The tester loads the BSDLs for every JTAG device on the board plus the netlist, generates the interconnect vectors, and runs them through the chain. **Boundary scan is mandatory** for any board with BGA balls, fine-pitch QFN pads, or HDI vias that a bed-of-nails or a flying probe cannot reach, which is most dense designs today.

### Test Purpose

Boundary scan produces a per-board structural result plus in-system programming:

- **Infrastructure**: every IDCODE on the chain, the BYPASS length, the total cell count
- **Interconnect test (EXTEST)**: walking patterns across every chain-reachable net, diagnosing shorts, opens and stuck-at faults to a net and a pin
- **Cluster test**: the DDR address and data lines exercised through the FPGA's cells
- **In-system programming**: the FPGA bitstream written through the same chain, with CRC readback
- **PCOLA-SOQ coverage**: the IPC score of what this test proves, per component and per net

![Bar chart of the mock netlist by class: 96 BGA-to-BGA, 48 BGA-to-connector, 34 BGA-to-passive and 39 cluster RAM nets reachable by EXTEST, 18 analog and 12 power nets not reachable, 217 of 247 nets covered for 87.9 percent against the 85 percent limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-d878c843-1cf5-4522-9954-e0086048a74f-wwBCARjBMOacqHVBD40Z68z3EBs73L.png)

The mock netlist by class: everything with a scan cell on at least one end is reachable, the analog and power nets are not. The 85 % coverage limit is a property of the board design, not of the unit, so a drop below it means the wrong netlist or BSDL was loaded.

The procedure catches what ICT physically cannot: **BGA cold solder, dry joint, missing via, MCU dead on arrival, bricked FPGA, wrong bitstream, bent connector pin, reversed BGA**. Combined with ICT and AOI it gives near-complete structural coverage on modern dense PCBAs.

Beyond the test content, this template demonstrates four framework mechanics: a **JSON measurement validated as a whole object** for the IDCODE table, a **list measurement validated against an empty list** for the fault diagnosis, an **attachment built in memory** for the per-net report, and **phase timeouts** around the two long-running steps.

## Equipment & Setup

To implement boundary scan testing on a production line, the following are required:

- **A JTAG controller (TAP pod)** with USB or Ethernet to the test station
- **A boundary scan engine** with vector generation, a BSDL library and flash drivers
- **A current-limited DC supply** for the bare board
- **The Device Under Test (DUT)**: a populated, unprogrammed PCB with JTAG access (header or probe pads)
- **CAD inputs**: BSDL per JTAG device, the board netlist, the schematic for non-JTAG clusters
- **A TofuPilot Framework procedure** to drive the engine, record per-net results and validate every metric
- **The TofuPilot Dashboard** to log per-net fault diagnostics and trend BGA defect signatures

![Open clamshell fixture holding a blue circuit board above its test contacts](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/boundary-scan-jtag-test-e3iRu0mM1lR5KNQjAuMUVDYeS3jijG.webp)

The board sits in a clamshell test fixture. The TAP controller, current-limited supply and test computer are not shown.

### Hardware Components

#### JTAG Controller and Engine

Four production tool stacks dominate: **ASSET InterTech ScanWorks**, **Goepel SYSTEM CASCON**, **XJTAG XJDeveloper / XJRunner**, and **JTAG Technologies ProVision** with its JT37x7 controllers. All of them take BSDLs and a netlist, generate the interconnect vectors, diagnose failures to net and pin, and program flash and FPGAs through the chain. Vendor "fast flash" modes upload a small helper into the target's RAM and write flash through it instead of bit-banging every byte through the boundary cells, which is where the 50× speedups come from. For simple chains on in-house boards, **OpenOCD** or **UrJTAG** cover chain scan and programming without the vector generation.

#### CAD Inputs

- **BSDL per IC**: the pin-to-cell map, instruction register length and IDCODE of every JTAG device
- **Board netlist** (ODB++, IPC-2581, or a CAD export)
- **Schematic** for non-JTAG clusters (RAM, level shifters, glue logic)
- **Cluster models** describing transparent buffers and muxes

**Missing BSDLs are the first blocker at NPI**: allow lead time to obtain them from silicon vendors, or have the tool vendor generate them under support.

### Custom Firmware

Boundary scan operates on bare boards with no firmware loaded; the vectors drive pins directly through the TAP. The ISP step of the same station programs the FPGA bitstream (and, on other designs, the bootloader and CPLD) as part of the same run, so structural test and first programming happen at one station.

## Test Procedure

### Overview

After the bare board is placed on the mat and the TAP cable plugged in, the procedure runs:

1. Setup: current-limited 3.3 V on the board, quiescent current checked before any chain access.
2. Infrastructure: scan the chain, compare every IDCODE with the BSDL expectations, check BYPASS length and cell count.
3. In parallel: EXTEST interconnect test over the netlist with the per-net report attached, and the RAM cluster test.
4. Program the FPGA bitstream through the chain and read the CRC back.
5. Score PCOLA-SOQ coverage from the components and nets the test proved.
6. Teardown: rail off before the TAP cable is unplugged.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  power_dut.py
  infrastructure.py
  interconnect.py
  cluster_test.py
  program_flash.py
  coverage.py
  power_off.py
plugs/
  psu.py
  jtag.py
firmware/
  fpga_bitstream_v1.3.bin
utils/
  netlist.py
  pcola.py
README.md
pyproject.toml
```

The controller plug is a mock of a three-device chain (an STM32H7 DAP, an Artix-7 FPGA, a CoolRunner-II CPLD) over a synthetic 247-net netlist, so the procedure runs green without a board. The bitstream is a real 256 KB file whose CRC32 the ISP phase expects. Run it with `tofupilot run . --no-tui --no-kiosk --json`. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-boundary-scan-jtag).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: Boundary Scan JTAG Test
version: 0.1.0
description: Checks the TAP chain, runs the EXTEST interconnect test and the RAM cluster test over a bare PCBA, programs the FPGA bitstream through the same chain, and scores PCOLA-SOQ coverage.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the bare-board data-matrix"
    placeholder: "PCB-000000"
    pattern: "^PCB-\\d{6}$"
    default_value: "PCB-018234"
  part_number:
    default_value: "PCB-CTRL-V3"

plugs:
  - name: DUT Supply
    description: Current-limited 3.3 V rail for the bare board (mock)
    python: plugs.psu:DutSupply
    key: psu
  - name: Boundary Scan Controller
    description: TAP pod plus vendor engine, three-device chain (mock)
    python: plugs.jtag:BoundaryScanController
    key: jtag

setup:
  - name: Power DUT
    key: power_dut
    python: phases.power_dut
    measurements:
      - name: Rail Voltage
        key: rail_voltage
        unit: V
        validators:
          - {operator: ">=", expected_value: 3.2}
          - {operator: "<=", expected_value: 3.4}
      - name: Quiescent Current
        key: quiescent_current
        unit: mA
        description: Unprogrammed board at 3.3 V. Above the limit means a short the chain would misdiagnose.
        validators:
          - {operator: "<=", expected_value: 80.0}

main:
  - name: Infrastructure
    key: infrastructure
    python: phases.infrastructure
    measurements:
      - name: Chain Devices
        key: chain_devices
        validators:
          - {operator: "==", expected_value: 3}
      - name: IDCODEs
        key: idcodes
        description: IDCODE read from each device in TDI to TDO order, compared as one object against the BSDL expectations.
        validators:
          - operator: "=="
            expected_value:
              U1: "0x6BA00477"
              U4: "0x0362D093"
              U7: "0x06E58093"
      - name: BYPASS Length
        key: bypass_length
        unit: bits
        validators:
          - {operator: "==", expected_value: 3}
      - name: Scan Cells
        key: scan_cells
        validators:
          - {operator: "==", expected_value: 1406}

  - name: Interconnect
    key: interconnect
    python: phases.interconnect
    depends_on: [infrastructure]
    timeout: 300s
    measurements:
      - name: Nets Total
        key: nets_total
      - name: Nets Reachable
        key: nets_reachable
      - name: Net Coverage
        key: net_coverage
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 85.0}
      - name: Vectors Applied
        key: vectors_applied
      - name: Faults
        key: faults
        description: Diagnosed faults as net, pin and fault model; empty on a good board.
        validators:
          - {operator: "==", expected_value: []}

  - name: Cluster Test
    key: cluster_test
    python: phases.cluster_test
    depends_on: [infrastructure]
    measurements:
      - name: Cluster Errors
        key: cluster_errors
        validators:
          - {operator: "==", expected_value: 0}
      - name: Cluster Lines
        key: cluster_lines
        validators:
          - {operator: "==", expected_value: 39}

  - name: Program Flash
    key: program_flash
    python: phases.program_flash
    depends_on: [interconnect, cluster_test]
    timeout: 600s
    ui:
      components:
        - key: isp_progress
          type: progress
          label: "FPGA bitstream"
          description: "256 KB through the fast-flash helper"
          default_value: 0
          max: 100
    measurements:
      - name: ISP Duration
        key: isp_seconds
        unit: s
        validators:
          - {operator: "<=", expected_value: 30.0}
      - name: ISP Throughput
        key: isp_throughput
        unit: KB/s
        validators:
          - {operator: ">=", expected_value: 100.0}
      - name: Flash CRC
        key: flash_crc
        validators:
          - {operator: "==", expected_value: "0xEFAA1C46"}
      - name: Flash CRC Match
        key: flash_crc_match
        validators:
          - {operator: "==", expected_value: true}

  - name: Coverage
    key: coverage
    python: phases.coverage
    depends_on: [interconnect, cluster_test]
    measurements:
      - name: PCOLA-SOQ
        key: pcola_soq
        unit: "%"
        description: IPC coverage score over components and nets for boundary scan alone; passives on the chain and unreachable nets cap it, ICT and AOI add the rest.
        validators:
          - {operator: ">=", expected_value: 50.0}
      - name: PCOLA-SOQ Breakdown
        key: pcola_breakdown
      - name: Net Coverage Check
        key: net_coverage_check
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 85.0}

teardown:
  - name: Power Off
    key: power_off
    python: phases.power_off
    measurements:
      - name: Rail Off
        key: rail_off
        unit: V
        validators:
          - {operator: "<=", expected_value: 0.1}
```

Framework features to notice:

1. **JSON measurement compared as a whole**: `idcodes` is a dict of reference designator to IDCODE; the `==` validator compares the entire object, so a wrong variant, a missing device or a swapped chain order all fail the same measurement with the full table on the run.
2. **List validated against `[]`**: `faults` carries the engine's diagnosis (net, pin, fault model). A good board records an empty list; a failed board records exactly what failed, on the run, without a separate report.
3. **Attachment from memory**: the per-net CSV is built in the phase and attached with `attach.data`, no temp file.
4. **Timeouts on the long steps**: 300 s for the interconnect run and 600 s for ISP bound a hung pod; the teardown still powers the board down.
5. **Fan-out and fan-in**: `interconnect` and `cluster_test` both depend on `infrastructure`; `program_flash` and `coverage` depend on both, so a broken chain skips everything downstream with one clear cause.

### Infrastructure

Before any vector runs, the chain must be electrically sane: each device's IDCODE must match its BSDL, BYPASS must shift one bit per device, and the cell count must be the sum of the BSDLs. A broken TAP at one device blocks everything downstream, so this phase isolates a chain-level cause before any per-net result is reported:

```python filename="phases/infrastructure.py"
def infrastructure(measurements, jtag, log):
    devices = jtag.scan_chain()
    idcodes = {d["ref"]: d["idcode"] for d in devices}
    log.info("Chain: " + ", ".join(f"{d['ref']} {d['part']} {d['idcode']}" for d in devices))

    measurements.chain_devices = len(devices)
    measurements.idcodes = idcodes
    measurements.bypass_length = jtag.bypass_length()
    measurements.scan_cells = sum(d["cells"] for d in devices)
```

### Interconnect

The engine loads EXTEST on every device and applies walking patterns across every reachable net. The phase records the coverage, the vector count and the fault list, and attaches the per-net report as CSV:

```python filename="phases/interconnect.py"
import csv
import io


def interconnect(measurements, jtag, attach, log):
    results, faults = jtag.extest_interconnect(8)
    reachable = [r for r in results if r["reachable"]]
    coverage = 100.0 * len(reachable) / len(results)
    log.info(f"EXTEST: {len(reachable)}/{len(results)} nets reachable ({coverage:.1f} %), {len(faults)} faults")

    measurements.nets_total = len(results)
    measurements.nets_reachable = len(reachable)
    measurements.net_coverage = coverage
    measurements.vectors_applied = sum(r["vectors"] for r in results)
    measurements.faults = faults

    buf = io.StringIO()
    writer = csv.DictWriter(buf, fieldnames=["name", "class", "reachable", "vectors", "result"])
    writer.writeheader()
    writer.writerows(results)
    attach.data(buf.getvalue().encode(), "interconnect_report.csv")
```

Plug methods take positional arguments only (they cross a process boundary as JSON), which is why the vector count is passed as `8` and not `vectors_per_net=8`.

### Program Flash

With the structure proven, the same chain programs the FPGA bitstream. The mock models a fast-flash helper at 180 KB/s; the CRC read back from the target is compared with the CRC of the file, and the progress bar tracks the transfer:

```python filename="phases/program_flash.py"
import zlib

BITSTREAM = "firmware/fpga_bitstream_v1.3.bin"


def program_flash(measurements, jtag, ui, log):
    expected = f"0x{zlib.crc32(open(BITSTREAM, 'rb').read()) & 0xFFFFFFFF:08X}"
    ui.isp_progress = 5
    crc, seconds, rate = jtag.program_flash(BITSTREAM)
    ui.isp_progress = 100
    readback = jtag.read_flash_crc()
    log.info(f"ISP {seconds:.1f} s at {rate:.0f} KB/s, CRC {readback} (image {expected})")

    measurements.isp_seconds = seconds
    measurements.isp_throughput = rate
    measurements.flash_crc = readback
    measurements.flash_crc_match = readback == expected
```

### Coverage

The IPC **PCOLA-SOQ** framework scores what a test proves per component (Presence, Correctness, Orientation, Live, Alignment) and per net (Shorts, Opens, Quality), each 0, 0.5 or 1. Boundary scan scores well on presence, correctness, orientation and live for the chain devices and on shorts and opens for reachable nets; it cannot judge alignment or solder quality, and it says little about a passive hanging off a chain net. The phase computes the score from the component and net counts and reads the interconnect coverage through **previous-results injection**:

```python filename="phases/coverage.py"
from utils.netlist import NET_CLASSES
from utils.pcola import score


def coverage(measurements, interconnect, log):
    components = {"jtag_device": 3, "cluster_ram": 2, "passive_on_chain": 34}
    reachable = sum(n for n, r in NET_CLASSES.values() if r)
    unreachable = sum(n for n, r in NET_CLASSES.values() if not r)
    pct, breakdown = score(components, {"reachable": reachable, "unreachable": unreachable})
    log.info(f"PCOLA-SOQ {pct:.1f} % (components {breakdown['component_pct']} %, nets {breakdown['net_pct']} %)")

    measurements.pcola_soq = pct
    measurements.pcola_breakdown = breakdown
    measurements.net_coverage_check = float(interconnect.net_coverage)
```

![PCOLA-SOQ breakdown for boundary scan alone: per-component presence, correctness, orientation, live and alignment scores for JTAG devices, cluster RAM and passives on the chain giving 35.9 percent, and per-net shorts, opens and quality for reachable and unreachable nets giving 58.6 percent, 53.8 percent overall against the 50 percent limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-0c860f2c-8a9f-4d62-b32e-b83782ab435d-Mosddq9Xp0XMBUmslStt5Rfs8Mo1we.png)

The honest number for boundary scan alone on this board: 53.8 %. The 34 passives on chain nets and the zero alignment and quality scores are what ICT, AOI and AXI add on top; the value of recording the score per run is that a board revision or a BSDL change moves it visibly.

### Mock Plugs

The controller mock carries the three-device chain, walks the synthetic netlist, and computes the target CRC from the same bitstream file. Its method surface is what a wrapper around XJRunner, ProVision or ScanWorks has to provide:

```python filename="plugs/jtag.py"
CHAIN = [
    {"ref": "U1", "part": "STM32H743 (ARM DAP)", "idcode": "0x6BA00477", "ir_length": 4, "cells": 0},
    {"ref": "U4", "part": "XC7A35T Artix-7", "idcode": "0x0362D093", "ir_length": 6, "cells": 1214},
    {"ref": "U7", "part": "XC2C64A CoolRunner-II", "idcode": "0x06E58093", "ir_length": 8, "cells": 192},
]


class BoundaryScanController:
    def __init__(self):
        self._nets = nets()
        print("TAP pod connected, TCK 10 MHz")

    def scan_chain(self):
        """Devices found on the chain in TDI -> TDO order, from their IDCODEs."""
        return [dict(d) for d in CHAIN]

    def bypass_length(self):
        """Bits shifted through with every device in BYPASS: one per device."""
        return len(CHAIN)

    def extest_interconnect(self, vectors_per_net=8):
        """Walking-1/0 interconnect test; returns per-net results and the fault list."""
        results = []
        faults = []
        for net in self._nets:
            if not net["reachable"]:
                results.append({**net, "vectors": 0, "result": "not_covered"})
                continue
            results.append({**net, "vectors": vectors_per_net, "result": "pass"})
        return results, faults
```

The ARM DAP on the MCU has no boundary cells (it is a debug port on the chain, not a boundary scan device), which is why its cell count is zero and why the interconnect vectors only drive the FPGA and CPLD pins. On a real bench, replace this class with the vendor engine's API; the phases, measurements, and limits stay exactly the same.


### Barometric Altimeter Pressure Calibration
URL: https://www.tofupilot.com/templates/barometric-altimeter-pressure-calibration
Calibrate a drone MEMS barometer for offset, scale, and thermal drift across the pressure-temperature envelope to ensure accurate altitude hold.

## Introduction

### Barometer Overview

A drone uses a **MEMS barometric pressure sensor** to estimate altitude. The sensor measures absolute atmospheric pressure at 50-100 Hz, and the autopilot maps pressure to altitude using the **ISA (International Standard Atmosphere)** model:

`h = 44330 · (1 − (P/P₀)^0.190284)` [m]

Near sea level, **1 Pa ≈ 8.3 cm of altitude**. GNSS vertical accuracy is 3-5 m and slow; baro is fast, low-noise, and bounded, but only if its offset, scale, and thermal drift are characterized at the factory. Altitude hold, terrain following, geo-fence ceiling enforcement, and EKF vertical fusion all rely on a calibrated baro.

![Flight-controller PCB with the metal-can barometer package and its pressure port highlighted, next to a 3 mm scale bar.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-43d7cc28-9ae7-45ee-8016-ee1471b55852-iBHZgfLXW2QQtXAqqoxP4wjwgRxyGk.jpg)

The metal-can barometer package and its pressure port on the flight-controller board.

Three sensors dominate drone production: the [TE MS5611](https://www.te.com/en/product-MS561101BA03-50.html) (0.012 mbar resolution at highest oversampling, 24-bit ΔΣ, factory PROM coefficients), the [Bosch BMP388](https://www.bosch-sensortec.com/media/boschsensortec/downloads/datasheets/bst-bmp388-ds001.pdf) (±8 Pa relative accuracy ≈ ±0.6 m, ±50 Pa absolute, TCO ±0.75 Pa/K, default for Betaflight and ArduPilot), and the [Infineon DPS310](https://www.infineon.com/cms/en/product/sensor/pressure-sensors/pressure-sensors-for-iot/dps310/) (±0.002 hPa precision ≈ 2 cm in high-precision mode, ±0.06 hPa relative accuracy, favored on PX4 and DJI). Each has different uncompensated drift, and **temperature compensation is mandatory** because the silicon diaphragm's Young's modulus drops with temperature, CTE mismatch between Si and the LGA package strains the membrane, and the ASIC adds its own offset and gain drift.

### Calibration Purpose

The procedure characterizes per-DUT pressure sensor performance across the **pressure × temperature envelope** and produces a compensation model:

- **Offset** at reference pressure and temperature (typically 100 kPa, 25 °C)
- **Scale** factor relating raw reading to true pressure
- **Temperature compensation coefficients** (linear or bivariate quadratic)
- **Noise floor** (Allan deviation at τ = 1 s, 10 s, 100 s)
- **Pressure-altitude conversion error** after applying the compensation model

![DUT minus reference pressure across 80-120 kPa at 0, 25 and 50 °C: before compensation a 140-195 Pa temperature-dependent offset, after the bivariate quadratic fit a flat residual of 0.30 Pa RMS within the ±5 Pa limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-d136a664-2cd4-4925-b55d-d1951fa9778d-pyKHMDKKwTUKW4DsHeYj0elmShVx9q.png)

Residual of the mock DUT over the full envelope before and after the compensation fit.

Without per-unit calibration, uncompensated drift can reach **tens of Pa across 0-50 °C**, enough to push the altitude estimate out of the EKF baro innovation gate (`EKF2_BARO_GATE` on PX4) and cause altitude jumps during takeoff in cold weather. With calibration, post-correction residual stays below **5 Pa (40 cm altitude)** across the entire envelope.

## Equipment & Setup

To implement barometer calibration on a production line, the following are required:

- **A traceable pressure reference** (controller or transducer) with **±0.01% FS class accuracy**
- **A climate chamber** covering 0-50 °C with pressure feedthrough
- **A sealed test cell** plumbed to the pressure controller, with electrical breakout for the DUT
- **The Device Under Test (DUT)**: autopilot board with baro mounted
- **A TofuPilot Framework procedure** to coordinate chamber, pressure, and DUT, fit the compensation model, and validate residuals
- **The TofuPilot Dashboard** to log per-unit coefficients and trend population statistics

### Hardware Components

#### Pressure Reference

The **Druck DPI 612 pFlex** is the bench-grade reference: accuracy from **0.005% FS** (module-dependent), **−95 to 2000 kPa** range, integrated pump and electrical port. For high-volume production lines, the [Mensor CPC4000](https://www.mensor.com/en-us/cpc4000.WIKA) is the standard: **0.02% IS-50** accuracy over a **−1 to 210 bar** span of available ranges, with parallel-port multi-DUT manifolds. Both are ISO 17025 traceable; recertification is typically annual (Mensor specifies a 365-day calibration interval).

![Sensor board inside a sealed pressure chamber with pneumatic and electrical feedthroughs](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/barometric-altimeter-pressure-calibration-physical-2026-0kx5UCZwe9ZiyNu97rK8W0OajGXLlz.webp)

The board sits on standoffs inside the sealed chamber. Separate pneumatic and electrical connections lead to external control and measurement equipment.

For portable / line-side spot checks, the **Fluke 3130-G2M** (−80 kPa to 2 MPa, ±0.025% rdg + 0.01% FS, integrated pump) is the field reference. A reference transducer sits **inside the test cell**, not outside the line, to eliminate pneumatic lag.

#### Climate Chamber

The **Vötsch VT4034** (335 L workspace, −40 to +180 °C) is the production standard for temperature coverage. The chamber needs a **pressure feedthrough bulkhead** with a pneumatic and an electrical pass-through. For lower volume, a **Memmert TTC256** environmental test chamber (256 L, −42 to +190 °C) or a custom Peltier cell handles the narrower 0-50 °C drone envelope at lower cost.

#### Test Cell

A machined **aluminum sealed cell** with an **O-ring lid**, a **4-pin pogo block** for electrical contact, and a **pressure tap** to the controller. Reference transducer mounted inside the cell. Cells must be **leak-tested** before each shift: a 1 kPa leak per minute is enough to invalidate the calibration.

### Custom Firmware

The DUT autopilot must expose a **raw baro streaming mode** at the maximum native rate (typically 50-100 Hz for BMP388, 200 Hz for DPS310). The stream includes:

- **Timestamp** (ms)
- **Raw pressure** (Pa, ADC counts with known scale, or already-corrected Pa from the sensor's factory PROM, depending on the chip)
- **Raw temperature** (°C from the sensor's built-in temperature channel)

The autopilot must **not apply on-board temperature compensation** during the calibration run, otherwise the calibration captures the sensor's own thermal compensation rather than the residual error.

## Test Procedure

### Overview

After mounting the DUT in the sealed cell and closing the chamber, the procedure runs:

1. Sweep pressure 80 → 120 kPa at 0, 25, and 50 °C, logging DUT, reference, and temperature.
2. Fit the per-DUT bivariate quadratic compensation model and validate the residuals.
3. Dwell at 100 kPa / 25 °C and compute the Allan deviation at 1 s, 10 s, and 100 s.
4. Write the coefficients to DUT non-volatile storage and verify the read-back.
5. Stream results to TofuPilot for traceability and analytics.

Total cycle is roughly 45 minutes per DUT on real hardware: three temperature soaks with stabilization, a ~10 minute pressure ramp at each, and the 10 minute noise dwell, amortized by batch parallelism (4-8 cells per chamber). The mock chamber compresses all of this into seconds so the template runs instantly; the phase structure and limits are what carry over to hardware.

This template exercises several framework capabilities on purpose: **plug config and shared plug scope**, **custom aggregation types** for the Allan deviation with their own validators, a **JSON measurement** carrying the coefficient set, a **phase timeout**, parallel phases via **depends_on**, and **previous-results injection**.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  pressure_sweep.py
  compensation_fit.py
  noise_characterization.py
  save_calibration.py
plugs/
  chamber.py
utils/
  allan.py
pyproject.toml
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-barometer-calibration). The mock chamber plug synthesizes the DUT error model in code, so the procedure runs end to end without hardware:

```bash
tofupilot run .
```

### The Procedure File

`procedure.yaml` declares the unit (with revision and batch defaults), the chamber plug with its `config` and `scope`, and four phases. `compensation_fit` and `noise_characterization` both depend only on the sweep, so the engine runs them **in parallel**; `save_calibration` waits for both. This is the exact file from the repository:

```yaml filename="procedure.yaml"
name: Barometric Altimeter Pressure Calibration
version: 0.1.0
description: Characterizes a MEMS barometer across the pressure-temperature envelope and fits a per-unit compensation model.

unit:
  auto_identify: true
  serial_number:
    default_value: "SN-BARO-0001"
  part_number:
    default_value: "FC-MAIN-V2"
  revision_number:
    default_value: "Rev C"
  batch_number:
    default_value: "BATCH-2026-031"

plugs:
  - name: Climate Chamber
    key: chamber
    description: Chamber + pressure controller. Execution scope (one instance shared by all slots). Newer engines add scope station to hold the chamber session across executions on a station daemon.
    python: plugs.chamber:MockChamber
    scope: all
    config:
      address: "192.168.1.60"

main:
  - name: Pressure Sweep
    key: pressure_sweep
    python: phases.pressure_sweep
    timeout: 5m
    measurements:
      - name: Sweep Samples
        key: sweep_samples
        validators:
          - operator: ">="
            expected_value: 1800

  - name: Compensation Fit
    key: compensation_fit
    python: phases.compensation_fit
    depends_on: [pressure_sweep]
    measurements:
      - name: Residual RMS
        key: residual_rms
        unit: Pa
        description: RMS of the post-fit residual over the full P x T envelope.
        validators:
          - operator: "<="
            expected_value: 5.0
      - name: Residual Max
        key: residual_max
        unit: Pa
        validators:
          - operator: "<="
            expected_value: 12.0
      - name: Scale Factor Error
        key: scale_error_pct
        unit: "%"
        validators:
          - operator: "<="
            expected_value: 0.5
      - name: Compensation Coefficients
        key: compensation_coefficients
        description: Bivariate quadratic model written to the DUT, stored as JSON for traceability.

  - name: Noise Characterization
    key: noise_characterization
    python: phases.noise_characterization
    depends_on: [pressure_sweep]
    measurements:
      - name: Noise Capture
        key: noise_capture
        unit: Pa
        description: 10 Hz dwell at 100 kPa / 25 C. Allan deviation is validated per averaging time.
        aggregations:
          - type: allan_1s
            unit: Pa
            validators:
              - operator: "<="
                expected_value: 0.5
          - type: allan_10s
            unit: Pa
            validators:
              - operator: "<="
                expected_value: 0.3
          - type: allan_100s
            unit: Pa
            validators:
              - operator: "<="
                expected_value: 0.2

  - name: Save Calibration
    key: save_calibration
    python: phases.save_calibration
    depends_on: [compensation_fit, noise_characterization]
    measurements:
      - name: Readback OK
        key: readback_ok
        validators:
          - operator: "=="
            expected_value: true
```

Three details worth noticing. The plug takes a **`config` mapping**: the chamber address lives in the YAML, not in Python. The plug's **`scope: all`** shares one chamber instance across all slots of an execution; on a station daemon, newer engines accept `scope: station` to keep the session open between executions so back-to-back units skip the reconnect. And the sweep phase carries a **`timeout: 5m`**: if the chamber hangs mid-ramp, the phase ends with a timeout outcome instead of blocking the line indefinitely.

### The Chamber Plug

The mock injects a known error model (offset, thermal terms, scale, noise) that the fit must recover. It also **caches the sweep**: the plug instance persists across phases, so the fit phase fetches the exact dataset the sweep phase captured instead of re-running a sweep that costs 45 minutes on real hardware. Swap the class for one driving your real chamber and pressure controller over SCPI; the interface stays identical:

```python filename="plugs/chamber.py"
import math
import random


class MockChamber:
    """Simulated climate chamber + pressure controller + DUT breakout.

    Declared with `scope: all` so one instance is shared by all slots.
    On a station daemon, newer engines accept `scope: station` to keep
    the chamber session open across executions.
    """

    # Injected DUT error model, unknown to the fit
    OFFSET = 118.0        # Pa at 25 C
    T_LIN = 0.8           # Pa per K
    T_QUAD = 0.012        # Pa per K^2
    SCALE = 1.0004
    NOISE_PA = 0.3

    def __init__(self, address):
        self.address = address
        self._saved = None
        self._last_sweep = None
        random.seed(388)
        print(f"Chamber connected at {address}")

    def sweep(self, temps_c, p_start, p_end, points_per_temp):
        """Ramp pressure at each temperature. Returns flat sample lists.

        The capture is also cached on the plug so later phases can fetch
        the same dataset with last_sweep() instead of re-running a sweep
        that takes ~45 minutes on real hardware.
        """
        ref, dut, temp = [], [], []
        for t_c in temps_c:
            for i in range(points_per_temp):
                p_true = p_start + (p_end - p_start) * i / (points_per_temp - 1)
                dt = t_c - 25.0
                err = self.OFFSET + self.T_LIN * dt + self.T_QUAD * dt * dt
                p_dut = self.SCALE * p_true + err + random.gauss(0, self.NOISE_PA)
                ref.append(p_true)
                dut.append(p_dut)
                temp.append(t_c + random.gauss(0, 0.05))
        self._last_sweep = {"ref": ref, "dut": dut, "temp": temp}
        return self._last_sweep

    def last_sweep(self):
        """Return the cached capture from the most recent sweep."""
        if self._last_sweep is None:
            raise RuntimeError("No sweep has been run yet")
        return self._last_sweep

    def dwell(self, seconds, rate_hz):
        """Hold 100 kPa / 25 C and stream DUT noise samples."""
        n = int(seconds * rate_hz)
        base = 100_000.0
        drift = 0.0
        out = []
        for i in range(n):
            drift += random.gauss(0, 0.004)          # slow random walk
            out.append(base + drift + random.gauss(0, self.NOISE_PA))
        return out

    def write_calibration(self, coefficients):
        self._saved = dict(coefficients)
        return True

    def read_calibration(self):
        return self._saved
```

### Pressure Sweep

At each temperature setpoint, ramp pressure 80 → 120 kPa while logging DUT and reference. On real hardware the slow ramp ensures the sealed cell tracks the controller without pneumatic lag:

```python filename="phases/pressure_sweep.py"
TEMPS_C = [0.0, 25.0, 50.0]
P_START = 80_000.0
P_END = 120_000.0
POINTS = 600


def pressure_sweep(chamber, measurements, log):
    log.debug(f"Sweeping {P_START/1000:.0f}-{P_END/1000:.0f} kPa at {TEMPS_C} C")
    data = chamber.sweep(TEMPS_C, P_START, P_END, POINTS)
    n = len(data["ref"])
    measurements.sweep_samples = n
    log.info(f"Captured {n} samples across {len(TEMPS_C)} temperature soaks")
```

![Pressure ramp from 80 to 120 kPa at the 50 °C soak: controller setpoint, in-cell reference transducer tracking it, and the DUT raw reading offset by about 180-195 Pa.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-08bcedc0-76de-4618-8365-be267499d543-vMPOwHYPccoL8rB6YFpfBsEKP6Xige.png)

One ramp of the sweep at the 50 °C soak; the inset shows the DUT error the fit has to absorb.

### Compensation Model Fit

For BMP388 and DPS310 (and MS5611 if rejecting factory PROM), fit a **bivariate quadratic** P_corr = a₀ + a₁·P_raw + a₂·T + a₃·P_raw·T + a₄·T². The fit consumes the sweep phase's dataset through the plug's `last_sweep()` cache, so both phases operate on the same samples. The coefficient set is recorded as a **JSON measurement**: the whole model travels with the run, queryable per unit years later:

```python filename="phases/compensation_fit.py"
import numpy as np


def compensation_fit(chamber, measurements, log):
    # Fetch the dataset the Pressure Sweep phase captured. The plug caches
    # it, so the fit consumes the exact samples that were counted, and a
    # real bench never pays for a second 45-minute sweep.
    data = chamber.last_sweep()
    p_raw = np.asarray(data["dut"])
    p_ref = np.asarray(data["ref"])
    t = np.asarray(data["temp"])

    # P_corr = a0 + a1*P_raw + a2*T + a3*P_raw*T + a4*T^2
    design = np.column_stack(
        [np.ones_like(p_raw), p_raw, t, p_raw * t, t**2]
    )
    coeffs, *_ = np.linalg.lstsq(design, p_ref, rcond=None)
    residuals = design @ coeffs - p_ref

    rms = float(np.sqrt(np.mean(residuals**2)))
    peak = float(np.max(np.abs(residuals)))
    scale_error = abs(coeffs[1] - 1.0) * 100.0

    log.debug(f"Fit coefficients: {coeffs}")
    log.info(f"Post-fit residual {rms:.2f} Pa RMS, {peak:.2f} Pa max")
    if rms > 2.5:
        log.warning(f"Residual RMS {rms:.2f} Pa above the 2.5 Pa watch level")

    measurements.residual_rms = rms
    measurements.residual_max = peak
    measurements.scale_error_pct = float(scale_error)
    measurements.compensation_coefficients = {
        "model": "bivariate_quadratic",
        "a0": float(coeffs[0]),
        "a1": float(coeffs[1]),
        "a2": float(coeffs[2]),
        "a3": float(coeffs[3]),
        "a4": float(coeffs[4]),
    }
```

### Allan Deviation with Custom Aggregations

The dwell at 100 kPa / 25 °C feeds an **Allan deviation** computation per IEEE 952-2020 conventions adapted for barometers. Excessive 1 s σ points to a noisy ASIC; high 100 s σ points to a leaking package or contaminated port.

Aggregation types are free strings, so the three averaging times are declared as `allan_1s`, `allan_10s`, and `allan_100s` on the `noise_capture` array measurement, each with its own validator. Python computes the values; the engine enforces the limits:

```python filename="utils/allan.py"
import numpy as np


def allan_deviation(samples, rate_hz, tau_s):
    """Non-overlapping Allan deviation at one averaging time."""
    samples = np.asarray(samples, dtype=float)
    m = int(tau_s * rate_hz)
    n_clusters = len(samples) // m
    if n_clusters < 3:
        raise ValueError(f"Need >= 3 clusters at tau={tau_s}s, got {n_clusters}")
    means = samples[: n_clusters * m].reshape(n_clusters, m).mean(axis=1)
    return float(np.sqrt(0.5 * np.mean(np.diff(means) ** 2)))
```

```python filename="phases/noise_characterization.py"
import numpy as np

from utils.allan import allan_deviation

RATE_HZ = 10
DWELL_S = 600


def noise_characterization(chamber, measurements, log):
    log.debug(f"Dwelling {DWELL_S}s at 100 kPa / 25 C for the noise capture")
    samples = chamber.dwell(DWELL_S, RATE_HZ)
    detrended = list(np.asarray(samples) - np.mean(samples))

    measurements.noise_capture = detrended
    aggs = measurements.noise_capture.aggregations
    aggs.allan_1s = allan_deviation(samples, RATE_HZ, 1)
    aggs.allan_10s = allan_deviation(samples, RATE_HZ, 10)
    aggs.allan_100s = allan_deviation(samples, RATE_HZ, 100)
    log.info(
        f"Allan deviation: {aggs.allan_1s:.3f} Pa @1s, "
        f"{aggs.allan_10s:.3f} Pa @10s, {aggs.allan_100s:.3f} Pa @100s"
    )
```

![Allan deviation log-log plot of the DUT pressure noise from the 600 s dwell: white-noise slope at short averaging times, minimum near 10 s, random-walk rise beyond, with allan_1s, allan_10s and allan_100s marked against their limits.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-f1a2a0b9-8c7e-4ef1-8ae3-263638736b93-MpvtMqW91ciJiWNnxU1vCNNctOTO5k.png)

Allan deviation of the mock dwell: white noise averages down until the slow random walk takes over past 10 s.

### Saving Calibration to DUT

The save phase reads the fitted model from the fit phase through **previous-results injection** (the parameter named `compensation_fit` exposes that phase's measurements), writes it through the plug, and verifies the read-back:

```python filename="phases/save_calibration.py"
def save_calibration(chamber, compensation_fit, measurements, log):
    coeffs = compensation_fit.compensation_coefficients
    chamber.write_calibration(coeffs)
    readback = chamber.read_calibration()
    ok = readback == coeffs
    if not ok:
        log.error("Calibration readback mismatch, refusing to pass the unit")
    measurements.readback_ok = ok
    log.info("Coefficients written to DUT non-volatile storage and verified")
```

On a real product the write targets the firmware's own calibration store: a dedicated flash page, EEPROM, or the sensor vendor's PROM, read by the driver at boot. On PX4, thermal compensation lives in the `TC_B*` parameters produced by the [sensor thermal calibration workflow](https://docs.px4.io/main/en/advanced_config/sensor_thermal_calibration.html), and the runtime applies it before pressure reaches the EKF baro innovation gate (`EKF2_BARO_GATE`; `EK3_ALT_M_NSE` is the ArduPilot equivalent noise setting). ArduPilot's runtime baro parameters (`BARO1_GND_PRESS`, `GND_ALT_OFFSET`) are not a factory-calibration store: the ground pressure is recalculated at every baro calibration and `GND_ALT_OFFSET` resets, so per-unit compensation belongs in your own firmware storage rather than those fields. After write, reboot the DUT and read back to confirm persistence.


### MCU Programming and Serialization
URL: https://www.tofupilot.com/templates/mcu-programming-and-serialization
Flash firmware, assign serial numbers, write MAC and calibration data to PCBAs at end-of-line with traceability and secure provisioning.


## Introduction

### Programming and Serialization Overview

**MCU programming and serialization** is the last point in PCBA manufacturing where the device is open for write access before functional test. Three operations sequence in this stage: **firmware flash**, **per-unit data write** (serial number, MAC address, calibration constants, keys), and **verify-readback**. It is frequently merged into the FCT station to save a fixture and one handling step, but logically separable: programming uses debug-port access whereas FCT exercises the running firmware. Many high-mix EMS lines keep them distinct so a programming failure does not block FCT throughput.

![Close-up of the PCB under programming: the microcontroller, the 10-pin SWD debug header with the ribbon cable plugged in, and the data-matrix serial sticker in the board corner.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-83340f62-a2b7-4a0b-a0e7-9a04f47a4126-cknNkwqyRJOYStG7kIWEhzWOlZuLrs.jpg)

The board as the station sees it: the data-matrix serial the operator scans, the 10-pin SWD header the fixture probes, the MCU that leaves this station with an identity it keeps for life.

The output: every shipped board carries a **unique identity** (serial, MAC, BLE address), a **production-validated firmware image** (CRC-verified against the release file), and **per-unit calibration constants** (ADC offset and gain trim, RTC trim, sensor offsets). This identity stays with the unit through its lifetime, indexed in MES and TofuPilot.

### Test Purpose

The programming and serialization procedure produces a per-board record of:

- **Target identity**: SW-DP IDCODE, device ID, flash size, so a wrong-variant MCU placed at SMT is rejected before anything is written
- **Allocated MAC address** from an IEEE-registered block, plus the pool level
- **Firmware flash** through the vendor programmer CLI, with its transcript attached to the run
- **Identity page** with serial, MAC and calibration block, recorded as JSON
- **Flash CRC** computed by the target against the CRC of the release image, firmware version, readback of the identity
- **Read-out protection** level after lock, and the rail state after teardown

![Flash map of the 1 MB mock target: 16 KB bootloader region at 0x08000000, 480 KB application region at 0x08008000, 128 KB identity sector at 0x080E0000, with the 32 KB release image and the identity page the station writes shown above.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-4c7f0a2b-5928-4342-ab76-d4c6258fbabe-mLKGphQiqqMjkDe51t00X8GmMDev2f.png)

The mock target's flash: the release image covers the start of the bootloader and application regions, the station writes the identity page in the last sector, and the CRC check covers the image the programmer wrote.

Per-unit programming eliminates **mis-flashed firmware revision, missing bootloader, duplicate serial number, blank identity page, MAC pool exhaustion, and unlocked production units**. Every shipped board is traceable from the production database to the customer.

Beyond the programming content, this template demonstrates five framework mechanics: an **executable phase** that runs the programmer CLI and passes on its exit code, **string, boolean and JSON measurements** next to the numeric ones, a **station-scoped plug** for the MES connection, **unit metadata written from Python**, and a **file attachment** of the programmer transcript on the run.

## Equipment & Setup

To implement programming and serialization on a production line, the following are required:

- **A programming tool** (single or gang) matched to the target MCU family
- **A test fixture** with the SWD / JTAG header on pogo pins and a switched 3.3 V rail
- **A barcode scanner** for the board serial and a **label printer** for the shipping label
- **MES integration** for MAC allocation, calibration lookup and result recording
- **The Device Under Test (DUT)**: an ICT-passed PCBA with an un-programmed MCU
- **A TofuPilot Framework procedure** to orchestrate probe, allocate, flash, write, verify and lock
- **The TofuPilot Dashboard** to log per-board programming history

![Open clamshell fixture holding a blue circuit board above its test contacts](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/boundary-scan-jtag-test-e3iRu0mM1lR5KNQjAuMUVDYeS3jijG.webp)

The board is located in a clamshell contact fixture. The programmer, barcode scanner, label printer and test computer are not shown.

### Hardware Components

#### Programming Tool

Selection depends on volume and MCU family:

- **STMicroelectronics ST-LINK V3** with **STM32CubeProgrammer**: SWD / JTAG plus a UART / I2C / SPI / CAN bridge to exercise the system bootloader from the same fixture. Its `STM32_Programmer_CLI` is the command this template's executable phase stands in for.
- **SEGGER J-Link family**: J-Link Plus / Pro for engineering and low volume, **Flasher Compact** for medium volume, **Flasher ATE** and **Flasher Hub** for gang programming, all scriptable from the test station over USB or Ethernet.
- **Microchip MPLAB PICkit 5 / ICD** with **MPLAB IPE** for PIC / dsPIC; **Nordic nrfjprog** and **pyOCD** for Nordic and generic Cortex-M targets.
- **Automated programmers** (BPM Microsystems, Data I/O) sit **upstream of SMT** to pre-program memories and MCUs on tape or tray before placement; the end-of-line station then only writes identity and verifies.

#### Protocols

- **SWD** (ARM 2-wire, pin-compatible subset of JTAG): default for Cortex-M
- **JTAG** (IEEE 1149.1): boundary scan plus in-system programming for higher-pin-count parts and FPGAs
- **System bootloader** in ROM (STM32: UART, SPI, I2C, USB-DFU, CAN depending on family), when the debug port is not accessible in the fixture
- **I2C / SPI EEPROM**: direct write to a side memory holding MAC and serial
- **USB-DFU**: when the device's USB port is the only test-accessible interface

### Identity Sources

The **serial number** comes from the data-matrix already on the board (scanned at identification). The **MAC address** comes from an IEEE-registered block (an MA-L, MA-M or MA-S assignment under the company's OUI) handed out by MES one address per request, so two stations never write the same one. **Calibration constants** come from upstream test phases keyed by serial. The firmware image is built per release with a fixed layout: bootloader, application, and a reserved identity page the station fills.

## Test Procedure

### Overview

After the board is loaded into the fixture and its data-matrix scanned, the procedure runs:

1. Power the 3.3 V rail, check idle current, read the SW-DP IDCODE, device ID and flash size.
2. Ask MES for a MAC address, record the pool level, stamp the MAC on the unit metadata.
3. Flash the release image through the programmer CLI (executable phase).
4. Write the identity page: serial, MAC, calibration block.
5. Read back the flash CRC and compare it with the CRC of the release image, read the firmware version, check the identity readback, attach the programmer transcript.
6. Set read-out protection level 1 and confirm the debug port is locked.
7. Teardown: rail off before the operator opens the fixture.

![Phase timeline of one run of the mock station: probe target 11 ms, allocate identity 4 ms, flash firmware 1344 ms through the programmer CLI, write identity 10 ms, verify image 33 ms, lock device 11 ms, power off 6 ms.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-6f51f579-e3a5-42fd-80e4-d809013da0df-43SJRHtAAjo46iz1NwX9mzYkZkrDak.png)

One run of the mock station as recorded by the CLI: the programmer transcript is the only phase that takes real time, everything around it is bookkeeping the framework records.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  probe_target.py
  allocate_identity.py
  write_identity.py
  verify_image.py
  lock_device.py
  power_off.py
plugs/
  psu.py
  swd.py
  mes.py
tools/
  mock_flash.sh
firmware/
  app_v2.4.1.hex
utils/
  intel_hex.py
  flash_map.py
README.md
pyproject.toml
```

The plugs are mocks and the programmer CLI is a shell script that prints a CubeProgrammer-style transcript, so the procedure runs green on any machine. The firmware file is a real 32 KB Intel HEX image whose CRC32 is the value the verify phase expects. Run it with `tofupilot run . --no-tui --no-kiosk --json`. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-mcu-programming-serialization).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: MCU Programming and Serialization
version: 0.1.0
description: Flashes the release firmware over SWD, allocates the unit identity from MES, writes the identity page, verifies the image CRC, and locks the debug port.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the data-matrix on the board"
    placeholder: "UAUT-00000"
    pattern: "^UAUT-\\d{5}$"
    default_value: "UAUT-04829"
  part_number:
    default_value: "PCB-MAIN-V2"
  metadata:
    mac_address:
      description: "Allocated by MES during the run"
    firmware_version:
      description: "Read back from the target after flashing"

plugs:
  - name: Fixture Power
    description: 3.3 V rail through the pogo pins (mock)
    python: plugs.psu:FixturePower
    key: power
  - name: SWD Probe
    description: Debug probe on the 10-pin header (mock ST-LINK / J-Link)
    python: plugs.swd:SwdProbe
    key: swd
  - name: MES Identity Service
    description: MAC pool and calibration lookup, connection held across units
    python: plugs.mes:IdentityService
    key: mes
    scope: station

main:
  - name: Probe Target
    key: probe_target
    python: phases.probe_target
    measurements:
      - name: Rail Voltage
        key: rail_voltage
        unit: V
        validators:
          - {operator: ">=", expected_value: 3.2}
          - {operator: "<=", expected_value: 3.4}
      - name: Idle Current
        key: idle_current
        unit: mA
        description: Unprogrammed target in reset. A short on the rail shows up here before any flash access.
        validators:
          - {operator: "<=", expected_value: 80.0}
      - name: SW-DP IDCODE
        key: idcode
        validators:
          - {operator: "==", expected_value: "0x2BA01477"}
      - name: Device ID
        key: device_id
        description: Rejects a wrong-variant MCU placed at SMT before anything is written.
        validators:
          - {operator: in, expected_value: ["0x413", "0x419"]}
      - name: Flash Size
        key: flash_size
        unit: KB
        validators:
          - {operator: "==", expected_value: 1024}

  - name: Allocate Identity
    key: allocate_identity
    python: phases.allocate_identity
    depends_on: [probe_target]
    measurements:
      - name: MAC Address
        key: mac_address
        validators:
          - {operator: matches, expected_value: "^70:B3:D5:[0-9A-F]{2}:[0-9A-F]{2}:[0-9A-F]{2}$"}
      - name: MAC Pool Remaining
        key: pool_remaining
        description: Addresses left in the OUI block; reorder below 500.
        validators:
          - {operator: ">=", expected_value: 500}

  - name: Flash Firmware
    key: flash_firmware
    depends_on: [allocate_identity]
    timeout: 120s
    executable:
      command: "sh tools/mock_flash.sh firmware/app_v2.4.1.hex"
      # Real bench: STM32_Programmer_CLI -c port=SWD mode=UR -d firmware/app_v2.4.1.hex -v

  - name: Write Identity
    key: write_identity
    python: phases.write_identity
    depends_on: [flash_firmware]
    measurements:
      - name: Identity Page Address
        key: identity_address
        validators:
          - {operator: "==", expected_value: "0x080E0000"}
      - name: Calibration Block
        key: calibration_block
        description: Per-unit constants written next to the identity, recorded as JSON for later comparison.

  - name: Verify Image
    key: verify_image
    python: phases.verify_image
    depends_on: [write_identity]
    measurements:
      - name: Flash CRC
        key: flash_crc
        description: CRC32 the target computes over its flash, compared to the CRC of the release image.
        validators:
          - {operator: "==", expected_value: "0x60A7D0AA"}
      - name: Firmware Version
        key: firmware_version
        validators:
          - {operator: in, expected_value: ["2.4.1"]}
      - name: Verify Mismatch Bytes
        key: verify_mismatch_bytes
        unit: B
        validators:
          - {operator: "==", expected_value: 0}
      - name: Identity Readback
        key: identity_readback
        validators:
          - {operator: "==", expected_value: true}

  - name: Lock Device
    key: lock_device
    python: phases.lock_device
    depends_on: [verify_image]
    measurements:
      - name: RDP Level
        key: rdp_level
        validators:
          - {operator: "==", expected_value: 1}
      - name: Debug Locked
        key: debug_locked
        validators:
          - {operator: "==", expected_value: true}

teardown:
  - name: Power Off
    key: power_off
    python: phases.power_off
    measurements:
      - name: Rail Off Voltage
        key: rail_off_voltage
        unit: V
        validators:
          - {operator: "<=", expected_value: 0.1}
```

Framework features to notice:

1. **Executable phase**: `flash_firmware` has no Python. The framework runs the command through the shell from the procedure directory, passes the phase on exit code 0 and fails it otherwise; the `timeout` bounds a programmer that hangs on a dead target. Swapping the mock script for `STM32_Programmer_CLI`, `JLinkExe` or `nrfjprog` is a one-line change.
2. **String measurements**: `idcode` validates with `==`, `device_id` with `in` (two accepted variants), `mac_address` with a `matches` regex pinned to the company's OUI, `flash_crc` and `firmware_version` with `==` and `in`.
3. **Boolean and JSON measurements**: `identity_readback` and `debug_locked` are booleans validated `== true`; `calibration_block` is a JSON measurement with no validator, recorded for trending.
4. **Station-scoped plug**: `scope: station` keeps the MES connection alive across units in station mode, so back-to-back boards do not pay the reconnection; in a one-shot `tofupilot run` it behaves as a normal per-execution plug.
5. **Unit metadata from Python**: `mac_address` and `firmware_version` are declared under `unit.metadata` and filled by the phases, so the dashboard unit carries both and can be filtered by either.

### Probe Target

The first phase powers the rail and reads what is on the other end of the SWD header. The SW-DP IDCODE identifies the debug port (0x2BA01477 is the ARM Cortex-M4 SW-DP), the device ID identifies the silicon (0x413 is STM32F405/407), and the flash size confirms the variant. Together they reject a wrong part before the programmer writes a byte:

```python filename="phases/probe_target.py"
def probe_target(measurements, power, swd, log):
    power.enable(3.3)
    measurements.rail_voltage = power.measure_rail()
    measurements.idle_current = power.measure_current_ma()

    idcode = swd.read_idcode()
    device_id = swd.read_device_id()
    flash_kb = swd.flash_size_kb()
    log.info(f"Target {device_id} IDCODE {idcode}, {flash_kb} KB flash")

    measurements.idcode = idcode
    measurements.device_id = device_id
    measurements.flash_size = flash_kb
```

### Allocate Identity

MES hands out the next MAC in the company's block and reports how many are left. The phase records both and stamps the MAC on the unit metadata:

```python filename="phases/allocate_identity.py"
def allocate_identity(measurements, mes, unit, log):
    identity = mes.allocate(unit.serial_number)
    log.info(f"Allocated MAC {identity['mac']} for {unit.serial_number}")

    measurements.mac_address = identity["mac"]
    measurements.pool_remaining = mes.pool_remaining()
    unit.metadata["mac_address"] = identity["mac"]
```

### Flash Firmware

The executable phase runs the programmer. On the real bench the command is the vendor CLI; the template ships a shell script that prints the same kind of transcript, writes it to `flash.log` for the verify phase to attach, and exits 0:

```text
      -------------------------------------------------------------------
                        Mock Programmer CLI v2.19.0
      -------------------------------------------------------------------
ST-LINK SN  : 0670FF3234424E3043212637
Voltage     : 3.29V
SWD freq    : 4000 KHz
Connect mode: Under reset
Device ID   : 0x413
Device name : STM32F405xx/F407xx/F415xx/F417xx
Flash size  : 1 MBytes
Memory Programming ...
  File          : firmware/app_v2.4.1.hex
  Size          : 32768 Bytes
  Address       : 0x08000000
Erasing memory corresponding to segment 0:
  Erasing internal memory sectors [0 1]
Download in Progress:
  10%
  ...
  100%
File download complete
Time elapsed during download operation: 00:00:01.240
Verifying ...
Download verified successfully
```

A non-zero exit from the programmer fails the phase, the dependent phases are skipped, and the teardown still powers the fixture down.

### Write Identity

The identity page holds the serial, the MAC just allocated, and the calibration constants MES keeps for this serial. The MAC comes from the previous phase through **previous-results injection**: naming a parameter after a completed phase's key injects its measurements:

```python filename="phases/write_identity.py"
def write_identity(measurements, swd, mes, unit, allocate_identity, log):
    calibration = mes.calibration_for(unit.serial_number)
    block = {"serial": unit.serial_number, "mac": allocate_identity.mac_address, **calibration}

    address = swd.write_identity(block)
    log.info(f"Identity page written at 0x{address:08X}")

    measurements.identity_address = f"0x{address:08X}"
    measurements.calibration_block = calibration
```

Assigning a `dict` to a measurement stores it as JSON, so the exact constants written to this board are on its run without flattening them into separate fields.

### Verify Image

The CRC the target computes over its flash is compared with the CRC32 of the release file, computed on the station from the Intel HEX. The firmware version is read back, the identity page is compared with what was allocated, the firmware version lands on the unit metadata, and the programmer transcript is attached to the run:

```python filename="phases/verify_image.py"
from utils.intel_hex import image_crc32

FIRMWARE = "firmware/app_v2.4.1.hex"


def verify_image(measurements, swd, unit, attach, allocate_identity, log):
    expected = image_crc32(FIRMWARE)
    actual = swd.read_flash_crc()
    version = swd.read_firmware_version()
    log.info(f"Flash CRC {actual}, release image {expected}, firmware {version}")

    measurements.flash_crc = actual
    measurements.firmware_version = version
    measurements.verify_mismatch_bytes = swd.verify_mismatch_bytes()
    measurements.identity_readback = swd.read_identity().get("mac") == allocate_identity.mac_address

    unit.metadata["firmware_version"] = version
    attach.file("flash.log", f"{unit.serial_number}_flash.log")
```

Two details worth knowing: a measurement read back from `measurements.<key>` returns a proxy, so keep the value in a local variable when you also write it to `unit.metadata` (metadata accepts strings, numbers and booleans only); and `attach.file` takes an optional second argument to name the attachment, here prefixed with the serial so the transcript is findable from the unit page.

The release image CRC is a plain utility over the HEX records:

```python filename="utils/intel_hex.py"
import zlib


def read_hex(path):
    ext = 0
    chunks = []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if not line.startswith(":"):
                continue
            raw = bytes.fromhex(line[1:])
            count, addr, rtype, data = raw[0], (raw[1] << 8) | raw[2], raw[3], raw[4:-1]
            if rtype == 0x04:
                ext = ((data[0] << 8) | data[1]) << 16
            elif rtype == 0x00:
                chunks.append((ext + addr, data))
    chunks.sort(key=lambda c: c[0])
    return b"".join(d for _, d in chunks)


def image_crc32(path):
    return f"0x{zlib.crc32(read_hex(path)) & 0xFFFFFFFF:08X}"
```

### Lock Device

After verification passes, read-out protection goes to level 1: the debug port is blocked, the flash cannot be read back, and the setting is reversible with a mass erase. Level 2 is permanent and belongs behind an explicit sign-off, which is why the template stops at 1:

```python filename="phases/lock_device.py"
def lock_device(measurements, swd, log):
    swd.set_rdp(1)
    level = swd.read_rdp()
    log.info(f"Read-out protection level {level}")
    measurements.rdp_level = level
    measurements.debug_locked = level >= 1
```

### Mock Plugs

The SWD mock returns the identifiers of an STM32F405, keeps the identity page it was given so the readback check has something to compare, and computes the "target" CRC from the same release file. The MES mock hands out sequential MACs from a fixed block and returns a fixed calibration block. Both carry the method surface a real pylink or MES client needs to implement:

```python filename="plugs/mes.py"
class IdentityService:
    OUI = "70:B3:D5"  # example IEEE MA-M style prefix
    POOL_SIZE = 4096

    def __init__(self):
        self._next = 0x1A2B00
        self._used = 0
        print("MES identity service connected")

    def allocate(self, serial_number):
        mac = f"{self.OUI}:{(self._next >> 16) & 0xFF:02X}:{(self._next >> 8) & 0xFF:02X}:{self._next & 0xFF:02X}"
        self._next += 1
        self._used += 1
        return {"serial": serial_number, "mac": mac, "ble_addr": mac}

    def pool_remaining(self):
        return self.POOL_SIZE - self._used

    def calibration_for(self, serial_number):
        """Constants measured upstream (ADC trim, RTC trim) keyed by serial."""
        return {"adc_offset_lsb": 8, "adc_gain": 1.012, "rtc_trim_ppm": -3.5}
```

On a real bench, point the `flash_firmware` command at the programmer CLI, replace `plugs/swd.py` with a pylink or pyOCD implementation, and `plugs/mes.py` with the MES client; the phases, measurements, and limits stay exactly the same.


### BMS Protection Trip Verification
URL: https://www.tofupilot.com/templates/bms-protection-trip-verification
Verify a 16S BMS PCBA's protections: OV and UV trip point and delay per channel, OCD and SCD by voltage injection, OTD by two-point NTC substitution.


## Introduction

### BMS Protection Overview

The protections on a battery management board are the reason the board exists. The analog front end (AFE) carries hardware comparators for cell over-voltage and under-voltage, for charge and discharge over-current on the sense resistor, for short-circuit discharge, and a temperature path through an external thermistor; each one has a programmable threshold and a programmable delay written into one-time-programmable memory at the end of the line, and each one, when it trips, opens the pack's charge or discharge FET. A board that leaves the factory with the wrong threshold, an unwritten configuration block, a swapped sense input, a mis-populated shunt or a thermistor from the wrong reel passes every voltage-accuracy test and fails the one time it matters.

![A BMS PCBA on a plate: the black AFE chip, two power MOSFETs, the flat shunt resistor between them with two sense wires, and the NTC thermistor bead on its two leads at the board's corner.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-323408f5-f450-40a2-ab4e-770124252cf1-LO3MvSk4Wbr77PJekcJJv8cmgzpwX9.jpg)

The parts under test on one board: the AFE with its comparators, the two power FETs it drives, the shunt whose voltage the over-current comparators read, and the thermistor whose resistance the over-temperature path converts to a temperature.

The datasheets are precise about what these comparators are allowed to do, and less precise than their front pages suggest. The **TI BQ76952** guarantees its over-voltage threshold to ±2 mV at 25 °C over the full setting range, and to **±25 mV from -40 to 85 °C in the 3.036 to 5.06 V band** where every real Li-ion OVP setting lives (see the [BQ76952 datasheet](https://www.ti.com/lit/ds/symlink/bq76952.pdf)); the thresholds themselves are set in **50.6 mV steps** and the delays in **3.3 ms steps**. Over-current accuracy is an absolute ±5 mV in the 56 to 100 mV band, which is ±8 % at a 60 mV setting; short-circuit detection is ±15 %; the short-circuit response at the fastest setting runs from **600 ns at 25 mV of overdrive to 8 µs at 3 mV**, before the recommended 100 Ω / 0.1 µF input filter adds its own 10 µs time constant. The automotive **BQ79616** is looser, not tighter, at ±24 mV on OV, and gets its safety rating from redundancy and a built-in comparator self-test rather than from precision. Every limit in this template is derived from these rows.

### Test Purpose

The procedure records one protection fingerprint per board:

- **Configuration** read bit-exact before any stimulus, as one object, plus its CRC
- **OV and UV trip point** on every channel by binary search on a cell simulator, and the **ALERT delay** on each
- **OCD1 and SCD thresholds** by voltage injection across the sense-resistor inputs, the **SCD response time** at a stated overdrive, the **DSG gate** after the short, and the **reset flags** that tell a trip from a crash
- **Temperature** reported at two NTC resistances, the **OTD trip** by a resistance sweep, and an **open thermistor** that must raise a fault
- The configuration read **again at teardown**, so a threshold the test touched can never ship lowered

![Left, OV trip point minus the 4250 mV setting per channel as bars inside the ±30 mV limit lines, channel 7 at -18 mV in orange. Right, the same for UV against 2800 mV, all within ±15 mV.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-b05410e1-83af-477f-a6ad-9cd88f849c98-AD1R1cWADSQs3LcMMZ7njGkU7zegle.png)

The mock board's sixteen OV and sixteen UV comparators against their settings: every trip inside the ±30 mV window derived from the datasheet's -40 to 85 °C row plus the simulator, channel 7's OV comparator 18 mV low and the one to watch across a batch.

The framework mechanics on show are JSON `==` on a whole configuration object before and after the test, multi-dimensional measurements with two curves and aggregations on both, limits copied from the AFE datasheet with delay limits expressed in quanta, a measurement that distinguishes a protection trip from a device reset, a two-point check that catches a wrong thermistor reel, and a search helper shared between two phases.

## Equipment & Setup

To run this verification on a production line, the following are required:

- **A battery cell simulator** with one isolated, four-quadrant channel per cell input and sub-millisecond settling
- **A current-sensor simulation channel**: a ±10 V analog output driven onto the filter side of the shunt's sense inputs
- **A programmable resistor** on the thermistor input, with open and short states
- **A timer input on the AFE's ALERT pin** for delay measurement, and a digitiser on the DSG gate for the short-circuit response
- **The Device Under Test (DUT)**: a 16S BMS PCBA with its production firmware and configuration
- **A TofuPilot Framework procedure** to sequence the stimuli, search the trip points and validate the limits
- **The TofuPilot Dashboard** to keep the per-board evidence that ISO 26262-7 asks a production line to keep

### Hardware Components

#### Cell Simulator and Injection Source

The [Chroma 87001](https://www.chromausa.com/pdf/87001%20Battery%20Cell%20Simulator%20052023.pdf) drives the sixteen cell inputs with 1 ms rise and fall and ±1 mV setting accuracy, and its software explicitly supports the OVP/UVP trip and release sequence. For the over-current tests the shunt is not driven with hundreds of amps; the AFE's OCC, OCD and SCD thresholds are all defined on **V(SRP) - V(SRN)**, a differential voltage at two pins, and the comparator cannot tell a source from a shunt. A current-sensor simulation channel (Bloomy lists one as a standard item: analog voltage, ±10 V, 16-bit, ±0.5 %; an NI analog output does the same) is driven onto the high-impedance filter side of the shunt, where the 100 Ω series resistors isolate it from the sub-milliohm shunt. On the [Chroma 8700](https://www.chromaate.com/en/product/bms_functional_ats_ev_8700_625) BMS test system the same function is an integrated current-sensing-unit stimulus.

![BMS protection board in a fixture with cell-sense wiring and protection-test probes](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/bms-protection-trip-verification-distinct-2026-9e3CDDBc33GjeKBEREQunNGniueU1L.webp)

The fixture holds the protection board and provides cell-sense, temperature-simulation and probe connections. The stimulus sources and timing instruments are external.

#### Programmable Resistor

The thermistor is substituted with a resistance, not a voltage. On a ratiometric AFE the trip temperature is a ratio of the thermistor to the on-board pull-up, so driving the pin with a DAC would bypass the pull-up and miss a wrong pull-up value. The [Pickering 40-297A-050](https://www.pickeringtest.com/product/40-297a-050-pxi-precision-resistor-module-6-channel-3R-to-1-51M) covers 3 Ω to 1.51 MΩ at 0.125 Ω resolution with programmable open and short states; Bloomy's temperature simulation module offers 24 channels from 10 Ω to 500 kΩ at 1 % accuracy. At 60 °C a 10 kΩ NTC reads about 2.5 to 3 kΩ and moves 90 Ω per kelvin, so a 1 Ω module resolves 0.01 K; a decade box has no place in volume production.

### Where the Limits Come From

None of IEC 62619, IEC 62133, UL 2054, UL 1973, ISO 6469 or GB 38031 specifies a BMS PCBA protection test; they apply abuse to cells and packs and judge the BMS by outcome. ISO 26262-7 requires that a safety-related characteristic be controlled in production and the evidence kept, which is why this test exists and is recorded per serial, but it hands out no millivolt. The limits are the AFE datasheet rows widened by the tester:

| Test | Limit | Basis |
|---|---|---|
| OV and UV trip point | setting ±30 mV | ±25 mV over -40..85 °C plus ±1 mV simulator, rounded |
| OV and UV delay | setting + 2 steps (≤ 107 ms at 100 ms) | 3.3 ms delay quantum, timed on the ALERT edge |
| OCD1 threshold | setting ±5 mV (55 to 65 mV) | ±5 mV absolute accuracy in the 56 to 100 mV band |
| OCD1 delay | ≤ 47 ms at 40 ms | 3.3 ms quantum |
| SCD threshold | setting ±20 % (160 to 240 mV) | ±15 % datasheet plus fixture |
| SCD response | ≤ 40 µs at +25 mV overdrive | 15 µs setting plus the input filter; the overdrive is part of the limit |
| Temperature at 25 and 60 °C, OTD trip | ±3 °C | ±0.5 % comparator (about ±1 K) plus 1 % resistor plus NTC tolerance |
| Configuration | exact match | no tolerance applies |

Two numbers in that table are traps. The OV front-page figure of ±2 mV describes the same part at 25 °C over the whole setting range; a limit built on it fails good boards. And an SCD response time without its overdrive and its filter is not a measurement: the same setting responds thirteen times faster at 25 mV of overdrive than at 3 mV, and the filter adds more than the whole comparator spec.

## Test Procedure

### Overview

The procedure maps the verification onto the framework's three stages. The configuration readback lives in `setup:` so no trip point is hunted on a board running the wrong protection block. The restore lives in `teardown:` so the stimulus goes off and the configuration is re-read even when a main phase fails.

1. Setup: protection block and CRC read bit-exact.
2. Main: OV trip point and ALERT delay, sixteen channels, binary search.
3. Main: UV trip point and delay, one channel lowered at a time.
4. Main: OCD1 and SCD by injection, SCD response, DSG gate, reset flags.
5. Main: temperature at two NTC points, OTD sweep, open thermistor.
6. Teardown: stimulus off, faults cleared, CRC again.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-board evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  config_readback.py
  ovp_trip.py
  uvp_trip.py
  overcurrent_injection.py
  otp_thermistor.py
  restore.py
plugs/
  protection_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-bms-protection-trips). The `ProtectionBench` plug is a mock of the simulator, the injection source, the programmable resistor and the DUT's UART together, synthesizing a healthy board with every comparator inside the datasheet window, one OV channel 18 mV low, the released configuration in OTP and a thermistor of the right B value, so the procedure runs end-to-end without a bench or a board connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug with its cell count as constructor argument, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: BMS Protection Trip Verification
version: 0.1.0
description: Protection trip verification of a 16S BMS PCBA. Config read bit-exact, OV and UV trip point and delay on every channel by binary search on a cell simulator, OCD and SCD by sense-resistor voltage injection with the reset flags checked, over-temperature by two-point NTC substitution and open-thermistor fault, config re-read at teardown.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the board data-matrix before it goes into the fixture"
    placeholder: "BMS16-00000"
    pattern: "^BMS16-\\d{5}$"
    default_value: "BMS16-01743"
  part_number:
    default_value: "PCB-BMS-16S-R3"
  batch_number:
    default_value: "LOT-2026-37"

plugs:
  - name: Protection Bench
    description: "Cell simulator, SRP/SRN injection source, programmable NTC resistor and DUT UART (mock, one plug per bench)"
    python: plugs.protection_bench:ProtectionBench
    key: bench
    config:
      cell_count: 16

setup:
  - name: Config Readback
    key: config_readback
    python: phases.config_readback
    measurements:
      - name: Protection Config
        key: protection_config
        description: Every protection register decoded to engineering units and compared as one object against the released configuration.
        validators:
          - operator: "=="
            expected_value:
              ov_mv: 4250
              ov_delay_ms: 100
              uv_mv: 2800
              uv_delay_ms: 100
              ocd1_mv: 60
              ocd1_delay_ms: 40
              scd_mv: 200
              scd_delay_us: 15
              otd_c: 60
      - name: Config CRC
        key: config_crc
        validators:
          - {operator: "==", expected_value: "0x3A7F"}

main:
  - name: OVP Trip
    key: ovp_trip
    python: phases.ovp_trip
    timeout: 3m
    measurements:
      - name: OV Trips
        key: ov_trips
        title: Over-voltage trip point and ALERT delay per channel
        x_axis:
          legend: Channel
        y_axis:
          - legend: Trip point
            key: trip
            unit: mV
            aggregations:
              - type: max_dev_mv
                unit: mV
                validators:
                  - {operator: "<=", expected_value: 30.0}
              - type: min_dev_mv
                unit: mV
                validators:
                  - {operator: ">=", expected_value: -30.0}
          - legend: Delay
            key: delay
            unit: ms
            aggregations:
              - type: max_ms
                unit: ms
                validators:
                  - {operator: "<=", expected_value: 107.0}

  - name: UVP Trip
    key: uvp_trip
    python: phases.uvp_trip
    depends_on: [ovp_trip]
    timeout: 3m
    measurements:
      - name: UV Trips
        key: uv_trips
        title: Under-voltage trip point and ALERT delay per channel
        x_axis:
          legend: Channel
        y_axis:
          - legend: Trip point
            key: trip
            unit: mV
            aggregations:
              - type: max_dev_mv
                unit: mV
                validators:
                  - {operator: "<=", expected_value: 30.0}
              - type: min_dev_mv
                unit: mV
                validators:
                  - {operator: ">=", expected_value: -30.0}
          - legend: Delay
            key: delay
            unit: ms
            aggregations:
              - type: max_ms
                unit: ms
                validators:
                  - {operator: "<=", expected_value: 107.0}

  - name: Overcurrent Injection
    key: overcurrent_injection
    python: phases.overcurrent_injection
    depends_on: [uvp_trip]
    measurements:
      - name: OCD1 Trip
        key: ocd1_trip_mv
        unit: mV
        description: Setting 60 mV; the AFE's OC accuracy is +-5 mV in the 56-100 mV band.
        validators:
          - {operator: ">=", expected_value: 55.0}
          - {operator: "<=", expected_value: 65.0}
      - name: OCD1 Delay
        key: ocd1_delay_ms
        unit: ms
        validators:
          - {operator: "<=", expected_value: 47.0}
      - name: SCD Trip
        key: scd_trip_mv
        unit: mV
        description: Setting 200 mV, +-20 % (+-15 % datasheet plus fixture).
        validators:
          - {operator: ">=", expected_value: 160.0}
          - {operator: "<=", expected_value: 240.0}
      - name: SCD Response
        key: scd_response_us
        unit: µs
        description: "Step to DSG gate off at +25 mV overdrive, 100 MS/s. The 15 us setting plus the input filter; without the stated overdrive the number means nothing."
        validators:
          - {operator: "<=", expected_value: 40.0}
      - name: DSG Gate After SCD
        key: dsg_gate_after_scd_v
        unit: V
        validators:
          - {operator: "<=", expected_value: 1.0}
      - name: Reset Flags After SCD
        key: reset_flags_after_scd
        description: "A device that rebooted on the SRP/SRN transient turns its FETs off too; only the POR and watchdog flags tell a trip from a crash."
        validators:
          - operator: "=="
            expected_value:
              por: false
              wdt: false

  - name: OTP Thermistor
    key: otp_thermistor
    python: phases.otp_thermistor
    depends_on: [overcurrent_injection]
    measurements:
      - name: Temperature At 25 C
        key: temp_at_25c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 23.0}
          - {operator: "<=", expected_value: 27.0}
      - name: Temperature At 60 C
        key: temp_at_60c
        unit: °C
        description: "10 kOhm NTCs all read 25 C at 25 C; a B=3977 reel where B=3435 belongs reads 55 C here and trips OTD late."
        validators:
          - {operator: ">=", expected_value: 57.0}
          - {operator: "<=", expected_value: 63.0}
      - name: OTD Trip
        key: otd_trip_c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 57.0}
          - {operator: "<=", expected_value: 63.0}
      - name: TS Open Flagged
        key: ts_open_flagged
        description: An open thermistor must raise a fault, not report a plausible temperature.
        validators:
          - {operator: "==", expected_value: true}

teardown:
  - name: Restore
    key: restore
    python: phases.restore
    measurements:
      - name: Faults After Restore
        key: faults_after_restore
        validators:
          - operator: "=="
            expected_value:
              ov: []
              uv: []
              ocd1: false
              scd: false
              otd: false
              ts_open: false
      - name: Config CRC After
        key: config_crc_after
        validators:
          - {operator: "==", expected_value: "0x3A7F"}
```

Framework features to notice:

1. **A whole object, compared once.** `protection_config` validates nine registers with one `==`; the report shows which key differed. The same operator validates the fault map after restore and the reset flags after the short.
2. **Two curves, both limited.** `ov_trips` records the trip point and the delay per channel; the trip has `max_dev_mv` and `min_dev_mv` aggregations, the delay has `max_ms`, each with its own limit.
3. **Limits in the datasheet's units.** ±30 mV, +2 delay quanta, ±5 mV absolute, ±20 %: the YAML carries the number and the description carries the datasheet row it came from.
4. **Trip versus crash.** `dsg_gate_after_scd_v` says the FET went off; `reset_flags_after_scd` says the device did not reboot to get there. Both are needed.
5. **Shared code across phases.** `uvp_trip.py` imports `find_trip` from `ovp_trip.py`; phases are plain Python modules.
6. **`teardown:` always runs.** The CRC re-read at the end catches a threshold the test lowered and never restored.

### Config Readback

Milliseconds, no stimulus, and the highest yield of any step in the sequence. The setup phase parks the pack at 3.6 V per cell, reads the protection block decoded to engineering units, records it as one object next to its CRC, and stamps the CRC onto the **unit metadata**:

```python filename="phases/config_readback.py"
from utils.recipe import CELL_COUNT, IDLE_MV


def config_readback(measurements, bench, unit, log):
    """Setup: the protection block read bit-exact before any stimulus. A
    board that left programming with the OTP unwritten fails here in
    milliseconds, with no fixture time spent hunting trip points."""
    bench.set_all_cells_mv(IDLE_MV)
    bench.dut_clear_faults()
    measurements.protection_config = bench.dut_read_config()
    measurements.config_crc = bench.dut_config_crc()
    unit.metadata["config_crc"] = bench.dut_config_crc()
    log.info(f"Board {unit.serial_number}: config CRC {bench.dut_config_crc()}, {CELL_COUNT} cells at {IDLE_MV} mV")
```

### OVP Trip

One channel is raised while the other fifteen sit at 3.6 V; a binary search from a 300 mV window to 1 mV takes seven steps, each followed by a wait longer than the configured delay and a read of the fault register. The ALERT pin on a timer input gives the delay; software polling cannot resolve the 3.3 ms quantum. The search helper is shared with the UV phase:

```python filename="phases/ovp_trip.py"
import numpy as np

from utils.recipe import CELL_COUNT, CONFIG, IDLE_MV, OV_SEARCH, SEARCH_RESOLUTION_MV


def find_trip(bench, channel, lo, hi, rising):
    """Binary search on one channel: seven steps from a 300 mV window to
    1 mV. The other fifteen channels stay at idle the whole time."""
    while hi - lo > SEARCH_RESOLUTION_MV:
        mid = 0.5 * (lo + hi)
        bench.dut_clear_faults()
        bench.set_cell_mv(channel, mid)
        tripped = bench.ov_tripped(channel) if rising else bench.uv_tripped(channel)
        if tripped:
            hi = mid if rising else hi
            lo = lo if rising else mid
        else:
            lo = mid if rising else lo
            hi = hi if rising else mid
    return 0.5 * (lo + hi)


def ovp_trip(measurements, bench, log):
    """Over-voltage trip point and delay on every channel, one channel
    raised at a time while the rest sit at 3.6 V."""
    trip = np.zeros(CELL_COUNT)
    delay = np.zeros(CELL_COUNT)
    for ch in range(1, CELL_COUNT + 1):
        trip[ch - 1] = find_trip(bench, ch, *OV_SEARCH, rising=True)
        delay[ch - 1] = bench.alert_delay_ms("ov")
        bench.set_cell_mv(ch, IDLE_MV)
    bench.dut_clear_faults()
    dev = trip - CONFIG["ov_mv"]

    measurements.ov_trips.x_axis = list(range(1, CELL_COUNT + 1))
    measurements.ov_trips.y_axis.trip = trip.round(1).tolist()
    measurements.ov_trips.y_axis.trip.aggregations.max_dev_mv = float(dev.max())
    measurements.ov_trips.y_axis.trip.aggregations.min_dev_mv = float(dev.min())
    measurements.ov_trips.y_axis.delay = delay.tolist()
    measurements.ov_trips.y_axis.delay.aggregations.max_ms = float(delay.max())
    worst = int(np.abs(dev).argmax()) + 1
    log.info(f"OV trips {trip.min():.0f}..{trip.max():.0f} mV against {CONFIG['ov_mv']}, worst channel {worst} at {dev[worst - 1]:+.0f} mV, delay up to {delay.max():.1f} ms")
```

Per channel is not optional: a swapped pair of sense inputs is invisible when only channel 1 is ramped. On a BQ76952 the OV threshold can also be trimmed in production with `CAL_COV`, applied between the top two cell pins, which corrects one comparator; the sixteen-channel search is still the test.

### UVP Trip

The same search downward, with one rule the mock cannot teach but the article can: lower **one** channel. Dropping all sixteen together trips the stack's shutdown and the station reports a communication timeout instead of a UV result.

```python filename="phases/uvp_trip.py"
import numpy as np

from phases.ovp_trip import find_trip
from utils.recipe import CELL_COUNT, CONFIG, IDLE_MV, UV_SEARCH


def uvp_trip(measurements, bench, log):
    """Under-voltage trip point and delay, one channel lowered at a time.
    Lowering all sixteen together trips the stack shutdown and the station
    reports a comms timeout instead of a UV result."""
    trip = np.zeros(CELL_COUNT)
    delay = np.zeros(CELL_COUNT)
    for ch in range(1, CELL_COUNT + 1):
        trip[ch - 1] = find_trip(bench, ch, *UV_SEARCH, rising=False)
        delay[ch - 1] = bench.alert_delay_ms("uv")
        bench.set_cell_mv(ch, IDLE_MV)
    bench.dut_clear_faults()
    dev = trip - CONFIG["uv_mv"]

    measurements.uv_trips.x_axis = list(range(1, CELL_COUNT + 1))
    measurements.uv_trips.y_axis.trip = trip.round(1).tolist()
    measurements.uv_trips.y_axis.trip.aggregations.max_dev_mv = float(dev.max())
    measurements.uv_trips.y_axis.trip.aggregations.min_dev_mv = float(dev.min())
    measurements.uv_trips.y_axis.delay = delay.tolist()
    measurements.uv_trips.y_axis.delay.aggregations.max_ms = float(delay.max())
    log.info(f"UV trips {trip.min():.0f}..{trip.max():.0f} mV against {CONFIG['uv_mv']}, delay up to {delay.max():.1f} ms")
```

### Overcurrent Injection

A voltage source on the filter side of the shunt sweeps V(SRP) - V(SRN) upward in 1 mV steps until OCD1 trips, then in 5 mV steps until SCD trips. The response time is then measured with a step to the trip point plus 25 mV, digitised on the DSG gate; the overdrive is stated with the result because the same comparator answers in 600 ns at 25 mV and 8 µs at 3 mV, and the input filter sits in front of both. After the short, the gate must be off **and the reset flags must be clear**: a device that browned out on the SRP/SRN transient reboots with its FETs off, which looks like a trip on the gate and is not one.

```python filename="phases/overcurrent_injection.py"
from utils.recipe import OCD_SWEEP_MV, SCD_OVERDRIVE_MV, SCD_SWEEP_MV


def overcurrent_injection(measurements, bench, log):
    """OCD1 and SCD thresholds by voltage injection across SRP-SRN: the
    comparators see a differential voltage and cannot tell a source from a
    shunt. Proves comparator, threshold config, fault path and FET drive;
    proves nothing about the shunt itself."""
    bench.dut_clear_faults()
    ocd_trip = None
    for mv in range(*OCD_SWEEP_MV):
        bench.inject_mv(mv)
        if bench.ocd1_tripped():
            ocd_trip = mv
            break
    ocd_delay = bench.alert_delay_ms("ocd1")
    bench.inject_mv(0)
    bench.dut_clear_faults()

    scd_trip = None
    for mv in range(*SCD_SWEEP_MV):
        bench.inject_mv(mv)
        if bench.scd_tripped():
            scd_trip = mv
            break
    bench.inject_mv(0)
    bench.dut_clear_faults()
    # Response time with a stated overdrive; without the overdrive the
    # number is not comparable to anything.
    bench.inject_mv(scd_trip + SCD_OVERDRIVE_MV)
    scd_us = bench.scd_response_us(SCD_OVERDRIVE_MV)
    gate_v = bench.dsg_gate_v()
    resets = bench.dut_reset_flags()
    bench.inject_mv(0)
    bench.dut_clear_faults()

    measurements.ocd1_trip_mv = float(ocd_trip)
    measurements.ocd1_delay_ms = ocd_delay
    measurements.scd_trip_mv = float(scd_trip)
    measurements.scd_response_us = scd_us
    measurements.dsg_gate_after_scd_v = gate_v
    measurements.reset_flags_after_scd = resets
    log.info(f"OCD1 at {ocd_trip} mV in {ocd_delay:.1f} ms, SCD at {scd_trip} mV, gate off in {scd_us:.1f} us at +{SCD_OVERDRIVE_MV} mV overdrive, resets {resets}")
```

![Three straight lines of equivalent pack current against injected millivolts for 0.5, 1.0 and 2.0 mΩ shunts, the OCD1 trip at 62 mV inside its 55 to 65 mV band and the SCD trip at 210 mV inside its 160 to 240 mV band marked as vertical lines.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-73113770-b23d-499d-9f71-1d46f9fc5bd4-c1QXd9bJiu7Hk0ZIJrFyMoqUR4TquB.png)

What injection proves and what it does not: the comparator trips at 62 and 210 mV, inside both windows. Which current that is depends entirely on the shunt, and the shunt is the one thing this phase never touches. A 1.0 mΩ resistor populated where 0.5 mΩ belongs trips at half the intended current (a nuisance, fails safe); 0.5 mΩ where 1.0 belongs trips at double (fails unsafe). Neither is visible here; a four-wire shunt measurement at ICT, or a real-current test at a temporarily lowered threshold, is the test for that.

### OTP Thermistor

Two resistances, not one. Every 10 kΩ NTC is 10 kΩ at 25 °C, so a room-temperature point cannot tell a B=3435 reel from a B=3977 one; at 60 °C they differ by 21 % in resistance, and the wrong reel, read through the released B model, reports 6 K low and trips over-temperature late. The phase reads the DUT's temperature at both points, then sweeps the resistance down until OTD trips, then opens the thermistor and requires a fault:

```python filename="phases/otp_thermistor.py"
import math

from utils.recipe import CONFIG, NTC_BETA_K, NTC_POINTS_C, NTC_R25_OHM, NTC_SWEEP_OHM


def ntc_ohm(temp_c):
    return NTC_R25_OHM * math.exp(NTC_BETA_K * (1.0 / (temp_c + 273.15) - 1.0 / 298.15))


def otp_thermistor(measurements, bench, log):
    """Over-temperature by resistance substitution on the TS pin, at two
    points: every 10 kOhm NTC reads 25 C at 25 C, only a hot point tells a
    B=3435 reel from a B=3977 one. Then a sweep down to the OTD trip, and
    an open thermistor that must raise a fault, not read a temperature."""
    bench.dut_clear_faults()
    reported = {}
    for t in NTC_POINTS_C:
        bench.set_ts_ohm(ntc_ohm(t))
        reported[t] = bench.dut_read_temperature_c()
        log.info(f"TS at {ntc_ohm(t):.0f} ohm ({t:.0f} C by the B={NTC_BETA_K:.0f} model): DUT reports {reported[t]:.1f} C")
    measurements.temp_at_25c = reported[25.0]
    measurements.temp_at_60c = reported[60.0]

    bench.dut_clear_faults()
    trip_c = None
    ohm = NTC_SWEEP_OHM[0]
    while ohm >= NTC_SWEEP_OHM[1]:
        bench.set_ts_ohm(ohm)
        if bench.otd_tripped():
            trip_c = NTC_BETA_K / (NTC_BETA_K / 298.15 + math.log(ohm / NTC_R25_OHM)) - 273.15
            break
        ohm -= NTC_SWEEP_OHM[2]
    measurements.otd_trip_c = float(trip_c)
    log.info(f"OTD trips at {ohm} ohm = {trip_c:.1f} C against {CONFIG['otd_c']} C")

    bench.dut_clear_faults()
    bench.open_ts()
    measurements.ts_open_flagged = bench.ts_open_flagged()
    bench.set_ts_ohm(NTC_R25_OHM)
    bench.dut_clear_faults()
```

![Two NTC resistance curves against temperature, B=3435 in green and B=3977 in dashed gray, crossing at 10 kΩ and 25 °C and separating to 2.98 versus 2.49 kΩ at 60 °C, the OTD setting at 60 °C as a vertical line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-3805b81c-fcd5-46ce-869c-ce3eec0b48ec-AgMHA8fTAsVgDJYgl3wPXrHKcJwPH9.png)

The two curves meet at the one point a single-point test would use. The Vishay NTCLE100E3103 is B=3977 K, the Murata NCP18XH103F is B=3434 K; both are "10k NTC" on a reel label.

### Restore

The teardown phase switches the injection off, parks the cells, clears the faults and reads the configuration CRC again. Lines that lower a threshold over the service port to test over-current at a modest real current depend on this last read: a threshold left lowered ships a pack whose protection trips in normal use.

```python filename="phases/restore.py"
from utils.recipe import IDLE_MV


def restore(measurements, bench, log):
    """Teardown: stimulus off, cells at idle, faults cleared, and the config
    CRC read again so a threshold the test lowered can never ship lowered."""
    bench.inject_mv(0)
    bench.set_all_cells_mv(IDLE_MV)
    bench.dut_clear_faults()
    measurements.faults_after_restore = bench.dut_read_faults()
    measurements.config_crc_after = bench.dut_config_crc()
    log.info(f"Restored: config CRC {bench.dut_config_crc()}, faults {bench.dut_read_faults()}")
```

### Mock Plug

`ProtectionBench` stands in for four instruments and the DUT because the DUT's fault registers depend on what the bench forces, and plugs run in separate processes. Its OV and UV comparators are drawn with a 7 mV standard deviation around the settings, channel 7's OV comparator is set 18 mV low, the OCD1 and SCD comparators sit 1.4 and 6 mV above their settings, the ALERT delay is the configured value plus a millisecond of latency, the SCD response is the 15 µs setting plus the input filter's response to the stated overdrive, and the thermistor model uses B=3435 K, the released part. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, split the class into the cell simulator over SCPI, the injection channel on an analog output, the programmable resistor over its driver, and the DUT over pyserial with the ALERT pin on a counter input. Keep the overdrive in the SCD result, keep balancing off during every trip test (it modifies the OV and UV timing on the BQ76952), and add a four-wire shunt measurement at ICT for the one defect injection cannot see. The phases, measurements and limits stay the same.


### Joint Encoder Zeroing and Mastering
URL: https://www.tofupilot.com/templates/joint-encoder-zeroing-and-mastering
Master joint encoders against mechanical references to align electronic and mechanical zero positions for accurate forward kinematics on every robot.


## Introduction

### Encoder Mastering Overview

Every robot joint reports its angle through an **encoder**, and every kinematic computation downstream assumes the **reported angle matches the true mechanical angle**. Mastering establishes this agreement: it aligns the **electronic zero** (encoder counts) with the **mechanical zero** (a physical reference: witness pin, alignment block, hardstop). Without correct mastering the transform chain accumulates angular offsets across joints, and on a 6-axis arm an offset on a base joint moves the tool by the whole downstream arm length.

![Tool centre point error caused by an angular offset on each joint alone, for 0.1, 0.02 and 0.002 degrees per joint on a 6-axis arm with 1.3 m of link length: a base joint offset moves the tool by millimetres, a pin-fixture mastering keeps it below 50 micrometres.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-41c43724-c2e1-407e-9595-3839a270473b-nK2NPQDBX0z5yisFjEqR0DxTYwE9MV.png)

Why the fixture matters: an unmastered 0.1° on J1 is over 2 mm at the tool, a witness mark by eye leaves half a millimetre, a pin fixture at 0.002° keeps every joint's contribution below 50 µm.

**Encoder types** found in production robots: optical absolute (Renishaw RESOLUTE, Heidenhain RCN), magnetic absolute off-axis rings (RLS AksIM, the joint-side encoder of most cobots), inductive (Celera IncOder) for dust, oil and EMI immunity, and resolver plus Hall on the motor side in harsh or low-cost designs. Modern joints carry a **dual-encoder architecture**: a motor-side encoder feeding the current loop and a joint-side encoder downstream of the harmonic or cycloidal reducer feeding the position loop. Comparing the two at rest is how this template reads **gearbox lost motion** without any extra instrument.

### Mastering Purpose

Per-joint mastering produces:

- **Encoder zero offset** per joint, written to the controller's mastering data
- **Motor-vs-joint encoder delta at rest**: the gearbox lost-motion signature, checked against each gearbox's own limit
- **Repeatability of the mastering pose**: jog away, back onto the pin, read again
- **Read-back of what was written**, the multi-turn counter state, and the mastering file attached to the run

![Mastering result of the mock arm: encoder offset written per joint in arcseconds with J4 at -291 arcseconds as the largest, motor-vs-joint lost motion per joint against the 0.5 arcminute cycloidal and 1.0 arcminute harmonic limits, and the remaster spread per joint under the 18 arcsecond limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-85da2cd4-f783-449e-b7ab-c98340365a36-DPYdV7V1OeCxNBxVNbZzLbeip0cocn.png)

One run on the mock arm: six offsets of a few arcminutes (J4's encoder sits slightly rotated on its shaft, still far inside the ±0.5° limit), lost motion under each gearbox's limit, and a remaster spread of a few arcseconds that says the fixture, not the operator, defines the zero.

Per-joint mastering catches: **an encoder installed rotated on its shaft, a gearbox cassette swapped from another joint without remastering, a corrupted mastering file, a multi-turn counter that wrapped while the battery was out during shipping**.

Beyond the metrology, this template demonstrates five framework mechanics: the **operator's badge bound to the run's operator field**, a **radio bound to a string measurement**, an **ordered setup stage** so nothing moves before the safety circuit is confirmed, a **multi-dimensional measurement with one custom aggregation per axis**, and a **disabled optional phase** for cells with a touch probe.

## Equipment & Setup

To implement joint mastering on a production line, the following are required:

- **A mechanical reference fixture**: witness pins, a machined alignment block, or a hardstop with shim, one geometry per joint, on a granite base
- **Optional touch probe or dial indicator** for a finishing pass on high-accuracy arms
- **The Device Under Test (DUT)**: a fully assembled arm with all joints connected, in service mode
- **A TofuPilot Framework procedure** to drive the per-joint sequence, capture both encoders, and validate
- **The TofuPilot Dashboard** to log per-unit offsets and trend assembly quality

### Hardware Components

#### Reference Fixture

The production-grade fixture is a **machined alignment block** with witness pins matching the joint geometry: ±5 µm on pin diameter, ±10 µm on hole concentricity, on a granite base plate square to gravity. Each joint has its own geometry, so the station carries a **fixture kit**; the kit in use is recorded on every run, because pin wear shows up as a slow drift of the remaster spread and has to be traceable to the kit.

#### Controller Service Interface

Mastering uses the controller's own service routine: **UR "Joint Zeroing"**, **KUKA "Mastering"**, **ABB "Calibration"**, **FANUC "Quick Master"**. The procedure drives it through the vendor API (UR dashboard and RTDE, KUKA Sunrise, ABB RobotWare, FANUC KAREL) and reads both encoders through the same interface. Cobots that offer a fixture-free gravity-based zeroing still benefit from the pin for the repeatability check.

### Custom Firmware

None. The controller runs production firmware in service mode; the procedure only needs brake release, jog, encoder read and offset write, which every vendor exposes.

## Test Procedure

### Overview

With the arm in service mode and the fixture kit on the bench, the procedure runs:

1. Setup: the operator scans their badge and picks the fixture kit; the safety circuit is checked.
2. For each joint: brake off, jog onto the witness pin, read the joint encoder and the motor encoder, write the offset, brake on.
3. Repeat the pin approach on every joint and compare: the procedure's repeatability.
4. Optional touch-probe finishing pass (disabled by default).
5. Read every offset back, confirm the multi-turn counters, attach the mastering file, mark the unit as mastered.
6. Teardown: every brake engaged before the fixture comes off.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
ui.json
phases/
  interlocks.py
  master_joints.py
  repeatability.py
  touch_probe.py
  verify_written.py
  brakes_engaged.py
plugs/
  robot.py
utils/
  joints.py
README.md
pyproject.toml
```

The controller plug is a mock six-joint arm with a real offset per joint and a lost motion per gearbox. Run it with `tofupilot run . --no-tui --no-kiosk --json --ui-values ui.json`, or drop `--ui-values` to scan a badge and pick the kit yourself. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-joint-encoder-mastering).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: Joint Encoder Zeroing and Mastering
version: 0.1.0
description: Masters the six joint encoders of an assembled arm against the witness-pin fixture, checks the motor-vs-joint encoder delta per gearbox, measures the procedure's own repeatability, and attaches the mastering file.

unit:
  auto_identify: true
  serial_number:
    description: "Robot serial from the base plate"
    placeholder: "RB-0000000"
    pattern: "^RB-\\d{7}$"
    default_value: "RB-2026041"
  part_number:
    default_value: "COBOT-5KG-R2"
  metadata:
    mastered:
      description: "Set by the station once the offsets are written and read back"

operated_by: {}

plugs:
  - name: Robot Controller
    description: Service interface of the controller under test (mock six-joint arm)
    python: plugs.robot:RobotController
    key: robot

setup:
  - name: Identify Operator
    key: identify_operator
    ui:
      components:
        - key: badge
          type: text_input
          label: "Scan your badge"
          required: true
          bind: run.operated_by
        - key: fixture_kit
          type: radio
          label: "Fixture kit on the bench"
          required: true
          bind: measurements.fixture_kit
          options:
            - label: "Kit A (serial FIX-A-012)"
              value: "kit_a"
            - label: "Kit B (serial FIX-B-007)"
              value: "kit_b"
    measurements:
      - name: Fixture Kit
        key: fixture_kit
        validators:
          - {operator: in, expected_value: ["kit_a", "kit_b"]}

  - name: Interlocks
    key: interlocks
    python: phases.interlocks
    depends_on: [identify_operator]
    measurements:
      - name: E-Stop Healthy
        key: estop_healthy
        validators:
          - {operator: "==", expected_value: true}

main:
  - name: Master Joints
    key: master_joints
    python: phases.master_joints
    ui:
      components:
        - key: mastering_progress
          type: progress
          label: "Mastering J1 to J6"
          default_value: 0
          max: 100
    measurements:
      - name: Mastering
        key: mastering
        title: Encoder offset and motor-vs-joint delta per joint
        x_axis: {legend: Joint}
        y_axis:
          - legend: Encoder offset
            key: offset
            unit: arcsec
            aggregations:
              - type: max_abs_deg
                unit: deg
                validators:
                  - {operator: "<=", expected_value: 0.5}
          - legend: Motor-vs-joint delta
            key: delta
            unit: arcmin
            aggregations:
              - type: worst_vs_limit
                validators:
                  - {operator: "<=", expected_value: 1.0}
      - name: Offsets
        key: offsets
        description: Encoder offset per joint in arcseconds, the fingerprint of this arm's assembly.

  - name: Repeatability
    key: repeatability
    python: phases.repeatability
    depends_on: [master_joints]
    measurements:
      - name: Remaster Spread
        key: remaster_spread
      - name: Remaster Worst
        key: remaster_worst
        unit: deg
        description: Largest difference between two successive masterings of the same joint.
        validators:
          - {operator: "<=", expected_value: 0.005}

  - name: Touch Probe Finish
    key: touch_probe
    python: phases.touch_probe
    enabled: false
    depends_on: [repeatability]
    measurements:
      - name: Probe Residual
        key: probe_residual
        unit: deg
        validators:
          - {operator: "<=", expected_value: 0.001}

  - name: Verify Written
    key: verify_written
    python: phases.verify_written
    depends_on: [repeatability]
    measurements:
      - name: Offsets Written
        key: offsets_written
        validators:
          - {operator: "==", expected_value: true}
      - name: Multi-Turn Valid
        key: multiturn_valid
        validators:
          - {operator: "==", expected_value: true}

teardown:
  - name: Brakes Engaged
    key: brakes_engaged
    python: phases.brakes_engaged
    measurements:
      - name: All Brakes Engaged
        key: all_brakes_engaged
        validators:
          - {operator: "==", expected_value: true}
```

Framework features to notice:

1. **Operator on the run**: `operated_by: {}` at the root declares the field; the badge text input carries `bind: run.operated_by`, so the scan becomes the run's operator with no Python, and later phases read it as `run.operated_by`.
2. **Ordered setup**: setup phases with satisfied dependencies run concurrently like main phases. `interlocks` depends on `identify_operator` so the badge is on record before the safety check logs it, and both must pass before any brake releases.
3. **One aggregation per axis**: the `mastering` series carries the offset and the delta per joint; `max_abs_deg` validates the offsets against the ±0.5° installation limit, and `worst_vs_limit` is the largest delta divided by that joint's own gearbox limit, so one validator covers two gearbox types.
4. **Disabled phase**: `touch_probe` ships with `enabled: false` and is skipped silently; a cell with a touch probe flips it on without touching code.
5. **JSON tables**: `offsets` and `remaster_spread` are recorded as per-joint objects, so the exact numbers are on the run without six measurements each.

### Master Joints

Per joint: brake off, jog until the witness pin engages, read the joint-side encoder (the offset to write) and the motor-side encoder divided by the gear ratio (the lost motion shows in the difference), write, brake on. The limit for the delta comes from the joint table, per gearbox:

```python filename="phases/master_joints.py"
from utils.joints import ARCSEC_PER_DEG, JOINTS


def master_joints(measurements, robot, ui, log):
    """Per joint: brake off, jog to the witness pin, read both encoders, write the offset, brake on."""
    offsets_arcsec, deltas_arcmin, worst_ratio = [], [], 0.0
    for i, joint in enumerate(JOINTS):
        name = joint["name"]
        robot.release_brake(name)
        robot.jog_to_reference(name)
        joint_deg = robot.raw_joint_encoder_deg(name)
        motor_deg = robot.motor_encoder_deg(name)
        robot.write_offset(name, joint_deg)
        robot.engage_brake(name)

        offset_arcsec = joint_deg * ARCSEC_PER_DEG
        delta_arcmin = abs(motor_deg - joint_deg) * 60.0
        offsets_arcsec.append(offset_arcsec)
        deltas_arcmin.append(delta_arcmin)
        worst_ratio = max(worst_ratio, delta_arcmin / joint["delta_limit_arcmin"])
        log.info(f"{name}: offset {offset_arcsec:+.0f}\" ({joint_deg:+.4f} deg), motor-vs-joint {delta_arcmin:.2f}' (limit {joint['delta_limit_arcmin']}')")
        ui.mastering_progress = int(100 * (i + 1) / len(JOINTS))

    measurements.mastering.x_axis = list(range(1, len(JOINTS) + 1))
    measurements.mastering.y_axis.offset = offsets_arcsec
    measurements.mastering.y_axis.delta = deltas_arcmin
    measurements.mastering.y_axis.offset.aggregations.max_abs_deg = max(abs(o) for o in offsets_arcsec) / ARCSEC_PER_DEG
    measurements.mastering.y_axis.delta.aggregations.worst_vs_limit = worst_ratio
    measurements.offsets = {j["name"]: round(o, 1) for j, o in zip(JOINTS, offsets_arcsec)}
```

The joint table pairs each joint with its gearbox and the lost-motion limit that goes with it:

```python filename="utils/joints.py"
JOINTS = [
    {"name": "J1", "gearbox": "cycloidal", "ratio": 121, "delta_limit_arcmin": 0.5},
    {"name": "J2", "gearbox": "cycloidal", "ratio": 121, "delta_limit_arcmin": 0.5},
    {"name": "J3", "gearbox": "harmonic", "ratio": 101, "delta_limit_arcmin": 1.0},
    {"name": "J4", "gearbox": "harmonic", "ratio": 101, "delta_limit_arcmin": 1.0},
    {"name": "J5", "gearbox": "harmonic", "ratio": 101, "delta_limit_arcmin": 1.0},
    {"name": "J6", "gearbox": "harmonic", "ratio": 101, "delta_limit_arcmin": 1.0},
]

ARCSEC_PER_DEG = 3600.0
```

### Repeatability

The joints are driven off the pin and back, and the encoder is read again. The difference with the offset just written is the repeatability of the mastering itself; a pin fixture holds it to a few arcseconds, a witness mark by eye to a few arcminutes. The first offsets arrive through **previous-results injection**:

```python filename="phases/repeatability.py"
from utils.joints import ARCSEC_PER_DEG, JOINTS


def repeatability(measurements, robot, master_joints, log):
    """Jog away, come back onto the pin, read again: the spread is the procedure's own repeatability."""
    first = master_joints.offsets
    diffs = []
    for joint in JOINTS:
        name = joint["name"]
        robot.release_brake(name)
        robot.jog_to_reference(name)
        second = robot.raw_joint_encoder_deg(name) * ARCSEC_PER_DEG
        robot.engage_brake(name)
        diffs.append(abs(second - float(first[name])))
    worst = max(diffs)
    log.info(f"Remaster spread per joint: {[round(d, 1) for d in diffs]} arcsec, worst {worst:.1f}\"")
    measurements.remaster_spread = {j["name"]: round(d, 1) for j, d in zip(JOINTS, diffs)}
    measurements.remaster_worst = worst / ARCSEC_PER_DEG
```

### Verify Written

Every offset is read back from the controller, the multi-turn counters are confirmed, the mastering file is attached to the run, and the unit is marked as mastered in its metadata so downstream stations (kinematic calibration, ISO 9283) can check for it:

```python filename="phases/verify_written.py"
import json

from utils.joints import JOINTS


def verify_written(measurements, robot, unit, attach, master_joints, log):
    """Read every offset back from the controller and attach the mastering file to the run."""
    readback = {j["name"]: robot.read_offset(j["name"]) for j in JOINTS}
    measurements.offsets_written = all(v is not None for v in readback.values())
    measurements.multiturn_valid = all(robot.multiturn_valid(j["name"]) for j in JOINTS)
    mastering_file = {"serial_number": unit.serial_number, "offsets_deg": readback, "offsets_arcsec": master_joints.offsets}
    attach.data(json.dumps(mastering_file, indent=2).encode(), f"{unit.serial_number}_mastering.json")
    unit.metadata["mastered"] = True
    log.info("Offsets read back and mastering file attached")
```

### Mock Plug

The controller mock carries one true offset per joint (J4's encoder is seated 0.08° rotated), a lost motion per gearbox drawn around a quarter of an arcminute, and a sub-arcsecond pin clearance. Its method surface is what a wrapper around the vendor service API has to provide:

```python filename="plugs/robot.py"
import numpy as np

from utils.joints import ARCSEC_PER_DEG, JOINTS


class RobotController:
    def __init__(self):
        self._rng = np.random.default_rng(6)
        # true encoder-zero to mechanical-zero offset per joint, degrees
        self._true_offset = {j["name"]: self._rng.normal(0.0, 0.05) for j in JOINTS}
        self._true_offset["J4"] = -0.08   # encoder seated slightly rotated on J4
        self._lost_motion = {j["name"]: abs(self._rng.normal(0.25, 0.08)) for j in JOINTS}  # arcmin
        self._brakes = {j["name"]: True for j in JOINTS}
        self._offset_written = {}
        print("Controller service mode, all brakes engaged")

    def raw_joint_encoder_deg(self, joint):
        """Joint-side encoder reading at the mechanical reference: the offset plus pin clearance."""
        return self._true_offset[joint] + self._rng.normal(0.0, 0.0008)

    def motor_encoder_deg(self, joint):
        """Motor-side reading divided by the gear ratio, at the same reference."""
        ratio = next(j["ratio"] for j in JOINTS if j["name"] == joint)
        lost = self._lost_motion[joint] / 60.0 * self._rng.choice([-1.0, 1.0])
        return self._true_offset[joint] + lost + self._rng.normal(0.0, 0.0005 / ratio)

    def write_offset(self, joint, offset_deg):
        self._offset_written[joint] = float(offset_deg)
        return True
```

On a real cell, `plugs/robot.py` maps to the vendor service API; the phases, measurements, and limits stay exactly the same.


### BMS Cell Simulator Functional Test
URL: https://www.tofupilot.com/templates/bms-cell-simulator-functional-test
Functional test of a 16S BMS PCBA on a battery cell simulator: identity readback, per-channel voltage accuracy, open-wire detection, sense tap resistance.


## Introduction

### BMS PCBA Test Overview

A battery management system board only does anything when it believes it is wired to a pack. Every cell-sense input expects a voltage in the 2 to 4.5 V window, stacked on top of the channel below it, so the top of a 16S board sits at 60 V above the bottom. A production functional test therefore starts with a **battery cell simulator**: one isolated, programmable channel per cell input, each able to source and sink current, stacked in series to reproduce the pack's common-mode ladder. With the pack replaced by a simulator, the tester reaches states a real pack might not reach in years (one cell at over-voltage next to one at under-voltage, an open sense wire, every channel at a distinct voltage) in seconds, and without a lithium cell inside the fixture.

![A 16S BMS PCBA on a fixture plate: the black AFE chip in the middle, a row of sixteen bleed resistors along one edge, the white cell-sense connector with a short ribbon harness plugged in.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-8324ed75-d8be-45d0-b970-1acbb3a7b1b2-PZ34k87VsTEQwhFqjztddd7FaELA5g.jpg)

A 16S BMS PCBA in the fixture: the cell-sense connector carries sixteen taps plus the stack return, the analog front end reads them, and the row of bleed resistors next to it is the balancing path this template borrows to measure each tap's resistance.

The analog front end (AFE) that reads the cells is specified to a few millivolts. The **TI BQ79616** main ADC, for instance, is guaranteed to -2.2/+1.5 mV at 25 °C and -3.0/+2.4 mV from -20 to 65 °C for cells between 2 and 4.5 V (see the [BQ79616 datasheet](https://www.ti.com/lit/ds/symlink/bq79616.pdf), VACC_MAIN_CELL). The **ADI ADBMS6815** claims better than 1.5 mV lifetime total measurement error. Everything downstream of the AFE, the divider or filter resistors on each input, the sense harness, the connector, the solder joints, the calibration constants written at the end of the line, is the board assembler's responsibility, and this is what the functional test proves.

### Test Purpose

The procedure records one functional fingerprint per board:

- **Identity and configuration**: firmware version, AFE part and cell count, CRC of the protection configuration block, all read over the service port
- **Cell voltage accuracy** on a monotonic staircase, per channel, against the simulator's own readback
- **Channel order**: a crossed pair in the sense harness breaks the staircase
- **Gain error and offset** per channel from a three-point sweep
- **Open-wire detection**: one channel opened, the DUT must flag the wire, and the fault map must be clean once reconnected
- **Sense tap resistance** per channel, measured through the bleed path
- **Ship-mode current** of the whole stack on the simulator's 250 µA range

![Left, the staircase: simulator readback in gray steps from 3.20 V on channel 1 to 4.10 V on channel 16, DUT readings as green dots on every step. Right, the reported-minus-readback error per channel as bars between the -4.0 and +3.5 mV limit lines, channel 12 highest at +2.9 mV.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-5f48e920-f494-4958-8b0f-b7005c3aca75-qlZPb2g0GxkNuxRCFt904a1WjOGCoA.png)

The mock board on the staircase: every channel reads its own step, the order is intact, and the error per channel stays inside the asymmetric -4.0/+3.5 mV window with channel 12 closest to the limit at +2.9 mV.

The framework mechanics on show are the `setup` and `teardown` stages, string and JSON validators (`matches` on the firmware version, `==` on a whole identity object), multi-dimensional measurements with custom aggregations validated in YAML, asymmetric limits, a sequential `depends_on` chain on one shared bench, and a plug `config` block that becomes constructor arguments.

## Equipment & Setup

To run this functional test on a production line, the following are required:

- **A battery cell simulator** with one isolated channel per cell input, four-quadrant, four-wire sense
- **A service link to the DUT** (UART, I²C or CAN, whatever the board exposes for calibration)
- **A pogo-pin fixture** with Kelvin contacts on every cell-sense point
- **The Device Under Test (DUT)**: a 16S BMS PCBA with its production firmware
- **A TofuPilot Framework procedure** to script the sequence, log the per-channel data and validate the limits
- **The TofuPilot Dashboard** to store the per-board results for traceability and per-channel trending

### Hardware Components

#### Cell Simulator

The [Chroma 87001](https://www.chromausa.com/pdf/87001%20Battery%20Cell%20Simulator%20052023.pdf) is the reference for this class: 16 channels, 0 to 5 V, bidirectional up to 5 A, setting accuracy ±0.02 % of full scale (±1 mV) with 0.5 mV setting resolution, **readback accuracy ±0.02 % of full scale with 0.1 mV resolution**, 1000 V channel-to-channel and channel-to-ground isolation, and units stackable to 240S2P below 1000 V. Two of its current ranges are designed for this exact test: **0 to 500 mA at ±100 µA** for passive balancing current, and **0 to 250 µA** for the quiescent draw of the cell-supervision circuit. The [Keysight SL1010A-803](https://www.keysight.com/us/en/products/hev-ev-grid-emulators-and-test-systems/sl1010a-series-scienlab-bms-environment.html) goes further on accuracy (±0.2 mV offset + 0.01 % of value, and it states that setting and measuring accuracy are identical) at ±5 A per channel. The [dSPACE EV1077](https://www.dspace.com/en/pub/home/products/special_solutions/battery_cell_voltage_emulation.cfm) holds ±0.5 mV over its whole working temperature range but only isolates 60 V between the cells of one module, so a 16S stack spans several modules.

Two instruments sold for BMS test sit five to seven times looser: the [Pickering 41-752A](https://www.pickeringtest.com/portal/web/256/content/downloads/datasheets/41-43-752AD.pdf) at ±5 mV setting, ±7 mV typical readback, and only over 11 to 31 °C, and the [Bloomy BS-1200](https://www.bloomy.com/products/battery-management-system-testing/battery-simulator-1200) at ±3 mV with remote sense required. A ±3 mV simulator cannot police a ±3 mV limit. The limits in this template assume a Chroma or Keysight class instrument; with a Pickering or Bloomy, widen the windows and say so in the procedure.

![Battery management board in a fixture with individual cell-sense contacts](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/bms-daisy-chain-signal-integrity-HgOfxGm9NqCjWsGejC74dvTHayMBTN.webp)

The fixture supports the BMS board and its individual cell-sense connections. The cell simulator and test computer connect externally and are not shown.

#### Fixture and Kelvin Sense

Every cell-sense point needs a four-wire contact. Two-wire probing folds pogo-pin and harness resistance into the forced voltage; at the tens of milliohms typical of a worn pin and the milliamps an AFE input draws that is small, but during the tap-resistance phase 80 mA flows through the same path and the error becomes the measurement. NI's production BMS test brief states the achieved figure plainly: four-wire Kelvin contacts on the cell voltage sense points are what get the fixture to sub-millivolt accuracy. Keep the simulator's own readback, taken at the sense pins, as the truth; the setpoint is what you asked for, the readback is what the DUT actually saw.

**Never build the cell ladder from a resistor divider across one supply.** TI devotes a section of [SLUAA81A](https://www.ti.com/lit/an/sluaa81a/sluaa81a.pdf) to why: the moment one channel balances, its bleed current collapses the divider, the neighbouring inputs move, the over-voltage protection trips, balancing is disabled, and on some parts a permanent-fail bit is set. That is a scrapped board, not a failed test.

### Where the Limits Come From

No standard specifies a BMS PCBA production test. IEC 62619, IEC 62133, UL 2054 and UL 1973 test cells, batteries and battery systems by outcome; ISO 26262-7 mandates that a safety-related characteristic is controlled in production and that the evidence is kept, but it does not hand out a millivolt. The limits therefore come from the AFE datasheet, widened by the simulator's accuracy. The staircase window below is the BQ79616's -3.0/+2.4 mV row for -20 to 65 °C plus ±1 mV of simulator, rounded to **-4.0/+3.5 mV**. It is asymmetric because the datasheet is asymmetric; a symmetric ±3.5 mV window would throw away yield on the negative side for no reason. The tap resistance limit of 150 mΩ and the ship current limit of 250 µA are internal specifications of the mock board and are the kind of number a customer sets from their own harness and AFE, not from a standard.

## Test Procedure

### Overview

The procedure maps the sequence onto the framework's three stages. Identity lives in `setup:` so no accuracy number is recorded against a board running the wrong firmware. Parking the board lives in `teardown:` so it runs even when a main phase fails and no board is left with a bleed path on.

1. Setup: pack to 3.6 V/cell, identity block, config CRC.
2. Main: staircase, per-channel error, channel order.
3. Main: three-point sweep, gain error and offset per channel.
4. Main: one channel opened, open-wire flag, fault map after reconnect.
5. Main: bleed current through each tap, series resistance per channel.
6. Teardown: ship mode, quiescent draw, outputs off.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for traceability and per-channel trending.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify_dut.py
  cell_voltage_accuracy.py
  gain_offset_sweep.py
  open_wire_check.py
  tap_resistance.py
  shutdown.py
plugs/
  bms_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-bms-cell-simulator-fct). The `BmsBench` plug is a mock of the simulator and the DUT's service port together, synthesizing a healthy 16S board with one channel near the offset limit and one sense tap with a marginal crimp, so the procedure runs end-to-end without a simulator or a board connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit (scanned data-matrix with a pattern), the bench plug with its **config** passed as constructor arguments, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: BMS Cell Simulator Functional Test
version: 0.1.0
description: Functional test of a 16S BMS PCBA on a battery cell simulator. Identity and config readback, per-channel voltage accuracy on a staircase, gain and offset from a three-point sweep, open-wire detection, sense tap resistance, ship-mode current.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the board data-matrix before it goes into the fixture"
    placeholder: "BMS16-00000"
    pattern: "^BMS16-\\d{5}$"
    default_value: "BMS16-01742"
  part_number:
    default_value: "PCB-BMS-16S-R3"
  batch_number:
    default_value: "LOT-2026-37"

plugs:
  - name: BMS Bench
    description: "16-channel cell simulator and DUT UART link (mock Chroma 87001-class, one plug per bench)"
    python: plugs.bms_bench:BmsBench
    key: bench
    config:
      cell_count: 16
      bleed_ma: 80

setup:
  - name: Identify DUT
    key: identify_dut
    python: phases.identify_dut
    measurements:
      - name: Firmware Version
        key: firmware_version
        description: Production firmware line is 2.x; a 1.x board was flashed from the wrong release.
        validators:
          - {operator: matches, expected_value: "^2\\.\\d+\\.\\d+$"}
      - name: AFE Identity
        key: afe_identity
        description: AFE part and cell count as one object; a 3S variant of the same PCB must fail here.
        validators:
          - operator: "=="
            expected_value:
              afe: BQ76952
              cell_count: 16
      - name: Config CRC
        key: config_crc
        description: CRC of the protection configuration block, must match the released config.
        validators:
          - {operator: "==", expected_value: "0x3A7F"}

main:
  - name: Cell Voltage Accuracy
    key: cell_voltage_accuracy
    python: phases.cell_voltage_accuracy
    measurements:
      - name: Staircase
        key: staircase
        title: Reported vs simulator readback per channel
        x_axis:
          legend: Channel
        y_axis:
          - legend: Forced
            key: forced
            unit: mV
          - legend: Reported
            key: reported
            unit: mV
          - legend: Error
            key: error
            unit: mV
            aggregations:
              - type: max_mv
                unit: mV
                validators:
                  - {operator: "<=", expected_value: 3.5}
              - type: min_mv
                unit: mV
                validators:
                  - {operator: ">=", expected_value: -4.0}
      - name: Channel Order
        key: order_ok
        description: Readback rises with channel index; a crossed pair in the sense harness breaks it.
        validators:
          - {operator: "==", expected_value: true}

  - name: Gain Offset Sweep
    key: gain_offset_sweep
    python: phases.gain_offset_sweep
    depends_on: [cell_voltage_accuracy]
    measurements:
      - name: Linearity
        key: linearity
        title: Gain error and offset per channel from a 3-point sweep
        x_axis:
          legend: Channel
        y_axis:
          - legend: Gain error
            key: gain_err
            unit: "%"
            aggregations:
              - type: max_abs_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 0.10}
          - legend: Offset
            key: offset
            unit: mV
            aggregations:
              - type: max_abs_mv
                unit: mV
                validators:
                  - {operator: "<=", expected_value: 3.0}

  - name: Open Wire Check
    key: open_wire_check
    python: phases.open_wire_check
    depends_on: [gain_offset_sweep]
    measurements:
      - name: Open Wire Flagged
        key: open_wire_flagged
        description: The DUT must raise the open-wire fault for the opened channel, not report a plausible voltage.
        validators:
          - {operator: "==", expected_value: true}
      - name: Open Wire Reading
        key: open_wire_reading_mv
        unit: mV
        description: What the AFE reads on the floating input, recorded for trending, no limit.
      - name: Faults After Reconnect
        key: faults_after_reconnect
        validators:
          - operator: "=="
            expected_value:
              open_wire: []
              ov: []
              uv: []

  - name: Tap Resistance
    key: tap_resistance
    python: phases.tap_resistance
    depends_on: [open_wire_check]
    timeout: 60s
    measurements:
      - name: Tap Path
        key: tap_path
        title: Sense tap series resistance per channel
        x_axis:
          legend: Channel
        y_axis:
          - legend: Resistance
            key: resistance
            unit: mΩ
            aggregations:
              - type: max_mohm
                unit: mΩ
                validators:
                  - {operator: "<=", expected_value: 150.0}

teardown:
  - name: Shutdown
    key: shutdown
    python: phases.shutdown
    measurements:
      - name: Ship Current
        key: ship_current_ua
        unit: µA
        description: Stack draw in ship mode on the simulator's 250 uA range; a stuck balancing FET or a wrong pull-up shows here.
        validators:
          - {operator: "<=", expected_value: 250.0}
```

Framework features to notice:

1. **String and JSON validators.** `firmware_version` validates with `matches` against a regex, `config_crc` with `==` on a string, and `afe_identity` with `==` on a whole object, so a 3S variant of the same PCB fails at setup with the reason in the report.
2. **Aggregations validated in YAML.** The staircase records three curves; the pass/fail lives on two aggregations of the error curve, `max_mv` and `min_mv`, computed in Python and named in the procedure file with their own limits.
3. **Asymmetric limits.** `<= 3.5` and `>= -4.0` on the same aggregation pair, copied from the AFE datasheet row rather than a single ± number.
4. **One bench, one chain.** Every main phase declares `depends_on` on the previous one because they share the simulator; without the chain the framework would run them concurrently and the readings would interleave.
5. **`teardown:` always runs.** The board ends in ship mode and the outputs go off whether the main phases passed or not.

### Identify DUT

The setup phase brings the simulated pack to 3.6 V per cell, reads the identity block over the service port, stamps the AFE die revision onto the **unit metadata**, and records the three identity measurements that gate the rest of the run:

```python filename="phases/identify_dut.py"
from utils.recipe import CELL_COUNT, IDLE_MV


def identify_dut(measurements, bench, unit, log):
    """Setup: bring the simulated pack to 3.6 V/cell and read the DUT's
    identity block. Every main phase talks to the board this phase proved alive."""
    bench.force_cells([IDLE_MV] * CELL_COUNT)
    ident = bench.dut_identify()
    log.info(f"DUT {unit.serial_number}: fw {ident['firmware']}, AFE {ident['afe']} rev {ident['afe_die_rev']}")
    unit.metadata["afe_die_rev"] = ident["afe_die_rev"]

    measurements.firmware_version = ident["firmware"]
    measurements.afe_identity = {"afe": ident["afe"], "cell_count": ident["cell_count"]}
    measurements.config_crc = ident["config_crc"]
```

Reading the config CRC is the cheapest, highest-yield check in the whole sequence: milliseconds, no stimulus, and it catches a board that left programming with the protection block unwritten.

### Cell Voltage Accuracy

The simulator forces a **distinct voltage on every channel**, 3.20 V on channel 1 rising 60 mV per channel to 4.10 V on channel 16. With all channels at the same 3.700 V, a crossed pair in the sense harness reads perfectly; on the staircase it breaks the order. The DUT's reading is compared against the simulator's readback, not its setpoint, and the three curves land in one multi-dimensional measurement with the limits on the error's aggregations:

```python filename="phases/cell_voltage_accuracy.py"
import numpy as np

from utils.recipe import STAIRCASE_MV


def cell_voltage_accuracy(measurements, bench, log):
    """Force a distinct voltage on every channel, compare the DUT's reading
    against the simulator's readback, and check the readback is monotonic."""
    bench.force_cells(STAIRCASE_MV)
    truth = np.array(bench.sim_readback_mv())
    reported = np.array(bench.dut_read_cells_mv())
    error = reported - truth

    channels = list(range(1, len(truth) + 1))
    measurements.staircase.x_axis = channels
    measurements.staircase.y_axis.forced = truth.tolist()
    measurements.staircase.y_axis.reported = reported.tolist()
    measurements.staircase.y_axis.error = error.round(2).tolist()
    measurements.staircase.y_axis.error.aggregations.max_mv = float(error.max())
    measurements.staircase.y_axis.error.aggregations.min_mv = float(error.min())

    # A crossed pair in the sense harness breaks the staircase order.
    order_ok = bool(np.all(np.diff(reported) > 0))
    measurements.order_ok = order_ok
    worst = int(np.abs(error).argmax()) + 1
    log.info(f"Error {error.min():+.2f} .. {error.max():+.2f} mV, worst channel {worst}, order {'ok' if order_ok else 'BROKEN'}")
```

### Gain Offset Sweep

Three points across the cell range, 3.0, 3.6 and 4.2 V on every channel, eight conversions averaged per point on both sides of the fixture. Gain error comes from the end points and offset from the mean residual, one number each per channel; fitting a line and extrapolating the intercept to 0 V amplifies the noise by the ratio of the operating point to the span and produces offsets that are not there:

```python filename="phases/gain_offset_sweep.py"
import numpy as np

from utils.recipe import CELL_COUNT, SWEEP_MV

AVERAGES = 8  # conversions averaged per point, both sides of the fixture


def gain_offset_sweep(measurements, bench, log):
    """Three points across the cell range on every channel. Gain error comes
    from the end points, offset from the mean residual, so one number each
    per channel instead of a fit extrapolated to 0 V."""
    truth = []
    reported = []
    for mv in SWEEP_MV:
        bench.force_cells([mv] * CELL_COUNT)
        truth.append(np.mean([bench.sim_readback_mv() for _ in range(AVERAGES)], axis=0))
        reported.append(np.mean([bench.dut_read_cells_mv() for _ in range(AVERAGES)], axis=0))
    truth = np.array(truth)  # (points, channels)
    reported = np.array(reported)

    gain_err_pct = ((reported[-1] - reported[0]) / (truth[-1] - truth[0]) - 1.0) * 100.0
    offset_mv = (reported - truth).mean(axis=0)

    measurements.linearity.x_axis = list(range(1, CELL_COUNT + 1))
    measurements.linearity.y_axis.gain_err = gain_err_pct.round(4).tolist()
    measurements.linearity.y_axis.gain_err.aggregations.max_abs_pct = float(np.abs(gain_err_pct).max())
    measurements.linearity.y_axis.offset = offset_mv.round(2).tolist()
    measurements.linearity.y_axis.offset.aggregations.max_abs_mv = float(np.abs(offset_mv).max())
    log.info(f"Gain error up to {np.abs(gain_err_pct).max():.3f} %, offset up to {np.abs(offset_mv).max():.2f} mV")
```

![Left, gain error per channel as bars within ±0.10 %, all below 0.07 %. Right, offset per channel within ±3.0 mV, channel 12 highest at 2.6 mV.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-f1312238-4ad5-45dc-ac36-0c5a7b8e695e-EyTtGdYu1jYUajc74BLoymWnVmycVu.png)

The sweep separates the two error terms: gain error is flat across the board (a wrong divider resistor on one input would stand out here), and the offset on channel 12 is the same 2 mV the staircase showed, now attributed to offset rather than gain.

### Open Wire Check

One simulator channel is opened. The DUT must raise the open-wire fault for that channel; what it reads on the floating input is recorded but not limited, because it floats toward the neighbours' midpoint and looks plausible, which is the point. Once reconnected, the fault map is validated as one object against an all-empty expectation:

```python filename="phases/open_wire_check.py"
from utils.recipe import CELL_COUNT, IDLE_MV, OPEN_WIRE_CHANNEL


def open_wire_check(measurements, bench, log):
    """Open one simulator channel and require the DUT to flag the wire, not
    report a plausible voltage. Then reconnect and require a clean fault map."""
    bench.force_cells([IDLE_MV] * CELL_COUNT)
    bench.open_channel(OPEN_WIRE_CHANNEL)
    reported = bench.dut_read_cells_mv()
    faults = bench.dut_read_faults()
    flagged = OPEN_WIRE_CHANNEL in faults["open_wire"]
    log.info(f"Channel {OPEN_WIRE_CHANNEL} open: DUT reads {reported[OPEN_WIRE_CHANNEL - 1]:.1f} mV, "
             f"open-wire flags {faults['open_wire']}")
    measurements.open_wire_flagged = flagged
    measurements.open_wire_reading_mv = float(reported[OPEN_WIRE_CHANNEL - 1])

    bench.close_channel(OPEN_WIRE_CHANNEL)
    measurements.faults_after_reconnect = bench.dut_read_faults()
```

On the BQ76952 the open-wire check sources 22 to 95 µA into the input under test and can register as a transient OV or UV on that cell, so the phase runs after the accuracy phases and before the fault map is read again.

### Tap Resistance

The one measurement in this template that is not in the usual BMS FCT list. The bleed path of each channel is switched on one at a time; on an external-balancing board the bleed current flows through the same harness wire, connector pin and solder joint the sense input hangs on. The voltage the AFE sees drops by I times R while the simulator's Kelvin readback at the fixture does not move, so the difference divided by the current the simulator sources on its 500 mA range is the **series resistance of that tap path**. It is the closest thing to a production test for a cold solder joint on a sense line, and it costs no extra instrument:

```python filename="phases/tap_resistance.py"
import numpy as np

from utils.recipe import CELL_COUNT, IDLE_MV


def tap_resistance(measurements, bench, log):
    """Series resistance of every sense tap: bleed current through the tap
    path drops the voltage the AFE sees, the simulator's Kelvin sense does
    not move. R = dV / I_bleed, one channel at a time."""
    bench.force_cells([IDLE_MV] * CELL_COUNT)
    idle = np.array(bench.dut_read_cells_mv())
    r_mohm = np.zeros(CELL_COUNT)
    for ch in range(1, CELL_COUNT + 1):
        bench.dut_balance(ch, True)
        loaded = bench.dut_read_cells_mv()[ch - 1]
        i_ma = bench.sim_channel_current_ma(ch)
        bench.dut_balance(ch, False)
        r_mohm[ch - 1] = (idle[ch - 1] - loaded) / i_ma * 1000.0

    measurements.tap_path.x_axis = list(range(1, CELL_COUNT + 1))
    measurements.tap_path.y_axis.resistance = r_mohm.round(1).tolist()
    measurements.tap_path.y_axis.resistance.aggregations.max_mohm = float(r_mohm.max())
    worst = int(r_mohm.argmax()) + 1
    log.info(f"Tap resistance {r_mohm.min():.0f} .. {r_mohm.max():.0f} mOhm, worst channel {worst}")
```

![Bars of tap path resistance per channel between 30 and 65 mΩ, channel 6 at 117 mΩ in orange with the annotation marginal crimp, the 150 mΩ limit line above.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-31a2d708-f54d-4379-90ef-b7d74c46d73b-GzkW6Cw0RES8ToL3ccuyObnFxDEOYd.png)

The mock's marginal crimp on channel 6: 117 mΩ against 30 to 65 mΩ on the other fifteen taps, still under the 150 mΩ limit, and exactly the kind of channel that passes every voltage test and shows up as a trend across a batch. Two conditions for the real bench: the AFE must convert while the bleed path is on (the LTC6811 family does this with the DCP bit; TI parts pause the ADC during balancing unless the cell-balance loop is slowed), and the board must be an external-balancing design, otherwise the bleed current flows through the input filter resistor and the drop is by design, not a defect.

### Shutdown

The teardown phase commands ship mode, reads the stack's quiescent draw on the simulator's 250 µA range, and turns the outputs off. A stuck balancing FET or a wrong pull-up shows here as tens of milliamps against a limit of 250 µA:

```python filename="phases/shutdown.py"
from utils.recipe import CELL_COUNT, IDLE_MV


def shutdown(measurements, bench, log):
    """Teardown: ship mode, quiescent draw on the 250 uA range, outputs off.
    Runs after a failed main phase too, so no board is left balancing."""
    bench.force_cells([IDLE_MV] * CELL_COUNT)
    bench.dut_ship_mode()
    ship_ua = bench.sim_total_current_ua()
    measurements.ship_current_ua = ship_ua
    log.info(f"Ship mode draw {ship_ua:.1f} uA across the stack")
    bench.outputs_off()
```

### Mock Plug

`BmsBench` stands in for both the simulator and the DUT's service port because the two have to share state (the DUT reads what the simulator forces) and plugs run in separate processes. Its `config` block in YAML becomes the constructor's `cell_count` and `bleed_ma` arguments. The mock draws a per-channel gain error with a 120 ppm standard deviation and an offset with a 0.6 mV standard deviation, both well inside the BQ79616 window, then sets channel 12 to +1.9 mV of offset and channel 6 to 115 mΩ of tap resistance so the charts have a story. Every method returns plain Python types, because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, split the class in two: the simulator over SCPI with pyvisa (`SOUR:VOLT` per channel, `MEAS:VOLT` and `MEAS:CURR` for the readback) and the DUT over pyserial or an I²C bridge. Keep the simulator's readback as the truth, run the accuracy phases with balancing off, and record the simulator model and calibration date in the run metadata, because a ±5 mV and a ±1 mV instrument produce numerically identical results that mean different things. The phases, measurements and limits stay the same.


### Functional Test Fixture (FCT)
URL: https://www.tofupilot.com/templates/functional-test-fixture-fct
Build a pogo-pin functional test fixture that powers, programs, and exercises a PCBA end-to-end before shipment to catch firmware and system-level defects.


## Introduction

### Functional Test Overview

**Functional Test (FCT)** exercises a PCBA as a complete working system under simulated end-use conditions: powers the board, boots firmware, drives every interface, and verifies outputs against spec. It answers **"does it work as intended"** rather than ICT's **"was it built correctly."** FCT catches what ICT and boundary scan miss because both are static or component-level: **firmware bugs, marginal regulator dropout, hardware-software interaction faults, brownout-induced corruption, and timing margins** only show up when the board runs at speed.

![Cutaway of a pogo-pin fixture: the PCB on tooling pins, spring probes in the probe plate contacting the test pads underneath, the wired backer plate below, and the top plate pressing the board down.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-02b488d7-7c56-46a5-82cc-2f05401625fd-JYWr5uQ5YVPUM0xALofSiV8P2057Ab.jpg)

Inside the fixture: tooling pins locate the board, spring probes in the probe plate reach the test pads from below, the backer plate carries the wiring to the instruments, the top plate holds the board down at a controlled stroke.

The typical production flow is **ICT first (structural)** then **FCT (system behavior)**. Defects that escape ICT, a wrong firmware revision, a peripheral solder defect that does not fail continuity, a regulator marginal under load, get caught at FCT before the unit ships.

### Test Purpose

The FCT procedure produces a per-board functional fingerprint:

- **Power-up behavior**: inrush peak and steady-state current on a soft-start ramp
- **Rail voltages and ripple** at every regulator output
- **Firmware handshake**: test mode entered, firmware version, hardware revision
- **GPIO loopback** on every channel, high and low
- **ADC linearity**: gain error, offset and R² over a stimulus sweep
- **Operator checks** for what the fixture cannot measure cheaply, backed by photodiodes
- **Stop-mode current** for battery-powered designs

![Inrush capture at 100 kS/s: Vin soft-start ramp from 0 to 12 V in 50 ms, input current peaking at 189 mA during the bulk capacitor charge under the 400 mA limit, then settling to 90 mA steady state inside the 70 to 110 mA window.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-4262c51a-e058-4353-a6f4-96a6455f9d09-8pLQPMVuVm2PKA0RnTcIFll0ga3kFr.png)

The power-on capture from the mock supply: constant capacitor current while Vin ramps, the regulators loading as they start, then the steady draw. A shorted output capacitor or a dead regulator changes this trace before any rail reads wrong.

Per-board FCT is the **last line of defense before shipment**. It eliminates the leading category of returns: "doesn't boot", "GPIO output stuck", "battery dies in three days", all invisible to ICT and AOI.

Beyond the test content, this template demonstrates five framework mechanics: a **setup stage** that gates power on the fixture interlock, a **phase DAG** where three phases fan out from the handshake and the sleep measurement fans them back in, **progress components** driven from Python during the sweeps, **operator switches bound to boolean measurements** with pre-baked answers for unattended runs, and **run metadata** carrying the fixture identity and cycle count.

## Equipment & Setup

To implement FCT on a production line, the following are required:

- **A bed-of-nails or clamshell fixture** with pogo pins contacting test pads and connectors
- **A programmable DC supply** for the DUT input rail, with a current capture or a shunt on a scope
- **A DAQ** for rail sense, GPIO loopback, analog stimulus and photodiode pickups
- **The Device Under Test (DUT)**: a populated, ICT-passed, programmed PCBA
- **A TofuPilot Framework procedure** to orchestrate phases, capture measurements, and validate against limits
- **The TofuPilot Dashboard** to log per-board results and trend the production line

![Blue circuit board in a manual test fixture beneath a raised transparent pressure plate](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/functional-test-fixture-fct-yuPL8DV5VlrozKQJRxiCd0BDF9AHyI.webp)

The fixture: a blue PCBA located in a support cartridge beneath a raised pressure plate. Spring probes contact test pads from below.

### Hardware Components

#### Fixture

Four fixture topologies, in order of cost and production readiness:

- **Manual clamshell / lever press**: hinged top plate, operator-actuated. Cheapest, fast cycle, the NPI standard. Angled closure causes probe drag, limiting density.
- **Vertical actuation press**: the last few millimetres travel straight down, eliminates drag, suits HDI boards.
- **Vacuum bed-of-nails**: a chamber pulls the DUT onto the probes; preferred when the top must stay clear or for dual-sided access. Higher build cost, and the chamber pressure doubles as a seating check.
- **Pneumatic clamshell**: air-cylinder closure, consistent force, automation-ready for production lines.

Probe vendors: **Ingun**, **Feinmetall**, **Everett Charles Technologies**, **QA Technology**, with cross-reference tables between them. Production designs use 100 mil (2.54 mm pitch) probes on through-hole and standard pads and 50 mil probes for fine pitch. Tip styles (spear, crown, cup, flat, serrated) are chosen per pad finish.

#### Instrument Suite

A programmable FCT bench: a **Rigol DP832-class** triple-output supply or **Keithley 2230** for Vin, an **NI cDAQ** chassis with analog input, analog output and DIO modules (or a USB DAQ at lower channel counts) for rails, stimulus and loopback, a **USB-serial bridge** for the firmware test mode, and photodiodes on the LEDs. All of it speaks SCPI, VISA or a vendor Python API from the test computer.

#### Safety Interlocks

- **Lid-close switch** wired to the e-stop bus and read by the setup phase
- **Vacuum pressure sensor** on the chamber to confirm the DUT is seated before power applies
- Fuse and **hardware overcurrent cutoff** on the Vin rail, independent of the software limits
- **Two-hand buttons** or a light curtain for high-force or automated fixtures

### Custom Firmware

The DUT firmware exposes a **production test mode** triggered by a fixture-pulled GPIO. In this mode, the firmware reports raw ADC readings over UART, toggles GPIOs on command, enters stop mode on request, and disables sleep timers that would interrupt the test. Production firmware ships with this mode compiled in but locked behind the fixture pin, so the shipped unit does not expose the test interface.

## Test Procedure

### Overview

After scanning the DUT barcode and closing the lid, the procedure runs:

1. Setup: confirm the lid interlock and the vacuum seating, record the fixture identity on the run.
2. Ramp Vin with a soft start, capture inrush, check every rail and the 3V3 ripple.
3. Enter the firmware test mode, read firmware version and hardware revision.
4. In parallel: GPIO loopback on 32 channels, ADC linearity sweep, operator checks with photodiode backing.
5. Command stop mode and measure sleep current.
6. Teardown: Vin off before the lid opens.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
ui.json
phases/
  fixture_check.py
  power_on.py
  dut_handshake.py
  gpio_loopback.py
  analog_linearity.py
  operator_checks.py
  sleep_current.py
  power_down.py
plugs/
  psu.py
  daq.py
  dut.py
utils/
  linearity.py
README.md
pyproject.toml
```

The plugs are mocks: the supply synthesizes the inrush capture, the DAQ loops the GPIOs back, the DUT answers with a 12-bit ADC that carries a small gain error for the linearity phase to find. Run it with `tofupilot run . --no-tui --no-kiosk --json --ui-values ui.json`, or drop `--ui-values` to answer the operator switches yourself. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-fct-fixture).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: Functional Test Fixture
version: 0.1.0
description: Powers a PCBA on a pogo-pin fixture with a soft-start ramp, checks rails and inrush, talks to the firmware test mode, exercises GPIO and ADC, records operator checks, and measures stop-mode current.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the board data-matrix"
    placeholder: "UAUT-00000"
    pattern: "^UAUT-\\d{5}$"
    default_value: "UAUT-04829"
  part_number:
    default_value: "PCB-MAIN-V2"
  metadata:
    firmware_version:
      description: "Reported by the DUT over UART"

plugs:
  - name: Fixture Supply
    description: Programmable DC supply on Vin with inrush capture (mock)
    python: plugs.psu:FixtureSupply
    key: psu
  - name: Fixture DAQ
    description: Rail sense, 32-channel GPIO loopback, AWG stimulus, LED photodiodes (mock)
    python: plugs.daq:FixtureDaq
    key: daq
  - name: DUT Serial
    description: Firmware test mode over the fixture UART (mock)
    python: plugs.dut:DutSerial
    key: dut

setup:
  - name: Fixture Check
    key: fixture_check
    python: phases.fixture_check
    measurements:
      - name: Lid Closed
        key: lid_closed
        validators:
          - {operator: "==", expected_value: true}
      - name: Vacuum
        key: vacuum_kpa
        unit: kPa
        description: Chamber pressure confirming the DUT is seated on the probes.
        validators:
          - {operator: "<=", expected_value: -50.0}

main:
  - name: Power On
    key: power_on
    python: phases.power_on
    measurements:
      - name: Inrush
        key: inrush
        title: Vin ramp and input current
        description: 50 ms soft-start ramp to 12 V, input current sampled at 100 kS/s.
        x_axis:
          legend: Time
          unit: ms
        y_axis:
          - legend: Vin
            key: vin
            unit: V
          - legend: Input current
            key: iin
            unit: mA
            aggregations:
              - type: peak_ma
                unit: mA
                validators:
                  - {operator: "<=", expected_value: 400.0}
              - type: steady_ma
                unit: mA
                validators:
                  - {operator: ">=", expected_value: 70.0}
                  - {operator: "<=", expected_value: 110.0}
      - name: 3V3 Rail
        key: rail_3v3
        unit: V
        validators:
          - {operator: ">=", expected_value: 3.135}
          - {operator: "<=", expected_value: 3.465}
      - name: 5V Rail
        key: rail_5v
        unit: V
        validators:
          - {operator: ">=", expected_value: 4.75}
          - {operator: "<=", expected_value: 5.25}
      - name: 1V8 Rail
        key: rail_1v8
        unit: V
        validators:
          - {operator: ">=", expected_value: 1.71}
          - {operator: "<=", expected_value: 1.89}
      - name: 3V3 Ripple
        key: ripple_3v3
        unit: mVpp
        validators:
          - {operator: "<=", expected_value: 30.0}

  - name: DUT Handshake
    key: dut_handshake
    python: phases.dut_handshake
    depends_on: [power_on]
    measurements:
      - name: Test Mode
        key: test_mode
        validators:
          - {operator: "==", expected_value: true}
      - name: Firmware Version
        key: firmware_version
        validators:
          - {operator: in, expected_value: ["2.4.1"]}
      - name: Hardware Revision
        key: hardware_revision
        validators:
          - {operator: in, expected_value: ["B", "C"]}

  - name: GPIO Loopback
    key: gpio_loopback
    python: phases.gpio_loopback
    depends_on: [dut_handshake]
    ui:
      components:
        - key: loopback_progress
          type: progress
          label: "GPIO loopback"
          description: "32 channels, high and low"
          default_value: 0
          max: 100
    measurements:
      - name: GPIO Pass Count
        key: gpio_pass_count
        validators:
          - {operator: "==", expected_value: 32}
      - name: GPIO Failed Channels
        key: gpio_failed_channels
        description: Channels whose loopback readback did not follow the DUT drive.
        validators:
          - {operator: "==", expected_value: []}

  - name: Analog Linearity
    key: analog_linearity
    python: phases.analog_linearity
    depends_on: [dut_handshake]
    ui:
      components:
        - key: sweep_progress
          type: progress
          label: "ADC sweep"
          description: "32 stimulus steps from 0.1 V to 3.2 V"
          default_value: 0
          max: 100
    measurements:
      - name: ADC Sweep
        key: adc_sweep
        title: ADC reading vs fixture stimulus
        x_axis:
          legend: Stimulus
          unit: V
        y_axis:
          - legend: Reading
            key: reading
            unit: V
            aggregations:
              - type: gain_error_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: -0.5}
                  - {operator: "<=", expected_value: 0.5}
              - type: offset_mv
                unit: mV
                validators:
                  - {operator: ">=", expected_value: -5.0}
                  - {operator: "<=", expected_value: 5.0}
              - type: r2
                validators:
                  - {operator: ">=", expected_value: 0.9999}
          - legend: Residual
            key: residual
            unit: mV

  - name: Operator Checks
    key: operator_checks
    python: phases.operator_checks
    depends_on: [dut_handshake]
    ui:
      components:
        - key: instructions
          type: text
          label: "Look at the board and listen"
          default_value: "The status LED must be green, the fault LED must stay off, and the buzzer chirps once. Tick what you observed."
        - key: status_led
          type: switch
          label: "Status LED is green"
          required: true
          bind: measurements.status_led_seen
        - key: fault_led
          type: switch
          label: "Fault LED is off"
          required: true
          bind: measurements.fault_led_off
        - key: buzzer
          type: switch
          label: "Buzzer chirped once"
          required: true
          bind: measurements.buzzer_heard
    measurements:
      - name: Status LED Seen
        key: status_led_seen
        description: Operator switch, bound straight to this measurement.
        validators:
          - {operator: "==", expected_value: true}
      - name: Fault LED Off
        key: fault_led_off
        validators:
          - {operator: "==", expected_value: true}
      - name: Buzzer Heard
        key: buzzer_heard
        validators:
          - {operator: "==", expected_value: true}
      - name: Status LED Photodiode
        key: status_led_mv
        unit: mV
        validators:
          - {operator: ">=", expected_value: 300.0}
      - name: Fault LED Photodiode
        key: fault_led_mv
        unit: mV
        validators:
          - {operator: "<=", expected_value: 20.0}

  - name: Sleep Current
    key: sleep_current
    python: phases.sleep_current
    depends_on: [gpio_loopback, analog_linearity, operator_checks]
    measurements:
      - name: Stop Mode Entered
        key: stop_mode_entered
        validators:
          - {operator: "==", expected_value: true}
      - name: Sleep Current
        key: sleep_current
        unit: µA
        description: Stop-mode current on the low-range shunt; a floating GPIO or an ungated clock shows up here.
        validators:
          - {operator: "<=", expected_value: 15.0}

teardown:
  - name: Power Down
    key: power_down
    python: phases.power_down
    measurements:
      - name: Vin Off
        key: vin_off
        unit: V
        validators:
          - {operator: "<=", expected_value: 0.1}
```

Framework features to notice:

1. **Setup stage**: `fixture_check` must pass before any main phase runs, so an open lid or an unseated board never sees Vin.
2. **Phase DAG**: `gpio_loopback`, `analog_linearity` and `operator_checks` all depend on `dut_handshake` and on nothing else; `sleep_current` depends on all three. The framework orders them from the graph, and a failed handshake skips the whole branch.
3. **Progress components**: the two sweeps declare a `progress` component and update it from Python with `ui.<key> = value`, so the operator sees the 32 channels and the 32 steps advance.
4. **Bound operator switches**: the three switches carry `bind: measurements.*`; the operator's answers land as boolean measurements validated `== true`, and `ui.json` pre-bakes them for unattended runs. The two photodiode measurements sit next to them so the operator's word is backed by a number.
5. **JSON measurement**: `gpio_failed_channels` records the list of failed channels and validates it against an empty list, which keeps the exact failing pins on a failed run instead of only a count.

### Fixture Check

The setup phase reads the interlock and the chamber pressure, and stamps the fixture identity, cycle count and line onto the **run metadata**, so probe wear can be trended per fixture later:

```python filename="phases/fixture_check.py"
def fixture_check(measurements, run, log):
    """Setup: lid interlock and DUT seating before any power is applied."""
    run.metadata["fixture_id"] = "FIX-07"
    run.metadata["fixture_cycles"] = 18420
    run.metadata["line"] = "SMT-2"

    measurements.lid_closed = True
    measurements.vacuum_kpa = -62.0
    log.info("Fixture FIX-07 closed, DUT seated")
```

### Power On

The supply ramps Vin over 50 ms and returns the input current sampled at 100 kS/s. The capture lands in a multi-dimensional measurement with two y-axes; the peak and the settled mean are computed in the phase and validated in YAML:

```python filename="phases/power_on.py"
import numpy as np

RAMP_MS = 50.0


def power_on(measurements, psu, daq, log):
    t_ms, vin, iin = psu.ramp_to(12.0, RAMP_MS)
    iin = np.asarray(iin)
    settled = np.asarray(t_ms) > 150.0

    measurements.inrush.x_axis = t_ms
    measurements.inrush.y_axis.vin = vin
    measurements.inrush.y_axis.iin = iin.tolist()
    measurements.inrush.y_axis.iin.aggregations.peak_ma = float(iin.max())
    measurements.inrush.y_axis.iin.aggregations.steady_ma = float(iin[settled].mean())
    log.info(f"Inrush peak {iin.max():.0f} mA, steady {iin[settled].mean():.0f} mA")

    measurements.rail_3v3 = daq.measure_rail("3v3")
    measurements.rail_5v = daq.measure_rail("5v")
    measurements.rail_1v8 = daq.measure_rail("1v8")
    measurements.ripple_3v3 = daq.measure_ripple_mv("3v3")
```

### GPIO Loopback

Every GPIO is driven high then low by the DUT and read back on the fixture side of the loopback jumper. The progress bar advances per channel, the failed channels are kept as a list:

```python filename="phases/gpio_loopback.py"
from plugs.daq import GPIO_CHANNELS


def gpio_loopback(measurements, dut, daq, ui, log):
    failed = []
    for ch in range(GPIO_CHANNELS):
        for level in (1, 0):
            dut.set_gpio(ch, level)
            daq.drive_gpio_from_dut(ch, level)  # mock: the loopback jumper
            if daq.read_gpio(ch) != level:
                failed.append(ch)
                break
        ui.loopback_progress = int((ch + 1) / GPIO_CHANNELS * 100)

    measurements.gpio_pass_count = GPIO_CHANNELS - len(failed)
    measurements.gpio_failed_channels = failed
    log.info(f"GPIO loopback: {GPIO_CHANNELS - len(failed)}/{GPIO_CHANNELS} channels")
```

![GPIO loopback matrix: 32 channels driven high and low by the DUT and read back on the fixture side, all 64 cells green.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-a077112d-286f-4dfd-aed1-4c4f35253294-p0TKJMoqiQhRdl0bCU3PFPFTIAEnvJ.png)

The loopback matrix of the mock run. On a real board a stuck pin shows as one red column, a solder bridge between neighbours as two.

### Analog Linearity

The fixture AWG steps the stimulus from 0.1 V to 3.2 V and the firmware reports the ADC code at each step. A least-squares fit gives gain, offset and R²; the readings and the residuals both go into the chart:

```python filename="phases/analog_linearity.py"
import numpy as np

from plugs.dut import ADC_BITS, ADC_FS_V
from utils.linearity import fit_linearity

STEPS_V = np.linspace(0.1, 3.2, 32)


def analog_linearity(measurements, dut, daq, ui, log):
    readings = []
    for i, v in enumerate(STEPS_V):
        daq.set_stimulus(float(v))
        code = dut.read_adc(float(v))
        readings.append(code / (2**ADC_BITS - 1) * ADC_FS_V)
        ui.sweep_progress = int((i + 1) / STEPS_V.size * 100)

    gain, offset_mv, r2 = fit_linearity(STEPS_V, readings)
    residual_mv = ((np.asarray(readings) - (gain * STEPS_V + offset_mv / 1000.0)) * 1000.0).tolist()
    log.info(f"ADC gain {gain:.4f}, offset {offset_mv:.1f} mV, R² {r2:.5f}")

    measurements.adc_sweep.x_axis = STEPS_V.tolist()
    measurements.adc_sweep.y_axis.reading = readings
    measurements.adc_sweep.y_axis.residual = residual_mv
    measurements.adc_sweep.y_axis.reading.aggregations.gain_error_pct = (gain - 1.0) * 100.0
    measurements.adc_sweep.y_axis.reading.aggregations.offset_mv = offset_mv
    measurements.adc_sweep.y_axis.reading.aggregations.r2 = r2
```

![ADC linearity sweep: 32 stimulus steps from 0.1 V to 3.2 V read back through the firmware test mode, fit gain 1.0020 and offset 1.8 mV with R² 0.999999, and the residuals in millivolts showing 12-bit quantization plus noise.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-e574b14d-60d5-4cb2-8ec2-bbb910bc660f-hbMNObJddoIVEDOhGFuGy1TORkDDlx.png)

The mock DUT carries a +0.2 % gain error and a 2 mV offset on purpose; the fit finds both, well inside the ±0.5 % and ±5 mV limits, and the residuals are the 12-bit quantization plus noise.

One plug boundary to know: plug method arguments cross a process boundary as JSON, so a phase passes plain values between plugs (the stimulus voltage here), never a plug object.

### Operator Checks

The fixture cannot cheaply tell a green LED from a red one or hear the buzzer, so the operator does, through three switches bound to boolean measurements. The phase adds the photodiode readings next to them:

```python filename="phases/operator_checks.py"
def operator_checks(measurements, daq, log):
    """Photodiode readings back the operator's checklist with numbers."""
    measurements.status_led_mv = daq.photodiode_mv("status_green")
    measurements.fault_led_mv = daq.photodiode_mv("fault_red")
    log.info("LED photodiodes read; operator switches recorded through their bindings")
```

For unattended runs the answers come from `ui.json`, keyed by phase and component:

```json filename="ui.json"
{
  "operator_checks": {
    "status_led": true,
    "fault_led": true,
    "buzzer": true
  }
}
```

### Sleep Current

With all three branches passed, the DUT is commanded into stop mode and the supply's low-range shunt reads the current. The 15 µA limit sits at three times the datasheet typical; a floating GPIO or a clock left running shows as a multiple of it:

```python filename="phases/sleep_current.py"
def sleep_current(measurements, dut, psu, log):
    measurements.stop_mode_entered = dut.enter_stop_mode()
    ua = psu.measure_sleep_current_ua()
    log.info(f"Stop-mode current {ua:.1f} uA")
    measurements.sleep_current = ua
```

### Mock Plugs

The supply mock builds the inrush trace from a 470 µF bulk capacitor charging at the ramp's dV/dt plus the regulator load starting up; the DUT mock applies a 1.0018 gain and a 2.1 mV offset before quantizing to 12 bits. Both carry the method surface a pyvisa supply and a pyserial test-mode client need to implement:

```python filename="plugs/dut.py"
import numpy as np

ADC_FS_V = 3.3
ADC_BITS = 12


class DutSerial:
    def __init__(self):
        self._rng = np.random.default_rng(3)
        self._test_mode = False
        self._gain = 1.0018   # ADC gain error the linearity phase must catch
        self._offset_v = 0.0021
        print("DUT UART open at 115200")

    def enter_test_mode(self):
        self._test_mode = True
        return True

    def firmware_version(self):
        return "2.4.1"

    def hardware_revision(self):
        return "C"

    def set_gpio(self, channel, level):
        return True

    def read_adc(self, stimulus_v):
        """ADC code the firmware reports for the given fixture stimulus."""
        v = float(stimulus_v) * self._gain + self._offset_v + self._rng.normal(0.0, 0.0006)
        code = int(round(np.clip(v, 0.0, ADC_FS_V) / ADC_FS_V * (2**ADC_BITS - 1)))
        return code

    def enter_stop_mode(self):
        return True
```

On a real bench, `plugs/psu.py` maps to a pyvisa supply with a current capture, `plugs/daq.py` to nidaqmx or a USB DAQ, `plugs/dut.py` to pyserial against the firmware test-mode protocol; the phases, measurements, and limits stay exactly the same.


### Pack EOL Hipot and Insulation Test
URL: https://www.tofupilot.com/templates/pack-eol-hipot-and-insulation-test
End-of-line electrical safety of a 96S traction pack: DC withstand on both HV terminals, 500 V insulation resistance in ohm per volt, 25 A ground bond.


## Introduction

### Pack Electrical Safety Overview

A traction or stationary storage pack is a 400 V DC source inside a metal enclosure that people touch. The end-of-line electrical safety test proves three things about every pack before it ships: that the insulation between the high-voltage bus and the enclosure survives a voltage well above anything it will see in service (dielectric withstand, or hipot), that this insulation is also high enough in resistance to keep the leakage current through a person below the touch limit (insulation resistance), and that every exposed conductive part is bonded to the protective earth stud with a resistance low enough to trip the upstream protection (ground bond). Of the eighteen standards routinely quoted around battery testing, this is the one test that a standard actually mandates on every unit: UL 1973 clause 40.3 requires a production-line dielectric withstand on packs above 60 V DC, with a shortened alternative at a higher test voltage so it fits a line takt. Everything else in those standards is a type test on a handful of samples.

![A traction battery pack case with two orange HV terminals on top, the red HV test clip on one terminal and the black four-wire ground bond clamp on the stud on the side.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-624ef8f4-40f5-42c8-bf71-1e33f469c21d-SIju77BTWnvgrFDgGIMAgvp0813n8g.jpg)

The three connections of the test on one pack: the analyzer's HV clip on the terminal under test, the return on the chassis, and the four-wire ground bond clamp on the PE stud.

Lithium packs are tested with **DC**, not AC. The HV bus sees the enclosure through the EMI filter capacitors and the cell-to-case capacitance, together in the order of 100 nF; at 50 Hz an AC test voltage drives a reactive current through that capacitance that swamps the real leakage by three orders of magnitude, so the analyzer trips on a healthy pack. Under DC the capacitance draws current only while the voltage ramps, then the current settles to the resistive leakage and the arc detector watches the dwell. Hioki's application notes for their ST5680 series say this outright, and it is why every EV pack line runs DC withstand.

### Test Purpose

The procedure records one safety fingerprint per pack:

- **Gate**: operator interlock confirmed, contactors open, pack at storage SOC, BMS isolation monitor switched off
- **DC withstand** on HV+ and on HV- to chassis: leakage current through the ramp and the dwell, peak and settled values, arc events, the analyzer's own verdict
- **Insulation resistance** at 500 V DC on both terminals: the 60 s value in MΩ and normalised to Ω per volt of working voltage
- **Ground bond** at 25 A, four-wire, from the PE stud to three exposed conductive parts
- **Residual voltage** after discharge, and the isolation monitor handed back to the BMS

![Leakage current in µA over 65 s for HV+ in green and HV- in blue: 51 to 54 µA during the 5 s ramp, a drop to 9 µA the moment the ramp ends, an exponential tail settling to 3 µA. The test voltage as a dotted line on the right axis rising to 2550 V. The 50 µA settled limit drawn over the last 30 s.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-7a833da3-1c70-417e-8972-b78b1b56642e-1vsuuIQtgctwMum1OPHyNwlJjK7XHw.png)

The mock pack under 2550 V DC: the 51 µA plateau during the ramp is 100 nF of Y capacitance charging at 510 V/s, the drop at 5 s is the ramp ending, the tail is dielectric absorption, and the 3 µA the current settles to is the pack's 850 MΩ of insulation. The 5 mA limit of ISO 16750-2 is off the chart; the customer's 50 µA settled limit is what a wet connector or a pinched harness would fail.

The framework mechanics on show are an operator switch bound to a validated measurement, a `setup` gate and a `teardown` that always discharges, a station-scoped plug held across packs, multi-dimensional measurements with custom aggregations where a standard's limit and an internal limit sit on the same curve, and string and integer validators on the instrument's own verdict.

## Equipment & Setup

To run this test on a pack line, the following are required:

- **An electrical safety analyzer** with DC withstand to at least 3 kV, insulation resistance at 500 or 1000 V DC, a four-wire ground bond source of 25 to 30 A, arc detection and an open/short check on the leads
- **A safety fixture**: hood with interlock, HV clips on both terminals, ground bond clamp on the PE stud, discharge path
- **A CAN link to the pack BMS** to switch the isolation monitor off and back on
- **The Device Under Test (DUT)**: an assembled pack at storage SOC, contactors open
- **A TofuPilot Framework procedure** to sequence the tests, log the curves and validate the limits
- **The TofuPilot Dashboard** to keep the per-pack evidence the standards ask you to keep

### Hardware Components

#### Safety Analyzer

The [Chroma 19032](https://www.chromausa.com/wp-content/uploads/2024/11/19032-Electrical-Safety-Analyzer-112024.pdf) does the whole sequence in one box: AC withstand 0.05 to 5 kV, **DC withstand 0.05 to 6 kV** with cutoff to 12 mA and 0.1 µA current resolution, insulation resistance 0.1 MΩ to 50 GΩ at 50 V to 1 kV, **ground bond 1 to 30 A four-wire** with 0.1 mΩ resolution over 10 to 510 mΩ, test time 0.3 to 999 s, programmable arc detection, and an Open/Short Check that catches the failure mode every hipot station fears: a lead that fell off reading as a clean pass. The [Hioki ST5680](https://www.hioki.com/global/products/safety-testers/withstanding-voltage-testers/id_412609) is the withstand specialist for lithium packs, DC to 8 kV and 20 mA, 0.001 µA current resolution, 500 kS/s waveform capture of the leakage during the dwell and arc detection. The [Chroma 19572](https://www.chromausa.com/product/ground-bond-tester-19572/) covers ground bond alone at 3 to 45 A when a line separates the two. Kikusui's TOS9200 and Vitrek's 95X are the usual alternatives.

![Battery pack secured inside a closed interlocked electrical-test enclosure](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/pack-eol-hipot-and-insulation-test-physical-2026-OVKyc1Mvodb6s5oKVsSAfxGRsgJGxp.webp)

The closed hood encloses the pack and its test connections. The pack case has a separate connection through the enclosure wall; the safety analyzer is not shown.

#### Test Voltages and Where the Limits Come From

The withstand voltage derives from the pack's maximum working voltage, not from a constant. For a 96S NMC pack, U_max is 96 × 4.2 = 403 V; the classic 2U + 1000 V AC rule gives 1806 V AC, and its DC equivalent (× 1.414) rounds to **2550 V DC**. A real CB report for an AC-DC supply used 4700 V AC for reinforced insulation and 2000 V AC for basic, both derived from the working voltage; copying any of these numbers into another product is the error to avoid.

The insulation limits are two different numbers that get conflated. The **regulatory floor** under ISO 6469-3 and UN R100 (and FMVSS 305 in the US) is **100 Ω/V for a DC bus with continuous isolation monitoring and 500 Ω/V for AC**; for a 403 V pack that is 40 kΩ, and it is a survival criterion after the crash and abuse tests, not a production gate. A **production gate** sits between 10 and 100 MΩ and is the customer's own specification. This template validates both on the same curve: the 60 s value ≥ 100 MΩ and the normalised value ≥ 500 Ω/V, so the report shows the pack cleared the floor by four orders of magnitude and the gate by a factor of eight. IEC 62133-2 clause 5.2 gives the portable-battery version of the same idea: ≥ 5 MΩ at 500 V DC read 60 s after the voltage is applied, which is where the 60 s dwell comes from.

Ground bond: ≤ 100 mΩ at 25 A is the common requirement, verified in the same CB report ("cannot be higher than 100 mΩ", 25 A for 1 minute). The four-wire method is not optional at that level: the clamp and lead resistance of a two-wire measurement are the same order as the limit.

#### Two Procedural Hazards

**Switch off the isolation monitor.** The BMS's own insulation monitoring device measures the same resistance the analyzer is about to measure, through a resistor of its own to chassis. Left on, it sits in parallel with the pack's insulation and the analyzer reads the IMD's resistor instead of the pack. GB 38031 says to disconnect it for the measurement; this template does it over CAN in the setup phase and validates that it happened.

**Never retest a pack that arced.** Hipot is the one end-of-line test where a retest-on-fail policy is unsafe. An arc across a contaminant, a solder ball, a strand of wire or flux residue on a creepage path, burns the contaminant away. The second test passes, the pack ships, and the creepage path is still short by whatever the contaminant bridged. The `arc_events == 0` validator is deliberately not wrapped in a retry.

## Test Procedure

### Overview

The procedure maps the sequence onto the framework's three stages. Safety lives in `setup:` so no high voltage leaves the analyzer before the operator has confirmed the hood and the clips and the BMS isolation monitor is off. The discharge lives in `teardown:` so it runs even when a withstand phase fails and no pack is left charged to 2.5 kV under the hood.

1. Setup: interlock switch, contactors open, pack voltage in the storage window, isolation monitor off.
2. Main: DC withstand HV+ to chassis, leakage logged, arc count, verdict.
3. Main: DC withstand HV- to chassis.
4. Main: insulation resistance at 500 V DC on both terminals, MΩ and Ω/V.
5. Main: ground bond at 25 A on three points.
6. Teardown: discharge, residual voltage, isolation monitor back on.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-pack evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  safety_gate.py
  hipot_hv_plus.py
  hipot_hv_minus.py
  insulation_resistance.py
  ground_bond.py
  discharge_release.py
plugs/
  safety_analyzer.py
  pack_fixture.py
utils/
  recipe.py
ui.json
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-pack-eol-hipot-insulation). The `SafetyAnalyzer` mock returns the whole 65 s dwell in one call and the `PackFixture` mock reports a pack at storage SOC with the contactors open, so the procedure runs end-to-end in seconds without a pack or an analyzer connected.

```bash
tofupilot run .
```

For CI or bench automation, `ui.json` pre-bakes the interlock switch and the run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json --ui-values ui.json --ui-timeout 60
```

### The Procedure File

`procedure.yaml` declares the unit, the two plugs (the analyzer at **station scope**, so its leads stay connected between packs), and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Pack EOL Hipot and Insulation
version: 0.1.0
description: End-of-line electrical safety of a 96S traction pack. Operator interlock and isolation-monitor gate, DC withstand on both HV terminals with the leakage logged, 500 V insulation resistance normalised to ohm per volt, 25 A ground bond, discharge and residual check.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the pack label before closing the fixture"
    placeholder: "PACK-96S-000000"
    pattern: "^PACK-96S-\\d{6}$"
    default_value: "PACK-96S-004217"
  part_number:
    default_value: "PK-96S-NMC-60"
  batch_number:
    default_value: "WK-2026-37"

plugs:
  - name: Safety Analyzer
    description: "5-in-1 electrical safety analyzer, leads stay on between packs (mock Chroma 19032-class)"
    python: plugs.safety_analyzer:SafetyAnalyzer
    key: analyzer
    scope: station
  - name: Pack Fixture
    description: Fixture PLC and CAN link to the pack BMS (mock)
    python: plugs.pack_fixture:PackFixture
    key: fixture

setup:
  - name: Safety Gate
    key: safety_gate
    python: phases.safety_gate
    ui:
      components:
        - key: interlock
          type: switch
          label: "Fixture closed and interlock armed"
          description: "Confirm the hood is down, the HV clips are on both terminals and the PE clip is on the chassis stud"
          required: true
          bind: measurements.interlock_armed
    measurements:
      - name: Interlock Armed
        key: interlock_armed
        validators:
          - {operator: "==", expected_value: true}
      - name: Contactors Open
        key: contactors_open
        validators:
          - {operator: "==", expected_value: true}
      - name: Pack Voltage
        key: pack_voltage_v
        unit: V
        description: Storage SOC window for a 96S NMC pack, 3.60 to 3.80 V per cell.
        validators:
          - {operator: ">=", expected_value: 345.0}
          - {operator: "<=", expected_value: 365.0}
      - name: Isolation Monitor Disabled
        key: imd_disabled
        description: The BMS isolation monitor's measuring resistance sits in parallel with the pack insulation; it must be off during the test.
        validators:
          - {operator: "==", expected_value: true}

main:
  - name: Hipot HV Plus
    key: hipot_hv_plus
    python: phases.hipot_hv_plus
    timeout: 5m
    measurements:
      - name: Leakage HV Plus
        key: leakage_plus
        title: DC withstand HV+ to chassis, 2550 V, 5 s ramp, 60 s dwell
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Leakage
            key: leak
            unit: µA
            aggregations:
              - type: peak_ua
                unit: µA
                validators:
                  - {operator: "<=", expected_value: 5000.0}
              - type: settled_ua
                unit: µA
                validators:
                  - {operator: "<=", expected_value: 50.0}
      - name: Arc Events HV Plus
        key: arc_events_plus
        description: "Arc detector count during the dwell. Any arc is a fail, and a failed pack is not retested: the arc burns away the contaminant that caused it."
        validators:
          - {operator: "==", expected_value: 0}
      - name: Judgment HV Plus
        key: judgment_plus
        validators:
          - {operator: "==", expected_value: PASS}

  - name: Hipot HV Minus
    key: hipot_hv_minus
    python: phases.hipot_hv_minus
    depends_on: [hipot_hv_plus]
    timeout: 5m
    measurements:
      - name: Leakage HV Minus
        key: leakage_minus
        title: DC withstand HV- to chassis, 2550 V, 5 s ramp, 60 s dwell
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Leakage
            key: leak
            unit: µA
            aggregations:
              - type: peak_ua
                unit: µA
                validators:
                  - {operator: "<=", expected_value: 5000.0}
              - type: settled_ua
                unit: µA
                validators:
                  - {operator: "<=", expected_value: 50.0}
      - name: Arc Events HV Minus
        key: arc_events_minus
        validators:
          - {operator: "==", expected_value: 0}
      - name: Judgment HV Minus
        key: judgment_minus
        validators:
          - {operator: "==", expected_value: PASS}

  - name: Insulation Resistance
    key: insulation_resistance
    python: phases.insulation_resistance
    depends_on: [hipot_hv_minus]
    timeout: 5m
    measurements:
      - name: IR HV Plus
        key: ir_plus
        title: Insulation resistance HV+ to chassis at 500 V DC
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Resistance
            key: ir
            unit: MΩ
            aggregations:
              - type: at_60s_mohm
                unit: MΩ
                validators:
                  - {operator: ">=", expected_value: 100.0}
              - type: ohm_per_volt
                unit: Ω/V
                validators:
                  - {operator: ">=", expected_value: 500.0}
      - name: IR HV Minus
        key: ir_minus
        title: Insulation resistance HV- to chassis at 500 V DC
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Resistance
            key: ir
            unit: MΩ
            aggregations:
              - type: at_60s_mohm
                unit: MΩ
                validators:
                  - {operator: ">=", expected_value: 100.0}
              - type: ohm_per_volt
                unit: Ω/V
                validators:
                  - {operator: ">=", expected_value: 500.0}

  - name: Ground Bond
    key: ground_bond
    python: phases.ground_bond
    depends_on: [insulation_resistance]
    measurements:
      - name: Bond Chassis Lug
        key: bond_chassis_lug_mohm
        unit: mΩ
        validators:
          - {operator: "<=", expected_value: 100.0}
      - name: Bond Service Cover
        key: bond_service_cover_mohm
        unit: mΩ
        validators:
          - {operator: "<=", expected_value: 100.0}
      - name: Bond HV Connector Bracket
        key: bond_hv_connector_bracket_mohm
        unit: mΩ
        validators:
          - {operator: "<=", expected_value: 100.0}

teardown:
  - name: Discharge Release
    key: discharge_release
    python: phases.discharge_release
    measurements:
      - name: Residual Voltage
        key: residual_voltage_v
        unit: V
        description: HV bus to chassis one second after the analyzer's discharge; below the 60 V DC touch limit before the hood opens.
        validators:
          - {operator: "<=", expected_value: 60.0}
      - name: Isolation Monitor Restored
        key: imd_restored
        validators:
          - {operator: "==", expected_value: true}
```

Framework features to notice:

1. **Operator switch bound to a measurement.** The interlock switch declares `bind: measurements.interlock_armed` and the measurement validates `== true`, so the report records what the operator confirmed, not that a button was pressed. `ui.json` pre-bakes it for headless runs.
2. **Station-scoped plug.** `scope: station` on the analyzer instantiates it once for the station's lifetime; the leads stay connected and the open/short check runs once, not per pack.
3. **Two limits on one curve.** The leakage curve carries `peak_ua <= 5000` (ISO 16750-2) and `settled_ua <= 50` (customer spec) as two aggregations; the report shows which one a failing pack broke.
4. **The instrument's verdict as a measurement.** `judgment_plus == PASS` and `arc_events_plus == 0` record what the analyzer itself decided, next to the framework's own limits on the curve.
5. **`teardown:` always runs.** The discharge and the residual check execute after a failed withstand too.

### Safety Gate

The setup phase reads what the fixture knows before the analyzer's output is enabled: contactors open, pack voltage inside the storage window, and the isolation monitor switched off over CAN and read back. The pack voltage at test goes onto the **unit metadata**:

```python filename="phases/safety_gate.py"
def safety_gate(measurements, fixture, unit, log):
    """Setup: operator interlock (switch bound to a measurement), contactors
    open, pack at storage SOC, BMS isolation monitor disabled. No high
    voltage leaves the analyzer before this phase passes."""
    measurements.contactors_open = fixture.contactors_open()
    v = fixture.pack_voltage()
    measurements.pack_voltage_v = v
    fixture.disable_isolation_monitor()
    measurements.imd_disabled = not fixture.isolation_monitor_enabled()
    unit.metadata["pack_voltage_at_test_v"] = v
    log.info(f"Pack {unit.serial_number} at {v:.1f} V, contactors open, isolation monitor disabled")
```

The hardware interlock on the hood stays wired to the analyzer's interlock input; the switch is the operator-facing layer, not the protection.

### Hipot HV Plus and HV Minus

Each withstand phase asks the analyzer for one capture: ramp to 2550 V DC in 5 s, hold 60 s, leakage sampled at 10 Hz. The peak is the Y capacitance charging during the ramp and is limited by the standard's 5 mA; the settled value is the mean over the last 30 s of the dwell and is limited by the customer's 50 µA. The arc count and the analyzer's own PASS/FAIL come back with the curve:

```python filename="phases/hipot_hv_plus.py"
import numpy as np

from utils.recipe import DCW_DWELL_S, DCW_RAMP_S, DCW_V


def hipot_hv_plus(measurements, analyzer, log):
    """DC withstand, HV+ to chassis: ramp to the test voltage, hold, log the
    leakage. The peak is the ramp charging the Y capacitance; the settled
    value is the real insulation current."""
    cap = analyzer.dc_withstand("hv_plus", DCW_V, DCW_RAMP_S, DCW_DWELL_S)
    leak = np.array(cap["leak_ua"])
    t = np.array(cap["time_s"])
    settled = float(leak[t >= DCW_RAMP_S + 30.0].mean())

    measurements.leakage_plus.x_axis = cap["time_s"]
    measurements.leakage_plus.y_axis.leak = cap["leak_ua"]
    measurements.leakage_plus.y_axis.leak.aggregations.peak_ua = float(leak.max())
    measurements.leakage_plus.y_axis.leak.aggregations.settled_ua = settled
    measurements.arc_events_plus = int(cap["arc_events"])
    measurements.judgment_plus = cap["judgment"]
    log.info(f"HV+ at {DCW_V} V DC: peak {leak.max():.1f} uA, settled {settled:.1f} uA, arcs {cap['arc_events']}")
```

`hipot_hv_minus.py` is the same phase on the other terminal, chained with `depends_on` because both use the analyzer. Testing the two terminals separately, rather than shorting HV+ to HV- and testing the bus once, keeps the pack's own voltage out of the measurement and tells the report which side of the bus leaks.

### Insulation Resistance

At 500 V DC the analyzer samples the resistance once a second for 60 s. The reading rises over the dwell as the dielectric absorbs charge, which is why the standards specify the value at 60 s and not at first contact. The phase records the curve per terminal and validates the 60 s value twice, in MΩ against the production gate and in Ω/V against the regulatory floor:

```python filename="phases/insulation_resistance.py"
from utils.recipe import IR_DWELL_S, IR_TEST_V, U_MAX_V


def insulation_resistance(measurements, analyzer, log):
    """500 V DC insulation resistance on both HV terminals, the 60 s value
    validated in MOhm and normalised to ohm per volt of working voltage."""
    for terminal, key in (("hv_plus", "ir_plus"), ("hv_minus", "ir_minus")):
        cap = analyzer.insulation_resistance(terminal, IR_TEST_V, IR_DWELL_S)
        ir_60 = float(cap["ir_mohm"][-1])
        curve = getattr(measurements, key)
        curve.x_axis = cap["time_s"]
        curve.y_axis.ir = cap["ir_mohm"]
        curve.y_axis.ir.aggregations.at_60s_mohm = ir_60
        curve.y_axis.ir.aggregations.ohm_per_volt = ir_60 * 1e6 / U_MAX_V
        log.info(f"{terminal}: {ir_60:.0f} MOhm at 60 s, {ir_60 * 1e6 / U_MAX_V:,.0f} ohm/V")
```

![Insulation resistance in MΩ over 60 s for both terminals, rising from 600 MΩ at t=0 to 851 MΩ (HV+) and 910 MΩ (HV-) at 60 s. The 100 MΩ customer limit as a dashed line near the bottom and the 0.20 MΩ regulatory floor as a dotted line on the axis.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-c6f353b4-b0bf-4fe5-a23c-a2d138669236-WUHk7qNBwstCH4PDNLP1U3xyHXR7BO.png)

The two curves of the mock pack: 30 % of dielectric absorption over the first 20 s, then flat. The regulatory floor of 500 Ω/V × 403 V = 0.20 MΩ is invisible at this scale, which is the point of validating both numbers: the floor is what the pack must survive after a crash, the gate is what a good pack looks like leaving the line.

### Ground Bond

25 A through the PE stud to each exposed conductive part, four-wire, the drop read at the clamp. Three points on this pack: the chassis lug, the service cover, and the bracket of the HV connector, each with its own scalar measurement and the same 100 mΩ limit:

```python filename="phases/ground_bond.py"
from utils.recipe import GROUND_BOND_A, GROUND_BOND_POINTS


def ground_bond(measurements, analyzer, log):
    """25 A 4-wire bond resistance from the PE stud to each exposed
    conductive part; a painted bracket or a missing star washer shows here."""
    for point in GROUND_BOND_POINTS:
        r = analyzer.ground_bond(point, GROUND_BOND_A)
        setattr(measurements, f"bond_{point}_mohm", r)
        log.info(f"{point}: {r:.1f} mOhm at {GROUND_BOND_A:.0f} A")
```

![Three bars of bond resistance: chassis lug 8.5 mΩ, service cover 31.2 mΩ, HV connector bracket 43.7 mΩ, all under the 100 mΩ dashed limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-238b6b91-51f2-425a-a7e9-0a3ef6c68daf-DLFzUNLKEDqFQshLf8aT0I1219aZbg.png)

The three bond points of the mock pack. The bracket is the one to watch across a batch: it bonds through a painted surface and a star washer, and a missing washer shows as a jump from 44 mΩ to several hundred.

### Discharge Release

The teardown phase discharges the bus through the analyzer, reads the residual one second later against the 60 V DC touch limit, and hands the isolation monitor back to the BMS:

```python filename="phases/discharge_release.py"
def discharge_release(measurements, analyzer, fixture, log):
    """Teardown: discharge the HV bus to chassis, verify the residual, hand
    the isolation monitor back to the BMS. Runs after a failed phase too, so
    no pack leaves the fixture charged to the test voltage."""
    residual = analyzer.discharge()
    measurements.residual_voltage_v = residual
    fixture.enable_isolation_monitor()
    measurements.imd_restored = fixture.isolation_monitor_enabled()
    log.info(f"Residual {residual:.1f} V after discharge, isolation monitor restored")
```

### Mock Plugs

`SafetyAnalyzer` models a pack with 100 nF of HV-to-chassis capacitance and 850 MΩ (HV+) and 910 MΩ (HV-) of insulation: the withstand current is C × dV/dt during the ramp, V/R plus a decaying absorption tail after it, and the IR curve is the same 850 MΩ approached from 30 % below with a 12 s time constant. The ground bond returns 8.5, 31 and 44 mΩ for the three points. `PackFixture` reports contactors open, 3.70 V per cell and an isolation monitor that obeys the disable command. Plug calls cross a JSON boundary, so the analyzer returns plain lists; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real line, `SafetyAnalyzer` becomes a pyvisa class speaking SCPI to a Chroma 19032, a Hioki ST5680 or a Kikusui TOS9200 (`DCW:VOLT`, `DCW:RAMP`, `START`, then a fetch of the leakage log), and `PackFixture` becomes the fixture PLC plus a python-can bus to the BMS. Keep the analyzer's arc detector and open/short check enabled, set `TIME_SCALE = 1.0` when the analyzer streams the dwell in real time, and keep the rule that a pack which arced goes to analysis, not back into the fixture. The phases, measurements and limits stay the same.


### Encoder and Commutation Calibration
URL: https://www.tofupilot.com/templates/encoder-and-commutation-calibration
Encoder and commutation calibration of a humanoid joint actuator: phase order, offset by a friction-cancelled d-axis walk, encoder self-cal checked, joint zero.


## Introduction

### Encoder and Commutation Calibration Overview

A frameless motor does not know where its rotor is. The drive commutates it from an encoder, and the number that ties the encoder's zero to the rotor's magnetic axis, the **commutation offset**, is set once, at the end of the line, per actuator. Get it wrong by 30 electrical degrees and the motor delivers 87 % of its torque and heats up; get it wrong by 90 and it delivers none. The encoder itself is a magnet ring glued to the rotor and read by a head a few hundred microns away, and its accuracy depends on how well the ring was centred: an eccentricity of 18 µm on a 30 mm ring is 120 arc-seconds of first-harmonic error, which the encoder's own self-calibration can remove and a mis-mounted ring cannot. And the joint's zero, the angle the robot's kinematics call zero, is a number found against a mechanical reference and written to the drive. Three calibrations, three writes to the actuator, each one verified before the next.

![A magnetic ring encoder on a plate: the dark magnet ring with fine segments on a shaft hub, and the small green readhead PCB held beside its rim by a bracket.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-edec7d2b-0ef0-4aec-8f61-9687008ef217-2dE07ZWMPZTaEYZaYle9djtH5UlvR4.jpg)

The rotor encoder: a magnetised ring on the shaft, read by a head on a bracket; the gap between the two and the centring of the ring are what the first phase checks.

The methods are the vendors'. Synapticon's drives run a **commutation offset detection** that finds the phase order, the electrical offset and the pole-pair count with the rotor free and unloaded ([Synapticon documentation](https://doc.synapticon.com/circulo_safe_motion/sw5.1/actuator_config/commutation_offset_detection.html)). Ben Katz's autocalibration for the MIT Cheetah actuators steps the d-axis forward and back and averages to cancel friction, low-passes over one electrical cycle to cancel cogging, and writes a 128-point lookup table ([build-its](https://build-its-inprogress.blogspot.com/2017/03/encoder-autocalibration-for-brushless.html)). The RLS AksIM-2 datasheet gives **±0.05° before installation, typically ±0.025° after self-calibration**, computes the ride height from the signal level to ±20 µm and flags a self-calibration run at over 300 rpm as invalid ([AksIM-2 datasheet](https://cdn.rls.si/documents/datasheets/MBD01_07_ds.pdf)). Apptronik's actuation test engineer owns "motor and encoder calibration" among the production test methods with pass/fail criteria; Figure's manufacturing software test engineer writes "end of line software for programming, calibrating, and verifying subassemblies". This template is that software.

### Test Purpose

The procedure writes three calibrations to one actuator and verifies each:

- **Encoder health** before anything is calibrated on it: signal level, ride height, error flag
- **Phase order and pole pairs** detected with the rotor free
- **Commutation offset** by the d-axis walk, run twice, agreement required; a lookup table written; the q-axis torque checked against expectation
- **Encoder self-calibration** run, then the accuracy verified against a reference table over one revolution
- **Joint zero** found against the hard stop, written and read back

![Left, the forward and backward d-axis walks as two nearly coincident lines of encoder electrical angle against commanded angle with the mean between them. Right, the residual after the constant offset as a cogging ripple of 1.77 electrical degrees peak-to-peak, the 128-point lookup table drawn through it, offset 138.07 electrical degrees and 99.7 % torque after the write.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-202a64c3-dba1-4e99-80bf-38453c126c16-1lmWncg1H7pbc7mbsYUNGwb1bo9NDu.png)

The mock actuator's d-axis walk: the rotor lags the field going forward and leads it coming back, by the same friction angle, so the mean of the two is the true relation; the constant part of that relation is the offset and the ripple is the cogging that the lookup table absorbs.

The framework mechanics on show are a calibration that writes to the DUT and then proves what it wrote, two independent runs required to agree as an aggregation with a limit, two curves in one measurement with the aggregations on one, string `==` validators on detection results, numbers recorded for the trend without a limit, and unit metadata the robot's bring-up reads later.

## Equipment & Setup

To run this calibration on an actuator line, the following are required:

- **The actuator's drive** over EtherCAT, on a real-time target or a bench controller with current control
- **Access to both encoders**, the rotor's magnetic ring and the output absolute encoder, over BiSS-C or the drive
- **A reference rotary table** or a reference encoder on the output, for the accuracy verification
- **A mechanical hard stop** on the output, at a known angle from the joint zero on the drawing
- **The Device Under Test (DUT)**: an assembled joint actuator with its production firmware, rotor free
- **A TofuPilot Framework procedure** to sequence the detections, compute the offsets, write them and verify them
- **The TofuPilot Dashboard** to trend the eccentricity before self-calibration, which is where a ring-gluing fixture drifting out of tolerance shows first

### Hardware Components

#### Drive and Encoders

Elmo and Kollmorgen drives run on Figure's benches; Synapticon's Circulo integrates the drive with the encoder interfaces and exposes the offset detection as CoE objects (phase order at 0x2003:5, offset at 0x2001, pole pairs detected in the same run). The rotor encoder modelled is the [RLS AksIM-2](https://cdn.rls.si/documents/datasheets/MBD01_07_ds.pdf), an on-axis magnetic absolute encoder to 20 bits with a self-calibration routine that fits and stores the eccentricity harmonics and reports pass/fail on its status line; its ride height, the gap between ring and head, is computed from the signal level and is the number the setup phase gates on. The output encoder is a second absolute device on the reducer output, 19 bits here, and is the one the joint zero is written against.

![Motor shaft with a mounted encoder ring and a readhead facing the rim](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/encoder-and-commutation-calibration-physical-2026-WcVTS2JE2edqlJuGp39DgND7j8s3el.webp)

The reference ring is secured to the shaft and the fixed readhead faces its engraved rim. Drive electronics and the encoder interface are not shown.

#### Reference Table

The accuracy sweep needs a truth the module's encoder is compared to: an Ideal Aerosmith rate table carrying the actuator, or a Heidenhain RCN or Renishaw RESOLUTE reference ring on the output. The vendor's ±0.025° typical after self-calibration is 90 arc-seconds; a reference at 1 arc-second or better keeps the fixture out of the measurement.

### Where the Limits Come From

No standard specifies an actuator calibration. The limits are the encoder and drive datasheets, plus the joint drawing:

| Test | Limit | Basis |
|---|---|---|
| Encoder signal level | ≥ 70 % | vendor status threshold |
| Ride height | 100 to 300 µm | vendor gap window for the ring size |
| Phase order, pole pairs | == UVW, == 21 | the motor's design |
| Offset repeatability | ≤ 1.5 °el between two runs | the walk's own noise floor plus margin |
| Lookup table peak-to-peak | ≤ 4 °el | cogging of a healthy stator; a magnet defect doubles it |
| Torque at offset | ≥ 99 % of expected | cos(8°) = 0.99: a residual offset error under 8 electrical degrees |
| Accuracy after self-cal | ≤ 120 arcsec max, first harmonic ≤ 20 arcsec | ±0.025° typical after self-cal, rounded up for the fixture; the first harmonic is the eccentricity the self-cal must have removed |
| Hard stop | 2.7 to 3.7° on the output encoder | 3.2° on the drawing, ±0.5° of assembly tolerance |

The eccentricity before self-calibration is recorded without a limit on purpose: it is the ring-gluing process, not the actuator, and a trend across a lot is how the fixture is caught drifting.

## Test Procedure

### Overview

The procedure maps the calibration onto the framework's three stages. The encoder health check lives in `setup:` so no calibration is stored on an encoder whose ring is mis-mounted. Disabling the drive lives in `teardown:` so the actuator is never left energised.

1. Setup: encoder signal level, ride height, error flag.
2. Main: phase order and pole pairs, rotor free.
3. Main: d-axis walk twice, offset, lookup table, torque check after the write.
4. Main: encoder self-calibration, accuracy sweep against the reference.
5. Main: hard stop, joint zero offset written and read back.
6. Teardown: drive disabled.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the actuator's calibration record.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  encoder_health.py
  phase_order.py
  commutation_offset.py
  encoder_self_cal.py
  output_zero.py
  release.py
plugs/
  cal_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-encoder-commutation-calibration). The `CalBench` plug is a mock of the drive, both encoders and the reference table together, synthesizing a healthy motor with a magnet ring 18 µm off-centre that self-calibration removes and a hard stop 3.2° from the joint zero, so the procedure runs end-to-end without a bench or an actuator connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Encoder and Commutation Calibration
version: 0.1.0
description: Encoder and commutation calibration of a humanoid joint actuator. Rotor encoder signal and ride height, phase order and pole-pair detection, electrical offset by a friction-cancelled d-axis walk run twice with a lookup table written, encoder self-calibration verified against a reference table, joint zero from the mechanical hard stop.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the actuator housing label"
    placeholder: "ACT-HIP-000000"
    pattern: "^ACT-[A-Z]{3}-\\d{6}$"
    default_value: "ACT-HIP-004813"
  part_number:
    default_value: "ACT-R100-60NM"
  batch_number:
    default_value: "WK-2026-37"

plugs:
  - name: Cal Bench
    description: "Actuator drive over EtherCAT, rotor and output absolute encoders, reference rotary table on the output (mock, one plug per bench)"
    python: plugs.cal_bench:CalBench
    key: bench

setup:
  - name: Encoder Health
    key: encoder_health
    python: phases.encoder_health
    measurements:
      - name: Signal Level
        key: signal_pct
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 70.0}
      - name: Ride Height
        key: ride_height_um
        unit: µm
        description: Magnet ring to readhead gap computed from the signal level; outside the window the ring is mis-mounted and the calibration will not hold over temperature.
        validators:
          - {operator: ">=", expected_value: 100.0}
          - {operator: "<=", expected_value: 300.0}
      - name: Encoder Error
        key: encoder_error
        validators:
          - {operator: "==", expected_value: false}

main:
  - name: Phase Order
    key: phase_order
    python: phases.phase_order
    measurements:
      - name: Phase Order
        key: phase_order
        validators:
          - {operator: "==", expected_value: UVW}
      - name: Pole Pairs
        key: pole_pairs
        validators:
          - {operator: "==", expected_value: 21}

  - name: Commutation Offset
    key: commutation_offset
    python: phases.commutation_offset
    depends_on: [phase_order]
    timeout: 2m
    measurements:
      - name: D-Axis Walk
        key: walk
        title: Mechanical angle read at each commanded electrical angle, forward and back
        x_axis:
          legend: Electrical angle
          unit: °
        y_axis:
          - legend: Forward
            key: forward
            unit: °
            aggregations:
              - type: offset_edeg
                unit: °
              - type: repeat_edeg
                unit: °
                validators:
                  - {operator: "<=", expected_value: 1.5}
              - type: lut_pp_edeg
                unit: °
                validators:
                  - {operator: "<=", expected_value: 4.0}
          - legend: Backward
            key: backward
            unit: °
      - name: Torque At Offset
        key: torque_at_offset_pct
        unit: "%"
        description: q-axis step after the offset is written, measured over expected; cos of the residual angle error, so 99 % is 8 electrical degrees off.
        validators:
          - {operator: ">=", expected_value: 99.0}

  - name: Encoder Self Cal
    key: encoder_self_cal
    python: phases.encoder_self_cal
    depends_on: [commutation_offset]
    measurements:
      - name: Eccentricity Before
        key: eccentricity_before_arcsec
        unit: arcsec
        description: First-harmonic error the ring's mounting eccentricity produced before self-calibration; recorded, the mounting tolerance is what it trends.
      - name: Self-Cal Status
        key: self_cal_status
        validators:
          - {operator: "==", expected_value: ok}
      - name: Accuracy
        key: accuracy
        title: Module encoder minus reference table over one revolution, after self-calibration
        x_axis:
          legend: Angle
          unit: °
        y_axis:
          - legend: Error
            key: error
            unit: arcsec
            aggregations:
              - type: max_abs_arcsec
                unit: arcsec
                validators:
                  - {operator: "<=", expected_value: 120.0}
              - type: first_harmonic_arcsec
                unit: arcsec
                validators:
                  - {operator: "<=", expected_value: 20.0}

  - name: Output Zero
    key: output_zero
    python: phases.output_zero
    depends_on: [encoder_self_cal]
    measurements:
      - name: Hard Stop
        key: hard_stop_deg
        unit: °
        description: Output encoder reading at the stop; the stop sits 3.2 deg from joint zero on the drawing, so a reading far from it is a mis-assembled stop or a wrong encoder.
        validators:
          - {operator: ">=", expected_value: 2.7}
          - {operator: "<=", expected_value: 3.7}
      - name: Zero Offset
        key: zero_offset_deg
        unit: °
      - name: Zero Offset Readback
        key: zero_offset_readback_ok
        validators:
          - {operator: "==", expected_value: true}

teardown:
  - name: Release
    key: release
    python: phases.release
```

Framework features to notice:

1. **Write, then prove.** The commutation offset is written and then a q-axis torque step is measured against expectation; the zero offset is written and read back. Each calibration phase ends with a measurement that would fail if the write had not landed.
2. **Two runs, one aggregation.** The d-axis walk runs twice; `repeat_edeg` is the disagreement between the two, with its own limit, so a noisy or slipping rotor fails on repeatability before it fails on torque.
3. **Recorded without a limit.** `offset_edeg`, `eccentricity_before_arcsec` and `zero_offset_deg` carry no validators; they are the numbers the dashboard trends across actuators.
4. **String `==` on detection results.** `phase_order == UVW` and `self_cal_status == ok` validate the drive's and the encoder's own verdicts as strings.
5. **Health before calibration.** The ride height gate in `setup:` refuses to calibrate a mis-mounted ring: a calibration stored on a ring outside its gap window holds on the bench and drifts in the robot.

### Encoder Health

The setup phase reads the rotor encoder's status line and stamps the encoder resolution onto the **unit metadata**:

```python filename="phases/encoder_health.py"
def encoder_health(measurements, bench, unit, log):
    """Setup: the rotor encoder's signal level and ride height before any
    calibration runs on it. A magnet ring mounted out of its gap window
    calibrates to a number and drifts with temperature."""
    st = bench.encoder_status()
    measurements.signal_pct = st["signal_pct"]
    measurements.ride_height_um = st["ride_height_um"]
    measurements.encoder_error = st["error"]
    unit.metadata["encoder_bits"] = st["bits"]
    log.info(f"Actuator {unit.serial_number}: encoder signal {st['signal_pct']} %, ride height {st['ride_height_um']} um")
```

### Phase Order

The drive pulses the phases in sequence and watches which way the encoder moves; the pole-pair count falls out of the same run. Both are validated before the offset search, because a swapped phase or a wrong pole-pair configuration makes the search converge on a plausible wrong number:

```python filename="phases/phase_order.py"
from utils.recipe import POLE_PAIRS


def phase_order(measurements, bench, log):
    """Phase order and pole-pair count from the drive's detection routine,
    rotor free. Swapped phases or a wrong pole-pair configuration make the
    offset search converge on nonsense; both are checked before it runs."""
    det = bench.detect_phase_order()
    measurements.phase_order = det["order"]
    measurements.pole_pairs = det["pole_pairs"]
    log.info(f"Phase order {det['order']}, {det['pole_pairs']} pole pairs (expected {POLE_PAIRS})")
```

### Commutation Offset

The rotor is locked to the d-axis at 6 A and the electrical angle is stepped through one electrical revolution in 16 steps, then back. Going forward the rotor lags the field by the friction angle; coming back it leads by the same amount; the mean of the two readings at each step is the friction-free relation. Its constant part is the offset, computed as a circular mean; its ripple is cogging, low-passed into a 128-point table. The whole thing runs twice and the two offsets must agree; then the offset and the table are written and a q-axis torque step is measured:

```python filename="phases/commutation_offset.py"
import numpy as np

from utils.recipe import CAL_STEPS_PER_EREV, D_AXIS_CURRENT_A, LUT_POINTS, POLE_PAIRS


def commutation_offset(measurements, bench, log):
    """Electrical offset by the d-axis walk: lock the rotor to the field,
    step the electrical angle forward through one electrical revolution
    and back, read the encoder at every step. The forward and backward
    readings are averaged to cancel friction; the remaining ripple is
    cogging and is low-passed away over the electrical cycle. Run twice
    and require agreement, then write the offset and a lookup table."""
    offsets = []
    for run in range(2):
        walk = bench.d_axis_walk(D_AXIS_CURRENT_A, CAL_STEPS_PER_EREV)
        e = np.array(walk["e_fwd"])
        m = 0.5 * (np.array(walk["m_fwd"]) + np.array(walk["m_bwd"])[::-1])
        # Electrical angle the encoder implies at each step, minus the
        # commanded field angle, is the offset; wrap and average.
        implied = (m * POLE_PAIRS) % 360.0
        diff = (e - implied + 180.0) % 360.0 - 180.0
        offsets.append(float(np.degrees(np.arctan2(np.mean(np.sin(np.deg2rad(diff))), np.mean(np.cos(np.deg2rad(diff)))))) % 360.0)
    offset = float(np.mean(offsets))
    repeat = float(abs((offsets[0] - offsets[1] + 180.0) % 360.0 - 180.0))

    # Residual per electrical step after the constant offset: the LUT.
    resid = (diff - (offset if offset < 180 else offset - 360.0) + 180.0) % 360.0 - 180.0
    lut = np.interp(np.linspace(0.0, 360.0, LUT_POINTS, endpoint=False), e, resid)
    bench.write_commutation(offset, lut.round(3).tolist())
    torque_pct = bench.torque_step_response()

    measurements.walk.x_axis = walk["e_fwd"]
    measurements.walk.y_axis.forward = walk["m_fwd"]
    measurements.walk.y_axis.backward = walk["m_bwd"][::-1]
    measurements.walk.y_axis.forward.aggregations.offset_edeg = offset
    measurements.walk.y_axis.forward.aggregations.repeat_edeg = repeat
    measurements.walk.y_axis.forward.aggregations.lut_pp_edeg = float(lut.max() - lut.min())
    measurements.torque_at_offset_pct = torque_pct
    log.info(f"Commutation offset {offset:.2f} edeg (runs {offsets[0]:.2f} / {offsets[1]:.2f}), LUT {lut.max() - lut.min():.2f} edeg p-p, q-axis torque {torque_pct:.1f} % of expected")
```

### Encoder Self Cal

The encoder's own routine runs (one turn at constant speed under 300 rpm, a harmonic fit of the eccentricity stored in the encoder), and its status is validated as a string. Then the reference table drives the output through one revolution and the module encoder is compared to it at every degree; the maximum error and the remaining first harmonic both carry limits, the second because it is exactly what the self-calibration must have removed:

```python filename="phases/encoder_self_cal.py"
import numpy as np

from utils.recipe import ACCURACY_ARCSEC_MAX


def encoder_self_cal(measurements, bench, log):
    """Run the encoder's built-in self-calibration (one turn, harmonic fit
    of the ring eccentricity), then verify against the reference table
    over one revolution. The vendor's typical figure after self-cal is
    +-0.025 deg; the sweep is what proves it on this unit."""
    cal = bench.self_calibrate()
    measurements.eccentricity_before_arcsec = cal["before_arcsec"]
    measurements.self_cal_status = cal["status"]
    sweep = bench.accuracy_sweep(360)
    err = np.array(sweep["error_arcsec"])
    measurements.accuracy.x_axis = sweep["angle_deg"]
    measurements.accuracy.y_axis.error = sweep["error_arcsec"]
    measurements.accuracy.y_axis.error.aggregations.max_abs_arcsec = float(np.abs(err).max())
    measurements.accuracy.y_axis.error.aggregations.first_harmonic_arcsec = float(2.0 * np.abs(np.fft.rfft(err))[1] / len(err))
    log.info(f"Eccentricity {cal['before_arcsec']:.0f} arcsec before self-cal; after, max |error| {np.abs(err).max():.0f} arcsec against the reference (limit {ACCURACY_ARCSEC_MAX:.0f})")
```

![Encoder error over one revolution: a dashed 124 arcsec first-harmonic sine for the eccentricity before self-calibration, and the measured error after it as a green trace within ±30 arcsec, the ±120 arcsec limits as dashed lines.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-d4254b9b-b18f-435e-ba64-62bf64a9dece-tA5EinZNvPsyXWZwo2k8RFFylNfeND.png)

The mock ring's 18 µm of eccentricity is 124 arc-seconds of first harmonic before self-calibration and gone after it; what remains is the second harmonic of the ring's ovality and the 21-per-revolution signature of the magnet poles, 31 arc-seconds at worst against a 120 limit.

### Output Zero

The output is driven slowly into its mechanical hard stop until the torque threshold; the output encoder's reading there, minus the stop's angle from joint zero on the drawing, is the zero offset. It is written to the drive and read back, and the reading at the stop is itself limited, because a stop far from 3.2° is a mis-assembled stop or the wrong encoder:

```python filename="phases/output_zero.py"
from utils.recipe import HARD_STOP_TORQUE_NM


def output_zero(measurements, bench, unit, log):
    """Joint zero: drive the output into its mechanical hard stop until the
    torque threshold, read the output encoder there, write the offset so
    the stop reads its design angle, read it back."""
    stop = bench.find_hard_stop(HARD_STOP_TORQUE_NM)
    design_stop_deg = 3.2  # the stop's angle from joint zero on the drawing
    offset = stop["stop_deg"] - design_stop_deg
    bench.write_zero_offset(offset)
    readback = bench.read_zero_offset()
    measurements.hard_stop_deg = stop["stop_deg"]
    measurements.zero_offset_deg = offset
    measurements.zero_offset_readback_ok = (readback == offset)
    unit.metadata["zero_offset_deg"] = round(offset, 4)
    log.info(f"Hard stop at {stop['stop_deg']:.4f} deg on the output encoder, zero offset {offset:+.4f} deg written and read back")
```

At robot level the same idea scales: Unitree's G1 has a calibration mode with the robot suspended and every joint in a zero-torque state, and a contact-constrained joint-offset calibration in the literature halved the foot-height error from 4.3 to 2.2 mm RMS. The per-actuator zero here is the input to that.

### Mock Plug

`CalBench` stands in for the drive, both encoders and the reference table. It synthesizes a true commutation offset of 137.4 °el, a friction lag of 1.8 °el each way and a cogging term of 0.9 °el at six times the electrical angle, a magnet ring with 18 µm of eccentricity on a 30 mm radius and a ride height of 210 µm, an accuracy after self-calibration of a second harmonic plus the 21-pole signature, and a hard stop at 3.2°. The torque step after the write returns cos of the residual offset error. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, the class speaks the drive's CoE objects through pysoem (the d-axis walk is a sequence of current-mode commands with the electrical angle forced; Synapticon and Elmo both expose it), reads the AksIM-2's status and runs its self-calibration over BiSS-C, and drives the reference table over its own interface. Keep the rotor free and unloaded for the offset detection, under 300 rpm for the self-calibration, and write the offsets to non-volatile memory with a read-back. The phases, measurements and limits stay the same.


### ARINC 429 LRU Acceptance Test
URL: https://www.tofupilot.com/templates/arinc-429-lru-acceptance-test
Automate a dual-bus avionics LRU acceptance test with two 28V supplies, an ARINC 429 analyzer, and a DMM, and trend the results to cut No-Fault-Found.


## Introduction

### The LRU and the ARINC 429 Bus

An **LRU** (line-replaceable unit) is a self-contained box in the avionics bay, swapped as a whole during maintenance and sent to a component shop for repair. Many LRUs sit on the aircraft's DC power distribution and report their status over **ARINC 429**, the standard digital databus for commercial avionics. A power distribution or bus control unit is a good example: it takes redundant 28 V DC feeds and transmits fault and status words to the flight computers and the central maintenance system.

![Close-up of an avionics line-replaceable unit: finned aluminium housing, two circular multi-pin connectors on the front face, and the twisted shielded pairs of the ARINC 429 buses plugged in.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-ff03c53a-cffe-4608-9c94-f60e68ec0687-X3Oa1xEkZnOPc8UuAD16wK5OMvwdsn.jpg)

A typical LRU: sealed aluminium housing, circular connectors carrying both 28 V buses and the ARINC 429 pairs, swapped as one part on the aircraft and tested as one part in the shop.

Two properties of this class of unit shape the whole test. First, it has **two segregated power inputs**, a primary and a backup bus, electrically and mechanically separated so the loss of one supply cannot take the unit down. Aircraft DC power is nominally **28 V DC**, produced by transformer-rectifier units from the 115 V AC buses. Second, it talks over ARINC 429, so the acceptance test has to speak that protocol to stimulate the unit and read it back.

ARINC 429 is a unidirectional, point-to-point databus: one transmitter drives up to twenty receivers on a twisted shielded pair, and a bidirectional link needs two buses. It runs at either **low speed (12.5 kbit/s)** or **high speed (100 kbit/s)**, and the two rates cannot share a bus. Each message is a **32-bit word** with five fields:

| Bits | Field | Purpose |
|---|---|---|
| 1–8 | Label | Octal code identifying the data type |
| 9–10 | SDI | Source/destination identifier |
| 11–29 | Data | The payload (BNR, BCD, or discrete) |
| 30–31 | SSM | Sign/status matrix (normal, failure warning, no-computed-data, functional test) |
| 32 | Parity | Odd parity over the word |

The differential signal swings ±10 V between the two legs, with defined high, null, and low bands and a specified rise/fall time (**1.5 ± 0.5 µs** at high speed, **10 ± 5 µs** at low speed). These are real, measurable pass/fail quantities on the bench.

![One ARINC 429 word captured on the bus: bipolar return-to-zero differential voltage swinging between +10 V, null and -10 V, 32 bits over 320 µs at 100 kbit/s, with the label, SDI, data, SSM and parity fields shaded.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-fc55266a-aefb-47b7-96e9-b45df6f23e04-119paQun7EofYMQhhiqgmKgDziN2vb.png)

Label 206 as the mock analyzer captures it: bipolar return-to-zero, every bit spends its first half at +10 V or -10 V and its second half at null, 32 bits in 320 µs at high speed, odd parity in bit 32.

### Purpose of the Test

When a removed unit arrives at the shop, the acceptance test per the OEM's CMM (component maintenance manual) has to prove three things before the unit is released on an EASA Form 1: it draws the right current on each bus, it transmits every expected ARINC 429 label correctly, and it fails over from one power bus to the other without dropping its output. This template builds that test as a structured, automated procedure so that every measured value, not just the pass/fail verdict, is recorded against the unit's serial number. The per-unit fingerprint:

- **Bus A and bus B current** at matched 28.0 V, and their mismatch
- **Label word error rate** and **response latency** on the ARINC 429 output
- **Bus waveform**: one captured word with its rise time and HI level
- **Output accuracy** of the decoded engineering-unit value
- **Failover**: word error rate and bus B current with bus A cut
- **Power-down state** of both buses, recorded by the teardown

The reason to keep the values matters as much as the test itself. A large share of avionics LRUs that come back to the shop test **No Fault Found**: field data puts avionics NFF as high as 20 percent of removals, and at the repair-bench level figures upward of 70 percent have been reported for some unit types. An NFF unit consumes a full bench cycle and returns to stock having taught the shop nothing, unless the measurements behind the pass are trended over time. ARINC 672 exists specifically as an industry guideline for reducing NFF, and structured test data is the raw material for it.

Beyond the avionics content, this template demonstrates four framework mechanics: **one driver class with one thin subclass per instrument address** for the two identical supplies, a **multi-dimensional measurement with custom aggregations** computed in Python and validated in YAML, a **derived measurement** combining two plugs, and a **teardown stage** that powers the unit down whatever happened before.

## Equipment & Setup

To run a dual-bus ARINC 429 acceptance test you will need:

* **Two identical programmable DC supplies**, one per power bus, each set to 28.0 V.
* **An ARINC 429 bus analyzer** with transmit ports (to stimulate the unit's receive labels), receive ports (to capture and time-tag the unit's transmit labels), and a waveform capture on the pair.
* **A bench digital multimeter (DMM)** for output voltage and insulation/dielectric checks.
* **The device under test (DUT)**: the LRU, wired to both supplies and to the analyzer's bus pairs.
* **A TofuPilot Framework procedure** to power the unit, drive the analyzer, evaluate every measurement against its CMM limits, and provide a global pass/fail status.
* **The TofuPilot Dashboard** to store results for traceability and analytics.

![Avionics line-replaceable unit secured to a fixture with connector harnesses](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/arinc-429-lru-acceptance-test-physical-2026-b1wAevenYZUKucH9wfXpRQgiKxEca1.webp)

The fixture secures the avionics unit and exposes its connector harnesses. The power supplies, ARINC analyzer and test computer are not shown.

### Why Two Identical Supplies

The unit under test has two power inputs by design, for redundancy, so the bench must feed each one from its **own independent supply**. Using two supplies of the same model, held at a matched 28.0 V, means any difference in current draw between bus A and bus B is a property of the unit, not an artifact of two mismatched instruments. It also lets the test prove the failover directly: cut supply A, and confirm the unit rides through on supply B without interrupting its ARINC 429 output.

A common bench choice is a pair of 3-channel programmable supplies (30 V / 3 A per channel, sub-0.1 percent programming accuracy, SCPI-controllable over LAN, USB, or RS-232), for example two Rigol DP832-class units. The important point for the software is not the model but that there are **two of them**, identical, each at a different address.

### ARINC 429 Analyzer

Production benches use a PCI/PXI or USB ARINC 429 interface with several transmit and receive channels (Astronics Ballard, AIM, or Holt evaluation boards). Word error rate and latency come from the channel's receive FIFO with hardware time tags; the waveform comes from an oscilloscope probe across the pair, or from the analyzer's own capture on interfaces that provide it. The template's mock synthesizes both from the same 32-bit encoder so the procedure runs without hardware.

## Test Procedure

### Overview

After connecting the unit to both supplies and to the analyzer pairs, the procedure runs:

1. Power both buses to 28.0 V and record per-bus current and their mismatch.
2. Measure the label 206 word error rate and response latency, capture one word and compute its rise time and HI level.
3. Decode an engineering-unit output and compare it to the commanded value.
4. Cut bus A, confirm the label error rate holds at zero on bus B alone, record bus B current.
5. Teardown: both supplies to 0 V, confirm both buses read below 0.5 V.
6. Validate every metric against its declared limits and stream the run to TofuPilot.

### Why TofuPilot Framework

TofuPilot Framework is a **YAML + Python** test framework built for hardware. You describe **what** the test does in a `procedure.yaml` file and **how** in small Python phase and plug files. The framework handles Python environment management, the operator UI, measurement validation and live charts, and process isolation between phases and each piece of equipment.

### Project Structure

```tree
procedure.yaml
phases/
  power_up.py
  bus_integrity.py
  output_accuracy.py
  bus_failover.py
  power_down.py
plugs/
  psu.py
  arinc.py
  dmm.py
utils/
  arinc_word.py
  waveform.py
README.md
pyproject.toml
```

The plugs are mocks: the supplies return fixed currents with noise, the analyzer synthesizes a word on the bus. Run it with `tofupilot run . --no-tui --no-kiosk --json`. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-arinc429-lru-acceptance).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI. It identifies the unit, declares the four instruments as **plugs**, lays out four main **phases** with their measurements and CMM limits, and a **teardown** phase:

```yaml filename="procedure.yaml"
name: ARINC 429 LRU Acceptance Test
version: 0.1.0
description: Functional acceptance test for a dual-bus avionics LRU, using two identical 28V supplies, an ARINC 429 bus analyzer, and a DMM.

unit:
  auto_identify: true
  serial_number:
    description: "Serial from the LRU nameplate"
    placeholder: "LRU000000"
    pattern: "^LRU\\d{6}$"
    default_value: "LRU000042"
  part_number:
    default_value: "622-1234-001"

plugs:
  - name: PSU Bus A
    description: 28V DC supply for redundant bus A (mock Rigol DP832-class, VISA 192.168.1.50)
    python: plugs.psu:PsuBusA
    key: psu_a
  - name: PSU Bus B
    description: 28V DC supply for redundant bus B (mock Rigol DP832-class, VISA 192.168.1.51)
    python: plugs.psu:PsuBusB
    key: psu_b
  - name: ARINC 429 Analyzer
    description: ARINC 429 bus analyzer for label capture, latency and waveform (mock)
    python: plugs.arinc:Arinc429Analyzer
    key: arinc
  - name: DMM
    description: Digital multimeter for output accuracy (mock)
    python: plugs.dmm:Dmm
    key: dmm

main:
  - name: Power Up
    key: power_up
    python: phases.power_up
    measurements:
      - name: Bus A Current
        key: bus_a_current
        unit: A
        validators:
          - {operator: "<=", expected_value: 1.8}
      - name: Bus B Current
        key: bus_b_current
        unit: A
        validators:
          - {operator: "<=", expected_value: 1.8}
      - name: Bus Current Mismatch
        key: bus_current_mismatch
        unit: A
        description: Absolute difference between the two bus currents at matched 28.0 V. Two identical supplies make this a property of the unit.
        validators:
          - {operator: "<=", expected_value: 0.2}

  - name: Bus Integrity
    key: bus_integrity
    python: phases.bus_integrity
    depends_on: [power_up]
    measurements:
      - name: Label Word Error Rate
        key: word_error_rate
        description: Fraction of label 206 words received corrupted or missing over the capture window.
        validators:
          - {operator: "<=", expected_value: 0.00001}
      - name: Response Latency
        key: response_latency
        unit: ms
        validators:
          - {operator: "<=", expected_value: 50}
      - name: Label 206 Waveform
        key: label_waveform
        title: ARINC 429 word on the bus (label 206, high speed)
        description: Differential voltage of one captured 32-bit word, bipolar return-to-zero at 100 kbit/s.
        x_axis:
          legend: Time
          unit: µs
        y_axis:
          - legend: Differential voltage
            key: v_diff
            unit: V
            aggregations:
              - type: rise_time_us
                unit: µs
                validators:
                  - {operator: ">=", expected_value: 1.0}
                  - {operator: "<=", expected_value: 2.0}
              - type: high_level_v
                unit: V
                validators:
                  - {operator: ">=", expected_value: 9.0}
                  - {operator: "<=", expected_value: 11.0}

  - name: Output Accuracy
    key: output_accuracy
    python: phases.output_accuracy
    depends_on: [bus_integrity]
    measurements:
      - name: Output Error
        key: output_error
        unit: "%"
        validators:
          - {operator: ">=", expected_value: -0.5}
          - {operator: "<=", expected_value: 0.5}

  - name: Bus Failover
    key: bus_failover
    python: phases.bus_failover
    depends_on: [output_accuracy]
    measurements:
      - name: Failover Word Error Rate
        key: failover_word_error_rate
        description: Label 206 error rate while bus A is cut and the unit rides on bus B alone.
        validators:
          - {operator: "<=", expected_value: 0.00001}
      - name: Bus B Current On Failover
        key: bus_b_failover_current
        unit: A
        description: Bus B carries the whole unit once bus A is gone.
        validators:
          - {operator: "<=", expected_value: 3.0}

teardown:
  - name: Power Down
    key: power_down
    python: phases.power_down
    measurements:
      - name: Bus A Off Voltage
        key: bus_a_off_voltage
        unit: V
        validators:
          - {operator: "<=", expected_value: 0.5}
      - name: Bus B Off Voltage
        key: bus_b_off_voltage
        unit: V
        validators:
          - {operator: "<=", expected_value: 0.5}
```

The limits shown are illustrative; on a real bench they come from the unit's CMM. Framework features to notice:

1. **Two plugs, one driver**: `psu_a` and `psu_b` point at two subclasses of the same driver class; each phase that needs both names them as two parameters.
2. **Multi-dimensional measurement with custom aggregations**: `label_waveform` stores the captured word as an x/y series; `rise_time_us` and `high_level_v` are free-named aggregations that Python computes on the series and the YAML validates against the bus spec.
3. **Derived measurement**: `bus_current_mismatch` is computed in the phase from two plug readings and validated like any other value.
4. **Teardown stage**: `power_down` runs after the main phases whatever their outcome, so a failed unit is never left energized on the bench.
5. **Unit serial pattern**: the nameplate serial is validated against `^LRU\d{6}$` before the run starts.

### Controlling Two Identical Instruments

Here is the question that trips up most people setting up a multi-instrument bench: **how do you drive two supplies of the same model without duplicating the driver code, and where do the addresses live?**

The framework instantiates a plug by calling the class with no arguments, so the cleanest pattern is **one shared driver class plus a thin subclass per address**. The driver logic is written once; each subclass just fixes which physical instrument it is:

```python filename="plugs/psu.py"
import random


class ProgrammablePsu:
    """Shared driver for a programmable DC supply at a given VISA address."""

    def __init__(self, address, idle_current_a):
        self.address = address
        self._idle_current_a = idle_current_a
        self._volts = 0.0
        self._extra_load_a = 0.0
        # self.inst = pyvisa.ResourceManager().open_resource(address)
        print(f"Connected PSU at {self.address}")

    def set_voltage(self, volts):
        # self.inst.write(f":SOUR:VOLT {volts}")
        self._volts = float(volts)

    def measure_voltage(self):
        # return float(self.inst.query(":MEAS:VOLT?"))
        return self._volts + random.gauss(0.0, 0.01)

    def measure_current(self):
        # return float(self.inst.query(":MEAS:CURR?"))
        if self._volts < 18.0:
            return 0.0
        return self._idle_current_a + self._extra_load_a + random.gauss(0.0, 0.01)

    def take_over_load(self, amps):
        """Mock only: the other bus dropped, this supply now carries its share."""
        self._extra_load_a = float(amps)


class PsuBusA(ProgrammablePsu):
    def __init__(self):
        super().__init__("TCPIP0::192.168.1.50::INSTR", idle_current_a=1.42)


class PsuBusB(ProgrammablePsu):
    def __init__(self):
        super().__init__("TCPIP0::192.168.1.51::INSTR", idle_current_a=1.39)
```

Each subclass is referenced as its own plug in the YAML, with its own key. The framework runs each plug in its own process and passes both into a phase by matching the parameter name to the plug key. So `psu_a` and `psu_b` are two independent connections sharing every line of driver code, and adding a third supply is three more lines.

### Power Up

Both buses go to 28.0 V and the phase records what each one draws. Two identical supplies at matched voltage make the per-bus comparison meaningful, so the mismatch is a measurement in its own right:

```python filename="phases/power_up.py"
def power_up(psu_a, psu_b, measurements, log):
    log.info("Powering redundant buses A and B to 28.0 V")
    psu_a.set_voltage(28.0)
    psu_b.set_voltage(28.0)

    i_a = psu_a.measure_current()
    i_b = psu_b.measure_current()
    log.info(f"Bus A {i_a:.2f} A, bus B {i_b:.2f} A")

    measurements.bus_a_current = i_a
    measurements.bus_b_current = i_b
    measurements.bus_current_mismatch = abs(i_a - i_b)
```

### Bus Integrity

The analyzer reports the label 206 error rate and the response latency, then captures one word on the pair. The phase pushes the raw samples into the multi-dimensional measurement and computes the two aggregations the YAML validates:

```python filename="phases/bus_integrity.py"
from utils.waveform import high_level, rise_time

LABEL = 0o206


def bus_integrity(arinc, measurements, log):
    log.info("Checking ARINC 429 label integrity and latency")
    measurements.word_error_rate = arinc.measure_word_error_rate(LABEL)
    measurements.response_latency = arinc.measure_latency_ms()

    t_us, v_diff = arinc.capture_word(LABEL)
    measurements.label_waveform.x_axis = t_us
    measurements.label_waveform.y_axis.v_diff = v_diff
    measurements.label_waveform.y_axis.v_diff.aggregations.rise_time_us = rise_time(t_us, v_diff)
    measurements.label_waveform.y_axis.v_diff.aggregations.high_level_v = high_level(t_us, v_diff)
    log.info(f"Rise time {rise_time(t_us, v_diff):.2f} us, HI level {high_level(t_us, v_diff):.2f} V")
```

The edge metrics live in a small utility module: the HI level is the mean of the settled top samples, the rise time is the 10-90 % crossing of the first null-to-HI edge:

```python filename="utils/waveform.py"
import numpy as np


def high_level(t_us, v_diff):
    """Mean of the settled HI half-bits (top 10 % of samples, noise-tolerant)."""
    v = np.asarray(v_diff)
    top = np.sort(v)[int(0.9 * v.size) :]
    return float(top.mean())


def rise_time(t_us, v_diff):
    """10-90 % rise time of the first null -> HI transition, in microseconds."""
    t = np.asarray(t_us)
    v = np.asarray(v_diff)
    high = high_level(t_us, v_diff)
    lo, hi = 0.1 * high, 0.9 * high
    i_lo = int(np.argmax(v >= lo))
    i_hi = int(np.argmax(v >= hi))
    return float(t[i_hi] - t[i_lo])
```

![Zoom on the first two bits of the captured word: the 10 to 90 percent rise time of 1.40 µs against the 1.5 ± 0.5 µs high-speed spec, and the 9.78 V HI level against the 9 to 11 V transmitter output limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-3c286a77-cfb5-4946-bf00-80667b624fc3-eyyKOdwPl8oC78dgdzLIWxptoj0MJQ.png)

The two aggregations on the captured word: 1.40 µs rise time inside the 1 to 2 µs window, 9.78 V HI level inside the 9 to 11 V transmitter output band. A driver with a failing output stage shows up here as a slow edge or a sagging HI level long before the label stops decoding.

### Output Accuracy

An engineering-unit output is decoded and compared to the commanded value:

```python filename="phases/output_accuracy.py"
def output_accuracy(dmm, measurements, log):
    log.info("Measuring output accuracy against reference")
    measurements.output_error = dmm.measure_output_error_pct()
```

### Bus Failover

The redundancy check is what a single-supply bench cannot do. With two independent supplies under software control, the phase drops bus A and confirms the unit holds its ARINC 429 output on bus B, and records how much current bus B carries alone:

```python filename="phases/bus_failover.py"
LABEL = 0o206


def bus_failover(psu_a, psu_b, arinc, measurements, log):
    log.info("Dropping bus A, confirming output holds on bus B")
    psu_a.set_voltage(0.0)
    psu_b.take_over_load(1.35)  # mock: bus B now carries the unit alone

    measurements.failover_word_error_rate = arinc.measure_word_error_rate(LABEL)
    i_b = psu_b.measure_current()
    log.info(f"Bus B {i_b:.2f} A with bus A cut")
    measurements.bus_b_failover_current = i_b
```

![Bar chart of supply currents: 1.42 A on bus A and 1.39 A on bus B at power up under the 1.8 A per-bus limit, then 0 A on bus A and 2.74 A on bus B during failover under the 3.0 A limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-67865a15-1dbc-4dc7-87ad-dd4e8789a462-9tphfAH5YmJgca4GTcLkFxD2zUAn3X.png)

Currents through the run: matched draw at power up (0.05 A mismatch against a 0.2 A limit), then the whole unit on bus B once bus A is cut. Because each supply is its own plug, cutting one and watching the other is a two-line phase, not a bench rewire.

### Power Down

The teardown always runs, even when a main phase failed, so no unit is left energized on the bench:

```python filename="phases/power_down.py"
def power_down(psu_a, psu_b, measurements, log):
    """Teardown: always runs, so the unit is never left powered on the bench."""
    log.info("Powering down both buses")
    psu_a.set_voltage(0.0)
    psu_b.set_voltage(0.0)
    measurements.bus_a_off_voltage = abs(psu_a.measure_voltage())
    measurements.bus_b_off_voltage = abs(psu_b.measure_voltage())
```

### Mock Plugs

The analyzer mock synthesizes the word it "captures": the 32 bits come from a small encoder that builds the label, SDI, data, SSM and odd parity in transmission order, then the bipolar return-to-zero waveform is shaped with first-order edges at 20 MS/s so the rise-time aggregation has something real to measure:

```python filename="utils/arinc_word.py"
def encode_word(label_octal, sdi, data, ssm):
    """Return the 32 bits of a word in transmission order (bit 1 first).

    Bits 1-8 label, transmitted MSB first (the label octal's high digit goes out
    first), bits 9-10 SDI, bits 11-29 data (LSB first), bits 30-31 SSM, bit 32
    odd parity over bits 1-31.
    """
    label = int(label_octal) & 0xFF
    bits = [(label >> (7 - i)) & 1 for i in range(8)]
    bits += [(sdi >> i) & 1 for i in range(2)]
    bits += [(data >> i) & 1 for i in range(19)]
    bits += [(ssm >> i) & 1 for i in range(2)]
    bits.append(0 if sum(bits) % 2 else 1)
    assert len(bits) == 32
    return bits
```

Running the procedure from its directory executes the five phases in order and validates every measurement:

```text
→ Phase: Power Up
→ Bus A 1.43 A, bus B 1.38 A
✓ Phase Power Up: PASS
→ Phase: Bus Integrity
→ Rise time 1.40 us, HI level 9.78 V
✓ Phase Bus Integrity: PASS
→ Phase: Output Accuracy
✓ Phase Output Accuracy: PASS
→ Phase: Bus Failover
→ Bus B 2.74 A with bus A cut
✓ Phase Bus Failover: PASS
→ Phase: Power Down
✓ Phase Power Down: PASS
✓ Run complete: PASS
```

To use it on a real bench, uncomment the pyvisa calls in `plugs/psu.py` and replace the analyzer mock with your interface's API; the phases, measurements, and limits stay exactly the same.


### BMS Cell Balancing Verification
URL: https://www.tofupilot.com/templates/bms-cell-balancing-verification
Passive balancing verification of a 16S BMS PCBA: off-state leakage per channel, bleed current in odd and even passes, command-to-channel attribution checked.


## Introduction

### Cell Balancing Overview

Cells in a series string drift apart. Passive balancing brings the high ones down: the AFE switches a bleed path across each cell, a resistor in series with a FET, and burns the excess charge as heat until the string is level again. On a 16S board that is sixteen bleed paths, each a resistor, a switch and a route back to the cell tap, and each one can be built wrong in a way no voltage-accuracy test sees: a resistor missing or of the wrong value, a FET open or, worse, stuck on, a command bit that lands on the wrong channel, firmware that never enables balancing at all. A stuck-on FET drains one cell in storage until the pack is over-discharged; a swapped channel balances the wrong cell for the life of the pack.

![A BMS PCBA close up: a row of bleed FETs in two lines along the board's edge, the AFE chip behind them, copper traces to the white sense connector.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-03ed3553-3321-4b3b-8b9b-251861467781-HJJL5VGtdqIW2vrU7UVZwsY7JNDJWY.jpg)

The bleed network of one board: a resistor and a discrete FET per channel, the AFE that switches them, and the copper back to the sense connector that every measurement in this template flows through.

Two vendor facts shape how the test has to be built. First, on an internal-balancing design there is no bleed resistor at all: the current flows through the cell input's own filter resistors and the AFE's on-die switch, and the BQ76952's internal balancing resistance is specified as **15 to 46 Ω** across temperature (28 Ω typical) with 100 mA absolute maximum per cell (see the [BQ76952 datasheet](https://www.ti.com/lit/ds/symlink/bq76952.pdf)); an external-FET design like the one in this template adds a discrete bleed resistor and gets ~80 mA through it plus ~16 mA through the internal path, ~96 mA total at 4 V per [TI SLUAA81A](https://www.ti.com/lit/an/sluaa81a/sluaa81a.pdf). Second, balancing is **duty-cycled by the AFE**: it is paused while the ADC scans the balanced cell and its neighbours, and for about 20 ms every second while the over- and under-voltage comparators are checked. A current read over a window shorter than that loop gives a number that depends on where the window landed.

### Test Purpose

The procedure records one balancing fingerprint per board:

- **Balancing configuration** as one object: the enable bit, the minimum cell voltage, the minimum delta and the maximum simultaneous cells, the four registers that silently disable balancing when wrong
- **Off-state leakage** on every channel with balancing commanded off, on the simulator's 250 µA range
- **Bleed current** on every channel at 4.0 V, integrated over one second, odd channels in one pass and even channels in the next
- **Command-to-channel attribution** from two sources: current on the commanded ammeters and none on the others, and the firmware's own active-cell mask
- **Balancing proven off** at teardown

![Sixteen bars of bleed current between 91.5 and 93.2 mA against the 80 to 110 mA limits, the median at 92.4 mA as a dashed line, the 97.5 mA DC value of the bleed network as a dotted line above, channel 11 in orange as the worst match.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-83db6053-0454-4266-9e01-f61eb4b26aef-P5j58vFfxlDvmYkh7zNpXZ0u93P75r.png)

The mock board's sixteen bleed paths: every channel within 1 % of the median, all of them about 95 % of the DC value the resistor network predicts, which is the AFE's duty cycling and not a defect. Channel 11's resistor sits at the top of its tolerance and is the one to watch across a batch.

The framework mechanics on show are JSON `==` on the configuration that gates the test and on the firmware's active-cell list, a multi-dimensional measurement with an absolute pair and a relative aggregation on the same curve, a boolean built from two independent sources, a disabled phase for commissioning the station, and `setup` and `teardown` stages with a `depends_on` chain on one bench.

## Equipment & Setup

To run this verification on a production line, the following are required:

- **A battery cell simulator** with one isolated, four-quadrant channel per cell input and a **per-channel ammeter** with a range for tens of milliamps and a range for microamps
- **A service link to the DUT** for the balancing command and the active-cell readback
- **A pogo-pin fixture** with Kelvin contacts on every cell-sense point
- **The Device Under Test (DUT)**: a 16S BMS PCBA with its production firmware
- **A TofuPilot Framework procedure** to sequence the passes, integrate the currents and validate the limits
- **The TofuPilot Dashboard** to trend every channel's bleed current across boards

### Hardware Components

#### Cell Simulator

The instrument is the ammeter. On a [Chroma 87001](https://www.chromausa.com/pdf/87001%20Battery%20Cell%20Simulator%20052023.pdf) each of the 16 channels sources and sinks up to 5 A and reads its own current on four ranges: 0 to 9 A, 0 to 5 A at ±1 mA, **0 to 500 mA at ±100 µA** for passive balancing, and **0 to 250 µA** for the off-state quiescent current of the cell-supervision circuit. Those last two ranges exist for exactly this test; the datasheet says so. The [Keysight SL1010A-803](https://www.keysight.com/us/en/products/hev-ev-grid-emulators-and-test-systems/sl1010a-series-scienlab-bms-environment.html) adds per-channel charge integration (±50 µC over 50 ms), which is what a PWM-balanced design needs to avoid aliasing. A [Pickering 41-752A](https://www.pickeringtest.com/portal/web/256/content/downloads/datasheets/41-43-752AD.pdf) can permit the balancing current but, on its simplified-sink variants, cannot measure it; its datasheet states the sink setting "may not represent precisely the current set by user".

![BMS board with two rows of balancing components and a cell-sense test harness](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/bms-cell-balancing-verification-distinct-2026-miPP6jhJgyHFrwqZCWXiFFrZVGYWdb.webp)

The fixture exposes the board's repeated balancing networks and cell-sense connections. The isolated cell simulator and per-channel current measurements are external.

#### Why Not a Resistor Divider

A ladder of resistors across one supply is the cheap way to fake sixteen cells, and it cannot test balancing. The moment one channel bleeds, its current collapses that rung of the divider, the neighbouring inputs move, the over-voltage comparator trips, and the AFE disables balancing; TI states that "any time an over-voltage condition occurs, cell balancing is immediately disabled", and on some parts the event sets a permanent-fail bit. Sixteen isolated channels that can sink current are the price of entry.

#### The Registers That Disable Balancing

A dozen configuration conditions switch balancing off silently. On a BQ76952 the `Cell Balance Min Cell V` register defaults to **3900 mV**, so a test that sets the simulator to 3.6 V commands sixteen channels and measures sixteen zeros; `Cell Balance Min Delta` defaults to a value that stops balancing when the cells are equal, which is precisely the condition of an accuracy test; the enable bit itself defaults off. The setup phase reads these four registers and validates them as one object before anything is commanded, and the pack goes to 4.0 V for the run.

### Where the Limits Come From

No standard specifies a balancing test; ISO 26262 makes a stuck-on bleed FET a failure mode against the over-discharge safety goal, which is why automotive AFEs ship a built-in CBFET diagnostic, and says nothing about an end-of-line current. The limits derive from the bleed network and the AFE:

| Test | Limit | Basis |
|---|---|---|
| Off-state leakage | ≤ 30 µA per channel | AFE input bias 8 to 12 µA; a stuck FET reads 90 000 |
| Bleed current | 80 to 110 mA | 97.5 mA DC from 4.0 V / 50 Ω + internal path, minus the duty cycle, ±10 % for 1 % resistors and FET spread |
| Channel match | within 5 % of the median | channels on one die match far better than the absolute spec; a single bad channel shows here |
| Attribution | exact | no tolerance applies |

For an internal-balancing design the current limit must come from the AFE's min/max on-resistance, not its typical: 15 to 46 Ω on a BQ76952 puts the silicon spread alone at ±22 % around the typical value, and a ±10 % window fails good boards. The relative match limit is what makes the test sensitive either way.

## Test Procedure

### Overview

The procedure maps the verification onto the framework's three stages. The configuration gate lives in `setup:` so no channel is commanded on a board whose firmware would ignore the command. Switching everything off lives in `teardown:` so no board leaves the fixture with a bleed path on.

1. Setup: balancing configuration as one object, pack to 4.0 V, all channels off.
2. Main: off-state leakage on every channel, 250 µA range.
3. Main: bleed current, odd pass then even pass, 1 s integration, attribution.
4. Main (disabled by default): 50 ms versus 1 s window on one channel.
5. Teardown: all off, proven off, cells parked.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-channel trending.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  balance_gate.py
  off_state_leakage.py
  bleed_current.py
  duty_cycle_check.py
  balancing_off.py
plugs/
  balance_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-bms-cell-balancing). The `BalanceBench` plug is a mock of the simulator's channels and ammeters and the DUT's service port together, synthesizing a board with 1 % bleed resistors, healthy FETs and a correct command-to-channel map, so the procedure runs end-to-end without a simulator or a board connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug with its cell count as constructor argument, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: BMS Cell Balancing Verification
version: 0.1.0
description: Passive balancing verification of a 16S BMS PCBA on a cell simulator. Balancing configuration read first, off-state leakage per channel on the 250 uA range, bleed current per channel in odd and even passes with command-to-channel attribution, currents integrated over 1 s against the AFE duty cycle.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the board data-matrix before it goes into the fixture"
    placeholder: "BMS16-00000"
    pattern: "^BMS16-\\d{5}$"
    default_value: "BMS16-01744"
  part_number:
    default_value: "PCB-BMS-16S-R3"
  batch_number:
    default_value: "LOT-2026-37"

plugs:
  - name: Balance Bench
    description: "16-channel cell simulator with per-channel ammeters and DUT UART (mock Chroma 87001-class, one plug per bench)"
    python: plugs.balance_bench:BalanceBench
    key: bench
    config:
      cell_count: 16

setup:
  - name: Balance Gate
    key: balance_gate
    python: phases.balance_gate
    measurements:
      - name: Balance Config
        key: balance_config
        description: "The four registers that silently disable balancing: enable bit, minimum cell voltage, minimum delta, maximum simultaneous cells."
        validators:
          - operator: "=="
            expected_value:
              cb_enabled: true
              min_cell_mv: 3900
              min_delta_mv: 0
              max_cells: 8

main:
  - name: Off State Leakage
    key: off_state_leakage
    python: phases.off_state_leakage
    measurements:
      - name: Off Leakage
        key: off_leakage
        title: Channel current with balancing commanded off, 250 µA range
        x_axis:
          legend: Channel
        y_axis:
          - legend: Leakage
            key: leak
            unit: µA
            aggregations:
              - type: max_ua
                unit: µA
                validators:
                  - {operator: "<=", expected_value: 30.0}
      - name: Active Cells Off
        key: active_cells_off
        description: Firmware's own view of which channels balance; must be empty here.
        validators:
          - operator: "=="
            expected_value: []

  - name: Bleed Current
    key: bleed_current
    python: phases.bleed_current
    depends_on: [off_state_leakage]
    timeout: 2m
    measurements:
      - name: Bleed
        key: bleed
        title: Bleed current per channel at 4.0 V, odd then even, 1 s integration
        x_axis:
          legend: Channel
        y_axis:
          - legend: Current
            key: current
            unit: mA
            aggregations:
              - type: min_ma
                unit: mA
                validators:
                  - {operator: ">=", expected_value: 80.0}
              - type: max_ma
                unit: mA
                validators:
                  - {operator: "<=", expected_value: 110.0}
              - type: match_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 5.0}
      - name: Attribution
        key: attribution_ok
        description: Every commanded channel and no other sourced current, and the firmware's active-cell mask agreed; a swapped bleed path or a wrong bitmap breaks it.
        validators:
          - {operator: "==", expected_value: true}

  - name: Duty Cycle Check
    key: duty_cycle_check
    python: phases.duty_cycle_check
    enabled: false
    depends_on: [bleed_current]
    measurements:
      - name: Short Window Spread
        key: short_window_spread_ma
        unit: mA
      - name: Long Window
        key: long_window_ma
        unit: mA

teardown:
  - name: Balancing Off
    key: balancing_off
    python: phases.balancing_off
    measurements:
      - name: Active Cells After
        key: active_cells_after
        validators:
          - operator: "=="
            expected_value: []
      - name: Max Leak After
        key: max_leak_after_ua
        unit: µA
        validators:
          - {operator: "<=", expected_value: 30.0}
```

Framework features to notice:

1. **The gate is a configuration object.** `balance_config` validates four registers with one `==` before the first channel is commanded; a board with `cb_enabled: false` fails in setup with the reason, not in main with sixteen zeros.
2. **An off-state measurement with a limit.** `off_leakage` is a multi-dimensional measurement with one aggregation, `max_ua <= 30`; it is the phase that finds a stuck-on FET.
3. **Absolute and relative on one curve.** `bleed` carries `min_ma` and `max_ma` from the network's DC value, and `match_pct` against the median; the third is the sensitive one.
4. **Empty-list equality.** `active_cells_off == []` and `active_cells_after == []` validate the firmware's own active-cell mask as a JSON value.
5. **`enabled: false`.** The duty-cycle check ships disabled; it runs once when the station's integration time is set.

### Balance Gate

The setup phase reads the balancing configuration, brings the pack to 4.0 V (above the 3900 mV minimum cell voltage below which nothing balances), commands every channel off, and stamps the test voltage onto the **unit metadata**:

```python filename="phases/balance_gate.py"
from utils.recipe import CELL_COUNT, TEST_MV


def balance_gate(measurements, bench, unit, log):
    """Setup: the configuration that silently disables balancing, read
    before anything is commanded. Min Cell V defaults to 3900 mV on a
    BQ76952: a test at 3.6 V would command sixteen channels and measure
    sixteen zeros. The pack goes to 4.0 V for the whole run."""
    cfg = bench.dut_balance_config()
    measurements.balance_config = cfg
    bench.set_all_cells_mv(TEST_MV)
    bench.dut_set_balancing([])
    unit.metadata["balance_test_mv"] = TEST_MV
    log.info(f"Board {unit.serial_number}: balancing enabled, min cell {cfg['min_cell_mv']} mV, {CELL_COUNT} channels at {TEST_MV} mV")
```

### Off State Leakage

Every channel commanded off, every ammeter on the 250 µA range. A healthy input draws the AFE's bias current, 8 to 12 µA on a BQ79616-class part; a bleed FET stuck on draws the full bleed current, three decades above the limit. The phase records the firmware's active-cell mask alongside, which must be empty:

```python filename="phases/off_state_leakage.py"
import numpy as np

from utils.recipe import CELL_COUNT


def off_state_leakage(measurements, bench, log):
    """Every channel with balancing commanded OFF, current on the 250 uA
    range. A stuck-on bleed FET reads tens of milliamps here and nowhere
    else; half the defect coverage of this template is this one phase."""
    bench.dut_set_balancing([])
    leak = np.array([bench.sim_channel_leak_ua(ch) for ch in range(1, CELL_COUNT + 1)])
    measurements.off_leakage.x_axis = list(range(1, CELL_COUNT + 1))
    measurements.off_leakage.y_axis.leak = leak.tolist()
    measurements.off_leakage.y_axis.leak.aggregations.max_ua = float(leak.max())
    measurements.active_cells_off = bench.dut_active_cells()
    log.info(f"Off-state leakage {leak.min():.1f}..{leak.max():.1f} uA, firmware reports active cells {bench.dut_active_cells()}")
```

![Sixteen bars of off-state current around 9 to 10 µA on a log axis, the 30 µA limit above them, and a dotted line at 90 000 µA labelled as what a stuck-on bleed FET reads.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-090d2f8a-4292-4eff-836c-9502c12895f8-KOkfLS8kWcpwvfNiR9dG1KYdV6WhAf.png)

The mock's sixteen off-state currents on a log axis, with the stuck-FET signature three decades above. A test that only measures the on state cannot see this defect, and it is the one with field consequences.

### Bleed Current

Two passes, odd channels then even channels. Turning all sixteen on at once is not an option: on the bq769x0 family TI states the part "is likely to be damaged" when every cell in a group balances together, and the BQ79616 refuses more than two consecutive FETs; the vendor's own diagnostic sequence is odd/even. Non-adjacent grouping keeps per-channel attribution, because each channel has its own ammeter, and cuts the passes from sixteen to two. Each current is integrated over one second so the AFE's duty cycling averages out. The phase checks, in both passes, that every commanded channel sources current, that no other channel does, and that the firmware's active-cell mask matches the command:

```python filename="phases/bleed_current.py"
import numpy as np

from utils.recipe import CELL_COUNT, EXPECTED_MA, INTEGRATION_S, PASSES


def bleed_current(measurements, bench, log):
    """Bleed current per channel in two passes, odd then even, never two
    adjacent channels at once. Each channel has its own ammeter, so the
    grouping keeps per-channel attribution: a command bit routed to the
    wrong channel shows as current on the wrong ammeter."""
    current = np.zeros(CELL_COUNT)
    attribution_ok = True
    for group in PASSES:
        bench.dut_set_balancing(group)
        if sorted(bench.dut_active_cells()) != sorted(group):
            attribution_ok = False
        for ch in range(1, CELL_COUNT + 1):
            i_ma = bench.sim_channel_current_ma(ch, INTEGRATION_S)
            if ch in group:
                current[ch - 1] = i_ma
            elif i_ma > 1.0:
                # Current on a channel that was not commanded: a swap.
                attribution_ok = False
                log.warning(f"Channel {ch} sources {i_ma:.1f} mA while not commanded")
        bench.dut_set_balancing([])

    median = float(np.median(current))
    measurements.bleed.x_axis = list(range(1, CELL_COUNT + 1))
    measurements.bleed.y_axis.current = current.tolist()
    measurements.bleed.y_axis.current.aggregations.min_ma = float(current.min())
    measurements.bleed.y_axis.current.aggregations.max_ma = float(current.max())
    measurements.bleed.y_axis.current.aggregations.match_pct = float(100.0 * (np.abs(current - median) / median).max())
    measurements.attribution_ok = attribution_ok
    worst = int(np.abs(current - median).argmax()) + 1
    log.info(f"Bleed {current.min():.1f}..{current.max():.1f} mA against {EXPECTED_MA:.1f} mA expected, median {median:.1f}, worst match channel {worst}, attribution {'ok' if attribution_ok else 'BROKEN'}")
```

The swap case is the argument for this structure. Commanding all channels and measuring the total current gives the same number whether cell 5's command bit drives cell 5 or cell 6; the AFE's built-in CBFET diagnostic cannot see it either, because the IC's view of "cell 5" is self-consistent whatever the PCB routed there. Only an external ammeter that knows the physical channel catches it.

### Duty Cycle Check

Disabled by default. It reads one channel ten times with a 50 ms window and once with the 1 s window, and reports the spread of the short readings. Run it when the station's integration time is chosen, not on every board.

```python filename="phases/duty_cycle_check.py"
from utils.recipe import INTEGRATION_S


def duty_cycle_check(measurements, bench, log):
    """Disabled by default. Reads one channel with a 50 ms window and with
    the 1 s window to show how much the AFE's duty-cycling moves a short
    reading; run it when the integration time of the station is set."""
    bench.dut_set_balancing([1])
    short = [bench.sim_channel_current_ma(1, 0.05) for _ in range(10)]
    long = bench.sim_channel_current_ma(1, INTEGRATION_S)
    bench.dut_set_balancing([])
    spread = max(short) - min(short)
    measurements.short_window_spread_ma = float(spread)
    measurements.long_window_ma = long
    log.info(f"50 ms windows spread {spread:.1f} mA on channel 1; 1 s window {long:.1f} mA")
```

![Instantaneous bleed current as a 97.5 mA square wave with short dropouts every 12.5 ms and a 20 ms gap before the one-second mark; three shaded 50 ms windows read 94, 94 and 56 mA depending on where they land, the 1 s mean at 91.7 mA as a dashed line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-e1c0cb95-af18-4f3b-bf5e-a2a499b00b1b-HA4FzSTx1KN6dbNqJ3UzSSGHeGWeW9.png)

Why the window is one second: the AFE suspends balancing for each ADC scan of the balanced cell and its neighbours, and for about 20 ms every second while it checks the over- and under-voltage comparators. Two 50 ms windows read 94 mA; the one that lands on the comparator pause reads 56 mA. Both are the same healthy channel. A limit derived from the DC equation and applied to a short reading produces a "flaky" balancing test that fails good boards.

### Balancing Off

The teardown phase commands every channel off, reads the firmware's mask and the worst off-state current again, and parks the cells at 3.6 V:

```python filename="phases/balancing_off.py"
from utils.recipe import CELL_COUNT


def balancing_off(measurements, bench, log):
    """Teardown: every bleed path off and proven off, cells back to 3.6 V."""
    bench.dut_set_balancing([])
    measurements.active_cells_after = bench.dut_active_cells()
    measurements.max_leak_after_ua = max(bench.sim_channel_leak_ua(ch) for ch in range(1, CELL_COUNT + 1))
    bench.set_all_cells_mv(3600)
    log.info(f"Balancing off, active cells {bench.dut_active_cells()}, cells parked at 3600 mV")
```

### Mock Plug

`BalanceBench` stands in for the simulator's channels and ammeters and the DUT's service port because the ammeters read what the DUT switches, and plugs run in separate processes. Its bleed resistors are 50 Ω ± 0.4 %, channel 11's at +0.9 %; the internal path is 2 × 100 Ω of filter plus a 28 Ω on-die switch with 8 % spread; the FETs add 50 mΩ; the off-state current is the AFE's 9.5 µA input bias, and a channel commanded on while it should be off would add 90 mA to it. A current read over one second is 95 % of the DC value, the mock's stand-in for the duty cycling; a 50 ms read scatters by ±8 %. The command-to-channel map is the identity. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, the class speaks SCPI to the simulator (`MEAS:CURR` per channel with the range selected explicitly, 500 mA on and 250 µA off) and pyserial to the DUT for the balancing command and the `CB_ACTIVE_CELLS` readback. Integrate over at least one second or slow the AFE's cell-balance loop, keep balancing off during any voltage-accuracy phase (eight balancing cells bias the top cell reading by 5 mV through 20 Ω filters), and for an active-balancing design measure the cell-side current and its direction, not the efficiency, which is a characterization measurement. The phases, measurements and limits stay the same.


### Reaction Wheel Acceptance Test
URL: https://www.tofupilot.com/templates/reaction-wheel-acceptance-test
Reaction wheel acceptance with TofuPilot Framework: wear-in drag, static and dynamic unbalance, bus power, coast-down friction and exported force sweep.

## Introduction

### Reaction Wheel Acceptance Overview

A reaction wheel is a flywheel on a pair of ball bearings, driven by a brushless motor, and it is the one mechanism on a small satellite that runs continuously for the whole mission. Every unit that leaves the vendor's line is accepted on the same set of numbers: how much the bearings drag once they have settled, how well the rotor is balanced, how much bus power it draws at speed, and how much force and torque it exports into the structure while it spins. The last one is the micro-vibration budget of the spacecraft; a wheel that shakes the optical bench at 100 Hz is a wheel the payload cannot use, whatever its momentum rating says.

![Close-up of a reaction wheel can with the lid removed, the flywheel rim visible inside, one connector on the side](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-c891930c-b821-4483-b6fc-59fece70cb9c-j6RXijnrnayhM2HXQRYN4FgpfQ6bGL.jpg)

The unit under test with its lid off: a rim-heavy flywheel on two bearings in a sealed can, one connector for power and telemetry. Everything the acceptance test measures is a property of the rotor and the two bearings.

The numbers are small. A 0.05 Nms-class wheel draws a few watts, its bearing drag is a few tenths of a milli-newton-metre, and its residual unbalance is a fraction of a gram-millimetre. Blue Canyon publishes a static unbalance under 1.2 g·mm and a dynamic unbalance under 20 g·mm² for its small wheels and ships the measured pair per serial (see [Blue Canyon's reaction wheel data](https://www.bluecanyontech.com/components)); NewSpace Systems quotes 4 W at 3000 rpm and 6 W at 6500 rpm steady state on a comparable unit. The force the unbalance exports rises with the square of the wheel speed, so a rotor at 0.35 g·mm pushes about 140 mN on the mounting plate at 6000 rpm. Reading that on a dynamometer that resolves 3 mN takes an isolated table and a charge amplifier, and the sweep is the part of the test that tells a good bearing from a damaged one.

### Test Purpose

The procedure records one acceptance record per wheel:

- **Firmware, bearing temperature at rest, and the serial's cumulative vibration log** against the bearing's qualification budget, before anything spins
- **A one-hour wear-in** through the speed profile with bearing drag and temperature logged, the settled drag validated
- **Static and dynamic unbalance** from the dynamometer against the vendor's per-serial acceptance figures
- **Bus power at 3000 and 6000 rpm**, then the **bearing friction torque** from a coast-down with the drive disabled
- **Exported force over a 500 to 6000 rpm sweep**, validated against an absolute ceiling and against the speed-squared law a balanced rotor follows

![Peak exported radial force in mN against wheel speed from 500 to 6000 rpm, green dots rising along a parabola from 6 to 173 mN, an orange dashed speed-squared fit under them, a red dashed 200 mN limit line at the top.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-dfcf438c-6d10-442c-9131-1cd8bdc68db1-sg1O9kIrlnCVzqodJfiMSFiWTkzt0K.png)

The mock wheel on the dynamometer: 173 mN at 6000 rpm against a 200 mN ceiling, and 5.9 mN worst deviation from the speed-squared law fitted to the sweep. A rotor that is simply out of balance moves the whole parabola up; a bearing with a damaged race adds a ball-pass line that does not scale with speed squared and shows as a bump the deviation aggregation catches.

The framework mechanics on show are a per-serial consumable budget validated in `setup:` and written to the unit metadata, a progress component on a time-scaled long phase with a `timeout`, two curves in one measurement each with its own aggregation and limit, a physical constant derived from a curve, a deviation-from-model aggregation next to an absolute one, and a `teardown:` that stops the wheel whatever happened.

## Equipment & Setup

To accept reaction wheels on a production bench, the following are required:

- **A multicomponent dynamometer** under the wheel's mounting plate, with a charge amplifier and a DAQ that covers 3 to 300 Hz
- **An isolated table** that keeps the building out of the force measurement
- **A bus power analyzer** on the 28 V supply and the wheel's own telemetry link for speed, current and bearing temperature
- **The Device Under Test (DUT)**: a 0.05 Nms-class reaction wheel, ±6000 rpm, 10 mNm
- **A TofuPilot Framework procedure** to sequence the wear-in, the balance reading, the power and coast-down, and the sweep
- **The TofuPilot Dashboard** to keep the vibration-exposure log per serial and trend drag, unbalance and exported force across the lot

### Hardware Components

#### Dynamometer

The [Kistler 9119AA1](https://www.kistler.com/INT/en/cp/multicomponent-dynamometers-9119aa1/P0000330) is the reference for wheel micro-vibration: a three-component force plate with a **threshold under 2 mN** on Fx and Fy, three moment channels, and a natural frequency high enough that the 3 to 300 Hz band of interest sits well below it. A **5070A charge amplifier** conditions the six piezo channels and the DAQ samples at 3200 S/s. The same plate reads the once-per-rev force and moment at balancing speed, which is where the static and dynamic unbalance come from, and the broadband force during the sweep. Anything that resolves a few mN at 100 Hz on a stiff plate does the job; what does not work is an accelerometer on the housing, because it reads the wheel's own response rather than what the wheel exports.

![Enclosed reaction wheel bolted to an adapter on a test platform](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/reaction-wheel-acceptance-test-physical-2026-0yhOSgmGxlJHZWu0hAU4kM2F3OXxtf.webp)

The reaction wheel is rigidly secured through an adapter bracket. Its cable exits the housing connector; power and measurement instruments are not shown.

#### Wheel Telemetry and Bus Power

The wheel reports speed, motor current and bearing temperature over its serial link at a few hertz; the procedure reads all three during the wear-in and at each step of the sweep. A Yokogawa WT-class analyzer on the 28 V bus gives the steady-state power at each speed, the number the spacecraft's power budget uses. Bearing temperature is the second output of the wear-in: a bearing whose lubricant is distributed runs cooler, and a bearing that climbs past the limit during an hour at 6000 rpm is not going to survive a mission.

#### Wear-In and the Vibration Log

SMC-S-016 lists wear-in as a required acceptance test for moving mechanical assemblies, and a wheel's bearings need the hour through the speed profile before the drag, the power and the coast-down mean anything. The second bookkeeping item is the cumulative vibration exposure of the bearings: the wheel's ICD caps how many seconds of acceptance vibration the bearings may see on the ground so that ground plus launch stays under what the bearing was qualified to. The procedure reads that log per serial in setup and refuses to proceed on a wheel that has already spent its allowance. ECSS-E-ST-10-03C leaves the micro-vibration characterisation to the customer's requirements, so the exported-force limits here are the wheel's own, not a standard's.

### Where the Limits Come From

The unbalance limits are the vendor's published acceptance figures for this wheel class: 1.2 g·mm static and 20 g·mm² dynamic. The settled drag limit of 0.30 mNm, the 0.35 mNm friction torque from the coast-down, and the 4.5 W and 7.5 W power ceilings sit above the vendor's steady-state curve with margin for a warm bench. The 200 mN exported-force ceiling is what a rotor at the unbalance limit pushes at full speed plus the bearing's contribution, and the 15 mN deviation limit is about five times the dynamometer's resolution. A customer sets the exported-force numbers from the spacecraft's micro-vibration budget, then tightens the deviation limit from the distribution the dashboard shows after a few dozen wheels.

## Test Procedure

### Overview

The procedure maps the acceptance test onto the framework's three stages. Identification and the vibration budget live in `setup:` so a wheel that fails the gate never spins. Stopping the wheel and reading the bearing temperature live in `teardown:` so the bench is safe to unload whether the test passed or not.

1. Setup: firmware, bearing temperature at rest, vibration log against the budget.
2. Main: one-hour wear-in through 1000, 3000, +6000 and -6000 rpm, drag and temperature logged.
3. Main: static and dynamic unbalance from the dynamometer.
4. Main: bus power at 3000 and 6000 rpm, coast-down from 6000 rpm, friction torque from the deceleration.
5. Main: exported force at each speed from 500 to 6000 rpm, checked against the speed-squared law.
6. Teardown: wheel stopped, bearing temperature read.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-serial trending across the lot.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  run_in.py
  balance.py
  power_and_friction.py
  exported_force.py
  stop.py
plugs/
  wheel_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-reaction-wheel-acceptance). The `WheelBench` plug is a mock of the wheel's telemetry link, the power analyzer and the dynamometer together, synthesizing a healthy wheel with 0.35 g·mm of static unbalance, bearings that settle from 0.31 to 0.19 mNm during the wear-in, and an exported force that follows the speed-squared law with the bearing's ball-pass harmonic on top, so the procedure runs end-to-end without a wheel or a dynamometer connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Reaction Wheel Acceptance Test
version: 0.1.0
description: Acceptance test of a smallsat reaction wheel. Telemetry and vibration-exposure log in setup, time-scaled one-hour wear-in with bearing drag and temperature, static and dynamic unbalance on the dynamometer, bus power at two speeds and bearing friction from a coast-down, exported force over a speed sweep against the speed-squared law.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the wheel's housing label"
    placeholder: "RW-050-0000"
    pattern: "^RW-050-\\d{4}$"
    default_value: "RW-050-0312"
  part_number:
    default_value: "RW-050-28V"
  batch_number:
    default_value: "LOT-2026-09"

plugs:
  - name: Wheel Bench
    description: "Wheel telemetry over RS-422, bus power analyzer, Kistler dynamometer under the mounting plate (mock, one plug per bench)"
    python: plugs.wheel_bench:WheelBench
    key: bench

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Firmware Version
        key: firmware_version
        validators:
          - {operator: matches, expected_value: "^1\\.\\d+\\.\\d+$"}
      - name: Bearing Temperature Start
        key: bearing_temp_start_c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 18.0}
          - {operator: "<=", expected_value: 26.0}
      - name: Vibration Log
        key: vibration_log_s
        unit: s
        description: Cumulative acceptance vibration already on this serial; the bearing's qualification budget for acceptance plus launch is finite and the ICD says so.
        validators:
          - {operator: "<=", expected_value: 1800.0}

main:
  - name: Run In
    key: run_in
    python: phases.run_in
    timeout: 2h
    ui:
      components:
        - key: run_in_progress
          type: progress
          label: "Wear-in"
          description: "60 min through 1000, 3000, +6000, -6000 rpm"
          default_value: 0
          max: 100
    measurements:
      - name: Wear-In
        key: wear_in
        title: Bearing drag torque and temperature through the wear-in
        x_axis:
          legend: Time
          unit: min
        y_axis:
          - legend: Drag
            key: drag
            unit: mNm
            aggregations:
              - type: final_mnm
                unit: mNm
                validators:
                  - {operator: "<=", expected_value: 0.30}
          - legend: Bearing temperature
            key: temperature
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 45.0}

  - name: Balance
    key: balance
    python: phases.balance
    depends_on: [run_in]
    measurements:
      - name: Static Unbalance
        key: static_unbalance_gmm
        unit: g·mm
        description: Vendor acceptance figure for this wheel class, shipped per serial.
        validators:
          - {operator: "<=", expected_value: 1.2}
      - name: Dynamic Unbalance
        key: dynamic_unbalance_gmm2
        unit: g·mm²
        validators:
          - {operator: "<=", expected_value: 20.0}

  - name: Power And Friction
    key: power_and_friction
    python: phases.power_and_friction
    depends_on: [balance]
    timeout: 15m
    measurements:
      - name: Power At 3000 rpm
        key: power_3000_w
        unit: W
        validators:
          - {operator: "<=", expected_value: 4.5}
      - name: Power At 6000 rpm
        key: power_6000_w
        unit: W
        validators:
          - {operator: "<=", expected_value: 7.5}
      - name: Coast-Down
        key: coast_down
        title: Speed after the drive is disabled at 6000 rpm
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Speed
            key: speed
            unit: rpm
            aggregations:
              - type: friction_mnm
                unit: mNm
                validators:
                  - {operator: "<=", expected_value: 0.35}
              - type: duration_s
                unit: s
                validators:
                  - {operator: ">=", expected_value: 120.0}

  - name: Exported Force
    key: exported_force
    python: phases.exported_force
    depends_on: [power_and_friction]
    timeout: 15m
    measurements:
      - name: EFT
        key: eft
        title: Peak exported radial force, 3 to 300 Hz, against wheel speed
        x_axis:
          legend: Speed
          unit: rpm
        y_axis:
          - legend: Force
            key: force
            unit: mN
            aggregations:
              - type: max_mn
                unit: mN
                validators:
                  - {operator: "<=", expected_value: 200.0}
              - type: max_deviation_mn
                unit: mN
                validators:
                  - {operator: "<=", expected_value: 15.0}

teardown:
  - name: Stop
    key: stop
    python: phases.stop
    measurements:
      - name: Bearing Temperature End
        key: bearing_temp_end_c
        unit: °C
        validators:
          - {operator: "<=", expected_value: 45.0}
```

Framework features to notice:

1. **A consumable budget in setup.** `vibration_log_s <= 1800` is not a property of the wheel's performance; it is how much of the bearing's ground allowance this serial has spent. It gates the run and lands on the unit metadata so the dashboard shows it per serial.
2. **Progress on a long phase.** The wear-in is an hour on real hardware; the `progress` component and the `timeout: 2h` make the phase visible and bounded without any UI code.
3. **Two curves, two limits.** `wear_in` records drag and bearing temperature against time; each curve carries its own aggregation, `final_mnm` on one and `max_c` on the other, with its own validator.
4. **A constant from a curve.** `coast_down.friction_mnm` is the deceleration slope of the first 60 s times the rotor's inertia; the framework stores the curve and validates the derived number.
5. **Model deviation next to an absolute.** `eft.max_mn <= 200` bounds what the wheel exports; `max_deviation_mn <= 15` bounds how far the sweep departs from the speed-squared law a balanced rotor obeys, which is what a bearing defect changes.

### Identify

The setup phase reads the firmware and the bearing temperature at rest over the telemetry link, then reads the serial's cumulative vibration log and writes it to the **unit metadata**. A wheel that has already spent its acceptance vibration allowance fails here and never spins:

```python filename="phases/identify.py"
from utils.recipe import VIBRATION_LIFE_S


def identify(measurements, bench, unit, log):
    """Setup: telemetry link, firmware, bearing temperature at rest, and
    the serial's cumulative vibration log against the bearing budget; a
    wheel that has already spent its acceptance vibration allowance is
    not re-vibrated whatever the sequence says."""
    ident = bench.identify()
    measurements.firmware_version = ident["firmware"]
    measurements.bearing_temp_start_c = ident["bearing_temp_c"]
    vib = bench.vibration_log_s()
    measurements.vibration_log_s = vib
    unit.metadata["vibration_exposure_s"] = vib
    log.info(f"Wheel {unit.serial_number}: fw {ident['firmware']}, bearings at {ident['bearing_temp_c']} C, {vib:.0f} s of vibration logged against a {VIBRATION_LIFE_S:.0f} s budget")
```

### Run In

One call returns the hour of wear-in as three lists; the phase converts time to minutes, records drag and bearing temperature as two curves of one measurement, and computes the settled drag as the mean of the last five minutes:

```python filename="phases/run_in.py"
import numpy as np

from utils.recipe import RUN_IN_MINUTES, RUN_IN_SPEEDS_RPM


def run_in(measurements, bench, ui, log):
    """Wear-in per SMC-S-016: the wheel through its speed profile for an
    hour, bearing drag and temperature logged. The drag at the end is
    the number the coast-down and the power test are judged against."""
    cap = bench.run_in(RUN_IN_SPEEDS_RPM, RUN_IN_MINUTES)
    ui.run_in_progress = 100
    drag = np.array(cap["drag_mnm"])
    t = np.array(cap["time_s"])
    final = float(drag[t >= t[-1] - 300.0].mean())

    measurements.wear_in.x_axis = (t / 60.0).round(2).tolist()
    measurements.wear_in.y_axis.drag = cap["drag_mnm"]
    measurements.wear_in.y_axis.drag.aggregations.final_mnm = final
    measurements.wear_in.y_axis.temperature = cap["bearing_temp_c"]
    measurements.wear_in.y_axis.temperature.aggregations.max_c = float(max(cap["bearing_temp_c"]))
    log.info(f"Wear-in {RUN_IN_MINUTES:.0f} min: drag {drag[0]:.3f} -> {final:.3f} mNm, bearings to {max(cap['bearing_temp_c']):.1f} C")
```

### Balance

The dynamometer at a fixed balancing speed gives the once-per-rev force and moment, which are the static and dynamic unbalance. The two scalars go straight against the vendor's acceptance figures:

```python filename="phases/balance.py"
def balance(measurements, bench, log):
    """Static and dynamic unbalance from the dynamometer at balancing
    speed. The vendor ships these two numbers per serial; the limits are
    the vendor's own acceptance figures."""
    bal = bench.balance_measurement()
    measurements.static_unbalance_gmm = bal["static_gmm"]
    measurements.dynamic_unbalance_gmm2 = bal["dynamic_gmm2"]
    log.info(f"Static {bal['static_gmm']:.2f} g-mm, dynamic {bal['dynamic_gmm2']:.1f} g-mm2")
```

### Power And Friction

Bus power at 3000 and 6000 rpm from the analyzer, then the drive is disabled at 6000 rpm and the speed logged until 500 rpm. The slope of the first 60 s times the rotor's inertia is the bearing friction torque at high speed, the number that separates a dry or damaged bearing from a good one:

```python filename="phases/power_and_friction.py"
import numpy as np

from utils.recipe import COAST_DOWN_FROM_RPM, MOMENTUM_NMS_AT_MAX, MAX_SPEED_RPM, POWER_W_MAX


def power_and_friction(measurements, bench, log):
    """Steady-state bus power at two speeds, then a coast-down from full
    speed with the drive disabled: the deceleration times the inertia is
    the bearing friction torque, the number that tells a dry or a
    damaged bearing from a good one."""
    for rpm in sorted(POWER_W_MAX):
        bench.spin_to(rpm)
        p = bench.steady_power_w(rpm)
        setattr(measurements, f"power_{int(rpm)}_w", p)
        log.info(f"{rpm:.0f} rpm: {p:.2f} W on the bus")

    bench.spin_to(COAST_DOWN_FROM_RPM)
    coast = bench.coast_down(COAST_DOWN_FROM_RPM)
    t = np.array(coast["time_s"])
    w = np.array(coast["speed_rpm"]) * 2 * np.pi / 60.0
    inertia = MOMENTUM_NMS_AT_MAX / (MAX_SPEED_RPM * 2 * np.pi / 60.0)
    # Friction torque at high speed from the first 60 s of the coast-down.
    m = t <= 60.0
    slope, _ = np.polyfit(t[m], w[m], 1)
    friction_mnm = float(-slope * inertia * 1000.0)

    measurements.coast_down.x_axis = coast["time_s"]
    measurements.coast_down.y_axis.speed = coast["speed_rpm"]
    measurements.coast_down.y_axis.speed.aggregations.friction_mnm = friction_mnm
    measurements.coast_down.y_axis.speed.aggregations.duration_s = float(t[-1])
    log.info(f"Coast-down from {COAST_DOWN_FROM_RPM:.0f} rpm: {t[-1]:.0f} s to 500 rpm, friction {friction_mnm:.3f} mNm at high speed")
```

![Left, bearing drag torque in green falling from 0.31 to 0.20 mNm over 60 minutes of wear-in under a red dashed 0.30 mNm limit with the last five minutes shaded, the bearing temperature as an orange dotted line rising from 21 to 32 °C on the right axis. Right, the coast-down speed in green falling from 6000 to 500 rpm over 181 s with an orange dashed line fitted to the first 60 s.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-cf3a4a76-17f0-4497-afb6-943a881c7032-m0uXYGOQyTVjEKr3HaiM0BmIMl6Hdu.png)

The mock wheel's hour: drag settles from 0.311 to 0.195 mNm as the lubricant distributes, bearings warm to 31.5 °C, and the coast-down from 6000 rpm takes 181 s with a friction torque of 0.290 mNm from the first minute. The bus draws 2.53 W at 3000 rpm and 2.90 W at 6000 rpm. A wheel whose drag does not settle, or whose coast-down is short, has a bearing problem the sweep will then locate.

### Exported Force

At each speed of the sweep the phase reads the peak exported radial force in the 3 to 300 Hz band, fits the speed-squared law to the whole sweep, and reports the maximum and the worst deviation from the fit:

```python filename="phases/exported_force.py"
import numpy as np

from utils.recipe import EFT_SWEEP_RPM


def exported_force(measurements, bench, log):
    """Exported force on the dynamometer at each speed of the sweep, 3 to
    300 Hz. A balanced rotor's force rises with speed squared; a bearing
    defect adds a line that does not, and shows as a bump in the curve."""
    force = []
    for rpm in EFT_SWEEP_RPM:
        bench.spin_to(rpm)
        force.append(bench.exported_force_mn(rpm))
    bench.stop()
    force = np.array(force)
    rpm = np.array(EFT_SWEEP_RPM, dtype=float)
    # Fit the unbalance law and report the worst deviation from it.
    k, _ = np.polyfit(rpm ** 2, force, 1)
    resid = force - (k * rpm ** 2 + force[0] - k * rpm[0] ** 2)

    measurements.eft.x_axis = EFT_SWEEP_RPM
    measurements.eft.y_axis.force = force.tolist()
    measurements.eft.y_axis.force.aggregations.max_mn = float(force.max())
    measurements.eft.y_axis.force.aggregations.max_deviation_mn = float(np.abs(resid).max())
    log.info(f"Exported force {force.min():.1f}..{force.max():.1f} mN over {rpm.min():.0f}..{rpm.max():.0f} rpm, worst deviation from the speed-squared law {np.abs(resid).max():.1f} mN")
```

### Mock Plug

`WheelBench` stands in for the wheel's telemetry link, the bus power analyzer and the dynamometer. It draws a rotor with 0.35 g·mm of static and 9 g·mm² of dynamic unbalance, bearing drag that decays exponentially from 0.31 to 0.19 mNm with a 15-minute time constant while the bearings warm from 21 to 32 °C, bus power as the drag times the speed over the drive efficiency plus 2.4 W of electronics, a coast-down integrated at 1 Hz from the drag over the inertia, and an exported force that is the unbalance times the speed squared plus a small bearing term. The wear-in is time-scaled and returns in one call. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real station, the class speaks the wheel's protocol over pyserial for speed, current and bearing temperature, SCPI to the power analyzer over pyvisa, and reads the six dynamometer channels through the charge amplifier and the DAQ with a 3 to 300 Hz band. Set `TIME_SCALE = 1.0` in `utils/recipe.py` for the real hour of wear-in, keep the vibration-exposure log in the dashboard per serial and add every acceptance vibration run to it, and take the unbalance limits from the vendor's data sheet for the wheel class. The phases, measurements and limits stay the same.

### ISO 9283 Repeatability and Accuracy Test
URL: https://www.tofupilot.com/templates/iso-9283-repeatability-and-accuracy-test
Run the ISO 9283 pose accuracy and repeatability test cycle on every robot at end-of-line and report per-unit performance against datasheet spec.


## Introduction

### ISO 9283 Overview

**ISO 9283:1998 "Manipulating industrial robots — Performance criteria and related test methods"** is the standard every robot datasheet cites for pose accuracy and repeatability. Originally written for type and acceptance testing, it has become the **contractual reference for end-of-line robot QA**: the number on the datasheet is the number each shipped unit has to reach.

![The 400 mm ISO 9283 test cube in the robot base frame with P1 at the centre and P2 to P5 on the diagonal plane, each 10 percent of the diagonal in from a corner, and the cycle path P1 to P5.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-f3d48b14-cb06-43e8-b90d-1b862d9c069e-o5Bc3SDAhuDCChhWKOl51fQ27zrLwL.png)

The test cube of this template: 400 mm, placed where the robot works most, P1 at the centre and P2 to P5 on the diagonal plane, each pulled 10 % of the diagonal in from its corner as the standard asks.

For two robots to be comparable under ISO 9283, **six conditions must match**: cube size and location, payload, velocity, path, cycle count, environment. The standard specifies **30 cycles per pose** at **100 % rated payload** and rated velocity, with additional runs at 50 % and 10 % velocity where the datasheet claims them. Pose accuracy AP is the distance between the commanded position and the barycentre of the attained ones; pose repeatability RP is the mean radial distance of the attained positions to that barycentre plus three standard deviations.

### Test Purpose

The ISO 9283 procedure produces a per-robot performance fingerprint:

- **Pose accuracy AP** and **pose repeatability RP** at each of the five poses, 30 cycles each
- **Multi-directional pose accuracy variation vAP** at P1, four approach directions
- **Distance accuracy AD** between two poses on the plane, expressed per metre
- **Test conditions**: payload, velocity, ambient temperature and humidity, recorded on the run
- **An acceptance record** attached to the run, the datasheet-style page that travels with the robot

![Scatter of the 30 attained P1 positions in the XY plane around the commanded point: the barycentre offset of 37 µm is the pose accuracy against a 150 µm limit, the dashed circle of 22 µm radius around the barycentre is the pose repeatability against a 30 µm limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-a5694f65-eed4-49af-bedb-b20d00d02e81-NGoiud7Ns8xud0V8HTvPJakz80ey1B.png)

P1 on the mock robot: the cloud of 30 attained positions sits 37 µm from the commanded point (accuracy) and spreads 22 µm around its own centre (repeatability). Accuracy is what kinematic calibration fixes; repeatability is what the gearboxes, encoders and brakes deliver.

Datasheet values for 5 kg-class cobots and small industrial arms sit between ±0.02 and ±0.03 mm repeatability. Every shipped robot has to hit its datasheet or go back for kinematic recalibration, gearbox inspection, or brake re-tuning.

Beyond the metrology, this template demonstrates four framework mechanics: **operator inputs bound to measurements** for the test conditions with pre-baked answers for unattended runs, an **automatic retry on error** for the tracker lock, **five multi-dimensional measurements with custom aggregations** carrying AP and RP per pose, and a **sequential DAG** that keeps the phases sharing one robot from running at the same time.

## Equipment & Setup

To implement ISO 9283 at end-of-line, the following are required:

- **A laser tracker** for 3-DOF (or 6-DOF with a probe) end-effector measurement
- **A retroreflector (SMR) in a magnetic nest** at the tool flange, or a 6-DOF probe
- **A reference payload** matching the robot's rated payload, bolted to the flange
- **The Device Under Test (DUT)**: a kinematically calibrated robot on its production pedestal
- **A TofuPilot Framework procedure** to drive the pose sequence, capture the tracker, and compute the ISO 9283 metrics
- **The TofuPilot Dashboard** to log per-unit results and trend production performance

![Robot wrist reflector opposite a laser tracker on a tripod](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/iso-9283-repeatability-and-accuracy-test-physical-2026-rRZ61fCporkNwETAPAATogQw8etmzU.webp)

A reflector on the robot wrist faces the laser tracker. The illustration shows the metrology arrangement; the test computer and measurement volume are not drawn.

### Hardware Components

#### Laser Tracker

The production reference is a **Hexagon Leica Absolute Tracker AT960-class** instrument (tens of micrometres of volumetric uncertainty, up to 1 kHz with the 6-DOF T-Mac probe) or a **FARO Vantage**. A **1.5 inch SMR** in a magnetic nest at the flange is the simplest target; the tracker averages the last 500 ms of each dwell to reject the settling transient. For linear-axis diagnostics a **Renishaw XL-80** interferometer and for circularity the **Renishaw QC20 ballbar** remain the specialised tools, not part of the ISO 9283 cycle itself.

#### Reference Payload

Measurements at 100 % rated payload need a calibrated reference mass matching the robot's specification: for a 5 kg cobot, a 5 kg machined steel block with a CMM-verified centre of gravity, bolted directly to the flange with no plastic adapters that would add compliance.

#### Environment

ISO 9283 asks for a stable cell temperature (20 °C ± 2 K is the usual acceptance window) and for the tracker to be warmed up and levelled. The procedure records the tracker's own environmental readings on the run so two results can be compared under the same conditions.

### Custom Firmware

The robot controller runs production firmware. The test station commands the trajectory through the vendor's motion API (URScript over RTDE, RAPID, KRL, KAREL, the vendor Python SDK); no special firmware is required.

## Test Procedure

### Overview

After mounting the SMR and the reference payload and levelling the tracker, the procedure runs:

1. The operator confirms the payload and picks the velocity in the UI.
2. The tracker locks on the reflector (retried automatically if the beam is not on the target yet), the environment is recorded, the robot gets its test conditions.
3. 30 cycles P1 → P5, one tracker reading per dwell; AP and RP per pose.
4. P1 approached from four directions, three cycles each; vAP.
5. P2 and P4 30 times; distance accuracy per metre.
6. The acceptance record is built from the three previous phases and attached.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
ui.json
phases/
  setup_cell.py
  pose_cycles.py
  multi_direction.py
  distance.py
  acceptance_report.py
plugs/
  robot.py
  tracker.py
utils/
  cube.py
  iso9283.py
README.md
pyproject.toml
```

The robot mock is a calibrated 5 kg cobot with small systematic offsets per pose and a 0.02 mm-class settle; the tracker mock fails its first lock the way a real one does when the beam is not on the reflector yet. Run it with `tofupilot run . --no-tui --no-kiosk --json --ui-values ui.json`. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-iso-9283-accuracy).

### The Procedure File

This is the exact file the template ships, verified end to end with the CLI. The five pose measurements share one shape, so P2 to P5 are elided here and shown in full in the repository:

```yaml filename="procedure.yaml"
name: ISO 9283 Repeatability and Accuracy Test
version: 0.1.0
description: Runs the ISO 9283 pose accuracy and repeatability cycle on a calibrated robot with a laser tracker, plus multi-directional accuracy and distance accuracy, and attaches the acceptance record.

unit:
  auto_identify: true
  serial_number:
    description: "Robot serial from the base plate"
    placeholder: "RB-0000000"
    pattern: "^RB-\\d{7}$"
    default_value: "RB-2026041"
  part_number:
    default_value: "COBOT-5KG-R2"

plugs:
  - name: Robot Controller
    description: Motion API of the robot under test (mock 5 kg cobot)
    python: plugs.robot:RobotController
    key: robot
  - name: Laser Tracker
    description: Tracker on the SMR at the tool flange (mock AT960-class)
    python: plugs.tracker:LaserTracker
    key: tracker

main:
  - name: Configure Test
    key: configure_test
    ui:
      components:
        - key: payload_kg
          type: number_input
          label: "Test payload (kg)"
          description: "Reference mass bolted to the flange, 100 % of rated payload per ISO 9283"
          default_value: 5
          bind: measurements.payload_kg
        - key: velocity_pct
          type: select
          label: "Velocity"
          bind: measurements.velocity_pct
          options:
            - label: "100 % rated"
              value: "100"
            - label: "50 %"
              value: "50"
            - label: "10 %"
              value: "10"
          default_value: "100"
    measurements:
      - name: Payload
        key: payload_kg
        unit: kg
        validators:
          - {operator: "==", expected_value: 5}
      - name: Velocity
        key: velocity_pct
        unit: "%"
        validators:
          - {operator: in, expected_value: ["100", "50", "10"]}

  - name: Setup Cell
    key: setup_cell
    python: phases.setup_cell
    depends_on: [configure_test]
    retry:
      limit: 2
      delay: 1s
    then:
      error: retry
    measurements:
      - name: Tracker Locked
        key: tracker_locked
        validators:
          - {operator: "==", expected_value: true}
      - name: Ambient Temperature
        key: ambient_temperature
        unit: °C
        description: ISO 9283 asks for 20 °C ± 2 K in the cell.
        validators:
          - {operator: ">=", expected_value: 18.0}
          - {operator: "<=", expected_value: 22.0}

  - name: Pose Cycles
    key: pose_cycles
    python: phases.pose_cycles
    depends_on: [setup_cell]
    timeout: 30m
    ui:
      components:
        - key: cycle_progress
          type: progress
          label: "30 cycles, P1 to P5"
          default_value: 0
          max: 100
    measurements:
      - name: Pose P1
        key: pose_p1
        title: P1 attained positions over 30 cycles
        x_axis: {legend: Cycle}
        y_axis:
          - {legend: dx, key: dx, unit: mm}
          - {legend: dy, key: dy, unit: mm}
          - {legend: dz, key: dz, unit: mm}
          - legend: Radial to barycentre
            key: radial
            unit: mm
            aggregations:
              - type: ap_mm
                unit: mm
                validators:
                  - {operator: "<=", expected_value: 0.15}
              - type: rp_mm
                unit: mm
                validators:
                  - {operator: "<=", expected_value: 0.03}
      # Pose P2 to Pose P5: same shape as Pose P1
      - name: RP Max
        key: rp_max
        unit: mm
        description: Worst pose repeatability over the five poses; the datasheet number.
        validators:
          - {operator: "<=", expected_value: 0.03}
      - name: AP Max
        key: ap_max
        unit: mm
        validators:
          - {operator: "<=", expected_value: 0.15}

  - name: Multi-Directional Accuracy
    key: multi_direction
    python: phases.multi_direction
    depends_on: [pose_cycles]
    measurements:
      - name: vAP at P1
        key: vap_p1
        unit: mm
        description: Spread of the P1 barycentres over four approach directions.
        validators:
          - {operator: "<=", expected_value: 0.15}

  - name: Distance Accuracy
    key: distance
    python: phases.distance
    depends_on: [multi_direction]
    measurements:
      - name: AD
        key: ad_mm
        unit: mm
      - name: AD per metre
        key: ad_mm_per_m
        unit: mm/m
        validators:
          - {operator: ">=", expected_value: -0.4}
          - {operator: "<=", expected_value: 0.4}

  - name: Acceptance Report
    key: acceptance_report
    python: phases.acceptance_report
    depends_on: [distance]
    measurements:
      - name: Report Attached
        key: report_attached
        validators:
          - {operator: "==", expected_value: true}
```

Framework features to notice:

1. **Operator inputs bound to measurements**: the payload and the velocity are entered once, land as validated measurements on the run, and `ui.json` pre-bakes them for unattended runs. Because select values are strings, the velocity is a string measurement validated with `in`.
2. **Retry on error**: `then: {error: retry}` with `retry: {limit: 2, delay: 1s}` on the tracker lock. A tracker that has not found the reflector yet raises; the framework re-runs the phase a second later instead of failing the robot. Both attempts stay on the run.
3. **Custom aggregations per pose**: each pose is a multi-dimensional measurement with the three axis deviations and the radial distance to the barycentre; `ap_mm` and `rp_mm` are computed in Python on that series and validated in YAML, so the chart and the limits live together.
4. **One robot, one phase at a time**: `pose_cycles`, `multi_direction` and `distance` are chained with `depends_on` even though none needs the other's result, because they share the same robot and tracker. Phases with satisfied dependencies run concurrently, and two phases moving one robot would interleave their commands.
5. **Run metadata**: the tracker's environment reading, the payload and the velocity are written to `run.metadata`, so runs can be filtered by test conditions on the dashboard.

### Setup Cell

The tracker lock is the step that fails on a real bench when the beam is not on the reflector; the retry rule handles it. The phase reads the operator's answers through **previous-results injection** from `configure_test`, hands the test conditions to the robot, and records the environment on the run:

```python filename="phases/setup_cell.py"
def setup_cell(measurements, tracker, robot, run, configure_test, log):
    payload = float(configure_test.payload_kg)
    velocity = int(configure_test.velocity_pct)
    robot.set_test_conditions(payload, velocity)

    measurements.tracker_locked = tracker.lock()
    env = tracker.environment()
    run.metadata["ambient_temp_c"] = env["temperature_c"]
    run.metadata["humidity_pct"] = env["humidity_pct"]
    run.metadata["payload_kg"] = payload
    run.metadata["velocity_pct"] = velocity
    measurements.ambient_temperature = env["temperature_c"]
    log.info(f"Tracker locked, {env['temperature_c']} C, payload {payload} kg, velocity {velocity} %")
```

In the JSON event stream the retry is explicit: attempt 1 ends `ERROR` with `beam not locked on the reflector`, attempt 2 ends `PASS` one second later.

### Pose Cycles

Thirty cycles through the five poses, one tracker reading at the end of each dwell. The per-pose series are filled through `getattr` on the measurements object, and the two ISO 9283 numbers are set as aggregations on the radial axis:

```python filename="phases/pose_cycles.py"
import numpy as np

from utils.cube import CYCLES, poses
from utils.iso9283 import pose_accuracy, pose_repeatability


def pose_cycles(measurements, robot, tracker, ui, log):
    """30 cycles P1 -> P5, tracker reading at the end of each dwell. AP and RP per pose."""
    targets = poses()
    attained = {name: [] for name in targets}

    for cycle in range(CYCLES):
        for name, xyz in targets.items():
            robot.move_to(name, xyz.tolist())
            attained[name].append(tracker.measure(robot.attained()))
        ui.cycle_progress = int(100 * (cycle + 1) / CYCLES)

    for name, xyz in targets.items():
        ap, bary = pose_accuracy(xyz, attained[name])
        rp, radial = pose_repeatability(attained[name])
        log.info(f"{name}: AP {ap:.3f} mm, RP {rp:.3f} mm")
        series = getattr(measurements, f"pose_{name.lower()}")
        series.x_axis = list(range(1, CYCLES + 1))
        series.y_axis.dx = [float(p[0] - xyz[0]) for p in attained[name]]
        series.y_axis.dy = [float(p[1] - xyz[1]) for p in attained[name]]
        series.y_axis.dz = [float(p[2] - xyz[2]) for p in attained[name]]
        series.y_axis.radial = radial.tolist()
        series.y_axis.radial.aggregations.ap_mm = ap
        series.y_axis.radial.aggregations.rp_mm = rp

    measurements.rp_max = max(pose_repeatability(a)[0] for a in attained.values())
    measurements.ap_max = max(pose_accuracy(targets[n], a)[0] for n, a in attained.items())
```

The ISO 9283 arithmetic lives in a small utility module:

```python filename="utils/iso9283.py"
import numpy as np


def pose_accuracy(commanded, attained):
    """AP: distance between the commanded position and the barycentre of the attained ones."""
    bary = np.mean(np.asarray(attained), axis=0)
    return float(np.linalg.norm(bary - np.asarray(commanded))), bary


def pose_repeatability(attained):
    """RP = l_mean + 3 * s_l, with l the radial distance of each attained position to the barycentre."""
    a = np.asarray(attained)
    bary = a.mean(axis=0)
    l = np.linalg.norm(a - bary, axis=1)
    s_l = np.sqrt(np.sum((l - l.mean()) ** 2) / (len(l) - 1))
    return float(l.mean() + 3.0 * s_l), l
```

![Pose accuracy and pose repeatability per pose P1 to P5 of the mock robot: AP between 27 and 59 µm under the 150 µm limit, RP between 17.6 and 23.3 µm under the 30 µm limit, over 30 cycles at 100 percent payload and velocity.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-c1bf4097-407f-490b-b17f-9306cecfd4b7-9loSsw77MUckcHkCmCLl9wx4KIVdzq.png)

The five poses of the mock robot: repeatability between 18 and 23 µm against the 30 µm datasheet limit, accuracy between 27 and 59 µm against 150 µm. The worst pose is the number that goes on the acceptance record.

### Multi-Directional Accuracy and Distance

vAP repeats P1 from four approach directions, three cycles each, and takes the largest distance between the four barycentres. Distance accuracy commands P2 and P4, a diagonal of the plane, thirty times and compares the attained distance with the commanded one, expressed per metre so the number is comparable across cube sizes:

```python filename="phases/distance.py"
from utils.cube import CYCLES, poses
from utils.iso9283 import distance_accuracy


def distance(measurements, robot, tracker, log):
    """Distance accuracy between P2 and P4 (a cube diagonal on the plane), 30 pairs."""
    targets = poses()
    a, b = [], []
    for _ in range(CYCLES):
        robot.move_to("P2", targets["P2"].tolist())
        a.append(tracker.measure(robot.attained()))
        robot.move_to("P4", targets["P4"].tolist())
        b.append(tracker.measure(robot.attained()))
    ad, d_cmd = distance_accuracy(targets["P2"], targets["P4"], a, b)
    log.info(f"AD over {d_cmd:.1f} mm: {ad:+.3f} mm ({1000 * ad / d_cmd:+.3f} mm/m)")
    measurements.ad_mm = ad
    measurements.ad_mm_per_m = 1000.0 * ad / d_cmd
```

### Acceptance Report

The last phase reads the three result phases through previous-results injection and attaches the acceptance record to the run, named after the serial so it is findable from the unit page:

```python filename="phases/acceptance_report.py"
import json


def acceptance_report(measurements, unit, run, pose_cycles, multi_direction, distance, attach, log):
    """Datasheet-style acceptance record attached to the run as JSON."""
    report = {
        "serial_number": unit.serial_number,
        "part_number": unit.part_number,
        "conditions": {k: run.metadata.get(k) for k in ("payload_kg", "velocity_pct", "ambient_temp_c", "humidity_pct")},
        "ap_max_mm": float(pose_cycles.ap_max),
        "rp_max_mm": float(pose_cycles.rp_max),
        "vap_p1_mm": float(multi_direction.vap_p1),
        "ad_mm_per_m": float(distance.ad_mm_per_m),
        "standard": "ISO 9283:1998, 30 cycles, 5 poses on the diagonal plane of a 400 mm cube",
    }
    attach.data(json.dumps(report, indent=2).encode(), f"{unit.serial_number}_iso9283.json")
    measurements.report_attached = True
    log.info("Acceptance record attached")
```

### Mock Plugs

The robot mock keeps one systematic offset per pose (what AP sees), adds a 4 µm-class settle on every move (what RP sees), and biases each approach direction slightly (what vAP sees). The tracker mock adds its own 4 µm noise and fails its first lock. Plug return values cross a process boundary as JSON, so the mock returns plain lists, never NumPy arrays:

```python filename="plugs/robot.py"
import numpy as np


class RobotController:
    def __init__(self):
        self._rng = np.random.default_rng(9283)
        self._offsets = {}
        self._pose = np.zeros(3)
        self._payload_kg = 0.0
        self._velocity_pct = 100
        print("Robot controller connected, servo on")

    def set_test_conditions(self, payload_kg, velocity_pct):
        self._payload_kg = float(payload_kg)
        self._velocity_pct = int(velocity_pct)

    def move_to(self, name, xyz, direction=0):
        """Move and settle; the attained position is what the tracker will see."""
        if name not in self._offsets:
            self._offsets[name] = self._rng.normal(0.0, 0.045, 3)
        bias = np.array([0.02, -0.015, 0.01]) * (direction - 1.5) / 1.5
        settle = self._rng.normal(0.0, 0.004, 3)
        self._pose = np.asarray(xyz) + self._offsets[name] + bias + settle
        return True

    def dwell_s(self):
        return 2.0

    def attained(self):
        return self._pose.tolist()
```

On a real cell, `plugs/robot.py` maps to the vendor motion API and `plugs/tracker.py` to the tracker SDK; the phases, measurements, and limits stay exactly the same.


### GNSS Receiver Sensitivity Verification
URL: https://www.tofupilot.com/templates/gnss-receiver-sensitivity-verification
Verify drone GNSS receiver sensitivity, TTFF, and C/N0 against a controlled RF simulator scenario to catch antenna, LNA, and firmware defects.


## Introduction

### GNSS in Drones

Every modern drone depends on a **Global Navigation Satellite System (GNSS)** receiver for position, velocity, and time. GPS alone is the legacy minimum; current production receivers track **GPS + GLONASS + Galileo + BeiDou** simultaneously, and high-end units add **L2/L5 dual-frequency** for centimeter-class **RTK** accuracy. The receiver chain is **antenna → LNA → SAW filter → RF front-end → baseband processor**, and any defect in the chain produces the same symptom: poor fix quality, slow time-to-first-fix (TTFF), and HDOP above the autopilot's arming threshold.

![Close-up of a drone GNSS receiver module: ceramic patch antenna, RF shield can over the LNA and receiver chip, U.FL antenna connector at the board edge and the JST cable to the flight controller.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-63fd8af0-d526-4374-b0cf-1afef5d1309a-zoknfQdoGhMi95U6VVo8bZINwYi098.jpg)

A typical drone GNSS module: patch antenna on top, the LNA and receiver under the shield can, and the U.FL connector where a cracked feedline or a bad crimp costs sensitivity.

A weak or noisy GNSS chain is a leading cause of drone fly-aways. ArduPilot's [pre-arm safety checks](https://ardupilot.org/copter/docs/common-prearm-safety-checks.html) block arming with a "High GPS HDOP" failure whenever HDOP exceeds 2.0 (the `GPS_HDOP_GOOD` parameter) in any GPS-requiring mode. Without GNSS, Return-to-Home is impossible. RTK-grade accuracy degrades if antenna-system gain drops below the 17 dB minimum u-blox specifies in the [ZED-F9P integration manual](https://content.u-blox.com/sites/default/files/ZED-F9P_IntegrationManual_UBX-18010802.pdf), or if the feedline cracks. The 5.8 GHz VTX radiates wideband power-amplifier noise that desenses the GNSS front end at L1 (1575.42 MHz) and L2 (1227.60 MHz), so every airframe ships with a receiver whose live noise floor is unknown unless verified.

### Test Purpose

Per-unit GNSS sensitivity verification produces a fingerprint covering:

- **Cold-start TTFF** (no almanac, no ephemeris, no position priors) at standardized signal power
- **Hot-start TTFF** (valid aiding data) for re-fix testing
- **Sensitivity floor**: the signal power at which the receiver loses lock
- **C/N₀ per satellite** across the power sweep, where band-skewed dropouts reveal matching network defects
- **Position and velocity accuracy** at reference power
- **HDOP at lock**

The limits in this template sit inside the margins the [u-blox NEO-M9N datasheet](https://content.u-blox.com/sites/default/files/NEO-M9N-00B_DataSheet_UBX-19014285.pdf) specifies (cold start 24-29 s, hot start 2 s, tracking sensitivity -167 dBm), so a healthy receiver passes with room to spare and a marginal RF chain does not.

![Mean C/N0 versus simulator output power: 42 dB-Hz at -130 dBm rolling off 0.35 dB per dB across the 3 dB steps down to -163 dBm, with the 38 dB-Hz aggregation limit and the -161 dBm sensitivity floor marked.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-ae7a3af4-27fd-43f6-9f9d-af0970b9893e-kvJdmRDwgPwPmn6Vx9iYTTfL08Mu7G.png)

The sweep as the mock simulator produces it: 42 dB-Hz at -130 dBm, a steady roll-off across the 3 dB steps, and a 3D fix held down to -161 dBm, 1 dB inside the limit.

Per-unit testing eliminates the leading GNSS-related field failures: dead LNA from reversed mount, feedline cracks from rough assembly, sub-spec antenna patterns from carbon-fiber frame shielding, and firmware misconfigs (constellation mask wrong, baud mismatch).

Beyond the RF content, this template demonstrates four framework capabilities that fit GNSS testing naturally: **automatic phase retry** for the flaky first cold start, a **regex validator** pinning the receiver firmware to the production release line, **sub-unit traceability** linking the GNSS module serial to the airframe, and a **phase timeout** bounding the power sweep.

## Equipment & Setup

To implement GNSS sensitivity verification on a production line, the following are required:

- **A GNSS RF simulator** capable of multi-constellation scenarios with calibrated output power
- **A shielded test environment** (Faraday tent or anechoic chamber) to block ambient GNSS leakage
- **A programmable step attenuator** for the RF path
- **The Device Under Test (DUT)** with the production-target GNSS module and antenna
- **A TofuPilot Framework procedure** to script the simulator, sweep power, collect NMEA / UBX telemetry, and validate metrics
- The **TofuPilot Dashboard** to log per-unit results and trend supplier quality

### Hardware Components

#### GNSS RF Simulator

The **Spirent GSS6300M** is the production reference: [36-channel, L1/E1 multi-GNSS](https://teleplanforsberg.com/wp-content/uploads/2022/04/GSS6300M_datasheet.pdf), IEEE-488 remote control, continuous-run scenarios for ATE. For higher-end labs the **Safran Skydel GSG-7 / GSG-8** (GPU + SDR, 1000 Hz simulation iteration rate, IQ playback) covers L1/L2/L5 with scripted scenarios. For budget production lines, the **Racelogic [LabSat 3 Wideband](https://www.labsat.co.uk/en/products/labsat-3-wideband)** is a record-replay box capable of 3 channels × 56 MHz that can play a recorded factory roof reference scenario at calibrated power. All three are quote-based; budget roughly $10k for the LabSat class up to high five figures for the Spirent and Skydel class.

![GNSS receiver board with coax and power connections inside an open RF shield box](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/gnss-receiver-sensitivity-verification-physical-2026-umTlTGCqqZo0zWjFm1Km6S2NLt1kOh.webp)

The receiver is shown in a conducted RF setup with coax and filtered electrical feedthroughs. The box is open for setup and closes for measurement; the RF source is external.

#### Shielded Environment

A **Faraday tent** (Select Fabricators, JRE Test) is the cost-effective production option vs a fixed anechoic chamber. Specify at least **60 dB isolation at L1** so no ambient GNSS leaks in and corrupts the scenario. A fixed **ETS-Lindgren anechoic chamber** is the higher-end alternative for OTA antenna pattern testing.

#### Attenuator

A **[JFW 50P-2014](https://www.jfwindustries.com/product/50p-2014-solid-state-programmable-attenuator-with-usb-control/)** solid-state programmable step attenuator: **0-95 dB in 1 dB steps**, 200-6000 MHz, USB or TTL controlled (the 50P-2013 is the 0-63 dB variant). Insertion accuracy ±0.5 dB up to 90 dB. Calibration certificate per channel-frequency required for production traceability.

#### Reference Receiver

A **u-blox EVK-NEO-M9N** runs the same scenario in parallel as a golden reference. Drift in the simulator output power shows up as a divergence between the reference and the DUT, isolating simulator faults from DUT faults.

### Custom Firmware

The DUT GNSS module runs **production firmware** with its normal constellation mask, sample rate, and dynamic model. The test station receives **UBX (u-blox) or proprietary protocol** messages and parses C/N₀, position, HDOP, fix status. No special firmware build is needed.

## Test Procedure

### Overview

After mounting the DUT inside the Faraday tent and connecting the simulator chain, the procedure runs:

1. Verify the reference receiver locks at -130 dBm (sanity check on the simulator).
2. Cold-start the DUT (clear ephemeris, position, time) and measure TTFF; retry automatically if the first fix exceeds the limit.
3. Hot-start the DUT and measure TTFF.
4. Sweep power down in 3 dB steps, log mean C/N₀ per step, and find the sensitivity floor.
5. Validate every metric against its declared limits.
6. Stream results to TofuPilot for traceability and analytics.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  verify_reference.py
  cold_start.py
  hot_start.py
  power_sweep.py
plugs/
  simulator.py
  receiver.py
utils/
  sensitivity.py
README.md
pyproject.toml
```

The mock plugs replay a deterministic scenario so the procedure runs without hardware; swap them for Spirent/Skydel (pyvisa) and pyubx2 implementations on a real bench. Run it with `tofupilot run . --no-tui --no-kiosk --json`. You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-gnss-sensitivity).

### The Procedure File

`procedure.yaml` declares the unit, the sub-unit, the equipment plugs, and the phases with their measurements and limits. This is the exact file the template ships, verified end to end with the CLI:

```yaml filename="procedure.yaml"
name: GNSS Receiver Sensitivity Verification
version: 0.1.0
description: Verifies GNSS receiver cold/hot-start TTFF, C/N0 tracking across a calibrated power sweep, and sensitivity floor against a simulator scenario.

unit:
  auto_identify: true
  serial_number:
    default_value: "DRN-00042"
  part_number:
    default_value: "FC-GNSS-M9N"
  sub_units:
    - label: "GNSS Module"
      serial_number:
        placeholder: "Scan GNSS module barcode"
        pattern: "^GNSS-\\d{6}$"
        default_value: "GNSS-104217"

plugs:
  - name: GNSS Simulator
    description: Mock Spirent-class simulator with calibrated output power and programmable step attenuator.
    python: plugs.simulator:GnssSimulator
    key: simulator
  - name: Receiver Telemetry
    description: Mock UBX telemetry stream from the receiver under test.
    python: plugs.receiver:ReceiverTelemetry
    key: receiver

main:
  - name: Verify Reference Lock
    key: verify_reference
    python: phases.verify_reference
    measurements:
      - name: Reference Locked
        key: reference_locked
        validators:
          - operator: "=="
            expected_value: true

  - name: Cold Start TTFF
    key: cold_start
    python: phases.cold_start
    depends_on: [verify_reference]
    retry:
      limit: 3
      delay: 1s
    then:
      fail: retry
    measurements:
      - name: Cold Start TTFF
        key: cold_start_ttff_s
        unit: s
        description: Median time to first 3D fix from cleared ephemeris at -130 dBm.
        validators:
          - operator: "<="
            expected_value: 30.0
      - name: Receiver Firmware
        key: firmware_version
        description: Firmware reported by UBX-MON-VER, pinned to the production release line.
        validators:
          - operator: matches
            expected_value: "^SPG \\d+\\.\\d{2}$"

  - name: Hot Start TTFF
    key: hot_start
    python: phases.hot_start
    depends_on: [cold_start]
    measurements:
      - name: Hot Start TTFF
        key: hot_start_ttff_s
        unit: s
        validators:
          - operator: "<="
            expected_value: 5.0

  - name: Power Sweep
    key: power_sweep
    python: phases.power_sweep
    depends_on: [hot_start]
    timeout: 60s
    measurements:
      - name: CN0 vs Power
        key: cn0_vs_power
        title: Average C/N0 vs Simulator Power
        description: Mean C/N0 across tracked satellites at each simulator output power step.
        x_axis:
          legend: Simulator Power
          unit: dBm
        y_axis:
          - legend: Mean C/N0
            key: cn0
            unit: dB-Hz
            aggregations:
              - type: max
                validators:
                  - operator: ">="
                    expected_value: 38.0
      - name: Sensitivity Floor
        key: sensitivity_floor_dbm
        unit: dBm
        description: Lowest simulator power with a held 3D fix.
        validators:
          - operator: "<="
            expected_value: -160.0
      - name: HDOP At Lock
        key: hdop_at_lock
        validators:
          - operator: "<"
            expected_value: 1.4
```

Three framework features to notice in the YAML:

1. **Sub-unit traceability**: the `sub_units` block collects the GNSS module serial (with a regex `pattern` on the barcode) and links it to the airframe unit. In Python the value is available as `unit.sub_units.gnss_module`; the framework snake_cases the label into the access key.
2. **Retry on the cold start**: `then: {fail: retry}` with `retry: {limit: 3, delay: 1s}`. A fresh board with a blank aiding flash routinely blows the 30 s TTFF limit once; instead of failing the unit, the framework re-runs the phase up to 3 times with a 1 s pause. Every attempt is preserved and visible on the dashboard with its own measurements.
3. **Regex validator**: the `matches` operator pins `firmware_version` to the `SPG x.yy` release line (SPG 4.04 is the NEO-M9N production firmware), catching mis-flashed receivers with a one-line validator instead of Python string handling.

Note the `hdop_at_lock < 1.4` limit is this template's production margin, chosen well inside ArduPilot's arming threshold of 2.0 so a shipped unit never sits at the edge of the autopilot's own check.

### Reference Lock Phase

The opening phase sanity-checks the simulator and logs which physical module this airframe carries, reading the sub-unit serial collected at identification:

```python filename="phases/verify_reference.py"
def verify_reference(measurements, simulator, log, unit):
    log.info(f"Verifying scenario for unit {unit.serial_number}")
    module_sn = unit.sub_units.gnss_module
    log.info(f"GNSS module sub-unit: {module_sn}")

    simulator.set_attenuation(0)
    measurements.reference_locked = simulator.reference_locked()
```

### Cold Start Phase

The phase writes its measurements and lets the declared validators decide the outcome. When `cold_start_ttff_s` exceeds 30 s, the measurement fails, the phase fails, and the `then: {fail: retry}` rule re-runs it:

```python filename="phases/cold_start.py"
def cold_start(measurements, receiver, log, unit):
    log.info(f"Cold-starting module {unit.sub_units.gnss_module}")
    receiver.clear_aiding_data()

    ttff = receiver.cold_start_ttff_s()
    log.info(f"Cold start TTFF: {ttff:.1f} s")
    if ttff > 30.0:
        log.warning("TTFF above limit, framework will retry this phase")

    measurements.cold_start_ttff_s = ttff
    measurements.firmware_version = receiver.firmware_version()
```

In the JSON event stream the retry is explicit: attempt 1 finishes `FAIL` with `measurement cold_start_ttff_s failed: <= 30.0`, attempt 2 finishes `PASS` one second later.

![Cold start time series: simulator held at -130 dBm, fix type rising from no fix to 3D at 41.8 s on the first attempt which fails the 30 s limit, then at 18.4 s on the automatic retry, with the satellites used climbing from 0 to 12 on each attempt.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-35500324-a914-4cff-95f9-8b681a74f233-Mj9dScVhg2obD7qXike3fErPTSW14R.png)

The two cold-start attempts of the mock receiver at -130 dBm: the first 3D fix lands at 41.8 s and fails the limit, the framework clears aiding data and re-runs the phase, the second fix lands at 18.4 s.

### Power Sweep Phase

The sweep steps attenuation up, records mean C/N₀ per power step into the multi-dimensional chart, then steps power down 1 dB at a time to find the sensitivity floor. The `timeout: 60s` on the phase bounds the whole sweep, including plug calls, so a hung simulator cannot stall the line:

```python filename="phases/power_sweep.py"
from utils.sensitivity import find_sensitivity_floor


def power_sweep(measurements, simulator, receiver, log):
    powers = []
    cn0s = []

    attenuation = 0
    while attenuation <= 35:
        simulator.set_attenuation(attenuation)
        power = simulator.output_power_dbm()
        powers.append(power)
        cn0s.append(simulator.mean_cn0_at(power))
        attenuation += 3

    measurements.cn0_vs_power.x_axis = powers
    measurements.cn0_vs_power.y_axis.cn0 = cn0s
    measurements.cn0_vs_power.y_axis.cn0.aggregations.max = max(cn0s)

    floor = find_sensitivity_floor(simulator, start_dbm=-150.0, stop_dbm=-165.0)
    log.info(f"Sensitivity floor: {floor:.0f} dBm")
    measurements.sensitivity_floor_dbm = floor
    measurements.hdop_at_lock = receiver.hdop()
```

The floor search lives in a plain utility module:

```python filename="utils/sensitivity.py"
def find_sensitivity_floor(simulator, start_dbm, stop_dbm, step_db=1.0):
    """Step power down until the receiver drops the 3D fix.

    Returns the lowest power at which the fix was still held.
    """
    power = start_dbm
    last_held = start_dbm
    while power >= stop_dbm:
        if not simulator.fix_held_at(power):
            break
        last_held = power
        power -= step_db
    return last_held
```

The **C/N₀ vs power curve** should be near-linear above the floor with slope ~1 dB/dB. Departure from linearity at high power indicates **LNA saturation** (rare); departure at low power indicates the **noise floor of the receiver chain**.

![Heatmap of C/N0 per satellite versus simulator power from -130 to -165 dBm: twelve satellites tracked uniformly down to the -161 dBm floor, one satellite dropping about 10 dB early and lost at -146 dBm, flagged as an antenna pattern null.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-0d9465db-242d-4217-a2ec-4e7a737c781d-krSMGm1J8FpcxlKfXUSi7zE6hUWvzu.png)

Per-satellite view of the same sweep on a real bench: a healthy chain loses satellites together at the floor, while one satellite dropping 10 dB early points at an antenna pattern null or a frame shadowing that elevation.

### Mock Plugs

The template ships mock plugs that replay a deterministic scenario, so the whole procedure runs green without hardware. The receiver mock makes the first cold start deliberately slow to exercise the retry path:

```python filename="plugs/receiver.py"
class ReceiverTelemetry:
    def __init__(self):
        self._cold_start_attempts = 0

    def clear_aiding_data(self):
        print("UBX-CFG-RST (navBbrMask 0xFFFF) sent, cold start forced")

    def cold_start_ttff_s(self):
        self._cold_start_attempts += 1
        if self._cold_start_attempts == 1:
            return 41.8  # blank aiding flash on first boot
        return 18.4

    def hot_start_ttff_s(self):
        return 2.1

    def firmware_version(self):
        return "SPG 4.04"

    def hdop(self):
        return 0.8
```

On real u-blox hardware, forcing a true cold start means sending **UBX-CFG-RST** with `navBbrMask 0xFFFF` to clear ephemeris, almanac, position, and time from battery-backed RAM (UBX-CFG-CFG only manages configuration, not aiding data). Plugs persist across retry attempts, which is what lets the mock model "slow first fix, fast second fix" with a simple counter. On a real bench, swap this class for a pyserial + pyubx2 implementation; the phases stay unchanged.


### LiPo Battery Discharge and IR Profiling
URL: https://www.tofupilot.com/templates/lipo-battery-discharge-and-ir-profiling
Run controlled discharge cycles on drone LiPo packs while logging cell voltage, internal resistance, sag, and temperature to grade pack health.


## Introduction

### LiPo Battery Overview

The lithium-polymer (LiPo) battery is one of the most expensive components in a drone bill of materials, and mishandled packs are the main fire hazard on a production floor. A 6S 22,000 mAh cargo-drone pack costs more than the autopilot, the camera, and the gimbal combined, and one defective cell can take down a $50,000 airframe mid-flight. Cell voltage swings from **4.20 V/cell fully charged** to **3.30 V/cell under-load cutoff**, with deep discharge below 3.0 V/cell causing irreversible damage. The pack is a series-parallel string, and **a single weak cell drags the entire pack**: capacity is gated by the worst cell, not the average.

![Drone 4S LiPo pack with the shrink wrap peeled back to show the four pouch cells in series and their tabs, the JST-XH balance lead and the XT60 main leads.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-3c9133cc-f7fb-47a1-8da6-19e877988259-y7DVEcAogrl4BFlNKTHRZEOMWrBn9K.jpg)

A 4S pack under the shrink wrap: four pouch cells in series, the balance lead tapping every cell junction, the XT60 main leads carrying the load current.

The cell vendor's COA covers cells before assembly; everything that happens after (tab welding, balance lead wiring, BMS configuration, storage at the wrong SOC) is on the pack assembler. End-of-line discharge profiling catches these defects per-pack before the customer's first crash.

### Test Purpose

The procedure produces a per-pack health fingerprint:

- **Capacity delivered** in mAh from full charge to cutoff at controlled C-rate
- **DC internal resistance** per cell, measured via short high-current pulse
- **Cell imbalance** maximum across the discharge envelope
- **Burst voltage sag** under a simulated motor demand transient
- **Temperature rise** during full discharge
- **OCV-SOC curve** for state-of-charge estimation calibration

![Per-cell voltage curves over the 0.5C discharge from 4.20 V to 3.30 V per cell: three cells track together, cell 3 with 4.2 mΩ diverges after 60% depth of discharge and ends 26 mV below the others at cutoff, inside the 30 mV limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-0fb375e4-40cb-499a-828f-199b68ecaf79-uC3KhczE9y9WTqKScPf05M8VnEe0vc.png)

The mock pack's discharge: three cells track to within a few millivolts, the weak cell 3 droops after 60% depth of discharge and finishes 26 mV below the others, just inside the 30 mV imbalance limit.

Per-pack profiling catches weak cells, internal micro-shorts (very low IR), tab cold-solder joints, miswired balance leads, and BMS misconfig. End-of-discharge state is set to **3.85 V/cell storage SOC** (commonly treated as ~50%) or **3.70-3.75 V/cell for air-shipped packs**: the [IATA Lithium Battery Guidance Document](https://www.iata.org/contentassets/05e6d8742b0047259bf3a700bc9d42b9/lithium-battery-guidance-document.pdf) (DGR 67th edition, 2026) requires batteries shipped alone under PI 965 at a state of charge not exceeding 30% of rated capacity, and since 2026 the same limit applies to packs shipped with equipment under PI 966. The rule is written in percent SOC, not volts, so verify the voltage targets against your cell's OCV-SOC table.

## Equipment & Setup

To implement LiPo discharge profiling on a production line, the following are required:

- **A programmable DC electronic load** with CC and pulse modes, sized for pack voltage and peak C-rate
- **A balance-lead breakout** for per-cell voltage sensing
- **Thermocouples** on cell surfaces (or pack thermistor)
- **The Device Under Test (DUT)**: a fully charged LiPo pack
- **A safety enclosure** (LiPo bunker) with thermal monitoring and emergency cutoff
- **A TofuPilot Framework procedure** to script the cycle, log measurements, and validate metrics
- **The TofuPilot Dashboard** to store per-pack health data for traceability and supplier-quality trending

### Hardware Components

#### Electronic Load

For production lines, the [Chroma 63200A series](https://www.chromausa.com/document-library/datasheet-high-power-dc-electronic-loads-63200a/) is the reference: CC / CR / CV / CP modes, voltage and time stops, **2-24 kW per unit** parallelable to 240 kW, up to **2,000 A per unit**, and **300% peak overpower** for burst pulses, with master/slave control for batch parallel discharge. For tight transient capture (burst sag), the [Kikusui PLZ-4W series](https://global.kikusui.co.jp/products-index/electronic-load/plz-4w/) offers **10 µs rise/fall**, which catches the full transient response of a 10C burst (the PLZ164WA and PLZ664WA are the series' 0 V-input models).

For lab and small-volume production, the **iCharger 4010 Duo** (2 × 40 A, up to 10S) is the affordable workhorse, with one important caveat for this procedure: its internal discharge capability is **130 W per channel**; the headline 1400 W figure applies to regenerative discharge into a supply battery. For incoming inspection of bare cells or 1S validation, the **ZKETech EBC-A40L** covers 40 A charge/discharge at ≤5 V.

![LiPo battery pack secured in an enclosure with separate power and sleeved balance leads](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/lipo-battery-discharge-and-ir-profiling-physical-2026-MunCBuyUuVlzjN9LTbR1TPo49Y4Bz2.webp)

The pack is supported in a cradle with separate main power and balance harnesses. The electronic load and measurement instruments are not shown.

#### Per-Cell Sensing

A **balance-lead breakout** (XH for ≤6S, PH for larger) routes each cell tap to a multiplexed ADC. Per-cell voltages are read at 1 Hz throughout discharge. Pack-only voltage is insufficient: a 4S pack reading 14.8 V can hide one cell at 3.0 V and three cells at 3.93 V.

#### Thermal Monitoring

**Type-K thermocouples** bonded to two cells (center and edge of the pack), or a pack-integrated thermistor pre-wired by the pack assembler. For visual QA, a **FLIR E8-XT thermal camera** (320×240, -20 to 550 °C, MSX overlay, IP54) captures post-discharge thermal images, flagging asymmetric heating from a weak cell or a bad tab weld.

#### Safety Bunker

LiPo discharge **must run inside a steel-walled bunker** lined with vermiculite or [CellBlockEX](https://cellblockfcs.com/cellblockex/) thermal-runaway suppressant. Adjacent fire-suppression: a sand bucket (CO₂ and dry chem don't stop lithium reignition). **Hardware interlocks** for OVP, UVP, and OTP wired to the load contactor, never software-only. The production line alarms and disconnects the load at **dT/dt > 5 °C/min** or **T > 60 °C**. These are deliberately conservative early-warning limits: formal thermal-runaway detection criteria in standards work trigger much later, around **dT/dt ≥ 1 °C/second** (see the [JRC evaluation of detection criteria](https://publications.jrc.ec.europa.eu/repository/handle/JRC132171)), at which point suppression, not prevention, is what remains.

### Cycle Parameters

Discharge at **0.5C constant current** (1C for high-C packs) from full charge to **3.30 V/cell** cutoff, logging per-cell voltage, current, and temperature at 1 Hz. The IR pulse applies **5C for 2 s at 50% SOC** after a 30 min rest; the burst applies **10C for 200 ms at 30% SOC**, captured at ≥1 kHz. The pack ends at the storage SOC target above. A 0.5C discharge takes ~2 hours regardless of capacity (rate-limited, not capacity-limited); batch-parallel discharge across 8-16 bunker cells per master/slave rack amortizes cycle time.

## Test Procedure

### Overview

The procedure maps the cycle onto the framework's three stages. Safety lives in `setup:`, so no current flows before the operator arms the interlock. The recharge lives in `teardown:`, so the framework runs it even when a main phase fails and no pack is ever left at full charge in the bunker.

1. Setup: interlock confirmation plus electrical sanity checks.
2. Main: CC capacity discharge with per-cell logging, bounded by a phase timeout.
3. Main: IR pulse and per-cell DC resistance computation.
4. Main: burst sag transient, split into ESR and polarization.
5. Main (disabled by default): optional FLIR thermal capture.
6. Teardown: recharge to storage SOC, always executed.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for traceability and analytics.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

This template additionally demonstrates **setup and teardown stages**, a **phase timeout**, a **disabled phase**, **unit metadata**, and an **operator interlock switch bound to a validated measurement**.

### Project Structure

```tree
procedure.yaml
phases/
  safety_check.py
  cc_discharge.py
  ir_pulse.py
  burst_sag.py
  thermal_capture.py
  recharge_storage.py
plugs/
  battery_bench.py
ui.json
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-lipo-discharge-profiling). The `BatteryBench` plug is a mock that synthesizes a 4S pack with one slightly weak cell, so the procedure runs end-to-end without a pack or an electronic load connected. The mock is **time-scaled**: it returns the whole 2-hour discharge in one call so a run finishes in seconds. This is also why the discharge phase carries `timeout: 120s`; a production procedure driving a real load would set e.g. `timeout: 3h` to bound the actual discharge duration.

```bash
tofupilot run .
```

For CI or bench automation, `ui.json` pre-bakes the operator inputs and the run executes headless:

```bash
tofupilot run . --no-tui --json --ui-values ui.json --ui-timeout 60
```

### The Procedure File

`procedure.yaml` declares the unit (auto-identified, with a **cell lot metadata field**), the bench plug with its **config** passed as constructor arguments, and the three stages. The main-phase measurement declarations are abridged here and shown in their sections below:

```yaml filename="procedure.yaml"
name: LiPo Discharge and IR Profiling
version: 0.1.0
description: Controlled discharge cycle on a drone LiPo pack with per-cell logging, DC internal resistance pulse, burst sag capture, and recharge to storage SOC.

plugs:
  - name: Battery Bench
    description: Electronic load and balance-lead ADC. Mock replays a synthetic 4S pack with one weak cell.
    python: plugs.battery_bench:BatteryBench
    key: bench
    config:
      cell_count: 4
      capacity_mah: 3000

unit:
  auto_identify: true
  serial_number:
    default_value: "PACK-000123"
  part_number:
    default_value: "LIPO-4S-3000"
  batch_number:
    default_value: "BATCH-2026-031"
  metadata:
    cell_lot:
      description: "Cell vendor lot code printed on the pouch edge"
      placeholder: "LOT-XXXX"
      default_value: "LOT-8842"

setup:
  - name: Safety Check
    key: safety_check
    python: phases.safety_check
    measurements:
      - name: Interlock Armed
        key: interlock_armed
        validators:
          - operator: "=="
            expected_value: true
      - name: Cells At Full Charge
        key: cells_full
        validators:
          - operator: "=="
            expected_value: true
    ui:
      components:
        - key: interlock
          type: switch
          label: "Bunker interlock armed"
          description: "Confirm the pack is inside the bunker and the hardware interlock is armed"
          required: true
          bind: measurements.interlock_armed

main:
  - name: CC Discharge
    key: cc_discharge
    python: phases.cc_discharge
    timeout: 120s  # mock is time-scaled; a real 0.5C discharge needs e.g. 3h

  - name: IR Pulse
    key: ir_pulse
    python: phases.ir_pulse
    depends_on: [cc_discharge]

  - name: Burst Sag
    key: burst_sag
    python: phases.burst_sag
    depends_on: [ir_pulse]

  - name: Thermal Capture
    key: thermal_capture
    python: phases.thermal_capture
    enabled: false
    depends_on: [burst_sag]

teardown:
  - name: Recharge To Storage
    key: recharge_storage
    python: phases.recharge_storage
    measurements:
      - name: Storage Voltage
        key: storage_voltage_v
        unit: V/cell
        validators:
          - operator: ">="
            expected_value: 3.80
          - operator: "<="
            expected_value: 3.90
```

Three framework behaviors carry the safety story:

- The **`setup:` stage** must pass before any main phase runs, so the interlock switch gates the discharge.
- The **`teardown:` stage** always runs, even after a failed or errored main phase, so the pack always ends at storage SOC.
- The **`timeout:`** on the discharge bounds the phase; a load stuck in CC mode ends the phase with a timeout outcome instead of running forever.

The `Thermal Capture` phase ships with `enabled: false`: it is skipped silently without recording results. Lines equipped with a FLIR camera flip it on without touching code.

### Safety Check Phase

The operator switch is **bound to the `interlock_armed` measurement** with `bind: measurements.interlock_armed`, and the measurement validates `== true`. The run report records what the operator actually confirmed, not just that a button was pressed. The phase verifies every cell sits at full charge (4.20 V ±0.02) before allowing the discharge, warns on pack temperature outside the 20-25 °C window, and stamps the BMS firmware onto the **unit metadata**, next to the cell lot collected from YAML:

```python filename="phases/safety_check.py"
def safety_check(measurements, bench, unit, log):
    """Pre-discharge check: interlock (operator switch, bound to a
    measurement) plus electrical sanity read from the bench."""
    log.info(f"Pack {unit.serial_number}, cell lot {unit.metadata.get('cell_lot')}")
    unit.metadata["bms_fw"] = "1.7.2"

    cells = bench.cell_voltages()
    measurements.cells_full = all(4.18 <= v <= 4.22 for v in cells)
    log.info(f"Cell voltages {[f'{v:.3f}' for v in cells]} V")

    temp = bench.pack_temperature()
    if not 20.0 <= temp <= 25.0:
        log.warning(f"Pack temperature {temp:.1f} C outside 20-25 C window")
    log.info(f"Pack temperature {temp:.1f} C")
```

Hardware interlocks for OVP, UVP, and OTP stay wired to the load contactor. The software check is the operator-facing layer, not the protection.

### CC Capacity Discharge

The load runs **0.5C constant current** to **3.30 V/cell**, logging per-cell voltages. The four cell curves land in a single **multi-dimensional measurement with four Y axes**, which the dashboard renders as one interactive chart. A weak cell diverging past 60% depth of discharge is visible at a glance, and the imbalance limit catches it numerically:

```python filename="phases/cc_discharge.py"
import numpy as np


def cc_discharge(measurements, bench, phase, log):
    """0.5C constant-current discharge to 3.30 V/cell with per-cell logging."""
    data = bench.discharge_capture(0.5, 3.30)

    # Hardware interlocks are the real protection; this software guard
    # alerts the operator before the bunker suppression has to act. It
    # sits after the capture only because the mock returns a completed
    # capture in one call; a real bench polls temperature_rate() inside
    # the acquisition loop and stops mid-discharge.
    if bench.temperature_rate() > 5.0:
        log.error("Thermal runaway onset detected, stopping run")
        phase.stop()

    cells = np.array(data["cells_v"])
    measurements.discharge_curves.x_axis = data["time_s"]
    measurements.discharge_curves.y_axis.cell_1 = data["cells_v"][0]
    measurements.discharge_curves.y_axis.cell_2 = data["cells_v"][1]
    measurements.discharge_curves.y_axis.cell_3 = data["cells_v"][2]
    measurements.discharge_curves.y_axis.cell_4 = data["cells_v"][3]

    measurements.capacity_delivered_pct = 100.0 * data["delivered_mah"] / data["capacity_mah"]
    measurements.imbalance_max_mv = float((cells.max(axis=0) - cells.min(axis=0)).max() * 1000.0)
    temps = np.array(data["pack_temp_c"])
    measurements.temp_rise_k = float(temps.max() - temps[0])
    log.info(f"Delivered {data['delivered_mah']:.0f} mAh")
```

`phase.stop()` aborts the entire run on early runaway warning (dT/dt above 5 K/min, well below the ~1 °C/s onset criteria used in standards work), and the teardown recharge still executes afterwards. In this mock the guard runs after the capture because the mock returns a completed capture in one call; a real bench polls the rate inside its acquisition loop. The corresponding measurement declarations:

```yaml filename="procedure.yaml"
measurements:
  - name: Discharge Curves
    key: discharge_curves
    title: Per-Cell Discharge Curves
    x_axis:
      legend: Time
      unit: s
    y_axis:
      - {legend: Cell 1, key: cell_1, unit: V}
      - {legend: Cell 2, key: cell_2, unit: V}
      - {legend: Cell 3, key: cell_3, unit: V}
      - {legend: Cell 4, key: cell_4, unit: V}
  - name: Capacity Delivered
    key: capacity_delivered_pct
    unit: "%"
    validators:
      - {operator: ">=", expected_value: 95.0}
  - name: Max Cell Imbalance
    key: imbalance_max_mv
    unit: mV
    validators:
      - {operator: "<=", expected_value: 30.0}
  - name: Temperature Rise
    key: temp_rise_k
    unit: K
    validators:
      - {operator: "<=", expected_value: 25.0}
```

![Stacked time series of the discharge: pack voltage from 16.8 V to 13.2 V over 120 minutes, per-cell voltages with cell 3 falling faster in the last 40%, and pack temperature rising from 22 °C to 36 °C under the 60 °C bunker cutoff.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-2bbc6da4-5473-45d8-a528-c4b4232d0efb-jFc7n3GEgGjqmSNyVrCCZelIoQGqwP.png)

What the station logs at 1 Hz over the two hours: pack voltage, the four cell voltages, and pack temperature, which rises 14 K against the 25 K limit and stays far from the 60 °C bunker cutoff.

### Internal Resistance Pulse

At 50% SOC, after a rest to settle polarization, the load applies a **5C pulse for 2 s**. The phase samples per-cell voltage just before the pulse (V_open) and 100-200 ms into the pulse (V_load). R_DC per cell = (V_open - V_load) / I_pulse. DC IR runs higher than 1 kHz AC IR because it includes charge-transfer resistance on top of the ohmic part, which is exactly why it predicts real-world voltage sag better (see [Hioki's white paper on Li-ion internal resistance testing](https://www.hioki.com/global/industries-solutions/manufacturing/liion-internal-test.html)):

```python filename="phases/ir_pulse.py"
import numpy as np


def ir_pulse(measurements, bench, log):
    """DC IR per cell from a 2 s 5C pulse at 50% SOC: R = dV / dI."""
    pulse = bench.ir_pulse(5.0, 2.0)
    dv = np.array(pulse["v_open"]) - np.array(pulse["v_load"])
    ir_mohm = dv / pulse["current_a"] * 1000.0
    for i, r in enumerate(ir_mohm):
        log.info(f"Cell {i + 1}: {r:.2f} mOhm")
    measurements.ir_max_mohm = float(ir_mohm.max())
    measurements.ir_spread_mohm = float(ir_mohm.max() - ir_mohm.min())
```

The declared limits: **max cell IR 2-8 mΩ** (a value below 2 suggests an internal micro-short, above 8 a bad tab weld) and **IR spread below 1.5 mΩ** across the pack:

```yaml filename="procedure.yaml"
measurements:
  - name: Max Cell IR
    key: ir_max_mohm
    unit: mΩ
    validators:
      - {operator: ">=", expected_value: 2.0}
      - {operator: "<=", expected_value: 8.0}
  - name: IR Spread
    key: ir_spread_mohm
    unit: mΩ
    validators:
      - {operator: "<=", expected_value: 1.5}
```

### Burst Sag Transient

At 30% SOC, a **10C burst for 200 ms** simulates a motor demand transient, captured at 1 kHz. The instantaneous step at burst onset is **ESR (ohmic)**; the slower exponential droop is **polarization**. The phase reports the worst-cell total sag and the ESR share:

```python filename="phases/burst_sag.py"
import numpy as np


def burst_sag(measurements, bench, log):
    """10C 200 ms burst at 30% SOC. The instantaneous step at burst onset
    is ohmic (ESR); the exponential droop that follows is polarization."""
    burst = bench.burst_capture(10.0, 200)
    v = np.array(burst["worst_cell_v"])
    sag_total = float(v[0] - v.min())
    esr_share = 100.0 * burst["esr_drop_v"] / sag_total
    measurements.sag_max_v = sag_total
    measurements.esr_share_pct = esr_share
    log.info(f"Sag {sag_total:.3f} V, ESR share {esr_share:.0f}%")
```

The worst-cell sag validates `<= 0.4 V`; the ESR share is recorded without validators, so it lands in the report with an UNSET outcome that does not affect pass/fail but stays trendable:

```yaml filename="procedure.yaml"
measurements:
  - name: Worst Cell Sag
    key: sag_max_v
    unit: V
    validators:
      - {operator: "<=", expected_value: 0.4}
  - name: ESR Share
    key: esr_share_pct
    unit: "%"
```

![1 kHz burst transient: load current stepping 0 to 30 A at 50 ms and held 200 ms, worst-cell voltage showing the instantaneous 126 mV ESR step, the exponential polarization droop to a 236 mV total sag, and the recovery when the current returns to zero.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-dc3614f2-4df4-43e1-8749-3411aa8b7599-sdZiCIKyuboIsbB4VrjtvFI4WCNkTs.png)

The burst on the mock's weak cell: 30 A into 4.2 mΩ gives the 126 mV ohmic step, the 80 ms polarization droop adds another 110 mV, and the 236 mV total stays under the 400 mV limit with the ESR share at 53%.

### Recharge to Storage SOC

The teardown phase recharges CC/CV to **3.85 V/cell** with active balancing (3.70-3.75 V/cell for air-shipped packs, per the IATA 30% SOC rule above), and validates the final per-cell voltage as the last QA check:

```python filename="phases/recharge_storage.py"
import numpy as np


def recharge_storage(measurements, bench, log):
    """Teardown: always runs, even when a main phase failed, so no pack
    is ever left at full charge in the bunker."""
    cells = bench.recharge_to_storage(3.85)
    storage_v = float(np.mean(cells))
    measurements.storage_voltage_v = storage_v
    log.info(f"Pack stored at {storage_v:.3f} V/cell")
```

### Mock Bench Plug

The `BatteryBench` plug simulates the load and balance ADC, generating a synthetic 4S pack whose cell 3 has elevated IR and droops past 60% depth of discharge. Its `config` block in YAML becomes constructor arguments, so pack chemistry lives in the procedure file, not the code. Swap the class for one speaking SCPI to a Chroma or Kikusui load; the phases stay unchanged.

Note two framework behaviors visible in the phase code: plug attributes are not readable across the process boundary (only method calls are proxied, which is why the mock returns `capacity_mah` inside the data dict), and a measurement read back from `measurements.<key>` returns a proxy, so keep a local variable when you need to format the value for a log line.


### Gimbal Three-Axis Alignment Calibration
URL: https://www.tofupilot.com/templates/gimbal-three-axis-alignment-calibration
Calibrate a drone 3-axis camera gimbal for IMU alignment, motor balance, and horizon accuracy to deliver stable cinematic and FPV video output.

## Introduction

### Three-Axis Gimbal Overview

A drone's **3-axis gimbal** uses three brushless motors (pitch, roll, yaw) under closed-loop control of an **IMU mounted on the camera tray** to keep the camera oriented in inertial space regardless of airframe motion. Stabilization error that reaches the camera shows up directly in the footage, as horizon tilt or rolling-shutter "jello" artifacts. Modern production gimbals achieve **±0.02° angular vibration range** ([DJI Ronin 2 spec](https://www.dji.com/ronin-2/info)) using encoder feedback on every motor and high-rate IMU fusion.

![Exploded view of a drone 3-axis gimbal: yaw motor at the top, roll and pitch motors below, the camera tray with its tray IMU at the bottom, and the coiled cable routing signal across the axes.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-b4c9c9e9-8b1f-4fe3-b3a9-7fa23a4b0c54-mhE3AwDpnIJZRhIkTM3dwpQH9EQ0I8.jpg)

Yaw, roll and pitch motors stacked above the camera tray that carries the gimbal IMU.

The controller is typically a **BaseCam SimpleBGC (AlexMos) 32-bit** or a custom STM32-class board running a proprietary stack (DJI, Skydio, Freefly). Two IMUs are common: the **camera-tray IMU** (closed-loop control reference) and a **frame IMU** (drone attitude input for feed-forward). Both must be calibrated to the airframe so that the gimbal levels the camera **against the actual horizon**, not against a board-mounted reference rotated by manufacturing tolerance.

### Calibration Purpose

Gimbal alignment calibration estimates:

- **IMU-to-airframe rotation**: the rotation between the gimbal IMU body frame and the drone airframe frame (level horizon, forward heading)
- **Motor balance / payload center of mass**: eliminates motor torque demand at zero rate so the gimbal can hold position with minimum current
- **Stabilization accuracy**: the residual horizon error under simulated airframe disturbance, judged against the product grade

![Split view: airframe roll and pitch during the simulated maneuver on the left, camera attitude tracked by the external reference on the right, holding 0.040° RMS against the ±0.1° cinema limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-3734c05c-c467-40bf-8402-c6999477c7fe-pHy6UwXTqIhKgJLsldIwgexU4wecUm.png)

Airframe motion on the left, what the camera actually sees on the right: the residual is what the customer notices.

Per-unit calibration addresses a leading source of cinematic drone returns: **horizon tilt** (camera not level when drone is level) and **mid-shot drift** (slow horizon walk during a pan). Both look identical to the customer and both come from per-unit IMU mounting tolerance plus motor balance error.

## Equipment & Setup

To implement gimbal calibration on a production line, the following are required:

- **A level reference fixture**: granite plate or precision tilt jig with sub-arc-minute leveling
- **An external attitude reference**: a precision tilt sensor or vision-based optical tracker
- **A motion stage**: robotic arm or air-bearing rotary table to apply known disturbances
- **The Device Under Test (DUT)**: drone with gimbal and target camera/lens combo installed
- **A TofuPilot Framework procedure** to script the calibration, capture telemetry, fit calibration parameters, and validate metrics
- **The TofuPilot Dashboard** to store calibration parameters for traceability and production-stability trending

### Hardware Components

#### Level Reference Fixture

A **granite surface plate** (Grade AA or A, flatness within a few micrometres at bench sizes per ASME B89.3.7; Grade B runs roughly 5 µm at 300 mm) provides the production-grade level reference. The drone clamps to the plate via a **non-magnetic fixture** that holds the airframe flat and forward-facing. Magnetic fixtures interfere with the magnetometer in the airframe and gimbal IMU, so production uses **aluminum or 3D-printed PETG/ABS** fixtures.

![Three-axis camera gimbal suspended over a reference plate opposite a target](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/gimbal-three-axis-alignment-calibration-physical-2026-RtH9Twcvv6KQCEzKaeXDpPFuK4pPvi.webp)

The gimbal hangs from its mounting interface above the reference plate. The target faces the camera, with its plain back visible in this view.

#### External Attitude Reference

The gimbal's own IMU cannot validate itself. Production uses an **external attitude reference**: a tactical-grade MEMS IMU such as the [Memsense MS-IMU3025](https://memsense.com/products/ms-imu3025/) (0.6-0.8 °/h gyro bias instability), a **Honeywell HG1700 ring-laser IMU** (RLG tactical grade, overkill for most drones), or a **vision-based optical tracker** ([OptiTrack PrimeX 41](https://www.optitrack.com/cameras/primex-41/specs.html): 2048×2048, 180 FPS, ±0.10 mm 3D accuracy, with retro-reflective markers on the camera tray). The optical tracker is the preferred production choice because it has no mass and adds no perturbation to the gimbal.

For lower volume or development benches, a **digital protractor** (Wixey WR365, ±0.1°) suffices for static angle measurement but cannot capture dynamic response.

#### Disturbance Source

To validate stabilization accuracy, the airframe must be **disturbed in a controlled way**. Production benches use a **6-DOF Stewart platform** or a **robotic arm** (a UR-series cobot such as the UR7e with a clamp end-effector) to drive sinusoidal roll, pitch, and yaw perturbations of ±10° at 0.5-5 Hz, matching typical flight disturbance spectra. For lower-cost lines, a **manual rocking jig** with a tach pickup is sufficient for go/no-go testing.

### Custom Firmware

Both the **drone autopilot** and the **gimbal controller** must expose telemetry of their respective IMU outputs in raw form, plus the encoder positions for each gimbal motor. The procedure also requires a **factory calibration mode** that disables runtime offset compensation so the calibration sees the unfiltered sensor outputs. PX4 autopilots expose this via `mavlink_send_param`, BaseCam SimpleBGC via the [SBGC32 Serial API](https://www.basecamelectronics.com/serialapi/). When writing calibration back to a SimpleBGC board, `CMD_WRITE_PARAMS_3` saves to EEPROM; firmware 2.70+ recommends wrapping writes in `CMD_WRITE_PARAMS_SET` configuration mode, and the board resets about one second after the write, which the read-back verification must wait out.

## Test Procedure

### Overview

After clamping the DUT to the granite reference and connecting the external attitude tracker, the procedure runs:

1. Mount Check: the operator confirms the mechanical preparation and selects the product grade.
2. Rotation Fit: static gravity capture and IMU-to-airframe misalignment computation.
3. Motor Balance: slow sweep per axis, current trace capture, DC offset extraction.
4. Stabilization Test: disturbance rejection measured by the external tracker, judged against the grade the operator selected.
5. Stream results to TofuPilot for traceability and analytics.

Total cycle ~3-5 minutes per DUT.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

This template additionally demonstrates the **operator UI components** (image reference, checklist, radio, textarea), **operator input captured as measurements via `bind`**, **previous-results injection** between phases, and **run metadata**.

### Project Structure

```tree
procedure.yaml
phases/
  mount_check.py
  rotation_fit.py
  motor_balance.py
  stabilization.py
plugs/
  gimbal_rig.py
images/
  mounting.png
ui.json
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-gimbal-alignment-calibration). The `GimbalRig` plug is a mock that simulates a slightly misaligned tray IMU and a mild pitch-axis imbalance, so the procedure runs end-to-end without hardware:

```bash
tofupilot run .
```

For CI or bench automation, `ui.json` pre-bakes the operator inputs and the run executes headless:

```bash
tofupilot run . --no-tui --json --ui-values ui.json --ui-timeout 60
```

### The Procedure File

`procedure.yaml` declares the unit, the rig plug, and four phases. The first phase is the operator-facing one, and its `ui.components` block carries the whole interaction: a reference **image**, a **checklist** of balance steps, a **radio** for the product grade, and a **textarea** for free-form notes. Every input component is **bound to a measurement**, so the run report records what the operator actually confirmed:

```yaml filename="procedure.yaml"
name: Gimbal Three-Axis Alignment Calibration
version: 0.1.0
description: Calibrates gimbal IMU-to-airframe alignment, checks motor balance, and validates stabilization residual against a grade selected by the operator.

plugs:
  - name: Gimbal Rig
    description: SBGC serial link plus external attitude reference. Mock simulates a slightly misaligned tray IMU and a mild pitch-axis imbalance.
    python: plugs.gimbal_rig:GimbalRig
    key: rig

unit:
  auto_identify: true
  serial_number:
    default_value: "GMB-000042"
  part_number:
    default_value: "GIMBAL-3X-PRO"

main:
  - name: Mount Check
    key: mount_check
    python: phases.mount_check
    measurements:
      - name: Balance Steps Done
        key: balance_steps_done
      - name: Grade Selected
        key: grade_selected
        validators:
          - operator: in
            expected_value: ["cinema", "fpv"]
      - name: Operator Notes
        key: operator_notes
    ui:
      components:
        - key: mounting_diagram
          type: image
          label: "Mounting Reference"
          description: "Clamp the airframe flat on the granite plate as shown"
          default_value: "./images/mounting.png"
          width: 50%
        - key: balance_steps
          type: checklist
          label: "Balance Adjustment Steps"
          description: "Complete every step before starting the capture"
          required: true
          bind: measurements.balance_steps_done
          options:
            - label: "Camera tray slid to neutral"
              value: "tray_neutral"
            - label: "Counterweight locked"
              value: "counterweight_locked"
            - label: "Cables routed slack across axes"
              value: "cables_slack"
        - key: grade
          type: radio
          label: "Product Grade"
          description: "Limits tighten for cinema-grade units"
          required: true
          bind: measurements.grade_selected
          options:
            - label: "Cinema (0.1° RMS)"
              value: "cinema"
            - label: "FPV (0.3° RMS)"
              value: "fpv"
        - key: notes
          type: textarea
          label: "Operator Notes"
          description: "Anything unusual about this unit"
          bind: measurements.operator_notes

  - name: Rotation Fit
    key: rotation_fit
    python: phases.rotation_fit
    depends_on: [mount_check]
    measurements:
      - name: Roll Misalignment
        key: roll_misalign_deg
        unit: deg
        validators:
          - {operator: ">=", expected_value: -1.5}
          - {operator: "<=", expected_value: 1.5}
      - name: Pitch Misalignment
        key: pitch_misalign_deg
        unit: deg
        validators:
          - {operator: ">=", expected_value: -1.5}
          - {operator: "<=", expected_value: 1.5}

  - name: Motor Balance
    key: motor_balance
    python: phases.motor_balance
    depends_on: [mount_check]
    measurements:
      - name: Pitch Current vs Angle
        key: pitch_current_curve
        title: Pitch Motor Current vs Angle
        description: Slow sweep through ±30°. A balanced axis is a zero-centered sinusoid; a DC offset is torque compensating a shifted center of mass.
        x_axis:
          legend: Angle
          unit: deg
        y_axis:
          - legend: Current
            key: current
            unit: mA
      - name: Pitch DC Offset
        key: pitch_dc_offset_ma
        unit: mA
        validators:
          - {operator: ">=", expected_value: -50.0}
          - {operator: "<=", expected_value: 50.0}

  - name: Stabilization Test
    key: stabilization
    python: phases.stabilization
    depends_on: [rotation_fit, motor_balance]
    measurements:
      - name: Stabilization Residual
        key: residual_rms_deg
        unit: deg
      - name: Within Grade Limit
        key: within_grade_limit
        validators:
          - operator: "=="
            expected_value: true
```

Note the shape of the dependency graph: `rotation_fit` and `motor_balance` both depend only on `mount_check`, so the framework runs them in parallel, and `stabilization` waits for both.

### Mount Check Phase

The Python side of the operator phase is nearly empty. The UI components collect the inputs, `bind` writes them into measurements, and the `grade_selected` validator (`in ["cinema", "fpv"]`) enforces that a grade was chosen:

```python filename="phases/mount_check.py"
def mount_check(log, unit):
    """Operator phase: mounting diagram, balance checklist, grade radio,
    and free-form notes. Every input lands in a measurement via bind, so
    the run report keeps what the operator actually confirmed."""
    log.info(f"Preparing gimbal {unit.serial_number} for calibration")
```

The image component sources `./images/mounting.png` relative to the procedure directory. The checklist value arrives as the array of checked option values, the radio as the selected value string. `balance_steps_done` and `operator_notes` declare no validators, so they record with an UNSET outcome: stored and queryable in the report, but never affecting pass/fail.

### Rotation Fit

With the airframe level on the granite plate, a perfectly mounted tray IMU reads gravity as [0, 0, -g]. Roll and pitch misalignment fall directly out of the measured vector components:

```python filename="phases/rotation_fit.py"
import numpy as np


def rotation_fit(measurements, rig, log):
    """Fit the tray-IMU-to-airframe rotation from the static gravity vector.

    With the airframe level on the granite plate, a perfectly mounted IMU
    reads [0, 0, -g]. Roll and pitch misalignment fall directly out of the
    measured vector components."""
    v = np.array(rig.static_gravity_capture(5, 200))
    g = np.linalg.norm(v)
    pitch = float(np.degrees(np.arcsin(-v[0] / g)))
    roll = float(np.degrees(np.arcsin(v[1] / (g * np.cos(np.radians(pitch))))))
    measurements.roll_misalign_deg = roll
    measurements.pitch_misalign_deg = pitch
    log.info(f"Misalignment roll {roll:+.2f} deg, pitch {pitch:+.2f} deg")
```

![Gravity vector read by the tray IMU versus the airframe vertical in the roll and pitch planes: 0.80° roll and -0.40° pitch misalignment, brought back onto vertical by the fitted rotation.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-691a1bdb-46c2-4415-9538-01691c2d1d83-OYYBIsbaJDh3IjDF7iJDPEay55E7yW.png)

The mock rig's tray IMU sits 0.80° off in roll and -0.40° in pitch; the fit recovers both to two decimals.

### Motor Balance

A well-balanced gimbal **draws zero motor current at rest** because gravity passes through the rotational axis. Production tolerance, lens swaps, and accessory mounts shift the COM by millimeters and the motor must work to compensate, wasting battery and limiting peak slew rate.

The phase sweeps the pitch axis through ±30° while logging motor current. A balanced axis shows a **sinusoidal current vs angle pattern centered on zero**; an unbalanced axis shows a **constant DC offset** proportional to torque demand. The full trace lands in a **multi-dimensional measurement**, rendered by the dashboard as an interactive chart, and the DC offset is validated numerically:

```python filename="phases/motor_balance.py"
import numpy as np


def motor_balance(measurements, rig, log):
    """Sweep the pitch axis through ±30° and split the current trace into
    its sinusoidal gravity term and the DC offset that indicates a shifted
    center of mass."""
    sweep = rig.sweep_axis("pitch", 30, 61)
    current = np.array(sweep["current_ma"])

    measurements.pitch_current_curve.x_axis = sweep["angles_deg"]
    measurements.pitch_current_curve.y_axis.current = sweep["current_ma"]

    dc_offset = float(current.mean())
    measurements.pitch_dc_offset_ma = dc_offset
    log.info(f"Pitch DC offset {dc_offset:+.1f} mA")
```

![Pitch motor current versus angle over a ±30° sweep: before balance adjustment a sinusoid riding on a +200 mA DC offset, after adjustment the same sinusoid with +29 mA DC offset inside the ±50 mA limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-bce4e39d-1f60-410e-9e80-44fdcf354b2d-7pEn0fHeTBcXfqIigKta8w0zLuIR5Y.png)

Same sweep before and after the operator's balance adjustment; only the DC offset moves.

### Stabilization Test

The disturbance source drives the airframe through a sinusoidal pattern while the external tracker records how far the camera tray strays from level. The pass limit depends on the grade the operator picked in Mount Check, read here through **previous-results injection**: naming a parameter after a completed phase's key injects that phase's measurements and outcome. The phase reads two upstream phases this way, taking the grade from `mount_check` and logging the misalignment the `rotation_fit` phase measured, then stamps the grade and the operator notes onto the **run metadata**, so the dashboard can filter runs by either:

```python filename="phases/stabilization.py"
import numpy as np

GRADE_LIMITS_DEG = {"cinema": 0.1, "fpv": 0.3}


def stabilization(measurements, rig, log, run, mount_check, rotation_fit):
    """Disturb the airframe and measure how far the camera tray strays from
    level. The pass limit comes from the grade the operator selected in the
    Mount Check phase, read here through previous-results injection."""
    grade = str(mount_check.grade_selected)
    limit = GRADE_LIMITS_DEG[grade]
    log.info(f"Grade {grade}: residual limit {limit} deg RMS")
    roll = float(rotation_fit.roll_misalign_deg)
    pitch = float(rotation_fit.pitch_misalign_deg)
    log.info(
        f"Rotation fit {rotation_fit.outcome}: compensating "
        f"roll {roll:+.2f} deg, pitch {pitch:+.2f} deg"
    )

    capture = rig.disturbance_capture(2.0, 10.0, 5.0)
    residual = np.array(capture["camera_deg"])
    rms = float(np.sqrt(np.mean(residual**2)))

    measurements.residual_rms_deg = rms
    measurements.within_grade_limit = rms <= limit

    run.metadata["grade"] = grade
    run.metadata["operator_notes"] = str(mount_check.operator_notes)
    log.info(f"Residual {rms:.3f} deg RMS")
```

The targets (**< 0.1° RMS** cinema, **< 0.3° RMS** FPV) are internal production limits, not industry standards; set them from your golden-unit data. Because the limit is grade-dependent, the boolean `within_grade_limit` measurement carries the pass/fail decision while `residual_rms_deg` keeps the raw value trendable across both grades.

![Stacked time series of the stabilization test: airframe roll oscillating ±10° at 2 Hz, the gimbal roll motor command rejecting it, and the camera tray attitude from the external tracker staying within ±0.08° for a 0.040° RMS residual.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-ae5a9dba-bee0-40c1-ba29-3a23220f2de0-PqzBcUdz0PDLxVc936w9PFdt4y5lO3.png)

Disturbance in, motor command out, and the camera residual the tracker measures: 0.040° RMS, inside the cinema grade.

### Mock Rig Plug

The `GimbalRig` plug simulates the SBGC serial link and the external attitude reference: a tray IMU mounted 0.8° off in roll and -0.4° in pitch, a pitch axis with a mild center-of-mass offset, and a disturbance-rejection loop with cinema-grade residual. Swap the class for one speaking the SBGC32 Serial API and your OptiTrack NatNet stream; the phases stay unchanged.


### Actuator End-of-Line Dyno Test
URL: https://www.tofupilot.com/templates/actuator-end-of-line-dyno-test
End-of-line dyno test of a humanoid joint actuator: run-in no-load torque, torque constant under stall, efficiency map, torque ripple and its dominant order.


## Introduction

### Actuator End-of-Line Test Overview

A humanoid has thirty to forty of them: a frameless motor, a strain-wave or cycloidal reducer, an output encoder and a drive, packed into one joint module. The actuator is the part of the robot that gets built in volume, and the joint module end-of-line test is where the actuator maker or the robot maker proves each one before it goes into a leg. The vocabulary in the job postings of Figure, Apptronik, Tesla and Agility is consistent: a **dyno** (dynamometer) with a **torque transducer** and a **power analyzer**, driven over EtherCAT from a real-time target, measuring **efficiency, stiffness, backlash, thermal limits** and producing **pass/fail criteria** per unit.

![A compact cylindrical joint actuator on a plate, its output flange with a bolt circle and the copper gear ring of the reducer facing the viewer, a small green drive PCB on the back of the can.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-5ce048f7-6ebe-4750-aeae-b9b07eef6c6a-JsPnJj1JgqFy7RaqiJkByCGKHHalY3.jpg)

One joint actuator: the output flange with its bolt circle, the reducer's gear ring behind it, the frameless motor inside the can, the drive PCB on the back.

The numbers to build the test on are the actuator's own datasheet, cross-checked against measurement. A characterization paper on a quasi-direct-drive actuator measured the motor torque constant at **0.105 ± 0.002 Nm/A under stall with a transducer** and mapped efficiency over 1838 operating points ([arXiv 2202.12395](https://ar5iv.labs.arxiv.org/html/2202.12395)); Harmonic Drive specifies the **no-load running torque of its gear units after a run-in of 2 h at 2000 rpm input** at 20 °C ([SHD catalog](https://www.harmonicdrive.net/_hd/content/documents/SHD-2UH-2SH_GearUnits.pdf)); a strain-wave reducer puts two cycles of torque ripple per input revolution on the output, so a healthy unit's dominant ripple order is twice the ratio. The Kistler 4503B transducer that reads all of this is class 0.05 with ±0.03° of angle, so the bench is never the limit.

### Test Purpose

The procedure records one performance fingerprint per actuator:

- **Drive identity and configuration** before the dyno moves: firmware, current limit, pole pairs, commutation mode
- **No-load running torque** through a run-in, input-referred, with its settled value limited
- **Output torque constant** from a locked-rotor current sweep, with the friction knee and the linearity
- **Efficiency** at sixteen operating points, four speeds by four torques, from shaft power over bus power
- **Torque ripple** over one output revolution, as a percentage of rated torque and as a dominant order
- **Housing temperature** at the start, after the run-in and at the end

![Left, the input-referred no-load torque settling from 0.19 to 0.12 Nm over the 10 minute run-in under the 0.20 Nm limit. Right, output torque against q-axis current under stall, six points on a line with a 1.7 Nm friction knee, the fit at 8.99 Nm/A next to the lossless 10.5 Nm/A line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-62e7e3ac-2203-4d1c-b8a4-c2582ff9eb13-elmhsDbLDXdCVq5H5Np6QBvRY3V6QI.png)

The mock actuator's first two main phases: the no-load torque settling as the grease finds its film, and the stall curve whose slope above the knee is the output-referred torque constant, 86 % of motor Kt times ratio.

The framework mechanics on show are a progress component on a time-scaled long phase, multi-dimensional measurements with derived aggregations, three curves in one measurement with the limits on one, an integer aggregation validated with `==` (the FFT order), string `matches` and JSON `==` on the drive before power, and `setup` and `teardown` stages around a `depends_on` chain.

## Equipment & Setup

To run this test on an actuator line, the following are required:

- **A dynamometer**: a four-quadrant servo load or an eddy-current brake sized for the actuator's peak torque and speed
- **An inline torque transducer** with an angle output, class 0.1 or better
- **A power analyzer** on the DC bus
- **A real-time target** running the actuator's EtherCAT drive
- **The Device Under Test (DUT)**: an assembled joint actuator with its production firmware
- **A TofuPilot Framework procedure** to sequence the dyno, compute the derived numbers and validate the limits
- **The TofuPilot Dashboard** to trend Kt, efficiency and ripple across actuators and lots

### Hardware Components

#### Dynamometer and Transducer

[Magtrol](https://www.magtrol.com/wp-content/uploads/PF-1263-Promotion-A04-EN-240116-NB.pdf) builds frameless-motor and actuator benches from a WB eddy-current dynamometer, a TM 300 inline torque transducer and a custom fixture; a servo motor on a second drive makes a four-quadrant load that can also back-drive the actuator. The [Kistler 4503B](https://www.kistler.com/INT/en/cp/shaft-torque-sensors-dual-range-torque-transducers-measuring-ranges-0.2-nm-to-5-knm-4503b/P0000238) is the reference transducer: dual range 0.2 to 5000 Nm, class 0.05, angle to ±0.03°, 10 kHz bandwidth; HBM's T40B is the alternative at 0.1 % of full scale. Figure's test-development posting names "analog torque cells (HBM)" and "high-resolution absolute encoders (BiSS-C)" on a Speedgoat target with Elmo and Kollmorgen drives; that is the bench this template models.

![Actuator and load motor coupled on a guarded dynamometer test bed](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/actuator-end-of-line-dyno-test-1zlPUoE1YxdfFqUMhahcFT36OSTnxI.webp)

The actuator and load motor share a common axis through an inline torque transducer, with a transparent coupling guard. Drive electronics and the analyzer are external.

#### Drive and Bus

The actuator's own EtherCAT drive commands q-axis current for the stall sweep and speed for the map; the bus power analyzer reads the electrical input. An Apptronik posting lists "peak current, thermal limits" and "motor and encoder calibration" among the things the actuation test engineer owns; the drive configuration is read before the dyno moves because a wrong pole-pair count or current limit runs, produces torque numbers, and produces the wrong ones.

### Where the Limits Come From

No standard specifies an actuator end-of-line test. GB/T 35089-2018 gives a bench method for robot precision reducers and GB/T 30819-2024 covers harmonic reducers for robots; IEC 60034-1 gives the motor's thermal classes; ISO 9283 tests the assembled robot. The limits in this template are the actuator's datasheet and the pack maker's own:

| Test | Limit | Basis |
|---|---|---|
| No-load running torque | ≤ 0.20 Nm input-referred at 2000 rpm, after run-in | reducer datasheet convention (value after 2 h at 2000 rpm) |
| Output torque constant | 8.5 to 9.6 Nm/A | motor Kt 0.105 × ratio 100 × locked-rotor efficiency 0.81 to 0.91 |
| Stall linearity | ≤ 2 % | transducer class 0.05, the rest is the actuator |
| Friction knee | ≤ 3 Nm | reducer static friction at the output |
| Efficiency at rated | ≥ 72 % | datasheet 80 % at 60 Nm / 30 rpm, minus tolerance |
| Efficiency map minimum | ≥ 50 % | light-load corner of the map |
| Torque ripple | ≤ 3 % of rated p-p | pack maker's spec; vendor guidance quotes 0.5 to 2 % |
| Dominant ripple order | == 200 | 2 × ratio for a strain-wave reducer |

The vendor-article figures that circulate ("65 to 80 % efficiency", "15 to 30 minutes per joint") are guidance without test data; the datasheet and the measured Kt are the anchors.

## Test Procedure

### Overview

The procedure maps the test onto the framework's three stages. The drive readback lives in `setup:` so no torque number is recorded on a mis-configured drive. Parking lives in `teardown:` so the dyno unloads and the drive disables whatever happened.

1. Setup: firmware, drive configuration, housing temperature.
2. Main: run-in at 2000 rpm input, no-load torque curve, settled value.
3. Main: locked-rotor current sweep, Kt, friction knee, linearity.
4. Main: sixteen-point efficiency map.
5. Main: one revolution at 3 rpm under load, ripple and dominant order.
6. Teardown: unload, disable, temperature.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for trending.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  run_in.py
  torque_constant.py
  efficiency_map.py
  torque_ripple.py
  park.py
plugs/
  actuator_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-actuator-eol-dyno). The `ActuatorBench` plug is a mock of the dyno, the transducer, the power analyzer and the drive together, synthesizing a healthy actuator on datasheet values, so the procedure runs end-to-end without a bench or an actuator connected. The run-in is **time-scaled**: the mock returns the ten minutes in one call.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Actuator End-of-Line Dyno Test
version: 0.1.0
description: End-of-line dynamometer test of a humanoid joint actuator (frameless motor, 100:1 harmonic reducer, output encoder). Drive identity and configuration, time-scaled run-in with the no-load running torque, output torque constant under stall, sixteen-point efficiency map, torque ripple with order tracking over one revolution.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the actuator housing label"
    placeholder: "ACT-HIP-000000"
    pattern: "^ACT-[A-Z]{3}-\\d{6}$"
    default_value: "ACT-HIP-004812"
  part_number:
    default_value: "ACT-R100-60NM"
  batch_number:
    default_value: "WK-2026-37"

plugs:
  - name: Actuator Bench
    description: "Four-quadrant dyno, inline torque transducer with angle, bus power analyzer, EtherCAT drive (mock, one plug per bench)"
    python: plugs.actuator_bench:ActuatorBench
    key: bench

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Firmware Version
        key: firmware_version
        validators:
          - {operator: matches, expected_value: "^3\\.\\d+\\.\\d+$"}
      - name: Drive Config
        key: drive_config
        description: Current limit, pole pairs, commutation mode and brake setting as one object; a wrong pole-pair count runs and produces numbers, all of them wrong.
        validators:
          - operator: "=="
            expected_value:
              current_limit_a: 30
              pole_pairs: 21
              commutation: sincos
              brake: none
      - name: Temperature Start
        key: temperature_start_c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 18.0}
          - {operator: "<=", expected_value: 28.0}

main:
  - name: Run In
    key: run_in
    python: phases.run_in
    timeout: 30m
    ui:
      components:
        - key: run_in_progress
          type: progress
          label: "Run-in at 2000 rpm input"
          description: "10 min unloaded, no-load torque logged every 10 s"
          default_value: 0
          max: 100
    measurements:
      - name: No-Load Torque
        key: no_load
        title: Input-referred no-load running torque during the run-in
        x_axis:
          legend: Time
          unit: min
        y_axis:
          - legend: Torque
            key: torque
            unit: Nm
            aggregations:
              - type: final_nm
                unit: Nm
                validators:
                  - {operator: "<=", expected_value: 0.20}
              - type: settle_pct
                unit: "%"
      - name: Temperature After Run-In
        key: temperature_after_run_in_c
        unit: °C
        validators:
          - {operator: "<=", expected_value: 45.0}

  - name: Torque Constant
    key: torque_constant
    python: phases.torque_constant
    depends_on: [run_in]
    measurements:
      - name: Stall
        key: stall
        title: Output torque against q-axis current, dyno locked
        x_axis:
          legend: Current
          unit: A
        y_axis:
          - legend: Torque
            key: torque
            unit: Nm
            aggregations:
              - type: kt_nm_per_a
                unit: Nm/A
                validators:
                  - {operator: ">=", expected_value: 8.5}
                  - {operator: "<=", expected_value: 9.6}
              - type: linearity_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 2.0}
              - type: friction_knee_nm
                unit: Nm
                validators:
                  - {operator: "<=", expected_value: 3.0}

  - name: Efficiency Map
    key: efficiency_map
    python: phases.efficiency_map
    depends_on: [torque_constant]
    timeout: 10m
    measurements:
      - name: Efficiency
        key: efficiency
        title: Efficiency at sixteen operating points, four speeds by four torques
        x_axis:
          legend: Point
        y_axis:
          - legend: Speed
            key: speed
            unit: rpm
          - legend: Torque
            key: torque
            unit: Nm
          - legend: Efficiency
            key: efficiency
            unit: "%"
            aggregations:
              - type: at_rated_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 72.0}
              - type: min_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 50.0}

  - name: Torque Ripple
    key: torque_ripple
    python: phases.torque_ripple
    depends_on: [efficiency_map]
    measurements:
      - name: Ripple
        key: ripple
        title: Torque over one output revolution at 3 rpm under 30 Nm
        x_axis:
          legend: Angle
          unit: °
        y_axis:
          - legend: Torque
            key: torque
            unit: Nm
            aggregations:
              - type: pp_pct_rated
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 3.0}
              - type: dominant_order
                validators:
                  - {operator: "==", expected_value: 200}

teardown:
  - name: Park
    key: park
    python: phases.park
    measurements:
      - name: Temperature End
        key: temperature_end_c
        unit: °C
        validators:
          - {operator: "<=", expected_value: 60.0}
```

Framework features to notice:

1. **Progress on a long phase.** The run-in declares a `progress` component and a 30 minute `timeout`; the phase writes `ui.run_in_progress` and the operator sees where the run is. The mock returns the ten minutes in one call.
2. **Derived aggregations.** The stall curve records six torque points; the pass/fail lives on `kt_nm_per_a`, `linearity_pct` and `friction_knee_nm`, all computed in Python from the fit and named in the YAML with their own limits.
3. **Three curves, limits on one.** The efficiency measurement records speed, torque and efficiency per point; `at_rated_pct` and `min_pct` carry the limits.
4. **An FFT order as an integer `==`.** `dominant_order == 200` is a physical signature: the wave generator of a 100:1 strain-wave reducer. A bearing defect or a damaged flexspline moves it.
5. **Drive readback before power.** `drive_config` compares four registers as one object; `firmware_version` validates with `matches`.

### Identify

The setup phase reads the drive's identity and configuration, the housing temperature, and stamps the reducer ratio and the encoder resolution onto the **unit metadata**:

```python filename="phases/identify.py"
def identify(measurements, bench, unit, log):
    """Setup: EtherCAT link up, firmware and drive configuration read
    before the dyno moves. A drive with the wrong pole-pair count or
    current limit would produce plausible torque numbers at the wrong
    operating point."""
    ident = bench.drive_identify()
    measurements.firmware_version = ident["firmware"]
    measurements.drive_config = bench.drive_config()
    measurements.temperature_start_c = bench.temperature_c()
    unit.metadata["reducer_ratio"] = ident["ratio"]
    unit.metadata["encoder_bits"] = ident["encoder_bits"]
    log.info(f"Actuator {unit.serial_number}: fw {ident['firmware']}, ratio {ident['ratio']}:1, {ident['encoder_bits']}-bit output encoder, {measurements.temperature_start_c} C")
```

### Run In

Ten minutes unloaded at 2000 rpm input, the no-load torque logged every 10 s. The value at the end is the one the reducer maker's datasheet is written against; the curve is recorded for the trend, and the percentage it settled by is an aggregation without a limit:

```python filename="phases/run_in.py"
import numpy as np

from utils.recipe import RUN_IN_INPUT_RPM, RUN_IN_MINUTES


def run_in(measurements, bench, ui, log):
    """Unloaded run-in at 2000 rpm input. The no-load running torque is
    only meaningful after the grease has redistributed, which is why the
    reducer maker specifies it after a run; the value at the end of the
    run is the one with a limit, the curve is there for the trend."""
    bench.enable()
    cap = bench.run_in(RUN_IN_INPUT_RPM, RUN_IN_MINUTES)
    t = np.array(cap["time_s"])
    torque = np.array(cap["torque_nm"])
    ui.run_in_progress = 100
    final = float(torque[t >= t[-1] - 60.0].mean())

    measurements.no_load.x_axis = (t / 60.0).round(2).tolist()
    measurements.no_load.y_axis.torque = cap["torque_nm"]
    measurements.no_load.y_axis.torque.aggregations.final_nm = final
    measurements.no_load.y_axis.torque.aggregations.settle_pct = float(100.0 * (torque[:3].mean() - final) / torque[:3].mean())
    measurements.temperature_after_run_in_c = bench.temperature_c()
    log.info(f"No-load torque {torque[0]:.3f} -> {final:.3f} Nm input-referred over {RUN_IN_MINUTES:.0f} min, {measurements.temperature_after_run_in_c} C")
```

### Torque Constant

Dyno locked, q-axis current stepped from 0 to 25 A, output torque from the transducer at each step. Below the friction knee the reducer's static friction eats the torque; the fit above 10 A gives the output-referred torque constant, and the intercept gives the knee:

```python filename="phases/torque_constant.py"
import numpy as np

from utils.recipe import KT_CURRENT_STEPS_A, KT_OUTPUT_EXPECTED


def torque_constant(measurements, bench, log):
    """Dyno locked, q-axis current stepped, output torque read on the
    transducer. The slope above the friction knee is the output-referred
    torque constant: motor Kt times ratio times the locked-rotor
    efficiency of the reducer."""
    torque = [bench.stall_torque_nm(i) for i in KT_CURRENT_STEPS_A]
    currents = np.array(KT_CURRENT_STEPS_A)
    t = np.array(torque)
    fit = currents >= 10.0  # above the friction knee
    slope, intercept = np.polyfit(currents[fit], t[fit], 1)
    resid = t[fit] - (slope * currents[fit] + intercept)
    linearity = float(100.0 * np.abs(resid).max() / t[fit].max())

    measurements.stall.x_axis = KT_CURRENT_STEPS_A
    measurements.stall.y_axis.torque = t.round(3).tolist()
    measurements.stall.y_axis.torque.aggregations.kt_nm_per_a = float(slope)
    measurements.stall.y_axis.torque.aggregations.linearity_pct = linearity
    measurements.stall.y_axis.torque.aggregations.friction_knee_nm = float(-intercept)
    log.info(f"Kt {slope:.2f} Nm/A output-referred ({KT_OUTPUT_EXPECTED:.1f} before losses), linearity {linearity:.2f} %, friction knee {-intercept:.2f} Nm")
```

### Efficiency Map

Four speeds by four torques, the dyno holding each point, shaft power from the transducer over electrical power from the bus analyzer:

```python filename="phases/efficiency_map.py"
import numpy as np

from utils.recipe import MAP_SPEEDS_RPM, MAP_TORQUES_NM, RATED_SPEED_RPM, RATED_TORQUE_NM


def efficiency_map(measurements, bench, log):
    """Sixteen operating points, four speeds by four torques, shaft power
    from the transducer over electrical power from the bus analyzer."""
    speeds, torques, effs = [], [], []
    for s in MAP_SPEEDS_RPM:
        for q in MAP_TORQUES_NM:
            op = bench.operating_point(s, q)
            speeds.append(s)
            torques.append(q)
            effs.append(100.0 * op["mech_w"] / op["elec_w"])
    bench.unload()
    effs = np.array(effs)
    i_rated = [i for i, (s, q) in enumerate(zip(speeds, torques)) if s == RATED_SPEED_RPM and q == RATED_TORQUE_NM][0]

    measurements.efficiency.x_axis = list(range(1, len(effs) + 1))
    measurements.efficiency.y_axis.speed = speeds
    measurements.efficiency.y_axis.torque = torques
    measurements.efficiency.y_axis.efficiency = effs.round(2).tolist()
    measurements.efficiency.y_axis.efficiency.aggregations.at_rated_pct = float(effs[i_rated])
    measurements.efficiency.y_axis.efficiency.aggregations.min_pct = float(effs.min())
    log.info(f"Efficiency {effs.min():.1f}..{effs.max():.1f} %, {effs[i_rated]:.1f} % at rated {RATED_TORQUE_NM:.0f} Nm / {RATED_SPEED_RPM:.0f} rpm")
```

![A four by four grid of efficiency percentages from 65.2 % at 15 Nm and 45 rpm to 80.3 % at 60 Nm and 5 rpm, the rated point at 30 rpm and 60 Nm outlined in red at 79.1 %.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-0c8bae87-277e-48f7-b185-10569e1f17e8-MjhFYHQlh0LKf674uuyYzywgBUHmzk.png)

The mock actuator's map: efficiency falls toward light load as the reducer's fixed losses dominate, and slightly above rated speed. The two limits sit on the rated point and on the map's minimum.

### Torque Ripple

One output revolution at 3 rpm under 30 Nm, torque against angle at 0.1°. The peak-to-peak is expressed as a percentage of rated torque; an FFT over the revolution gives the dominant order, which for a strain-wave reducer is twice the ratio, the wave generator's two lobes per input revolution:

```python filename="phases/torque_ripple.py"
import numpy as np

from utils.recipe import RATED_TORQUE_NM, RATIO, RIPPLE_LOAD_NM, RIPPLE_SPEED_RPM


def torque_ripple(measurements, bench, log):
    """One output revolution at 3 rpm under 30 Nm, torque against angle.
    Peak-to-peak ripple as a percentage of rated torque, and the dominant
    order from an FFT over the revolution: a healthy harmonic drive shows
    2 x ratio (the wave generator), a damaged flexspline or a bad bearing
    shows elsewhere."""
    cap = bench.ripple_capture(RIPPLE_SPEED_RPM, RIPPLE_LOAD_NM)
    torque = np.array(cap["torque_nm"])
    ripple = torque - torque.mean()
    spectrum = np.abs(np.fft.rfft(ripple))
    order = int(np.argmax(spectrum[1:]) + 1)  # cycles per output revolution

    measurements.ripple.x_axis = cap["angle_deg"]
    measurements.ripple.y_axis.torque = cap["torque_nm"]
    measurements.ripple.y_axis.torque.aggregations.pp_pct_rated = float(100.0 * (torque.max() - torque.min()) / RATED_TORQUE_NM)
    measurements.ripple.y_axis.torque.aggregations.dominant_order = order
    log.info(f"Ripple {100.0 * (torque.max() - torque.min()) / RATED_TORQUE_NM:.2f} % of rated p-p, dominant order {order} (2 x ratio = {2 * RATIO})")
```

![Left, torque over the first 60° of the revolution oscillating between 29.8 and 30.3 Nm, 1.06 % of rated peak-to-peak. Right, the order spectrum with a dominant line at order 200 and a smaller one at order 100.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-d61746d4-e8ae-4c75-852e-adda34f5bc77-HftIIwNSjOSoSNA0CroVDobAqxz8WF.png)

The mock's ripple: 1.06 % of rated peak-to-peak with the wave generator at order 200 and the input's once-per-revolution at order 100. A bearing defect shows at the bearing's ball-pass order, a damaged flexspline tooth at the tooth count; both move the dominant order and fail the `==`.

### Park

The teardown phase unloads the dyno, disables the drive and reads the housing temperature, so a hot actuator is not handed to the next station:

```python filename="phases/park.py"
def park(measurements, bench, log):
    """Teardown: dyno unloaded, drive disabled, housing temperature read
    so a hot actuator is not handed to the next station."""
    bench.unload()
    bench.disable()
    measurements.temperature_end_c = bench.temperature_c()
    log.info(f"Parked, housing at {measurements.temperature_end_c} C")
```

### Mock Plug

`ActuatorBench` stands in for the dyno, the transducer, the power analyzer and the drive because the readings depend on each other and plugs run in separate processes. It synthesizes a motor Kt of 0.105 Nm/A ± 0.6 %, a locked-rotor reducer efficiency of 0.86 with a 1.6 Nm friction knee, an efficiency of 0.80 at rated falling quadratically toward light load and linearly above rated speed, a no-load torque that settles from 0.19 to 0.11 Nm with a 3 minute time constant, and a ripple of 1.4 % dominated by the wave generator's second harmonic with the input's first harmonic at a quarter of it. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, the class speaks the drive's CoE objects through pysoem or a Speedgoat model, SCPI to the power analyzer, and reads the transducer through the DAQ with its angle channel; set `TIME_SCALE = 1.0` and the run-in timeout to match the reducer maker's specification. Add a backlash and stiffness phase from the companion template when the same bench carries an output-side reference encoder. The phases, measurements and limits stay the same.


### Solar Array Flash Test
URL: https://www.tofupilot.com/templates/solar-array-flash-test
Electrical acceptance of a solar panel under a pulsed solar simulator: reference pulses, 500 V insulation, string IV corrected to 25 C AM0, bypass diodes.


## Introduction

### Solar Array Flash Test Overview

A satellite's power budget is written against the power its panels deliver at 25 °C under one solar constant of the AM0 spectrum, and the flash test is where that number is measured on every flight panel. A large-area pulsed solar simulator (a LAPSS) fires a xenon pulse of a couple of milliseconds at the panel while an electronic load sweeps each string from short circuit to open circuit; the panel never heats, and a reference cell in the test plane reads the same pulse so every curve can be corrected to the standard conditions whatever the lamp delivered. The test finds cracked cells, open interconnects, shorted or open bypass diodes, strings wired wrong, insulation to the substrate that has broken down, and, when repeated after thermal cycling and vibration, the degradation those environments caused.

![One corner of a satellite solar panel on a plate: dark-blue cells with silver grid lines and interconnect tabs, a small black bypass diode between two of them, the substrate visible at the edge.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-b0c654f2-9696-4991-8b93-0c1d781fcd51-w6xxCLl4vXcb71766PmsjoLDgW74JD.jpg)

One corner of the panel: the cells, their interconnects, and the bypass diode that carries the string current around a shaded or failed cell group.

The standard is ECSS-E-ST-20-08C Rev.2, the photovoltaic assembly standard. Its Table 5-2 gives the acceptance sequence for a flight panel: visual inspection, health check, capacitance, electrical performance, bake-out, thermal cycling, and the same again after; its clause 10 defines the reference cells, a primary standard calibrated on a balloon or in orbit and secondary working standards traced to it; its example health-check values are **insulation resistance above 100 MΩ at 500 V** between circuit and substrate ([ECSS-E-ST-20-08C Rev.2](https://ecss.nl/wp-content/uploads/2023/08/ECSS-E-ST-20-08C_Rev.2(20April2023).pdf)). The flasher itself is characterised in the JPL LAPSS work and the [Spectrolab LAPSS II](https://www.spectrolab.com/illuminations/pdfs/LAPSSII.pdf) datasheet: ±1 % pulse-to-pulse, ±2 % uniformity over the test plane, 0.025 % voltage resolution. The power limit is not in the standard; it is project-defined in the PVA specification, and the test accuracy must be smaller than the calibration loss factor the power budget carries.

### Test Purpose

The procedure records one electrical fingerprint per panel:

- **Lamp intensity and pulse-to-pulse repeatability** from three pulses on the reference cell alone, and the panel temperature
- **Health check**: insulation resistance at 500 V on three paths and the panel capacitance
- **One flash per string**, the IV curve corrected to 25 °C and 1 SC AM0, with Isc, Voc, Pmax and fill factor per string, the panel Pmax and the string match
- **Forward drop of every bypass diode** at string current
- **Lamp pulses consumed**, logged against the flash tube's life

![Left, four corrected IV curves nearly on top of each other, 0.52 A flat to 40 V then falling to 48.6 V, string 3 slightly lower, the maximum-power points marked. Right, the four power curves peaking at 22 W with the 20 W string limit as a dashed line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-f432ce96-984b-49ad-9a1b-13da985fe075-ZqVadDDBad0oHlhdyRJrjwuh41A5gY.png)

The mock panel's four strings after correction: 22.0 to 22.1 W on three of them, 21.8 W on string 3 where one cell sits at the bottom of its grade, all above the 20 W specification and within 1.5 % of each other.

The framework mechanics on show are four curves in one measurement with aggregations and limits on each, a physics correction in `utils/` shared by the phases and the charts, a reference instrument validated in `setup:` on three pulses, a consumable tracked on the unit, and scalar limits taken from a standard's example values.

## Equipment & Setup

To run this acceptance test on a panel line, the following are required:

- **A large-area pulsed solar simulator** with a test plane larger than the panel, and its reference cell holder in the plane
- **A secondary standard reference cell** of the same cell technology as the panel, with a calibration certificate traced to a primary standard
- **An IV load** fast enough to sweep a string during the pulse, triggered by the flasher
- **A megohmmeter** at 500 V DC and an LCR meter for the health check
- **Thermocouples** on the panel substrate
- **The Device Under Test (DUT)**: a flight panel on its handling frame, in the test plane, at room temperature
- **A TofuPilot Framework procedure** to sequence the pulses, correct the curves and validate the limits
- **The TofuPilot Dashboard** to keep the before-and-after record every environmental test is judged against

### Hardware Components

#### Pulsed Solar Simulator

The [Spectrolab LAPSS II](https://www.spectrolab.com/illuminations/pdfs/LAPSSII.pdf) is the reference instrument: a 2 ms xenon pulse over a test plane of several metres, pulse-to-pulse repeatability better than ±1 %, spectral match to AM0 through filtering, voltage resolution 0.025 %. JPL's installation report ([JPL Publication 93-22](https://ntrs.nasa.gov/api/citations/19950015919/downloads/19950015919.pdf)) measured ±2 % uniformity and ±0.3 % repeatability on their unit. Angstrom Designs and Pasan build comparable flashers. Whatever the lamp, the reference cell in the plane on every pulse is what makes the number traceable: the lamp's intensity on that pulse is the reference cell's short-circuit current over its calibrated value.

![Solar panel facing an illumination head on a flash-test fixture](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/solar-array-flash-test-physical-2026-l27ax6UCtpTJVOuFLCBk2csTfSa3EB.webp)

The illumination head faces the panel's active surface. The illustration shows the optical arrangement; the electrical load and flash controls are omitted.

#### Reference Cell

ECSS-E-ST-20-08 clause 10 defines the chain: a primary standard calibrated under the extraterrestrial spectrum (balloon or orbit), secondary working standards of the same cell technology traced to it, recalibrated on a schedule. The reference cell's calibrated short-circuit current at 1 solar constant AM0 and 25 °C is the number every correction in this template divides by; it lives in the recipe and on the certificate, and the setup phase records which cell was in the plane.

#### Health-Check Instruments

A megohmmeter or a Keysight B2985 electrometer at 500 V DC for the circuit-to-substrate, adjacent-strings and thermal-sensor-to-substrate paths; an LCR meter for the panel capacitance, a dielectric health number that moves when the substrate insulation is damaged; type-T thermocouples on the substrate for the correction.

### Where the Limits Come From

ECSS-E-ST-20-08 gives the sequence and the health-check example values; the electrical limits are the project's PVA specification:

| Test | Limit | Basis |
|---|---|---|
| Pulse-to-pulse repeatability | ≤ 1.0 % over three pulses | flasher datasheet ±1 % |
| Lamp intensity | 0.95 to 1.05 SC | correction valid range |
| Panel temperature | 18 to 30 °C | correction valid range, thermocouple on the substrate |
| Insulation resistance | ≥ 100 MΩ at 500 V | ECSS-E-ST-20-08 example values |
| Capacitance | 80 to 160 nF | this panel's build, ±33 % |
| String Pmax | ≥ 20.0 W | PVA specification: 92.6 % of 21.6 W nominal |
| Fill factor | ≥ 0.80 | 3J cells at 0.85 typical; a cracked cell or a bad interconnect lowers it |
| Panel Pmax | ≥ 80 W | four strings at the specification |
| String match | ≤ 3 % | cells graded to match within a panel |
| Bypass diode Vf | 0.5 to 1.2 V at string current | open reads the cells' reverse breakdown, short reads zero |

The thermal-cycling acceptance criterion in the standard is written as a maximum Iop increment per the specification and "fewer than 2 % of cells cracked, or one cell, whichever is higher"; this template's before-and-after comparison is the number that criterion is applied to.

## Test Procedure

### Overview

The procedure maps the acceptance onto the framework's three stages. The reference pulses live in `setup:` so no panel curve is recorded under a lamp out of adjustment or on a warm panel. Shorting the panel lives in `teardown:` so it is never left at open circuit under the lamp.

1. Setup: three reference pulses, lamp intensity, repeatability, panel temperature.
2. Main: insulation resistance on three paths at 500 V, capacitance.
3. Main: one flash per string, corrected IV, Pmax, fill factor, panel Pmax, string match.
4. Main: forward drop of every bypass diode.
5. Teardown: panel shorted, pulses logged.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the panel's acceptance record.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  reference_check.py
  health_check.py
  flash_iv.py
  bypass_diodes.py
  release.py
plugs/
  flash_bench.py
utils/
  recipe.py
  correction.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-solar-array-flash-test). The `FlashBench` plug is a mock of the flasher, the reference cell, the IV load, the megohmmeter, the LCR meter and the thermocouples together, synthesizing a healthy four-string panel at 23 °C with one string 1.5 % low, so the procedure runs end-to-end without a flasher or a panel connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Solar Array Flash Test
version: 0.1.0
description: Electrical acceptance of a smallsat solar panel under a large-area pulsed solar simulator. Reference cell pulses for lamp intensity and repeatability, insulation resistance at 500 V and capacitance, one flash per string with the IV curve corrected to 25 C and 1 SC AM0, string match, every bypass diode's forward drop.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the panel's substrate label"
    placeholder: "SA-WING-000"
    pattern: "^SA-WING-\\d{3}$"
    default_value: "SA-WING-047"
  part_number:
    default_value: "SA-4S18-3G30"
  batch_number:
    default_value: "CELL-LOT-2026-19"

plugs:
  - name: Flash Bench
    description: "Pulsed solar simulator with reference cell, IV load, megohmmeter, LCR meter, thermocouples (mock LAPSS II-class, one plug per bench)"
    python: plugs.flash_bench:FlashBench
    key: bench

setup:
  - name: Reference Check
    key: reference_check
    python: phases.reference_check
    measurements:
      - name: Pulse Repeatability
        key: pulse_repeat_pct
        unit: "%"
        description: Spread of three reference-cell pulses; the flasher's own pulse-to-pulse figure is +-1 %.
        validators:
          - {operator: "<=", expected_value: 1.0}
      - name: Lamp Intensity
        key: lamp_intensity_sc
        unit: SC
        description: Solar constants delivered, from the reference cell; the correction handles 0.95 to 1.05, outside it the lamp is out of adjustment.
        validators:
          - {operator: ">=", expected_value: 0.95}
          - {operator: "<=", expected_value: 1.05}
      - name: Panel Temperature
        key: panel_temp_c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 18.0}
          - {operator: "<=", expected_value: 30.0}

main:
  - name: Health Check
    key: health_check
    python: phases.health_check
    measurements:
      - name: IR Circuit To Substrate
        key: ir_circuit_substrate_mohm
        unit: MΩ
        description: 500 V DC, ECSS-E-ST-20-08 example value greater than 100 MOhm at ambient.
        validators:
          - {operator: ">=", expected_value: 100.0}
      - name: IR Adjacent Strings
        key: ir_adjacent_strings_mohm
        unit: MΩ
        validators:
          - {operator: ">=", expected_value: 100.0}
      - name: IR Sensor To Substrate
        key: ir_sensor_substrate_mohm
        unit: MΩ
        validators:
          - {operator: ">=", expected_value: 100.0}
      - name: Capacitance
        key: capacitance_nf
        unit: nF
        description: Panel capacitance as a dielectric health number; it moves when the substrate insulation is damaged.
        validators:
          - {operator: ">=", expected_value: 80.0}
          - {operator: "<=", expected_value: 160.0}

  - name: Flash IV
    key: flash_iv
    python: phases.flash_iv
    depends_on: [health_check]
    timeout: 5m
    measurements:
      - name: IV Curves
        key: iv
        title: String IV curves corrected to 25 °C, 1 SC AM0
        x_axis:
          legend: Voltage
          unit: V
        y_axis:
          - legend: String 1
            key: string_1
            unit: A
            aggregations:
              - type: pmax_w
                unit: W
                validators:
                  - {operator: ">=", expected_value: 20.0}
              - type: isc_a
                unit: A
              - type: voc_v
                unit: V
              - type: fill_factor
                validators:
                  - {operator: ">=", expected_value: 0.80}
          - legend: String 2
            key: string_2
            unit: A
            aggregations:
              - type: pmax_w
                unit: W
                validators:
                  - {operator: ">=", expected_value: 20.0}
              - type: isc_a
                unit: A
              - type: voc_v
                unit: V
              - type: fill_factor
                validators:
                  - {operator: ">=", expected_value: 0.80}
          - legend: String 3
            key: string_3
            unit: A
            aggregations:
              - type: pmax_w
                unit: W
                validators:
                  - {operator: ">=", expected_value: 20.0}
              - type: isc_a
                unit: A
              - type: voc_v
                unit: V
              - type: fill_factor
                validators:
                  - {operator: ">=", expected_value: 0.80}
          - legend: String 4
            key: string_4
            unit: A
            aggregations:
              - type: pmax_w
                unit: W
                validators:
                  - {operator: ">=", expected_value: 20.0}
              - type: isc_a
                unit: A
              - type: voc_v
                unit: V
              - type: fill_factor
                validators:
                  - {operator: ">=", expected_value: 0.80}
      - name: Panel Pmax
        key: panel_pmax_w
        unit: W
        validators:
          - {operator: ">=", expected_value: 80.0}
      - name: String Match
        key: string_match_pct
        unit: "%"
        description: Spread of string Pmax over the mean; strings in one panel are graded to match and a mismatch is a cell at the wrong grade or a damaged one.
        validators:
          - {operator: "<=", expected_value: 3.0}

  - name: Bypass Diodes
    key: bypass_diodes
    python: phases.bypass_diodes
    depends_on: [flash_iv]
    measurements:
      - name: Diodes
        key: diodes
        title: Forward drop of every bypass diode at string current
        x_axis:
          legend: Diode
        y_axis:
          - legend: Vf
            key: vf
            unit: V
            aggregations:
              - type: max_v
                unit: V
                validators:
                  - {operator: "<=", expected_value: 1.2}
              - type: min_v
                unit: V
                validators:
                  - {operator: ">=", expected_value: 0.5}

teardown:
  - name: Release
    key: release
    python: phases.release
    measurements:
      - name: Pulses This Run
        key: pulses_this_run
        description: Lamp pulses consumed, logged against the flash tube's life.
```

Framework features to notice:

1. **Four strings, one measurement.** The `iv` measurement carries one corrected curve per string on a common voltage axis, each with its own `pmax_w`, `isc_a`, `voc_v` and `fill_factor` aggregations; the limits sit on Pmax and fill factor, the others are recorded for the trend.
2. **Correction in `utils/`.** `correct_to_stc` and `iv_metrics` live in a module both the phase and the article's charts import, so the chart is the run.
3. **The reference proves the lamp first.** Three pulses in `setup:` give the repeatability and the intensity; a lamp out of its window fails before a panel curve is recorded.
4. **A consumable on the unit.** The pulse count goes onto `unit.metadata` and into a teardown measurement without a limit; the flash tube's life is a station number, not a panel number.

### Reference Check

Three pulses on the reference cell alone. Their spread is the flasher's pulse-to-pulse repeatability, their mean over the calibrated value is the intensity the lamp delivers, and the panel thermocouple is read for the correction's validity window. The reference cell's identity goes onto the **unit metadata**:

```python filename="phases/reference_check.py"
import numpy as np

from utils.recipe import PULSE_REPEAT_PCT_MAX, REFERENCE_CELL_ISC_A


def reference_check(measurements, bench, unit, log):
    """Setup: three pulses on the reference cell alone. Their spread is the
    flasher's pulse-to-pulse repeatability, their mean over the calibrated
    value is the intensity the lamp delivers; the panel temperature must
    sit in the window the correction is valid for."""
    isc = np.array([bench.reference_pulse()["isc_a"] for _ in range(3)])
    spread = 100.0 * (isc.max() - isc.min()) / isc.mean()
    measurements.pulse_repeat_pct = float(spread)
    measurements.lamp_intensity_sc = float(isc.mean() / REFERENCE_CELL_ISC_A)
    measurements.panel_temp_c = bench.panel_temperature_c()
    unit.metadata["reference_cell"] = "WS-2026-07, secondary standard"
    log.info(f"Panel {unit.serial_number}: lamp {isc.mean() / REFERENCE_CELL_ISC_A:.4f} SC, pulse spread {spread:.2f} % (limit {PULSE_REPEAT_PCT_MAX}), panel {measurements.panel_temp_c} C")
```

### Health Check

The ECSS health check: insulation resistance at 500 V DC from the circuit to the substrate and from the thermal sensor to the substrate, at 250 V between adjacent strings, and the panel capacitance. Done here before the flash and repeated after every environment; a value that moved is a crack in the substrate insulation:

```python filename="phases/health_check.py"
from utils.recipe import IR_TEST_V


def health_check(measurements, bench, log):
    """Electrical health check per ECSS-E-ST-20-08: insulation resistance
    at 500 V on three paths and the panel capacitance. Done before the
    flash and repeated after environments; a value that moved is a crack."""
    measurements.ir_circuit_substrate_mohm = bench.insulation_mohm("circuit_substrate", IR_TEST_V)
    measurements.ir_adjacent_strings_mohm = bench.insulation_mohm("adjacent_strings", 250)
    measurements.ir_sensor_substrate_mohm = bench.insulation_mohm("sensor_substrate", IR_TEST_V)
    measurements.capacitance_nf = bench.capacitance_nf()
    log.info(f"IR circuit-substrate {measurements.ir_circuit_substrate_mohm} MOhm at {IR_TEST_V} V, capacitance {measurements.capacitance_nf} nF")
```

### Flash IV

One pulse per string, the load sweeping from short circuit to open circuit during the 2 ms flash, the reference cell reading the same pulse. Each curve is corrected to 25 °C and 1 SC AM0: current divided by the intensity and shifted by the Isc temperature coefficient, voltage shifted by 18 cells' worth of the Voc temperature coefficient. The four corrected curves land in one measurement; Pmax and fill factor per string carry the limits, the panel Pmax and the string match are scalars:

```python filename="phases/flash_iv.py"
import numpy as np

from utils.correction import correct_to_stc, iv_metrics
from utils.recipe import STRINGS

POINTS = 200


def flash_iv(measurements, bench, log):
    """One pulse per string, the load sweeping short circuit to open
    circuit during the 2 ms flash, every curve corrected to 25 C and 1 SC
    AM0 from the reference cell's reading of the same pulse and the panel
    thermocouple. Four corrected curves in one measurement, Pmax and fill
    factor per string as aggregations, the string match as a scalar."""
    pmax = []
    for s in range(1, STRINGS + 1):
        cap = bench.flash_iv(s, POINTS)
        v, i, intensity = correct_to_stc(cap["voltage_v"], cap["current_a"], cap["ref_isc_a"], cap["panel_temp_c"])
        m = iv_metrics(v, i)
        pmax.append(m["pmax_w"])
        if s == 1:
            measurements.iv.x_axis = v.round(3).tolist()
        setattr(measurements.iv.y_axis, f"string_{s}", i.round(4).tolist())
        y = getattr(measurements.iv.y_axis, f"string_{s}")
        y.aggregations.pmax_w = m["pmax_w"]
        y.aggregations.isc_a = m["isc_a"]
        y.aggregations.voc_v = m["voc_v"]
        y.aggregations.fill_factor = m["fill_factor"]
        log.info(f"String {s}: pulse {cap['pulse']} at {intensity:.4f} SC, {cap['panel_temp_c']:.1f} C -> Isc {m['isc_a']:.4f} A, Voc {m['voc_v']:.2f} V, Pmax {m['pmax_w']:.2f} W, FF {m['fill_factor']:.3f}")

    pmax = np.array(pmax)
    match = float(100.0 * (pmax.max() - pmax.min()) / pmax.mean())
    measurements.panel_pmax_w = float(pmax.sum())
    measurements.string_match_pct = match
    log.info(f"Panel Pmax {pmax.sum():.1f} W corrected, string match {match:.2f} %")
```

The correction itself:

```python filename="utils/correction.py"
import numpy as np

from utils.recipe import (CELLS_PER_STRING, D_ISC_A_PER_K, D_VOC_V_PER_K, REFERENCE_CELL_ISC_A, STC_TEMP_C)


def correct_to_stc(voltage_v, current_a, ref_isc_a, panel_temp_c):
    intensity = ref_isc_a / REFERENCE_CELL_ISC_A
    dt = STC_TEMP_C - panel_temp_c
    v = np.array(voltage_v) + CELLS_PER_STRING * D_VOC_V_PER_K * dt
    i = np.array(current_a) / intensity + D_ISC_A_PER_K * dt
    return v, i, intensity


def iv_metrics(v, i):
    p = v * i
    k = int(np.argmax(p))
    isc = float(i[0])
    voc = float(np.interp(0.0, i[::-1], v[::-1]))
    pmax = float(p[k])
    return {"isc_a": isc, "voc_v": voc, "pmax_w": pmax, "vmp_v": float(v[k]), "imp_a": float(i[k]),
            "fill_factor": pmax / (isc * voc)}
```

![The knee of string 1's IV curve between 44 and 49.5 V: the as-flashed curve dashed in gray at 0.9989 SC and 23.4 °C, the corrected curve in green shifted left by the temperature term and scaled by the intensity.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-cb57be85-cc0f-4710-b321-b4b82ddd3e37-VbkvkanawDRBncvtqM2T6WgvI2Ayvv.png)

What the correction does on the mock's string 1: the panel was 1.6 K below 25 °C and the lamp 0.1 % under one solar constant, so the corrected curve moves 0.17 V left and 0.1 % up. Small on a healthy pulse at room temperature, and the reason a panel measured at 30 °C under a lamp at 0.96 SC reports the same power as one measured at 20 °C under 1.03 SC.

### Bypass Diodes

Each bypass diode is made to carry the string current by reverse-biasing its cell group, and the drop across it is recorded. All 24 land in one measurement with the maximum and minimum limited: an open diode reads the reverse breakdown of the cells it was meant to protect, well above 1.2 V; a shorted one reads near zero:

```python filename="phases/bypass_diodes.py"
import numpy as np

from utils.recipe import CELL_ISC_A, DIODES_PER_STRING, STRINGS


def bypass_diodes(measurements, bench, log):
    """Every bypass diode carries the string current once, its forward
    drop recorded. An open diode reads the reverse breakdown of the
    shaded cells it was meant to protect; a shorted one reads zero."""
    vf = []
    for s in range(1, STRINGS + 1):
        for d in range(1, DIODES_PER_STRING + 1):
            vf.append(bench.bypass_diode_vf(s, d, CELL_ISC_A))
    vf = np.array(vf)
    measurements.diodes.x_axis = list(range(1, len(vf) + 1))
    measurements.diodes.y_axis.vf = vf.tolist()
    measurements.diodes.y_axis.vf.aggregations.max_v = float(vf.max())
    measurements.diodes.y_axis.vf.aggregations.min_v = float(vf.min())
    log.info(f"{len(vf)} bypass diodes: Vf {vf.min():.3f}..{vf.max():.3f} V at {CELL_ISC_A:.2f} A")
```

![Twenty-four bars of bypass diode forward drop between 0.74 and 0.86 V, grouped six per string by dotted lines, the 1.2 V and 0.5 V limits as dashed lines.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-14c655d5-646e-4b87-9128-7b2e1783410d-TKYcbWPZYfmPruDiWDawYtWbR5RVNP.png)

The mock panel's 24 diodes, all conducting between 0.74 and 0.86 V at 0.52 A.

### Release

The teardown phase shorts the panel and logs the lamp pulses used, on the run and on the unit:

```python filename="phases/release.py"
def release(measurements, bench, unit, log):
    """Teardown: panel shorted, lamp pulse count logged against the flash
    tube's rated life."""
    bench.short_panel()
    measurements.pulses_this_run = bench.pulses_used()
    unit.metadata["flash_pulses"] = bench.pulses_used()
    log.info(f"Panel shorted, {bench.pulses_used()} pulses this run")
```

### Mock Plug

`FlashBench` stands in for the flasher, the reference cell, the IV load, the megohmmeter, the LCR meter and the thermocouples. It synthesizes four strings of 18 triple-junction cells (0.52 A, 2.70 V, 1.2 W each at 25 °C) with per-string gains within 0.5 % and string 3 at 98.5 %, a lamp delivering 1 SC ± 0.4 % per pulse, a panel at 23.4 °C, insulation at 2 GΩ, 118 nF of capacitance and 24 diodes at 0.82 V ± 0.03. The IV shape is a single-diode model with a fill factor near 0.87. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, the class speaks to the flasher's controller for the pulse and to the IV load through its trigger, reads the reference cell's short-circuit current on every pulse, and drives the megohmmeter, the LCR meter and the thermocouple DAQ over SCPI. Use the reference cell's calibrated value from its certificate, keep the correction coefficients from the cell datasheet in the recipe, and record the reference cell's identity on every run. The phases, measurements and limits stay the same.


### Harness Continuity and Hipot Test
URL: https://www.tofupilot.com/templates/harness-continuity-and-hipot-test
Spacecraft harness acceptance with TofuPilot Framework: 4-wire continuity, 500 V insulation, DWV leakage, shield bonding and a per-connector mating log.

## Introduction

### Harness Acceptance Overview

A spacecraft harness is the one subsystem built entirely by hand, and every crimp, every pin insertion and every shield termination in it is a place where the drawing and the hardware can disagree. The acceptance test on a harness tester asks four questions of every net: is it connected to the pins the netlist says and nothing else, is its resistance what the wire gauge and the length predict, is it insulated from everything else at the test voltage, and does it hold that voltage without breaking down. A fifth question goes to each overall shield: is it bonded to its backshell. And a sixth goes to the record: how many times has each flight connector been mated, because this test costs one more.

![Close-up of one circular connector with its backshell and braided shield mated to an adapter block on the tester](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-aa375034-97dd-4d6a-bdcf-ce43fb294858-27ooA0jyPtR9DByAlO4iz4tks77ayO.jpg)

One connector mated to its adapter block on the tester. The adapter is a flight-representative receptacle wired to the tester's switch matrix; its own resistance is in the continuity limit.

The tests are written down. ECSS-Q-ST-20-30C, the electrical harness assembly standard, and IPC/WHMA-A-620 with its space addendum define continuity against the netlist, insulation resistance, dielectric withstanding voltage and shield termination as the acceptance inspection of a finished harness. The voltages, the resistance floor, the leakage ceiling and the number of mating cycles a flight connector is allowed are the programme's, taken from its harness specification and the connector's qualification, and this template holds them as recipe constants. The one number the recipe computes rather than states is the continuity limit: the wire's ohms per metre times the length plus two contacts plus the adapters, so that a harness of a different length gets the limit its wire deserves rather than a round figure.

### Test Purpose

The procedure records one acceptance record per harness:

- **Netlist revision against the drawing revision**, and each connector's **mating log** against the allowance, before the harness is mated to the tester
- **Continuity**: every net four-wire, opens and miswires counted, and the spread between nets of the same gauge and length
- **Insulation**: every net against every other net and the shields at 500 V DC, the worst pair per net
- **Dielectric withstanding**: 500 V DC for 60 s per net, leakage recorded, breakdowns counted
- **Shield bonding** to the backshell
- **The mate written back** to each connector's log in teardown

![Three panels of bars against net number: continuity resistance around 180 mΩ under a 231 mΩ red dashed limit, insulation around 10 GΩ on a log scale above a 100 MΩ floor, and DWV leakage under 6 µA against a 100 µA limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-619a143a-23a3-4fc4-8918-355d8c8b01ba-mLhxU2UfbeEOG2CgPCr5Ijgu1plTj2.png)

The mock harness on the tester: sixteen nets between 176 and 190 mΩ with a 14 mΩ spread, insulation no worse than 7.9 GΩ, leakage no higher than 5.5 µA, no opens, no miswires, no breakdowns, shields at 3.5 to 4.9 mΩ. A net at 260 mΩ is a crimp that did not take; a net at 30 mΩ over its neighbours is one that half took, and the spread limit is what catches it inside the absolute limit.

The framework mechanics on show are a limit derived in the recipe from physical constants, counts validated with `== 0` beside a resistance ceiling, a spread aggregation that catches a fault inside the absolute limit, a consumable per connector read in `setup:`, gated, and written back in `teardown:`, a boolean derived from two revisions and validated with `==`, and one curve per electrical test each with one judged aggregation.

## Equipment & Setup

To accept harnesses at the integration bench, the following are required:

- **A harness tester** with a switch matrix, four-wire continuity, insulation and DWV at the test voltage, and an API
- **Adapter blocks** for the harness's connectors, flight-representative receptacles wired to the matrix
- **The netlist** for the drawing revision, loaded on the tester
- **The Device Under Test (DUT)**: a harness segment, here two circular connectors, 16 nets of AWG 26 over 1.2 m, two overall shields
- **A TofuPilot Framework procedure** to sequence the gate, the four tests and the log
- **The TofuPilot Dashboard** to keep the mating log per serial as unit metadata and to see every net's resistance across the lot

### Hardware Components

#### Harness Tester

A Cirris CH2 or a CableEye HVX does all four electrical tests from one connection: four-wire continuity per net, insulation at up to 1500 V DC between every pin pair, DWV with leakage readback and breakdown detection, and low-resistance readings for the shields. The tester holds the netlist and reports per-net results, and the procedure asks for each test in turn and validates what comes back. The four-wire continuity matters at these resistances: a two-wire reading would add the matrix and the adapter cabling, which is more than the harness.

![Wire harness connected to continuity and electrical safety test fixtures](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/harness-continuity-and-hipot-test-4aTmzWPGFBLKhX8eSfaFMhBopYkUMg.webp)

The harness is mated to two supported connector adapters and secured along its length. The external tester performs the sequence without remating the harness.

#### Adapters and the Mating Log

Flight connectors are qualified for a finite number of mating cycles, and each programme allocates a share of them to ground testing. The adapter blocks are what the harness mates to on the bench, and every mate goes on the log. The procedure reads the log for both connectors in setup, refuses to mate a harness whose worst connector has no allowance left, and writes the mate back in teardown whatever the outcome, so a harness that failed continuity still has its log right.

#### The Continuity Limit

AWG 26 copper is 0.1339 Ω/m at 20 °C. Over 1.2 m that is 161 mΩ, plus two contacts at 5 mΩ each, plus a margin for the adapters and the matrix, gives 231 mΩ. The recipe computes it from those constants so a 3 m harness of the same design gets 470 mΩ without anyone editing a limit. The spread limit of 40 mΩ between the best and worst net is the number that finds a marginal crimp on a net that still passes the absolute limit.

### Where the Limits Come From

The structure of the test is ECSS-Q-ST-20-30C and IPC/WHMA-A-620. The 500 V DC insulation voltage and the 100 MΩ floor, the 500 V DC DWV for 60 s with 100 µA of leakage, the 10 mΩ shield bond and the 50-mate allowance are the programme's harness specification. The continuity limit is computed from the wire. A customer sets the voltages from the harness's working voltage and the connector's rating, and tightens the spread limit from the distribution the dashboard shows after a few dozen harnesses.

## Test Procedure

### Overview

The procedure maps the acceptance test onto the framework's three stages. The revision check and the mating gate live in `setup:` so the harness is never mated on a wrong netlist or past its allowance. The four electrical tests are chained with `depends_on` because they share one tester and one mate. The log write lives in `teardown:` so it happens whatever the outcome.

1. Setup: tester, netlist revision against the drawing, mates remaining.
2. Main: continuity, opens, miswires, spread.
3. Main: insulation at 500 V DC.
4. Main: DWV at 500 V DC for 60 s, leakage, breakdowns.
5. Main: shield bonding.
6. Teardown: mate written to each connector's log.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-serial trending across the lot.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  continuity.py
  insulation.py
  dwv.py
  shield_bond.py
  demate.py
plugs/
  harness_tester.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-harness-continuity-hipot). The `HarnessTester` plug is a mock of the tester and its adapters, synthesizing a healthy 16-net harness with nets around 180 mΩ, insulation over 5 GΩ, a few microamps of leakage, shields bonded under 5 mΩ and a mating log at 12 and 7 mates, so the procedure runs end-to-end without a tester or a harness connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the tester plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Harness Continuity and Hipot Test
version: 0.1.0
description: Acceptance of a spacecraft harness segment on a harness tester. Netlist revision and the connectors' mating log in setup, 4-wire continuity of every net against the gauge-and-length limit with opens and miswires counted, insulation at 500 V DC pin-pair, dielectric withstanding at 500 V DC with leakage and breakdowns, shield-to-backshell bonding, and the mate written back to the log in teardown.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the harness label"
    placeholder: "HRN-PWR-0000"
    pattern: "^HRN-PWR-\\d{4}$"
    default_value: "HRN-PWR-0142"
  part_number:
    default_value: "HRN-PWR-16N"
  batch_number:
    default_value: "LOT-2026-09"

plugs:
  - name: Harness Tester
    description: "Cirris or CableEye-class harness tester with adapters for both connectors, netlist loaded, mating log from the record (mock)"
    python: plugs.harness_tester:HarnessTester
    key: bench

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Tester
        key: tester_id
        validators:
          - {operator: matches, expected_value: "^CH2-\\d{2}$"}
      - name: Netlist Matches Drawing
        key: netlist_matches_drawing
        validators:
          - {operator: "==", expected_value: true}
      - name: Mates Remaining
        key: mates_remaining_min
        description: Mating cycles left on the worst connector before this test; the test costs one.
        validators:
          - {operator: ">=", expected_value: 1}

main:
  - name: Continuity
    key: continuity
    python: phases.continuity
    measurements:
      - name: Nets
        key: nets
        title: 4-wire resistance of every net
        x_axis:
          legend: Net
        y_axis:
          - legend: Resistance
            key: resistance
            unit: mΩ
            aggregations:
              - type: max_mohm
                unit: mΩ
                validators:
                  - {operator: "<=", expected_value: 230.7}
              - type: spread_mohm
                unit: mΩ
                validators:
                  - {operator: "<=", expected_value: 40.0}
      - name: Opens
        key: opens
        validators:
          - {operator: "==", expected_value: 0}
      - name: Miswires
        key: miswires
        validators:
          - {operator: "==", expected_value: 0}

  - name: Insulation
    key: insulation
    python: phases.insulation
    depends_on: [continuity]
    measurements:
      - name: IR
        key: ir
        title: Worst insulation resistance per net at 500 V DC
        x_axis:
          legend: Net
        y_axis:
          - legend: Worst pair
            key: worst
            unit: MΩ
            aggregations:
              - type: min_mohm
                unit: MΩ
                validators:
                  - {operator: ">=", expected_value: 100.0}

  - name: DWV
    key: dwv
    python: phases.dwv
    depends_on: [insulation]
    timeout: 30m
    measurements:
      - name: DWV
        key: dwv
        title: Leakage per net at 500 V DC for 60 s
        x_axis:
          legend: Net
        y_axis:
          - legend: Leakage
            key: leakage
            unit: µA
            aggregations:
              - type: max_ua
                unit: µA
                validators:
                  - {operator: "<=", expected_value: 100.0}
      - name: Breakdowns
        key: breakdowns
        validators:
          - {operator: "==", expected_value: 0}

  - name: Shield Bond
    key: shield_bond
    python: phases.shield_bond
    depends_on: [dwv]
    measurements:
      - name: Shields
        key: shields
        title: Shield to backshell resistance
        x_axis:
          legend: Shield
        y_axis:
          - legend: Resistance
            key: resistance
            unit: mΩ
            aggregations:
              - type: max_mohm
                unit: mΩ
                validators:
                  - {operator: "<=", expected_value: 10.0}

teardown:
  - name: Demate
    key: demate
    python: phases.demate
    measurements:
      - name: Mates Remaining After
        key: mates_remaining_after
        validators:
          - {operator: ">=", expected_value: 0}
```

Framework features to notice:

1. **A computed limit.** The 230.7 mΩ in the procedure file is the value `CONTINUITY_MOHM_MAX` takes in `utils/recipe.py` from the gauge, the length, the contacts and the adapters; the recipe is where it lives, the procedure file is where it is declared.
2. **Counts with `== 0`.** `opens`, `miswires` and `breakdowns` are integers the tester reports; the validator is equality with zero. A count is not a resistance and is not judged as one.
3. **A spread beside a ceiling.** `nets.resistance.max_mohm` bounds the worst net; `spread_mohm` bounds the difference between the best and the worst, which is where a half-taken crimp shows.
4. **A consumable through the run.** `mates_remaining_min >= 1` in setup gates the mate; `mates_remaining_after >= 0` in teardown records it; both connectors' counts sit on the unit metadata before and after.
5. **A boolean from two revisions.** `netlist_matches_drawing` is computed in the phase from the tester's loaded netlist and the drawing on the traveller, and validated with `== true`.

### Identify

The setup phase reads the tester's identity, compares the netlist revision to the drawing, reads the mating log for both connectors against the allowance, and puts the revision and the counts on the **unit metadata**:

```python filename="phases/identify.py"
from utils.recipe import CONNECTORS, MATES_ALLOWED


def identify(measurements, bench, unit, log):
    """Setup: tester and netlist revision against the drawing, and the
    mating log of each flight connector against the programme's
    allowance. This test costs one mate per connector; a harness that
    has none left is not mated again."""
    ident = bench.identify()
    measurements.tester_id = ident["tester"]
    measurements.netlist_matches_drawing = ident["netlist_rev"] == ident["drawing_rev"]
    remaining = [MATES_ALLOWED - ident["mates"][c] for c in CONNECTORS]
    measurements.mates_remaining_min = int(min(remaining))
    unit.metadata["netlist_rev"] = ident["netlist_rev"]
    for c in CONNECTORS:
        unit.metadata[f"mates_{c}"] = ident["mates"][c]
    log.info(f"Harness {unit.serial_number}: netlist rev {ident['netlist_rev']} vs drawing {ident['drawing_rev']}, mates " + ", ".join(f"{c} {ident['mates'][c]}/{MATES_ALLOWED}" for c in CONNECTORS))
```

### Continuity

Every net four-wire against the netlist; the resistances go on one curve with the maximum and the spread as aggregations, and the opens and miswires the tester reports go on two counts:

```python filename="phases/continuity.py"
import numpy as np

from utils.recipe import CONTINUITY_MOHM_MAX


def continuity(measurements, bench, log):
    """Every net 4-wire against the netlist. The resistance limit is the
    gauge and the length; an open or a miswire is a count that must be
    zero, not a resistance that is large."""
    c = bench.continuity()
    r = np.array(c["resistance_mohm"])
    measurements.nets.x_axis = c["net"]
    measurements.nets.y_axis.resistance = c["resistance_mohm"]
    measurements.nets.y_axis.resistance.aggregations.max_mohm = float(r.max())
    measurements.nets.y_axis.resistance.aggregations.spread_mohm = float(r.max() - r.min())
    measurements.opens = int(c["opens"])
    measurements.miswires = int(c["miswires"])
    log.info(f"{r.size} nets: {r.min():.1f}..{r.max():.1f} mohm (limit {CONTINUITY_MOHM_MAX:.0f}), {c['opens']} open, {c['miswires']} miswired")
```

### Insulation

Every net against every other net and the shields at 500 V DC; the tester reports the worst pair per net and the phase validates the minimum:

```python filename="phases/insulation.py"
import numpy as np

from utils.recipe import IR_V


def insulation(measurements, bench, log):
    """Insulation resistance of every net against every other net and
    the shields at 500 V DC; the tester reports the worst pair per net."""
    ir = bench.insulation(IR_V)
    worst = np.array(ir["worst_mohm"])
    measurements.ir.x_axis = ir["net"]
    measurements.ir.y_axis.worst = ir["worst_mohm"]
    measurements.ir.y_axis.worst.aggregations.min_mohm = float(worst.min())
    log.info(f"Insulation at {IR_V:.0f} V: worst pair {worst.min():.0f} Mohm")
```

### DWV

500 V DC for 60 s per net; the leakage goes on a curve with its maximum, and any breakdown the tester detected goes on a count:

```python filename="phases/dwv.py"
import numpy as np

from utils.recipe import DWV_S, DWV_V


def dwv(measurements, bench, log):
    """Dielectric withstanding voltage, 500 V DC for 60 s per net,
    leakage recorded and any breakdown counted."""
    d = bench.dwv(DWV_V, DWV_S)
    leak = np.array(d["leakage_ua"])
    measurements.dwv.x_axis = d["net"]
    measurements.dwv.y_axis.leakage = d["leakage_ua"]
    measurements.dwv.y_axis.leakage.aggregations.max_ua = float(leak.max())
    measurements.breakdowns = int(d["breakdowns"])
    log.info(f"DWV {DWV_V:.0f} V DC {DWV_S:.0f} s: leakage up to {leak.max():.1f} uA, {d['breakdowns']} breakdown")
```

### Shield Bond

Each overall shield to its backshell, low-resistance:

```python filename="phases/shield_bond.py"
import numpy as np


def shield_bond(measurements, bench, log):
    """Each overall shield to its backshell, low-resistance; a shield
    that is not bonded is an antenna."""
    s = bench.shield_bond()
    r = np.array(s["resistance_mohm"])
    measurements.shields.x_axis = s["shield"]
    measurements.shields.y_axis.resistance = s["resistance_mohm"]
    measurements.shields.y_axis.resistance.aggregations.max_mohm = float(r.max())
    log.info(f"Shield bonds {r.min():.1f}..{r.max():.1f} mohm")
```

### Demate

The teardown writes the mate this test cost to each connector's log and records the allowance left:

```python filename="phases/demate.py"
from utils.recipe import CONNECTORS, MATES_ALLOWED


def demate(measurements, bench, unit, log):
    """Teardown: the mate this test cost is written to the log of each
    connector, whatever the outcome."""
    mates = bench.record_mate()
    for c in CONNECTORS:
        unit.metadata[f"mates_{c}"] = mates[c]
    measurements.mates_remaining_after = int(min(MATES_ALLOWED - mates[c] for c in CONNECTORS))
    log.info("Demated, mates now " + ", ".join(f"{c} {mates[c]}/{MATES_ALLOWED}" for c in CONNECTORS))
```

### Mock Plug

`HarnessTester` stands in for the tester, its adapters and the mating log. It draws sixteen nets at the wire's resistance plus contacts, adapters and a few milliohms of scatter, insulation between 5 and 15 GΩ, leakage of 3 to 6 µA, two shields at 3 to 5 mΩ, no opens, miswires or breakdowns, and a mating log at 12 and 7 that the teardown increments. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real station, the class speaks the tester's API over USB or Ethernet: load the netlist for the drawing revision, run each test, read the per-net results back. Keep the mating log per connector in the dashboard as unit metadata and let every mate, this test's included, increment it. Set `IR_V`, `DWV_V`, `DWV_LEAK_UA_MAX` and `MATES_ALLOWED` to the programme's harness specification, and `WIRE_GAUGE_AWG`, `OHM_PER_M` and `LENGTH_M` to the harness's drawing. The phases, measurements and limits stay the same.

### TVAC Bakeout and Thermal Cycling
URL: https://www.tofupilot.com/templates/tvac-bakeout-and-thermal-cycling
Star tracker TVAC with TofuPilot Framework: pump-down, bakeout judged on the TQCM outgassing rate, four thermal vacuum cycles with a centroid check per dwell.

## Introduction

### Thermal Vacuum Overview

A star tracker is a camera whose lens must stay clean and whose focus must hold from -25 to +50 °C in vacuum. Two things on the ground threaten that. Everything inside the unit and around it outgasses when the air is gone, and what comes off the harness, the potting and the adhesives condenses on the coldest optical surface in reach. And every solder joint, every bond and every threaded insert sees its full thermal range for the first time under vacuum, where there is no convection to soften the gradients. The thermal vacuum run answers both: a bakeout with a witness that measures what the unit gives off, and acceptance cycles with a functional check at each extreme.

![Close-up of a star tracker unit with its lens baffle on the chamber plate, the TQCM crystal head on a post facing the lens](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-107c2118-ff41-4203-8457-a0e93e6063ff-G9oVEnQ1crPNywMyPRsDBTpBiuVSUo.jpg)

The unit on the chamber plate with the TQCM's crystal head on its post, facing the lens. The crystal is held cold so that whatever condenses on the optics also condenses on it, and its resonant frequency drops in proportion to the mass it collects.

The cycling is written down. ECSS-E-ST-10-03C Rev.1 asks for a pressure at or below 1e-5 hPa, four acceptance thermal vacuum cycles on a unit (eight for qualification), and a functional test at the hot and cold dwells (see [ECSS-E-ST-10-03C Rev.1, Testing](https://ecss.nl/standard/ecss-e-st-10-03c-rev-1-testing-31-may-2022/)). The bakeout is not: no standard gives its duration, because it depends on what the unit is made of. It ends when the outgassing rate the witness sees falls under the programme's figure and stays there. A thermoelectric quartz crystal microbalance (TQCM) at its collection temperature reads that rate as a frequency slope in hertz per hour; on this programme the crystal at -20 °C must read under 1 Hz/h for the last six hours of a bake of at least 36 h at +60 °C, and the total frequency shift over the bake, the mass collected, has its own ceiling.

### Test Purpose

The procedure records one thermal vacuum record per unit:

- **Pump-down** to the ECSS pressure, with the time it takes as a health figure for the chamber and for the unit's outgassing load
- **Bakeout** at +60 °C with the TQCM at -20 °C facing the unit: the frequency curve, the rate derived from it, the clean window at the end and the total mass collected
- **Four acceptance thermal vacuum cycles** with the tracker powered, its temperature logged, and at every hot and cold dwell the centroid error on a simulated star projected through the chamber window and the bus power
- **Backfill** with dry nitrogen, gated on the unit being above the room's dew point so the lens does not fog

![Two panels: on the left the chamber pressure falling on a log scale from 1000 hPa to 5e-6 hPa over five hours with a red dashed line at 1e-5 hPa; on the right the TQCM frequency shift rising in green to 320 Hz over a 44 hour bake with the outgassing rate as an orange dotted line falling below the 1 Hz/h red dashed line and the last nine hours shaded as the clean window.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-4a1f2e7c-9018-4856-bd75-be3793b07e6a-6BNomkXOM1dOrD3n7ifftSFfQnsX7U.png)

The mock unit's pump-down and bake: under 1e-5 hPa in 4.6 h, then 44 h at +60 °C during which the crystal collected 320 Hz while the rate decayed from 40 Hz/h to 0.2 Hz/h, clean for the last 9 h against the 6 h window. A unit with uncured potting or a contaminated harness keeps the rate above 1 Hz/h for days and hits the 72 h ceiling, which is a finding for the materials list, not a chamber problem.

The framework mechanics on show are a derived curve stored beside its raw one with the limits on the derivative, a duration window as two validators on one aggregation, a count validated with `==` beside a level validated as a window, a curve kept for the record with no aggregation next to two that are judged, very small numbers as limits, progress components on three time-scaled phases, and a teardown gate that protects the hardware.

## Equipment & Setup

To run thermal vacuum on optical units, the following are required:

- **A thermal vacuum chamber** with a shroud, a temperature-controlled plate, a turbo pump and an ion gauge, and a window the star simulator can see through
- **A TQCM** on a cold finger, positioned to see the same surfaces the optics see
- **A star simulator** outside the window, a collimated point source the tracker can centroid
- **The Device Under Test (DUT)**: a smallsat star tracker on its bracket, with its power and data harness through a feedthrough
- **A TofuPilot Framework procedure** to sequence the pump-down, the bake, the cycles and the backfill
- **The TofuPilot Dashboard** to keep the bake record per serial and trend the centroid error at the dwells across the lot

### Hardware Components

#### Chamber, Pump and Gauge

A bench chamber with a shroud and a plate covers a unit this size. The chamber controller exposes the shroud and plate set points and the valves over Modbus; the ion gauge reports over RS-232. The pressure requirement is ECSS's 1e-5 hPa; the time to reach it is not a requirement but a diagnostic the procedure records, because a chamber that used to pump down in four hours and now takes seven has a leak or a wet load.

![Star tracker on a platen inside an open thermal-vacuum chamber with separate feedthroughs](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/tvac-bakeout-and-thermal-cycling-physical-2026-524Bn3dtDLXyFt0rIDR4KhIBbhvqaD.webp)

The chamber is shown open for loading. Vacuum and electrical connections use separate feedthroughs; testing takes place with the door sealed.

#### TQCM

The witness is a quartz crystal held at a set temperature by a thermoelectric element, with its resonant frequency read by a controller of the CrystalTek or QCM Research class. Mass condensing on the crystal lowers the frequency; the procedure records the shift as a positive number. Two things matter: the crystal's temperature, which sets what condenses on it (on this programme -20 °C, cold enough to collect what would reach the lens), and its view, which must include the unit and not the chamber walls. The rate criterion and the total-shift ceiling are the programme's contamination control plan, and they are recipe constants here.

#### Star Simulator and Tracker Link

The functional check at each dwell is the tracker's own job: centroid a point source. A collimated source outside the window gives a star at a known angle, and the tracker reports the centroid over its Ethernet link along with its bus power and its temperature. The centroid error grows at the cold end as the lens contracts and the focus moves; the limit is the unit's own accuracy specification with margin.

### Where the Limits Come From

The pressure and the four cycles are ECSS-E-ST-10-03C. The acceptance levels are the operating range, -25 to +50 °C, with a 5 °C margin. The bake temperature, the 36 h minimum and 72 h maximum, the 1 Hz/h rate at -20 °C, the 6 h clean window and the 800 Hz total shift are the programme's contamination control plan for an optical unit. The 10 arcsec centroid ceiling and the 3.5 W power ceiling are the tracker's data sheet. The 15 °C backfill gate is the room's dew point with margin.

## Test Procedure

### Overview

The procedure maps the run onto the framework's three stages. Identification lives in `setup:` with the chamber still at ambient. The three long phases are chained with `depends_on` because they share one chamber. The backfill lives in `teardown:` so the chamber comes back to air safely whether the run passed or not, and only once the unit is warm enough.

1. Setup: firmware, unit temperature, chamber at ambient.
2. Main: pump-down, pressure curve, time to vacuum.
3. Main: bakeout, TQCM curve, rate, clean window, total shift.
4. Main: four thermal vacuum cycles, profile, centroid and power at each dwell.
5. Teardown: backfill with dry nitrogen once the unit is above the dew point.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-serial trending across the lot.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  pump_down.py
  bakeout.py
  tvac_cycling.py
  backfill.py
plugs/
  tvac_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-tvac-bakeout-cycling). The `TvacBench` plug is a mock of the chamber, the gauge, the TQCM and the tracker's link together, synthesizing a healthy unit that pumps down to 5e-6 hPa in five hours, outgasses at a rate that decays from 40 Hz/h with an 8 h time constant, and centroids within 6 arcsec at the cold dwells, so the procedure runs end-to-end without a chamber or a unit connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: TVAC Bakeout and Thermal Cycling
version: 0.1.0
description: Thermal vacuum run for a smallsat star tracker. Pump-down to the ECSS pressure, bakeout at +60 C with a TQCM witness judged on the outgassing rate and the mass collected, four acceptance thermal vacuum cycles with a centroid check on a simulated star at every dwell, and a dry-nitrogen backfill gated on the unit being above the dew point.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the tracker's label"
    placeholder: "ST-200-0000"
    pattern: "^ST-200-\\d{4}$"
    default_value: "ST-200-0088"
  part_number:
    default_value: "ST-200-FM"
  batch_number:
    default_value: "LOT-2026-09"

plugs:
  - name: TVAC Bench
    description: "Chamber shroud and plate, turbo pump and ion gauge, TQCM on a cold finger, star tracker telemetry and image link (mock, one plug per chamber)"
    python: plugs.tvac_bench:TvacBench
    key: bench

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Firmware Version
        key: firmware_version
        validators:
          - {operator: matches, expected_value: "^3\\.\\d+\\.\\d+$"}
      - name: Unit Temperature Start
        key: unit_temp_start_c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 18.0}
          - {operator: "<=", expected_value: 26.0}
      - name: Pressure Start
        key: pressure_start_hpa
        unit: hPa
        validators:
          - {operator: ">=", expected_value: 900.0}

main:
  - name: Pump Down
    key: pump_down
    python: phases.pump_down
    timeout: 12h
    ui:
      components:
        - key: pump_progress
          type: progress
          label: "Pump-down"
          description: "To 1e-5 hPa"
          default_value: 0
          max: 100
    measurements:
      - name: Pump-Down
        key: pumpdown
        title: Chamber pressure during pump-down
        x_axis:
          legend: Time
          unit: h
        y_axis:
          - legend: Pressure
            key: pressure
            unit: hPa
            aggregations:
              - type: final_hpa
                unit: hPa
                validators:
                  - {operator: "<=", expected_value: 1.0e-5}
              - type: time_to_vacuum_h
                unit: h
                validators:
                  - {operator: "<=", expected_value: 8.0}

  - name: Bakeout
    key: bakeout
    python: phases.bakeout
    depends_on: [pump_down]
    timeout: 24h
    ui:
      components:
        - key: bake_progress
          type: progress
          label: "Bakeout"
          description: "+60 °C, TQCM at -20 °C, until the rate is clean"
          default_value: 0
          max: 100
    measurements:
      - name: Bake
        key: bake
        title: TQCM frequency, outgassing rate and unit temperature through the bake
        x_axis:
          legend: Time
          unit: h
        y_axis:
          - legend: TQCM frequency shift
            key: tqcm
            unit: Hz
            aggregations:
              - type: duration_h
                unit: h
                validators:
                  - {operator: ">=", expected_value: 36.0}
                  - {operator: "<=", expected_value: 72.0}
              - type: total_shift_hz
                unit: Hz
                validators:
                  - {operator: "<=", expected_value: 800.0}
          - legend: Outgassing rate
            key: rate
            unit: Hz/h
            aggregations:
              - type: final_hz_per_h
                unit: Hz/h
                validators:
                  - {operator: "<=", expected_value: 1.0}
              - type: clean_window_h
                unit: h
                validators:
                  - {operator: ">=", expected_value: 6.0}
          - legend: Unit temperature
            key: unit
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 65.0}

  - name: TVAC Cycling
    key: tvac_cycling
    python: phases.tvac_cycling
    depends_on: [bakeout]
    timeout: 24h
    ui:
      components:
        - key: cycle_progress
          type: progress
          label: "Thermal vacuum cycling"
          description: "4 cycles, -30 to +55 °C, 2 h dwells, powered"
          default_value: 0
          max: 100
    measurements:
      - name: Profile
        key: profile
        title: Unit temperature through the four cycles
        x_axis:
          legend: Time
          unit: h
        y_axis:
          - legend: Unit
            key: unit
            unit: °C
            aggregations:
              - type: cycles
                validators:
                  - {operator: "==", expected_value: 4}
              - type: min_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: -29.0}
              - type: max_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: 54.0}
                  - {operator: "<=", expected_value: 58.0}
      - name: Dwells
        key: dwells
        title: Functional check at each hot and cold dwell
        x_axis:
          legend: Dwell
        y_axis:
          - legend: Temperature
            key: temperature
            unit: °C
          - legend: Centroid error
            key: centroid
            unit: arcsec
            aggregations:
              - type: max_arcsec
                unit: arcsec
                validators:
                  - {operator: "<=", expected_value: 10.0}
          - legend: Power
            key: power
            unit: W
            aggregations:
              - type: max_w
                unit: W
                validators:
                  - {operator: "<=", expected_value: 3.5}

teardown:
  - name: Backfill
    key: backfill
    python: phases.backfill
    measurements:
      - name: Unit Temperature At Backfill
        key: unit_temp_backfill_c
        unit: °C
        description: Dry nitrogen only comes in with the unit above the room's dew point; a cold lens fogs.
        validators:
          - {operator: ">=", expected_value: 15.0}
      - name: Pressure End
        key: pressure_end_hpa
        unit: hPa
        validators:
          - {operator: ">=", expected_value: 900.0}
```

Framework features to notice:

1. **A derived curve beside the raw one.** `bake` records the TQCM frequency as read and the outgassing rate the phase derives from it over a 4 h window; the limits sit on the rate's `final_hz_per_h` and `clean_window_h`, and on the raw curve's `total_shift_hz`. The dashboard shows both.
2. **A duration window.** `bake.tqcm.duration_h` carries a `>= 36` and a `<= 72`: too short and the bake did not run long enough to mean anything, too long and the unit is outgassing more than its materials list allows.
3. **A count and a level.** `profile.unit.cycles == 4` is the ECSS requirement as a number; `max_c` between 54 and 58 says the hot level was reached and not overshot.
4. **A curve for the record.** `dwells.temperature` has no aggregation; it is there so the centroid and the power at each dwell can be read against the temperature they were taken at.
5. **Very small numbers.** `final_hpa <= 1.0e-5` is a limit like any other; YAML reads the exponent.
6. **A teardown that gates the hardware.** `unit_temp_backfill_c >= 15` is not about the unit's quality; it fails the run if the chamber was backfilled onto a cold lens, which is the one mistake the run cannot undo.
7. **Phase timeouts cap at 24 h.** The bake and the cycles are time-scaled in the mock; on real hardware each runs as daily segments of its phase or is read back from the chamber's logger.

### Identify

The setup phase reads the tracker's firmware and temperature over its link and the chamber's pressure at ambient, and records the chamber on the **unit metadata**:

```python filename="phases/identify.py"
def identify(measurements, bench, unit, log):
    """Setup: tracker telemetry, firmware, chamber at ambient with the
    door closed, TQCM zeroed on the record."""
    ident = bench.identify()
    measurements.firmware_version = ident["firmware"]
    measurements.unit_temp_start_c = ident["unit_temp_c"]
    measurements.pressure_start_hpa = ident["pressure_hpa"]
    unit.metadata["chamber_id"] = "TVAC-1"
    log.info(f"Star tracker {unit.serial_number}: fw {ident['firmware']}, {ident['unit_temp_c']} C, chamber at {ident['pressure_hpa']:.0f} hPa")
```

### Pump Down

One call returns the pressure curve; the phase records it and finds the time the pressure first went under the requirement:

```python filename="phases/pump_down.py"
import numpy as np

from utils.recipe import P_VACUUM_HPA


def pump_down(measurements, bench, ui, log):
    """Pump-down to the ECSS pressure requirement; the time it takes is a
    health figure for the chamber and for the unit's outgassing load."""
    cap = bench.pump_down()
    ui.pump_progress = 100
    t = np.array(cap["time_h"]); p = np.array(cap["pressure_hpa"])
    below = np.flatnonzero(p <= P_VACUUM_HPA)
    t_vac = float(t[below[0]]) if below.size else float(t[-1])
    measurements.pumpdown.x_axis = cap["time_h"]
    measurements.pumpdown.y_axis.pressure = cap["pressure_hpa"]
    measurements.pumpdown.y_axis.pressure.aggregations.final_hpa = float(p[-1])
    measurements.pumpdown.y_axis.pressure.aggregations.time_to_vacuum_h = t_vac
    log.info(f"Pump-down: {p[-1]:.1e} hPa after {t[-1]:.1f} h, under {P_VACUUM_HPA:.0e} hPa at {t_vac:.2f} h")
```

### Bakeout

The bench holds the plate at +60 °C and the crystal at -20 °C and returns the frequency curve once its own end criterion is met or the maximum is reached. The phase derives the rate over a 4 h window, finds how long the rate has been under the figure at the end, and records the total shift:

```python filename="phases/bakeout.py"
import numpy as np

from utils.recipe import BAKEOUT_MAX_H, BAKEOUT_T_C, TQCM_CLEAN_WINDOW_H, TQCM_RATE_HZ_PER_H_MAX


def bakeout(measurements, bench, ui, log):
    """Bake at +60 C with the TQCM at -20 C facing the unit. The crystal
    frequency is the mass collected; its slope is the outgassing rate.
    The bake is judged on how long the rate has been under the
    programme's figure at the end, and on the total mass collected."""
    cap = bench.bakeout(BAKEOUT_MAX_H)
    ui.bake_progress = 100
    t = np.array(cap["time_h"]); f = np.array(cap["tqcm_hz"])
    # Rate over a 4 h window, Hz/h, from the frequency curve.
    w = 16
    rate = np.full(t.size, np.nan)
    rate[w:] = (f[w:] - f[:-w]) / (t[w:] - t[:-w])
    rate_curve = np.where(np.isnan(rate), rate[w], rate)
    clean = rate_curve <= TQCM_RATE_HZ_PER_H_MAX
    # Clean window at the end: hours since the rate last exceeded the figure.
    dirty = np.flatnonzero(~clean)
    clean_h = float(t[-1] - t[dirty[-1]]) if dirty.size else float(t[-1])

    measurements.bake.x_axis = cap["time_h"]
    measurements.bake.y_axis.tqcm = cap["tqcm_hz"]
    measurements.bake.y_axis.tqcm.aggregations.duration_h = float(t[-1])
    measurements.bake.y_axis.tqcm.aggregations.total_shift_hz = float(f[-1] - f[0])
    measurements.bake.y_axis.rate = rate_curve.round(3).tolist()
    measurements.bake.y_axis.rate.aggregations.final_hz_per_h = float(rate_curve[-1])
    measurements.bake.y_axis.rate.aggregations.clean_window_h = clean_h
    measurements.bake.y_axis.unit = cap["unit_c"]
    measurements.bake.y_axis.unit.aggregations.max_c = float(max(cap["unit_c"]))
    log.info(f"Bake {t[-1]:.1f} h at {BAKEOUT_T_C:.0f} C: TQCM {f[-1] - f[0]:.0f} Hz collected, rate {rate_curve[-1]:.2f} Hz/h at the end, clean for the last {clean_h:.1f} h (needs {TQCM_CLEAN_WINDOW_H:.0f})")
```

### TVAC Cycling

One call returns the four cycles as a temperature curve plus the centroid error and the power read at each dwell. The phase counts the cycles as the smaller of the hot and cold dwells reached, records the profile, and writes the dwell readings as three curves against the dwell index:

```python filename="phases/tvac_cycling.py"
import numpy as np

from utils.recipe import CYCLES, T_ACC_MAX_C, T_ACC_MIN_C


def tvac_cycling(measurements, bench, ui, log):
    """ECSS acceptance thermal vacuum cycling: four cycles between the
    acceptance levels, the tracker powered, a functional check at every
    dwell: centroid of a simulated star through the window, bus power."""
    cap = bench.thermal_cycles()
    ui.cycle_progress = 100
    u = np.array(cap["unit_c"])
    hot = sum(1 for d in cap["dwell_temp_c"] if d >= T_ACC_MAX_C - 1.0)
    cold = sum(1 for d in cap["dwell_temp_c"] if d <= T_ACC_MIN_C + 1.0)
    cycles = int(min(hot, cold))

    measurements.profile.x_axis = [round(m / 60.0, 2) for m in cap["time_min"]]
    measurements.profile.y_axis.unit = cap["unit_c"]
    measurements.profile.y_axis.unit.aggregations.cycles = cycles
    measurements.profile.y_axis.unit.aggregations.min_c = float(u.min())
    measurements.profile.y_axis.unit.aggregations.max_c = float(u.max())

    measurements.dwells.x_axis = list(range(1, len(cap["dwell_temp_c"]) + 1))
    measurements.dwells.y_axis.temperature = cap["dwell_temp_c"]
    measurements.dwells.y_axis.centroid = cap["dwell_centroid_arcsec"]
    measurements.dwells.y_axis.centroid.aggregations.max_arcsec = float(max(cap["dwell_centroid_arcsec"]))
    measurements.dwells.y_axis.power = cap["dwell_power_w"]
    measurements.dwells.y_axis.power.aggregations.max_w = float(max(cap["dwell_power_w"]))
    log.info(f"{cycles}/{CYCLES} cycles, unit {u.min():.1f}..{u.max():.1f} C, centroid error up to {max(cap['dwell_centroid_arcsec']):.1f} arcsec, power up to {max(cap['dwell_power_w']):.2f} W at the dwells")
```

![Two panels: on the left the unit temperature in green cycling four times between -30 and +55 °C over 23 hours with red dashed lines at the two levels and blue dots at the end of each dwell; on the right eight green bars of centroid error in arcseconds, 3.4 at the hot dwells and up to 6.1 at the cold ones, under a red dashed 10 arcsec limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-0b133f7f-d144-47ac-823b-6c6f0b943a51-L8zcqZrioBZDL0SmNqOl8icL5xbJOP.png)

The mock unit's cycles: four between -30 and +55 °C with 2 h dwells, the centroid error 3.4 arcsec at the hot dwells and 5.6 to 6.1 arcsec at the cold ones as the lens contracts, against a 10 arcsec limit, and the bus power under 2.8 W throughout. A tracker whose focus was set at ambient and drifts past the limit at -30 °C is a unit for the optics bench, not the chamber, and this is where it shows.

### Backfill

The teardown lets dry nitrogen in only once the unit is above the room's dew point, and records both the temperature at that moment and the pressure at the end:

```python filename="phases/backfill.py"
def backfill(measurements, bench, log):
    """Teardown: the chamber is backfilled with dry nitrogen only once
    the unit is above the room's dew point; the optics must not fog."""
    r = bench.backfill()
    measurements.unit_temp_backfill_c = r["unit_temp_c"]
    measurements.pressure_end_hpa = r["pressure_hpa"]
    log.info(f"Backfill at {r['unit_temp_c']} C on the unit, chamber at {r['pressure_hpa']:.0f} hPa")
```

### Mock Plug

`TvacBench` stands in for the chamber controller, the ion gauge, the QCM controller and the tracker's link. It draws a pump-down that decays to 5e-6 hPa in five hours, a bake whose outgassing rate starts at 40 Hz/h and decays with an 8 h time constant toward 0.15 Hz/h, with the bench ending the bake eight hours after the rate first went clean past the 36 h minimum, four cycles at 1.5 °C/min with 2 h dwells and a centroid error of 3.2 arcsec plus 0.055 arcsec per degree below 20 °C, and a backfill at ambient. The long phases are time-scaled and return in one call each. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real station, the class speaks Modbus to the chamber controller for the shroud, the plate and the valves, reads the ion gauge over RS-232 and the QCM controller over its serial protocol, and talks to the tracker over Ethernet for power, temperature and the centroid of the simulated star. Set `TIME_SCALE = 1.0` in `utils/recipe.py`, run the bake and the cycles as daily segments or read them back from the chamber's logger because a phase timeout caps at 24 h, and set the TQCM rate, window and total-shift figures to the programme's contamination control plan. The phases, measurements and limits stay the same.

### Battery Cell Screening with Vacuum
URL: https://www.tofupilot.com/templates/battery-cell-screening-with-vacuum
COTS 18650 cell screening with TofuPilot Framework: mass, OCV, ACIR and C/2 capacity before and after six hours under vacuum, NR-SRD-139 delta criteria.

## Introduction

### Cell Screening Overview

A smallsat battery is a few dozen commercial 18650 cells, and the programme that flies them has no say in how they were made. What it has is a screening: every cell measured, put through the part of the environment that finds a weak seal, and measured again. A cell that vents under vacuum loses mass; a cell with a soft short loses open-circuit voltage; a cell with damaged electrolyte loses capacity and gains impedance. The screening does not look for a bad cell in absolute terms, the vendor's data sheet does that. It looks for the cell that changed.

![Close-up of one 18650 cell in a four-wire spring-contact holder with the current and sense cables](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-f8db8906-5248-4e1d-a030-7ebcf7ab4c97-e9v5lHBXv9PXcJ0HtqYEJCZrJMU4BB.jpg)

One cell in its four-wire holder: two contacts carry the current, two read the voltage, so the discharge curve is the cell's and not the holder's.

The criteria are written down for the ISS. The Nanoracks payload safety screening for lithium-ion cells, NR-SRD-139, is the reference most CubeSat programmes follow for COTS cells: 100 % screening with a visual inspection, mass, open-circuit voltage and capacity before and after a vacuum exposure and a vibration, and acceptance on the change: **OCV within 0.1 %, capacity within 5 %, mass within 0.1 %**. The AC impedance at 1 kHz and its change are the programme's own addition here, because a cell whose impedance moved has a problem the other three may not show yet. The vibration is run as its own procedure between the two characterisations; the delta criteria are the same.

### Test Purpose

The procedure records one screening record per cell:

- **Visual inspection result** and the vendor code, before the cell touches an instrument
- **Before**: mass, OCV, AC impedance at 1 kHz, and a C/2 discharge to 3.0 V with the capacity from the curve, each against the data sheet
- **Six hours under vacuum**, open-circuit, with the pressure curve and the time under the threshold
- **After**: the same four, and the **four deltas** the cell is accepted on

![Two panels: on the left the C/2 discharge curve from 4.1 V down to 3.0 V over 3.4 Ah, before in gray and after in dashed green almost on top of each other, a red dashed line at the 3.3 Ah minimum; on the right four pairs of bars, the measured change in OCV, capacity, mass and ACIR as a fraction of its limit in green, each well under the red limit bar.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-408d7145-aa09-4f6e-bd32-20e009887f65-rmNWum5ABXzsqtIpHEN54j2Q4TNzA9.png)

The mock cell before and after six hours under vacuum: 3.400 then 3.380 Ah, a 0.59 % capacity change against 5 %, 0.016 % on OCV against 0.1 %, 0.010 % on mass against 0.1 %, 1.5 % on impedance against 20 %. A cell that vented reads 20 to 50 mg lighter, which is 0.04 to 0.1 % of its mass, and the balance sees it before the electrical numbers move.

The framework mechanics on show are one characterisation function shared by the before and after phases addressing prefixed measurement keys, absolute limits before and deltas after on the same four quantities, measurements kept without limits because the delta is the judgement, a boolean validated with `==`, a reference passed between phases through the plug, and time under a threshold as an aggregation of a pressure curve.

## Equipment & Setup

To screen cells at the pack line, the following are required:

- **A battery cycler channel** with a four-wire holder, for the full charge and the C/2 discharge
- **A battery tester** for OCV and AC impedance at 1 kHz
- **An analytical balance** reading to a milligram or better
- **A small vacuum chamber** with a gauge, a bell jar is enough, holding a rack of cells
- **The Device Under Test (DUT)**: a COTS 18650 cell, 3.5 Ah class, one at a time on the instruments and in a rack in the chamber
- **A TofuPilot Framework procedure** to sequence the two characterisations and the exposure
- **The TofuPilot Dashboard** to see the distribution of every delta across the lot, because a lot with a wide capacity spread is a lot to reject before matching

### Hardware Components

#### Cycler and Battery Tester

A Neware, Arbin or Chroma channel charges the cell to 4.2 V and discharges it at C/2 to 3.0 V with the voltage logged against the charge passed; the capacity is the last point of the curve. A Hioki BT3562 reads the open-circuit voltage to 0.1 mV and the AC impedance at 1 kHz to 0.01 mΩ through the same four-wire holder. The OCV criterion is 0.1 %, which is 3.6 mV, so the tester's resolution is not the constraint; the cell's rest time before the reading is, and the procedure reads OCV before the discharge both times so the state is the same.

![Cylindrical battery cells supported in a vacuum chamber with an external vacuum connection](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/battery-cell-screening-with-vacuum-physical-2026-yQjSpgp2k5fpejQTWIS3kXWPeKj7sD.webp)

The cell rack supports the cells during vacuum exposure. The hose connects to an external vacuum source; the balance and electrical tester are not shown.

#### Balance and Chamber

The mass criterion is 0.1 % of 48 g, which is 48 mg; a balance reading to 1 mg resolves a vent at a twentieth of the limit. A Sartorius or Mettler analytical balance over RS-232 does it. The chamber is a bell jar on a plate with a roughing pump and a Pirani gauge; the exposure is six hours at or below 1e-3 hPa, open-circuit, and the pressure curve is recorded so the time actually spent under the threshold is a number on the record rather than an assumption.

### Where the Limits Come From

The delta criteria on OCV, capacity and mass are NR-SRD-139. The impedance delta of 20 % is the programme's. The absolute limits before the exposure, 3.30 Ah, 3.55 to 3.75 V at the delivery state of charge, 30 mΩ and 47.5 to 49.0 g, are the vendor's data sheet with margin. The exposure of six hours at 1e-3 hPa is the programme's screening plan. A customer keeps the deltas as they are and tightens the absolute windows from the lot's distribution, because the cells that go into one pack have to match each other more than they have to meet the sheet.

## Test Procedure

### Overview

The procedure maps the screening onto the framework's three stages. The visual result lives in `setup:` so a dented cell never reaches the cycler. The three main phases are chained with `depends_on` because the after only makes sense after the exposure, and the exposure after the before. The last OCV lives in `teardown:` so the cell goes to storage with a known voltage on the record whatever the outcome.

1. Setup: visual result, vendor code.
2. Main: mass, OCV, ACIR, C/2 discharge; reference stored on the plug.
3. Main: six hours under vacuum, pressure curve.
4. Main: mass, OCV, ACIR, C/2 discharge; four deltas to the reference.
5. Teardown: OCV at storage.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-serial trending across the lot.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  characterise.py
  pre_screen.py
  vacuum.py
  post_screen.py
  release.py
plugs/
  cell_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-cell-screening-vacuum). The `CellBench` plug is a mock of the cycler, the tester, the balance and the chamber together, synthesizing a healthy 3.42 Ah cell at 3.652 V and 23 mΩ that loses 4 mg, 0.4 mV and 0.6 % of its capacity under vacuum, so the procedure runs end-to-end without an instrument or a cell connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Battery Cell Screening with Vacuum
version: 0.1.0
description: 100 % screening of COTS 18650 cells for a smallsat battery. Visual in setup, then mass, OCV, ACIR at 1 kHz and a C/2 discharge capacity before and after six hours under vacuum, the cell accepted on how little the four numbers changed, after the NR-SRD-139 criteria.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the cell's laser mark"
    placeholder: "CELL-35E-000000"
    pattern: "^CELL-35E-\\d{6}$"
    default_value: "CELL-35E-041872"
  part_number:
    default_value: "INR18650-35E"
  batch_number:
    default_value: "LOT-2026-09"

plugs:
  - name: Cell Bench
    description: "Cycler channel with a four-wire holder, AC impedance meter, analytical balance, vacuum chamber with gauge (mock, one plug per bench)"
    python: plugs.cell_bench:CellBench
    key: bench

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Visual
        key: visual_ok
        validators:
          - {operator: "==", expected_value: true}

main:
  - name: Pre Screen
    key: pre_screen
    python: phases.pre_screen
    timeout: 4h
    measurements:
      - name: Mass Before
        key: mass_pre_g
        unit: g
        validators:
          - {operator: ">=", expected_value: 47.5}
          - {operator: "<=", expected_value: 49.0}
      - name: OCV Before
        key: ocv_pre_v
        unit: V
        validators:
          - {operator: ">=", expected_value: 3.55}
          - {operator: "<=", expected_value: 3.75}
      - name: ACIR Before
        key: acir_pre_mohm
        unit: mΩ
        validators:
          - {operator: "<=", expected_value: 30.0}
      - name: Discharge Before
        key: discharge_pre
        title: C/2 discharge to 3.0 V before the vacuum exposure
        x_axis:
          legend: Capacity
          unit: Ah
        y_axis:
          - legend: Voltage
            key: voltage
            unit: V
            aggregations:
              - type: capacity_ah
                unit: Ah
                validators:
                  - {operator: ">=", expected_value: 3.30}

  - name: Vacuum
    key: vacuum
    python: phases.vacuum
    depends_on: [pre_screen]
    timeout: 12h
    ui:
      components:
        - key: vacuum_progress
          type: progress
          label: "Vacuum exposure"
          description: "6 h at or below 1e-3 hPa, open-circuit"
          default_value: 0
          max: 100
    measurements:
      - name: Exposure
        key: exposure
        title: Chamber pressure through the exposure
        x_axis:
          legend: Time
          unit: h
        y_axis:
          - legend: Pressure
            key: pressure
            unit: hPa
            aggregations:
              - type: min_hpa
                unit: hPa
                validators:
                  - {operator: "<=", expected_value: 1.0e-3}
              - type: hours_under_h
                unit: h
                validators:
                  - {operator: ">=", expected_value: 5.5}

  - name: Post Screen
    key: post_screen
    python: phases.post_screen
    depends_on: [vacuum]
    timeout: 4h
    measurements:
      - name: Mass After
        key: mass_post_g
        unit: g
      - name: OCV After
        key: ocv_post_v
        unit: V
      - name: ACIR After
        key: acir_post_mohm
        unit: mΩ
      - name: Discharge After
        key: discharge_post
        title: C/2 discharge to 3.0 V after the vacuum exposure
        x_axis:
          legend: Capacity
          unit: Ah
        y_axis:
          - legend: Voltage
            key: voltage
            unit: V
            aggregations:
              - type: capacity_ah
                unit: Ah
                validators:
                  - {operator: ">=", expected_value: 3.30}
      - name: OCV Delta
        key: ocv_delta_pct
        unit: "%"
        description: NR-SRD-139 acceptance on the change in open-circuit voltage.
        validators:
          - {operator: "<=", expected_value: 0.1}
      - name: Capacity Delta
        key: capacity_delta_pct
        unit: "%"
        validators:
          - {operator: "<=", expected_value: 5.0}
      - name: Mass Delta
        key: mass_delta_pct
        unit: "%"
        description: A cell that lost mass under vacuum vented; the balance sees it before anything else does.
        validators:
          - {operator: "<=", expected_value: 0.1}
      - name: ACIR Delta
        key: acir_delta_pct
        unit: "%"
        validators:
          - {operator: "<=", expected_value: 20.0}

teardown:
  - name: Release
    key: release
    python: phases.release
    measurements:
      - name: OCV At Storage
        key: ocv_storage_v
        unit: V
```

Framework features to notice:

1. **One function, two phases, prefixed keys.** `phases/characterise.py` writes `mass_<prefix>_g`, `ocv_<prefix>_v` and `acir_<prefix>_mohm` with `setattr` and takes the discharge measurement as an argument, so the before and the after are the same code.
2. **Absolutes before, deltas after.** The four quantities carry data-sheet limits on the `pre` side and none on the `post` side; the judgement after the exposure is the four `*_delta_pct` measurements.
3. **A boolean with `==`.** `visual_ok == true` gates the run on the operator's inspection result.
4. **A reference through the plug.** Phases run in separate processes and cannot read each other's measurements back, so `pre_screen` stores its readings on the bench plug and `post_screen` recalls them.
5. **Time under a threshold.** `exposure.pressure.hours_under_h` is computed from the pressure curve; the exposure is judged on what happened, not on how long the phase ran.

### Identify

The setup phase records the operator's visual inspection result and the vendor code:

```python filename="phases/identify.py"
def identify(measurements, bench, unit, log):
    """Setup: visual inspection result and vendor code on the record;
    a dented, scratched or leaking cell goes no further."""
    ident = bench.identify()
    measurements.visual_ok = ident["visual_ok"]
    unit.metadata["vendor_code"] = ident["vendor_code"]
    log.info(f"Cell {unit.serial_number}: {ident['vendor_code']}, visual {'ok' if ident['visual_ok'] else 'REJECT'}")
```

### Pre Screen

The before phase calls the shared function with the `pre` prefix and the `discharge_pre` measurement, and stores the readings on the plug:

```python filename="phases/pre_screen.py"
from phases.characterise import characterise


def pre_screen(measurements, bench, log):
    """Characterisation before the vacuum exposure; the reference kept on
    the bench plug because phases cannot read each other's measurements."""
    ref = characterise(measurements, bench, log, "pre", measurements.discharge_pre)
    bench.store("pre", ref)
```

The shared function reads mass, OCV and impedance, charges the cell, discharges it at C/2, records the curve and takes the capacity from its last point:

```python filename="phases/characterise.py"
import numpy as np


def characterise(measurements, bench, log, prefix, curve_meas):
    """Shared by the pre and post phases: mass, OCV, ACIR at 1 kHz, then
    a full charge and a C/2 discharge to the cutoff with the capacity
    read from the curve."""
    mass = bench.mass_g()
    ocv = bench.ocv_v()
    acir = bench.acir_mohm()
    bench.charge_full()
    cap = bench.discharge()
    q = np.array(cap["capacity_ah"])
    capacity = float(q[-1])

    setattr(measurements, f"mass_{prefix}_g", mass)
    setattr(measurements, f"ocv_{prefix}_v", ocv)
    setattr(measurements, f"acir_{prefix}_mohm", acir)
    curve_meas.x_axis = cap["capacity_ah"]
    curve_meas.y_axis.voltage = cap["voltage_v"]
    curve_meas.y_axis.voltage.aggregations.capacity_ah = capacity
    log.info(f"{prefix}: {mass:.4f} g, OCV {ocv:.4f} V, ACIR {acir:.2f} mohm, {capacity:.3f} Ah at C/2")
    return {"mass": mass, "ocv": ocv, "acir": acir, "capacity": capacity}
```

### Vacuum

The exposure returns the pressure curve; the phase records it with the lowest pressure reached and the hours spent under the threshold:

```python filename="phases/vacuum.py"
import numpy as np

from utils.recipe import VACUUM_H, VACUUM_HPA_MAX


def vacuum(measurements, bench, ui, log):
    """Six hours under vacuum, open-circuit. A cell with a weak seal or a
    gassing electrolyte vents, and the post characterisation sees it."""
    cap = bench.vacuum_exposure(VACUUM_H)
    ui.vacuum_progress = 100
    t = np.array(cap["time_h"]); p = np.array(cap["pressure_hpa"])
    under = t[p <= VACUUM_HPA_MAX]
    hours_under = float(under[-1] - under[0]) if under.size > 1 else 0.0
    measurements.exposure.x_axis = cap["time_h"]
    measurements.exposure.y_axis.pressure = cap["pressure_hpa"]
    measurements.exposure.y_axis.pressure.aggregations.min_hpa = float(p.min())
    measurements.exposure.y_axis.pressure.aggregations.hours_under_h = hours_under
    log.info(f"Vacuum {t[-1]:.1f} h, {p.min():.1e} hPa at best, {hours_under:.1f} h under {VACUUM_HPA_MAX:.0e} hPa")
```

### Post Screen

The after phase calls the same function with the `post` prefix, recalls the reference, and computes the four deltas:

```python filename="phases/post_screen.py"
from phases.characterise import characterise


def post_screen(measurements, bench, log):
    """Characterisation after the vacuum exposure, and the four deltas
    against the reference: the numbers the cell is accepted on."""
    post = characterise(measurements, bench, log, "post", measurements.discharge_post)
    ref = bench.recall("pre")
    measurements.ocv_delta_pct = float(100.0 * abs(post["ocv"] - ref["ocv"]) / ref["ocv"])
    measurements.capacity_delta_pct = float(100.0 * abs(post["capacity"] - ref["capacity"]) / ref["capacity"])
    measurements.mass_delta_pct = float(100.0 * abs(post["mass"] - ref["mass"]) / ref["mass"])
    measurements.acir_delta_pct = float(100.0 * abs(post["acir"] - ref["acir"]) / ref["acir"])
    log.info(f"Deltas: OCV {100.0 * abs(post['ocv'] - ref['ocv']) / ref['ocv']:.3f} %, capacity {100.0 * abs(post['capacity'] - ref['capacity']) / ref['capacity']:.2f} %, mass {100.0 * abs(post['mass'] - ref['mass']) / ref['mass']:.4f} %, ACIR {100.0 * abs(post['acir'] - ref['acir']) / ref['acir']:.1f} %")
```

### Mock Plug

`CellBench` stands in for the cycler channel, the battery tester, the balance and the chamber. It draws a cell with 3.42 Ah at C/2, 3.652 V, 23 mΩ and 48.15 g, a discharge curve with the plateau and the knee of an NMC cell less the IR drop at C/2, a chamber that pumps to 4e-4 hPa in minutes and holds, and an exposure that takes 4 mg, 0.4 mV, 0.6 % of capacity and adds 0.3 mΩ. The discharge and the exposure are time-scaled and return in one call each. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines, and the reference crosses phases through the plug's `store` and `recall`.

On a real station, the class speaks the cycler's API for the charge and the discharge with the curve read back, SCPI to the Hioki tester for OCV and ACIR, the balance's serial protocol, and the chamber's pump and gauge. Set `TIME_SCALE = 1.0` in `utils/recipe.py`, run the vibration screening as its own procedure between the two characterisations if the programme requires it, and take the absolute windows from the cell's data sheet. The phases, measurements and limits stay the same.

### Random Vibration Acceptance Test
URL: https://www.tofupilot.com/templates/random-vibration-acceptance-test
Random vibration acceptance with TofuPilot Framework: sine signature, GEVS 10 grms random run judged on the control channel, first-mode shift, three axes.

## Introduction

### Random Vibration Acceptance Overview

Every unit that flies is shaken before it ships. The random vibration acceptance test puts the unit on a shaker, drives the fixture to a specified spectrum for a minute in each axis, and reads two things: whether the shaker actually delivered the spectrum, and whether the unit is the same unit afterwards. The first is judged on the control accelerometer against the reference profile. The second is judged with a low-level sine sweep before and after each run, the resonance signature; a bolt that lost preload or a bracket that cracked moves the first mode, and the sweep is how the test sees it without opening the box.

![Close-up of an electronic unit bolted on a fixture plate with a control accelerometer on the plate and a response accelerometer on the unit](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-c56ade1c-9096-43db-8d1e-0c1a93829261-07tIppwrOuVSdQEHiBUV1I90tDogWM.jpg)

The unit on its fixture: four bolts at torque, the control accelerometer on the plate next to the unit and the response accelerometer on the unit's lid. The control channel closes the loop with the controller; the response channel is what the unit felt.

The levels are written down. NASA's GEVS gives generalized random vibration levels for components of 22.7 kg or less: **10.0 grms for acceptance, 14.1 grms for qualification and protoflight**, one minute per axis, a profile that rises at 6 dB/octave from 0.013 g²/Hz at 20 Hz to 0.08 g²/Hz at 50 Hz, holds to 800 Hz and falls at 6 dB/octave to 2000 Hz (see [GSFC-STD-7000A](https://standards.nasa.gov/standard/GSFC/GSFC-STD-7000)). SMC-S-016 sets the acceptance workmanship minimum at 6.9 grms, not the 6.8 that circulates. What the standards do not give is a number for the signature comparison: the 5 % shift in the first mode that most test plans use as the flag is customary, and this template treats it as a recipe constant, not a clause.

### Test Purpose

The procedure records one three-axis acceptance record per unit:

- **Fixture and instrumentation gate**: control accelerometer sensitivity against its calibration, unit mass, the four bracket bolts at torque
- Per axis, **a low-level sine signature**: first mode above the launcher's floor, amplification bounded
- Per axis, **the random run at the acceptance level**: control grms within its window, every spectral line within tolerance of the reference, a full minute at level, the response on the unit recorded and bounded
- Per axis, **the signature again**, with the first-mode shift and the amplification change validated against the pre-run curve

![Three log-log panels, X, Y and Z, each showing the transmissibility of the unit over 5 to 2000 Hz before the random run in gray and after in dashed green, first modes at 210, 260 and 340 Hz with a second mode above, red dashed lines at the 100 Hz first-mode floor and the Q of 30 ceiling.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-2ca5a95b-d24a-4b0a-bc01-b5e6ea407155-xSANYTuBK9tsEnQTpISbvxjQPCMCcM.png)

The mock unit's signatures: first modes at 210, 260 and 340 Hz with amplifications of 12 to 16, and after each random run a shift of 0.5 to 1 % as the bracket joint settles, against the 5 % flag. A bolt that backed off shifts the mode by several percent and drops the amplification; a cracked bracket adds a mode.

The framework mechanics on show are one Python function shared by three axis phases that address per-axis measurement keys, three curves in one measurement with aggregations on two of them, a window on an overall level next to a per-line tolerance, a before-versus-after comparison expressed as aggregations of the post curve, peak picking below the sweep's line spacing, and a fixture gate in `setup:` that keeps the shaker off until the sensor, the mass and the torques are right.

## Equipment & Setup

To run random vibration acceptance on electronic units, the following are required:

- **An electrodynamic shaker** with a head expander for the vertical axis and a slip table for the two lateral ones
- **A vibration controller** with at least one control and one response channel, sine and random modes, and an API the procedure can drive
- **IEPE accelerometers** with TEDS or a calibration sheet, one on the fixture for control and one on the unit for response
- **The Device Under Test (DUT)**: a smallsat onboard computer, 1.2 kg, on its four-bolt bracket
- **A TofuPilot Framework procedure** to sequence the gate, the three axes and the signature comparisons
- **The TofuPilot Dashboard** to keep the signature per serial across the lot and the vibration-exposure log the bearings and the crystals care about

### Hardware Components

#### Shaker and Controller

A shaker of the LDS V8 or Unholtz-Dickie class with a slip table covers a unit this size with margin. The controller does the closed-loop work: it drives the shaker so the control accelerometer's spectrum matches the reference, and it reports the control and response ASDs averaged over the minute at level, the overall grms and the elapsed time at full level. m+p VibControl, Crystal Instruments Spider and Siemens LMS all expose this over an API; the procedure asks for the run and reads the result, it does not close the loop itself.

![Electronic unit and accelerometer bolted to a shaker fixture plate](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/random-vibration-acceptance-test-physical-2026-2pVG3ociaWkIRcRJa5mnZldNjKRXEc.webp)

The unit and control accelerometer mount to the moving fixture plate. Cable service loops accommodate motion; the controller and acquisition equipment are not shown.

#### Accelerometers and the Setup Gate

The control accelerometer's sensitivity scales the entire test. A 10 mV/g sensor read as 9.5 mV/g drives the unit 5 % harder than the reference. The setup phase reads the sensitivity from the TEDS or the calibration sheet and compares it to the recipe; it also reads the unit's mass, because the reference profile is for a mass class, and the torque log of the four bracket bolts, because a bolt below torque is the most common reason a signature shifts and the test should not start on one.

#### The Signature

The low-level sine sweep, 0.5 g from 5 to 2000 Hz at 2 octaves per minute, gives the transmissibility of the unit on its fixture: response over control against frequency. The first peak above the noise is the first mode; its height is the amplification Q. The procedure picks the peak with a parabolic fit in log-log around the highest line, because the sweep's 1/24-octave lines are 3 % apart and the shift criterion is 5 %. The same sweep after the random run, on the same fixture with nothing touched, is the comparison the test is named for.

### Where the Limits Come From

The reference profile and the 10.0 grms are GEVS acceptance for components of 22.7 kg or less; a protoflight programme sets `RANDOM_GRMS_TARGET` to 14.1 and scales the profile. The 9.0 to 11.0 grms window is ±10 % on the overall level and the ±1.5 dB per line is the controller tolerance most test plans use. The 100 Hz first-mode floor is the launcher's secondary-structure requirement from the ICD. The Q ceiling of 30 and the 40 grms response ceiling are the unit's own qualification. The 5 % first-mode shift and the 20 % amplification change are the customary flags, and a customer tightens them from the distribution the dashboard shows after a few dozen units.

## Test Procedure

### Overview

The procedure maps the acceptance test onto the framework's three stages. The instrumentation and fixture gate lives in `setup:` so the shaker never runs on a wrong sensor or a loose bolt. The three axes run as three main phases chained with `depends_on`, because the fixture is rotated between them. Recording the exposure lives in `teardown:` so the serial's log is updated whatever happened.

1. Setup: controller, control accelerometer sensitivity, unit mass, bracket torques.
2. Main, X: signature, random run at 10.0 grms for 60 s, signature.
3. Main, Y: same.
4. Main, Z: same.
5. Teardown: shaker off, exposure added to the serial's record.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-serial trending across the lot.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  vibration.py
  axis_x.py
  axis_y.py
  axis_z.py
  unmount.py
plugs/
  shaker_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-random-vibration-acceptance). The `ShakerBench` plug is a mock of the controller, the shaker and the accelerometers together, synthesizing a healthy unit with a first mode at 210, 260 and 340 Hz in the three axes, a controller that holds the reference within a fraction of a decibel, and a bracket joint that settles under one percent per run, so the procedure runs end-to-end without a shaker or a unit connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit. The three axis phases are identical apart from their keys:

```yaml filename="procedure.yaml"
name: Random Vibration Acceptance Test
version: 0.1.0
description: Random vibration acceptance of a smallsat electronic unit, three axes. Fixture and instrumentation checked in setup, then per axis a low-level sine signature, the GEVS acceptance random run at 10.0 grms for one minute judged on the control channel against the reference, and the signature again with the first-mode shift validated.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the unit label"
    placeholder: "OBC-1U-0000"
    pattern: "^OBC-1U-\\d{4}$"
    default_value: "OBC-1U-0231"
  part_number:
    default_value: "OBC-1U-B"
  batch_number:
    default_value: "LOT-2026-09"

plugs:
  - name: Shaker Bench
    description: "Vibration controller, shaker and slip table, control and response accelerometers, bolt torque log (mock, one plug per bench)"
    python: plugs.shaker_bench:ShakerBench
    key: bench

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Controller
        key: controller_id
        validators:
          - {operator: matches, expected_value: "^VC-\\d$"}
      - name: Control Accelerometer Error
        key: control_accel_error_pct
        unit: "%"
        description: Sensitivity read from the TEDS or the calibration sheet against the recipe value; the whole test is scaled by it.
        validators:
          - {operator: ">=", expected_value: -5.0}
          - {operator: "<=", expected_value: 5.0}
      - name: Unit Mass
        key: unit_mass_kg
        unit: kg
        validators:
          - {operator: ">=", expected_value: 1.1}
          - {operator: "<=", expected_value: 1.3}
      - name: Bracket Torque Min
        key: bracket_torque_min_nm
        unit: Nm
        validators:
          - {operator: ">=", expected_value: 2.25}
      - name: Bracket Torque Max
        key: bracket_torque_max_nm
        unit: Nm
        validators:
          - {operator: "<=", expected_value: 2.75}

main:
  - name: Axis X
    key: axis_x
    python: phases.axis_x
    timeout: 30m
    measurements:
      - name: Survey Pre X
        key: survey_pre_x
        title: Low-level sine signature before the random run, X axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Transmissibility
            key: transmissibility
            unit: g/g
            aggregations:
              - type: f1_hz
                unit: Hz
                validators:
                  - {operator: ">=", expected_value: 100.0}
              - type: q
                validators:
                  - {operator: "<=", expected_value: 30.0}
      - name: Random X
        key: random_x
        title: Random acceptance run, control and response ASD against the reference, X axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Reference
            key: reference
            unit: g²/Hz
          - legend: Control
            key: control
            unit: g²/Hz
            aggregations:
              - type: grms
                unit: grms
                validators:
                  - {operator: ">=", expected_value: 9.0}
                  - {operator: "<=", expected_value: 11.0}
              - type: spectral_dev_db
                unit: dB
                validators:
                  - {operator: "<=", expected_value: 1.5}
              - type: duration_s
                unit: s
                validators:
                  - {operator: ">=", expected_value: 60.0}
          - legend: Response
            key: response
            unit: g²/Hz
            aggregations:
              - type: grms
                unit: grms
                validators:
                  - {operator: "<=", expected_value: 40.0}
      - name: Survey Post X
        key: survey_post_x
        title: Low-level sine signature after the random run, X axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Transmissibility
            key: transmissibility
            unit: g/g
            aggregations:
              - type: f1_shift_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 5.0}
              - type: q_change_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 20.0}

  - name: Axis Y
    key: axis_y
    python: phases.axis_y
    depends_on: [axis_x]
    timeout: 30m
    measurements:
      - name: Survey Pre Y
        key: survey_pre_y
        title: Low-level sine signature before the random run, Y axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Transmissibility
            key: transmissibility
            unit: g/g
            aggregations:
              - type: f1_hz
                unit: Hz
                validators:
                  - {operator: ">=", expected_value: 100.0}
              - type: q
                validators:
                  - {operator: "<=", expected_value: 30.0}
      - name: Random Y
        key: random_y
        title: Random acceptance run, control and response ASD against the reference, Y axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Reference
            key: reference
            unit: g²/Hz
          - legend: Control
            key: control
            unit: g²/Hz
            aggregations:
              - type: grms
                unit: grms
                validators:
                  - {operator: ">=", expected_value: 9.0}
                  - {operator: "<=", expected_value: 11.0}
              - type: spectral_dev_db
                unit: dB
                validators:
                  - {operator: "<=", expected_value: 1.5}
              - type: duration_s
                unit: s
                validators:
                  - {operator: ">=", expected_value: 60.0}
          - legend: Response
            key: response
            unit: g²/Hz
            aggregations:
              - type: grms
                unit: grms
                validators:
                  - {operator: "<=", expected_value: 40.0}
      - name: Survey Post Y
        key: survey_post_y
        title: Low-level sine signature after the random run, Y axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Transmissibility
            key: transmissibility
            unit: g/g
            aggregations:
              - type: f1_shift_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 5.0}
              - type: q_change_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 20.0}

  - name: Axis Z
    key: axis_z
    python: phases.axis_z
    depends_on: [axis_y]
    timeout: 30m
    measurements:
      - name: Survey Pre Z
        key: survey_pre_z
        title: Low-level sine signature before the random run, Z axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Transmissibility
            key: transmissibility
            unit: g/g
            aggregations:
              - type: f1_hz
                unit: Hz
                validators:
                  - {operator: ">=", expected_value: 100.0}
              - type: q
                validators:
                  - {operator: "<=", expected_value: 30.0}
      - name: Random Z
        key: random_z
        title: Random acceptance run, control and response ASD against the reference, Z axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Reference
            key: reference
            unit: g²/Hz
          - legend: Control
            key: control
            unit: g²/Hz
            aggregations:
              - type: grms
                unit: grms
                validators:
                  - {operator: ">=", expected_value: 9.0}
                  - {operator: "<=", expected_value: 11.0}
              - type: spectral_dev_db
                unit: dB
                validators:
                  - {operator: "<=", expected_value: 1.5}
              - type: duration_s
                unit: s
                validators:
                  - {operator: ">=", expected_value: 60.0}
          - legend: Response
            key: response
            unit: g²/Hz
            aggregations:
              - type: grms
                unit: grms
                validators:
                  - {operator: "<=", expected_value: 40.0}
      - name: Survey Post Z
        key: survey_post_z
        title: Low-level sine signature after the random run, Z axis
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Transmissibility
            key: transmissibility
            unit: g/g
            aggregations:
              - type: f1_shift_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 5.0}
              - type: q_change_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 20.0}

teardown:
  - name: Unmount
    key: unmount
    python: phases.unmount
    measurements:
      - name: Exposure Added
        key: exposure_added_s
        unit: s
```

Framework features to notice:

1. **One function, three phases.** `phases/vibration.py` runs the survey, the random run and the survey for one axis; `axis_x`, `axis_y` and `axis_z` each call it with their axis name, and the function addresses `survey_pre_<axis>`, `random_<axis>` and `survey_post_<axis>` with `getattr`.
2. **Three curves, two judged.** `random_<axis>` records the reference, the control and the response ASDs; the pass/fail lives on the control's `grms`, `spectral_dev_db` and `duration_s`, and on the response's `grms`. The reference curve is there for the chart and the record.
3. **A window and a per-line tolerance on the same data.** `control.grms` in 9.0 to 11.0 bounds the overall level; `spectral_dev_db <= 1.5` bounds the worst line. A run can pass one and fail the other.
4. **Before versus after as aggregations of the after curve.** `survey_post_<axis>` carries `f1_shift_pct` and `q_change_pct`, computed in the same phase from the pre-run signature it still has in hand; the dashboard shows both curves and the two numbers together.
5. **A gate on the fixture.** The setup measurements say nothing about the unit's quality; they say the sensor, the mass and the bolts are what the test assumes.
6. **Integrate in the phase.** Plug calls cross a JSON boundary, so the grms integration and the peak picking are phase-side numpy, not plug methods handed arrays.

### Identify

The setup phase reads the controller's identity, the control accelerometer's sensitivity against the recipe value, the unit's mass and the four bracket torques, and records the fixture on the **unit metadata**:

```python filename="phases/identify.py"
import numpy as np

from utils.recipe import BRACKET_TORQUE_NM, CONTROL_ACCEL_MV_PER_G


def identify(measurements, bench, unit, log):
    """Setup: control accelerometer sensitivity against its calibration,
    unit mass, the four bracket bolts at torque, controller identified.
    Nothing shakes until the fixture is right."""
    ident = bench.identify()
    torque = bench.bracket_torque_nm()
    measurements.controller_id = ident["controller"]
    measurements.control_accel_error_pct = float(100.0 * (ident["control_accel_mv_per_g"] - CONTROL_ACCEL_MV_PER_G) / CONTROL_ACCEL_MV_PER_G)
    measurements.unit_mass_kg = ident["unit_mass_kg"]
    measurements.bracket_torque_min_nm = float(min(torque))
    measurements.bracket_torque_max_nm = float(max(torque))
    unit.metadata["fixture"] = "FX-OBC-04"
    log.info(f"Unit {unit.serial_number}: {ident['unit_mass_kg']:.3f} kg, control accel {ident['control_accel_mv_per_g']:.3f} mV/g, bolts {min(torque):.2f}..{max(torque):.2f} Nm against {BRACKET_TORQUE_NM} Nm")
```

### Axis X, Y and Z

Each axis phase is three lines that call the shared function:

```python filename="phases/axis_x.py"
from phases.vibration import run_axis


def axis_x(measurements, bench, log):
    """X axis: survey, random at acceptance level, survey."""
    run_axis(measurements, bench, log, "x")
```

The shared function runs the survey, picks the first mode, runs the random test and computes the three control aggregations and the response level, runs the survey again and computes the shift and the amplification change:

```python filename="phases/vibration.py"
import numpy as np

from utils.recipe import RANDOM_DURATION_S


def grms(f, asd):
    """Overall level from an ASD: the square root of its integral."""
    return float(np.sqrt(np.trapezoid(np.asarray(asd), np.asarray(f))))


def signature(f, t):
    """First mode and its amplification from a transmissibility curve."""
    f = np.asarray(f); t = np.asarray(t)
    peaks = np.flatnonzero((t[1:-1] > t[:-2]) & (t[1:-1] > t[2:]) & (t[1:-1] > 3.0)) + 1
    i = peaks[0] if peaks.size else int(t.argmax())
    # Parabolic interpolation in log-log around the peak: the sweep's
    # 1/24 octave lines are 3 % apart, the shift criterion is 5 %.
    x = np.log(f[i - 1:i + 2]); y = np.log(t[i - 1:i + 2])
    a, b, c = np.polyfit(x, y, 2)
    xp = -b / (2.0 * a)
    return float(np.exp(xp)), float(np.exp(a * xp ** 2 + b * xp + c))


def run_axis(measurements, bench, log, axis):
    """One axis: low-level sine survey, random run at the acceptance
    level, sine survey again. The two signatures are compared; the
    random run is judged on the control channel against the reference."""
    bench.set_axis(axis)

    pre = bench.sine_survey()
    f1_pre, q_pre = signature(pre["freq_hz"], pre["transmissibility"])
    m = getattr(measurements, f"survey_pre_{axis}")
    m.x_axis = pre["freq_hz"]
    m.y_axis.transmissibility = pre["transmissibility"]
    m.y_axis.transmissibility.aggregations.f1_hz = f1_pre
    m.y_axis.transmissibility.aggregations.q = q_pre

    run = bench.random_run()
    f = np.array(run["freq_hz"])
    ctrl_grms = grms(f, run["control"])
    ref_grms = grms(f, run["reference"])
    resp_grms = grms(f, run["response"])
    dev_db = 10.0 * np.log10(np.array(run["control"]) / np.array(run["reference"]))
    m = getattr(measurements, f"random_{axis}")
    m.x_axis = run["freq_hz"]
    m.y_axis.reference = run["reference"]
    m.y_axis.control = run["control"]
    m.y_axis.control.aggregations.grms = ctrl_grms
    m.y_axis.control.aggregations.spectral_dev_db = float(np.abs(dev_db).max())
    m.y_axis.control.aggregations.duration_s = float(run["duration_s"])
    m.y_axis.response = run["response"]
    m.y_axis.response.aggregations.grms = resp_grms

    post = bench.sine_survey()
    f1_post, q_post = signature(post["freq_hz"], post["transmissibility"])
    m = getattr(measurements, f"survey_post_{axis}")
    m.x_axis = post["freq_hz"]
    m.y_axis.transmissibility = post["transmissibility"]
    m.y_axis.transmissibility.aggregations.f1_shift_pct = float(100.0 * abs(f1_post - f1_pre) / f1_pre)
    m.y_axis.transmissibility.aggregations.q_change_pct = float(100.0 * abs(q_post - q_pre) / q_pre)
    bench.stop()
    log.info(f"{axis.upper()}: f1 {f1_pre:.1f} Hz Q {q_pre:.1f}; random {ctrl_grms:.2f} grms control ({ref_grms:.2f} reference), worst line {np.abs(dev_db).max():.2f} dB, {resp_grms:.1f} grms on the unit, {run['duration_s']:.0f} s; post f1 {f1_post:.1f} Hz, shift {100.0 * abs(f1_post - f1_pre) / f1_pre:.2f} %")
```

![Log-log plot of ASD against frequency from 20 to 2000 Hz for the Z axis: the GEVS reference profile as a red dashed line inside a shaded plus and minus 1.5 dB band, the control spectrum in green tracking it, and the response on the unit in orange rising to a peak near 10 g²/Hz at 340 Hz.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-7c2a9f2d-f0c8-41f5-bfcb-590ac6280404-zHeQPTG5AHgKcSKeUT0m1TYL1cymhL.png)

The Z axis random run on the mock: the control channel at 9.98 grms against the 10.0 reference with a worst line 0.76 dB off, and the response on the unit at 22.6 grms with its first mode at 340 Hz amplifying the input by a factor of twelve at the peak. Across the three axes the control held 9.98 to 9.99 grms with worst lines of 0.69 to 0.97 dB, and the signatures shifted 0.52 to 0.95 %.

### Unmount

The teardown switches the shaker off and adds the three minutes of acceptance random to the serial's exposure record on the unit metadata:

```python filename="phases/unmount.py"
from utils.recipe import AXES, RANDOM_DURATION_S


def unmount(measurements, bench, unit, log):
    """Teardown: shaker off, exposure added to the serial's record."""
    bench.stop()
    exposure = float(len(AXES) * RANDOM_DURATION_S)
    measurements.exposure_added_s = exposure
    unit.metadata["vibration_exposure_added_s"] = exposure
    log.info(f"Shaker off, {exposure:.0f} s of acceptance random added to the serial's exposure log")
```

### Mock Plug

`ShakerBench` stands in for the controller, the shaker and the accelerometers. It draws a unit with a first mode at 210, 260 and 340 Hz in X, Y and Z with amplifications of 12 to 16 and a second mode at 2.6 times the first, a sine sweep on 1/24-octave lines from 5 to 2000 Hz, a random run whose control spectrum sits within a 0.3 dB standard deviation of the GEVS reference and whose response is the control times the transmissibility squared, and a bracket joint that lowers the first mode by 0.4 to 0.8 % at each random run. The sweeps and the run return in one call each. Every method returns plain Python types because plug calls cross a JSON boundary, which is also why the integration and the peak picking live in the phase.

On a real station, the class speaks the controller's API for the sine sweep and the random run with the control and response ASDs read back, reads the accelerometers' TEDS for the sensitivities, and reads the torque wrench log for the bolts. Set `TIME_SCALE = 1.0` in `utils/recipe.py`, set `RANDOM_PROFILE` and `RANDOM_GRMS_TARGET` to the programme's level, and keep the vibration-exposure log per serial in the dashboard. The phases, measurements and limits stay the same.

### Magnetorquer Dipole Moment Test
URL: https://www.tofupilot.com/templates/magnetorquer-dipole-moment-test
Torque rod acceptance with TofuPilot Framework: Helmholtz cage residual, coil resistance and insulation, dipole sweep with linearity and polarity, remanence.

## Introduction

### Magnetorquer Acceptance Overview

A magnetorquer is a coil on a high-permeability core. Pass a current through it and it becomes a magnet whose moment, crossed with the Earth's field, is the torque a small satellite uses to dump momentum and to point when the wheels are off. The attitude control law assumes a moment proportional to the current, with a known gain, a known sign along the rod's marked axis, and nothing left when the current is cut. The acceptance test measures those four things, plus the two electrical numbers that say the coil is whole and insulated from the case.

![Close-up of a torque rod in its V-block cradle with the +X marking, the fluxgate on a post in line with the rod at a fixed distance](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-f7c1ffde-5bf3-4740-962e-247eb1006372-YnZiXobfzUpVo2kUItnUv7lkx4Gl5H.jpg)

The rod in its cradle with the fluxgate on the axis at a fixed distance. The distance is the one number that scales every dipole reading, so it is machined into the fixture and written to the unit metadata.

The moment is read as a field. On the axis of a dipole, B = (μ0 / 2π) m / r³, so 1 A·m² at 0.5 m is 1600 nT, and the sensor reads that on top of whatever the room and the Earth give it, which is 50 000 nT. A Helmholtz cage nulls the ambient field to a few tens of nanotesla, and the zero-current reading is subtracted as the residual. The vendor sheets this rod class is modelled on (NewSpace Systems, ZARM, CubeSpace) quote the dipole at rated current, a linearity figure, the remanent dipole with the current off, the coil resistance and the insulation to the case; none of these come from a standard. The remanent dipole is the one that matters most and is the least advertised: the core keeps a fraction of a percent of the rated moment after a pulse, and that is a permanent disturbance torque the attitude model has to carry.

### Test Purpose

The procedure records one acceptance record per rod:

- **Cage residual** with the rod unpowered, and the rod's axis marking, before anything else
- **Coil resistance** four-wire and **insulation to the case** at 500 V DC
- **A current sweep** from -150 to +150 mA, the axial field at each step converted to a dipole, judged on the value at rated current, the gain of the straight fit, the worst deviation from it, and the polarity against the marking
- **The remanent dipole** with the current cut after a rated pulse

![Two panels: on the left the dipole moment in A·m² against coil current from -150 to +150 mA as green dots on an orange dashed straight fit through ±1 A·m²; on the right green bars of deviation from the fit in percent of rated within ±0.7 % between red dashed limits at ±2 %, and an orange dotted line for the remanent dipole.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-d35cb534-2381-4760-8c77-990a80f05b2c-5K7QPG3cox0s187ZMbj1QquL7BekI7.png)

The mock rod's sweep: 1.027 A·m² at +150 mA, a gain of 6.87 A·m²/A, a worst deviation of 0.70 % of rated as the core begins to saturate at the ends, and 4.3 mA·m² of remanence after the positive pulse. A rod with a cracked core reads low and bends early; a rod wired backwards reads a perfect curve of the wrong sign, which is what the polarity measurement exists for.

The framework mechanics on show are a physical conversion done in the phase so the record stores the dipole rather than the field, three aggregations on one curve (a value, a slope and a deviation), a string measurement validated with `==`, a facility floor validated in `setup:` and written to the unit metadata, a tolerance window as two validators, and a zero reference re-read in the phase that needs it.

## Equipment & Setup

To accept magnetorquers on a production bench, the following are required:

- **A three-axis Helmholtz cage** with bipolar supplies that null the ambient field at the sensor
- **A fluxgate magnetometer** on the rod's axis at a fixed, measured distance
- **A precision current source** for the coil, a **micro-ohm meter** and an **insulation tester** through a switch box
- **The Device Under Test (DUT)**: a 1 A·m² torque rod, 5 V, 150 mA rated
- **A TofuPilot Framework procedure** to sequence the cage check, the electrical tests, the sweep and the remanence reading
- **The TofuPilot Dashboard** to trend gain, linearity and remanence across the lot

### Hardware Components

#### Helmholtz Cage and Fluxgate

A cage of three nested coil pairs, driven by three bipolar supplies, cancels the Earth's field in a volume large enough to hold the rod and the sensor. The residual it leaves is the floor of the measurement, and the procedure reads it in setup and again as the zero of every sweep. A Bartington Mag-03 or Mag-13 fluxgate read through a Spectramag or an NI DAQ resolves well under a nanotesla, which against 1600 nT per A·m² at 0.5 m is a resolution of a fraction of a milli-ampere-square-metre. What limits the measurement in practice is the cage's stability over the minute the sweep takes, which is why the sweep is short and the zero is read right before it.

![Magnetorquer rod and magnetic probe between a pair of coaxial field coils](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/magnetorquer-dipole-moment-test-physical-2026-uod3OeiEs0YPYo6lIjH4qxI5hBcDAI.webp)

A simplified single-axis coil fixture illustrates the rod and probe arrangement. The procedure's full three-axis compensation setup and instruments are not shown.

#### Source and Meters

A Keithley 2450 drives the coil with the sweep currents and reads them back. A Hioki RM3545 reads the coil four-wire, which on a 33 Ω coil is not about resolution but about the connector and cable resistance a two-wire reading would add. A Megger MIT applies 500 V DC between the coil and the case for the insulation reading. A switch box routes the coil between the three instruments so the rod is connected once.

### Where the Limits Come From

The rated dipole, the gain, the 2 % linearity, the 10 mA·m² remanence, the 33 Ω coil with 5 % and the 100 MΩ insulation are the rod's data sheet. The 50 nT cage residual is what the facility achieves and what the dipole numbers are only as good as. A customer sets the remanence limit from the attitude control budget, because that is the disturbance the controller carries for the mission, and tightens the linearity from the distribution the dashboard shows after a few dozen rods.

## Test Procedure

### Overview

The procedure maps the acceptance test onto the framework's three stages. The cage check lives in `setup:` so no sweep is read against a bad zero. The three main phases are chained with `depends_on` because they share the rod, the source and the sensor. Cutting the current and reading the field one last time live in `teardown:` so the next rod starts from a known cage.

1. Setup: rod marking, sensor distance on the metadata, cage residual.
2. Main: coil resistance, insulation at 500 V.
3. Main: sweep from -150 to +150 mA, dipole at rated, gain, linearity, polarity.
4. Main: remanent dipole after a rated pulse.
5. Teardown: current off, field at the sensor.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-serial trending across the lot.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  electrical.py
  dipole_sweep.py
  residual.py
  power_off.py
plugs/
  torquer_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-magnetorquer-dipole-test). The `TorquerBench` plug is a mock of the cage, the fluxgate, the source and the meters together, synthesizing a healthy rod with a gain of about 7 A·m²/A, a core that begins to saturate gently at the ends of the sweep, 0.6 % remanence after a rated pulse, a 33 Ω coil, over 1 GΩ of insulation and a cage that nulls to 12 nT, so the procedure runs end-to-end without a cage or a rod connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Magnetorquer Dipole Moment Test
version: 0.1.0
description: Acceptance of a 1 A·m² torque rod in a Helmholtz cage. Cage residual and rod marking in setup, coil resistance and 500 V insulation, a current sweep from -150 to +150 mA with the axial field converted to dipole through the on-axis law and judged on the value at rated current, the straightness, the gain and the polarity, then the remanent dipole with the current cut.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the rod's label"
    placeholder: "MTQ-1-0000"
    pattern: "^MTQ-1-\\d{4}$"
    default_value: "MTQ-1-0416"
  part_number:
    default_value: "MTQ-1-5V"
  batch_number:
    default_value: "LOT-2026-09"

plugs:
  - name: Torquer Bench
    description: "Helmholtz cage, fluxgate on the rod's axis, coil current source, micro-ohm meter, insulation tester (mock, one plug per bench)"
    python: plugs.torquer_bench:TorquerBench
    key: bench

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Rod Marking
        key: rod_marking
        validators:
          - {operator: matches, expected_value: "^MTQ-1-[XYZ]$"}
      - name: Cage Residual
        key: cage_residual_nt
        unit: nT
        description: Field left at the sensor with the cage nulling and the rod unpowered; the floor of every dipole number below.
        validators:
          - {operator: "<=", expected_value: 50.0}

main:
  - name: Electrical
    key: electrical
    python: phases.electrical
    measurements:
      - name: Coil Resistance
        key: coil_resistance_ohm
        unit: Ω
        validators:
          - {operator: ">=", expected_value: 31.35}
          - {operator: "<=", expected_value: 34.65}
      - name: Insulation
        key: insulation_mohm
        unit: MΩ
        validators:
          - {operator: ">=", expected_value: 100.0}

  - name: Dipole Sweep
    key: dipole_sweep
    python: phases.dipole_sweep
    depends_on: [electrical]
    timeout: 10m
    measurements:
      - name: Sweep
        key: sweep
        title: Dipole moment against coil current, -150 to +150 mA
        x_axis:
          legend: Current
          unit: A
        y_axis:
          - legend: Dipole
            key: dipole
            unit: A·m²
            aggregations:
              - type: at_rated_am2
                unit: A·m²
                validators:
                  - {operator: ">=", expected_value: 0.95}
              - type: gain_am2_per_a
                unit: A·m²/A
                validators:
                  - {operator: ">=", expected_value: 6.3}
              - type: linearity_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 2.0}
      - name: Polarity
        key: polarity
        description: Sign of the moment at positive current against the rod's +X marking; a rod wired backwards passes every other number.
        validators:
          - {operator: "==", expected_value: "+X"}

  - name: Residual
    key: residual
    python: phases.residual
    depends_on: [dipole_sweep]
    measurements:
      - name: Residual Dipole
        key: residual_am2
        unit: A·m²
        validators:
          - {operator: "<=", expected_value: 0.010}

teardown:
  - name: Power Off
    key: power_off
    python: phases.power_off
    measurements:
      - name: Field After
        key: field_after_nt
        unit: nT
```

Framework features to notice:

1. **Field in, dipole out.** The instrument reads nanotesla; the phase applies the on-axis law with the sensor distance from the recipe, and the record stores ampere-square-metres. The dashboard trends the number the attitude engineer wants.
2. **A value, a slope and a deviation on one curve.** `sweep.dipole` carries `at_rated_am2`, `gain_am2_per_a` from the straight fit and `linearity_pct` as the worst departure from it, each with its own limit.
3. **A string with `==`.** `polarity` is `"+X"` or `"-X"`; the validator compares it to the marking. A number would have hidden the question.
4. **A facility floor in setup.** `cage_residual_nt` is not a property of the rod; it gates the run and goes on the unit metadata with the sensor distance, so every dipole on the record can be traced to its zero and its scale.
5. **Re-read the zero where it is used.** Phases run in separate processes and cannot read each other's measurements back, so `residual` reads the cage again instead of reaching for the setup value.

### Identify

The setup phase reads the rod's marking and the sensor distance, nulls the cage with the rod unpowered, and puts the distance on the **unit metadata**:

```python filename="phases/identify.py"
def identify(measurements, bench, unit, log):
    """Setup: rod marking, sensor distance on the record, cage nulled
    with the rod unpowered. The dipole numbers are only as good as the
    field the cage leaves at the sensor."""
    ident = bench.identify()
    measurements.rod_marking = ident["rod_marking"]
    measurements.cage_residual_nt = bench.cage_null()
    unit.metadata["sensor_distance_m"] = ident["sensor_distance_m"]
    log.info(f"Rod {unit.serial_number} marked {ident['rod_marking']}, sensor at {ident['sensor_distance_m']:.2f} m, cage residual {measurements.cage_residual_nt} nT")
```

### Electrical

Coil resistance four-wire and insulation to the case at 500 V, before any current goes through the rod:

```python filename="phases/electrical.py"
from utils.recipe import INSULATION_V


def electrical(measurements, bench, log):
    """Coil resistance 4-wire and insulation coil-to-case at 500 V DC,
    before any current goes through the rod."""
    r = bench.coil_resistance_ohm()
    ir = bench.insulation_mohm(INSULATION_V)
    measurements.coil_resistance_ohm = r
    measurements.insulation_mohm = ir
    log.info(f"Coil {r:.3f} ohm, insulation {ir:.0f} Mohm at {INSULATION_V:.0f} V")
```

### Dipole Sweep

The zero-current field is read first, then the current is stepped and the field read at each step. The phase subtracts the zero, converts to dipole, fits a straight line, and derives the four judged numbers:

```python filename="phases/dipole_sweep.py"
import numpy as np

from utils.recipe import (CAGE_RESIDUAL_NT_MAX, MU0_OVER_2PI, POLARITY_AXIS, RATED_CURRENT_A,
                          RATED_DIPOLE_AM2, SENSOR_DISTANCE_M, SWEEP_CURRENT_A)


def dipole_sweep(measurements, bench, log):
    """Current stepped from -rated to +rated, the axial field read at
    each step and converted to a dipole through the on-axis law. The
    zero-current field is subtracted as the cage residual. Judged on the
    dipole at rated current, the straightness of the curve, the gain and
    the polarity against the rod's marking."""
    bench.set_current(0.0)
    b0 = bench.field_nt()
    field = []
    for amps in SWEEP_CURRENT_A:
        bench.set_current(amps)
        field.append(bench.field_nt())
    bench.set_current(0.0)
    b = np.array(field) - b0
    m = b * 1e-9 * SENSOR_DISTANCE_M ** 3 / MU0_OVER_2PI  # A·m²
    i = np.array(SWEEP_CURRENT_A)
    gain, offset = np.polyfit(i, m, 1)
    resid_pct = 100.0 * np.abs(m - (gain * i + offset)).max() / RATED_DIPOLE_AM2
    at_rated = float(m[np.argmin(np.abs(i - RATED_CURRENT_A))])
    polarity = "+X" if at_rated > 0 else "-X"

    measurements.sweep.x_axis = SWEEP_CURRENT_A
    measurements.sweep.y_axis.dipole = m.round(4).tolist()
    measurements.sweep.y_axis.dipole.aggregations.at_rated_am2 = at_rated
    measurements.sweep.y_axis.dipole.aggregations.gain_am2_per_a = float(gain)
    measurements.sweep.y_axis.dipole.aggregations.linearity_pct = float(resid_pct)
    measurements.polarity = polarity
    log.info(f"Dipole {at_rated:.3f} A·m² at {RATED_CURRENT_A * 1000:.0f} mA, gain {gain:.2f} A·m²/A, worst deviation from straight {resid_pct:.2f} % of rated, polarity {polarity} (marking {POLARITY_AXIS})")
```

### Residual

The rod is driven to +150 mA, the current is cut, and the field is read against a fresh cage zero. What is left is the core's remanence:

```python filename="phases/residual.py"
from utils.recipe import MU0_OVER_2PI, RATED_CURRENT_A, SENSOR_DISTANCE_M


def residual(measurements, bench, log):
    """Remanent dipole: the rod driven to +rated, the current cut, the
    field read with the cage still nulling. What the core keeps is a
    permanent disturbance torque the attitude model has to carry. The
    zero reference is the cage residual read again here, not the setup
    value: phases cannot read each other's measurements back."""
    bench.set_current(RATED_CURRENT_A)
    bench.set_current(0.0)
    b = bench.field_nt() - bench.cage_null()
    m = b * 1e-9 * SENSOR_DISTANCE_M ** 3 / MU0_OVER_2PI
    measurements.residual_am2 = float(abs(m))
    log.info(f"Residual dipole {abs(m) * 1000:.1f} mA·m² after +{RATED_CURRENT_A * 1000:.0f} mA")
```

### Mock Plug

`TorquerBench` stands in for the cage, the fluxgate, the source and the two meters. It draws a rod whose moment follows a hyperbolic tangent of the current with a small-signal gain near 7 A·m²/A and a knee well above the rated current, a core that keeps 0.6 % of the rated moment after a rated pulse, a 33.2 Ω coil, 2.4 GΩ of insulation, a cage residual of 12 nT and 1.5 nT of sensor noise. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real station, the class reads the fluxgate through the DAQ, drives the cage's three supplies to null the field, speaks SCPI to the Keithley source and the Hioki meter and drives the insulation tester through the switch box. Set `SENSOR_DISTANCE_M` to the distance measured on the fixture, set `CAGE_RESIDUAL_NT_MAX` to what the facility achieves, and take the rod's limits from its data sheet. The phases, measurements and limits stay the same.

### PCDU Thermal Cycle and Burn-In
URL: https://www.tofupilot.com/templates/pcdu-thermal-cycle-and-burn-in
PCDU acceptance with TofuPilot Framework: FFT, four ECSS thermal cycles with RFT at every dwell, 200 h burn-in with a failure-free clock, RFT deltas.

## Introduction

### Electronic Unit Acceptance Environment Overview

Every electronic unit that goes onto a satellite is accepted the same way: it is tested at ambient, cycled through its acceptance temperatures while it operates, burned in powered for a fixed number of hours, and tested again at ambient to see whether the environment changed it. The power conditioning and distribution unit (PCDU) is the one unit everything else depends on, so it is the example here: a 28 V unregulated bus in, 3.3 V, 5 V and 12 V regulated rails out, 20 W of nominal load. The environment does two things. Thermal cycling finds the workmanship defects that show up as a solder joint or a connector that opens at temperature. Burn-in finds the infant mortalities that show up as a unit that resets itself after forty hours of running.

![Close-up of a PCDU board in its open aluminium frame, one bus input connector and three rail output connectors](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-e3d01e95-65d4-4495-af40-f50b306f7eeb-qQoYkYewnVqcdUHOiwjp9gpTQ02SFG.jpg)

The unit under test in its open frame: one bus input connector, three rail outputs, and the converters between them. The acceptance test reads the rails at the outputs and the telemetry the unit reports about itself.

The sequence is written down in the standards, and the numbers differ between them. ECSS-E-ST-10-03C Rev.1 asks for four acceptance thermal cycles on an electronic unit, a full functional test at ambient before and after, and a reduced functional test at the hot and cold dwells (see [ECSS-E-ST-10-03C Rev.1, Testing](https://ecss.nl/standard/ecss-e-st-10-03c-rev-1-testing-31-may-2022/)). SMC-S-016 asks for a 200 h powered burn-in with the last 100 h failure-free. NASA's GEVS asks for 1000 h at component level with 350 h at the hot plateau (see [GSFC-STD-7000A](https://standards.nasa.gov/standard/GSFC/GSFC-STD-7000)). The "300 h" figure that circulates is not in any of them. The durations and the cycle count are therefore recipe constants in this template, and the rail regulation and efficiency limits are the unit's own data sheet, not a standard's.

### Test Purpose

The procedure records one acceptance environment record per unit:

- **Full functional test at ambient before**: every rail across five load steps, load regulation per rail, efficiency at nominal load
- **Four acceptance thermal cycles** with the unit powered, chamber and board temperature logged, and the rails read at every hot and cold dwell
- **200 h powered burn-in** at the hot plateau, with the input current drift, the board temperature, and a failure-free clock that restarts at every reset the unit logs
- **Reduced functional test at ambient after**, each rail and the efficiency compared to the reading taken before the environment

![Chamber temperature in green cycling four times between -25 and +65 °C over 20 hours, the board temperature as an orange dotted line 7 °C above it, red dashed lines at the two acceptance levels, and blue dots at the end of each dwell where the reduced functional test was run.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-91c597c0-2be1-49f1-b937-04664d95c30c-YyqTNyLEWtMYWHGcCeIcrW4yWp9aiw.png)

The mock unit through its four cycles: the chamber between -25 and +65 °C, the board 7 °C above it under 20 W of load, and a reduced functional test at the end of each of the eight dwells with the worst rail at 1.13 % from nominal. The dwells are where a marginal joint opens; the RFT at each one is what catches it.

The framework mechanics on show are one Python function shared by two phases that write to different measurements, three curves with the same aggregation and limit each, a count validated with `==`, a failure-free window derived from an event counter curve, a finding with a ceiling that is not a failure, before-versus-after deltas next to absolute limits, a reference passed between phases through the plug, and progress components on two time-scaled long phases.

## Equipment & Setup

To run the acceptance environment on a PCDU, the following are required:

- **A thermal chamber** that covers the acceptance range with margin and reports its temperature over an interface
- **A bus source** at 28 V with current readback, and **one electronic load channel per rail**
- **A DMM on each rail's sense pair**, or the unit's own rail telemetry if it is accurate enough
- **The Device Under Test (DUT)**: a smallsat PCDU, 28 V in, 3.3 / 5 / 12 V out, 20 W nominal
- **A TofuPilot Framework procedure** to sequence the FFT, the cycles, the burn-in and the RFT
- **The TofuPilot Dashboard** to keep the before-and-after record per serial and trend the deltas across the lot

### Hardware Components

#### Thermal Chamber

Any bench chamber that reaches the acceptance levels with a few degrees of margin and holds a dwell to within a degree does the job. Vötsch and ESPEC chambers expose set point and readback over Ethernet or Modbus. The rate of change is what the chamber can do, not a requirement; the recipe's 3 °C/min is a typical bench chamber. The unit is powered at nominal load throughout the cycles, which is why the board runs above the chamber and why the board temperature is its own curve with its own limit from the derating analysis.

![Open thermal chamber holding blue power distribution boards](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/pcdu-thermal-cycle-and-burn-in-zHFDGonEEfkWfKlj7MMIXjF84j0isL.webp)

The illustration shows a chamber with supported board fixtures and wiring routed through its wall. It is open for loading; the bus source and loads are external.

#### Bus Source and Loads

A Keysight N6705-class source on the bus gives the input current readback the efficiency and the burn-in drift come from. One electronic load channel per rail steps each rail through 0, 25, 50, 75 and 100 % of its share of the 20 W. The rails are read four-wire at the output connector; a unit that reports its own rail voltages over telemetry can be used instead when the telemetry has been calibrated against the DMM.

#### Unit Telemetry

The PCDU reports rail voltages, board temperature and a reset counter over its serial link. The reset counter is the burn-in's most important channel: a watchdog reset during the burn-in is the infant mortality the test exists to find, and the procedure does not need to catch it live because the unit logs it. The failure-free window is computed from the counter curve at the end.

### Where the Limits Come From

The cycle count, four, and the structure of the sequence are ECSS-E-ST-10-03C. The acceptance levels are the operating range, -20 to +60 °C, with a 5 °C margin. The burn-in duration of 200 h with the last 100 h failure-free is SMC-S-016; a GEVS programme sets `BURN_IN_H` to 1000. The rail regulation limit of 1 % at ambient and 3 % at the dwells, the 88 % efficiency floor, the 75 °C board limit and the 2 % current drift are the unit's data sheet and its derating analysis. The 0.5 % before-versus-after delta on the rails is the limit a customer tightens after a few dozen units, because a unit that shifted is a unit with a story even when it is inside its limits.

## Test Procedure

### Overview

The procedure maps the acceptance environment onto the framework's three stages. Identification and the chamber check live in `setup:` so the bus is never switched on into an unknown chamber. Load off, bus off and chamber back at ambient live in `teardown:` so the unit is safe to unload whatever happened.

1. Setup: firmware, reset counter at zero, chamber at ambient, bus on.
2. Main: full functional test at ambient, the reference stored on the plug.
3. Main: four thermal cycles powered, chamber and board logged, rails read at every dwell.
4. Main: 200 h burn-in at the hot plateau, input current, board temperature and reset counter logged.
5. Main: reduced functional test at ambient, deltas to the reference.
6. Teardown: load off, bus off, chamber at ambient, reset counter read.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot for per-serial trending across the lot.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  functional.py
  fft_ambient.py
  thermal_cycling.py
  burn_in.py
  rft_ambient.py
  safe_state.py
plugs/
  thermal_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-pcdu-thermal-cycle-burn-in). The `ThermalBench` plug is a mock of the chamber, the bus source, the loads and the unit's telemetry together, synthesizing a healthy unit whose rails sit within 0.6 % of nominal with a small temperature coefficient, 90 % efficiency at nominal load, and one watchdog reset at hour 41 of the burn-in, so the procedure runs end-to-end without a chamber or a unit connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: PCDU Thermal Cycle and Burn-In
version: 0.1.0
description: Acceptance environment for a smallsat PCDU. Full functional test at ambient, four ECSS acceptance thermal cycles with a reduced functional test at every hot and cold dwell, 200 h powered burn-in with the last 100 h failure-free, then the reduced functional test at ambient compared rail by rail to the reference taken before.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the PCDU label"
    placeholder: "PCDU-28-0000"
    pattern: "^PCDU-28-\\d{4}$"
    default_value: "PCDU-28-0117"
  part_number:
    default_value: "PCDU-28-3R"
  batch_number:
    default_value: "LOT-2026-09"

plugs:
  - name: Thermal Bench
    description: "Thermal chamber, 28 V bus source, electronic load per rail, unit telemetry over serial (mock, one plug per bench)"
    python: plugs.thermal_bench:ThermalBench
    key: bench

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Firmware Version
        key: firmware_version
        validators:
          - {operator: matches, expected_value: "^2\\.\\d+\\.\\d+$"}
      - name: Reset Count Start
        key: reset_count_start
        validators:
          - {operator: "==", expected_value: 0}
      - name: Chamber Start
        key: chamber_start_c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 18.0}
          - {operator: "<=", expected_value: 28.0}

main:
  - name: FFT Ambient
    key: fft_ambient
    python: phases.fft_ambient
    measurements:
      - name: Rails Before
        key: rails_before
        title: Rail voltage across the load steps, ambient, before the environment
        x_axis:
          legend: Load
          unit: "%"
        y_axis:
          - legend: 3.3 V
            key: rail_3v3
            unit: V
            aggregations:
              - type: regulation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 1.0}
          - legend: 5 V
            key: rail_5v0
            unit: V
            aggregations:
              - type: regulation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 1.0}
          - legend: 12 V
            key: rail_12v
            unit: V
            aggregations:
              - type: regulation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 1.0}
      - name: Efficiency Before
        key: efficiency_before_pct
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 88.0}

  - name: Thermal Cycling
    key: thermal_cycling
    python: phases.thermal_cycling
    depends_on: [fft_ambient]
    timeout: 24h
    ui:
      components:
        - key: cycling_progress
          type: progress
          label: "Thermal cycling"
          description: "4 cycles, -25 to +65 °C, 2 h dwells, powered"
          default_value: 0
          max: 100
    measurements:
      - name: Profile
        key: profile
        title: Chamber and board temperature through the four cycles
        x_axis:
          legend: Time
          unit: h
        y_axis:
          - legend: Chamber
            key: chamber
            unit: °C
            aggregations:
              - type: cycles
                validators:
                  - {operator: "==", expected_value: 4}
              - type: min_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: -24.0}
              - type: max_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: 64.0}
                  - {operator: "<=", expected_value: 68.0}
          - legend: Board
            key: board
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 75.0}
      - name: Rails At Dwells
        key: rails_at_dwells
        title: Reduced functional test at each hot and cold dwell
        x_axis:
          legend: Dwell
        y_axis:
          - legend: 3.3 V
            key: rail_3v3
            unit: V
            aggregations:
              - type: max_deviation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 3.0}
          - legend: 5 V
            key: rail_5v0
            unit: V
            aggregations:
              - type: max_deviation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 3.0}
          - legend: 12 V
            key: rail_12v
            unit: V
            aggregations:
              - type: max_deviation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 3.0}

  - name: Burn In
    key: burn_in
    python: phases.burn_in
    depends_on: [thermal_cycling]
    timeout: 24h
    ui:
      components:
        - key: burn_in_progress
          type: progress
          label: "Burn-in"
          description: "200 h powered at 50 °C, last 100 h failure-free"
          default_value: 0
          max: 100
    measurements:
      - name: Burn-In
        key: burn_in
        title: Input current, board temperature and reset counter over the burn-in
        x_axis:
          legend: Time
          unit: h
        y_axis:
          - legend: Input current
            key: current
            unit: A
            aggregations:
              - type: hours
                unit: h
                validators:
                  - {operator: ">=", expected_value: 200.0}
              - type: drift_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: -2.0}
                  - {operator: "<=", expected_value: 2.0}
          - legend: Board
            key: board
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 75.0}
          - legend: Resets
            key: resets
            aggregations:
              - type: failure_free_h
                unit: h
                validators:
                  - {operator: ">=", expected_value: 100.0}
      - name: Reset Events
        key: reset_events
        description: Resets logged by the unit during the burn-in. One is a finding to be dispositioned, not a failure, as long as the last 100 h are clean; two restart the burn-in.
        validators:
          - {operator: "<=", expected_value: 1}

  - name: RFT Ambient
    key: rft_ambient
    python: phases.rft_ambient
    depends_on: [burn_in]
    measurements:
      - name: Rails After
        key: rails_after
        title: Rail voltage across the load steps, ambient, after the environment
        x_axis:
          legend: Load
          unit: "%"
        y_axis:
          - legend: 3.3 V
            key: rail_3v3
            unit: V
            aggregations:
              - type: regulation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 1.0}
          - legend: 5 V
            key: rail_5v0
            unit: V
            aggregations:
              - type: regulation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 1.0}
          - legend: 12 V
            key: rail_12v
            unit: V
            aggregations:
              - type: regulation_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 1.0}
      - name: Efficiency After
        key: efficiency_after_pct
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 88.0}
      - name: Rail Delta Max
        key: rail_delta_max_pct
        unit: "%"
        description: Worst rail shift between the FFT before and the RFT after, any load step; a unit inside its limits but shifted has something to explain.
        validators:
          - {operator: "<=", expected_value: 0.5}
      - name: Efficiency Delta
        key: efficiency_delta_pct
        unit: "%"
        validators:
          - {operator: ">=", expected_value: -1.0}
          - {operator: "<=", expected_value: 1.0}

teardown:
  - name: Safe State
    key: safe_state
    python: phases.safe_state
    measurements:
      - name: Chamber End
        key: chamber_end_c
        unit: °C
        validators:
          - {operator: "<=", expected_value: 30.0}
      - name: Reset Count End
        key: reset_count_end
```

Framework features to notice:

1. **One function, two phases.** `phases/functional.py` reads the rails across the load steps and computes the regulation per rail; `fft_ambient` and `rft_ambient` both call it, each with its own measurement target, so the before and after are the same code by construction.
2. **A count validated with `==`.** `profile.chamber.cycles == 4` is the ECSS requirement as a number; a chamber that aborted after three cycles fails here, not on a temperature.
3. **A failure-free window from an event curve.** The `resets` curve is the unit's own counter sampled over the burn-in; `failure_free_h` is the time from the last increment to the end, and the limit is the SMC-S-016 100 h.
4. **A finding with a ceiling.** `reset_events <= 1` lets one reset through as something to disposition while the failure-free clock decides the outcome; a second reset fails the unit.
5. **Deltas next to absolutes.** `rails_after` has the same 1 % regulation limit as `rails_before`, and `rail_delta_max_pct <= 0.5` sits beside it; the unit has to meet its limits and behave like it did.
6. **A reference through the plug.** Phases run in separate processes and cannot read each other's measurements back, so `fft_ambient` stores its readings on the bench plug and `rft_ambient` recalls them.
7. **Phase timeouts cap at 24 h.** The 200 h burn-in is time-scaled in the mock; on real hardware it runs as daily segments of this phase or is read back from the chamber's logger, with the same aggregations either way.

### Identify

The setup phase reads the firmware and the reset counter, which has to be zero before the burn-in means anything, checks the chamber is at ambient, records the chamber on the **unit metadata** and switches the bus on:

```python filename="phases/identify.py"
def identify(measurements, bench, unit, log):
    """Setup: unit telemetry, firmware, reset counter at zero, chamber at
    ambient before the bus is switched on."""
    ident = bench.identify()
    measurements.firmware_version = ident["firmware"]
    measurements.reset_count_start = ident["reset_count"]
    chamber = bench.chamber_temp_c()
    measurements.chamber_start_c = chamber
    unit.metadata["chamber_id"] = "TC-02"
    bench.bus_on()
    log.info(f"PCDU {unit.serial_number}: fw {ident['firmware']}, {ident['reset_count']} resets logged, chamber at {chamber:.1f} C, bus on")
```

### FFT Ambient

The full functional test calls the shared function with the `rails_before` measurement as its target and stores the readings on the plug for the RFT to compare against:

```python filename="phases/fft_ambient.py"
from phases.functional import functional_test


def fft_ambient(measurements, bench, log):
    """Full functional test at ambient before the environment: every
    rail across the load steps, load regulation per rail, efficiency at
    nominal load. The reference the post-test RFT is compared against,
    kept on the bench plug because phases run in separate processes and
    cannot read each other's measurements back."""
    readings, eff = functional_test(measurements, bench, log, measurements.rails_before, "efficiency_before_pct")
    bench.store("fft", {"rails": readings, "efficiency_pct": eff})
```

The shared function steps the load, reads every rail at every step, writes each rail as a curve with a `regulation_pct` aggregation, and computes the efficiency at nominal load from the source's readback:

```python filename="phases/functional.py"
import numpy as np

from utils.recipe import LOAD_STEPS_PCT, LOAD_W_NOMINAL, RAILS_V


def functional_test(measurements, bench, log, rails_meas, efficiency_key):
    """Shared by the FFT and the RFT: rails across the load steps, load
    regulation aggregation per rail, efficiency at nominal load."""
    readings = {k: [] for k in RAILS_V}
    eff = None
    for pct in LOAD_STEPS_PCT:
        bench.set_load_pct(pct)
        rails = bench.read_rails()
        for k in RAILS_V:
            readings[k].append(rails[k])
        if pct == 100:
            inp = bench.read_input()
            eff = 100.0 * LOAD_W_NOMINAL / (inp["bus_v"] * inp["current_a"])
    bench.set_load_pct(0)

    rails_meas.x_axis = LOAD_STEPS_PCT
    reg = {}
    for k, nom in RAILS_V.items():
        v = np.array(readings[k])
        setattr(rails_meas.y_axis, k, v.tolist())
        reg[k] = float(100.0 * np.abs(v - nom).max() / nom)
        getattr(rails_meas.y_axis, k).aggregations.regulation_pct = reg[k]
    setattr(measurements, efficiency_key, float(eff))
    log.info("Rails " + ", ".join(f"{k} {reg[k]:.2f} %" for k in RAILS_V) + f", efficiency {eff:.1f} % at {LOAD_W_NOMINAL:.0f} W")
    return readings, eff
```

### Thermal Cycling

One call returns the four cycles as a chamber and a board curve plus the rails read at each dwell. The phase counts the cycles as the smaller of the hot and cold dwells reached, records the profile with its aggregations, and writes the reduced functional test as one curve per rail across the eight dwells:

```python filename="phases/thermal_cycling.py"
import numpy as np

from utils.recipe import CYCLES, RAILS_V, T_ACC_MAX_C, T_ACC_MIN_C


def thermal_cycling(measurements, bench, ui, log):
    """ECSS-E-ST-10-03C acceptance thermal cycling: four cycles between
    the acceptance temperatures with the unit powered at nominal load,
    chamber and board temperature logged, and a reduced functional test
    (all rails) at every hot and cold dwell."""
    bench.set_load_pct(100)
    cap = bench.thermal_cycles()
    ui.cycling_progress = 100
    chamber = np.array(cap["chamber_c"])
    board = np.array(cap["board_c"])
    # A cycle is one hot dwell and one cold dwell reached.
    hot = int(sum(1 for d in cap["dwell_temp_c"] if d >= T_ACC_MAX_C - 1.0))
    cold = int(sum(1 for d in cap["dwell_temp_c"] if d <= T_ACC_MIN_C + 1.0))
    cycles = min(hot, cold)

    measurements.profile.x_axis = [round(m / 60.0, 2) for m in cap["time_min"]]
    measurements.profile.y_axis.chamber = cap["chamber_c"]
    measurements.profile.y_axis.chamber.aggregations.cycles = cycles
    measurements.profile.y_axis.chamber.aggregations.min_c = float(chamber.min())
    measurements.profile.y_axis.chamber.aggregations.max_c = float(chamber.max())
    measurements.profile.y_axis.board = cap["board_c"]
    measurements.profile.y_axis.board.aggregations.max_c = float(board.max())

    # Reduced functional test at each dwell: the worst rail deviation.
    measurements.rails_at_dwells.x_axis = list(range(1, len(cap["dwell_rails"]) + 1))
    worst = {}
    for k, nom in RAILS_V.items():
        v = np.array([d[k] for d in cap["dwell_rails"]])
        setattr(measurements.rails_at_dwells.y_axis, k, v.tolist())
        worst[k] = float(100.0 * np.abs(v - nom).max() / nom)
        getattr(measurements.rails_at_dwells.y_axis, k).aggregations.max_deviation_pct = worst[k]
    bench.set_load_pct(0)
    log.info(f"{cycles}/{CYCLES} cycles, chamber {chamber.min():.1f}..{chamber.max():.1f} C, board to {board.max():.1f} C, worst rail at a dwell " + ", ".join(f"{k} {worst[k]:.2f} %" for k in RAILS_V))
```

### Burn In

The chamber goes to the hot plateau, the unit runs at nominal load for 200 h, and the phase reads back the input current, the board temperature and the reset counter. The failure-free time is the time from the last reset to the end; the current drift is the last five hours against the first five:

```python filename="phases/burn_in.py"
import numpy as np

from utils.recipe import BURN_IN_H, BURN_IN_T_C


def burn_in(measurements, bench, ui, log):
    """SMC-S-016 acceptance burn-in: powered at nominal load at the hot
    plateau for 200 h, the last 100 h failure-free. Any reset logged by
    the unit restarts the failure-free clock; the aggregation is the
    clean time at the end, not the total."""
    bench.chamber_set(BURN_IN_T_C)
    bench.set_load_pct(100)
    cap = bench.burn_in(BURN_IN_H)
    ui.burn_in_progress = 100
    t = np.array(cap["time_h"])
    cur = np.array(cap["current_a"])
    resets = np.array(cap["reset_count"])
    events = np.flatnonzero(np.diff(resets) > 0)
    last_event_h = float(t[events[-1] + 1]) if events.size else 0.0
    failure_free_h = float(t[-1] - last_event_h)
    drift_pct = float(100.0 * (cur[-50:].mean() - cur[:50].mean()) / cur[:50].mean())

    measurements.burn_in.x_axis = cap["time_h"]
    measurements.burn_in.y_axis.current = cap["current_a"]
    measurements.burn_in.y_axis.current.aggregations.hours = float(t[-1])
    measurements.burn_in.y_axis.current.aggregations.drift_pct = drift_pct
    measurements.burn_in.y_axis.board = cap["board_c"]
    measurements.burn_in.y_axis.board.aggregations.max_c = float(max(cap["board_c"]))
    measurements.burn_in.y_axis.resets = cap["reset_count"]
    measurements.burn_in.y_axis.resets.aggregations.failure_free_h = failure_free_h
    measurements.reset_events = int(events.size)
    bench.set_load_pct(0)
    bench.chamber_set(23.0)
    log.info(f"Burn-in {t[-1]:.0f} h at {BURN_IN_T_C:.0f} C: {events.size} reset(s), last at {last_event_h:.1f} h, {failure_free_h:.0f} h failure-free, input current drift {drift_pct:+.2f} %")
```

![Bus input current in green around 797 to 800 mA over 200 hours of burn-in, a red dashed vertical line at 41 hours where the watchdog reset was logged, and the 159 hours after it shaded green as the failure-free window.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-83beb8de-7893-4591-b8e5-72fc21242e64-I7a8PYvHF1AdPFEeqpVsSI2UGd8Fb6.png)

The mock unit's burn-in: one watchdog reset at hour 41, 159 h failure-free after it against the 100 h limit, and an input current that drifts 0.53 % over the 200 h as the converters settle. A reset at hour 150 would leave 50 h clean and fail the unit; a second reset anywhere fails it on the count.

### RFT Ambient

The reduced functional test calls the same function with `rails_after` as its target, recalls the FFT readings from the plug, and computes the worst rail shift and the efficiency delta:

```python filename="phases/rft_ambient.py"
import numpy as np

from phases.functional import functional_test
from utils.recipe import RAILS_V


def rft_ambient(measurements, bench, log):
    """Reduced functional test at ambient after the environment: the same
    rails and efficiency, each compared to the FFT reading before the
    environment. The delta is the number that catches a unit that still
    meets its limits but no longer behaves like it did."""
    after, eff_after = functional_test(measurements, bench, log, measurements.rails_after, "efficiency_after_pct")
    ref = bench.recall("fft")
    deltas = {k: float(100.0 * np.abs(np.array(after[k]) - np.array(ref["rails"][k])).max() / RAILS_V[k]) for k in RAILS_V}
    measurements.rail_delta_max_pct = max(deltas.values())
    measurements.efficiency_delta_pct = float(eff_after - ref["efficiency_pct"])
    log.info("Before vs after: " + ", ".join(f"{k} {deltas[k]:.3f} %" for k in RAILS_V) + f", efficiency {ref['efficiency_pct']:.1f} -> {eff_after:.1f} %")
```

![Three panels, one per rail, showing the deviation from nominal in percent across the five load steps, the FFT before in gray and the RFT after in green almost on top of each other between 0 and -0.6 %, red dashed lines at plus and minus 1 %.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template-828d2c7b-9386-4269-b716-86652db5b4a3-wAIlNlSOpikuklt762i0mKEl46a8Dx.png)

Before and after on the mock unit: the three rails droop 0.4 to 0.6 % from no load to full load both times, the worst shift between the two is 0.049 % on the 12 V rail against the 0.5 % limit, and the efficiency moved from 89.7 to 89.9 %. A rail that came back 0.3 % different is inside every absolute limit and is still the trace of a component that changed in the chamber.

### Mock Plug

`ThermalBench` stands in for the chamber, the bus source, the loads and the unit's telemetry. It draws three rails with a fixed offset of a few tenths of a percent, a temperature coefficient of about -0.012 %/°C and 0.4 % of load regulation, 91 % converter efficiency plus 0.35 W of housekeeping, a board 7 °C above the chamber at full load, four cycles ramped at 3 °C/min with 2 h dwells and the rails read at each one, and a burn-in whose input current settles 0.6 % over the first days with one watchdog reset at hour 41. The cycles and the burn-in are time-scaled and return in one call each. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines, and the FFT reference crosses phases through the plug's `store` and `recall`.

On a real station, the class speaks the chamber's protocol for set point and readback, SCPI to the bus source and the loads over pyvisa, and the unit's protocol over pyserial for the rail telemetry and the reset counter. Set `TIME_SCALE = 1.0` in `utils/recipe.py` for the real cycles and burn-in, split the burn-in into daily segments or read it back from the chamber's logger because a phase timeout caps at 24 h, and set `BURN_IN_H` and `FAILURE_FREE_H` to the programme's standard. The phases, measurements and limits stay the same.

### Switch Port PRBS BER and Snake Test
URL: https://www.tofupilot.com/templates/switch-port-prbs-ber-and-snake-test
Accept a 32-port 400G switch: PRBS31 BER on all 256 lanes against the 802.3 budget, worst eye per port, a line-rate snake through every port, FEC and CMIS.

## Introduction

### Switch Port Acceptance Overview

A 32-port 400G top-of-rack switch carries 256 SerDes lanes on its faceplate, eight per QSFP-DD cage, each running 53.125 Gb/s of PAM4 at 26.5625 GBd. The ASIC, a Tomahawk 3 class device, does not see clean bits on those lanes: it sees a pre-FEC bit error ratio that the RS(544,514) forward error correction of IEEE 802.3 Clause 119 turns into a frame loss ratio, provided the errors are few enough and random enough. A port that ships with one lane at the edge of that budget links up in the customer's rack, passes every ping, and drops one frame in a billion under load six months later when the optic ages. The production test that catches it is not a link-up test: it is a bit error ratio measured per lane, an eye margin read from the SerDes, and a snake that pushes line-rate frames through every port at once.

![QSFP-DD electrical loopback module with a blue pull tab, closed front cap, and rear gold edge contacts.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/switch-port-prbs-ber-and-snake-test-matte-closeup-6ZOv4T0yZA1TF3XbmjyAoJfPu8GVjR.webp)

The parts this test exercises: the 256 SerDes transmitters and receivers, the PCB traces and QSFP-DD connectors between them and the cages, the FEC and MAC behind each port, the forwarding pipeline that moves frames port to port, the power stage feeding 32 modules, the fan trays that keep them cool, and the SONiC image and SAI package the whole thing boots on.

IEEE 802.3 gives the production test writer two numbers. The **pre-FEC BER a 400GBASE-R link may run at is 2.4e-4**, the point at which RS(544,514) still delivers a **frame loss ratio below 1.7e-12** for 64-octet frames (Clause 119 and the PMD clauses that cite it). The **electrical chip-to-module interface, 400GAUI-8, is budgeted at a symbol error ratio of 1e-5** (Annex 120E), two decades under the FEC limit, because the optical link is supposed to spend the rest. A 0 dB loopback plug in the cage adds nothing the host SerDes has to equalise, so a port tested through one should meet the AUI budget, not merely the FEC limit; Broadcom's own hardware design guideline for this ASIC family asks for a simulated pre-FEC BER better than 1e-6 as a guard band. Every limit in this template is one of those numbers, or says where it came from instead.

### Test Purpose

The procedure records one acceptance fingerprint per switch:

- **Serial, platform, ASIC, SONiC build and SAI package** as one object against the release, and every one of the 32 cages populated with the station's loopback module
- **PRBS31 on all 256 lanes for 60 s** with linkscan off, each port judged on its worst lane and on its count of errored lanes, the raw per-lane shell dump attached
- **The worst PAM4 eye per port**, height and width, from the SerDes dump
- **A line-rate snake** through every port the PRBS phase cleared, the frames it lost, its receive rate, and the **FEC corrected and uncorrected codewords per port** afterwards
- **Loopback module temperature and Vcc over CMIS** at the end of the snake, with the modules dissipating what a 400G optic would
- **PRBS off, linkscan on, the golden configuration reloaded and its checksum checked**, every port back up, in teardown

![Worst-lane pre-FEC BER per port on a log axis: 31 green bars between 2e-7 and 8e-7, port 17 orange at 3.0e-6, under the orange 1e-5 production line and the red 2.4e-4 FEC limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/switch-port-ber-chart-ber-sCRiA2Q7QQnnhxK2GTfNsp7Tw41prr.png)

The mock switch's worst lane on each port after 60 s of PRBS31: thirty-one ports between 2e-7 and 8e-7, and port 17 at 3.0e-6 on one lane, an order of magnitude above its neighbours and still three times under the production limit. That is the port the dashboard should show trending across the lot.

The framework mechanics on show are previous-results injection, with the snake phase receiving the PRBS phase's result object and leaving errored ports out of the chain; `attach.file` for the raw diag-shell dump next to the summarised measurement; `then: {fail: retry}` on the PRBS phase so a lane that lost lock once is re-run before the port is condemned; a plug at `scope: station` for the traffic generator next to a per-unit plug for the switch; `run.metadata` for what the station did and `unit.metadata` for what the unit is; and four multi-dimensional measurements indexed by port, each with two curves and their own aggregations.

## Equipment & Setup

To run this acceptance on a production line, the following are required:

- **32 QSFP-DD loopback modules** with programmable power dissipation and CMIS monitors, one per cage, left in place for the whole test
- **A 400GE traffic generator** with one QSFP-DD port on a direct-attach cable into port 1, sending at line rate and counting what returns
- **A management connection** to the switch: SSH into SONiC, and `bcmcmd` into the Broadcom diagnostic shell behind it
- **The Device Under Test (DUT)**: a 32-port 400G QSFP-DD switch on a Tomahawk 3 class ASIC, booted on the release SONiC image, with a golden `config_db.json` on the station
- **A TofuPilot Framework procedure** to sequence the steps and validate the limits
- **The TofuPilot Dashboard** to keep the per-port evidence and watch the worst lane of each port across the lot

### Hardware Components

#### Loopback Modules and Traffic Generator

The [MultiLane ML4062-LB-112](https://www.multilaneinc.com/product.php?pn=ML4062-LB-112) is a QSFP-DD loopback that returns each lane's transmitter to its own receiver through a short trace, with five heaters that emulate any QSFP-DD power class up to 16 W, a temperature monitor and Vcc sense behind a CMIS memory map, and a cut-off that shuts the heaters at 85 °C. Thirty-two of them in the faceplate turn the PRBS test into a test of the host SerDes and the PCB alone, and the snake into a thermal test of the chassis under the load a fully populated switch sees. On port 1, a direct-attach cable to a [Keysight AresONE 400GE](https://www.keysight.com/us/en/products/ethernet-traffic-emulation/hardware-accelerators-for-protocol-and-load-test/aresone.html) port sends 1518 byte frames at 99.99 % of line rate and reports frames sent, received and the receive rate; the 0.01 % left on the table is the room a ±100 ppm clock offset between the generator and the switch needs so that no hop drops a frame on its own.

![Isometric illustration of a 1U network switch with QSFP-DD electrical loopback modules installed in two rows of ports.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/switch-port-prbs-ber-and-snake-test-matte-cover-s3O994LfR0mfeg4Ta5t0IVllPWzMJr.webp)

The illustrated hardware: a 1U switch populated with electrical loopback modules. The traffic generator and management computer used by the procedure are omitted.

#### The Switch and Its Diagnostic Shell

The [Broadcom BCM56980 Tomahawk 3](https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56980-series) carries 256 Blackhawk SerDes lanes at 50G PAM4 that form 32 ports of 400GbE, each port eight lanes at 53.125 Gb/s with RS(544,514) FEC. SONiC drives it through the SAI, and the vendor's diagnostic shell, reached with `bcmcmd`, exposes the PRBS generator and checker on every lane (`phy diag cd0-cd31 prbs set p=3` for PRBS31, `prbs get` for the error counters), the SerDes dump with the eye margins (`dsc`), and `linkscan`, which must be turned off first: when the PRBS generator replaces data on the transmit lanes the PCS reports link-down, and a running linkscan re-initialises the SerDes under the checker. FEC counters come from `show interfaces counters fec`, the identity from `show platform syseeprom` and `show version`, and the CMIS monitors from the module EEPROM.

![Resolvable BER against PRBS accumulation time on log axes: the 95 percent confidence line from 1e-4 at 1 microsecond to 1e-13 at 1000 s, crossing the 2.4e-4, 1e-5 and 1e-12 lines, with the 60 s dwell marked at a floor of 9.4e-13.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/switch-port-ber-chart-dwell-K8kQ2hEnRulILOGAaptTJlvTrOZlXJ.png)

Why 60 s: a lane at 53.125 Gb/s accumulates 3.2e12 bits in a minute. With zero errors, that shows the lane is under 9.4e-13 with 95 % confidence, the grade of a post-FEC link, so the dwell measures every lane on the switch at its actual rate rather than merely checking it against the 1e-5 limit, which 6 µs would do. A port at 1e-6 collects three million errors in that time; the counters do not saturate.

### Where the Limits Come From

| Test | Limit | Basis |
|---|---|---|
| Identity, serial, presence | exact match | no tolerance applies |
| Worst-lane pre-FEC BER | ≤ 1e-5 (log10 ≤ −5.0) | IEEE 802.3 Annex 120E, the 400GAUI-8 C2M symbol error ratio budget; the FEC limit of 2.4e-4 (Cl. 119) is the line a port must never approach |
| Errored lanes | 0 | a lane that lost PRBS lock or crossed the budget |
| Eye height, eye width | ≥ 40 mV, ≥ 0.30 UI worst lane | this line's derivation from the known-good population of this board; the vendor's SerDes reports the smallest of the three PAM4 eyes |
| Ports in snake | 32 | every port the PRBS phase cleared |
| Frames lost | 0 | after every frame crossed 32 ports and 32 plugs |
| Throughput | ≥ 99.9 % of line rate | this line's derivation; the generator sends at 99.99 % |
| FEC corrected codewords | ≤ 1.3e9 per port | this line's derivation: the 1e-5 production BER over 300 s at 425 Gb/s, in 5440-bit codewords |
| FEC uncorrected codewords | 0 | any uncorrected codeword is a lost frame |
| Module temperature | ≤ 70 °C | this line's derivation: the 0 to 70 °C case-temperature class of the optics the loopback emulates, 15 °C under the module's own cut-off |
| Module Vcc | 3.135 to 3.465 V | QSFP-DD hardware specification, 3.3 V ± 5 % at the module connector |
| Config checksum, ports up | exact match, 32 | the golden configuration, every port back through its plug |

Two rows in that table deserve a note. The BER is recorded as log10 rather than as the ratio, because a dashboard trending 3e-6 against 2e-7 across a lot needs a linear axis to show the drift; the aggregation limit is −5.0, which is 1e-5. And the FEC ceiling is a translation, not a new limit: at 425 Gb/s a port sees 2.3e10 codewords in five minutes, and a lane at the 1e-5 production BER would have the decoder correct about 1.3e9 of them. A port under the BER limit in PRBS and over this ceiling in the snake is a port whose errors are bursty, which PRBS averages away and the FEC does not.

## Test Procedure

### Overview

The procedure maps the acceptance onto the framework's three stages. Identity and presence live in `setup:` so no lane is exercised on a switch booted on the wrong image or with an empty cage. The four main phases share one management session and one generator, and chain with `depends_on` so the snake cannot start while PRBS still owns the lanes. The restore lives in `teardown:` so PRBS is switched off and linkscan back on even when a main phase fails.

1. Setup: serial against the label, identity object, 32 cages with loopback modules.
2. Main: linkscan off, PRBS31 on 256 lanes, 60 s, BER per lane, worst lane and errored lanes per port, raw dump attached.
3. Main: eye height and width of the worst lane per port from the SerDes dump.
4. Main: PRBS off, snake through the ports PRBS cleared at 99.99 % of line rate for 5 min, frames lost, throughput, FEC counters per port.
5. Main: loopback module temperature and Vcc over CMIS.
6. Teardown: PRBS off, linkscan on, golden configuration reloaded, checksum and ports up.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-switch evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  prbs_ber.py
  eye_margin.py
  snake_test.py
  module_monitors.py
  restore_config.py
plugs/
  switch_dut.py
  traffic_generator.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-switch-port-ber). The `SwitchUnderTest` plug is a mock of the SONiC CLI and the diagnostic shell together, and `TrafficGenerator` a mock of the generator's API; between them they synthesize a healthy switch with 256 lanes between 1e-7 and 1e-6, one lane on port 17 at 3e-6, eyes that follow the BER, a FEC that corrects everything it sees and loopback modules warmest in the middle of the faceplate, so the procedure runs end-to-end without a switch or a generator connected. The 60 s dwell and the 5 min snake are time-scaled by 100 in the mock; the `timeout` on each phase is sized for the real durations.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the two plugs with the label serial and the management addresses as constructor arguments, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Switch Port PRBS BER and Snake Test
version: 0.1.0
description: Port acceptance of a 32-port 400G QSFP-DD top-of-rack switch on a Tomahawk 3 class ASIC running SONiC. Identity and loopback presence, PRBS31 on all 256 lanes for 60 s against the 802.3 budget with the raw dump attached, the worst eye per port, a line-rate snake through every port and plug with the PRBS results deciding which ports enter it, FEC counters, loopback module temperature and Vcc over CMIS, and a teardown that puts the switch back in its shipping configuration.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the serial label on the rear of the chassis"
    placeholder: "SW32D-00000000"
    pattern: "^SW32D-\\d{8}$"
    default_value: "SW32D-26380417"
  part_number:
    default_value: "TOR32X400G-R0"
  batch_number:
    default_value: "LOT-2026-38"

plugs:
  - name: Switch
    description: "The switch under test: SONiC CLI over SSH and the Broadcom diag shell for PRBS, dsc, FEC and CMIS (mock)"
    python: plugs.switch_dut:SwitchUnderTest
    key: switch
    config:
      serial_number: "SW32D-26380417"
      host: "192.168.10.41"
  - name: Traffic Generator
    description: "400GE traffic generator on a DAC into port 1 (mock, station scope: the session is held across units)"
    python: plugs.traffic_generator:TrafficGenerator
    key: tgen
    scope: station
    config:
      address: "192.168.10.9"
      port: 1

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Reported Serial
        key: reported_serial
        validators:
          - {operator: matches, expected_value: "^SW32D-\\d{8}$"}
      - name: Serial Matches Label
        key: serial_matches_label
        description: The serial in the syseeprom TLVs is the one on the chassis label.
        validators:
          - {operator: "==", expected_value: true}
      - name: Switch Identity
        key: switch_identity
        description: Platform, ASIC, SONiC build and SAI package as one object against the release.
        validators:
          - operator: "=="
            expected_value:
              platform: x86_64-tor32x400g-r0
              asic: BCM56980
              sonic: SONiC.202405.12
              sai: 10.1.28.0
      - name: Ports Present
        key: ports_present
        validators:
          - {operator: "==", expected_value: 32}
      - name: Loopback Plugs Detected
        key: loopback_plugs_detected
        description: Every cage reports a module, and every module's CMIS vendor PN is the station's loopback.
        validators:
          - {operator: "==", expected_value: true}

main:
  - name: PRBS BER
    key: prbs_ber
    python: phases.prbs_ber
    timeout: 5m
    retry:
      limit: 1
      delay: 2s
    then:
      fail: retry
    measurements:
      - name: Pre-FEC BER per Port
        key: ber_per_port
        title: Worst-lane pre-FEC BER and errored lanes per port, PRBS31 for 60 s
        x_axis:
          legend: Port
        y_axis:
          - legend: Worst lane BER (log10)
            key: worst_lane_log10
            unit: log10
            aggregations:
              - type: max_log10
                unit: log10
                validators:
                  - {operator: "<=", expected_value: -5.0}
          - legend: Errored lanes
            key: errored_lanes
            unit: lanes
            aggregations:
              - type: total
                unit: lanes
                validators:
                  - {operator: "==", expected_value: 0}

  - name: Eye Margin
    key: eye_margin
    python: phases.eye_margin
    depends_on: [prbs_ber]
    measurements:
      - name: Worst Eye per Port
        key: eye_per_port
        title: Worst-lane eye height and width per port from the SerDes dump
        x_axis:
          legend: Port
        y_axis:
          - legend: Eye height
            key: height_mv
            unit: mV
            aggregations:
              - type: min_mv
                unit: mV
                validators:
                  - {operator: ">=", expected_value: 40.0}
          - legend: Eye width
            key: width_ui
            unit: UI
            aggregations:
              - type: min_ui
                unit: UI
                validators:
                  - {operator: ">=", expected_value: 0.30}

  - name: Snake Test
    key: snake_test
    python: phases.snake_test
    depends_on: [eye_margin]
    timeout: 10m
    measurements:
      - name: Ports in Snake
        key: ports_in_snake
        description: "Ports the PRBS phase cleared, chained port to port through their loopback plugs. All 32, or the unit is not shippable."
        validators:
          - {operator: "==", expected_value: 32}
      - name: Frames Sent
        key: frames_sent
      - name: Frames Lost
        key: frames_lost
        description: Sent minus received at the generator after every frame crossed 32 ports and 32 plugs.
        validators:
          - {operator: "==", expected_value: 0}
      - name: Throughput
        key: throughput_pct
        unit: "%"
        description: "Receive rate at the generator as a fraction of 400G line rate. 99.9 % is this line's floor; the generator sends at 99.99 % to leave the +-100 ppm clock offset its room."
        validators:
          - {operator: ">=", expected_value: 99.9}
      - name: FEC Codewords per Port
        key: fec_per_port
        title: RS(544,514) corrected and uncorrected codewords per port over the 5 min snake
        x_axis:
          legend: Port
        y_axis:
          - legend: Corrected codewords
            key: corrected_cw
            unit: codewords
            aggregations:
              - type: max_cw
                unit: codewords
                validators:
                  - {operator: "<=", expected_value: 1300000000}
          - legend: Uncorrected codewords
            key: uncorrected_cw
            unit: codewords
            aggregations:
              - type: total
                unit: codewords
                validators:
                  - {operator: "==", expected_value: 0}

  - name: Module Monitors
    key: module_monitors
    python: phases.module_monitors
    depends_on: [snake_test]
    measurements:
      - name: Loopback Modules per Port
        key: modules_per_port
        title: Loopback module temperature and Vcc over CMIS at the end of the snake
        x_axis:
          legend: Port
        y_axis:
          - legend: Module temperature
            key: temp_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 70.0}
          - legend: Module Vcc
            key: vcc_v
            unit: V
            aggregations:
              - type: min_v
                unit: V
                validators:
                  - {operator: ">=", expected_value: 3.135}
              - type: max_v
                unit: V
                validators:
                  - {operator: "<=", expected_value: 3.465}

teardown:
  - name: Restore Config
    key: restore_config
    python: phases.restore_config
    measurements:
      - name: PRBS Disabled
        key: prbs_disabled
        validators:
          - {operator: "==", expected_value: true}
      - name: Linkscan Enabled
        key: linkscan_enabled
        validators:
          - {operator: "==", expected_value: true}
      - name: Config Checksum
        key: config_checksum
        description: SHA-256 of the running configuration after the golden config_db.json is reloaded.
        validators:
          - {operator: "==", expected_value: "sha256:9f3c1e7a4b2d0c8e5f6a7b8c9d0e1f2a3b4c5d6e7f8091a2b3c4d5e6f7a8b9c0"}
      - name: Ports Up
        key: ports_up
        validators:
          - {operator: "==", expected_value: 32}
```

Framework features to notice:

1. **Previous-results injection.** `snake_test` names `prbs_ber` as a parameter and receives the completed phase's measurements: it reads the `errored_lanes` curve, chains only the ports with none, and names the others in the log. `depends_on` guarantees the order; the snake reports on the ports that can carry traffic instead of failing on the one already known.
2. **`attach.file` next to the measurement.** The 256-line `prbs get` output is attached as `prbs_per_lane.txt`; the measurement keeps the worst lane and the errored count per port. The review gets both.
3. **`then: {fail: retry}` with `retry: {limit: 1, delay: 2s}`.** A PRBS checker that loses lock once at start reports a lane at 1e-3; the phase is re-run once before the port is condemned, and the second attempt is the one that counts.
4. **Two plug scopes.** The generator is `scope: station`, its session opened once and held across units; the switch plug is per unit, since every unit is a new SSH session.
5. **`run.metadata` and `unit.metadata`.** The PRBS polynomial, the dwell and the generator's identity describe what the station did on this run; the ASIC and the SONiC build describe what the unit is. Both stay queryable on the dashboard.
6. **Four measurements indexed by port.** `ber_per_port`, `eye_per_port`, `fec_per_port` and `modules_per_port` each carry two curves on one x-axis of 32 ports, with a different aggregation and limit on each curve.
7. **`teardown:` always runs.** PRBS off, linkscan on, golden configuration reloaded, whatever a main phase did.

### Identify

The first SSH session. The serial in the syseeprom TLVs is compared with the scanned label; the platform, ASIC, SONiC build and SAI package are compared as one object against the release; and every cage must report a module whose CMIS vendor part number is the station's loopback. A switch booted on a lab SAI passes PRBS and forwards frames differently in the field.

```python filename="phases/identify.py"
from utils.recipe import LOOPBACK_MODULE, PORTS


def identify(measurements, switch, unit, log):
    """Setup: the serial from the EEPROM against the label, platform, ASIC,
    SONiC build and SAI as one object against the release, and every cage
    populated with a loopback module before any lane is exercised. A switch
    booted on the wrong SAI passes PRBS and drops frames in the field."""
    eeprom = switch.syseeprom()
    identity = switch.identity()
    presence = switch.transceiver_presence()
    plugs_ok = presence["present"] == PORTS and presence["loopback_modules"] == PORTS and presence["module_pn"] == LOOPBACK_MODULE
    measurements.reported_serial = eeprom["serial"]
    measurements.serial_matches_label = eeprom["serial"] == unit.serial_number
    measurements.switch_identity = identity
    measurements.ports_present = presence["present"]
    measurements.loopback_plugs_detected = plugs_ok
    unit.metadata["sonic"] = identity["sonic"]
    unit.metadata["asic"] = identity["asic"]
    log.info(f"{eeprom['part']} {eeprom['serial']} (label {unit.serial_number}): {identity['asic']} on {identity['sonic']}, {presence['present']} cages, {presence['loopback_modules']} x {presence['module_pn']}")
```

### PRBS BER

Linkscan off, PRBS31 started on all 256 lanes at once, 60 s of accumulation, then the error counter and lock status of every lane. The phase computes each lane's BER as errors over bits, keeps the worst lane of each port as a log10 and counts the lanes that lost lock or crossed the budget, records both curves on one x-axis of 32 ports, stamps the polynomial and the dwell onto the run metadata, and attaches the raw shell dump.

```python filename="phases/prbs_ber.py"
import math

import numpy as np

from utils.recipe import BER_PRODUCTION_LIMIT, LANES_PER_PORT, PORT_NUMBERS, PRBS_DWELL_S, PRBS_POLYNOMIAL


def prbs_ber(measurements, attach, run, switch, log):
    """PRBS31 on all 256 lanes at once through the 0 dB loopback plugs,
    linkscan off, 60 s of accumulation, then BER = errors / bits per lane.
    Each port is judged on its worst lane against the 802.3 AUI budget of
    1e-5, two decades under the 2.4e-4 the FEC is built for; a lane that
    lost lock or crossed the budget counts as errored. The raw shell dump
    is attached for the review."""
    switch.set_linkscan(False)
    switch.prbs_start(PRBS_POLYNOMIAL)
    log.info(f"{PRBS_POLYNOMIAL} on {len(PORT_NUMBERS)} x {LANES_PER_PORT} lanes, {PRBS_DWELL_S:.0f} s dwell (mock: time-scaled)")
    errors, locked, bits = switch.prbs_accumulate(PRBS_DWELL_S)
    attach.file(switch.prbs_dump(), "prbs_per_lane.txt")
    run.metadata["prbs_polynomial"] = PRBS_POLYNOMIAL
    run.metadata["prbs_dwell_s"] = PRBS_DWELL_S

    e = np.asarray(errors, dtype=float)
    lock = np.asarray(locked, dtype=bool)
    ber = np.maximum(e, 1.0) / bits  # a lane with zero errors reports the resolution floor, not zero
    errored = (~lock) | (ber > BER_PRODUCTION_LIMIT)
    worst_log10 = np.log10(ber.max(axis=1))
    errored_lanes = errored.sum(axis=1)
    worst_port = int(np.argmax(worst_log10)) + 1
    for p, w, n in zip(PORT_NUMBERS, worst_log10, errored_lanes):
        log.info(f"Port {p:2d}: worst lane BER 1e{w:.2f}, {n} errored lanes")

    m = measurements.ber_per_port
    m.x_axis = PORT_NUMBERS
    m.y_axis.worst_lane_log10 = worst_log10.round(3).tolist()
    m.y_axis.worst_lane_log10.aggregations.max_log10 = round(float(worst_log10.max()), 3)
    m.y_axis.errored_lanes = errored_lanes.tolist()
    m.y_axis.errored_lanes.aggregations.total = int(errored_lanes.sum())
    log.info(f"Worst lane on the unit: port {worst_port} at BER {10 ** worst_log10.max():.1e} (limit {BER_PRODUCTION_LIMIT:.0e}), {int(errored_lanes.sum())} errored lanes, {int(e.sum())} errors over {math.log10(bits):.2f} decades of bits per lane")
```

A lane with zero errors is recorded at the resolution floor of the dwell, 3.1e-13, rather than at zero, so the log10 stays finite and the dashboard's trend shows what the measurement could resolve.

### Eye Margin

With PRBS still running, the SerDes dump on every lane: the smallest of the three PAM4 eyes in mV and its width in UI, as the SerDes firmware reports them. The worst lane of each port is kept. BER says which lane errs; the eye says how far the others are from erring, and a port whose eye closes over the lot while its BER holds is the early warning the BER alone does not give.

```python filename="phases/eye_margin.py"
import numpy as np

from utils.recipe import PORT_NUMBERS


def eye_margin(measurements, switch, log):
    """The SerDes dump with PRBS still running: per lane, the smallest of the
    three PAM4 eyes and its width, the worst lane kept per port. BER says a
    lane errs; the eye says how close the others are to erring."""
    height, width = switch.eye_scan()
    h = np.asarray(height).min(axis=1)
    w = np.asarray(width).min(axis=1)
    tight = int(np.argmin(h)) + 1
    m = measurements.eye_per_port
    m.x_axis = PORT_NUMBERS
    m.y_axis.height_mv = h.round(1).tolist()
    m.y_axis.height_mv.aggregations.min_mv = round(float(h.min()), 1)
    m.y_axis.width_ui = w.round(3).tolist()
    m.y_axis.width_ui.aggregations.min_ui = round(float(w.min()), 3)
    log.info(f"Smallest eye on port {tight}: {h.min():.1f} mV; narrowest {w.min():.3f} UI on port {int(np.argmin(w)) + 1}; median {np.median(h):.1f} mV x {np.median(w):.3f} UI")
```

![Worst-lane eye height against eye width per port: 31 green points clustered around 65 mV and 0.40 UI, port 17 orange at 58.1 mV and 0.357 UI, all above the 40 mV and 0.30 UI floor lines.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/switch-port-ber-chart-eye-6NV9kFoy8J4rnWZQTAztWQXN4Jnrzn.png)

The mock switch's worst eyes: a cluster around 65 mV by 0.40 UI, and port 17 at 58.1 mV by 0.357 UI, the lane at 3e-6 showing where the BER comes from. The floors are this line's; a switch whose cluster migrates toward them across a lot has a PCB lamination or a connector problem before any port fails.

### Snake Test

The PRBS phase's result object is injected. Ports with an errored lane are left out of the chain and named; the rest are configured as a snake, each port's ingress forwarded to the next port's egress, the last back to the first, so a frame from the generator crosses 32 ports and 32 loopback plugs before it returns. Five minutes at 99.99 % of line rate, then the generator's counters and the FEC counters of every port.

```python filename="phases/snake_test.py"
import numpy as np

from utils.recipe import PORT_NUMBERS, SNAKE_DURATION_S, SNAKE_FRAME_BYTES, SNAKE_RATE_PCT


def snake_test(measurements, run, switch, tgen, prbs_ber, log):
    """Line-rate traffic from one generator port, forwarded port to port
    through every loopback plug and back. The PRBS phase's results are
    injected: a port with an errored lane is left out of the chain and
    named, so the snake reports on the ports that can carry traffic rather
    than failing on the one already known. Frame loss must be zero and the
    FEC must have corrected everything it saw."""
    errored = prbs_ber.measurements.ber_per_port.y_axis.errored_lanes
    ports = [p for p, n in zip(PORT_NUMBERS, errored) if n == 0]
    skipped = [p for p, n in zip(PORT_NUMBERS, errored) if n]
    if skipped:
        log.warning(f"Ports left out of the snake after PRBS: {skipped}")
    switch.prbs_stop()
    switch.set_linkscan(True)
    hops = switch.snake_configure(ports)
    switch.clear_fec_counters()
    tgen.configure(SNAKE_FRAME_BYTES, SNAKE_RATE_PCT)
    run.metadata["traffic_generator"] = tgen.identity()
    log.info(f"{hops} hops, {SNAKE_FRAME_BYTES} B frames at {SNAKE_RATE_PCT} % of 400G for {SNAKE_DURATION_S / 60:.0f} min (mock: time-scaled)")
    stats = tgen.run(SNAKE_DURATION_S)
    corrected, uncorrected = switch.fec_counters(SNAKE_DURATION_S)

    c = np.asarray(corrected)
    u = np.asarray(uncorrected)
    measurements.ports_in_snake = hops
    measurements.frames_sent = stats["sent"]
    measurements.frames_lost = stats["lost"]
    measurements.throughput_pct = stats["rx_rate_pct"]
    m = measurements.fec_per_port
    m.x_axis = PORT_NUMBERS
    m.y_axis.corrected_cw = c.tolist()
    m.y_axis.corrected_cw.aggregations.max_cw = int(c.max())
    m.y_axis.uncorrected_cw = u.tolist()
    m.y_axis.uncorrected_cw.aggregations.total = int(u.sum())
    log.info(f"{stats['sent']} frames sent, {stats['lost']} lost, {stats['rx_rate_pct']:.3f} % of line rate; FEC corrected up to {c.max():.2e} codewords on port {int(np.argmax(c)) + 1}, {int(u.sum())} uncorrected")
```

![FEC corrected codewords per port after the 5 minute snake on a log axis: green bars between 1.2e7 and 3.2e7, port 17 orange at 1.55e8, under the 1.3e9 ceiling, with 9,751,950,585 frames sent and 0 lost.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/switch-port-ber-chart-fec-79g151rRKdLHt2iuS4lXXebVIgeYgA.png)

The FEC counters after the mock snake: 9.75 billion frames sent and none lost, every port between 1e7 and 4e7 corrected codewords, port 17 at 1.55e8, the same port and the same order of magnitude as its PRBS result, and zero uncorrected everywhere. The snake is where a forwarding, buffer or power-stage problem shows that PRBS cannot see, because PRBS never moves a frame.

### Module Monitors

At the end of the snake, with the 32 loopbacks having dissipated their emulated 12 W for five minutes, the CMIS temperature and Vcc of every module. The x-axis is the port, so the curve is a picture of the faceplate: a fan tray that has lost a rotor, or a power stage that sags under 32 modules, appears as a shape before it appears as an alarm.

```python filename="phases/module_monitors.py"
import numpy as np

from utils.recipe import LOOPBACK_POWER_W, PORT_NUMBERS


def module_monitors(measurements, switch, log):
    """CMIS temperature and Vcc from every loopback module at the end of the
    snake, with the plugs dissipating what a 400G optic would. A cage that
    runs hot here, or a Vcc rail that sags under 32 modules, is a fan tray
    or a power stage problem the optics will report as an alarm in the rack."""
    temp, vcc = switch.module_monitors()
    t = np.asarray(temp)
    v = np.asarray(vcc)
    m = measurements.modules_per_port
    m.x_axis = PORT_NUMBERS
    m.y_axis.temp_c = t.tolist()
    m.y_axis.temp_c.aggregations.max_c = round(float(t.max()), 1)
    m.y_axis.vcc_v = v.tolist()
    m.y_axis.vcc_v.aggregations.min_v = round(float(v.min()), 3)
    m.y_axis.vcc_v.aggregations.max_v = round(float(v.max()), 3)
    log.info(f"Modules at {LOOPBACK_POWER_W:.0f} W: warmest {t.max():.1f} C on port {int(np.argmax(t)) + 1}, coolest {t.min():.1f} C; Vcc {v.min():.3f} to {v.max():.3f} V")
```

### Restore Config

The teardown phase switches PRBS off on every lane, turns linkscan back on, reloads the golden `config_db.json` and records the checksum of the running configuration and the number of ports that came back up through their plugs. No switch leaves the station with a SerDes in a diagnostic mode.

```python filename="phases/restore_config.py"
def restore_config(measurements, switch, log):
    """Teardown: PRBS off, linkscan back on, the golden configuration
    reloaded and its checksum compared, every port back up through its
    plug. Runs whatever a main phase did, so no unit leaves the station
    with its SerDes in a diagnostic mode."""
    prbs_off = switch.prbs_stop()
    linkscan = switch.set_linkscan(True)
    reloaded = switch.config_reload()
    measurements.prbs_disabled = prbs_off
    measurements.linkscan_enabled = linkscan
    measurements.config_checksum = reloaded["checksum"]
    measurements.ports_up = reloaded["ports_up"]
    log.info(f"PRBS off, linkscan on, config {reloaded['checksum'][:19]}..., {reloaded['ports_up']} ports up")
```

### Mock Plugs

`SwitchUnderTest` stands in for the SONiC CLI and the diagnostic shell because PRBS, the eye dump, the FEC counters and the CMIS reads all describe the same 256 lanes. The unit's physical property is one pre-FEC BER per lane, drawn once at construction: a log-normal around 1.6e-7 clipped at 8e-7 on thirty-one ports, and on port 17 a spread around 1e-6 with lane 3 fixed at 3e-6. The PRBS errors are a Poisson draw on that BER over the 3.2e12 bits of the dwell, the eye height and width follow the BER with a little noise, the corrected codewords in the snake follow the port's mean BER over the 2.3e10 codewords of five minutes, and the module temperatures follow a bump centred on the middle of the faceplate. `TrafficGenerator` computes frames sent from the rate and the frame size and returns them all received. Both sleep the real dwell divided by 100. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, keep one SSH session per unit in `SwitchUnderTest` and run the SONiC commands and `bcmcmd` through it, parsing `prbs get` and `dsc` into the same lists; read the CMIS bytes through the platform's `sfputil` or the EEPROM page; and drive the generator through its REST API with one stream, frames sent and received per port. Keep the 60 s dwell and the 5 min snake at their real values and drop the time scale. The phases, measurements and limits stay the same.


### 400G DR4 Transceiver TP2 Test
URL: https://www.tofupilot.com/templates/400g-dr4-transceiver-tp2-test
End-of-line test of a 400GBASE-DR4 QSFP-DD transceiver: TP2 launch power, OMA, ER and TDECQ per lane, Rx sensitivity to BER 2.4e-4, SRS and temperature corners.

## Introduction

### 400G DR4 Transceiver Overview

A 400GBASE-DR4 transceiver carries 400 Gb/s over four parallel single-mode fibres at 1310 nm, 106.25 Gb/s PAM4 on each, for up to 500 m between a switch and the next tier of a data center fabric. In the QSFP-DD form factor the host side is eight electrical lanes of 53.125 Gb/s PAM4 (400GAUI-8) that a DSP gearboxes to the four optical lanes; the optical side is a silicon-photonics chip with four Mach-Zehnder modulators fed by one or two 1310 nm lasers, four germanium photodiodes, and an MPO-12 port that uses eight of its twelve fibres. One module that ships out of spec is not one dead link: it is a link that runs at a pre-FEC error rate the switch's RS(544,514) decoder cannot hold when the fibre plant adds its own loss, a lane that drops out at the top of the switch's temperature range, or a module whose power class lies to the host and pushes a 32-port line card over its cage budget.

![QSFP-DD optical transceiver with a recessed MPO receptacle, blue pull tab, and rear electrical contacts.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/400g-dr4-transceiver-tp2-test-matte-closeup-ahQxNqRahgoVp8w2oxbls4VfPuFQVX.webp)

The parts this test exercises: the CMIS management interface behind the gold edge contacts, the laser and modulator bias loops that set the launch power and the eye, the receiver photodiodes and the DSP's equaliser, the thermal path from the case to the laser TEC, and the MPO port that every result in the transmitter table is measured through.

[IEEE 802.3 Clause 124](https://www.ieee802.org/3/df/public/23_0523/dawe_3df_01_230523.pdf) specifies the physical layer as two tables at two test points. At TP2, the end of a short patch cord on the transmitter, Table 124-6 gives **average launch power -2.9 to +4.0 dBm, OMAouter -0.8 to +4.2 dBm, extinction ratio at least 3.5 dB, TDECQ at most 3.4 dB, and OMAouter minus TDECQ at least -2.2 dBm**, each per lane. TDECQ, the transmitter and dispersion eye closure for PAM4, is measured with the SSPRQ pattern through a fourth-order Bessel-Thomson reference receiver at 13.28 GHz and a five-tap feed-forward reference equaliser, so it is a number the scope computes, not one the operator reads off an eye. At TP3, the receiver, Table 124-7 gives **sensitivity as the OMAouter at which the BER reaches 2.4e-4**, the rate the FEC corrects, with the limit **max(-3.9 dBm, SECQ - 5.3 dB)** where SECQ is the eye closure of the reference transmitter used for the test, a **stressed receiver sensitivity of -1.9 dBm** with a conformance signal closed to 3.4 dB of SECQ, an average receive power range of -5.9 to +4 dBm and a damage threshold of 5 dBm. A vendor datasheet such as the [FS 400G QSFP-DD DR4 SiPh](https://img-en.fs.com/file/datasheet/400g-qsfp-dd-dr4-siph.pdf) restates those rows unchanged. The [QSFP-DD MSA hardware specification rev 7.0](http://www.qsfp-dd.com/wp-content/uploads/2023/09/QSFP-DD-Hardware-Rev7.0.pdf) adds the mechanical envelope, the 0 to 70 C commercial case range and the power classes, class 6 being at most 12 W. Every limit in this template is one of those rows, or says where it came from instead.

### Test Purpose

The procedure records one end-of-line fingerprint per module:

- **CMIS identity** as one object, the reported serial against the scanned label, the date code, the **power class and the module power** against the class ceiling, and the temperature after two minutes in high-power mode
- **The MPO connection** checked lane by lane on the power meter, with a dark fibre raising an error that the procedure retries after the operator cleans the ferrule
- **The TP2 transmitter table per lane**: average launch power, OMAouter, extinction ratio and TDECQ as four curves on one lane axis, each judged on its Clause 124 window, with the raw eye summary attached to the run
- **Receiver sensitivity per lane** by attenuator sweep from the reference transmitter until the BER crosses 2.4e-4, the whole BER curve kept for the worst lane, and the reference transmitter's SECQ and calibration id recorded so the -3.9 dBm limit is traceable
- **Stressed receiver sensitivity**: BER per lane with the conformance eye at -1.9 dBm
- **TDECQ and launch power at the 0 and 70 C case corners**, judged on the same windows and on their shift from the 25 C values injected from the earlier phase
- The **user EEPROM** stamped with the station and date at teardown, read back, and the module parked in low-power mode

![Four panels of the TP2 transmitter table per lane at 25 C: average launch power +1.07 to +1.46 dBm inside the -2.9 to +4.0 dBm window, OMAouter +1.02 to +1.66 dBm inside -0.8 to +4.2 dBm, extinction ratio 4.70 to 5.06 dB above the 3.5 dB line, and TDECQ 1.76 to 2.21 dB under the 3.4 dB line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/dr4-transceiver-chart-tx-tp2-Gx1bppokAf1R1EYfaD58MZECTXPcpQ.png)

The mock module's transmitter table against Clause 124: four lanes inside every window, the worst lane at 2.21 dB of TDECQ against a 3.4 dB limit and at -1.19 dBm of OMAouter minus TDECQ against -2.2 dBm.

The framework mechanics on show are `then: {error: retry}` with a `retry` limit and delay on the MPO connection, so the common false fail of a dirty ferrule costs a cleaning and not a module; previous-results injection, the temperature phase naming `tx_optical` as a parameter to get the 25 C curves it compares against; `attach.data` for the raw eye summary; `run.metadata` for the reference transmitter and its calibration record; `unit.metadata` written at teardown next to an EEPROM read-back; and multi-dimensional measurements with the lane as x-axis and a different aggregation set on each curve.

## Equipment & Setup

To run this test on a production line, the following are required:

- **A host evaluation board** with a QSFP-DD cage, a CMIS management port over I2C, and a current monitor on the 3.3 V rail
- **A case clamp** with a thermoelectric element holding the module case at 0, 25 and 70 C
- **An MPO-12 APC to LC fan-out** on a 1x4 optical switch, so one scope channel and one attenuator serve all four lanes
- **A sampling oscilloscope** with an optical reference receiver at 13.28 GHz and the PAM4 TDECQ analysis
- **A calibrated reference transmitter** with a stressed-eye path for the SRS conformance signal, driven by a BERT that generates PRBS13Q and SSPRQ and counts errors
- **A variable optical attenuator** and **an optical power meter**
- **The Device Under Test (DUT)**: a 400GBASE-DR4 QSFP-DD module with its production firmware
- **A TofuPilot Framework procedure** to sequence the stimuli and validate the limits
- **The TofuPilot Dashboard** to keep the per-module evidence, the BER curves and the eye summaries

### Hardware Components

#### Sampling Scope, BERT and Reference Transmitter

The transmitter table is a scope measurement: a Keysight N1092A DCA-M with the 400G PAM4 analysis option applies the 13.28 GHz Bessel-Thomson response and the five-tap FFE and reports TDECQ, OMAouter, ER and the average power per capture, triggered by the module's own SSPRQ pattern. The receiver side needs a reference transmitter whose eye is known: a Keysight M8040A BERT drives it with PRBS13Q, adds the sinusoidal jitter and interference that close the eye to the 3.4 dB SECQ of the conformance signal for the stressed test, and counts the errored bits behind the module's receiver through the host board. An EXFO FVA-600 attenuator sets the received OMAouter in 0.25 dB steps and a Keysight N7744C power meter reads the four lanes through the fan-out; the switch in front of the fan-out selects the lane for the scope and the attenuator.

![QSFP-DD optical transceiver secured in a host-board test fixture with a case clamp and MPO fiber connection.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/400g-dr4-transceiver-tp2-test-matte-cover-FMaKsWB5UOnM3h4ySzYLp6dOWsdDR9.webp)

The illustrated fixture: the transceiver in its host-board cage under a case clamp, with an MPO fiber connection. The BERT, optical switch, scope and attenuator are omitted.

#### Host Board and MPO Connection

The host board is where CMIS lives: page 00h for the identity, the power class in byte 200, the LowPwr control and the module state, page 03h for the user EEPROM the teardown writes. The MPO-12 APC connection is the one part of the station the operator touches on every module, and a single contaminated fibre end face is the common false fail on an optical line: the lane reads dark or several dB low, the module is good, and the fix is a fibre inspection scope and a cleaner. IEC 61300-3-35 gives the inspection criteria (scratches and defects by zone on the end face); the procedure raises on a dark lane and retries after the operator has inspected and cleaned, rather than recording a failed module.

### Where the Limits Come From

| Test | Limit | Basis |
|---|---|---|
| Identity, serial, date code | exact match, label match, `YYMMDD` | CMIS page 00h; no tolerance applies |
| Power class, module power | class 6, ≤ 12 W | QSFP-DD MSA rev 7.0 power classes |
| Module temperature | ≤ 60 °C after 2 min in a 25 °C cage | this line's derivation: a module hotter here reaches the 70 °C case ceiling in the customer's switch |
| MPO lane power | min ≥ -2.9 dBm, spread ≤ 1.0 dB, dark below -10 dBm raises | -2.9 dBm is the Clause 124 launch minimum; the spread and the dark threshold are this line's |
| Average launch power | -2.9 to +4.0 dBm per lane | Clause 124 Table 124-6 |
| OMAouter | -0.8 to +4.2 dBm per lane | Table 124-6 |
| Extinction ratio | ≥ 3.5 dB per lane | Table 124-6 |
| TDECQ | ≤ 3.4 dB per lane, SSPRQ, 13.28 GHz BT4, 5-tap FFE | Table 124-6 and 124-8 |
| OMAouter - TDECQ | ≥ -2.2 dBm, worst lane | Table 124-6 |
| Receiver sensitivity | ≤ -3.9 dBm OMAouter at BER 2.4e-4, worst lane | Table 124-7: max(-3.9 dBm, SECQ - 5.3 dB); the reference Tx SECQ is required ≤ 1.4 dB so the -3.9 dBm term governs |
| Stressed receiver sensitivity | BER ≤ 2.4e-4 on every lane at -1.9 dBm with 3.4 dB SECQ | Table 124-7 |
| Receiver power ceiling | ≤ 4 dBm presented during the sweep | Table 124-7 average receive power maximum; damage at 5 dBm |
| Temperature corners | TDECQ ≤ 3.4 dB and launch power in window at 0 and 70 °C; shift from 25 °C ≤ 0.6 dB TDECQ, ≤ 0.5 dB power | windows from Clause 124, case range from the MSA; the deltas are this line's derivation, sized to catch a bias or TEC loop that has lost its lock |
| EEPROM stamp, module state | exact | no tolerance applies |

Two rows in that table are traps. The TP2 table applies at the end of the patch cord, so the fan-out and the switch are inside the measurement: their loss has to be calibrated out per lane, or the station fails good modules on launch power by the 0.5 dB it eats. And the sensitivity limit depends on the bench: max(-3.9, SECQ - 5.3) is -3.9 dBm only while the reference transmitter's SECQ is at most 1.4 dB, which is why the phase records that SECQ from the calibration record and validates it, and why the calibration id goes into the run metadata. Two stations with different reference transmitters otherwise disagree on the same module.

## Test Procedure

### Overview

The procedure maps the test onto the framework's three stages. Identity and power live in `setup:` so the module's class and draw are known before the transmitter is enabled. The MPO connection opens `main:` with a retry, and the four optical phases chain on one switch and one attenuator. The EEPROM stamp and the park live in `teardown:` so the module goes to low-power mode even when a main phase fails.

1. Setup: CMIS identity, serial against the label, power class, module power and temperature after 2 min.
2. Main: transmitter on, four lane powers on the meter; a dark fibre raises and the phase retries.
3. Main: TP2 per lane on the scope, launch power, OMAouter, ER, TDECQ; eye summary attached.
4. Main: attenuator sweep per lane to BER 2.4e-4, curve kept for the worst lane, reference Tx recorded.
5. Main: stressed conformance eye at -1.9 dBm, BER per lane.
6. Main: case clamp at 0 and 70 C, TDECQ and launch power against the injected 25 C values.
7. Teardown: station and date into the user EEPROM, read back, transmitter off, low-power mode.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-module evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify_and_power.py
  mpo_connect.py
  tx_optical.py
  rx_sensitivity.py
  stressed_rx.py
  temperature_corners.py
  park_and_eeprom.py
plugs/
  dr4_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-qsfpdd-dr4-transceiver). The `Dr4Bench` plug is a mock of the host board, the case clamp, the switch and fan-out, the scope, the reference transmitter, the attenuator, the power meter and the BERT together, synthesizing a healthy module with about 1.3 dBm per lane, 4.9 dB of ER and 2 dB of TDECQ, a receiver reaching the FEC threshold near -5.6 dBm, and one contaminated fibre on the first MPO read, so the procedure runs end-to-end without a bench or a module connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug with the label serial as constructor argument, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: 400G DR4 Transceiver TP2 Test
version: 0.1.0
description: End-of-line test of a 400GBASE-DR4 QSFP-DD silicon-photonics transceiver against IEEE 802.3 Clause 124. CMIS identity and power class, the MPO connection checked lane by lane with a retry for a dirty ferrule, the TP2 transmitter table per lane (average launch, OMAouter, ER, TDECQ) on the sampling scope, receiver sensitivity by attenuator sweep to the 2.4e-4 FEC threshold with the BER curve kept for the worst lane, stressed receiver sensitivity at -1.9 dBm, TDECQ and launch power at the 0 and 70 C case corners against the 25 C values, and a teardown that stamps the station and date into the user EEPROM and parks the module in low-power mode.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the data-matrix on the module's pull tab"
    placeholder: "QDD-DR4-000000"
    pattern: "^QDD-DR4-\\d{6}$"
    default_value: "QDD-DR4-004417"
  part_number:
    default_value: "QDD-400G-DR4-S"
  batch_number:
    default_value: "LOT-2026-38"

plugs:
  - name: DR4 Bench
    description: "Host board with CMIS port, case clamp, MPO fan-out and switch, sampling scope, reference transmitter, attenuator, power meter and BERT (mock, one plug per bench)"
    python: plugs.dr4_bench:Dr4Bench
    key: bench
    config:
      serial_number: "QDD-DR4-004417"

setup:
  - name: Identify and Power
    key: identify_and_power
    python: phases.identify_and_power
    timeout: 5m
    measurements:
      - name: Module Identity
        key: module_identity
        description: "CMIS page 00h vendor, part, hardware revision, firmware and media as one object against the release."
        validators:
          - operator: "=="
            expected_value:
              vendor: TP-PHOTONICS
              part: QDD-400G-DR4-S
              hw_rev: B2
              firmware: 2.1.4
              media: SiPh 1310 nm MPO-12 APC
      - name: Reported Serial
        key: reported_serial
        validators:
          - {operator: matches, expected_value: "^QDD-DR4-\\d{6}$"}
      - name: Serial Matches Label
        key: serial_matches_label
        description: "The serial the module reports over CMIS is the one on its pull tab."
        validators:
          - {operator: "==", expected_value: true}
      - name: Date Code
        key: date_code
        description: "CMIS date code, YYMMDD."
        validators:
          - {operator: matches, expected_value: "^2\\d{5}$"}
      - name: Power Class
        key: power_class
        description: "QSFP-DD MSA rev 7.0 power class from page 00h byte 200; class 6 is the 12 W the host cage is budgeted for."
        validators:
          - {operator: "==", expected_value: 6}
      - name: Module Power
        key: module_power_w
        unit: W
        description: "Host 3.3 V rail current times voltage in high-power mode after the warm-up; the class 6 ceiling."
        validators:
          - {operator: "<=", expected_value: 12.0}
      - name: Module Temperature
        key: module_temperature_c
        unit: °C
        description: "Internal monitor after 2 min in high-power mode in a 25 C host cage. 60 C is this line's derivation; a module hotter here reaches the 70 C case ceiling in the customer's switch."
        validators:
          - {operator: "<=", expected_value: 60.0}

main:
  - name: MPO Connect
    key: mpo_connect
    python: phases.mpo_connect
    retry:
      limit: 2
      delay: 2s
    then:
      error: retry
    measurements:
      - name: MPO Lane Power
        key: mpo_lane_power
        title: Average power per lane at TP2 on the power meter after the cord is seated
        x_axis:
          legend: Lane
        y_axis:
          - legend: Average power
            key: avg_power_dbm
            unit: dBm
            aggregations:
              - type: min_dbm
                unit: dBm
                validators:
                  - {operator: ">=", expected_value: -2.9}
              - type: spread_db
                unit: dB
                validators:
                  - {operator: "<=", expected_value: 1.0}

  - name: Tx Optical
    key: tx_optical
    python: phases.tx_optical
    depends_on: [mpo_connect]
    measurements:
      - name: Transmitter at TP2
        key: tx_optical
        title: Average launch power, OMAouter, ER and TDECQ per lane at 25 C case
        x_axis:
          legend: Lane
        y_axis:
          - legend: Average launch power
            key: avg_power_dbm
            unit: dBm
            aggregations:
              - type: min_dbm
                unit: dBm
                validators:
                  - {operator: ">=", expected_value: -2.9}
              - type: max_dbm
                unit: dBm
                validators:
                  - {operator: "<=", expected_value: 4.0}
          - legend: OMAouter
            key: oma_outer_dbm
            unit: dBm
            aggregations:
              - type: min_dbm
                unit: dBm
                validators:
                  - {operator: ">=", expected_value: -0.8}
              - type: max_dbm
                unit: dBm
                validators:
                  - {operator: "<=", expected_value: 4.2}
          - legend: Extinction ratio
            key: er_db
            unit: dB
            aggregations:
              - type: min_db
                unit: dB
                validators:
                  - {operator: ">=", expected_value: 3.5}
          - legend: TDECQ
            key: tdecq_db
            unit: dB
            aggregations:
              - type: max_db
                unit: dB
                validators:
                  - {operator: "<=", expected_value: 3.4}
              - type: min_oma_minus_tdecq_dbm
                unit: dBm
                validators:
                  - {operator: ">=", expected_value: -2.2}

  - name: Rx Sensitivity
    key: rx_sensitivity
    python: phases.rx_sensitivity
    depends_on: [tx_optical]
    measurements:
      - name: BER Curve
        key: ber_curve
        title: BER against received OMAouter for the worst lane, reference transmitter through the attenuator
        x_axis:
          legend: Received OMAouter
          unit: dBm
        y_axis:
          - legend: BER
            key: ber
            aggregations:
              - type: sensitivity_dbm
                unit: dBm
                validators:
                  - {operator: "<=", expected_value: -3.9}
      - name: Rx Sensitivity
        key: rx_sensitivity
        title: Receiver sensitivity per lane, OMAouter at BER 2.4e-4
        x_axis:
          legend: Lane
        y_axis:
          - legend: Sensitivity
            key: sensitivity_dbm
            unit: dBm
            aggregations:
              - type: max_dbm
                unit: dBm
                validators:
                  - {operator: "<=", expected_value: -3.9}
      - name: Reference Tx SECQ
        key: reference_tx_secq_db
        unit: dB
        description: "Eye closure of the bench's reference transmitter from its calibration record. Clause 124 sets the sensitivity limit at max(-3.9 dBm, SECQ - 5.3 dB); at 1.4 dB or better the -3.9 dBm term governs."
        validators:
          - {operator: "<=", expected_value: 1.4}
      - name: Rx Power Ceiling
        key: rx_power_ceiling_dbm
        unit: dBm
        description: "Highest OMAouter the sweep presents to the receiver. The operating maximum is 4 dBm and the damage threshold 5 dBm."
        validators:
          - {operator: "<=", expected_value: 4.0}

  - name: Stressed Rx
    key: stressed_rx
    python: phases.stressed_rx
    depends_on: [rx_sensitivity]
    measurements:
      - name: Stressed Rx BER
        key: stressed_rx_ber
        title: BER per lane with the stressed conformance signal at -1.9 dBm OMAouter
        x_axis:
          legend: Lane
        y_axis:
          - legend: BER
            key: ber
            aggregations:
              - type: max_ber
                validators:
                  - {operator: "<=", expected_value: 0.00024}
      - name: SRS Bits per Lane
        key: srs_bits_per_lane
        unit: bit
        description: "Gate of 10 s per lane at 106.25 Gb/s; enough errors at the threshold to count, not to estimate."
        validators:
          - {operator: ">=", expected_value: 1000000000000}
      - name: All Lanes Under FEC Threshold
        key: all_lanes_under_fec_threshold
        validators:
          - {operator: "==", expected_value: true}

  - name: Temperature Corners
    key: temperature_corners
    python: phases.temperature_corners
    depends_on: [stressed_rx]
    timeout: 40m
    measurements:
      - name: Temperature Corners
        key: temperature_corners
        title: TDECQ and average launch power per lane at the 0 and 70 C case corners
        x_axis:
          legend: Lane
        y_axis:
          - legend: TDECQ at 0 C
            key: tdecq_0c
            unit: dB
            aggregations:
              - type: max_db
                unit: dB
                validators:
                  - {operator: "<=", expected_value: 3.4}
          - legend: TDECQ at 70 C
            key: tdecq_70c
            unit: dB
            aggregations:
              - type: max_db
                unit: dB
                validators:
                  - {operator: "<=", expected_value: 3.4}
          - legend: Average launch power at 0 C
            key: avg_power_0c
            unit: dBm
            aggregations:
              - type: min_dbm
                unit: dBm
                validators:
                  - {operator: ">=", expected_value: -2.9}
              - type: max_dbm
                unit: dBm
                validators:
                  - {operator: "<=", expected_value: 4.0}
          - legend: Average launch power at 70 C
            key: avg_power_70c
            unit: dBm
            aggregations:
              - type: min_dbm
                unit: dBm
                validators:
                  - {operator: ">=", expected_value: -2.9}
              - type: max_dbm
                unit: dBm
                validators:
                  - {operator: "<=", expected_value: 4.0}
      - name: Max TDECQ Delta
        key: max_tdecq_delta_db
        unit: dB
        description: "Largest change in TDECQ on any lane at either corner against the 25 C value from tx_optical. 0.6 dB is this line's derivation; a modulator bias loop that has lost its lock moves more."
        validators:
          - {operator: "<=", expected_value: 0.6}
      - name: Max Power Delta
        key: max_power_delta_db
        unit: dB
        description: "Largest change in average launch power on any lane at either corner against 25 C. 0.5 dB is this line's derivation; a laser TEC loop that has lost its lock moves more."
        validators:
          - {operator: "<=", expected_value: 0.5}

teardown:
  - name: Park and EEPROM
    key: park_and_eeprom
    python: phases.park_and_eeprom
    measurements:
      - name: EEPROM Stamp
        key: eeprom_stamp
        description: "Station id and test date written to the CMIS user EEPROM page 03h."
        validators:
          - {operator: matches, expected_value: "^OPT-EOL-\\d{2} \\d{4}-\\d{2}-\\d{2}$"}
      - name: EEPROM Readback Matches
        key: eeprom_readback_matches
        description: "The page reads back what was written."
        validators:
          - {operator: "==", expected_value: true}
      - name: Module State
        key: module_state
        description: "CMIS module state after the host asserts LowPwr, so the module is cold when the operator pulls it."
        validators:
          - {operator: "==", expected_value: LowPwr}
```

Framework features to notice:

1. **`then: {error: retry}` with a `retry` limit and delay.** `mpo_connect` raises on a dark lane; the framework waits two seconds, runs the phase again, up to two retries, and the run continues with the retried attempt. Every attempt is kept and shown on the run. On a real line a `ui` prompt in front of the retry tells the operator what to clean.
2. **Previous-results injection.** `temperature_corners` names `tx_optical` as a function parameter and receives that phase's results, the 25 C curves included, to compute its deltas; `depends_on` guarantees the order.
3. **`attach.data` for the raw file.** The PAM4 level means, the equaliser taps and the wavelength per lane travel with the run as `tp2_eye_summary.json`, so a TDECQ that drifts across a lot can be traced to a level or a tap.
4. **`run.metadata` for the bench's identity.** The reference transmitter and its calibration id are on the run, not on the unit: a sensitivity drift across a week can be split between modules and reference.
5. **`unit.metadata` and an EEPROM read-back.** The teardown writes the station and date into the module and onto the unit record, and a boolean says the module holds what was written.
6. **Four curves on one lane axis, each with its own aggregations.** `tx_optical` carries launch power, OMAouter, ER and TDECQ against the lane; the windows are `min_dbm` and `max_dbm`, the floor `min_db`, the ceiling `max_db`, and the combined rule `min_oma_minus_tdecq_dbm`.
7. **A whole curve judged on one interpolated number.** `ber_curve` keeps the sweep of the worst lane and validates `sensitivity_dbm`.

### Identify and Power

CMIS page 00h the moment the module is seated: the identity as one object against the release, the serial against the label, the date code, and the power class. Then the host releases low-power mode and the module settles for two minutes before the 3.3 V rail current and the internal temperature are read. The mock skips the two minutes; the `timeout` is sized for the real ones.

```python filename="phases/identify_and_power.py"
from utils.recipe import POWER_CLASS_LIMIT_W, WARM_UP_MIN


def identify_and_power(measurements, bench, unit, log):
    """Setup: CMIS page 00h read the moment the module is seated, then the
    host releases low-power mode and the module settles for two minutes
    before its power and temperature are read. A module that reports the
    wrong power class is a switch that oversubscribes its cage budget."""
    identity = bench.cmis_identity()
    serial = bench.cmis_serial()
    date_code = bench.cmis_date_code()
    power_class = bench.cmis_power_class()
    bench.set_low_power(False)
    bench.warm_up(WARM_UP_MIN)
    log.info(f"Warm-up: {WARM_UP_MIN} min in high-power mode (mock: instant)")
    power_w = bench.host_module_power_w()
    temp_c = bench.cmis_temperature_c()
    measurements.module_identity = identity
    measurements.reported_serial = serial
    measurements.serial_matches_label = serial == unit.serial_number
    measurements.date_code = date_code
    measurements.power_class = power_class
    measurements.module_power_w = power_w
    measurements.module_temperature_c = temp_c
    unit.metadata["firmware"] = identity["firmware"]
    unit.metadata["date_code"] = date_code
    log.info(f"{identity['vendor']} {identity['part']} rev {identity['hw_rev']} fw {identity['firmware']} reports {serial} ({date_code}), label {unit.serial_number}; class {power_class} ({POWER_CLASS_LIMIT_W[power_class]:.0f} W), draws {power_w:.2f} W, {temp_c:.1f} C after {WARM_UP_MIN} min")
```

### MPO Connect

The transmitter is enabled and the four lanes are read on the power meter through the fan-out. A lane below -10 dBm is dark: the cord is not seated, is keyed wrong, or has a contaminated end face on that fibre. The phase raises, and the procedure's `then: {error: retry}` runs it again after the operator has inspected and cleaned the ferrule per IEC 61300-3-35. Once all four lanes carry light, the phase records them and judges the lane-to-lane spread: four fibres of one clean MPO agree within a few tenths of a dB, and a single dirty end face shows as one lane a decibel low.

```python filename="phases/mpo_connect.py"
from utils.recipe import LANES, LIGHT_PRESENT_DBM


def mpo_connect(measurements, bench, log):
    """Setup: transmitter on, the four TP2 powers read on the power meter
    through the MPO fan-out. A dark fibre means the cord is not seated, is
    keyed wrong, or has a contaminated end face on that fibre: the phase
    raises, the operator inspects and cleans the ferrule (IEC 61300-3-35)
    and reseats, and the procedure's `then: error: retry` runs the phase
    again. A dirty connector does not fail the module."""
    bench.tx_enable(True)
    powers = bench.meter_lane_powers_dbm()
    dark = [lane for lane, p in zip(LANES, powers) if p < LIGHT_PRESENT_DBM]
    if dark:
        raise RuntimeError(f"Lane {dark} dark at the power meter ({powers} dBm): inspect the MPO end face per IEC 61300-3-35, clean, reseat and retry")
    spread = max(powers) - min(powers)
    m = measurements.mpo_lane_power
    m.x_axis = LANES
    m.y_axis.avg_power_dbm = powers
    m.y_axis.avg_power_dbm.aggregations.min_dbm = round(min(powers), 2)
    m.y_axis.avg_power_dbm.aggregations.spread_db = round(spread, 2)
    log.info(f"MPO seated: {powers} dBm at TP2 on the meter, lane spread {spread:.2f} dB")
```

The mock's first read shows lane 3 at -31.4 dBm; the retried attempt reads +1.34, +0.96, +1.43 and +1.18 dBm with a 0.47 dB spread.

### Tx Optical

Four captures on the sampling scope at 25 C case, one per lane through the switch: average launch power, OMAouter, extinction ratio and TDECQ with the SSPRQ pattern, the 13.28 GHz reference receiver and the five-tap FFE. The four series go on one lane axis; each carries the aggregations Clause 124 judges it on, and the TDECQ series also carries the worst OMAouter minus TDECQ. The level means, the taps and the wavelength are attached as one JSON file.

```python filename="phases/tx_optical.py"
import json

from utils.recipe import LANES, TX_FFE_TAPS, TX_PATTERN, TX_SCOPE_BW_GHZ


def tx_optical(measurements, bench, attach, log):
    """TP2 at 25 C case on the sampling scope: average launch power,
    OMAouter, ER and TDECQ per lane with the SSPRQ pattern, the 13.28 GHz
    Bessel-Thomson reference receiver and the 5-tap FFE of Clause 124.
    The raw eye summary (PAM4 level means, equaliser taps, wavelength)
    travels with the run as an attachment."""
    bench.set_case_temp_c(25.0)
    avg, oma, er, tdecq, raw = [], [], [], [], {}
    for lane in LANES:
        r = bench.scope_tx_lane(lane)
        avg.append(r["avg_dbm"])
        oma.append(r["oma_dbm"])
        er.append(r["er_db"])
        tdecq.append(r["tdecq_db"])
        raw[f"lane_{lane}"] = r
        log.info(f"Lane {lane}: {r['avg_dbm']:+.2f} dBm avg, OMAouter {r['oma_dbm']:+.2f} dBm, ER {r['er_db']:.2f} dB, TDECQ {r['tdecq_db']:.2f} dB at {r['wavelength_nm']:.1f} nm")
    oma_minus_tdecq = [o - t for o, t in zip(oma, tdecq)]

    m = measurements.tx_optical
    m.x_axis = LANES
    m.y_axis.avg_power_dbm = avg
    m.y_axis.avg_power_dbm.aggregations.min_dbm = round(min(avg), 3)
    m.y_axis.avg_power_dbm.aggregations.max_dbm = round(max(avg), 3)
    m.y_axis.oma_outer_dbm = oma
    m.y_axis.oma_outer_dbm.aggregations.min_dbm = round(min(oma), 3)
    m.y_axis.oma_outer_dbm.aggregations.max_dbm = round(max(oma), 3)
    m.y_axis.er_db = er
    m.y_axis.er_db.aggregations.min_db = round(min(er), 3)
    m.y_axis.tdecq_db = tdecq
    m.y_axis.tdecq_db.aggregations.max_db = round(max(tdecq), 3)
    m.y_axis.tdecq_db.aggregations.min_oma_minus_tdecq_dbm = round(min(oma_minus_tdecq), 3)

    raw["conditions"] = {"pattern": TX_PATTERN, "reference_receiver_ghz": TX_SCOPE_BW_GHZ, "ffe_taps": TX_FFE_TAPS, "case_c": 25.0}
    attach.data(json.dumps(raw, indent=2).encode(), "tp2_eye_summary.json")
    log.info(f"Worst lane: TDECQ {max(tdecq):.2f} dB, OMAouter - TDECQ {min(oma_minus_tdecq):+.2f} dBm, ER {min(er):.2f} dB")
```

OMAouter follows from the average power and the ER: OMA = 2 P (ER - 1) / (ER + 1) in linear units, so at 4.9 dB of ER it sits 0.1 dB above the average power. The OMAouter minus TDECQ rule is the one to watch on a silicon-photonics transmitter: a modulator biased off quadrature keeps its OMA and loses its eye, and fails this row before it fails TDECQ alone.

### Rx Sensitivity

Each lane's receiver against the reference transmitter through the attenuator: from -2 dBm down in 0.25 dB steps with one second of gating per step until the BER crosses 2.4e-4, then the sensitivity interpolated in log10(BER) between the last two steps. The whole curve is kept for the worst lane; every lane records its sensitivity. The reference transmitter's SECQ from its calibration record is validated at 1.4 dB or better, the condition under which the fixed -3.9 dBm term of the Clause 124 limit governs, and the calibration id goes into the run metadata.

```python filename="phases/rx_sensitivity.py"
import math

import numpy as np

from utils.recipe import BER_THRESHOLD, LANES, SWEEP_GATE_S, SWEEP_START_DBM, SWEEP_STEP_DB, SWEEP_STOP_DBM


def sweep_lane(bench, lane):
    """Attenuator down in 0.25 dB steps from -2 dBm until the BER crosses the
    FEC threshold; the sensitivity is interpolated in log10(BER) between the
    last two steps. Returns the sweep floor if the lane never crossed."""
    powers, bers = [], []
    p = SWEEP_START_DBM
    while p >= SWEEP_STOP_DBM:
        bench.set_rx_oma_dbm(lane, p)
        errors, bits = bench.ber_gate(lane, SWEEP_GATE_S)
        powers.append(p)
        bers.append(errors / bits)
        if bers[-1] > BER_THRESHOLD:
            break
        p = round(p - SWEEP_STEP_DB, 2)
    if bers[-1] <= BER_THRESHOLD:
        return powers, bers, SWEEP_STOP_DBM
    lb = [math.log10(max(b, 1e-15)) for b in bers[-2:]]
    frac = (math.log10(BER_THRESHOLD) - lb[0]) / (lb[1] - lb[0])
    return powers, bers, round(powers[-2] + frac * (powers[-1] - powers[-2]), 3)


def rx_sensitivity(measurements, bench, run, log):
    """Each lane's receiver against the calibrated reference transmitter
    through the attenuator, one second of gating per step. The whole BER
    curve is kept for the worst lane; every lane records its sensitivity.
    The Clause 124 limit is max(-3.9 dBm, SECQ - 5.3 dB) where SECQ is the
    reference transmitter's own eye closure; with a reference at 1.4 dB or
    better the -3.9 dBm term governs, so the reference's SECQ is recorded."""
    ref = bench.reference_tx()
    run.metadata["reference_tx"] = ref["id"]
    run.metadata["reference_tx_cal"] = ref["cal_id"]
    bench.srs_enable(False)
    curves, sens = {}, []
    for lane in LANES:
        powers, bers, s = sweep_lane(bench, lane)
        curves[lane] = (powers, bers)
        sens.append(s)
        log.info(f"Lane {lane}: {len(powers)} steps of {SWEEP_GATE_S:.0f} s (mock: instant), BER {bers[-1]:.2e} at {powers[-1]:+.2f} dBm, sensitivity {s:+.2f} dBm")
    worst = LANES[int(np.argmax(sens))]
    powers, bers = curves[worst]

    m = measurements.ber_curve
    m.x_axis = powers[::-1]
    m.y_axis.ber = bers[::-1]
    m.y_axis.ber.aggregations.sensitivity_dbm = sens[worst - 1]
    s = measurements.rx_sensitivity
    s.x_axis = LANES
    s.y_axis.sensitivity_dbm = sens
    s.y_axis.sensitivity_dbm.aggregations.max_dbm = round(max(sens), 3)
    measurements.reference_tx_secq_db = ref["secq_db"]
    measurements.rx_power_ceiling_dbm = SWEEP_START_DBM
    log.info(f"Worst lane {worst} at {max(sens):+.2f} dBm OMA; reference {ref['id']} ({ref['cal_id']}, SECQ {ref['secq_db']} dB)")
```

![BER against received OMAouter for lane 2, the worst lane, on a log scale from 1e-10 at -2.75 dBm to the 2.4e-4 FEC threshold at -5.46 dBm, with the -3.9 dBm sensitivity limit as a vertical line and the other three lanes' sensitivities of -5.72, -5.81 and -5.58 dBm as markers on the threshold.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/dr4-transceiver-chart-ber-curve-n7OwcPM0MOU9HDK3QOkLYdMD4mIcx9.png)

The mock module's worst lane reaches the FEC threshold at -5.46 dBm, 1.56 dB below the limit; the other three at -5.72, -5.81 and -5.58 dBm. The waterfall is the receiver's Gaussian tail: a decade of BER for about 0.6 dB of optical power near the threshold, which is why the sweep steps at 0.25 dB and why a limit at -3.9 dBm with a module at -5.5 dBm is a comfortable margin and a module at -4.2 dBm is not.

### Stressed Rx

The conformance signal, the reference eye closed to 3.4 dB of SECQ, at -1.9 dBm OMAouter on each lane for ten seconds. Every lane must stay under the FEC threshold. The stress is what the reference equaliser cannot recover; a receiver whose DSP recovers most of it reads a BER well under the limit, one that does not reads at it.

```python filename="phases/stressed_rx.py"
from utils.recipe import BER_THRESHOLD, LANES, SRS_GATE_S, SRS_OMA_DBM, SRS_SECQ_DB


def stressed_rx(measurements, bench, log):
    """Stressed receiver sensitivity: the conformance signal with 3.4 dB of
    SECQ at -1.9 dBm OMAouter, ten seconds of gating per lane. Every lane
    must stay under the FEC threshold; the DSP has to recover the eye that
    the reference equaliser cannot."""
    bench.srs_enable(True)
    bers = []
    bits = 0
    for lane in LANES:
        bench.set_rx_oma_dbm(lane, SRS_OMA_DBM)
        errors, bits = bench.ber_gate(lane, SRS_GATE_S)
        bers.append(errors / bits)
        log.info(f"Lane {lane}: {errors} errors in {bits:.3e} bits at {SRS_OMA_DBM} dBm with {SRS_SECQ_DB} dB SECQ, BER {bers[-1]:.2e} (gate {SRS_GATE_S:.0f} s, mock: instant)")
    bench.srs_enable(False)

    m = measurements.stressed_rx_ber
    m.x_axis = LANES
    m.y_axis.ber = bers
    m.y_axis.ber.aggregations.max_ber = max(bers)
    measurements.srs_bits_per_lane = float(bits)
    measurements.all_lanes_under_fec_threshold = max(bers) <= BER_THRESHOLD
    log.info(f"Worst stressed lane BER {max(bers):.2e}, a factor {BER_THRESHOLD / max(bers):.1f} under the {BER_THRESHOLD:.1e} threshold")
```

![Stressed receiver BER per lane on a log scale, four green bars between 7.7e-6 and 3.2e-5 with the stressed conformance eye at -1.9 dBm and 3.4 dB SECQ over a 10 s gate, all under the red 2.4e-4 limit line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/dr4-transceiver-chart-stressed-rx-AQftPodUgv96if4wY0Wkm7adI6GiFn.png)

The mock module's stressed BER per lane: 1.1e-5, 3.2e-5, 7.7e-6 and 2.0e-5, the worst a factor 7.5 under the threshold. A factor of ten on BER near the threshold is about half a decibel of optical margin, so the bars are read on a log scale and a lane an order of magnitude above its siblings is a lane to look at even when it passes.

### Temperature Corners

The case clamp goes to 0 C, soaks, and the four lanes are captured again on the scope; then 70 C. TDECQ and launch power at each corner are judged on the same Clause 124 windows, and lane by lane against the 25 C values the `tx_optical` phase recorded, injected here as a parameter. The mock skips the soaks; the `timeout` is sized for two real ones.

```python filename="phases/temperature_corners.py"
from utils.recipe import CASE_TEMPS_C, LANES, SOAK_MIN


def curve(result, key):
    """One y-axis series out of the injected multi-dimensional measurement."""
    return [float(v) for v in next(y for y in result["y_axis"] if y["key"] == key)["data"]]


def temperature_corners(measurements, bench, tx_optical, log):
    """TDECQ and average launch power re-measured at the 0 and 70 C case
    corners after a soak at each, and compared lane by lane with the 25 C
    values the tx_optical phase recorded, injected here by phase key. A
    laser TEC loop or a modulator bias loop that has lost its lock shows
    up as a delta long before it fails an absolute limit."""
    base = tx_optical.tx_optical
    base_tdecq = curve(base, "tdecq_db")
    base_avg = curve(base, "avg_power_dbm")
    corners = {}
    for temp in CASE_TEMPS_C:
        bench.set_case_temp_c(temp)
        bench.soak(SOAK_MIN)
        log.info(f"Case clamp at {temp:.0f} C, soak {SOAK_MIN} min (mock: instant)")
        tdecq, avg = [], []
        for lane in LANES:
            r = bench.scope_tx_lane(lane)
            tdecq.append(r["tdecq_db"])
            avg.append(r["avg_dbm"])
        corners[temp] = (tdecq, avg)
        log.info(f"{temp:.0f} C: TDECQ {tdecq} dB, avg {avg} dBm")
    bench.set_case_temp_c(25.0)
    t0, a0 = corners[0.0]
    t70, a70 = corners[70.0]
    d_tdecq = [t - b for t, b in zip(t0 + t70, base_tdecq + base_tdecq)]
    d_avg = [a - b for a, b in zip(a0 + a70, base_avg + base_avg)]

    m = measurements.temperature_corners
    m.x_axis = LANES
    m.y_axis.tdecq_0c = t0
    m.y_axis.tdecq_0c.aggregations.max_db = round(max(t0), 3)
    m.y_axis.tdecq_70c = t70
    m.y_axis.tdecq_70c.aggregations.max_db = round(max(t70), 3)
    m.y_axis.avg_power_0c = a0
    m.y_axis.avg_power_0c.aggregations.min_dbm = round(min(a0), 3)
    m.y_axis.avg_power_0c.aggregations.max_dbm = round(max(a0), 3)
    m.y_axis.avg_power_70c = a70
    m.y_axis.avg_power_70c.aggregations.min_dbm = round(min(a70), 3)
    m.y_axis.avg_power_70c.aggregations.max_dbm = round(max(a70), 3)
    measurements.max_tdecq_delta_db = round(max(abs(d) for d in d_tdecq), 3)
    measurements.max_power_delta_db = round(max(abs(d) for d in d_avg), 3)
    log.info(f"Largest shift from 25 C: TDECQ {max(d_tdecq, key=abs):+.2f} dB, launch power {max(d_avg, key=abs):+.2f} dB")
```

![Two panels across the 0, 25 and 70 C case corners: TDECQ per lane rising from 1.76 to 2.48 dB at 70 C under the 3.4 dB limit with a largest shift of 0.32 dB, and average launch power per lane between +0.81 and +1.46 dBm inside the -2.9 to +4.0 dBm window with a largest shift of 0.26 dB.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/dr4-transceiver-chart-temperature-mcE9BF4qlALx66XI4GbejTUS8DTBFh.png)

The mock module drifts the way a healthy silicon-photonics transmitter does: TDECQ up 0.31 dB and launch power down 0.24 dB at 70 C, with the lanes moving together. The delta limits are not in Clause 124; they are this line's derivation, sized so that a laser TEC loop or a modulator bias loop that has lost its lock, which moves a single lane by a decibel or more, fails here while it still passes the absolute windows.

### Park and EEPROM

The teardown writes the station id and the test date into the CMIS user EEPROM page and reads it back, so a module returned from the field names the line and the day that shipped it. Then the transmitter goes off and the host asserts LowPwr; the module state is recorded so the operator pulls a cold module.

```python filename="phases/park_and_eeprom.py"
from datetime import date

from utils.recipe import STATION_ID, USER_EEPROM_PAGE


def park_and_eeprom(measurements, bench, unit, log):
    """Teardown: the station id and the test date written into the user
    EEPROM page so a module returned from the field names the line that
    shipped it, read back, then the transmitter off and the module put in
    low-power mode before the operator pulls it. Runs whatever a main
    phase did."""
    today = date.today().isoformat()
    stamp = f"{STATION_ID} {today}"
    bench.eeprom_write_user(stamp)
    readback = bench.eeprom_read_user()
    bench.tx_enable(False)
    bench.set_low_power(True)
    state = bench.module_state()
    measurements.eeprom_stamp = stamp
    measurements.eeprom_readback_matches = readback == stamp
    measurements.module_state = state
    unit.metadata["eol_station"] = STATION_ID
    unit.metadata["eol_date"] = today
    unit.metadata["user_eeprom_page"] = USER_EEPROM_PAGE
    log.info(f"User EEPROM page {USER_EEPROM_PAGE} holds '{readback}', module {state}")
```

### Mock Plug

`Dr4Bench` stands in for the host board, the case clamp, the switch and fan-out, the scope, the reference transmitter, the attenuator, the power meter and the BERT because every reading depends on which lane is selected and what the attenuator and the clamp are doing at that instant, and plugs run in separate processes. Its transmitter is four lanes at 1.32, 1.05, 1.48 and 1.21 dBm with 4.9, 4.7, 5.1 and 4.8 dB of ER and 1.9, 2.2, 1.7 and 2.0 dB of TDECQ, with OMAouter derived from the power and the ER; its receiver reaches the FEC threshold at -5.72, -5.46, -5.81 and -5.58 dBm with a Q that scales with the linear optical power and a PAM4 BER of three quarters of the Gaussian tail, Poisson-counted over the gate; the stressed eye costs 2.95 dB of its 3.4 dB SECQ; the case corners add 0.12 and 0.31 dB of TDECQ and take 0.08 and 0.24 dB of launch power; and the first MPO read after the cord is seated shows lane 3 dark. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

```python filename="plugs/dr4_bench.py"
"""400G DR4 transceiver EOL bench (mock): the host board with the QSFP-DD
cage and its CMIS port, the case clamp, the MPO-12 fan-out and optical
switch, the sampling scope with the 13.28 GHz reference receiver, the
calibrated reference transmitter with its stressed-eye path, the variable
attenuator, the power meter and the BERT, one plug because every reading
depends on which lane the switch has selected and what the attenuator and
the case clamp are doing at that instant.

Maps to a Keysight N1092A DCA-M sampling scope with the PAM4 TDECQ
analysis, a Keysight M8040A BERT generating PRBS13Q and SSPRQ, an EXFO
FVA-600 attenuator, a Keysight N7744C power meter, an MPO-12 APC to LC
fan-out on a 1x4 optical switch, a thermoelectric case clamp, and a host
evaluation board driving CMIS over I2C. The mock synthesizes a healthy
module: about 1.3 dBm per lane with 4.9 dB of ER and 2 dB of TDECQ, a
receiver reaching the FEC threshold near -5.6 dBm OMA, a stressed eye
that the DSP mostly recovers, a few tenths of a dB of drift across the
case range, and one contaminated fibre on the first MPO read. Swap for
classes speaking SCPI and I2C; the phases stay unchanged.
"""

import math

import numpy as np

from utils.recipe import HOST_VCC_V, IDENTITY, LANE_GBPS, LANES


class Dr4Bench:
    # The module this bench synthesizes, per lane at 25 C case.
    AVG_DBM = [1.32, 1.05, 1.48, 1.21]
    ER_DB = [4.9, 4.7, 5.1, 4.8]
    TDECQ_DB = [1.9, 2.2, 1.7, 2.0]
    WAVELENGTH_NM = [1309.6, 1310.4, 1309.9, 1310.8]
    SENS_DBM = [-5.72, -5.46, -5.81, -5.58]  # OMAouter at BER 2.4e-4, unstressed
    SRS_PENALTY_DB = 2.95  # of the 3.4 dB of SECQ, what the DSP does not recover
    Q_AT_THRESHOLD = 3.414  # PAM4: 0.75 * 0.5 * erfc(Q / sqrt 2) = 2.4e-4
    # Drift from the 25 C values with the case temperature: TDECQ up, launch power down.
    TDECQ_DRIFT_DB = {0.0: 0.12, 25.0: 0.0, 70.0: 0.31}
    POWER_DRIFT_DB = {0.0: -0.08, 25.0: 0.0, 70.0: -0.24}

    def __init__(self, serial_number):
        self.serial_number = str(serial_number)
        self._rng = np.random.default_rng(1310)
        self._case_c = 25.0
        self._low_power = True
        self._tx_on = False
        self._warm = False
        self._mpo_reads = 0
        self._srs = False
        self._rx_oma_dbm = {lane: None for lane in LANES}
        self._user_eeprom = ""
        # self.i2c = smbus2...; self.scope = pyvisa...; self.bert = pyvisa...; self.att = pyvisa...; self.meter = pyvisa...
        print(f"DR4 bench connected: module {self.serial_number} seated, low-power mode, case clamp at 25 C, attenuator parked")

    # --- CMIS management, page 00h ---------------------------------------------

    def cmis_identity(self):
        return dict(IDENTITY)

    def cmis_serial(self):
        return self.serial_number

    def cmis_date_code(self):
        return "260914"

    def cmis_power_class(self):
        return 6

    def set_low_power(self, enabled):
        self._low_power = bool(enabled)
        if self._low_power:
            self._tx_on = False

    def module_state(self):
        return "LowPwr" if self._low_power else "Ready"

    def host_module_power_w(self):
        """Host 3.3 V rail current times voltage; a DR4 in high-power mode
        draws about 2.85 A, in low-power mode a few hundred milliamps."""
        amps = 0.42 if self._low_power else 2.85
        return round(HOST_VCC_V * (amps + self._rng.normal(0.0, 0.01)), 2)

    def warm_up(self, minutes):
        self._warm = True  # mock: instant

    def cmis_temperature_c(self):
        """Internal temperature monitor: about 22 C over the case once the
        module has been in high-power mode for a couple of minutes."""
        rise = 21.8 if self._warm else 4.0
        return round(self._case_c + rise + self._rng.normal(0.0, 0.2), 1)

    # --- transmitter at TP2 ---------------------------------------------------------

    def tx_enable(self, enabled):
        self._tx_on = bool(enabled) and not self._low_power

    def _avg_dbm(self, lane):
        return self.AVG_DBM[lane - 1] + self.POWER_DRIFT_DB[self._case_c]

    def meter_lane_powers_dbm(self):
        """Average power at TP2 on the power meter through the MPO fan-out,
        the four lanes in one read. The first read after the cord is seated
        shows lane 3 dark: a contaminated end face on that fibre."""
        self._mpo_reads += 1
        if not self._tx_on:
            return [-60.0] * len(LANES)
        p = [round(self._avg_dbm(lane) + self._rng.normal(0.0, 0.04), 2) for lane in LANES]
        if self._mpo_reads == 1:
            p[2] = -31.4
        return p

    def scope_tx_lane(self, lane):
        """One lane at TP2 on the sampling scope with the 13.28 GHz reference
        receiver and the 5-tap FFE: average power, OMAouter, ER, TDECQ,
        wavelength, the four PAM4 level means and the equaliser taps.
        OMAouter follows from the average power and the ER."""
        i = lane - 1
        avg = self._avg_dbm(lane) + self._rng.normal(0.0, 0.03)
        er = self.ER_DB[i] + self._rng.normal(0.0, 0.04)
        er_lin = 10 ** (er / 10.0)
        oma = avg + 10 * math.log10(2 * (er_lin - 1) / (er_lin + 1))
        tdecq = self.TDECQ_DB[i] + self.TDECQ_DRIFT_DB[self._case_c] + self._rng.normal(0.0, 0.03)
        p_avg = 10 ** (avg / 10.0)
        p0 = 2 * p_avg / (er_lin + 1)
        p3 = 2 * p_avg * er_lin / (er_lin + 1)
        step = (p3 - p0) / 3.0
        levels = [p0, p0 + 0.99 * step, p0 + 2.01 * step, p3]
        taps = (np.array([-0.05, 0.12, 0.86, 0.09, -0.02]) + self._rng.normal(0.0, 0.005, 5)).round(3).tolist()
        return {
            "avg_dbm": round(avg, 3),
            "oma_dbm": round(oma, 3),
            "er_db": round(er, 3),
            "tdecq_db": round(tdecq, 3),
            "wavelength_nm": round(self.WAVELENGTH_NM[i] + self._rng.normal(0.0, 0.02), 2),
            "levels_mw": [round(x, 4) for x in levels],
            "ffe_taps": taps,
        }

    # --- case temperature ---------------------------------------------------------------

    def set_case_temp_c(self, temp_c):
        self._case_c = float(temp_c)

    def soak(self, minutes):
        pass  # mock: instant

    # --- receiver at TP3 ----------------------------------------------------------------

    def reference_tx(self):
        """The calibrated reference transmitter on this bench and its
        calibration record, including the SECQ its conformance path presents."""
        return {"id": "REFTX-DR4-07", "cal_id": "CAL-2026-0914-REFTX07", "cal_date": "2026-09-14", "secq_db": 1.1}

    def srs_enable(self, enabled):
        self._srs = bool(enabled)

    def set_rx_oma_dbm(self, lane, dbm):
        self._rx_oma_dbm[lane] = float(dbm)

    def ber_gate(self, lane, seconds):
        """Errors and bits from the BERT after the gate. The mock's receiver:
        Q grows with the received optical power (linear), the PAM4 BER is
        three quarters of the Gaussian tail, and the stressed eye costs the
        part of its SECQ that the DSP cannot recover."""
        bits = int(LANE_GBPS * 1e9 * seconds)
        penalty = self.SRS_PENALTY_DB if self._srs else 0.0
        q = self.Q_AT_THRESHOLD * 10 ** ((self._rx_oma_dbm[lane] - penalty - self.SENS_DBM[lane - 1]) / 10.0)
        ber = 0.75 * 0.5 * math.erfc(q / math.sqrt(2.0))
        errors = int(self._rng.poisson(ber * bits))
        return [errors, bits]

    # --- user EEPROM and park -----------------------------------------------------------

    def eeprom_write_user(self, text):
        self._user_eeprom = str(text)[:128]

    def eeprom_read_user(self):
        return self._user_eeprom

    def __del__(self):
        print("Tx off, module in low-power mode, attenuator parked, case clamp at 25 C, bench released")
```

The recipe constants and the origin of every limit sit in one file:

```python filename="utils/recipe.py"
"""End-of-line recipe for a 400GBASE-DR4 QSFP-DD transceiver: the stimulus
applied at every step and the limit each result is judged on.

Limits come from IEEE 802.3-2022 Clause 124 (400GBASE-DR4) where it gives a
number: the TP2 transmitter table (average launch, OMAouter, ER, TDECQ,
OMAouter - TDECQ), the TP3 receiver table (sensitivity, stressed receiver
sensitivity, average receive power, damage threshold) and the 2.4e-4 BER
that the RS(544,514) FEC corrects. The power classes come from the QSFP-DD
MSA hardware specification rev 7.0. Everything else (the MPO lane spread,
the warm-up temperature ceiling, the temperature-corner deltas) is this
line's derivation and is marked as such in procedure.yaml."""

# Unit under test
LANES = [1, 2, 3, 4]
LANE_GBPS = 106.25  # PAM4, 53.125 GBd per lane
IDENTITY = {"vendor": "TP-PHOTONICS", "part": "QDD-400G-DR4-S", "hw_rev": "B2", "firmware": "2.1.4", "media": "SiPh 1310 nm MPO-12 APC"}
POWER_CLASS_LIMIT_W = {1: 1.5, 2: 3.5, 3: 7.0, 4: 8.0, 5: 10.0, 6: 12.0, 7: 14.0}  # QSFP-DD MSA rev 7.0
WARM_UP_MIN = 2  # module temperature read after 2 min in the host cage (mock: instant)
HOST_VCC_V = 3.3

# MPO connection, before any optical measurement (this line's derivation)
LIGHT_PRESENT_DBM = -10.0  # below this a fibre is dark: cord not seated, wrong key, or a blocked end face
LANE_SPREAD_MAX_DB = 1.0  # four fibres of one clean MPO agree within a few tenths of a dB

# Transmitter at TP2, Clause 124 Table 124-6, 25 C case
TX_SCOPE_BW_GHZ = 13.28  # fourth-order Bessel-Thomson reference receiver
TX_PATTERN = "SSPRQ"
TX_FFE_TAPS = 5  # reference equaliser for TDECQ
AVG_LAUNCH_DBM = (-2.9, 4.0)
OMA_OUTER_DBM = (-0.8, 4.2)
ER_MIN_DB = 3.5
TDECQ_MAX_DB = 3.4
OMA_MINUS_TDECQ_MIN_DBM = -2.2

# Receiver at TP3, Clause 124 Table 124-7
BER_THRESHOLD = 2.4e-4  # what RS(544,514) corrects
RX_SENS_MAX_DBM = -3.9  # max(-3.9, SECQ - 5.3): -3.9 governs while the reference Tx SECQ is at most 1.4 dB
REF_TX_SECQ_MAX_DB = 1.4
SWEEP_START_DBM = -2.0
SWEEP_STOP_DBM = -8.5
SWEEP_STEP_DB = 0.25
SWEEP_GATE_S = 1.0  # per attenuator step (mock: instant)
SRS_OMA_DBM = -1.9  # stressed receiver sensitivity, OMAouter at TP3
SRS_SECQ_DB = 3.4  # stressed eye closure of the conformance signal
SRS_GATE_S = 10.0  # per lane (mock: instant)
RX_AVG_POWER_DBM = (-5.9, 4.0)
RX_DAMAGE_DBM = 5.0

# Temperature corners, QSFP-DD commercial case range (this line's derivation for the deltas)
CASE_TEMPS_C = [0.0, 70.0]
SOAK_MIN = 10  # per corner (mock: instant)
TDECQ_DELTA_MAX_DB = 0.6
POWER_DELTA_MAX_DB = 0.5

# Teardown
STATION_ID = "OPT-EOL-03"
USER_EEPROM_PAGE = "03h"  # CMIS user EEPROM page, bytes 128 to 255
```

On a real bench, split the class into the CMIS port over I2C, the case clamp, the switch, the scope and the BERT over SCPI, the attenuator and the power meter over their drivers. Read the reference transmitter's calibration record from the bench, calibrate the fan-out and switch loss out per lane, keep the one-second gates in the sweep and the ten-second gates in the stressed test, hold the soaks at each corner, and put a `ui` prompt in front of the MPO retry. The phases, measurements and limits stay the same.

### Server Fan Tray End-of-Line Test
URL: https://www.tofupilot.com/templates/server-fan-tray-end-of-line-test
End-of-line test of a dual-rotor 12 V server fan tray: PWM to RPM curve, 13.2 V start-up current, tach integrity, airflow, vibration spectrum and dBA.

## Introduction

### Server Fan Tray Overview

A 2U server moves its heat with a row of hot-swap fan trays behind the front drive bays, each tray a sled with one or two 80 mm fans, a latch, and a blind-mate connector that carries 12 V, the PWM command and the tach return to the fan board. The dual-rotor version stacks two counter-rotating 80 mm rotors in one sled: the second rotor straightens the swirl of the first and roughly doubles the static pressure, which is what pushes air through a chassis packed with drives, DIMMs and a heat sink wall. The BMC drives every tray on one 25 kHz PWM and reads every tach, and it decides from those tachs whether a fan has failed and whether the chassis must go to full speed. One tray that ships out of spec is not one noisy fan: it is a chassis that throttles its CPUs because a rotor lags, a rack that runs its fans at 100 % because a tach reports half speed, or a node that loses a fan slot to a bearing at month six.

![Deep tandem server fan module with two coaxial rotor stages, a mounting sled, blue latch, and fan connector.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/server-fan-tray-end-of-line-test-matte-closeup-CLYu4XqwaSmrkLHRICPpIZBhmacBcz.webp)

The parts this test exercises: the sled and its insulation from the fan leads, the FRU EEPROM behind the hot-swap contacts, the PWM buffer that feeds each rotor, the two motor drives and their tach outputs, the two ball-bearing pairs, the blades, and the latch that tells the operator which airflow variant is in hand.

The Intel [4-Wire Pulse Width Modulation (PWM) Controlled Fans specification](https://www.konilabs.net/docs/standards/fan/intel_4wire_pwm_fans_specs_rev1_2.pdf), revision 1.2, is the document every server fan datasheet is written against. It sets the PWM frequency at **25 kHz, acceptable from 21 to 28 kHz** (section 2.4), requires the speed, as a percentage of the fan's maximum, to **match the PWM duty within 10 points** at every point (3.3), caps the vendor's minimum speed at **30 % of maximum** (3.2), defines the tach as **two pulses per revolution** on an open-collector output (2.3), caps the start-up current at **2.0 A at 13.2 V** for the fans of its era (2.2), and asks for 7 500 on/off cycles of endurance (5.1.4). A fan tray vendor's end-of-line sheet adds what the fan spec leaves to the datasheet: RPM at 20, 50 and 100 % duty within 10 %, input current, airflow in CFM and static pressure on an AMCA 210 / ISO 5801 chamber, sound pressure at 1 m by ISO 7779 (ECMA-74), insulation resistance of **10 MΩ at 500 Vdc** and a dielectric test at **500 Vac for one minute at 5 mA**. Every limit in this template is one of those numbers, or says where it came from instead.

### Test Purpose

The procedure records one end-of-line fingerprint per tray:

- **Insulation resistance and dielectric leakage** from the fan leads to the sled with the supply off, before anything spins
- **The FRU EEPROM** read through the hot-swap connector as one object against the release, its serial compared with the label, and the operator's answers on the latch colour and the finger guard
- **Both rotors through 20, 30, 50, 70 and 100 % duty**, speed and current on four curves, each rotor's speed in percent of its own full speed judged against the duty line, the 20 % point against the minimum-speed rule, the full speeds against the datasheet, and the two rotors against each other
- **A cold start at 13.2 V**, the tray current and both tachs recorded whole, judged on the peak and on the time each rotor takes to reach 90 % of the full speed the previous phase recorded
- **Tach pulses per revolution** against the fixture's optical tach, the tach duty, and the PWM frequency at the rotor pins behind the tray's buffer
- **Airflow and static pressure** across seven damper points on the chamber, judged in free air, at shut-off and at the chassis design point
- **The sled's vibration spectrum** at full speed, judged on the 1x tone of each rotor and the ball-pass tones of each bearing, and the **sound pressure at 1 m**
- A **teardown** that parks the PWM and the supply, puts the power cycles on the unit and the fixture's mating count on the run, and attaches the raw accelerometer capture

![Speed of rotor A in green and rotor B in blue against PWM duty from 20 to 100 percent, each inside the 10 point band around the duty line to its own full speed, worst deviations 4.03 and 4.02 points at 20 percent duty, both at 24.0 percent of full speed there against the 30 percent limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/fan-tray-chart-rpm-curve-KB02kn1FUnf68wAwGAvzAeWQsdXH70.png)

The mock tray's two rotors against the duty line: a minimum-speed clamp lifts both 4 points above it at 20 % duty, well inside the 10 point band, and both sit at 24 % of their own full speed there where the spec allows 30. The inlet rotor runs 1.2 % above its datasheet full speed and the outlet rotor 0.6 % below, which the two scalar full-speed measurements and the mismatch catch, not the curve.

The framework mechanics on show are an operator radio and a switch bound to validated measurements in a `setup:` phase, pre-baked from a JSON file for headless runs; a previous phase's result injected into the start-up phase as a parameter; a `station`-scoped plug for the chamber that keeps its damper homed and its mating counter running from one tray to the next; four multi-dimensional measurements with a different aggregation set on each axis; and a `teardown:` that writes `unit.metadata`, `run.metadata` and an `attach.data` file from a capture the bench plug kept.

## Equipment & Setup

To run this end-of-line test on a production line, the following are required:

- **A programmable 12 V supply** rated for the tray's start-up current at 13.2 V, with a current digitizer fast enough to catch a 100 ms peak
- **A DAQ with counters**: two PWM outputs at 25 kHz, two tach inputs, one optical tach input, and an IEPE analog input for the accelerometer
- **An AMCA 210 / ISO 5801 airflow chamber** whose inlet plate carries the tray fixture, with a motorised damper, nozzle differential pressure and chamber static pressure
- **An accelerometer** on the sled and **a class 1 sound level meter** at 1 m on the inlet axis, in a hemi-anechoic booth
- **An electrical safety analyzer** for insulation resistance at 500 Vdc and the dielectric test at 500 Vac, interlocked so the supply cannot enable while it is connected
- **An I2C adapter** on the hot-swap connector's SMBus pins for the FRU EEPROM
- **The Device Under Test (DUT)**: a dual-rotor 80 mm 12 V hot-swap fan tray with its production FRU programmed
- **A TofuPilot Framework procedure** to sequence the stimuli and validate the limits
- **The TofuPilot Dashboard** to keep the per-tray evidence and trend the bearing tones across the lot

### Hardware Components

#### Supply, Counters and Chamber

Two rotors at 4.0 and 3.7 A running current draw about 9.7 A at 13.2 V and peak above 12 A on a cold start, so the supply is a Keysight N6700C mainframe with an N6754A module (60 V, 20 A), whose digitizer records the current waveform the start-up phase judges. An NI USB-6363 gives the two PWM counter outputs, the two tach inputs and the optical tach input, and its analog input takes the accelerometer through an IEPE conditioner (a PCB Piezotronics 352C33 on a 482C05). The airflow chamber is a Long Win LW-9266-class multi-nozzle chamber, or an Airflow Measurement Systems bench: the tray fixture is the chamber's inlet plate, the damper on the outlet sets the operating point, and the nozzle bank's differential pressure gives the flow. The optical tach is a Monarch Instrument remote optical sensor on a reflective mark on each hub.

![Tandem server fan in a test fixture attached to a flow plenum, with a finger guard and differential-pressure hoses.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/server-fan-tray-end-of-line-test-matte-cover-D8Al1P0IrKQNmJW9q3r5bm5uEUmjAt.webp)

The illustrated fixture: a tandem fan attached to a flow plenum, with pressure tubing and a protective inlet guard. The supply, DAQ and acoustic measurement equipment are omitted.

#### Safety Analyzer and FRU

The [Chroma 19032](https://www.chromaate.com/en/product/electrical_safety_analyzer_19032) does the 500 Vdc insulation resistance and the 500 Vac dielectric test in one connection, with the fan leads tied together against the sled. The dielectric test is AC because the fan spec's vendors write it that way, at 500 V for one minute; a healthy tray leaks a fraction of a milliamp through the winding's capacitance to the frame, and a lead pinched under the sled leaks the full 5 mA and trips. The FRU EEPROM sits on the tray's small board with the PWM buffers and is read over the connector's SMBus pins in the IPMI FRU format; the part number, hardware revision and airflow variant in it are compared as one object with the release, and the serial with the label on the latch.

### Where the Limits Come From

| Test | Limit | Basis |
|---|---|---|
| Insulation resistance | ≥ 10 MΩ at 500 Vdc | vendor EOL item |
| Dielectric | ≤ 5 mA at 500 Vac, 60 s | vendor EOL item |
| PWM frequency at the rotor pins | 21 to 28 kHz | Intel 2.4 |
| Speed against the duty | in percent of the rotor's own full speed, within 10 points of the duty at every point | Intel 3.3 |
| Speed at 20 % duty | ≤ 30 % of the rotor's own full speed | Intel 3.2 |
| Full speed | 16 000 rpm rotor A, 14 500 rpm rotor B, within 10 % | datasheet, the vendor's EOL tolerance; Intel 3.1 leaves the maximum to the vendor |
| Rotor mismatch at 100 % | ≤ 5 % between the two normalised speeds | this line's derivation: a rotor that lags its twin is loaded |
| Current at 100 % | ≤ 4.4 A rotor A, ≤ 4.1 A rotor B | datasheet maximum, rated plus 10 % |
| Start-up peak at 13.2 V | ≤ 15.4 A for the tray | Intel 2.2 caps a single fan of its era at 2.0 A; this line applies the same ratio, two rated currents, to a 7.7 A tray |
| Spin-up | both rotors at 90 % of full speed within 3 s | this line's derivation: the BMC's fan-fault window; Intel 3.2 allows a 2 s start pulse |
| Tach | exactly 2 pulses per revolution, duty 40 to 60 % | Intel 2.3 for the count; the duty window is this line's |
| Airflow in free air | ≥ 157.5 CFM | datasheet 175 CFM less 10 %, the speed tolerance applied to flow, this line's |
| Static pressure at shut-off | ≥ 810 Pa | datasheet 900 Pa less 10 % |
| Pressure at the chassis design flow | ≥ 450 Pa at 100 CFM | this line's derivation from the chassis impedance |
| 1x vibration per rotor | ≤ 1.0 g | ISO 21940-11 grade G6.3 at 16 000 rpm is 6.3 mm/s, 1.06 g at 267 Hz; this line's derivation |
| Ball-pass tones per bearing | ≤ 0.15 g | this line's derivation |
| Sound pressure at 1 m | ≤ 74 dBA | ISO 7779 / ECMA-74 method; datasheet 71 dBA plus 3 dB, this line's |
| Power cycles applied | ≤ 3 | Intel 5.1.4 budgets 7 500; the count goes on the unit |
| FRU identity, serial, latch, guard | exact match | no tolerance applies |

Three rows in that table are traps. Intel's 2.0 A start-up cap cannot be applied as written to a rotor rated at 4 A: a tray that met it would have to soft-start so slowly that it failed the spin-up window, so the limit is scaled to the ratio the spec implies and the description says so. The same section allows the spike to exceed 1.0 A for no more than 1 s, which has no clean scaling to a tray that runs at 9.7 A at 13.2 V; this line replaces it with the spin-up time. And the bearing-band limit is not a standard: it is set from what new bearings show on this fixture, and its value is a trend to watch across the lot, not a number to argue with a supplier.

## Test Procedure

### Overview

The procedure maps the end-of-line test onto the framework's three stages. Insulation, the FRU read and the operator's answers live in `setup:` so no rotor turns on a tray with a pinched lead or the wrong airflow variant. The park and the counters live in `teardown:` so the PWM goes to zero and the supply off even when a main phase fails.

1. Setup: insulation resistance and dielectric with the supply off, FRU EEPROM, latch colour and finger guard from the operator.
2. Main: both rotors through five duty points, curves, minimum speed, rotor mismatch.
3. Main: cold start at 13.2 V, current and tachs whole, peak and spin-up time.
4. Main: tach pulses per revolution against the optical tach, tach duty, PWM frequency.
5. Main: airflow and static pressure across seven damper points on the chamber.
6. Main: vibration spectrum at full speed, 1x and ball-pass tones, sound pressure at 1 m.
7. Teardown: PWM to zero, supply off, cycles on the unit, mating count on the run, raw capture attached.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-tray evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify_and_insulation.py
  pwm_rpm_curve.py
  startup_current.py
  tach_integrity.py
  airflow_and_pressure.py
  vibration_and_acoustic.py
  park_and_cycles.py
plugs/
  fan_bench.py
  flow_bench.py
utils/
  recipe.py
ui.json
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-fan-tray-eol). The `FanBench` plug is a mock of the supply, the counters, the optical tach, the accelerometer, the sound level meter, the safety analyzer and the FRU link together, and `FlowBench` a mock of the chamber; between them they synthesize a healthy tray on the duty line with new bearings, so the procedure runs end-to-end without a bench or a tray connected. The 60 s dielectric dwell, the settling at each duty point and the spin-up return without waiting.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless, with the operator's two answers pre-baked from `ui.json`:

```bash
tofupilot run . --no-tui --no-kiosk --json --ui-values ui.json
```

### The Procedure File

`procedure.yaml` declares the unit, the two plugs (the chamber at `scope: station`), and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Server Fan Tray End-of-Line Test
version: 0.1.0
description: End-of-line test of a dual-rotor 80 mm 12 V hot-swap fan tray for a 2U server. Insulation and dielectric with the supply off, the FRU EEPROM read through the hot-swap connector against the release, both rotors through the PWM duty points with the speed in percent of full speed against the duty line, a cold start at 13.2 V with the current recorded whole, tach pulses per revolution against the fixture's optical tach, airflow and static pressure on the chamber, the vibration spectrum for imbalance and bearing tones, sound pressure at 1 m, and a teardown that parks the tray, counts its power cycles and attaches the raw accelerometer capture.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the label on the tray's latch"
    placeholder: "FT80-0000000"
    pattern: "^FT80-\\d{7}$"
    default_value: "FT80-0031842"
  part_number:
    default_value: "FT-2U-80CR-12"
  batch_number:
    default_value: "LOT-2026-38"

plugs:
  - name: Fan Bench
    description: "12 V supply with current digitizer, PWM and tach counters, optical tach, accelerometer, sound level meter, safety analyzer and the I2C link to the FRU EEPROM (mock, one plug per bench)"
    python: plugs.fan_bench:FanBench
    key: bench
    config:
      serial_number: "FT80-0031842"
  - name: Flow Chamber
    description: "AMCA 210 / ISO 5801 nozzle chamber whose inlet plate is the tray fixture, damper, pressures and the connector mating counter (mock, held across trays)"
    python: plugs.flow_bench:FlowBench
    key: flow_bench
    scope: station

setup:
  - name: Identify and Insulation
    key: identify_and_insulation
    python: phases.identify_and_insulation
    ui:
      components:
        - key: latch
          type: radio
          label: "Latch colour"
          description: "The latch colour is the airflow variant; this line builds front-to-back trays"
          required: true
          bind: measurements.latch_airflow
          options:
            - label: "Blue latch, front-to-back"
              value: "front-to-back"
            - label: "Red latch, back-to-front"
              value: "back-to-front"
        - key: grille
          type: switch
          label: "Finger guard seated on the outlet rotor"
          description: "The chamber reads the tray with its guard, as it ships"
          required: true
          bind: measurements.grille_seated
    measurements:
      - name: Latch Airflow Variant
        key: latch_airflow
        description: "Operator's answer. A back-to-front tray passes every electrical test and starves the chassis."
        validators:
          - {operator: "==", expected_value: "front-to-back"}
      - name: Finger Guard Seated
        key: grille_seated
        validators:
          - {operator: "==", expected_value: true}
      - name: Insulation Resistance
        key: insulation_mohm
        unit: MΩ
        description: "All lines tied together against the sled at 500 Vdc (vendor EOL item)."
        validators:
          - {operator: ">=", expected_value: 10.0}
      - name: Dielectric Leakage
        key: dielectric_leakage_ma
        unit: mA
        description: "500 Vac lines to sled for 60 s (vendor EOL item); the mock returns without the dwell."
        validators:
          - {operator: "<=", expected_value: 5.0}
      - name: FRU Identity
        key: fru_identity
        description: Part number, hardware revision, airflow variant and FRU format from the EEPROM as one object against the release.
        validators:
          - operator: "=="
            expected_value:
              part_number: FT-2U-80CR-12
              hw_rev: B2
              airflow: front-to-back
              fru_format: ipmi-1.0
      - name: EEPROM Serial Matches Label
        key: eeprom_serial_matches_label
        description: The serial the BMC will inventory is the one on the latch.
        validators:
          - {operator: "==", expected_value: true}

main:
  - name: PWM to RPM Curve
    key: pwm_rpm_curve
    python: phases.pwm_rpm_curve
    measurements:
      - name: PWM Curve
        key: pwm_curve
        title: Speed and current of both rotors against PWM duty at 12 V, speed judged in percent of full speed against the duty
        x_axis:
          legend: PWM duty
          unit: "%"
        y_axis:
          - legend: Rotor A speed
            key: rpm_a
            unit: rpm
            aggregations:
              - type: max_dev_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 10.0}
              - type: min_speed_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 30.0}
          - legend: Rotor B speed
            key: rpm_b
            unit: rpm
            aggregations:
              - type: max_dev_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 10.0}
              - type: min_speed_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 30.0}
          - legend: Rotor A current
            key: current_a
            unit: A
            aggregations:
              - type: at_100_pct_a
                unit: A
                validators:
                  - {operator: "<=", expected_value: 4.4}
          - legend: Rotor B current
            key: current_b
            unit: A
            aggregations:
              - type: at_100_pct_a
                unit: A
                validators:
                  - {operator: "<=", expected_value: 4.1}
      - name: Full Speed A
        key: full_speed_a_rpm
        unit: rpm
        description: "16 000 rpm at 100 % duty on the datasheet, within the datasheet's 10 % (Intel 3.1 leaves the maximum to the vendor); the reference for the spin-up time."
        validators:
          - {operator: ">=", expected_value: 14400.0}
          - {operator: "<=", expected_value: 17600.0}
      - name: Full Speed B
        key: full_speed_b_rpm
        unit: rpm
        description: "14 500 rpm at 100 % duty on the datasheet, within 10 %."
        validators:
          - {operator: ">=", expected_value: 13050.0}
          - {operator: "<=", expected_value: 15950.0}
      - name: Rotor Mismatch
        key: rotor_mismatch_pct
        unit: "%"
        description: "Each rotor's full speed as a fraction of its own nominal; the two fractions must agree within 5 %. This line's derivation: a rotor that lags its twin is loaded by a rubbing blade or a bearing, even while still inside its own 10 %."
        validators:
          - {operator: "<=", expected_value: 5.0}

  - name: Start-Up Current
    key: startup_current
    python: phases.startup_current
    depends_on: [pwm_rpm_curve]
    measurements:
      - name: Start-Up
        key: startup
        title: Tray current and both rotor speeds on a 0 to 100 percent step at 13.2 V (Intel 2.2)
        x_axis:
          legend: Time after the step
          unit: s
        y_axis:
          - legend: Tray current
            key: current
            unit: A
            aggregations:
              - type: peak_a
                unit: A
                validators:
                  - {operator: "<=", expected_value: 15.4}
          - legend: Rotor A speed
            key: rpm_a
            unit: rpm
            aggregations:
              - type: t90_s
                unit: s
                validators:
                  - {operator: "<=", expected_value: 3.0}
          - legend: Rotor B speed
            key: rpm_b
            unit: rpm
            aggregations:
              - type: t90_s
                unit: s
                validators:
                  - {operator: "<=", expected_value: 3.0}

  - name: Tach Integrity
    key: tach_integrity
    python: phases.tach_integrity
    depends_on: [startup_current]
    measurements:
      - name: Tach Pulses Per Revolution A
        key: tach_pulses_per_rev_a
        description: "Tach frequency over the optical tach's speed (Intel 2.3: two pulses per revolution)."
        validators:
          - {operator: "==", expected_value: 2}
      - name: Tach Duty A
        key: tach_duty_a_pct
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 40.0}
          - {operator: "<=", expected_value: 60.0}
      - name: PWM Frequency at Rotor A
        key: pwm_freq_a_khz
        unit: kHz
        description: "At the rotor pins behind the tray's buffer (Intel 2.4: 25 kHz target, 21 to 28 accepted)."
        validators:
          - {operator: ">=", expected_value: 21.0}
          - {operator: "<=", expected_value: 28.0}
      - name: Tach Pulses Per Revolution B
        key: tach_pulses_per_rev_b
        validators:
          - {operator: "==", expected_value: 2}
      - name: Tach Duty B
        key: tach_duty_b_pct
        unit: "%"
        validators:
          - {operator: ">=", expected_value: 40.0}
          - {operator: "<=", expected_value: 60.0}
      - name: PWM Frequency at Rotor B
        key: pwm_freq_b_khz
        unit: kHz
        validators:
          - {operator: ">=", expected_value: 21.0}
          - {operator: "<=", expected_value: 28.0}

  - name: Airflow and Static Pressure
    key: airflow_and_pressure
    python: phases.airflow_and_pressure
    depends_on: [tach_integrity]
    timeout: 5m
    measurements:
      - name: Fan Curve
        key: fan_curve
        title: Airflow and static pressure at 100 percent duty, damper from open to shut
        x_axis:
          legend: Damper opening
          unit: "%"
        y_axis:
          - legend: Airflow
            key: airflow
            unit: CFM
            aggregations:
              - type: free_air_cfm
                unit: CFM
                validators:
                  - {operator: ">=", expected_value: 157.5}
          - legend: Static pressure
            key: static_pressure
            unit: Pa
            aggregations:
              - type: shutoff_pa
                unit: Pa
                validators:
                  - {operator: ">=", expected_value: 810.0}
              - type: at_design_cfm_pa
                unit: Pa
                validators:
                  - {operator: ">=", expected_value: 450.0}

  - name: Vibration and Acoustic
    key: vibration_and_acoustic
    python: phases.vibration_and_acoustic
    depends_on: [airflow_and_pressure]
    measurements:
      - name: Vibration Spectrum
        key: spectrum
        title: Sled acceleration spectrum at 100 percent duty, 20 kHz capture
        x_axis:
          legend: Frequency
          unit: Hz
        y_axis:
          - legend: Amplitude
            key: amplitude
            unit: g
            aggregations:
              - type: one_x_a_g
                unit: g
                validators:
                  - {operator: "<=", expected_value: 1.0}
              - type: one_x_b_g
                unit: g
                validators:
                  - {operator: "<=", expected_value: 1.0}
              - type: bearing_a_g
                unit: g
                validators:
                  - {operator: "<=", expected_value: 0.15}
              - type: bearing_b_g
                unit: g
                validators:
                  - {operator: "<=", expected_value: 0.15}
      - name: Sound Pressure at 1 m
        key: sound_pressure_dba
        unit: dBA
        description: "ISO 7779 / ECMA-74 method, 1 m on the inlet axis, hemi-anechoic booth. Datasheet 71 dBA plus 3 dB, this line's derivation."
        validators:
          - {operator: "<=", expected_value: 74.0}

teardown:
  - name: Park and Cycles
    key: park_and_cycles
    python: phases.park_and_cycles
    measurements:
      - name: Current After Park
        key: current_after_park_a
        unit: A
        description: Supply off, nothing back-feeds the tray.
        validators:
          - {operator: "<=", expected_value: 0.01}
      - name: Power Cycles Applied
        key: power_cycles_applied
        description: "Two in this procedure, against the 7 500 on/off cycles of Intel 5.1.4; the count goes on the unit."
        validators:
          - {operator: "<=", expected_value: 3}
```

Framework features to notice:

1. **Operator answers as validated measurements.** The setup phase's `ui:` carries a radio bound to `measurements.latch_airflow` and a switch bound to `measurements.grille_seated`; both validate like any other measurement (`== front-to-back`, `== true`), and the run records what the operator actually answered. Headless runs take the two answers from `ui.json` through `--ui-values`.
2. **A previous result as a parameter.** `startup_current` names `pwm_rpm_curve` as a function parameter and reads `full_speed_a_rpm` and `full_speed_b_rpm` from it, so the spin-up time is measured against this tray's own full speed, not the datasheet's.
3. **`scope: station` for the chamber.** The `FlowBench` instance survives from one tray to the next: the damper stays homed and the connector mating counter keeps counting. In a one-shot `tofupilot run` it behaves as execution scope, which is why the mock reports a mating count of one.
4. **Four curves, four aggregation sets.** `pwm_curve` carries two speed axes judged on `max_dev_pct` and `min_speed_pct` and two current axes judged on `at_100_pct_a`; `startup` carries the current judged on `peak_a` and both tachs judged on `t90_s`; `fan_curve` and `spectrum` each judge one waveform on two to four numbers while the whole trace stays attached.
5. **A whole object, compared once.** `fru_identity` validates with one `==`; the report shows which key differed. `tach_pulses_per_rev_a` is an integer `== 2`.
6. **`teardown:` writes the record.** `unit.metadata["power_cycles"]`, `run.metadata["fixture_matings"]` and `attach.data(...)` of the raw accelerometer capture the bench plug kept, whatever a main phase did.

### Identify and Insulation

The tray is seated, the connector mated, and the safety analyzer runs with the supply off: all lines tied together against the sled at 500 Vdc for the insulation resistance, then 500 Vac for the dielectric leakage. The mock returns the dielectric reading without the 60 s dwell and the log says so. Then the first power-on at 0 % duty and the FRU EEPROM read: part number, hardware revision, airflow variant and format as one object, the serial against the label. The operator's two answers arrive through the bindings; the phase does not touch them.

```python filename="phases/identify_and_insulation.py"
from utils.recipe import DIELECTRIC_DWELL_S, DIELECTRIC_VAC, IR_TEST_VDC, SUPPLY_V


def identify_and_insulation(measurements, bench, flow_bench, unit, run, log):
    """Setup: the tray is seated on the chamber's inlet plate, the safety
    analyzer tests the lines against the sled with the supply off, then
    the first power-on reads the FRU EEPROM through the hot-swap
    connector. The operator's answers on the latch colour and the grille
    arrive as bound measurements. A tray with a pinched lead under the
    sled, or the wrong airflow variant, stops here before the rotors turn."""
    flow_bench.clamp()
    run.metadata["fixture_matings"] = flow_bench.mating_count()
    bench.supply_off()
    ir = bench.insulation_mohm(IR_TEST_VDC)
    leak = bench.dielectric_ma(DIELECTRIC_VAC, DIELECTRIC_DWELL_S)
    log.info(f"{unit.serial_number}: {ir:.0f} MOhm at {IR_TEST_VDC:.0f} Vdc, {leak:.3f} mA at {DIELECTRIC_VAC:.0f} Vac (mock: the {DIELECTRIC_DWELL_S:.0f} s dwell is not waited)")

    bench.supply_on(SUPPLY_V)
    bench.set_duty("a", 0.0)
    bench.set_duty("b", 0.0)
    fru = bench.fru_identity()
    serial = bench.fru_serial()
    measurements.insulation_mohm = ir
    measurements.dielectric_leakage_ma = leak
    measurements.fru_identity = fru
    measurements.eeprom_serial_matches_label = serial == unit.serial_number
    unit.metadata["hw_rev"] = fru["hw_rev"]
    log.info(f"FRU {fru['part_number']} rev {fru['hw_rev']} {fru['airflow']}, EEPROM serial {serial}, label {unit.serial_number}")
```

### PWM to RPM Curve

Both rotors are stepped through 20, 30, 50, 70 and 100 % duty at 12 V; at each point the tach gives the speed and the fixture's shunt the current on that rotor's feed. The spec's curve is the duty line: the speed as a percentage of the rotor's own full speed must match the duty within 10 points, and the 20 % point must sit at or below 30 %. The full speed itself is judged against the datasheet as a scalar, and the two full speeds against each other once normalised. A rotor still inside its own 10 % but 5 % behind its twin is dragging a blade or a bearing.

```python filename="phases/pwm_rpm_curve.py"
import numpy as np

from utils.recipe import DUTY_POINTS_PCT, NOMINAL_MAX_RPM, ROTORS, SETTLE_S, SUPPLY_V


def curve_stats(rotor, rpm, log):
    """Intel 3.3: the speed as a percentage of the rotor's own full speed
    must match the duty within 10 points; 3.2: the 20 % point at or below
    30 % of full speed."""
    pct_of_full = 100.0 * np.array(rpm) / rpm[-1]
    dev = pct_of_full - np.array(DUTY_POINTS_PCT)
    worst = dev[np.argmax(np.abs(dev))]
    log.info(f"Rotor {rotor.upper()}: worst deviation {worst:+.2f} points from the duty line, {pct_of_full[0]:.1f} % of full speed at {DUTY_POINTS_PCT[0]} % duty")
    return round(float(abs(worst)), 2), round(float(pct_of_full[0]), 1)


def pwm_rpm_curve(measurements, bench, log):
    """Both rotors through the duty points at 12 V, speed from each tach
    and current from each feed. Each rotor is judged against the duty line
    (speed in percent of its own full speed within 10 points of the duty,
    the 20 % point at or below 30 %), its full speed against the datasheet,
    and the two rotors against each other at 100 %: a rotor that lags its
    twin is loaded, even inside its own band."""
    bench.supply_on(SUPPLY_V)
    rpm = {r: [] for r in ROTORS}
    amps = {r: [] for r in ROTORS}
    for duty in DUTY_POINTS_PCT:
        for r in ROTORS:
            bench.set_duty(r, duty)
        bench.settle(SETTLE_S)
        for r in ROTORS:
            rpm[r].append(bench.tach_rpm(r))
            amps[r].append(bench.rotor_current_a(r))
        log.info(f"{duty:3d} %: A {rpm['a'][-1]:.0f} rpm {amps['a'][-1]:.2f} A, B {rpm['b'][-1]:.0f} rpm {amps['b'][-1]:.2f} A")

    dev_a, min_a = curve_stats("a", rpm["a"], log)
    dev_b, min_b = curve_stats("b", rpm["b"], log)
    m = measurements.pwm_curve
    m.x_axis = DUTY_POINTS_PCT
    m.y_axis.rpm_a = rpm["a"]
    m.y_axis.rpm_a.aggregations.max_dev_pct = dev_a
    m.y_axis.rpm_a.aggregations.min_speed_pct = min_a
    m.y_axis.rpm_b = rpm["b"]
    m.y_axis.rpm_b.aggregations.max_dev_pct = dev_b
    m.y_axis.rpm_b.aggregations.min_speed_pct = min_b
    m.y_axis.current_a = amps["a"]
    m.y_axis.current_a.aggregations.at_100_pct_a = amps["a"][-1]
    m.y_axis.current_b = amps["b"]
    m.y_axis.current_b.aggregations.at_100_pct_a = amps["b"][-1]

    full_a = rpm["a"][-1]
    full_b = rpm["b"][-1]
    mismatch = 100.0 * abs(full_a / NOMINAL_MAX_RPM["a"] - full_b / NOMINAL_MAX_RPM["b"])
    measurements.full_speed_a_rpm = full_a
    measurements.full_speed_b_rpm = full_b
    measurements.rotor_mismatch_pct = round(mismatch, 2)
    log.info(f"Full speed A {full_a:.0f} rpm, B {full_b:.0f} rpm, normalised mismatch {mismatch:.2f} %")
```

### Start-Up Current

Supply off, both duties at zero, then the supply at 13.2 V and both duties stepped to 100 % at once. The tray current and both tachs are recorded together from the supply's digitizer and the counters; the phase judges the peak against the scaled Intel limit, and the time each rotor takes to reach 90 % of the full speed the previous phase recorded, injected as the `pwm_rpm_curve` parameter.

```python filename="phases/startup_current.py"
import numpy as np

from utils.recipe import STARTUP_V, SUPPLY_V


def time_to_pct(t_s, rpm, target_rpm):
    """First sample at or above the target speed."""
    idx = np.where(np.asarray(rpm) >= target_rpm)[0]
    return float(t_s[idx[0]]) if len(idx) else float(t_s[-1])


def startup_current(measurements, bench, pwm_rpm_curve, log):
    """Cold start at 13.2 V, the supply's upper tolerance, both duties
    stepped from 0 to 100 % at once: the tray current recorded whole from
    the supply's digitizer, both tachs alongside. The peak is judged
    against the scaled Intel limit and each rotor against the time it
    takes to reach 90 % of the full speed the previous phase recorded."""
    bench.set_duty("a", 0.0)
    bench.set_duty("b", 0.0)
    bench.supply_off()
    t_s, amps, rpm_a, rpm_b = bench.startup_capture(STARTUP_V)
    bench.supply_on(SUPPLY_V)
    t = np.asarray(t_s)
    peak = float(np.max(amps))
    t90_a = time_to_pct(t, rpm_a, 0.9 * float(pwm_rpm_curve.full_speed_a_rpm))
    t90_b = time_to_pct(t, rpm_b, 0.9 * float(pwm_rpm_curve.full_speed_b_rpm))

    m = measurements.startup
    m.x_axis = t_s
    m.y_axis.current = amps
    m.y_axis.current.aggregations.peak_a = round(peak, 2)
    m.y_axis.rpm_a = rpm_a
    m.y_axis.rpm_a.aggregations.t90_s = round(t90_a, 3)
    m.y_axis.rpm_b = rpm_b
    m.y_axis.rpm_b.aggregations.t90_s = round(t90_b, 3)
    log.info(f"Start at {STARTUP_V} V: peak {peak:.2f} A at {t[int(np.argmax(amps))]:.3f} s, A at 90 % in {t90_a:.2f} s, B in {t90_b:.2f} s over {len(amps)} samples (mock: capture returned without waiting)")
```

![Tray current on the 0 to 100 percent step at 13.2 V, peaking at 12.26 A at 0.14 s and settling to 9.7 A under the 15.4 A limit line, with rotor A reaching 90 percent of full speed at 1.02 s and rotor B at 1.28 s against the 3 s limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/fan-tray-chart-startup-rM29ayYLsnjtyY6IUQDlKv8zOSogIo.png)

The mock tray's cold start: each drive holds its winding at twice rated current until the rotor's back-EMF builds, so the tray peaks at 12.3 A at 140 ms and settles to 9.7 A at 13.2 V, the current the spec's 1 s spike clause was written around for a fan a tenth this size. A rotor with a seized bearing holds its start-up current and never reaches 90 %; a drive with its current limit mis-set peaks above the line.

### Tach Integrity

At 50 % duty, each rotor's tach frequency is counted for one second next to the optical tach on its hub; the ratio is the pulses per revolution and must be exactly two. The tach duty is read from the counter and the PWM frequency at the rotor's pins, behind the tray's buffer, on the same DAQ. A tray whose buffer regenerates the PWM at the wrong frequency runs its rotors outside the curve the BMC expects.

```python filename="phases/tach_integrity.py"
from utils.recipe import ROTORS, SETTLE_S, SUPPLY_V, TACH_CHECK_DUTY_PCT


def tach_integrity(measurements, bench, log):
    """At 50 % duty, each rotor's tach frequency against the fixture's
    optical tach on the hub: pulses per revolution must be exactly two,
    the tach duty inside the window, and the PWM at the rotor pins, behind
    the tray's buffer, inside the Intel frequency window. A BMC counting
    a one-pulse tach reads the rotor at half speed and drives the whole
    chassis to full speed for nothing."""
    bench.supply_on(SUPPLY_V)
    for r in ROTORS:
        bench.set_duty(r, TACH_CHECK_DUTY_PCT)
    bench.settle(SETTLE_S)
    for r in ROTORS:
        tach_hz = bench.tach_frequency_hz(r)
        optical_rpm = bench.optical_rpm(r)
        pulses = tach_hz * 60.0 / optical_rpm
        duty = bench.tach_duty_pct(r)
        pwm_khz = bench.pwm_frequency_khz(r)
        log.info(f"Rotor {r.upper()}: tach {tach_hz:.1f} Hz, optical {optical_rpm:.0f} rpm, {pulses:.3f} pulses/rev, tach duty {duty:.1f} %, PWM {pwm_khz:.2f} kHz at the pins")
        if r == "a":
            measurements.tach_pulses_per_rev_a = int(round(pulses))
            measurements.tach_duty_a_pct = duty
            measurements.pwm_freq_a_khz = pwm_khz
        else:
            measurements.tach_pulses_per_rev_b = int(round(pulses))
            measurements.tach_duty_b_pct = duty
            measurements.pwm_freq_b_khz = pwm_khz
```

### Airflow and Static Pressure

Both rotors at 100 %, the damper walked from fully open to shut in seven steps, airflow from the nozzle bank and static pressure from the chamber tap at each. Free-air flow and shut-off pressure are judged against the datasheet less the tolerance; the curve is interpolated at the chassis design flow, the point the server actually needs. On a real chamber each point settles for seconds, which is what the phase's `timeout` is for.

```python filename="phases/airflow_and_pressure.py"
import numpy as np

from utils.recipe import CHASSIS_DESIGN_CFM, DAMPER_POINTS, ROTORS, SETTLE_S, SUPPLY_V


def airflow_and_pressure(measurements, bench, flow_bench, log):
    """Both rotors at 100 % on the chamber, the damper walked from fully
    open to shut: airflow from the nozzle bank and static pressure from
    the chamber tap at each point. Free-air flow and shut-off pressure
    are judged against the datasheet, and the curve is interpolated at
    the chassis design flow, the point the server needs. On a real
    chamber each point settles for seconds; the mock returns at once."""
    bench.supply_on(SUPPLY_V)
    for r in ROTORS:
        bench.set_duty(r, 100.0)
    bench.settle(SETTLE_S)
    cfm, pa = flow_bench.fan_curve(DAMPER_POINTS)
    opening = np.linspace(100.0, 0.0, DAMPER_POINTS).round(1).tolist()
    # the curve is sampled with flow decreasing; interp wants it increasing
    at_design = float(np.interp(CHASSIS_DESIGN_CFM, cfm[::-1], pa[::-1]))
    for o, q, p in zip(opening, cfm, pa):
        log.info(f"Damper {o:5.1f} % open: {q:6.1f} CFM at {p:4.0f} Pa")

    m = measurements.fan_curve
    m.x_axis = opening
    m.y_axis.airflow = cfm
    m.y_axis.airflow.aggregations.free_air_cfm = cfm[0]
    m.y_axis.static_pressure = pa
    m.y_axis.static_pressure.aggregations.shutoff_pa = pa[-1]
    m.y_axis.static_pressure.aggregations.at_design_cfm_pa = round(at_design, 0)
    log.info(f"Free air {cfm[0]:.1f} CFM, shut-off {pa[-1]:.0f} Pa, {at_design:.0f} Pa at the chassis design point of {CHASSIS_DESIGN_CFM:.0f} CFM")
```

![Fan curve at 100 percent duty over seven damper points from 171.4 CFM in free air to 887 Pa at shut-off, against the 157.5 CFM and 810 Pa limits and the chassis design point of 100 CFM needing 450 Pa, where the curve gives 549 Pa.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/fan-tray-chart-fan-curve-NOZihvZYBvIo6BZuc5gRwBgViewwlH.png)

The mock tray's fan curve: 171.4 CFM in free air, 887 Pa at shut-off, 549 Pa at the 100 CFM the chassis draws. A rotor with a bent blade loses the shut-off end first; a tray with a guard that is not seated loses the free-air end.

### Vibration and Acoustic

One accelerometer capture on the sled at full speed, 0.5 s at 20 kHz, the spectrum by FFT with a Hann window. The shaft frequency of each rotor comes from its tach, so the 1x tone is read where it is, not where the datasheet says; the ball-pass frequencies of the outer and inner race come from the bearing geometry in the recipe. The blade-pass tone at seven times the shaft frequency is logged and not judged: it is the aerodynamics, not a defect. Then the sound level meter at 1 m.

```python filename="phases/vibration_and_acoustic.py"
import numpy as np

from utils.recipe import BLADES, ROTORS, SETTLE_S, VIB_SAMPLE_HZ, VIB_SPECTRUM_MAX_HZ, bearing_defect_frequencies


def peak_near(freqs, spectrum, target_hz, tolerance_bins=3):
    """Largest bin within a few bins of the target, so a speed that drifts
    a few rpm during the capture still lands on the tone."""
    idx = int(np.argmin(np.abs(freqs - target_hz)))
    lo = max(0, idx - tolerance_bins)
    hi = min(len(spectrum), idx + tolerance_bins + 1)
    return float(np.max(spectrum[lo:hi]))


def vibration_and_acoustic(measurements, bench, log):
    """Both rotors at 100 %, one accelerometer capture on the sled, the
    spectrum by FFT. The 1x tone of each rotor, at the shaft frequency
    the tach gives, is the imbalance; the ball-pass tones of each rotor's
    bearing are the bearing check. The blade-pass tone is logged, not
    judged: it is the fan's aerodynamics, not a defect. Then the
    A-weighted sound pressure at 1 m on the inlet axis."""
    bench.settle(SETTLE_S)
    samples = np.asarray(bench.vibration_capture())
    window = np.hanning(len(samples))
    spectrum = np.abs(np.fft.rfft(samples * window)) * 2.0 / np.sum(window)
    freqs = np.fft.rfftfreq(len(samples), 1.0 / VIB_SAMPLE_HZ)
    one_x = {}
    bearing = {}
    for r in ROTORS:
        f_r = bench.tach_rpm(r) / 60.0
        d = bearing_defect_frequencies(f_r)
        one_x[r] = peak_near(freqs, spectrum, f_r)
        bpfo = peak_near(freqs, spectrum, d["bpfo"])
        bpfi = peak_near(freqs, spectrum, d["bpfi"])
        bearing[r] = max(bpfo, bpfi)
        blade = peak_near(freqs, spectrum, BLADES * f_r)
        log.info(f"Rotor {r.upper()} at {f_r * 60:.0f} rpm: 1x {one_x[r]:.3f} g at {f_r:.0f} Hz, BPFO {bpfo:.3f} g at {d['bpfo']:.0f} Hz, BPFI {bpfi:.3f} g at {d['bpfi']:.0f} Hz, blade pass {blade:.3f} g at {BLADES * f_r:.0f} Hz")
    spl = bench.sound_pressure_dba()

    keep = freqs <= VIB_SPECTRUM_MAX_HZ
    m = measurements.spectrum
    m.x_axis = freqs[keep].round(1).tolist()
    m.y_axis.amplitude = spectrum[keep].round(4).tolist()
    m.y_axis.amplitude.aggregations.one_x_a_g = round(one_x["a"], 3)
    m.y_axis.amplitude.aggregations.one_x_b_g = round(one_x["b"], 3)
    m.y_axis.amplitude.aggregations.bearing_a_g = round(bearing["a"], 3)
    m.y_axis.amplitude.aggregations.bearing_b_g = round(bearing["b"], 3)
    measurements.sound_pressure_dba = spl
    log.info(f"{spl:.1f} dBA at 1 m, {int(keep.sum())} bins to {VIB_SPECTRUM_MAX_HZ:.0f} Hz")
```

![Sled acceleration spectrum at 100 percent duty to 2.1 kHz: the 1x tones of rotor A at 270 Hz (0.418 g) and rotor B at 240 Hz (0.338 g) under the 1.0 g limit segments, the ball-pass tones under 0.05 g against the 0.15 g limit segments, and the blade-pass tones near 1.7 and 1.9 kHz marked as logged only.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/fan-tray-chart-spectrum-IZGJtuiIxu9RA3J9HfDxaSCwocjXGP.png)

The mock tray's spectrum: two 1x tones under half the limit, ball-pass tones of new bearings at 0.04 g, the two blade-pass tones above the bearing limit and ignored by design. A dented outer race puts a tone at 2.38 times the shaft frequency that grows week by week in the field; this phase catches it at 0.15 g on the line, and the raw capture attached at teardown lets it be re-analysed later with a different band.

### Park and Cycles

The teardown phase sets both duties to zero and switches the supply off, releases the tray from the chamber plate, and records what the test did to the unit: the power cycles applied, against the 7 500 of the fan spec's 5.1.4, go on the unit; the fixture's mating count goes on the run, so the connector on the fixture is replaced on a count and not on a failure; and the raw accelerometer capture the bench plug kept is attached as CSV.

```python filename="phases/park_and_cycles.py"
import io

import numpy as np

from utils.recipe import ROTORS


def park_and_cycles(measurements, bench, flow_bench, unit, run, attach, log):
    """Teardown: PWM to 0 on both rotors, supply off, the tray released
    from the chamber plate, whatever happened before. The power cycles
    this test applied go on the unit against its 7 500-cycle budget, the
    fixture's mating count on the run for connector wear, and the raw
    accelerometer capture is attached so a bearing can be re-analysed
    later without the tray."""
    for r in ROTORS:
        bench.set_duty(r, 0.0)
    bench.supply_off()
    residual = bench.supply_current_a()
    flow_bench.release()
    cycles = bench.power_cycles()
    matings = flow_bench.mating_count()
    measurements.current_after_park_a = residual
    measurements.power_cycles_applied = cycles
    unit.metadata["power_cycles"] = cycles
    run.metadata["fixture_matings"] = matings

    raw = bench.last_vibration_capture()
    if raw:
        buf = io.StringIO()
        np.savetxt(buf, np.asarray(raw["samples"]), fmt="%.5f", header=f"acceleration_g at {raw['sample_rate_hz']:.0f} Hz", comments="# ")
        attach.data(buf.getvalue().encode(), "vibration_raw_20khz.csv")
    log.info(f"Parked: {residual:.4f} A with the supply off, {cycles} power cycles applied, fixture mating {matings}, raw capture {'attached' if raw else 'absent'}")
```

### Mock Plugs

`FanBench` stands in for six instruments and the DUT because every reading depends on the duty and the supply voltage at that instant, and plugs run in separate processes. Its rotors follow the duty line with a 4 point lift at 20 % duty, the inlet rotor 1.2 % fast and the outlet rotor 0.6 % slow on full speed, its currents follow speed to the power 2.5 from a 0.15 A idle, its cold start holds each winding at twice rated current until the back-EMF builds, its tachs are clean two-pulse square waves, its spectrum carries the 1x, 2x, blade-pass and ball-pass tones of each rotor on 0.02 g of noise, and it keeps the last capture for the teardown. `FlowBench` holds the chamber at `scope: station`: a fan curve from 172 CFM in free air to 885 Pa at shut-off, and a mating counter that increments on every clamp. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, split `FanBench` into the supply over SCPI, the DAQ through nidaqmx for the PWM outputs, the tach and optical inputs and the IEPE channel, the sound level meter over its API, the safety analyzer over SCPI and the FRU EEPROM over smbus; keep `FlowBench` on the chamber's serial protocol and at station scope. Interlock the safety analyzer against the supply, let each damper point settle before reading, and wait the full 60 s dielectric dwell. The phases, measurements and limits stay the same.


### GPU Baseboard Diagnostic Ladder
URL: https://www.tofupilot.com/templates/gpu-baseboard-diagnostic-ladder
L10 diagnostic ladder for an 8-GPU HGX baseboard: DCGM levels 1 to 4 as executable phases, targeted power to TDP, NCCL all-reduce, HBM memtest, 24 h soak.

## Introduction

### GPU Baseboard Overview

An AI server in the HGX or OCP OAM form factor carries its eight accelerators on one baseboard: eight GPU modules on a universal baseboard (UBB) joined by NVLink switches, with the PCIe links to the host CPUs, the power sequencing for each module and the thermal interface to the cold plates or the heatsinks all on that one board. The baseboard reaches the integrator as an assembled part and is built into a node through the L6 to L9 levels; at L10 the node boots an operating system and is tested as a system for the first time. That is where a module with a marginal HBM stack, an NVLink lane that trains at reduced width, a cold plate that is not seated, or a PCIe slot that enumerates at gen4 is found, before the node is racked at L11 and cabled into a cluster whose all-reduce runs at the speed of its slowest link.

![SXM GPU module with a central GPU and HBM package, surrounding power components, and mounting holes.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/gpu-baseboard-diagnostic-ladder-matte-closeup-TKM1tVjILbFAx5Dc02Q3B7e3zRJabG.webp)

The parts this test exercises: each module's HBM stacks and the ECC and row-remapping logic in front of them, the PCIe link to the host, the NVLink links between modules, the voltage regulators that must hold TDP for as long as the workload asks, and the thermal path from the die to the coolant.

The test flow is NVIDIA's [Data Center GPU Manager diagnostics](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/dcgm-diagnostics.html), a ladder of four levels invoked as `dcgmi diag -r <level>`: **level 1** is software only (driver, NVML, CUDA libraries, persistence mode, denylist) and runs in seconds; **level 2** adds PCIe and NVLink, a memory allocation test and a memory bandwidth test; **level 3** adds the diagnostic matrix, targeted stress, **targeted power** that drives each GPU to a power target, nvbandwidth, nccl_tests and a short EUD; **level 4** adds the **HBM memtest patterns**, the pulse test that exercises the voltage regulators, and the full EUD, at about two hours for eight GPUs. DCGM's example configuration for an H100 PCIe card sets a `target_power` of 350 W, a memory bandwidth minimum of 1,230,000 MB/s and a PCIe link of at least gen3 x16; those are the only numbers NVIDIA publishes as limits, and they are for a different module than the SXM-class board here. The [OCP UBB r2.0 base specification](https://www.opencompute.org/documents/oai-ubb-base-specification-r2-0-v1-0-20230919-pdf) defines the power sequencing signals of the baseboard (PWREN, PWROK, PWRBRK#, THERMTRIP#) but no test flow. The acceptance rules that integrators apply on top, **zero uncorrectable ECC, a ceiling on correctable ECC and on row-remap events during the soak, and a soak of 24 to 72 hours at elevated intake temperature**, are practice, not a standard. Every limit in this template is DCGM's example, the board's golden configuration, or this line's derivation, and the procedure file says which.

### Test Purpose

The procedure records one diagnostic fingerprint per baseboard:

- **DCGM level 1** as an executable phase whose exit code is the result, nothing to parse
- **Inventory**: eight GPUs enumerated, PCIe generation and width per GPU, the firmware set as one object, NVLink links up per GPU, the baseboard serial against the label, and the ECC counters as a baseline
- **DCGM level 2** as an executable phase, then **memory bandwidth and PCIe bandwidth per GPU** parsed from its report and judged against this line's floors
- **DCGM level 3** with targeted_power for 300 s, then **power reached per GPU as a percentage of TDP**, the hottest sensor, the throttle count and the duration DCGM reports
- **nccl_tests all_reduce_perf** across the eight GPUs, the **bus bandwidth curve** from 8 B to 8 GiB judged at the top of the sweep against the line's reference
- **DCGM level 4** memtest, then the **ECC and row-remap counters minus the inventory baseline**
- A **24 hour soak** at 35 °C intake sampled hourly, the hottest GPU and the board power over time, per GPU the maximum temperature and the minimum SM clock, the **ECC counters minus the memtest phase's**, and zero XID in the kernel log
- A teardown that **attaches the four DCGM reports and the NCCL log**, stamps the eight GPU serials on the unit, and clears the volatile counters

![Memory bandwidth per GPU from the DCGM level 2 test as eight green bars between 3.07 and 3.13 TB/s, above the line's 2.85 TB/s floor drawn as a red dashed line and the DCGM example floor of 1.23 TB/s as an orange dotted line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/gpu-baseboard-chart-membw-adMhZ1u2Fo0XcaxdmtOSvcbBKKy9PS.png)

The mock board's level 2 result: eight modules between 3.07 and 3.13 TB/s, a 1.9 % spread, against the floor this line derives from the module's datasheet figure. The DCGM example floor for a PCIe card is drawn for scale; a board that passes it can still be a board with one module 20 % under its siblings.

The framework mechanics on show are executable phases judged on their exit code, one per DCGM level and one for nccl_tests, each chained through `depends_on` to a Python phase that parses the report the command wrote; previous-results injection, with the memtest phase receiving the inventory phase's ECC baseline and the soak receiving the memtest's counters; per-GPU multi-dimensional measurements with custom aggregations validated in the YAML; `attach.file` for the raw reports, `unit.metadata` for the GPU serials and `run.metadata` for the DCGM version and the soak length; and a `timeout` sized per level.

## Equipment & Setup

To run this ladder on an L10 line, the following are required:

- **The node under test**, the baseboard built into its chassis with CPUs, memory, NICs and cooling, booting the line's OS image with the driver, DCGM and nccl_tests installed
- **A test rack position** with the intake temperature controlled at 35 °C for the soak, and a metered rack PDU to log the node's power
- **A station computer** with SSH to the node and a route to the BMC, running the TofuPilot Framework station
- **The BMC on the management network**, for the baseboard serial, the intake temperature and the power state over Redfish
- **The Device Under Test (DUT)**: the 8-GPU baseboard, identified by the serial the BMC reports under the chassis
- **A TofuPilot Framework procedure** to run the levels in order and validate the limits
- **The TofuPilot Dashboard** to keep the per-board evidence, the raw DCGM reports and the ECC history of every module

### Hardware Components

#### Diagnostics on the Node

[NVIDIA DCGM](https://developer.nvidia.com/dcgm) ships `dcgmi`, whose `diag` command runs the four levels and writes a JSON report with `-j`; the test names in the report (PCIe, Memory Bandwidth, Targeted Power, Memtest) are the plugin names the parsers here key on, and `-p targeted_power.test_duration=300` sets the line's stress duration. [nccl_tests](https://github.com/NVIDIA/nccl-tests) provides `all_reduce_perf`, which prints one line per message size with the algorithm and bus bandwidth and the count of wrong values when validation is on. Both are invoked exactly as written, over SSH from the station or from a station process on the node itself, and redirected to a file under `results/` that the next phase reads.

![Eight-GPU baseboard with two rows of four heatsinks, interconnect heatsinks, and power connections.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/gpu-baseboard-diagnostic-ladder-matte-cover-z9Xf6ltr7Lolzyi1K35H3V2vmwjJiU.webp)

The illustrated hardware: an eight-GPU baseboard with its heatsinks and interconnects visible. Diagnostic run-in and soak tests use the complete node with the chassis closed.

#### Node Agent

Between the diagnostic runs the station reads the node: `nvidia-smi -q -x` for the inventory, the PCIe link state, the VBIOS and the ECC and row-remapper counters, `nvidia-smi nvlink -s` for the links up per GPU, the BMC's [Redfish](https://www.dmtf.org/standards/redfish) API for the baseboard serial and the intake sensor, and `journalctl -k` for XID events. The DCGM Python bindings can replace the nvidia-smi calls on a line that prefers a library to a parsed command.

### Where the Limits Come From

| Test | Limit | Basis |
|---|---|---|
| GPU count, PCIe link | 8 GPUs, gen5 x16 on every module | the board's golden configuration; DCGM's example asks for gen3 x16 at least |
| Firmware | driver, VBIOS, DCGM as one object, exact | the release; no tolerance applies |
| NVLink | 18 links up per GPU | the board's topology |
| Memory bandwidth | ≥ 2,850,000 MB/s per GPU, spread ≤ 5 % | this line's derivation: 85 % of the module's 3.35 TB/s datasheet figure; DCGM's example floor is 1,230,000 MB/s for an H100 PCIe card |
| PCIe bandwidth | ≥ 45 GB/s host to device | this line's, from the golden node's first article |
| Targeted power | ≥ 95 % of the 700 W power limit, 0 throttle events, 300 s | DCGM's test; the power limit is what nvidia-smi reports for the module; 95 % and 300 s are this line's |
| Temperature | ≤ 85 °C at the hottest sensor, stress and soak | this line's ceiling |
| NCCL all_reduce | bus bandwidth ≥ 432 GB/s at the top of the sweep | 90 % of this line's 480 GB/s reference, the median of its golden nodes; the 90 % rule is integrator practice |
| Memtest | 0 errors per GPU | DCGM's test |
| ECC | uncorrectable 0, rows pending remap 0, correctable ≤ 10 per GPU per stage | integrator practice for the rule; the 10 is this line's |
| SM clock | ≥ 1500 MHz sustained under the soak load | this line's, from the golden node |
| Soak | 24 h at 35 °C intake, 0 XID | integrator range 24 to 72 h; this line runs 24 h |

Two rows deserve a note. The memory bandwidth floor is not DCGM's: the example configuration's 1,230,000 MB/s is for a PCIe card with a third of this module's HBM bandwidth, and a board judged on it passes with a module that has lost a stack. And the NCCL reference is not a theoretical number: NVLink's raw rate is not what a ring all-reduce achieves, so the line measures its first golden nodes, takes the median, and judges every board at 90 % of that.

## Test Procedure

### Overview

The ladder maps onto the framework's three stages. Level 1 and the inventory live in `setup:` so no stress runs on a node whose driver is broken or whose topology is wrong; the inventory waits on level 1 through `depends_on`. Each main level is two phases, the command and its parser, chained one after the other on the single node. The teardown attaches the reports and clears the counters whatever a main phase did.

1. Setup: `dcgmi diag -r 1`, exit code only.
2. Setup: inventory, PCIe link, firmware, NVLink, serial, ECC baseline.
3. Main: `dcgmi diag -r 2`, then memory and PCIe bandwidth per GPU parsed.
4. Main: `dcgmi diag -r 3` with targeted_power for 300 s, then power, temperature and throttles parsed.
5. Main: `all_reduce_perf` across the 8 GPUs, then the bus bandwidth curve parsed.
6. Main: `dcgmi diag -r 4`, then memtest results and ECC minus the baseline.
7. Main: 24 h soak sampled hourly, ECC minus the memtest counters, XID.
8. Teardown: reports attached, GPU serials on the unit, counters cleared.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-board evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  inventory.py
  dcgm_r2.py
  dcgm_r3_targeted_power.py
  nccl_allreduce.py
  hbm_memtest.py
  soak.py
  clear_and_report.py
plugs/
  node_agent.py
  mock_node_state.py
scripts/
  dcgm_diag_mock.py
  nccl_allreduce_mock.py
utils/
  recipe.py
  dcgm.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-gpu-baseboard-diag). The `NodeAgent` plug is a mock of nvidia-smi, the BMC and the kernel log, and the two scripts under `scripts/` stand in for `dcgmi diag` and `all_reduce_perf`: they print DCGM-shaped JSON and the all_reduce_perf table and exit 0, so the executable phases judge a real exit code and the parsers read real files. The mock synthesizes a healthy board with eight modules between 3.07 and 3.13 TB/s, 98.5 % of TDP reached, 462 GB/s of all-reduce bus bandwidth, and a few correctable ECC events on GPU 4. Time is scaled, one hour of the real ladder to one second, so the procedure runs end to end in under a minute without a node connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the node plug with the label serial as constructor argument, and the three stages with every command, measurement and limit:

```yaml filename="procedure.yaml"
name: GPU Baseboard Diagnostic Ladder
version: 0.1.0
description: L10 diagnostic ladder for an 8-GPU HGX / OAM-UBB baseboard in its server node, wrapped around NVIDIA DCGM. Inventory and NVLink topology against the golden configuration, dcgmi diag levels 1 to 4 as executable phases judged on their exit code with Python phases that parse the JSON reports, targeted_power to TDP, nccl_tests all_reduce across the fabric, the HBM memtest patterns with the ECC counters subtracted from the inventory baseline, a 24 hour soak at elevated intake with the counters subtracted again, and a teardown that attaches every raw report and clears the counters.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the data-matrix on the baseboard's edge, or read it from the BMC"
    placeholder: "BB8-000000-0000"
    pattern: "^BB8-\\d{6}-\\d{4}$"
    default_value: "BB8-260922-0417"
  part_number:
    default_value: "GPU-BASEBOARD-8X-HBM3"
  batch_number:
    default_value: "LOT-2026-38"

plugs:
  - name: Node Agent
    description: "nvidia-smi, the BMC's Redfish API and the kernel log of the node under test (mock, one plug per station)"
    python: plugs.node_agent:NodeAgent
    key: node
    config:
      serial_number: "BB8-260922-0417"

setup:
  - name: DCGM Level 1
    key: dcgm_r1
    description: "Software checks only: driver, NVML, CUDA libraries, persistence mode, denylist. Nothing to parse; the exit code is the result. On a real node the command is dcgmi diag -r 1 -j."
    executable:
      command: "mkdir -p results && python3 scripts/dcgm_diag_mock.py -r 1 -j > results/dcgm_r1.json"
    timeout: 2m

  - name: Inventory
    key: inventory
    python: phases.inventory
    depends_on: [dcgm_r1]
    measurements:
      - name: GPU Count
        key: gpu_count
        description: GPUs nvidia-smi enumerates on the baseboard.
        validators:
          - {operator: "==", expected_value: 8}
      - name: PCIe Link
        key: pcie_link
        title: PCIe link generation and width per GPU
        x_axis:
          legend: GPU
        y_axis:
          - legend: Generation
            key: gen
            aggregations:
              - type: min_gen
                validators:
                  - {operator: "==", expected_value: 5}
          - legend: Width
            key: width
            unit: lanes
            aggregations:
              - type: min_width
                validators:
                  - {operator: "==", expected_value: 16}
      - name: Firmware
        key: firmware
        description: "Driver, VBIOS and DCGM version as one object against the release. DCGM's example configuration asks for gen3 x16 at least; this board's golden configuration is gen5 x16."
        validators:
          - operator: "=="
            expected_value:
              driver: 570.86.15
              vbios: 96.00.89.00.01
              dcgm: 3.3.9
      - name: NVLink
        key: nvlink
        title: NVLink links up per GPU
        x_axis:
          legend: GPU
        y_axis:
          - legend: Links up
            key: links_up
            aggregations:
              - type: min_links
                validators:
                  - {operator: "==", expected_value: 18}
      - name: All NVLinks Up
        key: all_nvlinks_up
        description: Every GPU reports the full link count of this board's topology.
        validators:
          - {operator: "==", expected_value: true}
      - name: Serial Matches Label
        key: serial_matches_label
        description: The baseboard serial the BMC reports is the one on the label.
        validators:
          - {operator: "==", expected_value: true}
      - name: ECC Baseline
        key: ecc_baseline
        description: "Volatile ECC and row-remap counters per GPU before any stress, kept for the memtest phase to subtract."

main:
  - name: DCGM Level 2
    key: dcgm_r2_run
    description: "PCIe, NVLink, memory allocation and memory bandwidth. Executable phase, judged on the exit code; the report is parsed by the next phase. On a real node the command is dcgmi diag -r 2 -j."
    executable:
      command: "python3 scripts/dcgm_diag_mock.py -r 2 -j > results/dcgm_r2.json"
    timeout: 15m

  - name: Level 2 Bandwidth
    key: dcgm_r2
    python: phases.dcgm_r2
    depends_on: [dcgm_r2_run]
    measurements:
      - name: DCGM Level 2 All Pass
        key: dcgm_r2_all_pass
        validators:
          - {operator: "==", expected_value: true}
      - name: Memory Bandwidth
        key: memory_bandwidth
        title: HBM bandwidth per GPU from the DCGM memory_bandwidth test
        x_axis:
          legend: GPU
        y_axis:
          - legend: Memory bandwidth
            key: mb_s
            unit: MB/s
            aggregations:
              - type: min_mb_s
                unit: MB/s
                validators:
                  - {operator: ">=", expected_value: 2850000}
              - type: spread_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 5.0}
      - name: PCIe Bandwidth
        key: pcie_bandwidth
        title: Host-to-device bandwidth per GPU from the DCGM PCIe test
        x_axis:
          legend: GPU
        y_axis:
          - legend: Host to device
            key: h2d_gb_s
            unit: GB/s
            aggregations:
              - type: min_gb_s
                unit: GB/s
                validators:
                  - {operator: ">=", expected_value: 45.0}

  - name: DCGM Level 3
    key: dcgm_r3_run
    description: "Targeted stress and targeted_power for the line's 300 s. On a real node the command is dcgmi diag -r 3 -j -p targeted_power.test_duration=300."
    executable:
      command: "python3 scripts/dcgm_diag_mock.py -r 3 -j -p targeted_power.test_duration=300 > results/dcgm_r3.json"
    depends_on: [dcgm_r2]
    timeout: 30m

  - name: Level 3 Targeted Power
    key: dcgm_r3_targeted_power
    python: phases.dcgm_r3_targeted_power
    depends_on: [dcgm_r3_run]
    measurements:
      - name: DCGM Level 3 All Pass
        key: dcgm_r3_all_pass
        validators:
          - {operator: "==", expected_value: true}
      - name: Targeted Power
        key: targeted_power
        title: Power reached and hottest sensor per GPU at the TDP target
        x_axis:
          legend: GPU
        y_axis:
          - legend: Achieved power
            key: achieved_w
            unit: W
            aggregations:
              - type: min_pct_of_tdp
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 95.0}
              - type: min_w
                unit: W
          - legend: Max temperature
            key: max_temp_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 85.0}
      - name: Throttle Events
        key: throttle_events
        description: "Clock throttle events across the 8 GPUs during targeted_power. A module that reaches TDP by throttling has a cold plate that is not seated."
        validators:
          - {operator: "==", expected_value: 0}
      - name: Targeted Power Duration
        key: targeted_power_duration_s
        unit: s
        description: "The duration DCGM reports, which must be the one asked for."
        validators:
          - {operator: "==", expected_value: 300}

  - name: NCCL All-Reduce
    key: nccl_run
    description: "nccl_tests all_reduce_perf across the 8 GPUs, 8 B to 8 GiB doubling, with validation. On a real node the command is all_reduce_perf -b 8 -e 8G -f 2 -g 8 -c 1."
    executable:
      command: "python3 scripts/nccl_allreduce_mock.py -b 8 -e 8G -f 2 -g 8 -c 1 > results/nccl_allreduce.txt"
    depends_on: [dcgm_r3_targeted_power]
    timeout: 15m

  - name: NCCL Bus Bandwidth
    key: nccl_allreduce
    python: phases.nccl_allreduce
    depends_on: [nccl_run]
    measurements:
      - name: All-Reduce Bus Bandwidth
        key: allreduce_busbw
        title: all_reduce_perf bus and algorithm bandwidth against message size
        x_axis:
          legend: Message size
          unit: B
        y_axis:
          - legend: Bus bandwidth
            key: busbw_gb_s
            unit: GB/s
            aggregations:
              - type: peak_gb_s
                unit: GB/s
                validators:
                  - {operator: ">=", expected_value: 432.0}
              - type: peak_pct_of_ref
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 90.0}
          - legend: Algorithm bandwidth
            key: algbw_gb_s
            unit: GB/s
      - name: NCCL Wrong Values
        key: nccl_wrong_values
        description: Elements the validation pass found wrong, out-of-place and in-place summed over the sweep.
        validators:
          - {operator: "==", expected_value: 0}

  - name: DCGM Level 4
    key: dcgm_r4_run
    description: "HBM memtest patterns and the pulse test, about two hours on 8 GPUs. On a real node the command is dcgmi diag -r 4 -j."
    executable:
      command: "python3 scripts/dcgm_diag_mock.py -r 4 -j > results/dcgm_r4.json"
    depends_on: [nccl_allreduce]
    timeout: 3h

  - name: HBM Memtest and ECC
    key: hbm_memtest
    python: phases.hbm_memtest
    depends_on: [dcgm_r4_run]
    measurements:
      - name: DCGM Level 4 All Pass
        key: dcgm_r4_all_pass
        validators:
          - {operator: "==", expected_value: true}
      - name: Memtest
        key: memtest
        title: Memtest errors and duration per GPU
        x_axis:
          legend: GPU
        y_axis:
          - legend: Errors
            key: errors
            aggregations:
              - type: max_errors
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: Duration
            key: duration_s
            unit: s
            aggregations:
              - type: max_s
                unit: s
      - name: ECC After Memtest
        key: ecc_after_memtest
        title: ECC and row-remap counters per GPU, memtest minus inventory baseline
        x_axis:
          legend: GPU
        y_axis:
          - legend: Correctable
            key: correctable_delta
            aggregations:
              - type: max_per_gpu
                validators:
                  - {operator: "<=", expected_value: 10}
          - legend: Uncorrectable
            key: uncorrectable_delta
            aggregations:
              - type: max_per_gpu
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: Rows pending remap
            key: row_remap_pending
            aggregations:
              - type: max_per_gpu
                validators:
                  - {operator: "==", expected_value: 0}
      - name: ECC Counters After Memtest
        key: ecc_counters_after_memtest
        description: "The counters as read after the memtest, kept for the soak phase to subtract."

  - name: Soak
    key: soak
    python: phases.soak
    depends_on: [hbm_memtest]
    measurements:
      - name: Soak Trace
        key: soak_trace
        title: Hottest GPU, intake temperature and board power over the soak
        x_axis:
          legend: Time
          unit: h
        y_axis:
          - legend: Hottest GPU
            key: hottest_gpu_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 85.0}
          - legend: Intake
            key: intake_c
            unit: °C
            aggregations:
              - type: mean_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: 33.0}
                  - {operator: "<=", expected_value: 37.0}
          - legend: Board power
            key: board_power_kw
            unit: kW
            aggregations:
              - type: mean_kw
                unit: kW
      - name: Soak Per GPU
        key: soak_per_gpu
        title: Per-GPU maximum temperature, minimum SM clock and ECC counters over the soak
        x_axis:
          legend: GPU
        y_axis:
          - legend: Max temperature
            key: max_temp_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 85.0}
          - legend: Min SM clock
            key: min_sm_clock_mhz
            unit: MHz
            aggregations:
              - type: min_mhz
                unit: MHz
                validators:
                  - {operator: ">=", expected_value: 1500}
          - legend: Correctable ECC
            key: correctable_delta
            aggregations:
              - type: max_per_gpu
                validators:
                  - {operator: "<=", expected_value: 10}
          - legend: Uncorrectable ECC
            key: uncorrectable_delta
            aggregations:
              - type: max_per_gpu
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: Rows pending remap
            key: row_remap_pending
            aggregations:
              - type: max_per_gpu
                validators:
                  - {operator: "==", expected_value: 0}
      - name: XID Events
        key: xid_events
        description: XID lines in the kernel log over the soak. Any XID is a reason to look, whatever the counters say.
        validators:
          - {operator: "==", expected_value: 0}

teardown:
  - name: Clear and Report
    key: clear_and_report
    python: phases.clear_and_report
    measurements:
      - name: ECC Counters Cleared
        key: ecc_counters_cleared
        description: Volatile counters at zero after the reset, so the first event the customer reads happened in their rack.
        validators:
          - {operator: "==", expected_value: true}
      - name: Reports Attached
        key: reports_attached
        description: The four DCGM reports and the NCCL log on the run.
        validators:
          - {operator: "==", expected_value: 5}
```

Framework features to notice:

1. **Executable phases judged on the exit code.** `dcgm_r1`, `dcgm_r2_run`, `dcgm_r3_run`, `nccl_run` and `dcgm_r4_run` have no Python: `executable.command` runs through `sh`, stdout and stderr land in the phase log, and a non-zero exit fails the phase. Level 1 is nothing but that.
2. **A parser chained to each command.** `dcgm_r2`, `dcgm_r3_targeted_power`, `nccl_allreduce` and `hbm_memtest` declare `depends_on` on their executable and read the file it wrote. The command is the only line that changes on a real node.
3. **Previous-results injection.** `hbm_memtest(measurements, node, inventory, log)` receives the inventory phase's results by naming the phase key as a parameter and subtracts its `ecc_baseline`; `soak` does the same with `hbm_memtest.ecc_counters_after_memtest`. Each stage's ECC ceiling applies to that stage alone.
4. **Per-GPU measurements with custom aggregations.** Every per-GPU result is one multi-dimensional measurement with the GPU index on the x-axis: `pcie_link` judged on `min_gen` and `min_width`, `memory_bandwidth` on `min_mb_s` and `spread_pct`, `targeted_power` on `min_pct_of_tdp` and `max_c`, `ecc_after_memtest` and `soak_per_gpu` on `max_per_gpu`.
5. **Attachments and metadata.** The teardown calls `attach.file` on the four DCGM reports and the NCCL log, writes the eight GPU serials into `unit.metadata`, and the inventory writes the DCGM version, the soak length and the mock time scale into `run.metadata`.
6. **A `timeout` per level.** Level 4 is allowed 3 hours because DCGM's own figure is about two hours on eight GPUs; the soak carries no timeout because it is longer than the 24 h cap and controls its own duration.

### DCGM Level 1 and Inventory

Level 1 is the only phase in the procedure with no measurement: the software checks pass or fail, and `sh` reports the exit code. The inventory that follows reads what the node says it is, keeps the PCIe link and the NVLink count per GPU as multi-dimensional measurements judged on their minimum, compares the firmware set as one object, checks the BMC's serial against the label, and records the ECC counters as the baseline the memtest phase will subtract:

```python filename="phases/inventory.py"
from utils.recipe import GPU_COUNT, NVLINK_LINKS_PER_GPU, SOAK_HOURS, TIME_SCALE


def inventory(measurements, node, unit, run, log):
    """Setup: what the node says it is, before anything is stressed. Eight
    GPUs enumerated at the golden PCIe generation and width, the firmware
    set as one object against the release, every NVLink up, and the ECC
    counters as they stand, which the memtest phase later subtracts."""
    gpus = node.gpu_inventory()
    fw = node.firmware()
    links = node.nvlink_status()
    ecc = node.ecc_counters()
    serial = node.baseboard_serial()

    measurements.gpu_count = len(gpus)
    m = measurements.pcie_link
    m.x_axis = [g["index"] for g in gpus]
    m.y_axis.gen = [g["pcie_gen"] for g in gpus]
    m.y_axis.gen.aggregations.min_gen = min(g["pcie_gen"] for g in gpus)
    m.y_axis.width = [g["pcie_width"] for g in gpus]
    m.y_axis.width.aggregations.min_width = min(g["pcie_width"] for g in gpus)
    measurements.firmware = fw
    n = measurements.nvlink
    n.x_axis = [g["index"] for g in gpus]
    n.y_axis.links_up = links
    n.y_axis.links_up.aggregations.min_links = min(links)
    measurements.all_nvlinks_up = all(l == NVLINK_LINKS_PER_GPU for l in links)
    measurements.serial_matches_label = serial == unit.serial_number
    measurements.ecc_baseline = ecc

    unit.metadata["driver"] = fw["driver"]
    unit.metadata["vbios"] = fw["vbios"]
    run.metadata["dcgm_version"] = fw["dcgm"]
    run.metadata["soak_hours"] = SOAK_HOURS
    run.metadata["mock_time_scale"] = TIME_SCALE
    log.info(f"{serial}: {len(gpus)} of {GPU_COUNT} GPUs, PCIe gen {min(g['pcie_gen'] for g in gpus)} x{min(g['pcie_width'] for g in gpus)}, driver {fw['driver']}, {min(links)} NVLinks up per GPU, correctable ECC baseline {[v['correctable'] for v in ecc.values()]}")
```

### DCGM Level 2 and Bandwidth

`dcgmi diag -r 2 -j > results/dcgm_r2.json` runs PCIe, NVLink, memory allocation and memory bandwidth; the phase after it reads the report, keeps the memory bandwidth and the host-to-device bandwidth per GPU, and judges each on its minimum. The spread aggregation catches a board whose eight modules pass the floor but do not match each other:

```python filename="phases/dcgm_r2.py"
from utils.dcgm import all_passed, load_report, number_after, test_results
from utils.recipe import MEM_BW_FLOOR_MB_S, PCIE_BW_FLOOR_GB_S, RESULTS_DIR


def dcgm_r2(measurements, log):
    """Level 2 parsed: the report dcgm_r2_run wrote. The exit code already
    said pass or fail; this phase keeps the numbers, memory bandwidth and
    PCIe bandwidth per GPU, and judges each against this line's floor so a
    board that passes DCGM's default floor but sits 20 % under its siblings
    is caught here."""
    report = load_report(RESULTS_DIR / "dcgm_r2.json")
    mem = test_results(report, "Memory Bandwidth")
    pcie = test_results(report, "PCIe")
    mem_bw = [number_after(r["info"], "memory bandwidth") for r in mem]
    h2d = [number_after(r["info"], "host-to-device") for r in pcie]
    gpus = list(range(len(mem)))

    measurements.dcgm_r2_all_pass = all_passed(report)
    m = measurements.memory_bandwidth
    m.x_axis = gpus
    m.y_axis.mb_s = mem_bw
    m.y_axis.mb_s.aggregations.min_mb_s = min(mem_bw)
    m.y_axis.mb_s.aggregations.spread_pct = round(100.0 * (max(mem_bw) - min(mem_bw)) / max(mem_bw), 2)
    p = measurements.pcie_bandwidth
    p.x_axis = gpus
    p.y_axis.h2d_gb_s = h2d
    p.y_axis.h2d_gb_s.aggregations.min_gb_s = min(h2d)
    log.info(f"DCGM r2 {report['version']}: memory bandwidth {min(mem_bw):,.0f} to {max(mem_bw):,.0f} MB/s against a {MEM_BW_FLOOR_MB_S:,} floor, PCIe host-to-device {min(h2d):.1f} GB/s min against {PCIE_BW_FLOOR_GB_S:.0f}")
```

The DCGM report is one object under `DCGM Diagnostic` with a list of test categories, each holding tests with a per-GPU list of results; the numbers travel in each result's `info` string. `utils/dcgm.py` holds the readers, so a change of wording between DCGM versions is a change of label in one file:

```python filename="utils/dcgm.py"
"""Readers for the DCGM diagnostic report (`dcgmi diag -j`) and for the
numbers DCGM puts in each result's info string."""

import json
import re


def load_report(path):
    """The report as dcgmi writes it: one object under 'DCGM Diagnostic'."""
    with open(path) as f:
        return json.load(f)["DCGM Diagnostic"]


def test_results(report, test_name):
    """The per-GPU results of one named test, in GPU order."""
    for category in report["test_categories"]:
        for test in category["tests"]:
            if test["name"] == test_name:
                return sorted(test["results"], key=lambda r: int(r.get("gpu_id", 0)))
    raise KeyError(f"test {test_name!r} not in the report")


def all_passed(report):
    return all(r["status"] == "Pass" for c in report["test_categories"] for t in c["tests"] for r in t["results"])


def number_after(info, label):
    """The number that follows a label in an info string, as a float:
    number_after("achieved 694.2 W over 300 s", "achieved") is 694.2.
    DCGM's wording changes between versions; adjust the labels here, not
    the phases."""
    m = re.search(re.escape(label) + r"\s*(-?[\d,]+(?:\.\d+)?)", info)
    if not m:
        raise ValueError(f"no number after {label!r} in {info!r}")
    return float(m.group(1).replace(",", ""))


def number_before(info, label):
    m = re.search(r"(-?[\d,]+(?:\.\d+)?)\s*" + re.escape(label), info)
    if not m:
        raise ValueError(f"no number before {label!r} in {info!r}")
    return float(m.group(1).replace(",", ""))
```

### DCGM Level 3 and Targeted Power

`dcgmi diag -r 3 -j -p targeted_power.test_duration=300` drives every GPU to its power target for the line's 300 s. The parser records the power reached per GPU as a percentage of the 700 W limit, the hottest sensor, the throttle count summed over the eight modules and the duration DCGM reports, which must be the one asked for:

```python filename="phases/dcgm_r3_targeted_power.py"
from utils.dcgm import all_passed, load_report, number_after, number_before, test_results
from utils.recipe import GPU_TDP_W, RESULTS_DIR, TARGETED_POWER_S


def dcgm_r3_targeted_power(measurements, log):
    """Level 3 parsed: targeted_power drove every GPU to its power limit for
    the line's duration. Achieved power per GPU is judged as a percentage
    of TDP, the hottest sensor against the line's ceiling, and the throttle
    count must be zero: a module that reaches TDP by throttling is a module
    whose cold plate is not seated."""
    report = load_report(RESULTS_DIR / "dcgm_r3.json")
    rows = test_results(report, "Targeted Power")
    achieved = [number_after(r["info"], "achieved") for r in rows]
    temps = [number_after(r["info"], "max temperature") for r in rows]
    throttles = [number_before(r["info"], "throttle events") for r in rows]
    duration = number_after(rows[0]["info"], "over")
    pct = [100.0 * w / GPU_TDP_W for w in achieved]

    measurements.dcgm_r3_all_pass = all_passed(report)
    m = measurements.targeted_power
    m.x_axis = list(range(len(rows)))
    m.y_axis.achieved_w = achieved
    m.y_axis.achieved_w.aggregations.min_pct_of_tdp = round(min(pct), 2)
    m.y_axis.achieved_w.aggregations.min_w = min(achieved)
    m.y_axis.max_temp_c = temps
    m.y_axis.max_temp_c.aggregations.max_c = max(temps)
    measurements.throttle_events = int(sum(throttles))
    measurements.targeted_power_duration_s = duration
    log.info(f"DCGM r3 targeted_power: {duration:.0f} s at {GPU_TDP_W:.0f} W target (asked {TARGETED_POWER_S} s), achieved {min(achieved):.1f} to {max(achieved):.1f} W ({min(pct):.1f} % of TDP min), hottest {max(temps):.0f} C, {int(sum(throttles))} throttle events")
```

![Power reached per GPU during the DCGM targeted_power test as eight green bars between 689.8 and 696.1 W, under the 700 W TDP line and above the 665 W floor at 95 percent of TDP, with each GPU's hottest sensor from 69 to 76 degrees C printed in its bar.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/gpu-baseboard-chart-power-KzjmR7qJTiUKMT0YjTfFpoqiZpFZC6.png)

The mock board at the power target: 689.8 to 696.1 W, 98.5 % of TDP on the lowest module, 69 to 76 °C at the hottest sensor with the intake at room temperature. A module that reaches the target only by throttling, or that sits 10 °C above its siblings at the same power, is a cold plate that is not seated; both show here before the soak.

### NCCL All-Reduce

`all_reduce_perf -b 8 -e 8G -f 2 -g 8 -c 1 > results/nccl_allreduce.txt` runs the collective across the eight GPUs over 31 message sizes with validation on. The parser reads the data rows of the table, keeps the bus and algorithm bandwidth curves whole, and judges the bus bandwidth at the top of the sweep against the line's reference:

```python filename="phases/nccl_allreduce.py"
from utils.recipe import NCCL_BUSBW_FLOOR_PCT, NCCL_BUSBW_REF_GB_S, RESULTS_DIR


def parse_all_reduce_perf(path):
    """The data rows of all_reduce_perf: size in bytes, out-of-place algbw
    and busbw in GB/s, and the wrong-value count of both passes. Comment
    lines start with '#'."""
    sizes, algbw, busbw, wrong = [], [], [], 0
    with open(path) as f:
        for line in f:
            if line.startswith("#") or not line.strip():
                continue
            cols = line.split()
            sizes.append(int(cols[0]))
            algbw.append(float(cols[6]))
            busbw.append(float(cols[7]))
            wrong += int(cols[8]) + int(cols[12])
    return sizes, algbw, busbw, wrong


def nccl_allreduce(measurements, log):
    """The NVLink fabric under the collective the customer will run:
    all_reduce_perf across the 8 GPUs over the message-size sweep, judged on
    the bus bandwidth it reaches at the top of the sweep against the line's
    expected value. A GPU with one NVLink down passes DCGM's link test and
    loses a fifth of its bus bandwidth here."""
    sizes, algbw, busbw, wrong = parse_all_reduce_perf(RESULTS_DIR / "nccl_allreduce.txt")
    peak = max(busbw)
    pct = 100.0 * peak / NCCL_BUSBW_REF_GB_S

    m = measurements.allreduce_busbw
    m.x_axis = sizes
    m.y_axis.busbw_gb_s = busbw
    m.y_axis.busbw_gb_s.aggregations.peak_gb_s = round(peak, 2)
    m.y_axis.busbw_gb_s.aggregations.peak_pct_of_ref = round(pct, 2)
    m.y_axis.algbw_gb_s = algbw
    measurements.nccl_wrong_values = wrong
    log.info(f"all_reduce_perf: {len(sizes)} sizes from {sizes[0]} B to {sizes[-1] >> 30} GiB, bus bandwidth peaks at {peak:.1f} GB/s, {pct:.1f} % of the {NCCL_BUSBW_REF_GB_S:.0f} GB/s reference (floor {NCCL_BUSBW_FLOOR_PCT:.0f} %), {wrong} wrong values")
```

![all_reduce_perf bus bandwidth in green and algorithm bandwidth in blue against message size from 8 B to 8 GiB on a log axis, the bus bandwidth rising past 2 MiB to a 462.2 GB/s peak, above the 432 GB/s floor at 90 percent of the 480 GB/s reference.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/gpu-baseboard-chart-nccl-3i70A0PsyJw3b3QMpFfNtQQUoCVLjL.png)

The mock fabric: latency-bound below a megabyte, saturating past 1 GiB at 462 GB/s, 96.3 % of the reference. A GPU with one NVLink lane trained down passes DCGM's link check and moves the top of this curve by a fifth, which is why the bus bandwidth is judged and not the link count alone.

### DCGM Level 4 and ECC

`dcgmi diag -r 4 -j > results/dcgm_r4.json` walks the HBM memtest patterns and the pulse test over about two hours on eight GPUs. The parser reads the errors and the duration per GPU, then reads the ECC and row-remap counters a second time and subtracts the inventory phase's baseline, which the framework injects because the function names `inventory` as a parameter:

```python filename="phases/hbm_memtest.py"
from utils.dcgm import all_passed, load_report, number_after, number_before, test_results
from utils.recipe import ECC_CORRECTABLE_CEILING, RESULTS_DIR


def ecc_delta(after, before, field):
    """Per-GPU change of one ECC counter between two reads, in GPU order."""
    return [after[k][field] - before[k][field] for k in sorted(after, key=int)]


def hbm_memtest(measurements, node, inventory, log):
    """Level 4 parsed, then the field acceptance rule. The memtest patterns
    walked every HBM row; the report says how long and how many errors.
    The ECC counters are read again and the inventory phase's baseline is
    subtracted: zero uncorrectable, no row pending a remap, and a ceiling
    on correctable events per GPU, because a stack that corrects on every
    pattern is a stack that will remap rows in the customer's rack."""
    report = load_report(RESULTS_DIR / "dcgm_r4.json")
    rows = test_results(report, "Memtest")
    errors = [number_before(r["info"], "errors") for r in rows]
    durations = [number_after(r["info"], "errors,") for r in rows]
    after = node.ecc_counters()
    before = inventory.ecc_baseline
    corr = ecc_delta(after, before, "correctable")
    uncorr = ecc_delta(after, before, "uncorrectable")
    pending = [after[k]["row_remap_pending"] for k in sorted(after, key=int)]
    gpus = list(range(len(rows)))

    measurements.dcgm_r4_all_pass = all_passed(report)
    m = measurements.memtest
    m.x_axis = gpus
    m.y_axis.errors = errors
    m.y_axis.errors.aggregations.max_errors = max(errors)
    m.y_axis.duration_s = durations
    m.y_axis.duration_s.aggregations.max_s = max(durations)
    e = measurements.ecc_after_memtest
    e.x_axis = gpus
    e.y_axis.correctable_delta = corr
    e.y_axis.correctable_delta.aggregations.max_per_gpu = max(corr)
    e.y_axis.uncorrectable_delta = uncorr
    e.y_axis.uncorrectable_delta.aggregations.max_per_gpu = max(uncorr)
    e.y_axis.row_remap_pending = pending
    e.y_axis.row_remap_pending.aggregations.max_per_gpu = max(pending)
    measurements.ecc_counters_after_memtest = after
    log.info(f"DCGM r4 memtest: {max(durations):.0f} s longest, {int(sum(errors))} errors; ECC since inventory: correctable {corr} (ceiling {ECC_CORRECTABLE_CEILING} per GPU), uncorrectable {uncorr}, rows pending remap {pending}")
```

The counters as read are also kept as a JSON measurement with no validator, `ecc_counters_after_memtest`, so the soak can subtract them in turn.

### Soak

The node under its burn-in load at 35 °C intake for 24 hours, sampled once an hour through the node agent. The phase keeps a trace of the hottest GPU, the intake and the board power against time, and per GPU the maximum temperature and the minimum SM clock over the day. At the end the counters are read a third time and the memtest phase's counters subtracted, and the kernel log must hold no XID. The mock scales one hour to one second and says so in the phase log:

```python filename="phases/soak.py"
import time

from phases.hbm_memtest import ecc_delta
from utils.recipe import SOAK_HOURS, SOAK_INTAKE_C, SOAK_SAMPLE_MIN, TEMP_CEILING_C, TIME_SCALE


def soak(measurements, node, hbm_memtest, log):
    """The soak: the node under its burn-in load at elevated intake
    temperature for the line's 24 h, sampled once an hour. Two views are
    kept: the trace of the hottest GPU, the intake and the board power
    against time, and per GPU the maximum temperature and the minimum SM
    clock over the day. At the end the ECC counters are read a third time
    and the memtest phase's counters subtracted, so the ceiling applies to
    the soak alone, and the kernel log must hold no XID. The mock scales
    one hour to one second."""
    samples = list(range(0, SOAK_HOURS + 1, max(1, SOAK_SAMPLE_MIN // 60)))
    hottest, intake, board_kw = [], [], []
    per_gpu_max_c = None
    per_gpu_min_mhz = None
    log.info(f"Soak: {SOAK_HOURS} h at {SOAK_INTAKE_C:.0f} C intake, sampled every {SOAK_SAMPLE_MIN} min (mock: {SOAK_HOURS * 3600 / TIME_SCALE:.0f} s)")
    for hour in samples:
        s = node.soak_sample(hour)
        hottest.append(max(s["temp_c"]))
        intake.append(s["intake_c"])
        board_kw.append(round(sum(s["power_w"]) / 1000.0, 3))
        per_gpu_max_c = s["temp_c"] if per_gpu_max_c is None else [max(a, b) for a, b in zip(per_gpu_max_c, s["temp_c"])]
        per_gpu_min_mhz = s["sm_clock_mhz"] if per_gpu_min_mhz is None else [min(a, b) for a, b in zip(per_gpu_min_mhz, s["sm_clock_mhz"])]
        time.sleep(3600.0 / TIME_SCALE)
    after = node.ecc_counters()
    before = hbm_memtest.ecc_counters_after_memtest
    corr = ecc_delta(after, before, "correctable")
    uncorr = ecc_delta(after, before, "uncorrectable")
    pending = [after[k]["row_remap_pending"] for k in sorted(after, key=int)]
    xids = node.xid_events()

    t = measurements.soak_trace
    t.x_axis = samples
    t.y_axis.hottest_gpu_c = hottest
    t.y_axis.hottest_gpu_c.aggregations.max_c = max(hottest)
    t.y_axis.intake_c = intake
    t.y_axis.intake_c.aggregations.mean_c = round(sum(intake) / len(intake), 2)
    t.y_axis.board_power_kw = board_kw
    t.y_axis.board_power_kw.aggregations.mean_kw = round(sum(board_kw) / len(board_kw), 3)
    g = measurements.soak_per_gpu
    g.x_axis = list(range(len(per_gpu_max_c)))
    g.y_axis.max_temp_c = per_gpu_max_c
    g.y_axis.max_temp_c.aggregations.max_c = max(per_gpu_max_c)
    g.y_axis.min_sm_clock_mhz = per_gpu_min_mhz
    g.y_axis.min_sm_clock_mhz.aggregations.min_mhz = min(per_gpu_min_mhz)
    g.y_axis.correctable_delta = corr
    g.y_axis.correctable_delta.aggregations.max_per_gpu = max(corr)
    g.y_axis.uncorrectable_delta = uncorr
    g.y_axis.uncorrectable_delta.aggregations.max_per_gpu = max(uncorr)
    g.y_axis.row_remap_pending = pending
    g.y_axis.row_remap_pending.aggregations.max_per_gpu = max(pending)
    measurements.xid_events = xids
    log.info(f"Soak done: hottest GPU {max(hottest):.1f} C (ceiling {TEMP_CEILING_C:.0f}), board {sum(board_kw) / len(board_kw):.2f} kW mean, SM clock floor {min(per_gpu_min_mhz)} MHz; ECC over the soak: correctable {corr}, uncorrectable {uncorr}, rows pending {pending}; {xids} XID")
```

![Correctable ECC events per GPU as paired bars, blue for the memtest stage and green for the 24 hour soak, at most 2 and 3 on GPU 4, far under the ceiling of 10 per GPU per stage drawn as a red dashed line.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/gpu-baseboard-chart-ecc-RK2rC7ORHZrFNlLqafpt8rzH1GMSnZ.png)

The mock board's ECC story: four correctable events during the memtest and six during the soak, five of them on GPU 4, zero uncorrectable and no row pending a remap. GPU 4 is the module to watch on this board; its serial is on the unit metadata, so a re-test that adds events to the same module is traceable to it.

### Clear and Report

The teardown attaches the five raw files to the run, writes the eight GPU serials into the unit metadata so a module traced from a field failure leads back to this board and this run, resets the volatile ECC counters, and records that they read zero:

```python filename="phases/clear_and_report.py"
from utils.recipe import RESULTS_DIR

REPORTS = ["dcgm_r1.json", "dcgm_r2.json", "dcgm_r3.json", "dcgm_r4.json", "nccl_allreduce.txt"]


def clear_and_report(measurements, node, unit, attach, log):
    """Teardown: the raw reports go onto the run as attachments, the GPU
    serials onto the unit, and the volatile ECC counters back to zero so
    the first event the customer reads happened in their rack. Runs
    whatever a main phase did."""
    attached = 0
    for name in REPORTS:
        path = RESULTS_DIR / name
        if path.exists():
            attach.file(str(path), name)
            attached += 1
    for g in node.gpu_inventory():
        unit.metadata[f"gpu{g['index']}_serial"] = g["serial"]
    cleared = node.reset_ecc_counters()
    after = node.ecc_counters()
    measurements.ecc_counters_cleared = cleared and all(v["correctable"] == 0 and v["uncorrectable"] == 0 for v in after.values())
    measurements.reports_attached = attached
    log.info(f"{attached} reports attached, {len(after)} GPU serials on the unit, ECC counters cleared")
```

### Mock Plug

`NodeAgent` stands in for nvidia-smi, the BMC and the kernel log; the diagnostics themselves are not in the plug, they are the executable phases. In the mock, `scripts/dcgm_diag_mock.py` and `scripts/nccl_allreduce_mock.py` print what `dcgmi diag -j` and `all_reduce_perf` print and exit 0, and they share the node's state with the plug through `plugs/mock_node_state.py`, one JSON file under `results/` that the level 4 script advances with the memtest's correctable events and the plug reads back as counters. The plug reports eight modules at gen5 x16 with 18 links each, the soak temperatures rising over the first three hours and settling about 3 °C above the stress figures, SM clocks not below 1612 MHz, and a board power of 5.5 kW. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines. The recipe with the golden configuration, the floors and their basis:

```python filename="utils/recipe.py"
"""Diagnostic ladder for an 8-GPU HGX / OAM-UBB baseboard at L10: the DCGM
level each phase runs, the stimulus, and the limit each result is judged on.

The ladder is NVIDIA DCGM's: `dcgmi diag -r 1` software, `-r 2` PCIe,
NVLink, memory and memory bandwidth, `-r 3` the targeted power and stress
tests plus nvbandwidth and nccl_tests, `-r 4` the HBM memtest patterns and
the pulse test. DCGM ships an example configuration for an H100 PCIe card
(target_power 350 W, memory_bandwidth minimum 1,230,000 MB/s, PCIe at
least gen3 x16); this line's board carries SXM-class modules and sets its
own floors from its golden node. The field acceptance rules, zero
uncorrectable ECC and a ceiling on correctable ECC and row-remap events
during the soak, are integrator practice. Every limit here is either the
DCGM example, the board's golden configuration, or this line's derivation,
and procedure.yaml says which."""

from pathlib import Path

# Unit under test: the baseboard's golden configuration
GPU_COUNT = 8
PCIE_GEN = 5  # DCGM's example minimum is gen3 x16; this board's links are gen5 x16
PCIE_WIDTH = 16
NVLINK_LINKS_PER_GPU = 18  # the topology of this board; the count nvidia-smi nvlink -s reports per GPU
FIRMWARE = {"driver": "570.86.15", "vbios": "96.00.89.00.01", "dcgm": "3.3.9"}

# Level 2: memory bandwidth and PCIe bandwidth floors (this line's)
MEM_BW_FLOOR_MB_S = 2_850_000  # 85 % of the module's 3.35 TB/s datasheet figure; DCGM's example floor is 1,230,000 MB/s for a PCIe card
PCIE_BW_FLOOR_GB_S = 45.0  # host-to-device bandwidth on a gen5 x16 link, from the golden node's first article

# Level 3: targeted_power drives every GPU to its power limit
GPU_TDP_W = 700.0  # the module's power limit as nvidia-smi reports it
TARGETED_POWER_S = 300  # this line's; DCGM's default is shorter
POWER_FLOOR_PCT_OF_TDP = 95.0
TEMP_CEILING_C = 85.0  # this line's ceiling at the module's hottest sensor during stress and soak

# NCCL all_reduce_perf across the 8 GPUs
NCCL_MSG_MIN = "8"
NCCL_MSG_MAX = "8G"
NCCL_BUSBW_REF_GB_S = 480.0  # this line's expected bus bandwidth at 8 GiB, the median of its golden nodes
NCCL_BUSBW_FLOOR_PCT = 90.0

# Level 4: HBM memtest patterns, then the field acceptance rules on ECC
ECC_CORRECTABLE_CEILING = 10  # per GPU, per stage (memtest, soak); integrator practice, the number is this line's
ROW_REMAP_PENDING_CEILING = 0

# Soak at elevated intake temperature
SOAK_HOURS = 24  # 24 to 72 h is the integrator range; this line runs 24 h at L10
SOAK_INTAKE_C = 35.0
SOAK_SAMPLE_MIN = 60
SM_CLOCK_FLOOR_MHZ = 1500  # sustained SM clock under the soak load, from the golden node

# Mock time scale: one hour of the real ladder is one second here
TIME_SCALE = 3600.0

RESULTS_DIR = Path(__file__).resolve().parents[1] / "results"
```

```python filename="plugs/node_agent.py"
"""L10 node agent (mock): what the test station reads from the node under
test between the diagnostic runs. nvidia-smi for the inventory, the PCIe
link, the NVLink status, the ECC and row-remap counters and the telemetry
samples; the BMC's Redfish API for the baseboard serial and the intake
temperature; the kernel log for XID events.

Maps to `nvidia-smi -q -x` and `nvidia-smi nvlink -s` over SSH (or the
DCGM Python bindings, `pydcgm`), a Redfish client on the BMC, and
`journalctl -k` for the XIDs. The diagnostics themselves, `dcgmi diag` and
`all_reduce_perf`, are not here: they are executable phases in
procedure.yaml, and the mock scripts they run share this mock's state
through plugs/mock_node_state.py. Swap for classes speaking SSH, Redfish
and pydcgm; the phases stay unchanged.
"""

import random

from plugs import mock_node_state as node
from utils.recipe import FIRMWARE, GPU_COUNT, NVLINK_LINKS_PER_GPU, PCIE_GEN, PCIE_WIDTH, SOAK_INTAKE_C


class NodeAgent:
    def __init__(self, serial_number):
        self.serial_number = str(serial_number)
        self._rng = random.Random(8100)
        node.reset()
        # self.ssh = paramiko.SSHClient(...); self.bmc = redfish.redfish_client(...)
        print(f"Node agent connected: baseboard {self.serial_number}, {GPU_COUNT} GPUs enumerated, BMC reachable")

    # --- inventory, nvidia-smi -q and Redfish ----------------------------------

    def baseboard_serial(self):
        """The baseboard serial the BMC reports under Chassis/Baseboard."""
        return self.serial_number

    def gpu_inventory(self):
        """One record per GPU as nvidia-smi -q -x lists them: serial, PCIe
        link generation and width, VBIOS."""
        return [
            {"index": i, "serial": node.SERIALS[i], "pcie_gen": PCIE_GEN, "pcie_width": PCIE_WIDTH, "vbios": FIRMWARE["vbios"]}
            for i in range(GPU_COUNT)
        ]

    def firmware(self):
        """Driver, VBIOS and DCGM version as one object."""
        return dict(FIRMWARE)

    def nvlink_status(self):
        """Links up per GPU from nvidia-smi nvlink -s."""
        return [NVLINK_LINKS_PER_GPU] * GPU_COUNT

    # --- ECC, row remap and XID, read between the diagnostic runs -------------

    def ecc_counters(self):
        """Volatile ECC and row-remapper counters per GPU: correctable,
        uncorrectable, rows pending a remap."""
        s = node.load()
        return {
            str(i): {"correctable": s["correctable"][i], "uncorrectable": s["uncorrectable"][i], "row_remap_pending": s["row_remap_pending"][i]}
            for i in range(GPU_COUNT)
        }

    def reset_ecc_counters(self):
        """nvidia-smi -p 0: the volatile counters back to zero so the first
        event the customer reads happened in their rack."""
        s = node.load()
        s["correctable"] = [0] * GPU_COUNT
        s["uncorrectable"] = [0] * GPU_COUNT
        node.save(s)
        return True

    def xid_events(self):
        """XID lines in the kernel log since the node booted."""
        return node.load()["xid_events"]

    # --- soak telemetry, sampled once an hour -----------------------------------

    def soak_sample(self, hour):
        """One telemetry sample at the stated soak hour: per-GPU temperature
        at the hottest sensor, SM clock and power under the soak load, and
        the intake temperature from the BMC. The mock advances the node's
        ECC counters for that hour."""
        node.apply_soak_hour(hour)
        temps = [round(t - 3.0 + 3.0 * min(1.0, hour / 3.0) + self._rng.uniform(-0.6, 0.6), 1) for t in node.SOAK_TEMP_C]
        clocks = [int(c + self._rng.randint(-8, 8)) for c in node.SOAK_SM_CLOCK_MHZ]
        power = [round(p - 4.0 + self._rng.uniform(-2.0, 2.0), 1) for p in node.POWER_W]
        return {"temp_c": temps, "sm_clock_mhz": clocks, "power_w": power, "intake_c": round(SOAK_INTAKE_C + self._rng.uniform(-0.3, 0.3), 1)}

    def __del__(self):
        print("Node agent released")
```

On a real node, replace the five `python3 scripts/...` commands in `procedure.yaml` with `dcgmi diag -r 1 -j`, `-r 2`, `-r 3 -p targeted_power.test_duration=300`, `-r 4` and `all_reduce_perf -b 8 -e 8G -f 2 -g 8 -c 1`, each redirected to the same file under `results/`, run over SSH or from a station process on the node. Split the plug into an SSH client for nvidia-smi and the kernel log and a Redfish client for the BMC, or use the DCGM Python bindings for the counters. Set the labels in `utils/dcgm.py` to the DCGM version's wording, or point the parsers at the per-test stats DCGM writes with `--statspath`. Delete `plugs/mock_node_state.py` and `scripts/`. The phases, measurements and limits stay the same.


### ORV3 PSU Acceptance Test
URL: https://www.tofupilot.com/templates/orv3-psu-acceptance-test
Acceptance of an OCP ORV3 12 kW PSU module: hipot, inrush, efficiency vs Titanium, ripple, load step, hold-up at 100 and 120 %, OVP OCP OTP trips, blackbox.

## Introduction

### ORV3 Power Supply Overview

A rack in an OCP Open Rack V3 data center has no power supply in any server. A power shelf at the bottom or the middle of the rack converts the facility's 230 to 277 Vac single phase into a 48 V class busbar that runs the height of the rack, and every compute tray hangs off that busbar with a blind-mate clip. The shelf holds a row of hot-pluggable PSU modules, 12 kW each in the high-power (HPR) generation, sharing the load in N+1 or N+2. One module that ships out of spec is not one broken server: it is a shelf that sheds load when the grid blinks, a busbar that overshoots into the trays' OVP, or a rack whose power capping is wrong because the module lies about its own current.

![Slender hot-swap rack power supply with a closed metal enclosure, front fan grille, and blue extraction handle.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/orv3-psu-acceptance-test-matte-closeup-jTvv6PUcwfmMV5rJTMo3JNUogI4wYz.webp)

The parts this test exercises: the AC inlet and the earth path behind it, the bulk capacitors that carry the output through a mains dropout, the PFC and LLC stages that set the efficiency curve, the output blade connector, and the controller behind the Modbus port that holds the protections and the blackbox.

The [OCP Open Rack V3 HPR V2 12 kW PSU module specification](https://www.opencompute.org/documents/open-rack-v3-hpr-v2-12kw-psu-module-spec-v1-0-0-pdf) is unusually explicit for a production test writer. Efficiency is a floor and a peak, not a single point: **at least 96.5 % from 30 to 100 % load, 94 % from 10 to 30 %, and a peak above 97.5 %**, measured at 230 to 277 Vac after 30 minutes at full load with the fans running (section 4.8). Hold-up is a table: **20 ms at 100 % load, 16.67 ms at 120 %, 13.33 ms at 150 %** (Table 8, section 4.17), and the AC_Loss_L signal must lead the collapse by 5 ms. Cold-start inrush is capped at **60 A RMS and 85 A peak** (4.7); output ripple at **500 mVpp over 5 Hz to 100 MHz** with a 20 MHz scope bandwidth (4.14); the dynamic load table asks for 70 to 130 % steps at 3 A/µs settling within 3 ms (4.15); output over-voltage must trip **above 52.5 V within 200 ms**, with a fast comparator at 54 V (4.22); and section 10.2 makes hipot and ground continuity a **100 % production test** with a stamp on the chassis. The 80 PLUS Titanium points that the industry quotes, 90 / 94 / 96 / 91 % at 10 / 20 / 50 / 100 % load for a 230 V internal-redundant supply, all sit under the ORV3 floor. Every limit in this template is one of those rows, or says where it came from instead.

### Test Purpose

The procedure records one acceptance fingerprint per module:

- **Hipot leakage and ground bond** with the AC source off, before anything else
- **Model, firmware and serial** over Modbus as one object, the reported serial compared with the scanned label, and the no-load output
- **Cold-start inrush** at 277 Vac closed at the 90 degree phase angle, the whole waveform, judged on its peak and its first-cycle RMS
- **Efficiency at six load points at 230 and 277 Vac**, each curve judged on the stair floor and the peak, with the **output-current telemetry error** on a third curve
- **Ripple** at full load, and the **70 to 130 % load step** in both directions, judged on deviation and settle time
- **Hold-up at 100 and 120 % load**, the output traces whole, and the **AC_Loss_L warning margin**
- **OVP, OCP, OTW and OTP** trip points found in test mode, the OVP response and the OCP restart delay, and the output back in regulation afterwards
- The **blackbox** read at teardown, required to hold exactly the events the test caused, then cleared

![Efficiency against load at 277 Vac in green and 230 Vac in blue, both peaking at 97.8 % at 50 % load, above the ORV3 stair floor of 94 % below 30 % load and 96.5 % above, above the 97.5 % peak line, with the four 80 PLUS Titanium points far below as gray triangles.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/orv3-psu-chart-efficiency-fBZnCM4ippg2AdzlqfE0cwgOUDPQOk.png)

The mock module's two efficiency curves against the spec: the stair floor, the peak line, and the Titanium points that a marketing sheet would quote, all comfortably under a curve that peaks at 97.8 % at half load.

The framework mechanics on show are a `setup:` stage used as a safety gate, with the identity phase waiting on the hipot through `depends_on`; a multi-dimensional measurement with three curves and a different set of aggregations on each, validated in the YAML; four waveforms recorded whole and each judged on one or two numbers; JSON `==` on a whole object three times; a `timeout` sized for the spec's warm-up; and a `teardown:` that always parks the load and the source.

## Equipment & Setup

To run this acceptance on a production line, the following are required:

- **A programmable AC source** covering 230 to 277 Vac at the module's full input current, with phase-angle start for the inrush test
- **A 12 kW DC electronic load** with a 3 A/µs slew and a 120 % overload range
- **A power analyzer** on the input and output for the efficiency sweep
- **A hipot and ground-bond tester** interlocked so the AC source cannot enable while it is connected
- **An oscilloscope** on the output with a 20 MHz bandwidth limit, and on an inrush current transformer at the input
- **A USB-RS485 adapter** on the module's Modbus management port
- **The Device Under Test (DUT)**: an ORV3 HPR 12 kW PSU module with its production firmware, in a test-mode build that accepts an output trim and a thermistor override over the bus
- **A TofuPilot Framework procedure** to sequence the stimuli and validate the limits
- **The TofuPilot Dashboard** to keep the per-module evidence and the weekly Cpk the spec's section 10.1 asks for

### Hardware Components

#### AC Source and Electronic Load

At 12 kW the input draws about 45 A at 277 Vac, and the [Chroma 61512](https://www.chromaate.com/en/product/programmable_ac_source_61500_series) (12 kVA, phase-angle start and stop) or a [Pacific Power 3120AFX](https://pacificpower.com/products/afx-series/) cover it; the phase-angle start is what makes the inrush test repeatable, because a contactor closed at random gives a different peak every time. On the output, a [Chroma 63200A-series](https://www.chromaate.com/en/product/dc_electronic_load_63200a_series) 12 kW module or two in parallel at 240 A, with the slew set to 3 A/µs for the load step and the CC range reaching the 130 % point. A [Yokogawa WT5000](https://tmi.yokogawa.com/solutions/products/power-analyzers/wt5000-precision-power-analyzer/) on both sides gives the 0.03 % that a Titanium-class efficiency needs to be measured rather than asserted.

![Long hot-swap rack power supply secured in a blind-mate test fixture with enclosed rear connections and heavy output cables.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/orv3-psu-acceptance-test-matte-cover-hPM6nt1DIwpAvG69rwz3bhgGt4gFHK.webp)

The illustrated fixture: a hot-swap PSU in a guided blind-mate adapter with protected rear connections. The AC source, electronic load, power analyzer and scope are omitted.

#### Hipot Tester and Management Bus

The [Chroma 19032](https://www.chromaate.com/en/product/electrical_safety_analyzer_19032) does the DC withstand and the 40 A ground bond in one connection and reports leakage in microamps. The test is DC because the module's Y-capacitors would draw milliamps at 1500 Vac and hide a real fault; at 2121 Vdc a healthy unit reads tens of microamps and a pinched primary lead reads a milliamp. The management bus on the ORV3 PSU is Modbus RTU over RS-485 at 115.2 kbps, with the blackbox fault log the spec requires; the test-mode registers for output trim and thermistor override are the vendor's, and the identity, faults and blackbox registers are the spec's.

### Where the Limits Come From

| Test | Limit | Basis |
|---|---|---|
| Hipot leakage | ≤ 1.0 mA at 2121 Vdc, 1 s | IEC 62368-1 routine test, 1500 Vac equivalent, basic insulation to an earthed chassis; ORV3 10.2 requires 100 % |
| Ground bond | ≤ 100 mΩ at 40 A, 2 s | IEC 62368-1 |
| Inrush | ≤ 85 A peak, ≤ 60 A RMS first cycle | ORV3 4.7 |
| Efficiency | ≥ 94 % (10 to 30 %), ≥ 96.5 % (30 to 100 %), peak ≥ 97.5 % | ORV3 4.8 |
| Iout telemetry | ≤ 2 % of reading | this line's derivation: the rack's power capping budget |
| Ripple | ≤ 500 mVpp, 20 MHz | ORV3 4.14 |
| Load step | ≤ 1.5 V deviation, settled within 3 ms | ORV3 4.15 for the stimulus and the settle time; 1.5 V keeps the overshoot under the 52.5 V OVP threshold |
| Hold-up | ≥ 20 ms at 100 %, ≥ 16.67 ms at 120 % | ORV3 Table 8 |
| AC_Loss_L lead | ≥ 5 ms | ORV3 4.17 |
| OVP | 52.5 to 54.0 V, ≤ 200 ms | ORV3 4.22 |
| Constant-current limit | 115 to 125 % of rated | ORV3: CC at 120 %; ±5 % is this line's |
| OCP restart | 9 to 11 s out of regulation | ORV3: 10 s; ±1 s is this line's |
| OTW / OTP | 85 ± 2 °C / 95 ± 2 °C, OTP − OTW ≥ 4 °C | the module's configuration; the 4 °C margin is M-CRPS 8.1.9 |
| Identity, blackbox | exact match | no tolerance applies |

Two rows in that table are traps. The 150 % hold-up line of Table 8 needs a load that sinks 18 kW, which this station does not have; the procedure tests the two rows it can and says so, rather than silently testing one. And the OVP trip on a module that regulates to 50.5 V cannot be found by raising the output externally, because the module's own stage fights the source; it is found by raising the module's set-point in test mode, which is why the DUT needs a test-mode build.

## Test Procedure

### Overview

The procedure maps the acceptance onto the framework's three stages. Hipot and bond live in `setup:` so no AC is applied to a module that fails withstand, and the identity phase depends on them so the setup stage cannot run them concurrently. The blackbox read and the park live in `teardown:` so the load goes to zero and the source off even when a main phase fails.

1. Setup: DC withstand and ground bond, AC source off.
2. Setup: first power-on, identity and serial over Modbus, no-load output.
3. Main: cold-start inrush at 277 Vac, 90 degree close.
4. Main: efficiency at six load points, 230 and 277 Vac, telemetry error.
5. Main: ripple at full load, 70 to 130 % load step both ways.
6. Main: AC loss at 100 and 120 % load, AC_Loss_L lead.
7. Main: OVP, OCP, OTW and OTP in test mode, recovery.
8. Teardown: blackbox checked and cleared, load off, AC off.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-module evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  hipot_and_bond.py
  identify.py
  inrush.py
  efficiency.py
  ripple_and_transient.py
  hold_up.py
  protections.py
  blackbox_and_park.py
plugs/
  psu_bench.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-orv3-psu-acceptance). The `PsuBench` plug is a mock of the AC source, the load, the analyzer, the hipot tester, the scope and the Modbus port together, synthesizing a healthy module with Titanium-class efficiency peaking at half load, 7.2 mF of bulk capacitance, every comparator inside its window and a blackbox that records what the test does to it, so the procedure runs end-to-end without a bench or a module connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug with the label serial as constructor argument, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: ORV3 PSU Acceptance Test
version: 0.1.0
description: Acceptance of an OCP Open Rack V3 HPR 12 kW PSU module. Hipot and ground bond before AC is applied, identity over Modbus, cold-start inrush at the 90 degree phase angle, efficiency at 230 and 277 Vac against the ORV3 floor and the Titanium points, ripple and the 70 to 130 percent load step, hold-up at 100 and 120 percent load with the AC_Loss_L lead, OVP OCP OTW and OTP trips in test mode, and a blackbox that must hold exactly what the test did before it is cleared.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the data-matrix on the module's handle"
    placeholder: "ORV3-12K-000000"
    pattern: "^ORV3-12K-\\d{6}$"
    default_value: "ORV3-12K-004217"
  part_number:
    default_value: "PSU-ORV3-12K-277"
  batch_number:
    default_value: "LOT-2026-38"

plugs:
  - name: PSU Bench
    description: "AC source, 12 kW DC load, power analyzer, hipot tester, scope and the PSU's Modbus port (mock, one plug per bench)"
    python: plugs.psu_bench:PsuBench
    key: bench
    config:
      serial_number: "ORV3-12K-004217"

setup:
  - name: Hipot and Ground Bond
    key: hipot_and_bond
    python: phases.hipot_and_bond
    measurements:
      - name: Hipot Leakage
        key: hipot_leakage_ma
        unit: mA
        description: "2121 Vdc primary to chassis for 1 s (IEC 62368-1 routine test, 1500 Vac equivalent). Y-capacitors pass nothing at DC; a healthy module leaks tens of microamps."
        validators:
          - {operator: "<=", expected_value: 1.0}
      - name: Ground Bond
        key: ground_bond_mohm
        unit: mΩ
        description: "Inlet earth pin to chassis at 40 A for 2 s."
        validators:
          - {operator: "<=", expected_value: 100.0}

  - name: Identify
    key: identify
    python: phases.identify
    depends_on: [hipot_and_bond]
    measurements:
      - name: PSU Identity
        key: psu_identity
        description: Model, firmware and management protocol as one object against the release.
        validators:
          - operator: "=="
            expected_value:
              model: PSU-ORV3-12K-277
              fw: 2.4.1
              protocol: modbus-rtu
      - name: Reported Serial
        key: reported_serial
        validators:
          - {operator: matches, expected_value: "^ORV3-12K-\\d{6}$"}
      - name: Serial Matches Label
        key: serial_matches_label
        description: The serial the module reports over Modbus is the one on its label.
        validators:
          - {operator: "==", expected_value: true}
      - name: Output At No Load
        key: vout_no_load_v
        unit: V
        validators:
          - {operator: ">=", expected_value: 50.0}
          - {operator: "<=", expected_value: 51.0}

main:
  - name: Inrush
    key: inrush
    python: phases.inrush
    measurements:
      - name: Inrush Current
        key: inrush
        title: Cold-start inrush at 277 Vac, closed at 90 degrees
        x_axis:
          legend: Time
          unit: ms
        y_axis:
          - legend: Input current
            key: current
            unit: A
            aggregations:
              - type: peak_a
                unit: A
                validators:
                  - {operator: "<=", expected_value: 85.0}
              - type: first_cycle_rms_a
                unit: A
                validators:
                  - {operator: "<=", expected_value: 60.0}

  - name: Efficiency
    key: efficiency
    python: phases.efficiency
    depends_on: [inrush]
    timeout: 45m
    measurements:
      - name: Efficiency
        key: efficiency
        title: Efficiency and output-current telemetry across the load range
        x_axis:
          legend: Load
          unit: "%"
        y_axis:
          - legend: Efficiency at 277 Vac
            key: eff_277
            unit: "%"
            aggregations:
              - type: min_30_100_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 96.5}
              - type: min_10_30_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 94.0}
              - type: peak_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 97.5}
          - legend: Efficiency at 230 Vac
            key: eff_230
            unit: "%"
            aggregations:
              - type: min_30_100_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 96.5}
              - type: min_10_30_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 94.0}
              - type: peak_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 97.5}
          - legend: Iout telemetry error at 277 Vac
            key: iout_telemetry_err_pct
            unit: "%"
            aggregations:
              - type: max_abs_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 2.0}

  - name: Ripple and Load Step
    key: ripple_and_transient
    python: phases.ripple_and_transient
    depends_on: [efficiency]
    measurements:
      - name: Output Ripple
        key: ripple_mvpp
        unit: mVpp
        description: "Full load, 20 MHz bandwidth, 0.1 uF at the probe tip (spec 4.14)."
        validators:
          - {operator: "<=", expected_value: 500.0}
      - name: Load Step
        key: load_step
        title: Output deviation on the 70 to 130 percent load step at 3 A/us
        x_axis:
          legend: Time
          unit: ms
        y_axis:
          - legend: Step up
            key: step_up
            unit: V
            aggregations:
              - type: min_v
                unit: V
                validators:
                  - {operator: ">=", expected_value: -1.5}
              - type: settle_ms
                unit: ms
                validators:
                  - {operator: "<=", expected_value: 3.0}
          - legend: Step down
            key: step_down
            unit: V
            aggregations:
              - type: max_v
                unit: V
                validators:
                  - {operator: "<=", expected_value: 1.5}
              - type: settle_ms
                unit: ms
                validators:
                  - {operator: "<=", expected_value: 3.0}

  - name: Hold-Up
    key: hold_up
    python: phases.hold_up
    depends_on: [ripple_and_transient]
    measurements:
      - name: Hold-Up
        key: hold_up
        title: Output voltage after AC loss at 100 and 120 percent load
        x_axis:
          legend: Time after AC loss
          unit: ms
        y_axis:
          - legend: Output at 100 % load
            key: vout_100
            unit: V
            aggregations:
              - type: hold_up_ms
                unit: ms
                validators:
                  - {operator: ">=", expected_value: 20.0}
          - legend: Output at 120 % load
            key: vout_120
            unit: V
            aggregations:
              - type: hold_up_ms
                unit: ms
                validators:
                  - {operator: ">=", expected_value: 16.67}
      - name: AC Loss Warning Margin
        key: ac_loss_warning_margin_ms
        unit: ms
        description: "Time from AC_Loss_L asserting to the output leaving regulation, worst of the two loads. The rack needs 5 ms to shed load."
        validators:
          - {operator: ">=", expected_value: 5.0}

  - name: Protections
    key: protections
    python: phases.protections
    depends_on: [hold_up]
    measurements:
      - name: OVP Trip
        key: ovp_trip_v
        unit: V
        description: "Set-point raised in 0.1 V steps in test mode until the output latches off. Spec 4.22: trip above 52.5 V, the fast comparator at 54 V."
        validators:
          - {operator: ">=", expected_value: 52.5}
          - {operator: "<=", expected_value: 54.0}
      - name: OVP Response
        key: ovp_response_ms
        unit: ms
        validators:
          - {operator: "<=", expected_value: 200.0}
      - name: Constant-Current Limit
        key: cc_limit_a
        unit: A
        description: "Load ramped in 2 A steps until the output leaves regulation. 120 % of 237.6 A rated, +-5 % of this line's derivation."
        validators:
          - {operator: ">=", expected_value: 273.0}
          - {operator: "<=", expected_value: 297.0}
      - name: OCP Restart Delay
        key: ocp_restart_s
        unit: s
        description: "Time out of regulation before the module shuts down and restarts, 10 s in the spec, +-1 s here."
        validators:
          - {operator: ">=", expected_value: 9.0}
          - {operator: "<=", expected_value: 11.0}
      - name: OTW Trip
        key: otw_trip_c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 83.0}
          - {operator: "<=", expected_value: 87.0}
      - name: OTP Trip
        key: otp_trip_c
        unit: °C
        validators:
          - {operator: ">=", expected_value: 93.0}
          - {operator: "<=", expected_value: 97.0}
      - name: OTP Minus OTW
        key: otp_minus_otw_c
        unit: °C
        description: "The warning must lead the shutdown by at least 4 C (M-CRPS 8.1.9), or the rack learns of the problem by losing the module."
        validators:
          - {operator: ">=", expected_value: 4.0}
      - name: Recovered After Trips
        key: recovered_after_trips
        description: Output back in regulation after the last fault is cleared.
        validators:
          - {operator: "==", expected_value: true}

teardown:
  - name: Blackbox and Park
    key: blackbox_and_park
    python: phases.blackbox_and_park
    measurements:
      - name: Blackbox After Test
        key: blackbox_after_test
        description: "Exactly the events this test caused: two AC losses, one of each trip, no fan fault. Anything else happened on its own."
        validators:
          - operator: "=="
            expected_value:
              ac_loss: 2
              ovp: 1
              ocp: 1
              otw: 1
              otp: 1
              fan: 0
      - name: Blackbox After Clear
        key: blackbox_after_clear
        description: Ships empty, so the first entry the customer reads happened in their rack.
        validators:
          - operator: "=="
            expected_value:
              ac_loss: 0
              ovp: 0
              ocp: 0
              otw: 0
              otp: 0
              fan: 0
```

Framework features to notice:

1. **`setup:` as a safety gate.** Setup phases run concurrently unless told otherwise; `identify` declares `depends_on: [hipot_and_bond]` so the AC source cannot be enabled while the hipot tester is on the inlet.
2. **Three curves, three aggregation sets.** `efficiency` carries the 277 Vac curve, the 230 Vac curve and the telemetry error on one x-axis; the first two are judged on `min_30_100_pct`, `min_10_30_pct` and `peak_pct`, the third on `max_abs_pct`.
3. **The spec's shape, not a single number.** A stair floor and a peak for efficiency, a table of hold-up times per load, a peak and an RMS for inrush: each is one aggregation with its own limit, and the whole waveform stays attached for the review.
4. **A whole object, compared once.** `psu_identity`, `blackbox_after_test` and `blackbox_after_clear` each validate with one `==`; the report shows which key differed.
5. **`matches` next to `==`.** `reported_serial` must look like a serial; `serial_matches_label` says it is this module's.
6. **A `timeout` for the warm-up.** The efficiency phase is allowed 45 minutes because the spec's 30 minute soak is part of the measurement, not overhead.
7. **`teardown:` always runs.** Load to zero, blackbox read and cleared, source off, whatever a main phase did.

### Hipot and Ground Bond

Three seconds, no AC, and the highest consequence of any step in the sequence. The setup phase forces the source off and the load to zero, applies 2121 Vdc from the shorted line and neutral to the chassis for one second, then 40 A from the inlet earth pin to the chassis for two, and stamps the withstand voltage onto the **unit metadata**:

```python filename="phases/hipot_and_bond.py"
from utils.recipe import GROUND_BOND_A, GROUND_BOND_DWELL_S, HIPOT_DWELL_S, HIPOT_VDC


def hipot_and_bond(measurements, bench, unit, log):
    """Setup: withstand and ground bond with the AC source off. A module
    with a pinched primary lead or a missing earth screw fails here in
    three seconds, before any of the 12 kW ever flows through it."""
    bench.ac_off()
    bench.set_load_a(0.0)
    leak = bench.hipot_dc(HIPOT_VDC, HIPOT_DWELL_S)
    bond = bench.ground_bond(GROUND_BOND_A, GROUND_BOND_DWELL_S)
    measurements.hipot_leakage_ma = leak
    measurements.ground_bond_mohm = bond
    unit.metadata["hipot_vdc"] = HIPOT_VDC
    log.info(f"{unit.serial_number}: {HIPOT_VDC:.0f} Vdc for {HIPOT_DWELL_S:.0f} s leaks {leak:.3f} mA, bond {bond:.1f} mOhm at {GROUND_BOND_A:.0f} A")
```

### Identify

The first power-on, at no load, and the first Modbus transaction. The identity is compared as one object against the release; the serial the module reports is checked against the one on its label. A module carrying another unit's serial passes every electrical test and breaks the rack's inventory the day it is installed.

```python filename="phases/identify.py"
from utils.recipe import AC_INPUTS_V


def identify(measurements, bench, unit, log):
    """Setup: first power-on at 277 Vac with no load, then the model,
    firmware and serial read over Modbus. The serial the PSU reports must
    be the one on its label: a module programmed with another unit's
    identity is a rack that cannot be inventoried."""
    bench.ac_on(AC_INPUTS_V[-1])
    identity = bench.modbus_identity()
    serial = bench.modbus_serial()
    vout = bench.vout_v()
    measurements.psu_identity = identity
    measurements.reported_serial = serial
    measurements.serial_matches_label = serial == unit.serial_number
    measurements.vout_no_load_v = vout
    unit.metadata["fw"] = identity["fw"]
    log.info(f"{identity['model']} fw {identity['fw']} reports {serial}, label {unit.serial_number}, {vout:.2f} V at no load")
```

### Inrush

The source is switched off, the load is zero, and the source closes at the 90 degree phase angle where the bulk capacitors see the full peak. The waveform is recorded whole from the inrush CT; the phase computes the peak and the RMS of the first mains cycle and attaches both as aggregations with the spec's limits.

```python filename="phases/inrush.py"
import numpy as np

from utils.recipe import AC_INPUTS_V, MAINS_HZ


def inrush(measurements, bench, log):
    """Cold start at 277 Vac closed at the 90 degree phase angle, the worst
    case for the bulk capacitors, on the inrush CT. Peak and the RMS of the
    first mains cycle, both limits from section 4.7 of the spec."""
    bench.ac_off()
    bench.set_load_a(0.0)
    t_ms, i_a = bench.inrush_capture(90.0)
    bench.ac_on(AC_INPUTS_V[-1])
    t = np.asarray(t_ms)
    i = np.asarray(i_a)
    first_cycle = i[t < 1000.0 / MAINS_HZ]
    peak = float(np.abs(i).max())
    rms = float(np.sqrt(np.mean(first_cycle**2)))

    measurements.inrush.x_axis = t_ms
    measurements.inrush.y_axis.current = i_a
    measurements.inrush.y_axis.current.aggregations.peak_a = round(peak, 2)
    measurements.inrush.y_axis.current.aggregations.first_cycle_rms_a = round(rms, 2)
    log.info(f"Inrush at 90 deg: peak {peak:.1f} A, first-cycle RMS {rms:.1f} A over {len(i_a)} samples")
```

![Inrush current over 40 ms after the contactor closes at the 90 degree phase angle: a 61.9 A peak decaying through three half-cycles to the steady no-load current, under the 85 A peak limit line, the first 20 ms cycle shaded for the 60 A RMS limit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/orv3-psu-chart-inrush-tTGExnqx8qAIFMnXe03SshcUaMlpEM.png)

The mock module's cold start: 61.9 A on the first half-cycle, decaying with the bulk capacitor charge, 20.9 A RMS over the first cycle. A module with its NTC inrush limiter bypassed shows the same shape at twice the height.

### Efficiency

Full load for the spec's 30 minutes, then six load points at each end of the input range, input and output power from the analyzer. At each point the phase also reads the output current the module reports over Modbus and records the error against the load's own reading: the rack's power capping runs on that telemetry, and a module that reports 2 % low lets the shelf oversubscribe. The mock skips the soak; the `timeout` is sized for the real one.

```python filename="phases/efficiency.py"
import numpy as np

from utils.recipe import AC_INPUTS_V, LOAD_POINTS_PCT, RATED_A, WARM_UP_MIN


def sweep(bench, vin, log):
    """Six load points at one input voltage: analyzer in and out, and the
    output current the PSU reports next to the load's own reading."""
    bench.ac_on(vin)
    eff = []
    telemetry_err = []
    for pct in LOAD_POINTS_PCT:
        amps = pct / 100.0 * RATED_A
        bench.set_load_a(amps)
        pin, pout = bench.analyzer_pin_pout_w(pct)
        eff.append(100.0 * pout / pin)
        telemetry_err.append(100.0 * (bench.modbus_iout_a() - amps) / amps)
        log.info(f"{vin:.0f} Vac, {pct:3d} %: {pin:.0f} W in, {pout:.0f} W out, {eff[-1]:.2f} %, Iout telemetry {telemetry_err[-1]:+.2f} %")
    return np.array(eff), np.array(telemetry_err)


def efficiency(measurements, bench, log):
    """Efficiency at both ends of the input range after the spec's 30 min
    warm-up at full load. The ORV3 floor is a stair (94 % below 30 % load,
    96.5 % above) and a peak (97.5 %); the Titanium points sit under it."""
    bench.set_load_a(RATED_A)
    log.info(f"Warm-up: {WARM_UP_MIN} min at {RATED_A:.0f} A (mock: instant)")
    curves = {}
    for vin in AC_INPUTS_V:
        curves[vin] = sweep(bench, vin, log)
    pts = np.array(LOAD_POINTS_PCT)
    low = pts < 30
    high = pts >= 30

    m = measurements.efficiency
    m.x_axis = LOAD_POINTS_PCT
    e277, err = curves[277.0]
    e230, _ = curves[230.0]
    m.y_axis.eff_277 = e277.round(2).tolist()
    m.y_axis.eff_277.aggregations.min_30_100_pct = round(float(e277[high].min()), 2)
    m.y_axis.eff_277.aggregations.min_10_30_pct = round(float(e277[low].min()), 2)
    m.y_axis.eff_277.aggregations.peak_pct = round(float(e277.max()), 2)
    m.y_axis.eff_230 = e230.round(2).tolist()
    m.y_axis.eff_230.aggregations.min_30_100_pct = round(float(e230[high].min()), 2)
    m.y_axis.eff_230.aggregations.min_10_30_pct = round(float(e230[low].min()), 2)
    m.y_axis.eff_230.aggregations.peak_pct = round(float(e230.max()), 2)
    m.y_axis.iout_telemetry_err_pct = err.round(2).tolist()
    m.y_axis.iout_telemetry_err_pct.aggregations.max_abs_pct = round(float(np.abs(err).max()), 2)
    log.info(f"Peak {e277.max():.2f} % at 277 Vac, {e230.max():.2f} % at 230 Vac; floor {min(e277[high].min(), e230[high].min()):.2f} % above 30 % load")
```

### Ripple and Load Step

Ripple at full load in the spec's bandwidth, then the dynamic load of section 4.15: from 70 to 130 % of rated at 3 A/µs and back. The deviation from the set-point is recorded whole in both directions; the phase finds the last sample outside the ±1 % band and reports the next one as the settle time.

```python filename="phases/ripple_and_transient.py"
import numpy as np

from utils.recipe import DYN_CAPTURE_MS, DYN_LOAD_PCT, DYN_SLEW_A_PER_US, RATED_A, RIPPLE_BW_MHZ, SETTLE_BAND_V


def settle_ms(t_ms, dev_v):
    """Last excursion outside the +-1 % band; settled from the next sample."""
    t = np.asarray(t_ms)
    outside = np.where(np.abs(np.asarray(dev_v)) > SETTLE_BAND_V)[0]
    return float(t[outside[-1] + 1]) if len(outside) else 0.0


def ripple_and_transient(measurements, bench, log):
    """Ripple at full load in the spec's 20 MHz bandwidth, then the dynamic
    load step of section 4.15 in both directions: 70 to 130 % of rated at
    3 A/us. The deviation limit keeps the overshoot under the OVP threshold
    and the undershoot inside the regulation window."""
    bench.set_load_a(RATED_A)
    ripple = bench.ripple_mvpp(RIPPLE_BW_MHZ)
    measurements.ripple_mvpp = ripple

    lo, hi = DYN_LOAD_PCT
    bench.set_load_a(lo / 100.0 * RATED_A)
    t_ms, up = bench.transient_capture(lo, hi, DYN_SLEW_A_PER_US)
    _, down = bench.transient_capture(hi, lo, DYN_SLEW_A_PER_US)
    bench.set_load_a(RATED_A)
    up_settle = settle_ms(t_ms, up)
    down_settle = settle_ms(t_ms, down)

    m = measurements.load_step
    m.x_axis = t_ms
    m.y_axis.step_up = up
    m.y_axis.step_up.aggregations.min_v = round(float(min(up)), 3)
    m.y_axis.step_up.aggregations.settle_ms = round(up_settle, 2)
    m.y_axis.step_down = down
    m.y_axis.step_down.aggregations.max_v = round(float(max(down)), 3)
    m.y_axis.step_down.aggregations.settle_ms = round(down_settle, 2)
    log.info(f"Ripple {ripple:.0f} mVpp; step {lo}->{hi} % undershoots {min(up):.2f} V, settles in {up_settle:.2f} ms; {hi}->{lo} % overshoots {max(down):+.2f} V, settles in {down_settle:.2f} ms over a {DYN_CAPTURE_MS:.0f} ms capture")
```

![Output deviation on the 70 to 130 % load step, left, undershooting 0.81 V and settling into the 1 % band at 0.57 ms, and on the 130 to 70 % step, right, overshooting 0.75 V and settling at 0.13 ms, both inside the 1.5 V and 3 ms limits.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/orv3-psu-chart-loadstep-eiCgdRDAzHN6xh3xINn70F6tzCVlui.png)

The two steps on the mock module: a 0.9 kHz loop with a 1.2 ms envelope, inside the band in well under a millisecond. The 1.5 V limit is not in the spec; it is the distance from the 50.5 V set-point to the 52.5 V OVP threshold with half a volt to spare, so a module that passes here cannot trip its own OVP on a tray hot-plug.

### Hold-Up

AC removed at full load and at 120 %, the output recorded whole each time. The module regulates from its bulk capacitors until the bus reaches UVLO, then the output capacitors discharge into the load; the phase reports the time the output stayed above the regulation floor as the hold-up, and the worst margin between the AC_Loss_L assertion and the collapse as the warning the rack gets.

```python filename="phases/hold_up.py"
import numpy as np

from utils.recipe import AC_INPUTS_V, HOLDUP_LOADS_PCT, RATED_A, REGULATION_LOW_V


def hold_up(measurements, bench, log):
    """AC removed at full load and at 120 % (Table 8): the output must stay
    in regulation for 20 ms and 16.67 ms, and AC_Loss_L must lead the
    collapse by at least 5 ms so the rack can shed load. Hold-up is a
    capacitor-energy budget; a bulk capacitor from the wrong reel shows up
    here and nowhere else."""
    margins = []
    traces = {}
    for pct in HOLDUP_LOADS_PCT:
        bench.set_load_a(pct / 100.0 * RATED_A)
        t_ms, v, ac_loss_ms = bench.ac_loss_capture(pct)
        bench.ac_on(AC_INPUTS_V[-1])
        t = np.asarray(t_ms)
        below = np.where(np.asarray(v) < REGULATION_LOW_V)[0]
        held = float(t[below[0]]) if len(below) else float(t[-1])
        margins.append(held - ac_loss_ms)
        traces[pct] = (t_ms, v, held)
        log.info(f"AC loss at {pct} %: output above {REGULATION_LOW_V} V for {held:.1f} ms, AC_Loss_L after {ac_loss_ms:.1f} ms")
    m = measurements.hold_up
    m.x_axis = traces[100][0]
    m.y_axis.vout_100 = traces[100][1]
    m.y_axis.vout_100.aggregations.hold_up_ms = round(traces[100][2], 1)
    m.y_axis.vout_120 = traces[120][1]
    m.y_axis.vout_120.aggregations.hold_up_ms = round(traces[120][2], 1)
    measurements.ac_loss_warning_margin_ms = round(min(margins), 1)
    bench.set_load_a(RATED_A)
```

![Output voltage after AC loss: flat at 50.5 V for 23.0 ms at 100 % load in green and 19.2 ms at 120 % in blue, then collapsing, against the 20 ms and 16.67 ms limit lines and the AC_Loss_L assertion at 2.6 ms.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/orv3-psu-chart-holdup-kNOSY3hZRY7XNebC0XwDFrvQ8werwR.png)

Hold-up is an energy budget: 7.2 mF between 400 V and 290 V holds 12 kW for 23 ms. A bulk capacitor at the low end of its tolerance, or one from the wrong reel, moves the knee left and fails the 120 % row first. The signal margin, 16.6 ms on the mock against the 5 ms the rack needs, is what lets the shelf's other modules pick up the load before the busbar sags.

### Protections

Three trips, each cleared before the next. OVP is found by raising the module's own set-point in 0.1 V steps over the management bus until the output latches off, with the response timed on the scope; OCP by ramping the load in 2 A steps until the output leaves regulation, then timing the restart from the fault log; OTW and OTP by overriding the thermistor reading in 0.2 °C steps. After the last clear, the output must be back in regulation.

```python filename="phases/protections.py"
from utils.recipe import OCP_RAMP_STEP_A, OTP_SWEEP_C, OVP_TRIM_STEP_V, RATED_A, REGULATION_LOW_V, VOUT_SET_V


def protections(measurements, bench, log):
    """OVP by raising the set-point over the management bus in test mode,
    OCP by ramping the load into constant current and timing the restart,
    OTW and OTP by overriding the thermistor reading. Each trip is cleared
    and the output checked back in regulation before the next."""
    bench.set_load_a(0.5 * RATED_A)

    v = VOUT_SET_V
    while not bench.modbus_faults()["ovp"]:
        v = round(v + OVP_TRIM_STEP_V, 2)
        bench.modbus_trim_vout(v)
    ovp_ms = bench.ovp_response_ms()
    bench.modbus_clear_faults()
    log.info(f"OVP at {v:.2f} V, output off in {ovp_ms:.0f} ms")

    amps = RATED_A
    while not bench.modbus_faults()["ocp"]:
        amps += OCP_RAMP_STEP_A
        bench.set_load_a(amps)
    cc_pct = 100.0 * amps / RATED_A
    ocp_s = bench.ocp_shutdown_s()
    bench.set_load_a(0.5 * RATED_A)
    bench.modbus_clear_faults()
    log.info(f"Constant current from {amps:.0f} A ({cc_pct:.0f} %), restart after {ocp_s:.1f} s out of regulation")

    otw_c = None
    otp_c = None
    temp = OTP_SWEEP_C[0]
    while temp <= OTP_SWEEP_C[1]:
        bench.modbus_override_temp_c(temp)
        faults = bench.modbus_faults()
        if faults["otw"] and otw_c is None:
            otw_c = temp
        if faults["otp"]:
            otp_c = temp
            break
        temp = round(temp + OTP_SWEEP_C[2], 1)
    bench.modbus_clear_faults()
    recovered = bench.vout_v() > REGULATION_LOW_V
    log.info(f"OTW at {otw_c:.1f} C, OTP at {otp_c:.1f} C, output back at {bench.vout_v():.2f} V")

    measurements.ovp_trip_v = v
    measurements.ovp_response_ms = ovp_ms
    measurements.cc_limit_a = round(amps, 1)
    measurements.ocp_restart_s = ocp_s
    measurements.otw_trip_c = otw_c
    measurements.otp_trip_c = otp_c
    measurements.otp_minus_otw_c = round(otp_c - otw_c, 1)
    measurements.recovered_after_trips = recovered
```

The 4 °C rule between OTW and OTP is the one to keep: a module whose warning and shutdown thresholds are the same value never warns, and the rack learns about the fan failure by losing 12 kW.

### Blackbox and Park

The teardown phase reads the blackbox and records it as one object: two AC losses, one of each trip, no fan fault. Anything else in that log happened on its own during the test and is a reason to look. Then the log is cleared, so the first entry the customer reads was written in their rack, and the load and source go off.

```python filename="phases/blackbox_and_park.py"
def blackbox_and_park(measurements, bench, log):
    """Teardown: the blackbox must hold exactly the events this test caused,
    then it is cleared so the log the customer reads starts in their rack.
    Load to zero and AC off last, whatever happened before."""
    bench.set_load_a(0.0)
    before = bench.modbus_blackbox()
    bench.modbus_clear_blackbox()
    after = bench.modbus_blackbox()
    bench.ac_off()
    measurements.blackbox_after_test = before
    measurements.blackbox_after_clear = after
    log.info(f"Blackbox held {before}, cleared to {after}, AC off")
```

### Mock Plug

`PsuBench` stands in for six instruments and the DUT because every reading depends on what the source and the load are doing at that instant, and plugs run in separate processes. Its efficiency is a table peaking at 97.8 % at half load with 0.03 % of noise, its inrush a 61.8 A peak decaying over 8 ms, its bulk capacitance 7.2 mF between a 400 V bus and a 290 V UVLO, its load-step response a 0.9 kHz loop with a 1.2 ms envelope, its comparators at 53.12 V, 120.4 % of rated, 85.4 °C and 95.2 °C, and its blackbox a counter that increments on every cleared fault and every AC loss. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, split the class into the AC source and the load over SCPI, the analyzer and the hipot tester over their drivers, the scope for the inrush, ripple, load-step, hold-up and OVP captures, and the Modbus port through pymodbus. Time the AC_Loss_L edge and the OVP response on the scope, not on a polled register; interlock the hipot tester against the source; and add the 150 % hold-up row when the station gets a load that can sink it. The phases, measurements and limits stay the same.


### Cold Plate Leak and Thermal Qualification
URL: https://www.tofupilot.com/templates/cold-plate-leak-and-thermal-qualification
Qualify a direct-to-chip cold plate: N2 pressure decay, hydrostatic at 1x and 3x MOP, pressure drop as loop minus bypass, thermal resistance and a flush.

## Introduction

### Cold Plate Qualification Overview

A direct-to-chip liquid-cooled server has no fan moving the heat off its accelerators. A copper cold plate sits on each 1 kW class GPU or CPU through a thermal interface, coolant enters through one quick-disconnect stub, passes a field of microchannel fins a few hundred micrometres wide, and leaves through the other stub about 10 K warmer. Dozens of these plates hang in series and parallel off a tray manifold, the tray off a rack manifold, the rack off a coolant distribution unit, and the whole secondary loop runs PG25, a 25 % propylene glycol water mix with a corrosion inhibitor package. One plate that ships out of spec is not one warm chip: it is a weep at a lid braze that puts glycol on a 10 kW board, a blocked channel that throttles one accelerator in a training job of a thousand, a burr of brazing flux that lodges in the next plate's fins, or a plate shipped wet that freezes in the container.

![Sealed copper cold plate with two fluid ports, a lid seam, and mounting ears.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/cold-plate-leak-and-thermal-qualification-matte-closeup-gZo0eXYIhHbGuvU3O7guY0Ksa2KkU3.webp)

The parts this test exercises: the two quick-disconnect stubs and their seals, the lid-to-body braze or weld that holds the maximum operating pressure, the fin field that sets the thermal resistance and the pressure drop, and whatever the machining and brazing left inside it.

The [OCP Cold Plate Development and Qualification white paper](https://www.opencompute.org/documents/ocp-cold-plate-development-and-qualification-with-integrated-comments-pdf) gives a production test writer most of the numbers. Leak tightness is a **pressure decay under 0.5 %** over the hold (the EN 1779 pressure-change method); burst margin is a **hydrostatic dwell at 1x the maximum operating pressure for 5 min then 3x for 2 min**, quoted from IEC 62368-1; thermal performance is **R = (Tcase - Tinlet) / P** measured at several flows; pressure drop is **the loop with the plate minus the same loop through a bypass**, so hoses and quick-disconnects do not end up in the plate's number; cleanliness is **no particle above 50 um** after the flush; and the coolant velocity stays **under 1.5 m/s** so the copper does not erode. The [OCP ACS cold plate requirements](https://www.opencompute.org/documents/ocp-acs-liquid-cooling-cold-plate-requirements-pdf) set the interfaces, the OCP Liquid Cooling Integration and Logistics white paper puts a gaseous leak test at L10 before any coolant enters a server, and the [Catalina compute tray specification](https://www.opencompute.org/documents/catalina-compute-tray-ocp-specification-v1d33-pdf-1) fixes the rack boundary this plate lives in: **40 C supply, 1.25 to 1.5 LPM/kW**, 15 psi across the rack loop. Every limit in this template is one of those numbers, or says where it came from instead.

### Test Purpose

The procedure records one qualification fingerprint per plate:

- **Coolant conductivity and supply temperature** of the bench loop, stamped on the run as metadata, the conductivity also judged, and the **dry tare mass** of the plate
- **Dry N2 pressure decay** at the maximum operating pressure, the whole 96 s trace, judged on the decay over the 60 s hold, with one retry for a stub not fully seated
- **Hydrostatic** at 1x MOP for 5 min then 3x for 2 min with the pump isolated, the trace whole, judged on the minimum of each dwell, and the **operator's deformation check** as a switch bound to a measurement
- **Pressure drop** at six flows around the design flow, the loop, the bypass and the difference, judged at the design flow only, with the **velocity in the stubs**
- **Thermal resistance** at four power steps to 1 kW at the design flow, case and inlet temperature and R at each, with the **energy balance** closed at the outlet
- **Particle counts** per size bin after a 10 min flush, judged on the bin above 50 um, and the inline filter's dP settled
- The **residual coolant** after the blow-down against the dry tare, the caps confirmed, and the **raw pressure log** attached to the run

![Case rise above the 40 °C inlet against heater power at 1.5 LPM: 4.0, 7.9, 12.0 and 16.0 K at 250, 500, 750 and 1000 W on a 0.0159 K/W line, under the 0.020 K/W limit line that reaches 20 K at 1 kW, with the case temperatures 44.1 to 56.2 °C annotated.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/cold-plate-chart-thermal-rAZnkWAc9CWAbyQjXILK9I3dqFAMVm.png)

The mock plate's thermal curve: 16.0 K of case rise at 1 kW on a 40.1 C supply, 0.016 K/W, against a limit that allows 20 K. A plate with one channel blocked by flux runs the same line steeper and closes the heat balance short.

The framework mechanics on show are `then: {fail: retry}` with a retry limit of one on the pressure decay, so the common false fail gets a second seating before the run stops; an operator switch bound to a boolean measurement with `bind:`; `run.metadata` for the bench's coolant state; previous-results injection, the teardown reading the dry tare from the identify phase by naming it as a parameter; `attach.data` for the raw pressure log; and multi-dimensional measurements where the loop and the bypass curves are recorded next to the plate curve that is judged.

## Equipment & Setup

To run this qualification on a production line, the following are required:

- **A dry N2 supply** with a regulator and a pressure transducer resolving 0.01 kPa at 300 kPa gauge, for the decay
- **A fill and hydrostatic pump** reaching 3x the plate's MOP with an isolation valve, so the dwells are held by the plate and not by the pump
- **A flow loop** with a flow meter, a bypass valve, two differential pressure cells and a recirculating chiller holding a 40 C PG25 supply
- **A heater block** with a PID controller to 1 kW, a class A RTD in its pedestal for the case temperature, and inlet and outlet RTDs in the loop
- **An inline 50 um filter** with its own dP cell and a **liquid particle counter** on the effluent
- **A conductivity probe** in the loop and a **scale** under the fixture resolving 0.1 g at 1 kg
- **The Device Under Test (DUT)**: a copper microchannel cold plate for a 1 kW class device with two quick-disconnect stubs, dry, as it leaves brazing and machining
- **A TofuPilot Framework procedure** to sequence the stimuli and validate the limits
- **The TofuPilot Dashboard** to keep the per-plate evidence and the pressure logs a customer's incoming inspection will ask for

### Hardware Components

#### Flow Loop, Pressure Cells and Chiller

The plate's own pressure drop is a subtraction, so the two differential pressure cells matter more than the flow meter: an [Endress+Hauser](https://www.endress.com/) Deltabar or an equivalent wet-wet dP cell spanning 0 to 50 kPa across the plate and across the bypass, with the same tapping points, and a Coriolis or ultrasonic flow meter from [Bronkhorst](https://www.bronkhorst.com/) or [Keyence](https://www.keyence.com/) resolving 0.005 LPM at 1.5. The chiller has to hold the supply within 0.1 K through a 1 kW step, which is a [Julabo](https://www.julabo.com/) or [Huber](https://www.huber-online.com/) recirculator sized well above the heater, not a bench chiller at its limit. The N2 side is a regulator, a solenoid, and a transducer good to 0.01 kPa; the hydrostatic side is a small piston or diaphragm pump reaching 1 MPa and a valve to isolate it.

![Sealed copper cold plate clamped to a heater-block fixture with two connected fluid hoses and a sensor lead.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/cold-plate-leak-and-thermal-qualification-matte-cover-DT9k34kcJPGwDHTS1RRipnt3RdFrmr.webp)

The illustrated fixture: a sealed cold plate clamped to a heater block, with both fluid ports connected. The pump, nitrogen supply, coolant loop and weighing equipment are omitted.

#### Heater Block, Particle Counter and Scale

The heater block stands in for the accelerator: a copper pedestal with cartridge heaters driven by a [Watlow](https://www.watlow.com/) controller, a class A RTD in a hole a millimetre under the contact face, and the same thermal interface material the product will use. The case temperature is measured there and nowhere else, so every R this bench reports carries the same interface. The particle counter is a light-extinction instrument on the flush effluent, a [PAMAS](https://www.pamas.de/) S40 or a Beckman Coulter HIAC class, reporting counts per 100 mL in size bins; the inline filter ahead of the counter is the 50 um element the rack's CDU would carry anyway, with a dP cell across it. The scale is a [Mettler Toledo](https://www.mt.com/) precision balance under the fixture, tared once per plate before the stubs go on and read again after the blow-down. The quick-disconnects on the bench side are the rack's, Amphenol UQD, Parker or CPC depending on the program, so a stub that seals here seals in the tray.

### Where the Limits Come From

| Test | Limit | Basis |
|---|---|---|
| Coolant conductivity | 3000 to 6000 uS/cm | inhibited PG25 reads about 4700 uS/cm at 20 C on supplier data sheets, and the OCP PG25 base spec limits the blend water to 25 uS/cm; the window is this line's |
| Dry mass | 695 to 745 g | the plate's drawing, 720 g nominal |
| Pressure decay | under 0.5 % over the 60 s hold, from at least 285 kPa | OCP cold plate qualification (EN 1779 method) |
| Hydrostatic | min 285 kPa over the 1x dwell (5 min), min 855 kPa over the 3x dwell (2 min) | IEC 62368-1 as OCP quotes it for the pressures and times; the 5 % floor is this line's |
| Deformation | none visible | OCP hydrostatic acceptance |
| Pressure drop | at or under 15 kPa across the plate at 1.5 LPM | OCP for the loop-minus-bypass method and the Catalina rack boundary for 1.5 LPM/kW; the 15 kPa is this line's budget, two plates in series and the tray manifold inside the rack's 15 psi |
| Stub velocity | at or under 1.5 m/s | OCP erosion-corrosion limit for copper |
| Thermal resistance | mean and full-power R at or under 0.020 K/W, case at or under 60 C, inlet 38 to 42 C | OCP for R = (Tcase - Tinlet) / P; 0.020 K/W is this line's derivation, a 60 C case ceiling on the Catalina 40 C supply |
| Heat balance | 90 to 110 % at 1 kW | this line's, closing the energy balance at the outlet RTD |
| Particulate | zero counts above 50 um, filter dP risen less than 0.2 kPa | OCP for the 50 um rule; the filter criterion is this line's |
| Residual coolant | at or under 3.0 g | this line's shipping limit, about 10 % of the 28 mL fill |

Two rows in that table are traps. The decay percentage depends on when the hold starts: gas compressed into a cold plate warms, then cools back over tens of seconds, and a hold that starts too early reads a thermal transient as a leak. The procedure waits 30 s after the ramp and judges the next 60 s, and the chart shows both windows. And the thermal resistance depends on the interface as much as on the plate, so the bench uses the product's own TIM and the same pedestal for every plate; a number from a different interface is a different measurement.

## Test Procedure

### Overview

The procedure maps the qualification onto the framework's three stages. The tare and the coolant state live in `setup:` with the N2 decay chained after them, so the plate is weighed dry before the stubs go on and no coolant enters a plate that leaks gas. The drain, the blow-down and the residual live in `teardown:` so a plate goes back on the scale dry even when a main phase fails.

1. Setup: dry tare on the scale, coolant conductivity and supply temperature on the run.
2. Setup: dry N2 to 300 kPa, 30 s stabilise, 60 s hold, one retry.
3. Main: fill with PG25, 300 kPa for 5 min then 900 kPa for 2 min, deformation switch.
4. Main: six flows from 0.75 to 2.0 LPM, loop and bypass dP, plate dP and stub velocity.
5. Main: 250 to 1000 W at 1.5 LPM, R at each step, heat balance at 1 kW.
6. Main: 10 min flush at 3 LPM, particle counts per bin, filter dP settled.
7. Teardown: drain, blow-down, caps, residual against the tare, pressure log attached.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-plate evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  identify.py
  dry_pressure_decay.py
  hydrostatic.py
  flow_pressure_drop.py
  thermal_resistance.py
  particulate_flush.py
  drain_and_cap.py
plugs/
  cold_plate_bench.py
utils/
  recipe.py
ui.json
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-cold-plate-qualification). The `ColdPlateBench` plug is a mock of the N2 supply, the pump, the flow loop with its dP cells, the heater block with its RTDs, the filter, the particle counter, the conductivity probe and the scale together, synthesizing a healthy plate that decays 0.08 % over the hold, drops 9.96 kPa at 1.5 LPM and runs 0.016 K/W at 1 kW, so the procedure runs end-to-end without a bench or a plate connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless, with the deformation switch answered from `ui.json`:

```bash
tofupilot run . --no-tui --no-kiosk --json --ui-values ui.json --ui-timeout 60
```

### The Procedure File

`procedure.yaml` declares the unit, the bench plug with the label serial as constructor argument, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Cold Plate Leak and Thermal Qualification
version: 0.1.0
description: Qualification of a direct-to-chip liquid cold plate for a 1 kW class GPU or CPU, OCP ACS style, PG25 coolant. Coolant conductivity and supply temperature on the run, dry N2 pressure decay before any coolant with one retry for a QD not seated, hydrostatic at 1x then 3x the maximum operating pressure with the operator's deformation check bound to a switch, the plate's own pressure drop as loop minus bypass across a flow sweep, thermal resistance at four power steps with the energy balance closed at the outlet, a flush judged on nothing above 50 um, and a teardown that drains, caps, weighs the residual against the dry tare and attaches the raw pressure log.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the laser mark on the plate's lid"
    placeholder: "CP-1K-000000"
    pattern: "^CP-1K-\\d{6}$"
    default_value: "CP-1K-003118"
  part_number:
    default_value: "CP-1K-CU-QD6"
  batch_number:
    default_value: "LOT-2026-38"

plugs:
  - name: Cold Plate Bench
    description: "N2 supply, fill and hydrostatic pump, flow loop with bypass and two dP cells, heater block with RTDs, inline filter and particle counter, conductivity probe and scale (mock, one plug per bench)"
    python: plugs.cold_plate_bench:ColdPlateBench
    key: bench
    config:
      serial_number: "CP-1K-003118"

setup:
  - name: Identify
    key: identify
    python: phases.identify
    measurements:
      - name: Coolant Conductivity
        key: coolant_conductivity_us_cm
        unit: µS/cm
        description: "The bench loop's PG25, at the conductivity probe. Inhibited PG25 reads about 4700 uS/cm (supplier data sheets); a low reading is a loop topped up with water, a high one is contamination. The window is this line's."
        validators:
          - {operator: ">=", expected_value: 3000.0}
          - {operator: "<=", expected_value: 6000.0}
      - name: Dry Mass
        key: dry_mass_g
        unit: g
        description: "Dry tare on the scale before anything is connected, 720 g nominal. A plate outside the window is missing a fin field or carries the wrong lid; the teardown subtracts this from the wet mass."
        validators:
          - {operator: ">=", expected_value: 695.0}
          - {operator: "<=", expected_value: 745.0}

  - name: Dry Pressure Decay
    key: dry_pressure_decay
    python: phases.dry_pressure_decay
    depends_on: [identify]
    retry:
      limit: 1
    then:
      fail: retry
    measurements:
      - name: Pressure Decay
        key: pressure_decay
        title: Dry N2 at the maximum operating pressure, 30 s stabilise, 60 s hold
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Pressure
            key: pressure
            unit: kPa
            aggregations:
              - type: hold_start_kpa
                unit: kPa
                validators:
                  - {operator: ">=", expected_value: 285.0}
              - type: decay_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 0.5}

main:
  - name: Hydrostatic
    key: hydrostatic
    python: phases.hydrostatic
    ui:
      components:
        - key: deformation_check
          type: switch
          label: "No visible deformation after the 3x dwell"
          description: "Lid flat under the straightedge, no bulge over the fin field, no weep at the stubs or the lid seam"
          required: true
          bind: measurements.no_deformation
    measurements:
      - name: Hydrostatic Pressure
        key: hydrostatic
        title: Filled with PG25, 1x MOP for 5 min then 3x MOP for 2 min, pump isolated during each dwell
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Pressure
            key: pressure
            unit: kPa
            aggregations:
              - type: min_1x_dwell_kpa
                unit: kPa
                validators:
                  - {operator: ">=", expected_value: 285.0}
              - type: min_3x_dwell_kpa
                unit: kPa
                validators:
                  - {operator: ">=", expected_value: 855.0}
      - name: No Deformation
        key: no_deformation
        description: "The operator's inspection after the 3x dwell, recorded through the switch bound to this measurement."
        validators:
          - {operator: "==", expected_value: true}

  - name: Flow and Pressure Drop
    key: flow_pressure_drop
    python: phases.flow_pressure_drop
    depends_on: [hydrostatic]
    measurements:
      - name: Flow Curve
        key: flow_curve
        title: Pressure drop across the plate as loop minus bypass, six flows around 1.5 LPM
        x_axis:
          legend: Flow
          unit: LPM
        y_axis:
          - legend: Plate dP (loop minus bypass)
            key: plate_dp
            unit: kPa
            aggregations:
              - type: at_design_flow_kpa
                unit: kPa
                validators:
                  - {operator: "<=", expected_value: 15.0}
          - legend: Loop dP with plate
            key: loop_dp
            unit: kPa
          - legend: Bypass dP
            key: bypass_dp
            unit: kPa
          - legend: Velocity in the QD stubs
            key: stub_velocity
            unit: m/s
            aggregations:
              - type: at_design_flow_m_s
                unit: m/s
                validators:
                  - {operator: "<=", expected_value: 1.5}

  - name: Thermal Resistance
    key: thermal_resistance
    python: phases.thermal_resistance
    depends_on: [flow_pressure_drop]
    timeout: 20m
    measurements:
      - name: Thermal Curve
        key: thermal_curve
        title: Case and inlet temperature and R = (Tcase - Tinlet) / P at four power steps, 1.5 LPM, 40 C supply
        x_axis:
          legend: Heater power
          unit: W
        y_axis:
          - legend: Case temperature
            key: tcase_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 60.0}
          - legend: Inlet temperature
            key: tin_c
            unit: °C
            aggregations:
              - type: mean_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: 38.0}
                  - {operator: "<=", expected_value: 42.0}
          - legend: Thermal resistance
            key: r_k_per_w
            unit: K/W
            aggregations:
              - type: mean_k_per_w
                unit: K/W
                validators:
                  - {operator: "<=", expected_value: 0.020}
              - type: at_full_power_k_per_w
                unit: K/W
                validators:
                  - {operator: "<=", expected_value: 0.020}
      - name: Coolant Heat Balance
        key: coolant_heat_balance_pct
        unit: "%"
        description: "Heat carried away by the coolant (mass flow, cp, outlet minus inlet) as a share of the heater power at 1 kW. Below 90 % a channel is blocked or the case RTD sits on a hot spot; above 110 % the flow meter or an RTD is wrong."
        validators:
          - {operator: ">=", expected_value: 90.0}
          - {operator: "<=", expected_value: 110.0}

  - name: Particulate Flush
    key: particulate_flush
    python: phases.particulate_flush
    depends_on: [thermal_resistance]
    timeout: 20m
    measurements:
      - name: Particle Counts
        key: particle_counts
        title: Effluent particle counts per 100 mL by size bin after a 10 min flush at 3 LPM
        x_axis:
          legend: Size bin lower edge
          unit: µm
        y_axis:
          - legend: Count per 100 mL
            key: count
            aggregations:
              - type: over_50um
                validators:
                  - {operator: "==", expected_value: 0}
      - name: Filter dP Stable
        key: filter_dp_stable
        description: "The inline 50 um filter's dP rose by less than 0.2 kPa over the flush, so the plate had stopped shedding when the flush ended."
        validators:
          - {operator: "==", expected_value: true}

teardown:
  - name: Drain and Cap
    key: drain_and_cap
    python: phases.drain_and_cap
    measurements:
      - name: Residual Coolant
        key: residual_coolant_g
        unit: g
        description: "Wet mass after the N2 blow-down minus the dry tare from identify. About 10 % of the 28 mL fill is this line's shipping limit for a plate shipped drained."
        validators:
          - {operator: "<=", expected_value: 3.0}
      - name: QDs Capped
        key: qds_capped
        validators:
          - {operator: "==", expected_value: true}
```

Framework features to notice:

1. **`then: {fail: retry}` with `retry: {limit: 1}`.** A stub not fully seated on the bench's quick-disconnect is the common reason a good plate fails the decay. The phase gets one more attempt before the run stops; every attempt is kept, and the dashboard shows both traces side by side.
2. **A switch bound to a measurement.** The operator's inspection after the 3x dwell is a boolean the YAML validates with `==`, through `bind: measurements.no_deformation`. The form is up while the trace records; the phase waits for the answer once it is done.
3. **`run.metadata` for the bench state.** The coolant, its conductivity and the supply temperature go onto the run, so every R in the dashboard can be filtered by the loop it was measured on.
4. **Previous results by parameter name.** `drain_and_cap` takes `identify` as a parameter and reads `identify.dry_mass_g`; the residual is a subtraction across two phases without a global.
5. **`attach.data` for the raw log.** The decay and the hydrostatic traces go with the run as one JSON file, so a customer's incoming inspection gets the transducer samples, not a summary.
6. **Curves recorded next to the one judged.** `flow_curve` carries the loop, the bypass and the difference on one x-axis; only `plate_dp` has an aggregation with a limit, the other two stay for the review.
7. **`setup:` chained, `teardown:` always.** The decay depends on the identify phase so the plate is weighed before the stubs go on; the drain runs whatever a main phase did.

### Identify

The plate is on the scale, dry, with nothing connected. The bench loop's conductivity and supply temperature are read once and written to the run as metadata; the conductivity is also a measurement with a window, because a loop topped up with water has half the inhibitor and a diluted coolant transfers heat differently from the one the rack will run. The dry mass is the tare the teardown will subtract, and a plate outside its window is missing a fin field or carries the wrong lid.

```python filename="phases/identify.py"
from utils.recipe import COOLANT


def identify(measurements, bench, unit, run, log):
    """Setup: the plate is on the scale, dry, with nothing connected. The
    bench's coolant conductivity and supply temperature go onto the run as
    metadata, so a thermal resistance measured against a diluted or a warm
    loop can be found later without opening the trace."""
    cond = bench.coolant_conductivity_us_cm()
    tin = bench.inlet_temp_c()
    dry = bench.dry_mass_g()
    run.metadata["coolant"] = COOLANT
    run.metadata["coolant_conductivity_us_cm"] = cond
    run.metadata["coolant_supply_c"] = tin
    measurements.coolant_conductivity_us_cm = cond
    measurements.dry_mass_g = dry
    unit.metadata["dry_mass_g"] = dry
    log.info(f"{unit.serial_number}: {dry:.1f} g dry, {COOLANT} at {cond:.0f} uS/cm, supply {tin:.2f} C")
```

### Dry Pressure Decay

The OCP L10 gaseous leak step, before any coolant. Dry N2 to the maximum operating pressure, then 30 s for the gas to give back the heat of compression, then a 60 s hold judged on the first sample minus the last. The trace is recorded whole; the phase attaches the pressure at the start of the hold, so a hold that never reached the test pressure fails on its own aggregation rather than passing on a small decay of a small number.

```python filename="phases/dry_pressure_decay.py"
import numpy as np

from utils.recipe import DECAY_HOLD_S, DECAY_STABILISE_S, DECAY_TEST_KPA


def dry_pressure_decay(measurements, bench, log):
    """Setup: the OCP L10 gaseous leak step before any coolant. Dry N2 to
    the test pressure, 30 s for the gas to give back its compression heat,
    then a 60 s hold judged on the decay between its first and last sample.
    A quick-disconnect not fully seated is the common false fail, which is
    why the YAML retries this phase once before stopping the run."""
    t_s, p_kpa = bench.n2_pressure_decay(DECAY_TEST_KPA, DECAY_STABILISE_S, DECAY_HOLD_S)
    bench.vent()
    t = np.asarray(t_s)
    p = np.asarray(p_kpa)
    hold_start = t[-1] - DECAY_HOLD_S
    hold = p[t >= hold_start]
    decay_pct = 100.0 * (hold[0] - hold[-1]) / hold[0]

    m = measurements.pressure_decay
    m.x_axis = t_s
    m.y_axis.pressure = p_kpa
    m.y_axis.pressure.aggregations.hold_start_kpa = round(float(hold[0]), 2)
    m.y_axis.pressure.aggregations.decay_pct = round(float(decay_pct), 3)
    log.info(f"N2 hold from {hold[0]:.2f} to {hold[-1]:.2f} kPa over {DECAY_HOLD_S:.0f} s: {decay_pct:.3f} % (mock: the {DECAY_STABILISE_S + DECAY_HOLD_S:.0f} s trace is returned at once)")
```

![Dry N2 pressure over 96 s: a ramp to 305.5 kPa, a 30 s stabilise window where the compression heat relaxes to 300 kPa, then a 60 s hold from 299.98 kPa decaying 0.084 %, well above the 0.5 % limit floor at 298.48 kPa drawn across the hold only.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/cold-plate-chart-decay-IDwAXoSmoHs1MUA6mzn00WCxdPlH5H.png)

The mock plate's decay: 5.5 kPa of compression heat relaxing with a 7 s time constant through the stabilise window, then 0.25 kPa lost over the hold, 0.084 % against 0.5 %. The mock returns the 96 s trace in one call; a real bench records it. A stub seated on its first detent but not its second shows a decay ten times steeper, and the YAML's `then: {fail: retry}` gives the operator one re-seat before the run stops.

### Hydrostatic

Filled with PG25 from the bench pump, ramped to 1x the maximum operating pressure and held for 5 min with the pump isolated, then to 3x for 2 min, then vented. The phase finds each dwell in the trace and reports its minimum; a leak, a weeping braze or a lid lifting off its seal shows as a falling trace during the dwell, not as a pump working harder. The operator's inspection after the 3x dwell lands as `no_deformation` through the switch bound to it.

```python filename="phases/hydrostatic.py"
import numpy as np

from utils.recipe import HYDRO_1X_DWELL_S, HYDRO_3X_DWELL_S, MOP_KPA


def dwell_min(t, p, target_kpa, dwell_s):
    """Lowest pressure inside the dwell at the stated target: the samples
    within 2 % of the target, the last dwell_s worth of them."""
    at_target = np.where(np.abs(p - target_kpa) < 0.02 * target_kpa)[0]
    end = t[at_target[-1]]
    window = (t > end - dwell_s) & (t <= end)
    return float(p[window].min())


def hydrostatic(measurements, bench, log):
    """Filled with PG25, 1x the maximum operating pressure for 5 min then 3x
    for 2 min (IEC 62368-1 as OCP quotes it), pump isolated during each
    dwell so a leak shows as a falling trace. The operator's inspection of
    the plate after the 3x dwell lands as the no_deformation measurement
    through the switch bound to it; the phase waits for that answer once the
    trace is recorded."""
    bench.fill()
    t_s, p_kpa = bench.hydrostatic(MOP_KPA, HYDRO_1X_DWELL_S, HYDRO_3X_DWELL_S)
    t = np.asarray(t_s)
    p = np.asarray(p_kpa)
    min_1x = dwell_min(t, p, MOP_KPA, HYDRO_1X_DWELL_S)
    min_3x = dwell_min(t, p, 3.0 * MOP_KPA, HYDRO_3X_DWELL_S)

    m = measurements.hydrostatic
    m.x_axis = t_s
    m.y_axis.pressure = p_kpa
    m.y_axis.pressure.aggregations.min_1x_dwell_kpa = round(min_1x, 1)
    m.y_axis.pressure.aggregations.min_3x_dwell_kpa = round(min_3x, 1)
    log.info(f"1x MOP dwell held {min_1x:.1f} kPa of {MOP_KPA:.0f}, 3x dwell {min_3x:.1f} kPa of {3 * MOP_KPA:.0f} (mock: the {t[-1]:.0f} s trace is returned at once)")
```

![Hydrostatic pressure over 460 s with the plate filled with PG25: a ramp to 300 kPa held for 5 min with a minimum of 298.2 kPa above the 285 kPa floor, a ramp to 900 kPa held for 2 min with a minimum of 897.0 kPa above the 855 kPa floor, then the vent.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/cold-plate-chart-hydrostatic-SsjRrn3nycRVrIvAH8zS2oCGynCwyF.png)

The mock plate holds 298.2 kPa of 300 and 897.0 of 900, the small settle coming from the hoses' compliance. The 5 and 2 min dwells are the spec's; the mock returns the 460 s trace in one call, so the run does not wait the seven minutes a real bench does.

### Flow and Pressure Drop

Six flows from half to a third above the design flow. At each point the loop dP is read with the plate in, then again through the bypass with the plate isolated, and the plate's own dP is the difference: the hoses, the quick-disconnects and the fixture are the same in both readings and cancel. The velocity in the stubs comes from the measured flow and the bore, and is the OCP erosion limit for copper.

```python filename="phases/flow_pressure_drop.py"
import math

import numpy as np

from utils.recipe import DESIGN_FLOW_LPM, FLOW_SWEEP_LPM, PORT_ID_MM


def flow_pressure_drop(measurements, bench, log):
    """Flow sweep at six points around the design flow. At each point the
    loop dP is read with the plate in and again through the bypass, and the
    plate's own dP is the difference (OCP), so the hoses, the QDs and the
    fixture do not end up in the plate's number. The velocity in the QD
    stubs comes from the measured flow and the bore."""
    area_m2 = math.pi * (PORT_ID_MM / 2000.0) ** 2
    flows, net, loop, bypass, velocity = [], [], [], [], []
    for q in FLOW_SWEEP_LPM:
        bench.set_flow_lpm(q)
        measured = bench.flow_lpm()
        with_plate = bench.dp_kpa("plate")
        through_bypass = bench.dp_kpa("bypass")
        flows.append(measured)
        loop.append(with_plate)
        bypass.append(through_bypass)
        net.append(with_plate - through_bypass)
        velocity.append(measured / 60000.0 / area_m2)
        log.info(f"{measured:.3f} LPM: loop {with_plate:.2f} kPa, bypass {through_bypass:.2f} kPa, plate {net[-1]:.2f} kPa, {velocity[-1]:.2f} m/s in the stubs")
    bench.set_flow_lpm(DESIGN_FLOW_LPM)
    at_design = int(np.argmin(np.abs(np.asarray(flows) - DESIGN_FLOW_LPM)))

    m = measurements.flow_curve
    m.x_axis = [round(q, 3) for q in flows]
    m.y_axis.plate_dp = [round(v, 3) for v in net]
    m.y_axis.plate_dp.aggregations.at_design_flow_kpa = round(net[at_design], 2)
    m.y_axis.loop_dp = loop
    m.y_axis.bypass_dp = bypass
    m.y_axis.stub_velocity = [round(v, 3) for v in velocity]
    m.y_axis.stub_velocity.aggregations.at_design_flow_m_s = round(velocity[at_design], 3)
    log.info(f"Plate dP {net[at_design]:.2f} kPa at {flows[at_design]:.2f} LPM, {velocity[at_design]:.2f} m/s in a {PORT_ID_MM:.0f} mm bore")
```

![Pressure drop against flow from 0.75 to 2.0 LPM: the loop with the plate in gray, the bypass dashed gray, and the plate as loop minus bypass in green reaching 9.96 kPa at the 1.5 LPM design flow, under the 15 kPa limit marker drawn at the design flow only.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/cold-plate-chart-flow-Vtt6z5xQ9taAQNQ7NuDBMKw8cu2pOT.png)

The mock plate: 9.96 kPa at 1.5 LPM and 0.88 m/s in a 6 mm bore, with the bypass taking 4 kPa of the 14 the loop shows. The curve is a laminar term in the fins plus a square term in the manifolds, so it bends upward; the limit applies at the design flow only and is drawn there only, because the same plate at 2 LPM reads 15 kPa and is not failing anything.

### Thermal Resistance

Four power steps to 1 kW at the design flow, the supply held at 40 C, 180 s of settling at each on a real bench. R = (Tcase - Tinlet) / P at every step; the mean and the full-power value each carry the limit. The outlet RTD closes the energy balance at 1 kW: the coolant must carry away what the heater put in, within 10 %, or a channel is blocked and the case RTD is reading a hot spot the plate is not cooling. The inlet mean is judged too, since an R measured against a drifting supply is not an R.

```python filename="phases/thermal_resistance.py"
import numpy as np

from utils.recipe import DESIGN_FLOW_LPM, PG25_CP_J_PER_KG_K, PG25_DENSITY_KG_L, POWER_STEPS_W, STEADY_STATE_S


def thermal_resistance(measurements, bench, log):
    """Heater block at four power steps up to 1 kW at the design flow, the
    supply held at 40 C. R = (Tcase - Tinlet) / P at each step (OCP). The
    outlet RTD closes the energy balance at full power: the coolant must
    carry away what the heater put in, or a channel is blocked and the
    case RTD is reading a hot spot the plate is not cooling."""
    bench.set_flow_lpm(DESIGN_FLOW_LPM)
    m_cp = DESIGN_FLOW_LPM / 60.0 * PG25_DENSITY_KG_L * PG25_CP_J_PER_KG_K
    tin_c, tcase_c, r = [], [], []
    balance = None
    for p in POWER_STEPS_W:
        bench.set_heater_w(p)
        log.info(f"{p:.0f} W: settle {STEADY_STATE_S:.0f} s (mock: instant)")
        tin, tout, tcase = bench.rtds_c()
        tin_c.append(tin)
        tcase_c.append(tcase)
        r.append((tcase - tin) / p)
        balance = 100.0 * m_cp * (tout - tin) / p
        log.info(f"{p:.0f} W: inlet {tin:.2f} C, outlet {tout:.2f} C, case {tcase:.2f} C, R {r[-1]:.4f} K/W, coolant carries {balance:.1f} %")
    bench.set_heater_w(0.0)

    m = measurements.thermal_curve
    m.x_axis = POWER_STEPS_W
    m.y_axis.tcase_c = tcase_c
    m.y_axis.tcase_c.aggregations.max_c = round(max(tcase_c), 2)
    m.y_axis.tin_c = tin_c
    m.y_axis.tin_c.aggregations.mean_c = round(float(np.mean(tin_c)), 2)
    m.y_axis.r_k_per_w = [round(v, 5) for v in r]
    m.y_axis.r_k_per_w.aggregations.mean_k_per_w = round(float(np.mean(r)), 4)
    m.y_axis.r_k_per_w.aggregations.at_full_power_k_per_w = round(r[-1], 4)
    measurements.coolant_heat_balance_pct = round(balance, 1)
```

The 0.020 K/W limit is not in the OCP paper; it is what a 60 C case ceiling leaves on a 40 C supply for a 1 kW device, and a plate that passes it at this interface has the headroom the product's own TIM tolerance will consume.

### Particulate Flush

Ten minutes at twice the design flow through the inline 50 um filter, the effluent through the particle counter. The counts per 100 mL in each size bin are recorded as one curve, and the bin above 50 um is the one judged, at zero. The filter's own dP is read at the start and at the end of the flush: a filter still loading when the flush ends is a plate still shedding, whatever the counter said in the last sample.

```python filename="phases/particulate_flush.py"
from utils.recipe import FILTER_DP_RISE_MAX_KPA, FLUSH_FLOW_LPM, FLUSH_MIN, PARTICLE_BINS_UM


def particulate_flush(measurements, bench, log):
    """Flush at twice the design flow through the inline 50 um filter with
    the effluent through the particle counter. OCP's rule is nothing above
    50 um; the counts per 100 mL in every size bin are recorded, and the
    filter's own dP must have stopped rising, or the flush ended while the
    plate was still shedding."""
    counts, dp_start, dp_end = bench.flush(FLUSH_FLOW_LPM, FLUSH_MIN)
    rise = dp_end - dp_start
    m = measurements.particle_counts
    m.x_axis = PARTICLE_BINS_UM
    m.y_axis.count = counts
    m.y_axis.count.aggregations.over_50um = counts[-1]
    measurements.filter_dp_stable = rise < FILTER_DP_RISE_MAX_KPA
    log.info(f"{FLUSH_MIN:.0f} min at {FLUSH_FLOW_LPM:.1f} LPM (mock: instant): counts {counts} per 100 mL from {[int(b) for b in PARTICLE_BINS_UM]} um, filter dP {dp_start:.2f} to {dp_end:.2f} kPa")
```

### Drain and Cap

The teardown drains the plate, blows it down with N2 for 60 s, fits the shipping caps and puts the plate back on the scale. The residual is the wet mass minus the dry tare that the identify phase recorded, read through the injected result of that phase. The raw pressure traces of the decay and the hydrostatic go with the run as one JSON attachment.

```python filename="phases/drain_and_cap.py"
import json

from utils.recipe import BLOW_DOWN_S


def drain_and_cap(measurements, bench, attach, identify, log):
    """Teardown: drain, N2 blow-down, caps on both quick-disconnects, the
    plate back on the scale. The residual is the wet mass minus the dry
    tare the identify phase recorded, read through the injected result of
    that phase. The raw pressure traces of the decay and the hydrostatic
    go with the run as one JSON attachment."""
    bench.drain_and_blow_down(BLOW_DOWN_S)
    capped = bench.cap_qds()
    residual = bench.wet_mass_g() - identify.dry_mass_g
    measurements.residual_coolant_g = round(residual, 2)
    measurements.qds_capped = capped
    attach.data(json.dumps(bench.pressure_log()).encode(), "pressure-log.json")
    log.info(f"Blow-down {BLOW_DOWN_S:.0f} s (mock: instant), {residual:.2f} g of coolant left, caps {'fitted' if capped else 'missing'}, pressure log attached")
```

### Mock Plug

`ColdPlateBench` stands in for the whole bench and the plate because every reading depends on what the pump and the heater are doing at that instant, and plugs run in separate processes. Its decay is 5.5 kPa of compression heat with a 7 s time constant over a leak of 0.0027 kPa/s, its hydrostatic dwells settle 0.9 and 2.5 kPa on the hose compliance, its pressure drop is 4.2 kPa/LPM in the fins plus 1.6 kPa/LPM squared in the manifolds with a bypass of 1.78 kPa/LPM squared, its thermal resistance 0.0157 K/W rising 0.0003 per kW, its coolant carrying 97 % of the heater power, its effluent 286, 71, 6 and 0 counts per 100 mL from 5, 10, 25 and 50 um, and its residual 1.8 g. Time is scaled: the 96 s decay, the 460 s hydrostatic, the 180 s settles and the 10 min flush are returned as traces or values in one call, and each phase logs that it did so. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real bench, split the class into the N2 side, the pump and its isolation valve, the flow loop with its meter and dP cells, the chiller, the heater controller and the RTD reader, the particle counter, the conductivity probe and the scale, over Modbus, SCPI or their vendor drivers. Record the pressure traces at the transducer's rate and let the phases compute the decay and the dwell minima; wait the real 180 s per power step and the real 10 min flush, the `timeout` values are sized for them; and keep the same heater pedestal and thermal interface for every plate. The phases, measurements and limits stay the same.


### Rack L11 Integration Test
URL: https://www.tofupilot.com/templates/rack-l11-integration-test
Integrate a liquid-cooled AI rack at L11: build record, cable map vs LLDP, gaseous leak test, PG25 fill and flow, ports at rate, NCCL all-reduce, 24 h soak.

## Introduction

### Rack Integration Overview

A GPU rack does not leave the integrator as eight servers. The OCP liquid cooling integration and logistics white paper defines the levels: L6 is a board in a chassis, L10 is a server that boots and passes its own witness test, **L11 is that server racked with its network and its coolant manifold attached**, and L12 adds the customer's software environment. L11 is where the rack becomes one unit: eight GPU nodes on the busbar of an ORV3 power shelf, two top-of-rack switches with 32 cables to the nodes, a manifold with 16 quick-disconnects and a CDU behind it. Everything that can go wrong between eight good servers and one good rack goes wrong here: a cable pair swapped between two nodes, a quick-disconnect that did not seat, a node whose BIOS was flashed before the release, a transceiver with a dirty endface, a branch of the manifold that starves one node.

![Paired supply and return coolant manifolds with color-coded quick-disconnect couplings and main hoses.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/rack-l11-integration-test-matte-closeup-F8lTi3dHONpdGdVo9noaz4Ri3Xq9Rd.webp)

The parts this test exercises: the manifold and its quick-disconnects, the CDU and the coolant loop through eight sets of cold plates, the 32 fabric links between the nodes and the ToRs, the NVLink domain inside every node and the fabric between them, and the power shelf on the busbar.

There is no published specification for L11. The white paper puts a **gaseous leak test of the servers, the manifold and the CDU before the fill**, with nitrogen or forming gas and a hydrogen sniffer, and lets the rack ship wet or drained. The 0.5 % pressure-decay figure comes from the [OCP cold plate development and qualification](https://www.opencompute.org/documents/ocp-cold-plate-development-and-qualification-with-integrated-comments-pdf) document, which cites EN 1779. The rack's coolant boundary comes from the [Meta Catalina compute tray specification](https://www.opencompute.org/documents/catalina-compute-tray-ocp-specification-v1d33-pdf-1): **PG25 at 40 °C supply, at most 100 LPM and 15 psi at the rack, 1.25 to 1.5 LPM per kW**. The rest is integrator practice, consistent across the cluster acceptance guides: the cable map checked against LLDP, every port at rate with nothing uncorrected, the NVLink domain checked with `nvidia-smi` and `nvbandwidth`, NCCL `all_reduce_perf` at 90 to 95 % of theoretical, a 24 to 72 h soak with no throttling and no ECC growth, and a rack-loop pressure hold with no decay the transducer can see. Every limit in this template is one of those, or says whose derivation it is.

### Test Purpose

The procedure records one integration fingerprint per rack:

- **The build record** read from the hardware: rack asset tag, elevation revision, node, switch and PSU serials as one object
- **The cable map** against the LLDP neighbour tables of both ToRs: mismatched, missing and extra links, and the negotiated speed on every mapped port
- **A forming-gas pressure hold** on the rack loop, judged on temperature-compensated decay, and the **hydrogen reading at each of the 16 quick-disconnects**
- **The fill volume**, the **flow and inlet temperature at every node's branch**, and the rack's total flow, pressure drop and supply temperature
- **Nodes reachable** after power-on, the **inventory of every node collapsed into one object** against the golden configuration, and the shelf's idle power
- **FEC and PCS counters per port** after 5 min at 400 GbE line rate, and frames lost
- **NVLinks down** and the **all-reduce bus bandwidth** across 64 ranks against the node NIC ceiling
- **Per-node soak results** over 24 h: hottest GPU, throttle events, ECC counters, and the leak sensors and the shelf's fault log
- At teardown, **the loop state next to the operator's declaration**, the shelf blackbox, the BMC event logs, and **the final cable map attached** as a report

![all_reduce_perf bus bandwidth across 64 ranks against message size from 1 MiB to 8 GiB, rising from 21 GB/s to a 187.4 GB/s plateau, 93.7 % of the 200 GB/s node NIC ceiling line, above the 180 GB/s limit line, with algorithm bandwidth reaching 95.2 GB/s in blue.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/rack-l11-chart-nccl-3tQzZYE6lAKHOp5dRCeslYF34qqEBX.png)

The mock rack's all-reduce sweep: bus bandwidth climbs out of the latency floor and plateaus at 187.4 GB/s, 93.7 % of the 200 GB/s that the node's four 400 GbE NICs can carry. The ceiling is the fabric's, not the GPUs': an inter-node ring cannot move more than the node's NICs, so the limit is stated as a fraction of that number.

The framework mechanics on show are a `depends_on` DAG where the cable check and the leak test run concurrently on different instruments while the fill waits on both; `then: {fail: retry}` with `retry: {limit: 1}` on the cable check, so one re-seat and re-run stays in the report; operator UI `select` and `switch` components bound to measurements in the teardown, validated with `in` and `==`; `attach.file` of a report the phase wrote; JSON `==` on whole objects, including an inventory collapsed from eight nodes; and long phases time-scaled in the mock with `timeout` sized for the real ones.

## Equipment & Setup

To run this integration on a rack line, the following are required:

- **A forming-gas panel** (95 % N2, 5 % H2) with a regulator, a shut-off valve and a gauge-pressure transducer on the manifold's supply stub, and a gas thermocouple on the return stub
- **A hydrogen sniffer** with a hand probe for the 16 quick-disconnects
- **The rack's CDU**, or a bench CDU on the rack's supply and return stubs, with a PG25 reservoir, a flow meter and a differential-pressure transducer, and per-branch flow meters and inlet thermistors on the manifold
- **Management access to both ToRs** for LLDP and the per-port PCS and FEC counters, and a traffic agent on every node for the run at rate
- **Redfish access to the eight BMCs** and the rack manager's API for the shelf and the leak sensors
- **An SSH agent on every node** with `nvidia-smi`, `nvbandwidth`, NCCL tests and DCGM
- **The Device Under Test (DUT)**: an 8-node liquid-cooled AI rack after L10, cabled and plumbed, with its build record and its topology map
- **A TofuPilot Framework procedure** to sequence the gates and validate the limits
- **The TofuPilot Dashboard** to keep the per-rack evidence, the cable map and the soak history

### Hardware Components

#### Leak Bench and CDU

The gaseous test needs a gas the sniffer can see and a pressure the loop can take: forming gas at 5 % hydrogen is non-flammable and an Inficon Sensistor Sentrac reads it to a fraction of a ppm above background. The transducer matters more than the gas: a 30 min hold at 200 kPa on a loop of 42 L moves a few hundredths of a kilopascal for a leak that would matter, so the phase averages the first and last minute and compensates the gas temperature before it computes the decay. The CDU is the rack's own or a bench unit on the same stubs; Vertiv and Motivair in-rack CDUs expose the supply set-point, flow and dP over Modbus TCP, and the manifold's per-branch turbine meters sit on the same bus.

![Liquid-cooled server rack with two top-of-rack switches, eight server trays, paired coolant manifolds, and a bottom cooling unit.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/rack-l11-integration-test-matte-cover-fxwRXuQ07VOvytWBkK25vNeaCSBxUI.webp)

The illustrated hardware: eight server trays, two top-of-rack switches and paired coolant connections in one rack. The management computer and external facility connections are omitted.

#### Fabric and Node Tooling

The ToRs are 400 GbE switches whose management API returns LLDP neighbours and per-port RS(544,514) FEC counters: corrected codewords, uncorrected codewords and PCS symbol errors. A link that corrects a lot is a dirty endface or a marginal transceiver and still passes traffic; a link that has anything uncorrected is dropping frames. On the nodes, `nvidia-smi nvlink -s` and [nvbandwidth](https://github.com/NVIDIA/nvbandwidth) cover the NVLink domain, [nccl-tests](https://github.com/NVIDIA/nccl-tests) `all_reduce_perf` covers the fabric between nodes, and [DCGM](https://developer.nvidia.com/dcgm) diagnostics at the power limit cover the soak with the temperature, throttle and ECC counters the phase records.

### Where the Limits Come From

| Test | Limit | Basis |
|---|---|---|
| Build record | exact match | no tolerance applies |
| Cable map | 0 mismatched, 0 missing, 0 extra, every mapped port at 400 Gb/s | integrator practice (LLDP against the topology map) |
| Pressure hold | ≤ 0.5 % decay over 30 min at 200 kPa, gas drift within ±1 °C | OCP cold plate qualification, EN 1779; the hold pressure is this loop's MOP; the drift window is this line's |
| Hydrogen at QDs | ≤ 5 ppm above background | this line's: ten times the sniffer's noise floor |
| Fill volume | 42 L ± 10 % | the rack's design volume; the tolerance is this line's |
| Node flow | ≥ 8.75 LPM per node | Catalina: 1.25 LPM/kW × 7 kW of heat to liquid per node |
| Rack flow, dP | ≤ 100 LPM, ≤ 15 psi | Catalina rack boundary |
| Supply, inlets | 40 °C ± 2 | Catalina: 40 °C PG25 supply; ±2 °C is this line's |
| Inventory | exact match to the golden configuration, 8 nodes, 64 GPUs | no tolerance applies |
| Idle power | 8 to 14 kW | this line's window for eight nodes and two switches |
| Ports at rate | ≥ 99 % of line rate, ≤ 1,000,000 corrected codewords per port in 5 min, 0 uncorrected, 0 symbol errors, 0 frames lost | integrator practice for the zeros; the corrected ceiling is this line's, about 8e-9 pre-FEC BER, four decades under the 2.4e-4 budget of the code |
| NCCL | ≥ 180 GB/s = 90 % of the 200 GB/s node NIC ceiling at the largest message, 0 NVLinks down | integrator practice quotes 90 to 95 % of theoretical; the ceiling is the node's 4 × 400 GbE |
| Soak | hottest GPU ≤ 85 °C, 0 throttle events, 0 uncorrectable ECC, 0 leak events, 0 PSU faults over 24 h | integrator practice for the zeros; 85 °C is this line's, 5 °C under the GPU's slowdown threshold |
| Ship prep | loop state and declaration each in [wet, drained], QDs capped, blackbox and SEL empty | this line's |

Two rows deserve a note. The pressure-decay limit is a cold-plate qualification figure applied to a rack loop: the loop is a hundred times the plate's volume, so 0.5 % of it is a large leak, and the practice on this line is zero decay within the transducer's resolution, with the 0.5 % as the number the report validates. The NCCL ceiling is the node NIC bandwidth and not the GPU's NVLink bandwidth: in a ring across nodes every rank sends the same amount, and one rank per node per ring sends it over a NIC, so the bus bandwidth cannot exceed what the node's NICs carry.

## Test Procedure

### Overview

The procedure maps the integration onto the framework's three stages. The build record lives in `setup:` so nothing proceeds on a rack that is not the one on the drawing. The cable map and the leak test are the first two main phases with no dependency between them: one runs on the ToRs, the other on the gas panel, and the fill waits for both. From the fill on, the chain is sequential because each phase needs the last one's state: coolant before power, power before traffic, traffic before NCCL, NCCL before the soak. Ship prep lives in `teardown:` so the CDU goes into ship mode and the cable map is attached whatever happened.

1. Setup: build record from the hardware, one object.
2. Main: LLDP tables against the topology map, one retry after a re-seat.
3. Main, concurrent with 2: forming-gas hold and hydrogen sniff at 16 QDs.
4. Main: PG25 fill, flow and inlet temperature per node, rack boundary.
5. Main: power-on, inventory collapsed against the golden set, idle power.
6. Main: 32 ports at rate for 5 min, FEC and PCS counters.
7. Main: NVLink status, all-reduce sweep against the NIC ceiling.
8. Main: 24 h soak, per node.
9. Teardown: loop state, declaration, blackbox, SEL, cable map attached.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-rack evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
ui.json
phases/
  rack_identity.py
  cabling_verification.py
  gaseous_leak.py
  fill_and_flow.py
  power_on_inventory.py
  network_at_rate.py
  nccl_cluster.py
  soak.py
  ship_prep.py
plugs/
  rack_manager.py
  fabric.py
  leak_bench.py
  cdu.py
utils/
  recipe.py
  topology.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-rack-l11-integration). The four plugs are mocks of the rack manager with the BMCs and the shelf, the two ToRs with the node traffic agents, the leak bench, and the CDU with the manifold instrumentation, synthesizing a rack built to its record: a tight loop, eight branches within 10 % of each other, FEC correcting a healthy number of codewords with nothing uncorrected, 93.7 % of the all-reduce ceiling and a soak with the hottest GPU at 78.7 °C, so the procedure runs end-to-end without a rack connected.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless, with the operator's ship-prep answers pre-baked in `ui.json`:

```bash
tofupilot run . --no-tui --no-kiosk --json --ui-values ui.json --ui-timeout 60
```

### The Procedure File

```yaml filename="procedure.yaml"
name: Rack L11 Integration Test
version: 0.1.0
description: L11 integration of a liquid-cooled AI rack, eight GPU nodes, two ToR switches and an ORV3 power shelf on a 48 V busbar, racked with the network and the coolant manifold attached. Build record against the hardware, cable map against LLDP with one re-seat retry, gaseous leak test of the rack loop before the fill, PG25 fill and per-node flow against the CDU boundary, power-on and inventory against the golden configuration, every port at rate with zero uncorrected errors, NCCL all-reduce against the node NIC ceiling, a 24 h soak with nothing growing, and a ship-prep teardown that records the loop state and attaches the final cable map.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the rack asset label on the front door"
    placeholder: "RACK-AI8-00000"
    pattern: "^RACK-AI8-\\d{5}$"
    default_value: "RACK-AI8-00217"
  part_number:
    default_value: "AIR-8N-2T-LC"
  batch_number:
    default_value: "BUILD-2026-38"

plugs:
  - name: Rack Manager
    description: "Rack manager, the eight BMCs over Redfish, the power shelf's Modbus port and a node agent for the GPU tooling (mock)"
    python: plugs.rack_manager:RackManager
    key: rack
    config:
      address: "10.0.0.10"
  - name: Fabric
    description: "Both ToR switches' management API and the node traffic agents (mock)"
    python: plugs.fabric:Fabric
    key: fabric
    config:
      tor1: "10.0.0.11"
      tor2: "10.0.0.12"
  - name: Leak Bench
    description: "Forming-gas panel, pressure and gas-temperature transducers, hydrogen sniffer (mock); station scoped because the sniffer's warm-up outlasts a rack's test"
    python: plugs.leak_bench:LeakBench
    key: leak
    scope: station
    config:
      gas_panel: "/dev/ttyUSB0"
      sniffer: "/dev/ttyUSB1"
  - name: CDU
    description: "In-rack CDU, manifold branch flow meters and inlet thermistors, leak-rope sensors (mock)"
    python: plugs.cdu:Cdu
    key: cdu
    config:
      address: "10.0.0.20"

setup:
  - name: Rack Identity
    key: rack_identity
    python: phases.rack_identity
    measurements:
      - name: Build Record
        key: build_record
        description: "Rack asset tag, elevation drawing revision, node, switch and PSU serials read from the hardware, as one object against the build record."
        validators:
          - operator: "=="
            expected_value:
              rack: RACK-AI8-00217
              elevation_rev: C
              nodes: [N8G-26-04117, N8G-26-04118, N8G-26-04119, N8G-26-04120, N8G-26-04121, N8G-26-04122, N8G-26-04123, N8G-26-04124]
              switches: [TOR-26-00871, TOR-26-00872]
              psus: [ORV3-12K-004217, ORV3-12K-004218, ORV3-12K-004221, ORV3-12K-004224, ORV3-12K-004230, ORV3-12K-004231]
      - name: Rack Serial Matches Label
        key: rack_serial_matches_label
        description: The asset tag the rack manager reports is the one scanned on the door.
        validators:
          - {operator: "==", expected_value: true}

main:
  - name: Cabling Verification
    key: cabling_verification
    python: phases.cabling_verification
    retry:
      limit: 1
    then:
      fail: retry
    measurements:
      - name: Link Mismatches
        key: link_mismatches
        description: "Mapped ports whose LLDP neighbour is another node or another NIC: a swapped pair of cables."
        validators:
          - {operator: "==", expected_value: 0}
      - name: Missing Links
        key: missing_links
        description: "Mapped ports with no LLDP neighbour: an unseated cable or a dead transceiver."
        validators:
          - {operator: "==", expected_value: 0}
      - name: Extra Links
        key: extra_links
        description: "Ports off the map that have a neighbour."
        validators:
          - {operator: "==", expected_value: 0}
      - name: Link Speed
        key: link_speed
        title: Expected and negotiated speed on the 32 mapped ToR ports
        x_axis:
          legend: Mapped port
        y_axis:
          - legend: Expected
            key: expected_gbps
            unit: Gb/s
          - legend: Seen
            key: seen_gbps
            unit: Gb/s
            aggregations:
              - type: ports_below_expected
                validators:
                  - {operator: "==", expected_value: 0}

  - name: Gaseous Leak
    key: gaseous_leak
    python: phases.gaseous_leak
    timeout: 60m
    measurements:
      - name: Pressure Hold
        key: pressure_hold
        title: Rack loop at 200 kPa of forming gas, 30 min hold after a 10 min settle
        x_axis:
          legend: Time
          unit: min
        y_axis:
          - legend: Gauge pressure
            key: pressure_kpa
            unit: kPa
            aggregations:
              - type: decay_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 0.5}
              - type: hold_start_kpa
                unit: kPa
                validators:
                  - {operator: ">=", expected_value: 195.0}
          - legend: Gas temperature
            key: gas_temp_c
            unit: °C
            aggregations:
              - type: drift_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: -1.0}
                  - {operator: "<=", expected_value: 1.0}
      - name: QD Sniff
        key: qd_sniff
        title: Hydrogen at each of the 16 quick-disconnects
        x_axis:
          legend: Quick-disconnect
        y_axis:
          - legend: H2 above background
            key: h2_ppm
            unit: ppm
            aggregations:
              - type: max_ppm
                unit: ppm
                validators:
                  - {operator: "<=", expected_value: 5.0}

  - name: Fill and Flow
    key: fill_and_flow
    python: phases.fill_and_flow
    depends_on: [gaseous_leak, cabling_verification]
    measurements:
      - name: Fill Volume
        key: fill_volume_l
        unit: L
        description: "Litres taken from the reservoir to fill the loop, against the 42 L design volume +-10 %. A short fill is a node whose quick-disconnects did not open."
        validators:
          - {operator: ">=", expected_value: 37.8}
          - {operator: "<=", expected_value: 46.2}
      - name: Node Flow
        key: node_flow
        title: Coolant flow and inlet temperature at each node's manifold branch
        x_axis:
          legend: Node
        y_axis:
          - legend: Flow
            key: flow_lpm
            unit: LPM
            aggregations:
              - type: min_lpm
                unit: LPM
                validators:
                  - {operator: ">=", expected_value: 8.75}
              - type: min_lpm_per_kw
                unit: LPM/kW
                validators:
                  - {operator: ">=", expected_value: 1.25}
          - legend: Inlet temperature
            key: inlet_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 42.0}
              - type: min_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: 38.0}
      - name: Rack Flow
        key: rack_flow_lpm
        unit: LPM
        description: "Total at the CDU, inside the 100 LPM rack boundary (Catalina)."
        validators:
          - {operator: "<=", expected_value: 100.0}
      - name: Rack Pressure Drop
        key: rack_dp_psi
        unit: psi
        description: "Supply to return across the rack at the CDU, inside the 15 psi boundary (Catalina)."
        validators:
          - {operator: "<=", expected_value: 15.0}
      - name: Supply Temperature
        key: supply_c
        unit: °C
        description: "40 C PG25 supply (Catalina), +-2 C is this line's."
        validators:
          - {operator: ">=", expected_value: 38.0}
          - {operator: "<=", expected_value: 42.0}

  - name: Power-On and Inventory
    key: power_on_inventory
    python: phases.power_on_inventory
    depends_on: [fill_and_flow]
    measurements:
      - name: Nodes Reachable
        key: nodes_reachable
        validators:
          - {operator: "==", expected_value: 8}
      - name: Rack Inventory
        key: rack_inventory
        description: "Every node's BIOS, BMC, VBIOS and NIC firmware and its GPU and DIMM counts, collapsed into one object: a field that differs between nodes becomes a list and fails the match on that key."
        validators:
          - operator: "=="
            expected_value:
              bios: 2.4.1
              bmc: 1.12.0
              vbios: 96.00.89.00.01
              nic_fw: 28.41.1000
              gpus: 8
              dimm_gb: 2048
      - name: GPUs Seen
        key: gpus_seen
        validators:
          - {operator: "==", expected_value: 64}
      - name: Idle Power
        key: idle_power_kw
        unit: kW
        description: "Shelf output with every node at the OS prompt, this line's window for eight nodes and two switches."
        validators:
          - {operator: ">=", expected_value: 8.0}
          - {operator: "<=", expected_value: 14.0}

  - name: Network at Rate
    key: network_at_rate
    python: phases.network_at_rate
    depends_on: [power_on_inventory]
    timeout: 15m
    measurements:
      - name: Port Counters
        key: port_counters
        title: Every mapped port at 400 GbE line rate for 5 min, counters cleared before and read after
        x_axis:
          legend: Mapped port
        y_axis:
          - legend: Line rate achieved
            key: line_rate_pct
            unit: "%"
            aggregations:
              - type: min_pct
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 99.0}
          - legend: FEC corrected codewords
            key: fec_corrected
            aggregations:
              - type: max_per_port
                validators:
                  - {operator: "<=", expected_value: 1000000}
          - legend: FEC uncorrected codewords
            key: fec_uncorrected
            aggregations:
              - type: total
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: PCS symbol errors
            key: symbol_errors
            aggregations:
              - type: total
                validators:
                  - {operator: "==", expected_value: 0}
      - name: Frames Lost
        key: frames_lost
        description: "Sent minus received across every pair over the 5 min."
        validators:
          - {operator: "==", expected_value: 0}

  - name: NCCL Cluster
    key: nccl_cluster
    python: phases.nccl_cluster
    depends_on: [network_at_rate]
    timeout: 30m
    measurements:
      - name: NVLinks Down
        key: nvlink_links_down
        description: "nvidia-smi nvlink -s on all 64 GPUs; a link down halves a node's NVLink domain."
        validators:
          - {operator: "==", expected_value: 0}
      - name: All-Reduce
        key: all_reduce
        title: all_reduce_perf across 64 ranks over the fabric, 1 MiB to 8 GiB
        x_axis:
          legend: Message size
          unit: MiB
        y_axis:
          - legend: Bus bandwidth
            key: busbw_gbs
            unit: GB/s
            aggregations:
              - type: peak_gbs
                unit: GB/s
                validators:
                  - {operator: ">=", expected_value: 180.0}
              - type: pct_of_ceiling
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 90.0}
          - legend: Algorithm bandwidth
            key: algbw_gbs
            unit: GB/s

  - name: Soak
    key: soak
    python: phases.soak
    depends_on: [nccl_cluster]
    measurements:
      - name: Node Soak
        key: node_soak
        title: 24 h at the power limit, per node
        x_axis:
          legend: Node
        y_axis:
          - legend: Hottest GPU
            key: max_gpu_temp_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 85.0}
          - legend: Throttle events
            key: throttle_events
            aggregations:
              - type: total
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: ECC uncorrectable
            key: ecc_uncorrectable
            aggregations:
              - type: total
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: ECC corrected
            key: ecc_corrected
      - name: Leak Sensor Events
        key: leak_sensor_events
        description: "Leak-rope and drip-tray events since the fill."
        validators:
          - {operator: "==", expected_value: 0}
      - name: PSU Faults
        key: psu_faults
        description: "Fault entries across the shelf's six modules over the soak."
        validators:
          - {operator: "==", expected_value: 0}

teardown:
  - name: Ship Prep
    key: ship_prep
    python: phases.ship_prep
    ui:
      components:
        - key: ship_state
          type: select
          label: "Ship state"
          description: "Per the shipping order: wet ships the loop full of PG25 with the stubs capped; drained ships it blown down"
          required: true
          bind: measurements.ship_state
          options:
            - {label: "Wet, loop filled", value: wet}
            - {label: "Drained, loop blown down", value: drained}
        - key: qds_capped
          type: switch
          label: "All 16 quick-disconnects and both stubs capped"
          description: "Dust caps on every node QD and the supply and return stubs"
          required: true
          bind: measurements.qds_capped
    measurements:
      - name: Ship State
        key: ship_state
        description: "The operator's declaration, from the shipping order."
        validators:
          - {operator: in, expected_value: [wet, drained]}
      - name: Loop State
        key: loop_state
        description: "What the CDU's level sensor says is in the loop: wet above 90 % of the design volume, drained under 5 %, partial in between. Read it next to the declaration."
        validators:
          - {operator: in, expected_value: [wet, drained]}
      - name: Coolant In Loop
        key: coolant_in_loop_l
        unit: L
      - name: QDs Capped
        key: qds_capped
        validators:
          - {operator: "==", expected_value: true}
      - name: Shelf Blackbox
        key: shelf_blackbox
        description: "The six modules' blackbox counters summed. A rack that ships with an AC loss or a trip in its log has a story the customer will ask about."
        validators:
          - operator: "=="
            expected_value:
              ac_loss: 0
              ovp: 0
              ocp: 0
              otw: 0
              otp: 0
              fan: 0
      - name: SEL Critical Entries
        key: sel_critical
        description: "System event log entries of severity Critical across the eight BMCs since power-on."
        validators:
          - {operator: "==", expected_value: 0}
```

Framework features to notice:

1. **Two gates on two instruments, one fill.** `cabling_verification` and `gaseous_leak` declare no dependencies, so they start together, one on the ToRs and one on the gas panel; `fill_and_flow` declares `depends_on: [gaseous_leak, cabling_verification]` and waits for both. Everything after the fill chains on the phase before it.
2. **One re-seat, kept in the report.** `cabling_verification` has `retry: {limit: 1}` and `then: {fail: retry}`: a swapped or unseated cable fails the first attempt, the operator re-seats what the log lists, the phase runs again, and the dashboard keeps both attempts with their measurements.
3. **Operator answers as measurements.** The teardown's `select` binds to `ship_state` and its `switch` to `qds_capped`; the answers are validated with `in` and `==` like any other measurement, and `ui.json` pre-bakes them for headless runs. The CDU's own reading of the loop is recorded next to the declaration.
4. **A report attached by the phase.** `ship_prep` writes the final cable map, the LLDP table and their diff to a file and calls `attach.file`, so the rack's run carries the map it shipped with.
5. **Eight nodes, one object.** `rack_inventory` collapses the inventory of every node into one dict: a field that agrees across nodes keeps its value, a field that differs becomes a list and fails the `==` on that key. `build_record` and `shelf_blackbox` use the same `==`.
6. **Long tests, time-scaled.** The 30 min hold, the 5 min run at rate and the 24 h soak are instant in the mock; `timeout` is sized for the real leak test, network run and NCCL sweep, and the soak carries none because 24 h is the framework's cap.
7. **A station-scoped plug.** The leak bench is `scope: station` so the sniffer, whose warm-up outlasts a rack's test, stays open between racks.

### Rack Identity

The setup phase reads the rack's asset tag and elevation revision from the rack manager, the node serials from the BMC FRUs, the switch serials from their management API and the PSU serials from the shelf, and records all of it as one object against the build record. A node from another rack, a shelf missing a module or a rack built to the previous elevation revision fails here, before any gas, coolant or power goes in. The elevation revision is stamped on the **unit metadata**.

```python filename="phases/rack_identity.py"
from utils.recipe import NODES, PSU_MODULES, TORS


def rack_identity(measurements, rack, unit, log):
    """Setup: what is in the rack, read from the hardware, against the build
    record. Rack asset tag, elevation revision, eight node serials, two switch
    serials and six PSU serials as one object: a node from another rack, or
    a shelf missing a module, fails here before any coolant or power goes in."""
    labels = rack.read_build_labels()
    measurements.build_record = labels
    measurements.rack_serial_matches_label = labels["rack"] == unit.serial_number
    unit.metadata["elevation_rev"] = labels["elevation_rev"]
    log.info(f"{labels['rack']} elevation rev {labels['elevation_rev']}: {len(labels['nodes'])}/{NODES} nodes, {len(labels['switches'])}/{TORS} switches, {len(labels['psus'])}/{PSU_MODULES} PSUs read; label {unit.serial_number}")
```

### Cabling Verification

Both ToRs' LLDP neighbour tables against the topology map in `utils/topology.py`, which lists the node and NIC every ToR port should see and at what speed. The phase counts mismatched, missing and extra links, records expected and seen speed per mapped port as a multi-dimensional measurement, and logs every discrepancy in the words the operator needs to re-seat it. On a fail the framework runs it once more.

```python filename="phases/cabling_verification.py"
from utils.recipe import PORTS
from utils.topology import diff, speeds


def cabling_verification(measurements, fabric, log):
    """LLDP neighbour tables from both ToRs against the topology map. A
    swapped pair of cables is a mismatch, an unseated one is a missing link,
    and a port that came up below 400 GbE is a dirty or wrong transceiver. On
    a fail the phase retries once after the operator re-seats what it lists."""
    seen = fabric.lldp_table()
    d = diff(seen)
    expected, actual = speeds(seen)
    for m in d["mismatched"]:
        log.warning(f"{m['expected']['tor']} {m['expected']['port']}: expected {m['expected']['node']} {m['expected']['nic']}, saw {m['seen']['node']} {m['seen']['nic']}")
    for e in d["missing"]:
        log.warning(f"{e['tor']} {e['port']}: no neighbour, expected {e['node']} {e['nic']}")
    below = sum(1 for e, a in zip(expected, actual) if a < e)

    measurements.link_mismatches = len(d["mismatched"])
    measurements.missing_links = len(d["missing"])
    measurements.extra_links = len(d["extra"])
    m = measurements.link_speed
    m.x_axis = list(range(1, PORTS + 1))
    m.y_axis.expected_gbps = expected
    m.y_axis.seen_gbps = actual
    m.y_axis.seen_gbps.aggregations.ports_below_expected = below
    log.info(f"{len(seen)} neighbours on {PORTS} mapped ports: {len(d['mismatched'])} mismatched, {len(d['missing'])} missing, {len(d['extra'])} extra, {below} below 400 GbE")
```

The map and the diff are plain Python so the report can carry them:

```python filename="utils/topology.py"
"""The rack's topology map: which ToR port carries which node NIC at what
speed, and the diff of an LLDP neighbour table against it."""

from utils.recipe import BUILD_RECORD, NODES, PORT_GBPS, PORTS_PER_NODE


def expected_links():
    links = []
    for n in range(NODES):
        for nic in range(PORTS_PER_NODE):
            tor = nic // 2 + 1  # eth0 and eth1 on ToR 1, eth2 and eth3 on ToR 2
            port = 2 * n + nic % 2 + 1  # ToR ports 1 to 16
            links.append({"tor": f"tor{tor}", "port": f"Ethernet{port}", "node": BUILD_RECORD["nodes"][n], "nic": f"eth{nic}", "gbps": PORT_GBPS})
    return links


EXPECTED_LINKS = expected_links()


def diff(seen):
    """A map row is mismatched when the neighbour on that port is not the
    node and NIC the map expects, missing when the port has no neighbour;
    a row is extra when a port off the map has one."""
    by_port = {(r["tor"], r["port"]): r for r in seen}
    mismatched = []
    missing = []
    for e in EXPECTED_LINKS:
        s = by_port.pop((e["tor"], e["port"]), None)
        if s is None:
            missing.append(e)
        elif (s["node"], s["nic"]) != (e["node"], e["nic"]):
            mismatched.append({"expected": e, "seen": s})
    return {"mismatched": mismatched, "missing": missing, "extra": list(by_port.values())}


def speeds(seen):
    """Expected and seen speed per map row, in map order; 0 when the port has no neighbour."""
    by_port = {(r["tor"], r["port"]): r for r in seen}
    expected = [e["gbps"] for e in EXPECTED_LINKS]
    actual = [by_port.get((e["tor"], e["port"]), {}).get("gbps", 0) for e in EXPECTED_LINKS]
    return expected, actual
```

### Gaseous Leak

The rack loop, empty, is charged with forming gas to 200 kPa and left 10 min for the compression heating to settle, then held 30 min with pressure and gas temperature sampled every 10 s. The phase averages the first and last minute, corrects the end pressure to the start temperature through the ideal gas law, and records the decay as an aggregation on the trace. Then the hydrogen sniffer visits all 16 quick-disconnects and the loop is vented.

```python filename="phases/gaseous_leak.py"
import numpy as np

from utils.recipe import ATM_KPA, FORMING_GAS, HOLD_MIN, QD_COUNT, SAMPLE_S, STABILISE_MIN, TEST_PRESSURE_KPA


def gaseous_leak(measurements, leak, log):
    """Rack loop pressurised with forming gas before any coolant goes in,
    a 30 min hold judged on temperature-compensated decay, then the hydrogen
    sniffer at all 16 quick-disconnects. The hold is time-scaled in the mock."""
    leak.pressurise(TEST_PRESSURE_KPA)
    log.info(f"{FORMING_GAS} to {TEST_PRESSURE_KPA:.0f} kPa gauge, {STABILISE_MIN} min to settle (mock: instant)")
    leak.stabilise(STABILISE_MIN)
    t_min, p_kpa, temp_c = leak.hold_trace(HOLD_MIN, SAMPLE_S)
    p = np.asarray(p_kpa)
    tk = np.asarray(temp_c) + 273.15
    n = int(60 / SAMPLE_S)  # first and last minute averaged against the transducer noise
    p0 = float(p[:n].mean())
    p_end_corrected = float(((p[-n:] + ATM_KPA) * tk[:n].mean() / tk[-n:].mean() - ATM_KPA).mean())
    decay_pct = 100.0 * (p0 - p_end_corrected) / p0
    drift_c = float(np.asarray(temp_c)[-n:].mean() - np.asarray(temp_c)[:n].mean())

    m = measurements.pressure_hold
    m.x_axis = t_min
    m.y_axis.pressure_kpa = p_kpa
    m.y_axis.pressure_kpa.aggregations.decay_pct = round(decay_pct, 3)
    m.y_axis.pressure_kpa.aggregations.hold_start_kpa = round(p0, 2)
    m.y_axis.gas_temp_c = temp_c
    m.y_axis.gas_temp_c.aggregations.drift_c = round(drift_c, 2)
    log.info(f"Hold {HOLD_MIN} min from {p0:.2f} kPa: raw end {p[-n:].mean():.2f} kPa, gas {drift_c:+.2f} C, compensated decay {decay_pct:.3f} %")

    ppm = [leak.sniff(i) for i in range(1, QD_COUNT + 1)]
    s = measurements.qd_sniff
    s.x_axis = list(range(1, QD_COUNT + 1))
    s.y_axis.h2_ppm = ppm
    s.y_axis.h2_ppm.aggregations.max_ppm = round(max(ppm), 2)
    leak.vent()
    log.info(f"Sniffer on {QD_COUNT} QDs: max {max(ppm):.2f} ppm at QD {ppm.index(max(ppm)) + 1}, loop vented")
```

![Rack loop gauge pressure over a 30 min hold at 200 kPa of forming gas: the raw reading in green falls 0.32 kPa as the gas cools 0.30 C, the temperature-compensated trace in blue stays flat with 0.010 % decay, both far above the 0.5 % decay limit line at 199.0 kPa.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/rack-l11-chart-pressure-hold-QbMempmKWEJp0RHuO8Kwp1a6PhJca3.png)

The mock loop over the hold: the raw reading loses 0.32 kPa as the shop cools the gas 0.3 °C, which an uncompensated test would report as 0.16 % decay; corrected to the start temperature the loop shows 0.010 %, which is the transducer's noise. The sniffer reads under 0.5 ppm at fifteen quick-disconnects and 1.8 ppm at QD 11, node 6's return, under the 5 ppm limit and worth a look at the next rack from the same manifold lot.

### Fill and Flow

Coolant goes in only after the loop held. The CDU fills from its reservoir with the bleed open and reports the litres taken, which is the first check: a fill 10 % short of the design volume is a node whose quick-disconnects did not open. Then the supply goes to 40 °C, and the phase records the flow and inlet temperature at each node's branch as a multi-dimensional measurement judged on its minimum and its extremes, and the rack's total flow, pressure drop and supply temperature against the CDU boundary.

```python filename="phases/fill_and_flow.py"
from utils.recipe import COOLANT, NODES, NODE_FLOW_MIN_LPM, NODE_LIQUID_KW, SUPPLY_C


def fill_and_flow(measurements, cdu, log):
    """Fill with PG25 from the CDU reservoir, supply to 40 C, then the flow
    the manifold delivers to each node against the 1.25 LPM/kW floor, with
    the rack's total flow and pressure drop inside the CDU boundary."""
    litres = cdu.fill(COOLANT)
    cdu.set_supply_c(SUPPLY_C)
    flows = cdu.node_flow_lpm()
    inlets = cdu.node_inlet_c()
    rack_flow = cdu.rack_flow_lpm()
    dp = cdu.rack_dp_psi()
    supply = cdu.supply_c()

    measurements.fill_volume_l = litres
    m = measurements.node_flow
    m.x_axis = list(range(1, NODES + 1))
    m.y_axis.flow_lpm = flows
    m.y_axis.flow_lpm.aggregations.min_lpm = round(min(flows), 2)
    m.y_axis.flow_lpm.aggregations.min_lpm_per_kw = round(min(flows) / NODE_LIQUID_KW, 3)
    m.y_axis.inlet_c = inlets
    m.y_axis.inlet_c.aggregations.max_c = round(max(inlets), 2)
    m.y_axis.inlet_c.aggregations.min_c = round(min(inlets), 2)
    measurements.rack_flow_lpm = rack_flow
    measurements.rack_dp_psi = dp
    measurements.supply_c = supply
    log.info(f"{litres:.1f} L of {COOLANT} in; supply {supply:.2f} C, rack {rack_flow:.1f} LPM at {dp:.2f} psi; node flow {min(flows):.2f} to {max(flows):.2f} LPM against a {NODE_FLOW_MIN_LPM:.2f} LPM floor, lowest on node {flows.index(min(flows)) + 1}")
```

![PG25 flow per node as bars from 9.32 to 10.42 LPM with node 6 lowest in orange, all above the 8.75 LPM floor line, and inlet temperature per node from 40.16 to 40.53 C on a secondary axis inside the 38 to 42 C supply window, with the rack at 80.0 LPM and 9.3 psi quoted in the legend.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/rack-l11-chart-node-flow-zrj0xRIRqBnJpyaUTcNeMjAR1mWImn.png)

The eight branches of the mock rack: node 6, at the end of the longest hose run on this elevation, gets 9.32 LPM, 6.5 % above the floor; the rack draws 80 LPM of its 100 and drops 9.3 psi of its 15. The floor is 1.25 LPM per kW of heat to liquid, so the number depends on what each node puts into the coolant at its power limit, not on its nameplate.

### Power-On and Inventory

With coolant flowing, the nodes power on over Redfish and the phase waits for their agents. Every node's BIOS, BMC, VBIOS and NIC firmware and its GPU and DIMM counts are read, then collapsed into one object: a field with one value across the rack keeps it, a field with several becomes the sorted list of what was seen, and the `==` against the golden configuration fails on that key with the list in the report. The shelf's idle power closes the phase.

```python filename="phases/power_on_inventory.py"
from utils.recipe import NODES


def collapse(inventories):
    """One object for the rack: a field whose value is the same on every
    node keeps that value; a field that differs becomes the sorted list of
    what was seen, so the == against the golden set fails on that key."""
    out = {}
    for key in inventories[0]:
        seen = sorted({str(inv[key]) for inv in inventories})
        out[key] = inventories[0][key] if len(seen) == 1 else seen
    return out


def power_on_inventory(measurements, rack, log):
    """Nodes powered on over Redfish with the coolant flowing, all eight
    reachable, inventory of every node collapsed into one object against the
    golden configuration, and the shelf's idle power."""
    reachable = rack.power_on_nodes()
    inventories = [rack.node_inventory(i) for i in range(NODES)]
    rack_inventory = collapse(inventories)
    gpus = sum(inv["gpus"] for inv in inventories)
    idle_kw = rack.shelf_power_kw()

    measurements.nodes_reachable = reachable
    measurements.rack_inventory = rack_inventory
    measurements.gpus_seen = gpus
    measurements.idle_power_kw = idle_kw
    log.info(f"{reachable}/{NODES} nodes up, {gpus} GPUs, BIOS {rack_inventory['bios']} BMC {rack_inventory['bmc']} VBIOS {rack_inventory['vbios']}, shelf {idle_kw:.2f} kW at idle")
```

### Network at Rate

Every mapped port loaded to line rate in both directions for 5 min, with the counters cleared before and read after. The phase records four curves indexed by port: line rate achieved, judged on its minimum; FEC corrected codewords, judged on the highest port; FEC uncorrected codewords and PCS symbol errors, judged on their totals, which must be zero. Frames lost across every pair is a fifth number.

```python filename="phases/network_at_rate.py"
from utils.recipe import PORTS, RATE_TEST_MIN


def network_at_rate(measurements, fabric, log):
    """Every mapped port at line rate in both directions for 5 min
    (time-scaled in the mock). Corrected FEC codewords are recorded per port
    and capped; uncorrected codewords, PCS symbol errors and lost frames
    must total zero."""
    log.info(f"{PORTS} ports at rate for {RATE_TEST_MIN} min (mock: instant)")
    r = fabric.run_at_rate(RATE_TEST_MIN)
    corrected = r["fec_corrected"]

    m = measurements.port_counters
    m.x_axis = list(range(1, PORTS + 1))
    m.y_axis.line_rate_pct = r["line_rate_pct"]
    m.y_axis.line_rate_pct.aggregations.min_pct = round(min(r["line_rate_pct"]), 2)
    m.y_axis.fec_corrected = corrected
    m.y_axis.fec_corrected.aggregations.max_per_port = max(corrected)
    m.y_axis.fec_uncorrected = r["fec_uncorrected"]
    m.y_axis.fec_uncorrected.aggregations.total = sum(r["fec_uncorrected"])
    m.y_axis.symbol_errors = r["symbol_errors"]
    m.y_axis.symbol_errors.aggregations.total = sum(r["symbol_errors"])
    measurements.frames_lost = r["frames_lost"]
    log.info(f"Line rate {min(r['line_rate_pct']):.2f} % worst; corrected codewords {min(corrected)} to {max(corrected)} (port {corrected.index(max(corrected)) + 1} highest); uncorrected {sum(r['fec_uncorrected'])}, symbol errors {sum(r['symbol_errors'])}, frames lost {r['frames_lost']}")
```

![FEC corrected codewords per port on a log scale for the 32 mapped ToR ports after 5 min at 400 GbE line rate, between 30,586 and 612,400 with port 19 highest in orange, under the 1,000,000 per-port limit line, with zero uncorrected codewords, zero symbol errors and zero frames lost quoted in the legend.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/rack-l11-chart-fec-per-port-G7XD5nPeuHRNBzky6tn36VusQvfynx.png)

Symbol errors on a healthy 400 GbE link are all zero, so the chart shows what is not: the corrected-codeword count per port, which spans a decade across the rack and peaks on mapped port 19, node 5's eth2 on ToR 2. That port passes; it is the endface to inspect if the count grows at the next run. The corrected ceiling of a million in 5 min corresponds to a pre-FEC bit-error rate around 8e-9, four decades under the 2.4e-4 the code is specified to correct, so a port that reaches it has a problem long before it drops a frame.

### NCCL Cluster

`nvidia-smi nvlink -s` on all 64 GPUs first, then `all_reduce_perf` across the 64 ranks over the fabric with a message-size sweep from 1 MiB to 8 GiB. The phase records bus and algorithm bandwidth per size and judges the peak against 90 % of the node NIC ceiling.

```python filename="phases/nccl_cluster.py"
from utils.recipe import BUSBW_MIN_PCT, MSG_SIZES_B, NODE_NIC_GBS, RANKS


def nccl_cluster(measurements, rack, log):
    """NVLink status on every GPU, then all_reduce_perf across the 64 ranks
    over the fabric with a message-size sweep. Bus bandwidth at the largest
    message is judged against 90 % of the node NIC ceiling: an inter-node
    ring cannot move more than the node's NICs carry."""
    links_down = rack.nvlink_links_down()
    bus, alg = rack.nccl_all_reduce(MSG_SIZES_B)
    peak = max(bus)
    pct = 100.0 * peak / NODE_NIC_GBS

    measurements.nvlink_links_down = links_down
    m = measurements.all_reduce
    m.x_axis = [s / 2**20 for s in MSG_SIZES_B]
    m.y_axis.busbw_gbs = bus
    m.y_axis.busbw_gbs.aggregations.peak_gbs = round(peak, 2)
    m.y_axis.busbw_gbs.aggregations.pct_of_ceiling = round(pct, 2)
    m.y_axis.algbw_gbs = alg
    log.info(f"{RANKS} ranks, {links_down} NVLinks down; busbw {bus[0]:.1f} GB/s at 1 MiB to {peak:.1f} GB/s at 8 GiB, {pct:.1f} % of the {NODE_NIC_GBS:.0f} GB/s ceiling (floor {BUSBW_MIN_PCT:.0f} %)")
```

The mock rack reaches 187.4 GB/s at 8 GiB, 93.7 % of the 200 GB/s its four 400 GbE NICs per node can carry. A rack with one NIC on a slow lane, or a ring that NCCL had to route through fewer NICs because of the cable map, plateaus lower and fails here after passing every port test individually.

### Soak

Twenty-four hours at the power limit on every node under DCGM diagnostics, instant in the mock. The phase records per node the hottest GPU, throttle events and ECC counters as one multi-dimensional measurement judged on the maximum temperature and the totals, and reads the leak sensors under the manifold and the shelf's fault log.

```python filename="phases/soak.py"
from utils.recipe import GPU_TEMP_MAX_C, NODES, SOAK_H


def soak(measurements, rack, cdu, log):
    """24 h at the power limit on every node (time-scaled in the mock):
    the hottest GPU per node, throttle events, ECC counters, the leak sensors
    under the manifold and the shelf's fault log. Nothing may grow."""
    log.info(f"Soak {SOAK_H} h at the power limit on {NODES} nodes (mock: instant)")
    r = rack.soak(SOAK_H)
    temps = r["max_gpu_temp_c"]

    m = measurements.node_soak
    m.x_axis = list(range(1, NODES + 1))
    m.y_axis.max_gpu_temp_c = temps
    m.y_axis.max_gpu_temp_c.aggregations.max_c = round(max(temps), 1)
    m.y_axis.throttle_events = r["throttle_events"]
    m.y_axis.throttle_events.aggregations.total = sum(r["throttle_events"])
    m.y_axis.ecc_uncorrectable = r["ecc_uncorrectable"]
    m.y_axis.ecc_uncorrectable.aggregations.total = sum(r["ecc_uncorrectable"])
    m.y_axis.ecc_corrected = r["ecc_corrected"]
    measurements.leak_sensor_events = cdu.leak_events()
    measurements.psu_faults = rack.psu_faults()
    log.info(f"Hottest GPU {max(temps):.1f} C on node {temps.index(max(temps)) + 1} (limit {GPU_TEMP_MAX_C:.0f} C), {sum(r['throttle_events'])} throttle events, {sum(r['ecc_uncorrectable'])} uncorrectable ECC, {sum(r['ecc_corrected'])} corrected, {cdu.leak_events()} leak events, {rack.psu_faults()} PSU faults")
```

The hottest GPU on the mock rack is on node 6 at 78.7 °C, the node with the least flow, which is the correlation the two measurements exist to show. Corrected ECC is recorded without a limit; uncorrectable ECC, throttling, leaks and PSU faults must be zero.

### Ship Prep

The teardown reads the loop's volume from the CDU and classifies it wet, drained or partial, puts the CDU into ship mode, reads the shelf blackbox as one object and the BMC event logs for critical entries, and writes the final cable map, LLDP table and diff to a file that it attaches to the run. The operator's ship-state declaration and the capped-QDs confirmation arrive through the phase's UI, bound to their measurements.

```python filename="phases/ship_prep.py"
import json
import tempfile
from pathlib import Path

from utils.recipe import LOOP_VOLUME_L
from utils.topology import EXPECTED_LINKS, diff


def ship_prep(measurements, rack, cdu, fabric, attach, unit, log):
    """Teardown: the loop's state as the CDU sees it next to the state the
    operator declared, the CDU into ship mode, the shelf blackbox and the
    BMC event logs empty, and the final cable map attached as a report."""
    litres = cdu.loop_volume_l()
    if litres >= 0.9 * LOOP_VOLUME_L:
        loop_state = "wet"
    elif litres <= 0.05 * LOOP_VOLUME_L:
        loop_state = "drained"
    else:
        loop_state = "partial"
    cdu.ship_mode()
    measurements.loop_state = loop_state
    measurements.coolant_in_loop_l = litres
    measurements.shelf_blackbox = rack.shelf_blackbox()
    measurements.sel_critical = rack.sel_critical()

    seen = fabric.lldp_table()
    report = {"rack": unit.serial_number, "map": EXPECTED_LINKS, "lldp": seen, "diff": diff(seen)}
    path = Path(tempfile.gettempdir()) / f"topology-diff-{unit.serial_number}.json"
    path.write_text(json.dumps(report, indent=2))
    attach.file(str(path), "topology-diff.json")
    log.info(f"Loop {loop_state} with {litres:.1f} L, CDU in ship mode, blackbox and SEL clean, cable map attached")
```

The headless run answers the UI from `ui.json`:

```json filename="ui.json"
{
  "ship_prep": {
    "ship_state": "wet",
    "qds_capped": true
  }
}
```

### Mock Plugs

Four plugs, one per link, because each is a different connection on a real line and plugs run in separate processes. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

`RackManager` stands in for the rack manager, the eight BMCs, the shelf and the node agents: it returns the build record and the golden inventory, reports 10.6 kW at idle, models the all-reduce as an 85 µs latency floor plus the bytes each rank moves over its share of the node's NICs at 93.5 % of their rate, and returns a soak whose hottest GPU is on node 6, 78.3 °C before noise.

```python filename="plugs/rack_manager.py"
"""Rack manager, node BMCs and the node agents (mock): asset labels and FRUs,
Redfish power and inventory, the power shelf over its Modbus port, and an
agent on every node that runs nvidia-smi, nvbandwidth, NCCL tests and DCGM.

Maps to the rack manager's API for the shelf and the leak GPIOs, Redfish on
the eight BMCs for power, FRU, SEL and sensors, and an SSH agent on every node
for the GPU tooling. The mock synthesizes a rack built to its record: every
node at the golden configuration, all NVLinks up, ring all-reduce reaching
93 % of the node NIC ceiling at 8 GiB, and 24 h of soak with the hottest GPU
at 78 C, no throttling, no uncorrectable ECC and a silent shelf.
"""

import numpy as np

from utils.recipe import BUILD_RECORD, GOLDEN_CONFIG, NODES, NODE_NIC_GBS, RANKS


class RackManager:
    T0_S = 85e-6  # all-reduce latency floor across 64 ranks
    BUSBW_FRACTION = 0.935  # of the node NIC ceiling, once the message is large

    def __init__(self, address):
        self.address = address
        self._rng = np.random.default_rng(1400)
        self._powered = False
        # self.redfish = [redfish.redfish_client(bmc) for bmc in ...]; self.shelf = ModbusTcpClient(address)
        print(f"Rack manager at {address}: 8 BMCs enumerated, shelf on the bus")

    def read_build_labels(self):
        """Rack asset tag and elevation revision from the rack manager, node
        serials from the BMC FRUs, switch serials from their management API,
        PSU serials from the shelf's Modbus registers."""
        return {k: (list(v) if isinstance(v, list) else v) for k, v in BUILD_RECORD.items()}

    def power_on_nodes(self):
        """Power every node on over Redfish and wait for the OS agents. Nodes reachable."""
        self._powered = True
        return NODES

    def node_inventory(self, index):
        """BIOS, BMC, VBIOS and NIC firmware, GPU and DIMM counts from Redfish and the agent."""
        return dict(GOLDEN_CONFIG)

    def shelf_power_kw(self):
        return round(10.6 + self._rng.normal(0.0, 0.1), 2)

    def nvlink_links_down(self):
        """nvidia-smi nvlink -s on every GPU: links reported inactive."""
        return 0

    def nccl_all_reduce(self, sizes_b):
        """all_reduce_perf across the 64 ranks over the fabric, one row per
        message size. Ring time is a latency floor plus the bytes each rank
        moves over its share of the node's NICs."""
        factor = 2.0 * (RANKS - 1) / RANKS
        bus = []
        alg = []
        for s in sizes_b:
            t = self.T0_S + factor * s / (self.BUSBW_FRACTION * NODE_NIC_GBS * 1e9)
            t *= 1.0 + self._rng.normal(0.0, 0.004)
            alg.append(round(s / t / 1e9, 2))
            bus.append(round(factor * s / t / 1e9, 2))
        return bus, alg

    def soak(self, hours):
        """DCGM diagnostics at the power limit on every node for the stated
        time (mock: instant): the hottest GPU temperature per node, clock
        throttle events, ECC counters."""
        max_c = [76.4, 74.9, 77.8, 75.2, 76.1, 78.3, 75.7, 76.9]
        return {
            "max_gpu_temp_c": [round(t + self._rng.normal(0.0, 0.2), 1) for t in max_c],
            "throttle_events": [0] * NODES,
            "ecc_uncorrectable": [0] * NODES,
            "ecc_corrected": [0, 1, 0, 0, 2, 0, 0, 1],
        }

    def psu_faults(self):
        """Fault entries across the shelf's modules since power-on."""
        return 0

    def shelf_blackbox(self):
        """The six modules' blackbox counters, summed."""
        return {"ac_loss": 0, "ovp": 0, "ocp": 0, "otw": 0, "otp": 0, "fan": 0}

    def sel_critical(self):
        """System event log entries of severity Critical across the eight BMCs since power-on."""
        return 0

    def __del__(self):
        print("Nodes left powered, BMC sessions closed")
```

`Fabric` returns an LLDP table that matches the map and a run at rate where corrected codewords are log-uniform between 30,000 and 400,000 per port with port 19 at 612,400, and nothing uncorrected.

```python filename="plugs/fabric.py"
"""The two top-of-rack switches and the nodes' NICs (mock): the LLDP
neighbour tables, the port counters, and a line-rate traffic run driven from
the nodes.

Maps to the ToRs' management API (SONiC over gNMI or the vendor's REST) for
LLDP and the per-port PCS and FEC counters, and to a traffic agent on every
node (ib_write_bw or iperf3 in a full mesh, or the fabric's own diagnostic
traffic) for the run at rate. The mock synthesizes a rack cabled exactly to
its map, every port at 400 GbE, and RS(544,514) FEC correcting a healthy
number of codewords with nothing uncorrected: port 19 corrects more than the
others and is the endface to clean next time.
"""

import numpy as np

from utils.recipe import PORTS
from utils.topology import EXPECTED_LINKS


class Fabric:
    def __init__(self, tor1, tor2):
        self.tors = [tor1, tor2]
        self._rng = np.random.default_rng(1300)
        # self.api = [SonicClient(tor1), SonicClient(tor2)]; self.agents = [...]
        print(f"ToRs at {tor1} and {tor2} reachable, node traffic agents idle")

    def lldp_table(self):
        """Neighbour rows from both ToRs: local port, neighbour system name
        (the node serial) and port id (the NIC), negotiated speed."""
        return [{"tor": e["tor"], "port": e["port"], "node": e["node"], "nic": e["nic"], "gbps": e["gbps"]} for e in EXPECTED_LINKS]

    def run_at_rate(self, minutes):
        """Every port loaded to line rate in both directions for the stated
        time (mock: instant); the counters are cleared before and read after.
        Corrected codewords are log-uniform across the ports; port 19 sits
        higher; nothing is uncorrected and no PCS symbol error is seen."""
        rate = 99.7 + self._rng.normal(0.0, 0.05, PORTS)
        corrected = np.exp(self._rng.uniform(np.log(3e4), np.log(4e5), PORTS)).astype(int)
        corrected[18] = 612_400
        return {
            "line_rate_pct": rate.round(2).tolist(),
            "fec_corrected": corrected.tolist(),
            "fec_uncorrected": [0] * PORTS,
            "symbol_errors": [0] * PORTS,
            "frames_lost": 0,
        }

    def __del__(self):
        print("Traffic agents stopped, ToR sessions closed")
```

`LeakBench` holds a loop whose pressure follows the gas temperature, with a 0.02 % leak under the transducer's 0.03 kPa of noise, and a sniffer that reads 1.6 ppm at QD 11 and a quarter of that elsewhere.

```python filename="plugs/leak_bench.py"
"""Gaseous leak bench (mock): the forming-gas panel on the rack manifold's
supply stub, a gauge-pressure transducer and a gas thermocouple on the
return stub, and a hydrogen sniffer probe.

Maps to a regulated N2/H2 (95/5) panel with a shut-off valve, a 0 to 400 kPa
transducer read over Modbus, a type-T thermocouple in the return stub, and an
Inficon Sensistor Sentrac hydrogen leak detector with its hand probe. Station
scoped: the sniffer's warm-up is longer than a rack's test, so the station
keeps it open between racks. The mock synthesizes a tight loop: a pressure
that follows the gas temperature, no leak the transducer can resolve, and one
quick-disconnect that reads above the others and still well under the limit.
"""

import numpy as np

from utils.recipe import ATM_KPA


class LeakBench:
    def __init__(self, gas_panel, sniffer):
        self.gas_panel = gas_panel
        self.sniffer = sniffer
        self._rng = np.random.default_rng(1100)
        self._p_kpa = 0.0
        # self.panel = minimalmodbus.Instrument(gas_panel, 1); self.sniffer = serial.Serial(sniffer, 9600)
        print(f"Leak bench: gas panel on {gas_panel} closed, sniffer on {sniffer} warm")

    def pressurise(self, kpa):
        """Open the panel until the loop reads the target gauge pressure, then shut."""
        self._p_kpa = float(kpa)

    def stabilise(self, minutes):
        """Wait out the compression heating so the hold starts from a settled gas (mock: instant)."""
        return minutes

    def hold_trace(self, minutes, sample_s):
        """Gauge pressure and gas temperature over the hold. The shop cools
        0.3 C over the half hour and the gas follows it; a 0.02 % real
        leak sits under the transducer's 0.03 kPa of noise."""
        n = int(minutes * 60 / sample_s) + 1
        t_min = np.arange(n) * sample_s / 60.0
        temp_c = 23.4 - 0.3 * t_min / minutes + self._rng.normal(0.0, 0.02, n)
        p_abs0 = self._p_kpa + ATM_KPA
        p_abs = p_abs0 * (temp_c + 273.15) / (temp_c[0] + 273.15) * (1.0 - 0.0002 * t_min / minutes)
        p_kpa = p_abs - ATM_KPA + self._rng.normal(0.0, 0.03, n)
        return t_min.round(3).tolist(), p_kpa.round(3).tolist(), temp_c.round(3).tolist()

    def sniff(self, qd_index):
        """Hydrogen at one quick-disconnect, in ppm above the background the
        probe was zeroed on. QD 11 (node 6 return) reads higher than the rest."""
        base = 1.6 if qd_index == 11 else 0.25
        return round(base + abs(self._rng.normal(0.0, 0.08)), 2)

    def vent(self):
        self._p_kpa = 0.0

    def __del__(self):
        print("Gas panel closed, loop vented, sniffer left warm")
```

`Cdu` fills to 1.4 % over the design volume, delivers eight branch flows between 9.3 and 10.4 LPM with node 6 lowest, and holds the supply 0.12 °C over its set-point.

```python filename="plugs/cdu.py"
"""Rack CDU and manifold instrumentation (mock): the fill pump and the PG25
reservoir, the supply temperature control, the CDU's own flow meter and
differential-pressure transducer, one turbine flow meter and one inlet
thermistor per node on the manifold, and the leak-rope sensors under the
manifold and in every node's drip tray.

Maps to an in-rack CDU with a Modbus TCP register map (Vertiv CoolChip,
Motivair or the rack vendor's), the manifold's per-branch flow meters on the
same bus, and the leak sensors on the rack manager's GPIO. The mock
synthesizes a loop that fills to its design volume, eight branches within
10 % of each other with node 6 a little restricted, a supply within 0.2 C of
its set-point and a rack dP well under the boundary.
"""

import numpy as np

from utils.recipe import LOOP_VOLUME_L, NODES

# Branch flow at the design pump speed, litres per minute, node 1 to 8. Node 6
# has the longest hose run on this elevation.
_BRANCH_LPM = [10.2, 9.9, 10.4, 9.7, 10.1, 9.3, 10.0, 9.8]


class Cdu:
    def __init__(self, address):
        self.address = address
        self._rng = np.random.default_rng(1200)
        self._litres = 0.0
        self._supply_c = 0.0
        self._pump = False
        # self.modbus = ModbusTcpClient(address)
        print(f"CDU at {address}: pump off, reservoir full, loop empty")

    def fill(self, coolant):
        """Fill from the reservoir with the air bleed open until the return
        runs clear, then close the bleed. Litres taken from the reservoir."""
        self._litres = round(LOOP_VOLUME_L * 1.014 + self._rng.normal(0.0, 0.2), 1)
        return self._litres

    def set_supply_c(self, celsius):
        self._supply_c = float(celsius)
        self._pump = True

    def supply_c(self):
        return round(self._supply_c + 0.12 + self._rng.normal(0.0, 0.03), 2)

    def node_flow_lpm(self):
        return [round(f + self._rng.normal(0.0, 0.05), 2) for f in _BRANCH_LPM] if self._pump else [0.0] * NODES

    def node_inlet_c(self):
        """Manifold branch temperature at each node, warmer down the rack as
        the supply picks up heat from the return manifold beside it."""
        return [round(self._supply_c + 0.12 + 0.06 * i + self._rng.normal(0.0, 0.03), 2) for i in range(NODES)]

    def rack_flow_lpm(self):
        return round(sum(_BRANCH_LPM) + 0.3 + self._rng.normal(0.0, 0.1), 1) if self._pump else 0.0

    def rack_dp_psi(self):
        return round(9.4 + self._rng.normal(0.0, 0.05), 2) if self._pump else 0.0

    def leak_events(self):
        """Leak-rope and drip-tray events since the fill, from the rack manager."""
        return 0

    def loop_volume_l(self):
        return self._litres

    def ship_mode(self):
        """Pump off, isolation valves closed on both stubs, coolant left where it is."""
        self._pump = False

    def __del__(self):
        print("CDU pump off, valves closed")
```

The constants and where each limit comes from live in one file:

```python filename="utils/recipe.py"
"""Integration recipe for a liquid-cooled AI rack at L11: eight GPU nodes,
two top-of-rack switches and one ORV3 power shelf on a 48 V busbar, racked
with the network and the coolant manifold attached, after every node passed
its L10 system test and before the rack ships.

Nothing at L11 is a published spec. The level itself is the OCP liquid
cooling integration white paper's (L10 boots and passes its witness test,
L11 is racked with network and manifold, L12 adds the software environment),
and the same paper puts a gaseous leak test of the servers, the manifold and
the CDU before the fill. The 0.5 % pressure-decay figure is the OCP cold plate
qualification's (EN 1779). The rack's coolant boundary is Meta's Catalina
example: PG25, 40 C supply, at most 100 LPM and 15 psi at the rack, 1.25 to
1.5 LPM per kW. Everything else is this line's practice, and each constant
says so where it is used and again in procedure.yaml."""

# The rack
NODES = 8
NODE_LIQUID_KW = 7.0  # heat to liquid per node at its power limit
TORS = 2
PORTS_PER_NODE = 4  # 400GbE NICs per node, two on each ToR
PORT_GBPS = 400
PSU_MODULES = 6  # ORV3 HPR 12 kW modules, N+1 for the rack's 58 kW

BUILD_RECORD = {
    "rack": "RACK-AI8-00217",
    "elevation_rev": "C",
    "nodes": ["N8G-26-04117", "N8G-26-04118", "N8G-26-04119", "N8G-26-04120",
              "N8G-26-04121", "N8G-26-04122", "N8G-26-04123", "N8G-26-04124"],
    "switches": ["TOR-26-00871", "TOR-26-00872"],
    "psus": ["ORV3-12K-004217", "ORV3-12K-004218", "ORV3-12K-004221",
             "ORV3-12K-004224", "ORV3-12K-004230", "ORV3-12K-004231"],
}
GOLDEN_CONFIG = {"bios": "2.4.1", "bmc": "1.12.0", "vbios": "96.00.89.00.01",
                 "nic_fw": "28.41.1000", "gpus": 8, "dimm_gb": 2048}

# Gaseous leak before the fill (OCP integration WP), decay figure from the
# OCP cold plate qualification (EN 1779). The hold is time-scaled in the mock.
FORMING_GAS = "N2 with 5 % H2"
TEST_PRESSURE_KPA = 200.0  # gauge, this loop's maximum operating pressure
ATM_KPA = 101.3
STABILISE_MIN = 10  # compression heating settles before the hold starts
HOLD_MIN = 30
SAMPLE_S = 10
DECAY_LIMIT_PCT = 0.5
QD_COUNT = 2 * NODES  # one supply and one return quick-disconnect per node
H2_LIMIT_PPM = 5.0  # this line's: ten times the sniffer's noise floor above background

# Fill and flow (Meta Catalina rack boundary)
COOLANT = "PG25"
SUPPLY_C = 40.0
SUPPLY_TOL_C = 2.0
RACK_FLOW_MAX_LPM = 100.0
RACK_DP_MAX_PSI = 15.0
LPM_PER_KW = 1.25
NODE_FLOW_MIN_LPM = LPM_PER_KW * NODE_LIQUID_KW  # 8.75 LPM per node
LOOP_VOLUME_L = 42.0  # manifold, hoses and eight nodes of cold plates, from the rack's design
FILL_TOL_PCT = 10.0  # a fill far short of the design volume is a node that did not open

# Power-on and inventory
IDLE_POWER_KW = (8.0, 14.0)  # this line's: eight nodes idling and two switches on this shelf

# Network at rate (time-scaled in the mock)
PORTS = NODES * PORTS_PER_NODE  # 32
RATE_TEST_MIN = 5
LINE_RATE_MIN_PCT = 99.0
FEC_CORRECTED_MAX = 1_000_000  # per port over the 5 min, this line's: about 8e-9 pre-FEC BER on RS(544,514), four decades under the 2.4e-4 budget

# NCCL across the rack
RANKS = NODES * 8
MSG_SIZES_B = [2**k for k in range(20, 34)]  # 1 MiB to 8 GiB
NODE_NIC_GBS = PORTS_PER_NODE * PORT_GBPS / 8.0  # 200 GB/s: the inter-node ceiling of ring all-reduce bus bandwidth
BUSBW_MIN_PCT = 90.0

# Soak (time-scaled in the mock)
SOAK_H = 24
GPU_TEMP_MAX_C = 85.0  # this line's: 5 C under the GPU's slowdown threshold
```

On a real line, replace each plug with the class that speaks to the link it mocks: the rack manager's API and Redfish for `RackManager`, with an SSH agent per node for the GPU tooling; the ToRs' gNMI or REST for `Fabric`, with a traffic agent per node; Modbus on the gas panel and serial on the sniffer for `LeakBench`; Modbus TCP on the CDU for `Cdu`. Keep the temperature compensation in the phase, clear the port counters before the run at rate and read them after, let the hold and the soak run their full time, and remove `--ui-values` so the operator answers the teardown. The phases, measurements and limits stay the same.


### Server L10 System Test and Run-In
URL: https://www.tofupilot.com/templates/server-l10-system-test-and-run-in
L10 test of a 2U dual-socket server: Redfish inventory against the golden config, memtest, PCIe links, fio, NIC loopback, TDP stress and a 24 h run-in.

## Introduction

### Server L10 Overview

A server reaches the end of its integrator's line as an L10 unit: the chassis with its motherboard (L6), the CPUs, DIMMs, drives and adapters installed (L7 to L9), a BMC that answers on its management port, and an operating system it can boot. The [OCP liquid-cooling integration and logistics white paper](https://www.opencompute.org/documents/ocp-liquid-cooling-integration-and-logistics-white-paper-revision-1-0-1-pdf) defines the levels: L6 is the board in a chassis with no I/O, L6 to L9 add the cards, drives, CPUs and memory, L10 is the unit that boots an OS and receives the final witness test, L11 is the rack with its network and manifold, L12 the rack with its software environment. The L10 test is the last time the server is seen alone, with every part reachable, before it is racked and cabled into an L11 where a DIMM that corrects errors or a drive on a x2 link becomes one node in a cluster that is slow for a reason nobody can name.

![DDR5 server RDIMM with DRAM chips, supporting components, a keyed gold contact edge, and side retention notches.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/server-l10-system-test-and-run-in-matte-closeup-chC7Y0ZkGoPVcTednBnADfIjxFVHq8.webp)

The parts this test exercises: the BMC and the firmware set it reports, the two CPUs and their heatsinks, the 32 DIMMs and the 16 channels behind them, the 8 NVMe drives and the 2 network adapters with the PCIe links they trained on, the fans and the power supplies under a 24 h soak at the chamber's 40 °C.

No public standard sets an L10 pass mark. The content is integrator practice, described the same way by [AI Data Center Guide's system-integration chapter](https://aidatacenterguide.com/part-7-compute-silicon-and-system-integration/7-14-server-and-system-integration) and by the job postings that hire for it: **BMC and Redfish inventory against a golden configuration**, a **memory test**, **PCIe link width and generation**, **NIC loopback**, a **thermal soak at the power limit**, and a **run-in** of 24 to 72 hours at elevated ambient with no normative number behind it. [Supermicro's rack services brochure](https://www.supermicro.com/manuals/brochure/Brochure-Rack-Services.pdf) calls its version sBurn and rBurn; a [Jabil test-engineering posting](https://jobs.jabil.com/en/job/florence/principal-product-test-engineer-rack-and-server/626/96836667856) asks for L10 server and L11 rack procedures and reports first-pass yield on them. What the tools define is the method, not the limit: [MemTest86's test descriptions](https://www.memtest86.com/tech_individual-test-descr.html) for the memory patterns (moving inversions, walking ones and zeros, modulo-X, bit fade, row hammer), [SNIA PTS-E](https://www.snia.org/sites/default/files/technical-work/pts/release/SNIA-SSS-PTS-Enterprise-v1.1.pdf) for the SSD steady state (purge, preconditioning at twice the capacity in 128 KiB sequential writes, five rounds inside a 20 % window), and the [OCP Datacenter NVMe SSD specification v2.6](https://www.opencompute.org/documents/datacenter-nvme-ssd-specification-v2-6-2-pdf) for the SMART fields, including the PLP capacitor health of the extended SMART log C0h. Every limit in this template is this line's, and the table below says where each one comes from.

### Test Purpose

The procedure records one L10 fingerprint per server:

- **BMC reachable, firmware set, processors** as whole objects against the golden configuration, the **DIMM, NVMe and NIC counts**, the total memory and the slowest DIMM speed, and the **SEL cleared** before the first test
- **Negotiated PCIe generation and width** of every NIC and every NVMe drive, judged on the worst device per class, and the **AER correctable count**
- **MemTest86 bit errors**, **correctable and uncorrectable ECC per DIMM**, and a **STREAM triad** at rising thread counts against a floor
- **fio preconditioning** as an executable phase judged on its exit code, then **4 KiB random read IOPS and 128 KiB sequential MB/s per drive** at steady state, and **SMART media errors, percentage used and capacitor health per drive**
- **Frames sent, lost and FCS errors** on both 100 GbE ports through loopback plugs, and the **link state** of both ports as one object
- **Package power per socket against the TDP**, the **hottest core**, the **fan duty at the end** and the **throttle counter** over 10 minutes of all-core load
- The **24 h run-in trace** at 40 °C inlet, and the **ECC and media-error deltas per DIMM and per drive** against what the earlier phases recorded, with the SEL, throttle and PSU redundancy counters
- At teardown, the **SEL cleared again**, the **BIOS checksum** and **boot configuration** of the shipping profile, the MAC addresses and part serials in the unit's metadata, and the **Redfish inventory attached**

![24 h run-in at 40 C inlet: package power of both sockets at 330 to 339 W under the 350 W TDP line, the hottest core at 80 to 82 C under the 90 C limit, the hottest NVMe at 56 to 58 C under the 70 C limit, and the inlet at 39.5 to 40.6 C inside the 40 plus or minus 2 C window.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/server-l10-chart-run-in-UKH76XcLKGixs56IGhFfU23VzLaQJH.png)

The mock server's run-in: both packages at 333 to 339 W under the 350 W TDP the shipping BIOS profile sets as PL1, the hottest core at 81 °C with the inlet at 40 °C, the hottest drive at 58 °C, and nothing added to any counter over the 24 hours.

The framework mechanics on show are previous-results injection, with `run_in` receiving the `memory_test` and `nvme_test` results as parameters and computing its deltas from them; an executable phase for the fio preconditioning, judged on its exit code and chained with `depends_on` and a `timeout` like any Python phase; JSON `==` on whole objects for the firmware set, the processors, the link state and the boot configuration; `attach.file` for the Redfish inventory; a `timeout` sized per phase up to the framework's 24 h cap; and per-device multi-dimensional measurements judged on `min` and `max` aggregations.

## Equipment & Setup

To run this L10 test on an integration line, the following are required:

- **A network path to the server's BMC** with Redfish enabled, and to the test OS image over the data network
- **A PXE-booted test OS image** carrying the memory tester, `edac-util`, STREAM, `fio`, `nvme-cli`, a loopback traffic generator and the stress load
- **Two 100 GbE loopback plugs** (QSFP28, 0 dB) for the NIC ports
- **A thermal chamber or a hot aisle** holding 40 °C inlet for the run-in, with the server on the line's power feed through both PSUs
- **The Device Under Test (DUT)**: a 2U dual-socket x86 server with 2 CPUs, 32 DIMMs, 8 NVMe drives, 2 x 100 GbE adapters, the release BIOS, BMC and CPLD, and the test BIOS profile that exposes the counters
- **A TofuPilot Framework procedure** to sequence the phases and validate the limits
- **The TofuPilot Dashboard** to keep the per-server evidence and the first-pass yield per line

### Hardware Components

#### BMC and Test OS Image

The BMC is the only instrument this station owns. Over [DMTF Redfish](https://www.dmtf.org/standards/redfish) it answers the firmware inventory (`/redfish/v1/UpdateService/FirmwareInventory`), the memory, storage and network collections under `/Systems/1`, the system event log under `/Managers/1/LogServices/SEL`, the BIOS settings and the thermal and power sensors that the stress and run-in phases sample. Everything else runs in the test OS: `lspci -vv` for the negotiated PCIe links, the memory tester and `edac-util` for the ECC counters, [STREAM](https://www.cs.virginia.edu/stream/) for the bandwidth, [fio](https://github.com/axboe/fio) with `--output-format=json` for the preconditioning and the steady-state rounds, [nvme-cli](https://github.com/linux-nvme/nvme-cli) for `smart-log` and the OCP `ocp smart-add-log`, and [stress-ng](https://github.com/ColinIanKing/stress-ng) for the all-core load. The image is PXE-booted so the drives stay blank; the customer's OS is installed at L11 or on site.

![Open dual-processor rack server showing two CPU heatsinks, DIMM banks, a fan wall, and rear expansion slots.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/server-l10-system-test-and-run-in-matte-cover-5ozil0oXk9nTo8Cko8LKN6H7azsEtQ.webp)

The illustrated hardware: a dual-processor server with its cover removed to show the heatsinks, memory and fan wall. Run-in uses the closed unit with the specified network connections and loopback plugs.

#### Loopback Plugs, Chamber and Traffic

A 100 GbE loopback plug returns every frame the port sends to the same port through the four lanes of the cage, so a frame lost in 60 s at line rate is a lane, a retimer or a cage that will drop frames in the customer's rack. The generator can be the NIC vendor's own loopback tool or [Cisco TRex](https://trex-tgn.cisco.com/) on the test image. The run-in needs 40 °C at the inlet with the server under load; a reach-in chamber from [Thermotron](https://www.thermotron.com/) or [Weiss Technik](https://www.weiss-technik.com/) holds it to ±2 °C, and a hot-aisle enclosure with a heater does the same for a row of units at once. The chamber's setpoint and the soak duration are written into the run's metadata so a unit tested at a different profile can be found later.

### Where the Limits Come From

| Test | Limit | Basis |
|---|---|---|
| Firmware, processors, counts, speed | exact match with the golden configuration | the release; no tolerance applies |
| PCIe links | Gen5 x16 on every NIC, Gen5 x4 on every NVMe, AER correctable = 0 | the board's block diagram; a link that trained lower is a seating or a signal-integrity fault |
| MemTest86, ECC | 0 bit errors, 0 correctable, 0 uncorrectable | this line's: a DIMM that corrects at L10 will correct more in the rack |
| STREAM triad | peak ≥ 400 GB/s | this line's derivation: 65 % of the 614 GB/s theoretical of 16 channels at 4800 MT/s; a channel that trained slow or in a degraded mode drops the triad below it |
| fio at steady state | min ≥ 2.0 M IOPS (4 KiB random read), min ≥ 11.2 GB/s (128 KiB sequential read) | this line's: 80 % of the drive's datasheet; SNIA PTS-E defines how to measure, not a pass mark |
| SMART | media errors = 0, percentage used ≤ 1 %, PLP capacitor health ≥ 100 % | OCP Datacenter NVMe SSD v2.6 for the fields (SMART 02h, extended SMART C0h); the values are this line's for a new drive |
| NIC loopback | frames lost = 0, FCS errors = 0, ≥ 480 M frames in 60 s, 100000 Mb/s full duplex RS-FEC on both ports | this line's: line rate at 1518 B frames is 8.1 M frames/s; the link state is the release |
| CPU stress | max ≤ 350 W per socket, settled mean ≥ 300 W, hottest core ≤ 85 °C, fan duty ≤ 80 % at the end, throttle events = 0 | the TDP is the CPU's and PL1 of the shipping profile; 85 % of TDP, 85 °C at 25 °C inlet and the fan headroom are this line's |
| Run-in | 24 h at 40 ± 2 °C, max ≤ 350 W, hottest core ≤ 90 °C, hottest NVMe ≤ 70 °C, every delta = 0 | integrator practice for the duration and ambient (24 to 72 h, no normative number); 90 °C is 10 °C under PROCHOT, 70 °C is 10 °C under the drive's warning threshold, this line's |
| BIOS checksum, boot config, SEL | exact match, 0 entries | the shipping profile of the release |

Two rows in that table are traps. The STREAM floor is not the datasheet's bandwidth: a triad reaches 70 to 75 % of the theoretical on a healthy platform, so a floor set at the theoretical fails every unit and a floor set at 50 % passes a server with one channel down. And the ECC rule is zero at L10 because the counter is the only early signal a DIMM gives; a line that allows a few correctable errors here has no baseline to compare the run-in delta against, which is why `run_in` receives the per-DIMM counts from `memory_test` rather than a total.

## Test Procedure

### Overview

The procedure maps the L10 test onto the framework's three stages. The inventory and the PCIe links live in `setup:` so a unit with the wrong firmware or a drive on a x2 link never enters the two hours of memory and drive testing; `pcie_links` depends on `bmc_and_inventory` because both use the same server sessions. The main phases are chained in order on the one server. The SEL clear, the shipping BIOS profile and the inventory attachment live in `teardown:` so a unit pulled for repair after a failed run-in still leaves with the test profile removed.

1. Setup: BMC reachable, firmware and processors against the golden configuration, counts, SEL cleared.
2. Setup: PCIe generation and width per NIC and NVMe, AER count.
3. Main: MemTest86 patterns, ECC per DIMM, STREAM triad.
4. Main: fio preconditioning as an executable phase.
5. Main: fio at steady state and SMART per drive.
6. Main: 100 GbE loopback on both ports.
7. Main: all-core stress against the TDP, hottest core, fan duty.
8. Main: 24 h run-in at 40 °C, deltas against phases 3 and 5.
9. Teardown: SEL cleared, shipping BIOS profile, metadata, inventory attached.

Every metric validates against limits declared in `procedure.yaml`, and results stream to TofuPilot as the per-server evidence.

### Why TofuPilot Framework?

TofuPilot Framework is a **YAML + Python** test framework built for hardware manufacturing. Instead of writing all your test logic, measurements, and limits inside Python code, you describe **what** the test does in a `procedure.yaml` file, and **how** in small Python phase files. The framework handles:

* Automatic Python environment management (via `uv`)
* Operator UI (no frontend code needed)
* Measurement validation and live charts
* Process isolation between phases and equipment plugs

### Project Structure

```tree
procedure.yaml
phases/
  bmc_and_inventory.py
  pcie_links.py
  memory_test.py
  nvme_test.py
  nic_loopback.py
  cpu_stress_and_power.py
  run_in.py
  clear_and_ship_config.py
plugs/
  server.py
scripts/
  fio_precondition.py
utils/
  recipe.py
pyproject.toml
README.md
```

You can find the full source on [GitHub](https://github.com/tofupilot/template-framework-server-l10-test). The `ServerUnderTest` plug is a mock of the BMC's Redfish session and the test OS's SSH session together, synthesizing a healthy build with every firmware at the golden version, every link at full generation and width, zero errors on every counter and 335 W per socket under the 350 W TDP, so the procedure runs end-to-end without a server connected. The `nvme_precondition` phase runs `scripts/fio_precondition.py`, which prints the JSON fio prints and exits 0, in place of the real `fio --output-format=json`.

```bash
tofupilot run .
```

For CI or bench automation, the same run executes headless:

```bash
tofupilot run . --no-tui --no-kiosk --json
```

### The Procedure File

`procedure.yaml` declares the unit, the server plug with the BMC address and the label serial as constructor arguments, and the three stages with every measurement and limit:

```yaml filename="procedure.yaml"
name: Server L10 System Test and Run-In
version: 0.1.0
description: L10 system test of a 2U dual-socket x86 server after L6 to L9 assembly. BMC inventory and firmware against the golden configuration over Redfish, PCIe link gen and width of every NIC and NVMe, MemTest86 patterns with per-DIMM ECC and a STREAM triad, fio preconditioning as an executable phase then steady-state IOPS and MB/s with SMART per drive, 100 GbE loopback on both ports, an all-core stress against the TDP, a 24 h run-in at 40 C whose counters are compared with the earlier phases, and a teardown that ships the BIOS profile and attaches the Redfish inventory.

unit:
  auto_identify: true
  serial_number:
    description: "Scan the service tag on the front bezel"
    placeholder: "2U-DS-000000"
    pattern: "^2U-DS-\\d{6}$"
    default_value: "2U-DS-004217"
  part_number:
    default_value: "SRV-2U-DS-G5"
  batch_number:
    default_value: "LOT-2026-39"

plugs:
  - name: Server Under Test
    description: "The BMC over Redfish and the test OS over SSH (mock, one plug per server)"
    python: plugs.server:ServerUnderTest
    key: server
    config:
      bmc_host: "10.20.30.41"
      serial_number: "2U-DS-004217"

setup:
  - name: BMC and Inventory
    key: bmc_and_inventory
    python: phases.bmc_and_inventory
    measurements:
      - name: BMC Reachable
        key: bmc_reachable
        description: "Redfish session opened on the BMC's management port."
        validators:
          - {operator: "==", expected_value: true}
      - name: Firmware Versions
        key: firmware_versions
        description: "BIOS, BMC and CPLD from the Redfish firmware inventory as one object against the golden configuration."
        validators:
          - operator: "=="
            expected_value:
              bios: 2.4.1
              bmc: 1.12.0
              cpld: "0x1a"
      - name: Processors
        key: processors
        validators:
          - operator: "=="
            expected_value:
              count: 2
              model: x86-64 64C/128T 350W
      - name: DIMM Count
        key: dimm_count
        validators:
          - {operator: "==", expected_value: 32}
      - name: Memory Total
        key: memory_total_gb
        unit: GB
        validators:
          - {operator: "==", expected_value: 2048}
      - name: DIMM Minimum Speed
        key: dimm_min_speed_mt_s
        unit: MT/s
        description: "The slowest configured DIMM speed. One DIMM from a slower reel drags the whole channel down to its speed."
        validators:
          - {operator: "==", expected_value: 4800}
      - name: NVMe Count
        key: nvme_count
        validators:
          - {operator: "==", expected_value: 8}
      - name: NIC Count
        key: nic_count
        validators:
          - {operator: "==", expected_value: 2}
      - name: SEL Entries After Clear
        key: sel_entries_after_clear
        description: "The system event log is cleared so the run-in starts from zero."
        validators:
          - {operator: "==", expected_value: 0}

  - name: PCIe Links
    key: pcie_links
    python: phases.pcie_links
    depends_on: [bmc_and_inventory]
    measurements:
      - name: NIC PCIe Links
        key: pcie_nic_links
        title: Negotiated PCIe link per NIC
        x_axis:
          legend: Device
        y_axis:
          - legend: Generation
            key: gen
            aggregations:
              - type: min_gen
                validators:
                  - {operator: "==", expected_value: 5}
          - legend: Width
            key: width
            unit: lanes
            aggregations:
              - type: min_width
                validators:
                  - {operator: "==", expected_value: 16}
      - name: NVMe PCIe Links
        key: pcie_nvme_links
        title: Negotiated PCIe link per NVMe drive
        x_axis:
          legend: Device
        y_axis:
          - legend: Generation
            key: gen
            aggregations:
              - type: min_gen
                validators:
                  - {operator: "==", expected_value: 5}
          - legend: Width
            key: width
            unit: lanes
            aggregations:
              - type: min_width
                validators:
                  - {operator: "==", expected_value: 4}
      - name: AER Correctable Errors
        key: aer_correctable_errors
        description: "PCIe Advanced Error Reporting correctable count since boot, all devices."
        validators:
          - {operator: "==", expected_value: 0}

main:
  - name: Memory Test
    key: memory_test
    python: phases.memory_test
    timeout: 3h
    measurements:
      - name: MemTest Bit Errors
        key: memtest_bit_errors
        description: "Seven MemTest86 patterns, two passes each, over all 2 TB."
        validators:
          - {operator: "==", expected_value: 0}
      - name: ECC Per DIMM
        key: ecc_per_dimm
        title: ECC counters per DIMM after the pattern test
        x_axis:
          legend: DIMM
        y_axis:
          - legend: Correctable
            key: correctable
            aggregations:
              - type: max
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: Uncorrectable
            key: uncorrectable
            aggregations:
              - type: max
                validators:
                  - {operator: "==", expected_value: 0}
      - name: Correctable ECC Total
        key: correctable_ecc_total
        validators:
          - {operator: "==", expected_value: 0}
      - name: STREAM Bandwidth
        key: stream_bandwidth
        title: STREAM triad against thread count
        x_axis:
          legend: Threads
        y_axis:
          - legend: Triad
            key: triad
            unit: GB/s
            aggregations:
              - type: peak_gb_s
                unit: GB/s
                validators:
                  - {operator: ">=", expected_value: 400.0}

  - name: NVMe Precondition
    key: nvme_precondition
    depends_on: [memory_test]
    timeout: 2h
    executable:
      command: "python3 scripts/fio_precondition.py --drives 8 --capacity-tb 7.68 --loops 2 --output-format=json"

  - name: NVMe Test
    key: nvme_test
    python: phases.nvme_test
    depends_on: [nvme_precondition]
    timeout: 1h
    measurements:
      - name: NVMe Performance
        key: nvme_performance
        title: Steady-state fio per drive after PTS-E preconditioning
        x_axis:
          legend: Drive
        y_axis:
          - legend: 4 KiB random read
            key: rand_read_4k_iops
            unit: IOPS
            aggregations:
              - type: min
                unit: IOPS
                validators:
                  - {operator: ">=", expected_value: 2000000}
          - legend: 128 KiB sequential read
            key: seq_read_128k_mb_s
            unit: MB/s
            aggregations:
              - type: min
                unit: MB/s
                validators:
                  - {operator: ">=", expected_value: 11200}
      - name: NVMe SMART
        key: nvme_smart
        title: SMART 02h and OCP extended SMART C0h per drive
        x_axis:
          legend: Drive
        y_axis:
          - legend: Media errors
            key: media_errors
            aggregations:
              - type: max
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: Percentage used
            key: percentage_used
            unit: "%"
            aggregations:
              - type: max
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 1}
          - legend: PLP capacitor health
            key: capacitor_health_pct
            unit: "%"
            aggregations:
              - type: min
                unit: "%"
                validators:
                  - {operator: ">=", expected_value: 100}
      - name: Media Errors Total
        key: media_errors_total
        validators:
          - {operator: "==", expected_value: 0}

  - name: NIC Loopback
    key: nic_loopback
    python: phases.nic_loopback
    depends_on: [nvme_test]
    measurements:
      - name: NIC Loopback
        key: nic_loopback
        title: Line-rate loopback per port for 60 s
        x_axis:
          legend: Port
        y_axis:
          - legend: Frames sent
            key: frames_sent
            aggregations:
              - type: min
                validators:
                  - {operator: ">=", expected_value: 480000000}
          - legend: Frames lost
            key: frames_lost
            aggregations:
              - type: max
                validators:
                  - {operator: "==", expected_value: 0}
          - legend: FCS errors
            key: fcs_errors
            aggregations:
              - type: max
                validators:
                  - {operator: "==", expected_value: 0}
      - name: NIC Link State
        key: nic_link_state
        description: "Speed, duplex and FEC mode of both ports as ethtool reports them, as one object."
        validators:
          - operator: "=="
            expected_value:
              port1: {speed_mb_s: 100000, duplex: full, fec: rs}
              port2: {speed_mb_s: 100000, duplex: full, fec: rs}

  - name: CPU Stress and Power
    key: cpu_stress_and_power
    python: phases.cpu_stress_and_power
    depends_on: [nic_loopback]
    timeout: 20m
    measurements:
      - name: CPU Stress
        key: cpu_stress
        title: All cores at 100 percent for 10 min at 25 C inlet
        x_axis:
          legend: Time
          unit: s
        y_axis:
          - legend: Package 0 power
            key: package_0_w
            unit: W
            aggregations:
              - type: max_w
                unit: W
                validators:
                  - {operator: "<=", expected_value: 350.0}
              - type: settled_mean_w
                unit: W
                validators:
                  - {operator: ">=", expected_value: 300.0}
          - legend: Package 1 power
            key: package_1_w
            unit: W
            aggregations:
              - type: max_w
                unit: W
                validators:
                  - {operator: "<=", expected_value: 350.0}
              - type: settled_mean_w
                unit: W
                validators:
                  - {operator: ">=", expected_value: 300.0}
          - legend: Hottest core
            key: hottest_core_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 85.0}
          - legend: Fan duty
            key: fan_duty_pct
            unit: "%"
            aggregations:
              - type: end_pct
                unit: "%"
                validators:
                  - {operator: "<=", expected_value: 80.0}
      - name: Throttle Events
        key: throttle_events
        description: "PROCHOT and power-limit throttle counters over the stress, both sockets."
        validators:
          - {operator: "==", expected_value: 0}

  - name: Run-In
    key: run_in
    python: phases.run_in
    depends_on: [cpu_stress_and_power]
    timeout: 24h
    measurements:
      - name: Run-In Trace
        key: run_in_trace
        title: 24 h run-in at 40 C inlet under the all-core load
        x_axis:
          legend: Time
          unit: h
        y_axis:
          - legend: Inlet
            key: inlet_c
            unit: °C
            aggregations:
              - type: min_c
                unit: °C
                validators:
                  - {operator: ">=", expected_value: 38.0}
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 42.0}
          - legend: Package 0 power
            key: package_0_w
            unit: W
            aggregations:
              - type: max_w
                unit: W
                validators:
                  - {operator: "<=", expected_value: 350.0}
          - legend: Package 1 power
            key: package_1_w
            unit: W
            aggregations:
              - type: max_w
                unit: W
                validators:
                  - {operator: "<=", expected_value: 350.0}
          - legend: Hottest core
            key: hottest_core_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 90.0}
          - legend: Hottest NVMe
            key: hottest_nvme_c
            unit: °C
            aggregations:
              - type: max_c
                unit: °C
                validators:
                  - {operator: "<=", expected_value: 70.0}
      - name: ECC Delta Per DIMM
        key: ecc_delta_per_dimm
        title: Correctable ECC added per DIMM over the run-in
        x_axis:
          legend: DIMM
        y_axis:
          - legend: Correctable added
            key: correctable_delta
            aggregations:
              - type: max
                validators:
                  - {operator: "==", expected_value: 0}
      - name: Media Error Delta Per Drive
        key: media_error_delta_per_drive
        title: NVMe media errors added per drive over the run-in
        x_axis:
          legend: Drive
        y_axis:
          - legend: Media errors added
            key: media_error_delta
            aggregations:
              - type: max
                validators:
                  - {operator: "==", expected_value: 0}
      - name: SEL Entries Added
        key: sel_entries_added
        description: "System event log entries written during the run-in. The log was cleared at setup, so every entry is the run-in's."
        validators:
          - {operator: "==", expected_value: 0}
      - name: Throttle Events
        key: throttle_events
        validators:
          - {operator: "==", expected_value: 0}
      - name: PSU Redundancy Events
        key: psu_redundancy_events
        description: "Redundancy-lost events from either PSU over the run-in."
        validators:
          - {operator: "==", expected_value: 0}

teardown:
  - name: Clear and Ship Config
    key: clear_and_ship_config
    python: phases.clear_and_ship_config
    measurements:
      - name: SEL Entries After Clear
        key: sel_entries_after_clear
        description: "Ships empty, so the first entry the customer reads was written in their rack."
        validators:
          - {operator: "==", expected_value: 0}
      - name: BIOS Checksum
        key: bios_checksum
        description: "Checksum of the BIOS settings after the shipping profile is applied, against the release."
        validators:
          - {operator: "==", expected_value: "sha256:7c3e19b0"}
      - name: Boot Config
        key: boot_config
        description: "Boot order, RAID mode and Secure Boot as one object against the shipping configuration."
        validators:
          - operator: "=="
            expected_value:
              boot_order: [Pxe, Hdd]
              raid: none
              secure_boot: true
```

Framework features to notice:

1. **Previous results as parameters.** `run_in(measurements, server, run, memory_test, nvme_test, log)` names two earlier phases in its signature and receives their results: `memory_test.measurements.ecc_per_dimm.y_axis.correctable` and `nvme_test.measurements.nvme_smart.y_axis.media_errors` are the baselines its deltas are computed from. `depends_on` guarantees they have completed.
2. **An executable phase in the chain.** `nvme_precondition` has no Python: it runs a command, passes on exit code 0, and carries `depends_on`, `timeout: 2h` and a place in the sequence like every other phase. `nvme_test` waits for it.
3. **A whole object, compared once.** `firmware_versions`, `processors`, `nic_link_state` and `boot_config` each validate with one `==`; the report shows which key differed.
4. **`attach.file` for the as-shipped record.** The teardown writes the Redfish inventory to a file and attaches it under the unit's serial, next to the MAC addresses and part serials it writes into `unit.metadata`.
5. **`timeout` sized per phase.** 3 h for the memory patterns, 2 h for the preconditioning, 1 h for the drives, 20 min for the stress, and `24h` on the run-in, which is the framework's cap.
6. **Per-device multi-dimensional measurements.** `pcie_nic_links`, `pcie_nvme_links`, `ecc_per_dimm`, `nvme_performance`, `nvme_smart` and `nic_loopback` carry one point per device and are judged on the worst one through a `min` or `max` aggregation, so the report names the drive or the DIMM.
7. **`run.metadata` for the soak profile.** The run-in's hours and ambient are written on the run, so units soaked at a different profile can be filtered apart on the dashboard.

### BMC and Inventory

The first Redfish transaction, and the highest-yield step in the sequence. The firmware set and the processor description are recorded as objects and compared with the golden configuration in one validator each; the DIMMs, drives and adapters are counted and the slowest DIMM speed is recorded, because one DIMM from a slower reel drags its whole channel down to its speed without any other symptom. The SEL is cleared so every entry the run-in finds is the run-in's.

```python filename="phases/bmc_and_inventory.py"
from utils.recipe import DIMM_CAPACITY_GB, DIMM_COUNT


def bmc_and_inventory(measurements, server, unit, log):
    """Setup: the BMC answers over Redfish, the firmware set is the golden
    one as a single object, and the populated parts are counted. A unit
    that carries one DIMM from the wrong reel or a BIOS one build behind
    the release fails here, before any test that would take an hour to
    say the same thing. The SEL is cleared so the run-in starts empty."""
    reachable = server.redfish_reachable()
    firmware = server.redfish_firmware()
    cpus = server.redfish_processors()
    dimms = server.redfish_memory()
    drives = server.redfish_storage()
    nics = server.redfish_network()
    server.redfish_sel_clear()
    sel = server.redfish_sel_count()

    total_gb = sum(d["capacity_gb"] for d in dimms)
    min_speed = min(d["speed_mt_s"] for d in dimms)
    measurements.bmc_reachable = reachable
    measurements.firmware_versions = firmware
    measurements.processors = cpus
    measurements.dimm_count = len(dimms)
    measurements.memory_total_gb = total_gb
    measurements.dimm_min_speed_mt_s = min_speed
    measurements.nvme_count = len(drives)
    measurements.nic_count = len(nics)
    measurements.sel_entries_after_clear = sel
    unit.metadata["bios"] = firmware["bios"]
    unit.metadata["bmc_fw"] = firmware["bmc"]
    log.info(f"{unit.serial_number}: BIOS {firmware['bios']}, BMC {firmware['bmc']}, CPLD {firmware['cpld']}; {cpus['count']}x {cpus['model']}")
    log.info(f"{len(dimms)}/{DIMM_COUNT} DIMMs, {total_gb} GB at {min_speed} MT/s min (expected {DIMM_COUNT}x{DIMM_CAPACITY_GB} GB); {len(drives)} NVMe, {len(nics)} NICs; SEL cleared, {sel} entries")
```

### PCIe Links

The negotiated link of every NIC and NVMe from the OS after boot. A drive seated at an angle in its bay trains at x2 and passes every functional test at half its bandwidth; a NIC that trained at Gen4 on a Gen5 slot passes the loopback and delivers half its throughput. Both are found only by reading the link, which is why this phase gates the main stage. The AER correctable count is a separate scalar: a retimer that corrects its way through training is a return in six months.

```python filename="phases/pcie_links.py"
def pcie_links(measurements, server, log):
    """Setup: the negotiated PCIe link of every NIC and every NVMe from the
    OS. A drive seated at an angle trains at x2 and passes every
    functional test at half its bandwidth; only the link width says so.
    AER correctable errors after boot must be zero: a retimer or a
    connector that corrects its way through is a field return."""
    nics, nvme = server.os_pcie_links()
    aer = server.os_aer_correctable_errors()

    m = measurements.pcie_nic_links
    m.x_axis = [d["device"] for d in nics]
    m.y_axis.gen = [d["gen"] for d in nics]
    m.y_axis.gen.aggregations.min_gen = min(d["gen"] for d in nics)
    m.y_axis.width = [d["width"] for d in nics]
    m.y_axis.width.aggregations.min_width = min(d["width"] for d in nics)

    m = measurements.pcie_nvme_links
    m.x_axis = [d["device"] for d in nvme]
    m.y_axis.gen = [d["gen"] for d in nvme]
    m.y_axis.gen.aggregations.min_gen = min(d["gen"] for d in nvme)
    m.y_axis.width = [d["width"] for d in nvme]
    m.y_axis.width.aggregations.min_width = min(d["width"] for d in nvme)
    measurements.aer_correctable_errors = aer

    links = ", ".join(f"{d['device']} Gen{d['gen']} x{d['width']}" for d in nics + nvme)
    log.info(f"Links after training: {links}; AER correctable {aer}")
```

### Memory Test

Seven MemTest86 patterns, two passes each, over all 2 TB, then the ECC counters read per DIMM and a STREAM triad at 16 to 128 threads. The bit-error count and the uncorrectable count are zero or the unit stops; the correctable count is recorded per DIMM so the run-in can compute a delta per slot. The triad rises with the thread count until every channel has requests in flight and saturates at the platform's real bandwidth; a channel that trained slow moves the plateau down, and the floor catches it. Two passes of seven patterns take about 90 minutes on the line; the mock is time-scaled and the `timeout` is sized for the real one.

```python filename="phases/memory_test.py"
from utils.recipe import MEMTEST_PASSES, MEMTEST_PATTERNS, STREAM_ARRAY_GB, STREAM_THREADS


def memory_test(measurements, server, log):
    """The MemTest86 pattern set over all 2 TB, then the ECC counters per
    DIMM and a STREAM triad at rising thread counts. Bit errors and
    uncorrectable ECC are zero or the unit stops; correctable ECC is
    recorded per DIMM so a slot that starts correcting during the run-in
    can be named. The triad floor catches a channel that trained slow.
    Two passes of seven patterns take about 90 min on the line; the mock
    is time-scaled."""
    errors = 0
    for pattern in MEMTEST_PATTERNS:
        e = server.os_memtest(pattern, MEMTEST_PASSES)
        errors += e
        log.info(f"{pattern} x{MEMTEST_PASSES}: {e} errors (mock: instant)")
    correctable, uncorrectable = server.os_ecc_counters()
    triad = [server.os_stream_triad(n) for n in STREAM_THREADS]

    measurements.memtest_bit_errors = errors
    m = measurements.ecc_per_dimm
    m.x_axis = list(range(1, len(correctable) + 1))
    m.y_axis.correctable = correctable
    m.y_axis.correctable.aggregations.max = max(correctable)
    m.y_axis.uncorrectable = uncorrectable
    m.y_axis.uncorrectable.aggregations.max = max(uncorrectable)
    measurements.correctable_ecc_total = sum(correctable)
    s = measurements.stream_bandwidth
    s.x_axis = STREAM_THREADS
    s.y_axis.triad = triad
    s.y_axis.triad.aggregations.peak_gb_s = max(triad)
    log.info(f"ECC: {sum(correctable)} correctable, {sum(uncorrectable)} uncorrectable over {len(correctable)} DIMMs; STREAM triad on {STREAM_ARRAY_GB:.0f} GB arrays peaks at {max(triad):.0f} GB/s with {STREAM_THREADS[triad.index(max(triad))]} threads")
```

![STREAM triad bandwidth against thread count: 234 GB/s at 16 threads rising to 457 GB/s at 128 threads, above the 400 GB/s floor and under the 614 GB/s theoretical line of 16 DDR5-4800 channels.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/server-l10-chart-stream-xlyH9dWkvciNYh1rpAvkj5QsX26g5f.png)

The mock's triad: 234 GB/s at 16 threads, 432 GB/s at 64, a plateau at 457 GB/s from 96 threads on, 74 % of the theoretical 614 GB/s. A server with one channel in a degraded mode plateaus near 400 GB/s and fails on the peak.

### NVMe Precondition

An executable phase, not a Python one. SNIA PTS-E requires a purge and then twice the drive's capacity written in 128 KiB sequential blocks before any steady-state number means anything, and that is a `fio` command line, not a measurement. The phase runs the command, passes on exit code 0, and fails on anything else: a drive that fails to open or reports a write error makes fio exit non-zero. On the line the command is `fio --name=precondition --filename=/dev/nvme1n1:... --rw=write --bs=128k --loops=2 --output-format=json`, about 45 minutes for eight 7.68 TB drives in parallel; the template runs a stand-in that prints the same JSON shape:

```python filename="scripts/fio_precondition.py"
"""Stand-in for the SNIA PTS-E preconditioning command, run by the
executable phase `nvme_precondition`:

    fio --name=precondition --filename=/dev/nvme1n1:/dev/nvme2n1:... \
        --rw=write --bs=128k --ioengine=io_uring --iodepth=32 --direct=1 \
        --loops=2 --output-format=json

The real command writes twice the capacity of every drive in 128 KiB
sequential blocks after a purge, about 45 minutes for eight 7.68 TB
drives in parallel. This script prints the same JSON shape fio prints
and exits 0; a drive that fails to open or reports a write error makes
fio exit non-zero, which fails the phase."""

import argparse
import json
import sys


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--drives", type=int, default=8)
    parser.add_argument("--capacity-tb", type=float, default=7.68)
    parser.add_argument("--loops", type=int, default=2)
    parser.add_argument("--output-format", default="json")
    args = parser.parse_args()

    written_bytes = int(args.capacity_tb * 1e12 * args.loops)
    bw_kib_s = 6_400_000  # 6.4 GB/s of 128 KiB sequential writes per drive
    runtime_ms = int(written_bytes / (bw_kib_s * 1024) * 1000)
    jobs = [
        {
            "jobname": f"precondition-nvme{i + 1}",
            "error": 0,
            "write": {"io_bytes": written_bytes, "bw": bw_kib_s, "iops": bw_kib_s / 128, "runtime": runtime_ms},
        }
        for i in range(args.drives)
    ]
    report = {"fio version": "fio-3.37 (mock)", "global options": {"rw": "write", "bs": "128k", "loops": str(args.loops)}, "jobs": jobs}
    json.dump(report, sys.stdout, indent=2)
    print()
    sys.exit(0 if all(j["error"] == 0 for j in jobs) else 1)


if __name__ == "__main__":
    main()
```

### NVMe Test

Every drive at steady state: five 60 s rounds of 4 KiB random read and of 128 KiB sequential read, the mean of the rounds recorded per drive and the slowest drive judged against 80 % of the datasheet. Then SMART per drive: media errors, percentage used and the PLP capacitor health from the OCP extended log, each judged on the worst drive. A drive that reports 3 % used at L10 has a history the customer did not buy; a capacitor health under 100 % on a new drive is a power-loss-protection bank that will not hold the write cache through a PSU event.

```python filename="phases/nvme_test.py"
from utils.recipe import NVME_COUNT, NVME_ROUND_S, NVME_STEADY_STATE_ROUNDS


def nvme_test(measurements, server, log):
    """After the executable preconditioning, every drive at steady state:
    4 KiB random read IOPS and 128 KiB sequential read MB/s as the mean of
    the PTS-E rounds, judged on the slowest drive against 80 % of the
    datasheet. Then SMART: media errors, percentage used and the PLP
    capacitor health from the OCP extended log, per drive. Five 60 s
    rounds on eight drives are 40 min on the line; the mock is
    time-scaled."""
    drives = list(range(1, NVME_COUNT + 1))
    iops, mb_s, media, used, cap, temp = [], [], [], [], [], []
    for d in drives:
        i, m = server.os_fio_steady_state(d, NVME_STEADY_STATE_ROUNDS, NVME_ROUND_S)
        smart = server.os_nvme_smart(d)
        iops.append(i)
        mb_s.append(m)
        media.append(smart["media_errors"])
        used.append(smart["percentage_used"])
        cap.append(smart["capacitor_health_pct"])
        temp.append(smart["temperature_c"])
        log.info(f"nvme{d}: {i / 1e6:.2f} M IOPS, {m:.0f} MB/s; SMART media errors {smart['media_errors']}, used {smart['percentage_used']} %, capacitor {smart['capacitor_health_pct']} %, {smart['temperature_c']:.0f} C")

    p = measurements.nvme_performance
    p.x_axis = drives
    p.y_axis.rand_read_4k_iops = iops
    p.y_axis.rand_read_4k_iops.aggregations.min = min(iops)
    p.y_axis.seq_read_128k_mb_s = mb_s
    p.y_axis.seq_read_128k_mb_s.aggregations.min = min(mb_s)
    s = measurements.nvme_smart
    s.x_axis = drives
    s.y_axis.media_errors = media
    s.y_axis.media_errors.aggregations.max = max(media)
    s.y_axis.percentage_used = used
    s.y_axis.percentage_used.aggregations.max = max(used)
    s.y_axis.capacitor_health_pct = cap
    s.y_axis.capacitor_health_pct.aggregations.min = min(cap)
    measurements.media_errors_total = sum(media)
    log.info(f"Slowest drive: {min(iops) / 1e6:.2f} M IOPS, {min(mb_s):.0f} MB/s; {sum(media)} media errors over {NVME_COUNT} drives")
```

![fio per drive at steady state: eight bars of 4 KiB random read between 2.24 and 2.29 million IOPS over the 2.0 million floor, and eight bars of 128 KiB sequential read between 12.9 and 13.1 GB/s over the 11.2 GB/s floor, with the datasheet values as dotted lines.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/server-l10-chart-nvme-o0AGrZkHelv2bvmhNSrSwcWX3Ze2tr.png)

The mock's eight drives at steady state: 2.24 to 2.29 M IOPS and 12.9 to 13.1 GB/s, all within 3 % of each other, over floors at 2.0 M IOPS and 11.2 GB/s. A drive on a x2 link shows up in this chart at half the sequential number, which is why the link is checked first: the two faults are told apart by which phase failed.

### NIC Loopback

Both QSFP28 ports with a loopback plug, line-rate frames for 60 s. Every frame must return, the FCS counter must stay at zero, and the link must have negotiated 100 GbE full duplex with RS-FEC, recorded as one object for both ports. A port that linked at 40G because one lane of the cage is dirty, or drops one frame in ten million from a marginal retimer, is found here and not in the customer's rack. The frame count is a multi-dimensional measurement per port with a floor, so a run that ended early is also caught.

```python filename="phases/nic_loopback.py"
from utils.recipe import NIC_COUNT, NIC_LOOPBACK_S


def nic_loopback(measurements, server, log):
    """Both QSFP28 ports with a fibre loopback plug: line-rate frames for
    60 s, every frame must come back, and the link must have negotiated
    100 GbE full duplex with RS-FEC. A port that links at 40G because one
    lane of the cage is dirty, or drops one frame in ten million from a
    marginal retimer, is found here and not in the customer's rack."""
    ports = list(range(1, NIC_COUNT + 1))
    sent, lost, fcs, link = [], [], [], {}
    for p in ports:
        r = server.os_nic_loopback(p, NIC_LOOPBACK_S)
        sent.append(r["frames_sent"])
        lost.append(r["frames_lost"])
        fcs.append(r["fcs_errors"])
        link[f"port{p}"] = {"speed_mb_s": r["speed_mb_s"], "duplex": r["duplex"], "fec": r["fec"]}
        log.info(f"port{p}: {r['frames_sent']:,} frames in {NIC_LOOPBACK_S} s, {r['frames_lost']} lost, {r['fcs_errors']} FCS errors, {r['speed_mb_s'] // 1000} GbE {r['duplex']} {r['fec']} (mock: instant)")

    m = measurements.nic_loopback
    m.x_axis = ports
    m.y_axis.frames_sent = sent
    m.y_axis.frames_sent.aggregations.min = min(sent)
    m.y_axis.frames_lost = lost
    m.y_axis.frames_lost.aggregations.max = max(lost)
    m.y_axis.fcs_errors = fcs
    m.y_axis.fcs_errors.aggregations.max = max(fcs)
    measurements.nic_link_state = link
```

### CPU Stress and Power

All cores at 100 % for 10 minutes at the room's 25 °C, sampled from the BMC every 5 s. Each socket's package power must reach the power limit in the second half of the run and not exceed the TDP the shipping profile sets as PL1; the hottest core must stay under 85 °C; the fan controller must end with headroom; and the throttle counters must stay at zero. A heatsink with a missing thermal pad shows as one socket hot and its package power capped under the other's; a socket that never reaches 300 W exercised fewer cores than it should, or has a power cap the profile did not set.

```python filename="phases/cpu_stress_and_power.py"
import numpy as np

from utils.recipe import CPU_STRESS_S, CPU_STRESS_SAMPLE_S, CPU_TDP_W


def cpu_stress_and_power(measurements, server, log):
    """All cores at 100 % for 10 min at the room's 25 C: package power per
    socket must reach the power limit and not exceed the TDP, the hottest
    core must stay under 85 C, the fan controller must end with headroom,
    and nothing may throttle. A heatsink with a missing thermal pad shows
    up as one socket hot and its package power capped under the other.
    The mock is time-scaled."""
    t, pkg0, pkg1, temp, fan, throttles = server.os_stress_all_cores(CPU_STRESS_S, CPU_STRESS_SAMPLE_S)
    settled = np.asarray(t) >= CPU_STRESS_S / 2  # the second half, once the heatsinks are hot
    p0 = np.asarray(pkg0)
    p1 = np.asarray(pkg1)

    m = measurements.cpu_stress
    m.x_axis = t
    m.y_axis.package_0_w = pkg0
    m.y_axis.package_0_w.aggregations.max_w = round(float(p0.max()), 1)
    m.y_axis.package_0_w.aggregations.settled_mean_w = round(float(p0[settled].mean()), 1)
    m.y_axis.package_1_w = pkg1
    m.y_axis.package_1_w.aggregations.max_w = round(float(p1.max()), 1)
    m.y_axis.package_1_w.aggregations.settled_mean_w = round(float(p1[settled].mean()), 1)
    m.y_axis.hottest_core_c = temp
    m.y_axis.hottest_core_c.aggregations.max_c = max(temp)
    m.y_axis.fan_duty_pct = fan
    m.y_axis.fan_duty_pct.aggregations.end_pct = fan[-1]
    measurements.throttle_events = throttles
    log.info(f"{CPU_STRESS_S} s all-core (mock: instant): package {p0[settled].mean():.0f} / {p1[settled].mean():.0f} W settled against a {CPU_TDP_W:.0f} W TDP, hottest core {max(temp):.1f} C, fans at {fan[-1]:.0f} % at the end, {throttles} throttle events")
```

![All cores at 100 percent for 10 min: package power stepping from 62 W idle to 335 and 338 W settled under the 350 W TDP line and over the 300 W floor, the hottest core rising from 41 to 72.9 C, and the fan duty rising from 34 to 65 percent.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/template/server-l10-chart-cpu-stress-U8By3QiHb4tjnAoI6x94JKTtiqRcL8.png)

The mock's stress: both packages step from 62 W idle to 335 and 338 W and stay there, the hottest core climbs from 41 °C to 72.9 °C with a 90 s time constant, and the fans settle at 65 % duty. The settled window for the mean is the second half, once the heatsinks are hot.

### Run-In

Twenty-four hours in the chamber at 40 °C inlet under the all-core load, the trace recorded whole and judged on the inlet window, the package power, the hottest core and the hottest drive. Then the counters that must not have moved, compared with what the earlier phases recorded: the phase receives `memory_test` and `nvme_test` as parameters and computes the correctable ECC delta per DIMM and the media-error delta per drive from their measurements, then reads the SEL, throttle and PSU redundancy counters. The mock is time-scaled; `timeout: 24h` is the framework's cap, and a line that needs the full 24 h with margin chains two 12 h phases, each reading the counters at its end.

```python filename="phases/run_in.py"
from utils.recipe import CPU_TDP_W, RUN_IN_AMBIENT_C, RUN_IN_HOURS


def run_in(measurements, server, run, memory_test, nvme_test, log):
    """24 h in the chamber at 40 C inlet under the all-core load, the trace
    recorded whole, then every counter compared with what memory_test and
    nvme_test recorded: correctable ECC per DIMM, media errors per drive,
    SEL entries, throttle and PSU redundancy events. The deltas are the
    result; the baselines come from the earlier phases through the
    framework, not from a file. The mock is time-scaled."""
    run.metadata["run_in_hours"] = RUN_IN_HOURS
    run.metadata["run_in_ambient_c"] = RUN_IN_AMBIENT_C
    log.info(f"Run-in: {RUN_IN_HOURS:.0f} h at {RUN_IN_AMBIENT_C:.0f} C inlet under the all-core load (mock: instant)")
    r = server.run_in(RUN_IN_HOURS, RUN_IN_AMBIENT_C)

    m = measurements.run_in_trace
    m.x_axis = r["t_h"]
    m.y_axis.inlet_c = r["inlet_c"]
    m.y_axis.inlet_c.aggregations.min_c = min(r["inlet_c"])
    m.y_axis.inlet_c.aggregations.max_c = max(r["inlet_c"])
    m.y_axis.package_0_w = r["pkg0_w"]
    m.y_axis.package_0_w.aggregations.max_w = max(r["pkg0_w"])
    m.y_axis.package_1_w = r["pkg1_w"]
    m.y_axis.package_1_w.aggregations.max_w = max(r["pkg1_w"])
    m.y_axis.hottest_core_c = r["cpu_max_c"]
    m.y_axis.hottest_core_c.aggregations.max_c = max(r["cpu_max_c"])
    m.y_axis.hottest_nvme_c = r["nvme_max_c"]
    m.y_axis.hottest_nvme_c.aggregations.max_c = max(r["nvme_max_c"])

    ecc_before = list(memory_test.measurements.ecc_per_dimm.y_axis.correctable)
    ecc_after, _ = server.os_ecc_counters()
    ecc_delta = [a - b for a, b in zip(ecc_after, ecc_before)]
    d = measurements.ecc_delta_per_dimm
    d.x_axis = list(range(1, len(ecc_delta) + 1))
    d.y_axis.correctable_delta = ecc_delta
    d.y_axis.correctable_delta.aggregations.max = max(ecc_delta)

    media_before = list(nvme_test.measurements.nvme_smart.y_axis.media_errors)
    media_after = [server.os_nvme_smart(i + 1)["media_errors"] for i in range(len(media_before))]
    media_delta = [a - b for a, b in zip(media_after, media_before)]
    n = measurements.media_error_delta_per_drive
    n.x_axis = list(range(1, len(media_delta) + 1))
    n.y_axis.media_error_delta = media_delta
    n.y_axis.media_error_delta.aggregations.max = max(media_delta)

    measurements.sel_entries_added = r["sel_added"]
    measurements.throttle_events = r["throttle_events"]
    measurements.psu_redundancy_events = r["psu_redundancy_events"]
    log.info(f"Inlet {min(r['inlet_c']):.1f} to {max(r['inlet_c']):.1f} C, package max {max(r['pkg0_w']):.0f} / {max(r['pkg1_w']):.0f} W against {CPU_TDP_W:.0f} W, hottest core {max(r['cpu_max_c']):.1f} C, hottest NVMe {max(r['nvme_max_c']):.1f} C")
    log.info(f"Deltas over {RUN_IN_HOURS:.0f} h: ECC {sum(ecc_delta)} (baseline {sum(ecc_before)}), media errors {sum(media_delta)} (baseline {sum(media_before)}), SEL +{r['sel_added']}, throttles {r['throttle_events']}, PSU events {r['psu_redundancy_events']}")
```

The deltas are the result. A DIMM that corrected nothing during the pattern test and 12 errors during the soak is a DIMM that fails with temperature, and its slot is named in the report; the same DIMM read as a total over 32 slots is a number nobody can act on.

### Clear and Ship Config

The teardown clears the SEL the run-in wrote into, applies the shipping BIOS profile and checks its checksum and its boot configuration against the release, writes the MAC addresses and the serials of every DIMM and drive into the unit's metadata, and attaches the Redfish inventory as the as-shipped record. It runs whatever a main phase did, so a unit pulled for repair is never left with the test profile in its BIOS.

```python filename="phases/clear_and_ship_config.py"
import json
import tempfile

from utils.recipe import SHIPPING_BIOS_PROFILE


def clear_and_ship_config(measurements, server, unit, attach, log):
    """Teardown: the SEL the run-in wrote into is cleared, the BIOS goes
    to the shipping profile and its checksum and boot configuration are
    checked, the MAC addresses and the serials of every DIMM and drive go
    into the unit's metadata, and the whole Redfish inventory is attached
    as the as-shipped record. Runs whatever a main phase did, so a unit
    pulled for repair is never left with the test profile in its BIOS."""
    server.redfish_sel_clear()
    server.redfish_bios_apply_profile(SHIPPING_BIOS_PROFILE)
    checksum = server.redfish_bios_checksum()
    boot = server.redfish_boot_config()
    inventory = server.redfish_inventory()

    measurements.sel_entries_after_clear = server.redfish_sel_count()
    measurements.bios_checksum = checksum
    measurements.boot_config = boot
    for nic in inventory["network"]:
        unit.metadata[f"mac_port{nic['port']}"] = nic["mac"]
    unit.metadata["dimm_serials"] = ",".join(d["serial"] for d in inventory["memory"])
    unit.metadata["nvme_serials"] = ",".join(d["serial"] for d in inventory["storage"])

    with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f:
        json.dump(inventory, f, indent=2)
    attach.file(f.name, f"redfish-inventory-{unit.serial_number}.json")
    log.info(f"SEL cleared, BIOS {SHIPPING_BIOS_PROFILE} {checksum}, boot {boot['boot_order']} RAID {boot['raid']}, {len(inventory['memory'])} DIMM and {len(inventory['storage'])} NVMe serials in unit metadata, inventory attached")
```

### Mock Plug

`ServerUnderTest` stands in for the BMC's Redfish session and the test OS's SSH session because the run-in reads both at once, and plugs run in separate processes. Its firmware set is the golden one, its 32 DIMMs report 64 GB at 4800 MT/s, its links train at Gen5 x16 and x4, its STREAM triad saturates at 456 GB/s with a 22-thread time constant, its drives deliver 2.24 to 2.29 M IOPS and 12.9 to 13.1 GB/s with capacitor health between 104 and 111 %, its all-core load draws 335 and 338 W with a 90 s thermal time constant, and its chamber holds 40 °C with a 6 h drift of ±0.3 °C. Every method returns plain Python types because plug calls cross a JSON boundary; a measurement read back from `measurements.<key>` returns a proxy, so the phases keep locals for their log lines.

On a real line, split the class into the Redfish client and the SSH session, replace `scripts/fio_precondition.py` in the `nvme_precondition` command with the real `fio` invocation, drive the chamber from a plug of its own with `scope: station` if it is shared between stations, and read the throttle counters from `turbostat` or the BMC rather than a polled register. The phases, measurements and limits stay the same.


## Changelog

### Improve chart readability and date filtering
URL: https://www.tofupilot.com/changelog/improve-chart-readability-and-date-filtering

Charts lacked axis labels and there was no easy way to filter data by date range.

- Added x-axis labels to control chart and histogram
- Added calendar and date range picker for filtering data
- Moved search params to page header for cleaner layout

### Improve dashboard stability
URL: https://www.tofupilot.com/changelog/improve-dashboard-stability

Several UI issues were surfaced during customer demos.

- Fixed multiple front-end errors affecting dashboard interactions
- Improved overall page stability

### Track measurements at the phase level
URL: https://www.tofupilot.com/changelog/track-measurements-at-the-phase-level

Until now, measurements were only available at the run level. Phase-level tracking gives you finer-grained visibility into each step of a test.

- Added phase-level measurements replacing the legacy report view
- Added automatic extraction of part number and revision from serial numbers

### Manage test stations from the dashboard
URL: https://www.tofupilot.com/changelog/manage-test-stations-from-the-dashboard

Teams needed a central place to create, organize, and monitor their test stations.

- Added full station management: create, update, delete, and archive
- Added station activity tracker showing current runs per station

### Explore the API with built-in documentation
URL: https://www.tofupilot.com/changelog/explore-the-api-with-built-in-documentation

Integrating with TofuPilot is easier when you can browse the API directly from the app.

- Added interactive API documentation with auto-generated schema
- Added API call tracking so you can monitor usage and debug integrations

### Get help without leaving the app
URL: https://www.tofupilot.com/changelog/get-help-without-leaving-the-app

Reaching support required navigating to an external page. Now you can get help directly from the dashboard.

- Added in-app support dialog with streamlined help request form
- Added quick links to API documentation and resources

### Analyze multi-dimensional measurements
URL: https://www.tofupilot.com/changelog/analyze-multi-dimensional-measurements

Some tests produce measurements across multiple dimensions. You can now capture and visualize all of them.

- Added support for multi-dimensional measurements
- Added temporal graphs for phases and improved Cpk charts
- Improved control chart display and measurement filtering

### Upgrade to Pro with integrated billing
URL: https://www.tofupilot.com/changelog/upgrade-to-pro-with-integrated-billing

Teams can now unlock advanced features by upgrading directly from the dashboard.

- Added Pro subscription with integrated billing
- Added subscription management with upgrade and downgrade flows

### Monitor tests in real time with the operator UI
URL: https://www.tofupilot.com/changelog/monitor-tests-in-real-time-with-the-operator-ui

Operators on the factory floor need live visibility into running tests without switching between tools.

- Added real-time operator UI with live data streaming
- Added operator UI support in Python scripts
- Improved connection reliability and API navigation

### Navigate the dashboard with a redesigned layout
URL: https://www.tofupilot.com/changelog/navigate-the-dashboard-with-a-redesigned-layout

The app layout has been redesigned for faster navigation and a cleaner experience.

- Redesigned app layout with improved navigation structure
- Redesigned procedure cards and subscription badge
- Reorganized settings pages with cleaner URLs

### Sign in with Microsoft, Google, and GitHub
URL: https://www.tofupilot.com/changelog/sign-in-with-microsoft-google-and-github

Users signing in with enterprise identity providers experienced login issues. All sign-in methods now work reliably.

- Fixed Microsoft Entra ID and Google OAuth login flows
- Added account linking for multiple auth providers

### Deploy TofuPilot on your own infrastructure
URL: https://www.tofupilot.com/changelog/deploy-tofupilot-on-your-own-infrastructure

Enterprise teams with strict data residency requirements can now run TofuPilot on-premise.

- Added self-hosting deployment with Docker
- Added configuration for on-premise instances

### Visualize measurement trends over time
URL: https://www.tofupilot.com/changelog/visualize-measurement-trends-over-time

Understanding how measurements evolve across test runs helps catch regressions early.

- Added temporal graph visualization for phase measurements over time
- Improved Cpk chart calculations and multi-unit comparison views

### See live test activity per station
URL: https://www.tofupilot.com/changelog/see-live-test-activity-per-station

Teams wanted to know which tests are running on each station right now.

- Added station activity page showing current and past test runs
- Added real-time current runs display per station

### Load insights page faster
URL: https://www.tofupilot.com/changelog/load-insights-page-faster

The insights page was slow to load for teams with large datasets.

- Improved page load speed on the insights page
- Fixed cycle time chart and batch filter issues

### Browse runs with infinite scroll and logs
URL: https://www.tofupilot.com/changelog/browse-runs-with-infinite-scroll-and-logs

Navigating large lists of test runs was slow, and there was no way to explore logs. This update makes both much easier.

- Added infinite scroll pagination on the Runs page
- Added log explorer with filtering, sorting, and color-coded levels
- Added log support for Python vanilla scripts

### Add phases to existing runs via the Python SDK
URL: https://www.tofupilot.com/changelog/add-phases-to-existing-runs-via-the-python-sdk

You can now append phase data to runs that were already uploaded, making incremental test reporting easier.

- Added API endpoint to add phases to existing runs
- Added automatic testing of Python scripts before releases

### Retrieve run attachments by serial number
URL: https://www.tofupilot.com/changelog/retrieve-run-attachments-by-serial-number

When querying runs by serial number, attachment data was missing from the response.

- Added attachment data to the get runs by serial number endpoint
- Fixed response format to match documentation

### Launch the documentation site
URL: https://www.tofupilot.com/changelog/launch-the-documentation-site

All TofuPilot documentation is now in one place with full-text search and auto-generated API reference.

- Added documentation site with markdown support and full-text search
- Added auto-generated API documentation

### Integrate faster with API v2
URL: https://www.tofupilot.com/changelog/integrate-faster-with-api-v2

The new API version simplifies integrations with cleaner endpoint paths and a consistent response format.

- Added API v2 with new endpoint paths and response format
- Simplified runs list endpoint for cleaner integrations

### Load Cpk charts faster on large datasets
URL: https://www.tofupilot.com/changelog/load-cpk-charts-faster-on-large-datasets

Cpk charts were slow to render for organizations with thousands of measurements.

- Improved Cpk chart computation performance
- Removed redundant calculations when all values are identical
- Added sorted values in Cpk chart filter

### Keep using get_runs with API v1 compatibility
URL: https://www.tofupilot.com/changelog/keep-using-get-runs-with-api-v1-compatibility

Teams using the v1 API needed continued access to the get_runs method during migration.

- Restored legacy get_runs method in Python client for v1 API compatibility

### Improve date display and image handling
URL: https://www.tofupilot.com/changelog/improve-date-display-and-image-handling

Date formatting was inconsistent across the app, and some images failed to load in API responses.

- Unified date formatting and display across the app
- Fixed date range picker behavior for all range options
- Improved image handling in API responses

### Fix revision settings page
URL: https://www.tofupilot.com/changelog/fix-revision-settings-page

Opening revision settings returned errors for some users.

- Fixed 404 error when opening revision settings
- Fixed permission errors when accessing revisions via API

### Get more detail from control charts
URL: https://www.tofupilot.com/changelog/get-more-detail-from-control-charts

Control charts now show richer information on hover, making it easier to investigate individual data points.

- Added detailed tooltips showing unit and run data on control charts
- Improved error states with contextual messaging

### Discover TofuPilot with new landing page
URL: https://www.tofupilot.com/changelog/discover-tofupilot-with-new-landing-page

The landing page and website have been refreshed to better showcase what TofuPilot does.

- Redesigned landing page with dark mode support
- Added SEO metadata across docs and web apps
- Updated about page and navigation

### List units programmatically with get_units
URL: https://www.tofupilot.com/changelog/list-units-programmatically-with-get-units

You can now retrieve your unit inventory directly from your Python scripts.

- Added get_units endpoint to the Python SDK

### Fix GitLab repository picker
URL: https://www.tofupilot.com/changelog/fix-gitlab-repository-picker

Fixed an issue where linking a GitLab repository could fail with a generic "Failed to fetch metadata" error.

### Guide operators with checklists
URL: https://www.tofupilot.com/changelog/guide-operators-with-checklists

Station procedures can now include visual checklists and require operator input at specific steps.

- Added image choice and image checklist components for station procedures
- Added requires_input toggle for operator prompts during test execution

### Distribute self-hosted via Docker
URL: https://www.tofupilot.com/changelog/distribute-self-hosted-via-docker

Self-hosted customers now receive Docker images gated by their license, simplifying deployment and updates.

- Added license-gated Docker image access for self-hosted customers
- Added automated Docker image builds for dev and production releases

### Fix part filtering on Batches and Units pages
URL: https://www.tofupilot.com/changelog/fix-part-filtering-on-batches-and-units-pages

Filtering by part on the Batches and Units pages caused errors for some organizations.

- Fixed errors when filtering by part on Batches and Units pages
- Fixed permission-related errors affecting filtered views

### Improve login reliability
URL: https://www.tofupilot.com/changelog/improve-login-reliability

Some users experienced authentication issues preventing them from signing in.

- Fixed authentication errors affecting specific user accounts
- Improved user tracking for newly signed up users

### Show accurate API error messages
URL: https://www.tofupilot.com/changelog/show-accurate-api-error-messages

API error responses sometimes showed unrelated messages, making debugging harder than it needed to be.

- Fixed API error messages to show the actual error
- Improved error clarity across all endpoints

### Speed up run creation by 17%
URL: https://www.tofupilot.com/changelog/speed-up-run-creation-by-17

We migrated to a faster database engine, resulting in faster test uploads and better reliability under load.

- Improved run creation speed by 17%
- Completed migration with zero data loss and zero downtime

### Strengthen security across all services
URL: https://www.tofupilot.com/changelog/strengthen-security-across-all-services

We performed a comprehensive security review and patched all identified issues.

- Updated all dependencies to latest secure versions
- Hardened security configuration across all services

### Share filtered views via URL
URL: https://www.tofupilot.com/changelog/share-filtered-views-via-url

You can now bookmark or share a filtered page — the filters are preserved in the URL.

- Pages now load with correct filtered data from URL search params
- Fixed filters to always refresh data when changed

### Control test execution with phase dependencies
URL: https://www.tofupilot.com/changelog/control-test-execution-with-phase-dependencies

Complex test procedures need phases to run in a specific order with retries and timeouts. Studio now supports all of that.

- Added phase dependencies, setup/teardown nesting, timeouts, and retries
- Added operator UI with real-time component rendering
- Improved execution bar with scope selectors and kill hotkey

### Run Studio reliably on Windows and Linux
URL: https://www.tofupilot.com/changelog/run-studio-reliably-on-windows-and-linux

Several platform-specific issues were preventing smooth test execution on Windows and Linux.

- Fixed console windows appearing during test execution on Windows
- Fixed Python environment initialization on Linux
- Fixed deep links on Windows desktop builds

### Get started faster with automatic Python setup
URL: https://www.tofupilot.com/changelog/get-started-faster-with-automatic-python-setup

Setting up a Python environment was a manual and error-prone step. Studio now handles it automatically.

- Switched to automatic Python environment management
- Added auto-created project files for new projects
- Added signed Linux packages for secure distribution

### Run Studio on ARM64 Linux devices
URL: https://www.tofupilot.com/changelog/run-studio-on-arm64-linux-devices

Teams deploying on edge devices like NVIDIA Jetson Nano can now run Studio natively.

- Added ARM64 Linux build target
- Expanded platform coverage for edge deployment scenarios

### Choose which version to deploy
URL: https://www.tofupilot.com/changelog/choose-which-version-to-deploy

Self-hosted customers previously always got the latest version. You can now pin a specific version during deployment.

- Added version selector to the deploy script
- Users can now choose specific versions instead of always installing the latest

### Upgrade from v1 to v2 safely
URL: https://www.tofupilot.com/changelog/upgrade-from-v1-to-v2-safely

Upgrading between major versions can be risky. The new upgrade script handles the migration with built-in safety checks.

- Added upgrade script migrating from v1 to v2 database and storage
- Added safety features to prevent data loss during migration
- Added partial upgrade recovery for failed attempts

### Run OpenHTF tests with real-time operator UI
URL: https://www.tofupilot.com/changelog/run-openhtf-tests-with-real-time-operator-ui

OpenHTF users can now execute tests through Studio with live operator feedback and automatic dashboard uploads.

- Added OpenHTF test runner integration with real-time operator UI
- Connected Studio execution engine with dashboard upload pipeline

### Browse Framework and API docs in one place
URL: https://www.tofupilot.com/changelog/browse-framework-and-api-docs-in-one-place

All documentation — Framework guides, API reference, and examples — now lives on a single site.

- Added docs site with landing page and support resources
- Added Framework documentation with structure, phases, and plugs
- Improved operator UI and plug documentation with examples

### Learn Framework faster with better docs
URL: https://www.tofupilot.com/changelog/learn-framework-faster-with-better-docs

The Framework docs have been restructured to help you get productive faster.

- Restructured docs with environments and execution sections
- Added download cards for all platforms
- Updated template covers and configuration

### Sync procedures from Git, localize Studio
URL: https://www.tofupilot.com/changelog/sync-procedures-from-git-localize-studio

Teams storing procedures in Git can now sync them directly into Studio. The entire UI is also now translatable.

- Added GitHub/GitLab repository sync for cloning repos locally
- Completed i18n migration of all UI strings and error messages
- Renamed "runs" to "reports" with improved report UI

### Launch Studio with editor and execution
URL: https://www.tofupilot.com/changelog/launch-studio-with-editor-and-execution

Studio is a new desktop app for creating and running test procedures. Define your phases in YAML, execute them, and stream results in real time.

- Added procedure editor with phase configuration
- Added test execution with real-time log streaming
- Available on macOS with signed and notarized builds

### Aggregate data and validate test results
URL: https://www.tofupilot.com/changelog/aggregate-data-and-validate-test-results

New chart views help you spot trends across runs, and validators let you define pass/fail criteria for measurements.

- Added aggregation and validator views to the dashboard
- Fixed phase timestamp display for legacy data
- Added string validator compatibility for migrated stations

### Fix charts crashing on large datasets
URL: https://www.tofupilot.com/changelog/fix-charts-crashing-on-large-datasets

Organizations with many parts experienced chart crashes when loading histograms or filtering.

- Fixed histogram and sidebar filters crashing on large datasets
- Added smart limits to chart series and lazy rendering for filter panels

### Connect GitLab repositories for test uploads
URL: https://www.tofupilot.com/changelog/connect-gitlab-repositories-for-test-uploads

Teams using GitLab can now connect their repositories to TofuPilot, just like GitHub users.

- Added GitLab as a supported Git provider alongside GitHub
- Enabled seamless token refresh for GitLab connections

### Deploy and upgrade self-hosted instances faster
URL: https://www.tofupilot.com/changelog/deploy-and-upgrade-self-hosted-instances-faster

Repeated deployments are now faster and more reliable, with better error handling throughout.

- Sped up repeated deploys by skipping already-completed steps
- Fixed authentication secret generation and environment variable handling
- Added clear error messages and fail-fast on migration errors

### Manage self-hosted versions with v0.1.1
URL: https://www.tofupilot.com/changelog/manage-self-hosted-versions-with-v011

The deploy script now shows your current version and detects upgrades or downgrades automatically.

- Added version display with upgrade/downgrade detection in deploy script
- Added automatic cleanup of old Docker images after successful deploy
- Fixed version listing with authenticated registry access

### Run tests locally or upload to the cloud
URL: https://www.tofupilot.com/changelog/run-tests-locally-or-upload-to-the-cloud

Studio now supports two modes: desktop mode for local-only testing (no API key needed) and cloud mode for uploading results to your dashboard.

- Added desktop mode and cloud mode for test execution
- Improved error reporting when test scripts fail

### Use pytest and Robot Framework in Studio
URL: https://www.tofupilot.com/changelog/use-pytest-and-robot-framework-in-studio

Studio now supports pytest and Robot Framework alongside OpenHTF, so you can use your preferred test framework.

- Added pytest and Robot Framework as test framework connectors
- Enabled real-time test status, phase tracking, and log streaming

### Download Studio for Apple Silicon and ARM Linux
URL: https://www.tofupilot.com/changelog/download-studio-for-apple-silicon-and-arm-linux

Studio is now available as native ARM64 builds for Apple Silicon Macs and ARM-based Linux devices.

- Added ARM64 builds for Apple Silicon and ARM-based Linux devices
- Updated download page with new platform options

### Sort unit lists 12x faster
URL: https://www.tofupilot.com/changelog/sort-unit-lists-12x-faster

Sorting large unit lists was slow for teams with thousands of units, causing noticeable delays when navigating.

- Reduced unit list sort time from 518ms to 43ms
- Biggest improvement for organizations with 15,000+ units

### Set up self-hosted for air-gapped envs
URL: https://www.tofupilot.com/changelog/set-up-self-hosted-for-air-gapped-envs

Self-hosted installations now work in restricted environments without internet access or SMTP configuration.

- Added Azure Government cloud authentication support
- Made analytics and email sending optional for air-gapped environments
- Added "Share invite link" dialog when SMTP is not configured

### Control access with teams and roles
URL: https://www.tofupilot.com/changelog/control-access-with-teams-and-roles

Growing teams need fine-grained control over who can do what. You can now assign roles and restrict access per team.

- Added team-based access control with admin, member, and viewer roles
- Added support for email, Google, GitHub, and Microsoft Entra ID sign-in
- Added station API key authentication for automated test uploads

### Improve mobile layout and team creation
URL: https://www.tofupilot.com/changelog/improve-mobile-layout-and-team-creation

The procedures list is now responsive on mobile, and team creation respects your subscription tier.

- Fixed team creation button to respect subscription tier
- Made procedures list responsive for mobile
- Improved page load speed across the app

### Upgrade to Python SDK v2.0
URL: https://www.tofupilot.com/changelog/upgrade-to-python-sdk-v20

The Python SDK v2.0 aligns with the new API v2, bringing a simpler authentication flow and updated endpoint paths.

- Migrated to new API key authentication scheme
- Updated all endpoint paths for v2 API compatibility

### Launch tofupilot.com
URL: https://www.tofupilot.com/changelog/launch-tofupilotcom

The new tofupilot.com website is live with a product overview and staff-only access to internal tools.

- Added landing page with product overview and hero section
- Added authentication with staff-only access for internal tools

### Navigate the website with a cleaner structure
URL: https://www.tofupilot.com/changelog/navigate-the-website-with-a-cleaner-structure

The website has been reorganized into clearer sections with direct navigation links.

- Restructured pages into route groups for products, resources, and company
- Updated navbar with direct links instead of dropdown menus
- Aligned hero section width with content layout

### Use TofuPilot in your language
URL: https://www.tofupilot.com/changelog/use-tofupilot-in-your-language

The website is now available in multiple languages, and the Orbit product page has a fresh new look.

- Added internationalization for landing, pricing, about, orbit, and footer pages
- Redesigned Orbit hero with new illustration and layout

### Refresh email branding
URL: https://www.tofupilot.com/changelog/refresh-email-branding

Emails from TofuPilot now feature the updated mascot icon and branding.

- Replaced email header logo with new TofuPilot mascot icon
- Updated email template to use com domain for assets

### Improve website SEO and page performance
URL: https://www.tofupilot.com/changelog/improve-website-seo-and-page-performance

Search engines and users now find TofuPilot pages faster with optimized metadata and improved load times.

- Optimized meta descriptions across 77 pages and consolidated sitemap
- Fixed canonical URLs, heading structure, and broken links for better crawlability
- Added dynamic social preview images and improved page load speed

### Access docs at tofupilot.com/docs
URL: https://www.tofupilot.com/changelog/access-docs-at-tofupilotcomdocs

All documentation now lives under the main tofupilot.com domain for a unified experience.

- Unified documentation under the main tofupilot.com domain
- Fixed version mismatches across documentation pages

### Fix docs page loading errors
URL: https://www.tofupilot.com/changelog/fix-docs-page-loading-errors

Some documentation pages were intermittently failing to load.

- Fixed intermittent loading errors on documentation pages
- Improved docs routing reliability

### Simplify self-hosting setup documentation
URL: https://www.tofupilot.com/changelog/simplify-self-hosting-setup-documentation

Getting started with self-hosting is now a single command instead of cloning a repository.

- Replaced git clone instructions with one-line curl deploy command
- Added GitHub and GitLab OAuth provider setup guides
- Added version management and uninstall flags documentation

### View and track your quotes in the customer hub
URL: https://www.tofupilot.com/changelog/view-and-track-your-quotes-in-the-customer-hub

Customers can now view and track their quotes directly from the Orbit portal.

- Added customer hub for viewing and tracking quotes
- Redesigned Orbit hero illustration

### Show the real cause of CLI connection errors
URL: https://www.tofupilot.com/changelog/show-the-real-cause-of-cli-connection-errors

When a station could not reach its TofuPilot instance, the CLI printed a generic "error sending request" message with no way to tell what actually failed.

- Fixed connection errors to report the underlying cause, such as a rejected TLS certificate, a DNS failure or a proxy issue
- Added the full error chain to the message, for example "invalid peer certificate: UnknownIssuer" when the instance certificate is not trusted

### Fix license error messages during deployment
URL: https://www.tofupilot.com/changelog/fix-license-error-messages-during-deployment

The self-hosted deployment script showed a generic "Invalid license key" error for all failures, making it difficult to diagnose connection problems versus actual license issues.

- Fixed the deployment script to show specific error messages: malformed key format, unrecognized key, expired or suspended license, or network connectivity problems
- Improved error messages for expired or suspended licenses to display the reason from the licensing server

### Ship test data from Rust
URL: https://www.tofupilot.com/changelog/ship-test-data-from-rust

You can now push test runs to TofuPilot from Rust. The SDK covers the full V2 API with async builders, typed errors, retries, and file upload helpers.

```rust
let client = TofuPilot::new("your-api-key");

client.runs().create()
    .procedure_id("FVT-001")
    .serial_number("SN-042")
    .part_number("PART-001")
    .outcome(Outcome::Pass)
    .send()
    .await?;
```

Source on [GitHub](https://github.com/tofupilot/rust), package on [crates.io](https://crates.io/crates/tofupilot).

### Track phase retries in test runs
URL: https://www.tofupilot.com/changelog/track-phase-retries-in-test-runs

When a test phase fails and is retried, you can now store every attempt with a `retry_count` field and filter retries out of your analytics.

- Added `retry_count` to phases: set it to track which attempt each phase represents (0 = first, 1 = first retry). Defaults to 0 when omitted.
- Added "Exclude retries" toggle on all insights charts (Cpk, cycle time, failures, control charts, duration) to focus on final attempts only
- Retried phases show expandable attempt tabs in the phase detail view
- OpenHTF retries are detected automatically from repeated phase names, no changes needed in test scripts

### Improve API reliability and performance
URL: https://www.tofupilot.com/changelog/improve-api-reliability-and-performance

Rebuilt our REST API layer for better error handling, faster request routing, and more accurate OpenAPI documentation. This improves integration stability for all API and SDK users.

### Deploy test procedures from GitLab
URL: https://www.tofupilot.com/changelog/deploy-test-procedures-from-gitlab

Connect your GitLab repositories to automatically deploy test procedures to your stations and trace every test run back to its source code.

- Added GitLab integration supporting both GitLab.com and self-hosted instances
- Added automatic procedure deployment to stations when pushing to GitLab
- Added webhook-based synchronization to keep branches and commits in sync

### Upload test results from MATLAB
URL: https://www.tofupilot.com/changelog/upload-test-results-from-matlab

You can now upload runs, manage parts, and query test data directly from MATLAB -- no middleware, no CSV exports.

- Upload runs with measurements, limits, and attachments in a few lines of MATLAB code
- Query and filter units, batches, and procedures from your test scripts
- Automatic retries on transient errors so uploads don't fail mid-production
- Ships as a .mltbx toolbox -- install once, available across all your projects

### Process control for every measurement
URL: https://www.tofupilot.com/changelog/process-control-for-every-measurement

The Insights page is now Process Control, rebuilt from scratch.

- Added interactive control charts with spec limits, control limits, histograms, and capability indices (Cp, Cpk, Pp, Ppk) for numeric, string, and boolean measurements
- Added sidebar filters for outcome, value range, retries, and all standard run filters, with a measurement selector sorted by failures, fail rate, or Cpk
- Added measurement table with checkbox selection to drill down into matching runs or units in one click
- Faster loads. Charts and capability indices now compute server-side.

### Attach files to runs and units in one call
URL: https://www.tofupilot.com/changelog/attach-files-to-runs-and-units-in-one-call

Uploading files to test runs and units used to require three API calls (initialize, PUT, finalize) plus a fourth to link the attachment. Now it's one call across all five SDK clients.

- Added `runs.attachments.upload()` and `units.attachments.upload()` helpers that handle the full upload flow in a single call across Python, C#, Rust, C++, and MATLAB
- Added `units.attachments.delete()` to remove attachments from units by ID
- Added `runs.attachments.download()` and `units.attachments.download()` to download attachments to local files

```python
from tofupilot.v2 import TofuPilot

client = TofuPilot()

# Upload a file to a run
client.runs.attachments.upload(id=run.id, file="data/test-report.pdf")

# Upload a file to a unit
client.units.attachments.upload(serial_number="SN-0001", file="data/calibration.pdf")

# Download
client.runs.attachments.download(attachment, dest="local-report.pdf")

# Delete (units only)
client.units.attachments.delete(serial_number="SN-0001", ids=[attachment_id])
```

### Stations v2.0
URL: https://www.tofupilot.com/changelog/stations-v20

A headless-first rewrite of TofuPilot Stations: more hardware architectures supported, real-time control from the web, isolated artifacts, and push-to-deploy with environments.

- Rebuilt on a headless core that runs on Raspberry Pi (arm64), Linux (amd64/arm64), Windows, and macOS: pick a kiosk UI, terminal UI for SSH and edge boxes, or no UI at all
- Streamed every station live to the web for remote control and telemetry, with the offline queue and full Framework UI components carried over from v1
- Replaced repo-pull with a managed artifact system so stations never touch your Git provider directly
- Auto-deployed on merge to your configured environments, with skippable preview environments matched by branch pattern
- Run TofuPilot Framework procedures end-to-end from your repo to your stations

### Better monorepo venv support
URL: https://www.tofupilot.com/changelog/better-monorepo-venv-support

Stations now find the right Python environment every time, including monorepo deployments where the venv lives next to a sub-package.

- Improved monorepo support: venv is resolved deterministically at the package directory, no more walk-up search
- Fixed run failures on stations where divergent venv resolvers picked the wrong interpreter
- Added regression tests covering monorepo, single-package, and local-path runs

### Add V2 Python SDK examples to docs
URL: https://www.tofupilot.com/changelog/add-v2-python-sdk-examples-to-docs

All dashboard documentation pages now include working Python V2 SDK code examples. V1 client syntax has been removed and replaced with `tofupilot.v2` throughout.

### Add strongly-typed validators to all SDKs
URL: https://www.tofupilot.com/changelog/add-strongly-typed-validators-to-all-sdks

All SDK clients now expose typed validator objects instead of untyped JSON, making it easier to build and validate measurement validators with IDE autocomplete and compile-time checks.

- Simplified validators API schema for cleaner OpenAPI output, removing legacy string union type
- Regenerated C++, Rust, C#, and Python SDK clients with strongly-typed validator structs and builders
- Fixed C# generator to support CancellationToken in all async methods and synced SDK version to 2.3.0
- Fixed Speakeasy generation script to use correct config and synced version to 2.2.2

### Fix workflow trigger accuracy
URL: https://www.tofupilot.com/changelog/fix-workflow-trigger-accuracy

Workflows could incorrectly trigger from unrelated events.

- Fixed workflow execution to correctly fire only on matching events
- Improved organization-level filtering in workflow endpoints

### Fix SPC sidebar filters with multiple values
URL: https://www.tofupilot.com/changelog/fix-spc-sidebar-filters-with-multiple-values

Fixed a bug that could cause SPC sidebar filters to silently ignore selections when more than one value was picked. Multi-value filters now narrow the chart as expected.

### Override the entry point per procedure
URL: https://www.tofupilot.com/changelog/override-the-entry-point-per-procedure

Procedures can now declare which file or directory the station hands to the framework runner. Useful when your repo doesn't follow the default layout, or when you want pytest to scan a specific subdirectory.

- Added an Entry point field to the Build configuration section on every procedure page
- Forwarded the value through the deployment manifest so the station picks it up on the next pull
- Defaulted to the framework convention when left blank: main.py for OpenHTF and plain Python, the package directory for pytest, procedure.yaml for the TofuPilot Framework

### Detect pytest repos without a main.py
URL: https://www.tofupilot.com/changelog/detect-pytest-repos-without-a-mainpy

Pytest-only repos with no sentinel main.py now show up in the New Procedure picker on their own. The audit treats a leaf test_*.py next to pyproject.toml as enough signal to commit to the pytest framework.

- Added test_*.py / *_test.py as a procedure signal in the repo audit
- Defaulted detected pytest procedures to entry point "." (the package directory)
- Dropped the sentinel main.py from the Pytest Starter template

### Default new stations to kiosk mode
URL: https://www.tofupilot.com/changelog/default-new-stations-to-kiosk-mode

Operator-facing stations come up in kiosk mode out of the box, with a tighter install command and a polished repo list on the New Procedure flow.

- Defaulted new station deployments to kiosk mode (full-screen operator UI on boot)
- Refactored the install command snippet so the copy-paste flow is shorter
- Polished the linked-repo list on the procedure detail page

### Self-hosting license management
URL: https://www.tofupilot.com/changelog/self-hosting-license-management

TofuPilot now handles license validation, plan enforcement, and usage tracking for self-hosted instances directly in the dashboard.

- Added feature-based license gating. SSO, SCIM, and Teams are now tied to your plan tier and enforced at runtime.
- Built a self-hosted usage page showing station count, user count, and monthly runs against plan limits.
- Added air-gapped license activation for instances without internet access.
- License card on the plan page shows current plan, limits, and expiry at a glance.
- Fixed station duplicate check and limit-reached modal that blocked test uploads silently.

### Export and copy data from list pages
URL: https://www.tofupilot.com/changelog/export-and-copy-data-from-list-pages

You can now export data from any list page without writing a query.

- Added Copy, Export to CSV, and Export to JSON on Runs, Units, Logs, and API Activity tables.
- Added the same options in the selection bar to copy or download just the rows you pick.

### Fix datetime filters and simplify finalize
URL: https://www.tofupilot.com/changelog/fix-datetime-filters-and-simplify-finalize

You can now pass `datetime` objects to date filters on `units.list()` and `batches.list()`. Previously these only accepted strings, while `runs.list()` already handled `datetime`. All list endpoints are now consistent.

We also cleaned up `attachments.finalize()`. It no longer asks for an empty `request_body` parameter. If you were passing `request_body={}`, you can drop it.

### Auto-append /api to custom server URLs
URL: https://www.tofupilot.com/changelog/auto-append-api-to-custom-server-urls

All SDK clients now automatically append `/api` to custom server URLs, so self-hosted servers no longer need to include it manually — passing `https://demo.tofupilot.sh` now works out of the box without getting redirected to the signin page.

### Compare measurements on SPC charts
URL: https://www.tofupilot.com/changelog/compare-measurements-on-spc-charts

Control charts were limited to one measurement at a time. You can now overlay multiple compatible measurements on a single chart to spot correlations and trends across your process.

- Added multi-measurement selection with Shift-click on the measurement overview, with compatibility checks (same type and units)
- Added phase filter in sidebar showing data point counts per phase when multiple measurements are selected
- Added compatibility tooltips showing which criteria match or mismatch when selecting measurements
- Limits and Cpk tab now hide automatically when selected measurements have different spec limits

### Redesign run sheet measurement table
URL: https://www.tofupilot.com/changelog/redesign-run-sheet-measurement-table

The phase/measurement table on the run sheet page got a full redesign. Measurements now open in a dedicated sidebar panel with charts, validators, and aggregations.

- Restyled phase/measurement table with consistent row spacing, sticky header, and unified column grid matching other tables across the app
- Added measurement and phase sidebar panels with full detail views: metadata, validators table, aggregations table, JSON code block, and multi-dimensional charts with reference lines
- Migrated multi-dimensional charts to recharts with legend, axis labels, toggleable aggregation reference lines, and a fullscreen dialog
- Added outcome filters (fail, error, skip), search, sort, and relative/duration time toggle to the section header

### Add named data series and multi-axis charts
URL: https://www.tofupilot.com/changelog/add-named-data-series-and-multi-axis-charts

Multi-dimensional measurements now support named data series with independent units, enabling richer time-series visualizations.

- Added `name` field to data series, allowing each axis to carry a label (e.g. "Frequency", "Voltage")
- Moved `units` from measurement level to individual data series for per-axis unit control
- Added multi-axis Y chart support with independent scales when series have different units
- Added interactive series toggle in chart legend to show/hide individual data series

### Deploy from uv-workspace monorepos
URL: https://www.tofupilot.com/changelog/deploy-from-uv-workspace-monorepos

Procedures living inside a uv-workspace monorepo can now be deployed without restructuring the repo, by pointing the CLI at the package directory.

- Added a `package_directory` setting on procedures so deployments resolve dependencies from a sub-package instead of the repo root.
- Updated the CLI and deployer to walk up from the package directory to find the workspace venv and lockfile.
- Consolidated the deploy pipeline behind a typed v1 manifest, with framework auto-detected from disk at runtime.

### Fix measurement outcome rollup
URL: https://www.tofupilot.com/changelog/fix-measurement-outcome-rollup

Measurements with a recorded value but no validators were reaching the dashboard as `UNSET` instead of `PASS`, mostly affecting string measurements used for traceability.

- Fixed outcome rollup to match OpenHTF semantics: null value stays `UNSET`, any failing validator forces `FAIL`, otherwise `PASS`.
- Added unit tests covering the parity rules.

### Reliability and kiosk fixes
URL: https://www.tofupilot.com/changelog/reliability-and-kiosk-fixes

A round of fixes covering the auto-update flow, kiosk polish on Windows, the git provider setup, and how stations refresh to the latest CLI.

- Fixed auto-update segfaults and the boot ENOENT race that occasionally crashed stations on first launch after an upgrade
- Fixed the Windows TUI double-registering keystrokes and the desktop shortcut launching the procedure runner instead of the station daemon
- Fixed the GitHub and GitLab provider setup screens losing state on reload, and surfaced underlying provider errors instead of a generic toast
- Refreshed installs to the latest CLI faster: removed the CDN cache on the version endpoint and suppressed spurious "update failed" warnings on subsequent boots

### Fix billing seat count to exclude banned users
URL: https://www.tofupilot.com/changelog/fix-billing-seat-count-to-exclude-banned-users

Organizations were being overcharged because banned users were incorrectly counted as active seats in billing calculations.

- Fixed billing to exclude banned users from seat counts
- Fixed unbanning users to correctly restore their seat in billing

### Sign Windows CLI builds with a code certificate
URL: https://www.tofupilot.com/changelog/sign-windows-cli-builds-with-a-code-certificate

Windows now trusts the TofuPilot CLI out of the box, removing the "unknown publisher" SmartScreen warning that appeared when running downloaded builds.

- Signed the Windows CLI executable with a new code-signing certificate issued to TofuPilot SA
- Verified every release build carries a valid signature before it ships

### Fix batch unit count for batches over 100 units
URL: https://www.tofupilot.com/changelog/fix-batch-unit-count-for-batches-over-100-units

Batch lists were showing incorrect unit counts for batches with more than 100 units, making it difficult to track production progress accurately.

- Fixed batch unit counts to show the correct total instead of capping at 100
- Updated batch list API response to return `unit_count` as an integer
- Removed unnecessary unit details from batch list responses to reduce payload size

### Fix missing API pages and add CLI snippets
URL: https://www.tofupilot.com/changelog/fix-missing-api-pages-and-add-cli-snippets

You can now copy `tofupilot` CLI commands straight from the API reference.

- Added a CLI tab next to Python, cURL, and the other code samples on every endpoint.
- Fixed missing endpoints and the empty cURL tab in the v2 reference.

### Clean up README and improve release automation
URL: https://www.tofupilot.com/changelog/clean-up-readme-and-improve-release-automation

The README contained outdated example code and placeholder text that didn't reflect the current v2 SDK, making it harder for new users to get started.

- Updated README with working v2 SDK examples and removed boilerplate content
- Added automatic release notes on the repository when new versions are published
- Fixed version tag format to match existing convention

### Remove console banner and version check
URL: https://www.tofupilot.com/changelog/remove-console-banner-and-version-check

The Python SDK no longer prints a banner or checks PyPI on every client init — cleaner output, faster startup, no network call at import time.

### Speed up procedure analytics page
URL: https://www.tofupilot.com/changelog/speed-up-procedure-analytics-page

The procedure analytics page now loads faster, even on procedures with tens of thousands of runs.

- Improved performance across charts, KPIs, and filter sidebars
- Added a guardrail that prompts you to narrow the date range on very large queries

### GitLab real-time sync and soft delete
URL: https://www.tofupilot.com/changelog/gitlab-real-time-sync-and-soft-delete

GitLab sync relied on polling and deleted branches disappeared from deployment history, losing audit context.

- Added real-time webhook sync for GitLab push and merge request events, replacing polling
- Preserved deleted branches and commits in deployment history with visual indicators instead of removing them
- Replaced OAuth setup with a token-first flow supporting both Group and Personal Access Tokens

### Add meetings section to Orbit
URL: https://www.tofupilot.com/changelog/add-meetings-section-to-orbit

Enterprise customers can now view their meetings with TofuPilot directly in Orbit, including agenda, summary, and linked action items with assignees.

### Ship test data from C# and .NET
URL: https://www.tofupilot.com/changelog/ship-test-data-from-c-and-net

You can now push test runs to TofuPilot from C# and .NET. The SDK covers the full V2 API with typed responses and errors.

```csharp
var client = new TofuPilot(apiKey: "your-api-key");

await client.Runs.CreateAsync(new RunCreateRequest
{
    ProcedureId = "FVT-001",
    SerialNumber = "SN-042",
    PartNumber = "PART-001",
    Outcome = RunCreateOutcome.Pass,
});
```

API docs now show C# examples next to Python. Source on [GitHub](https://github.com/tofupilot/csharp), package on [NuGet](https://www.nuget.org/packages/TofuPilot).

Thanks to community member [@Hylaean](https://github.com/Hylaean) for building the original 1.x client that got this started.

### Add Rust code snippets to API reference
URL: https://www.tofupilot.com/changelog/add-rust-code-snippets-to-api-reference

Every API reference page now includes Rust code samples alongside Python, cURL, and C#.

- Added Rust builder-pattern snippets for all 44 V2 API endpoints
- Fixed C# snippet generation for add and remove sub-unit operations

### Rewrite self-hosting docs and deploy script
URL: https://www.tofupilot.com/changelog/rewrite-self-hosting-docs-and-deploy-script

The self-hosting documentation has been fully rewritten to match the actual deploy flow, and the deploy script now auto-generates a `.env` template on first run.

- Rewrote self-hosting docs to cover the real deploy flow end-to-end
- Deploy script auto-generates `.env` template when missing, instead of failing silently
- Added documentation for license management, air-gapped activation, and Orbit monitoring
- Merged Admin and Enterprise sidebar sections for clearer navigation
- Fixed license refresh endpoint that caused 500 errors during validation
- Fixed station duplicate check and limit-reached modal

### Improve process control and chart export
URL: https://www.tofupilot.com/changelog/improve-process-control-and-chart-export

Process control gets smarter defaults, better filtering, and a new PDF export for sharing charts outside TofuPilot.

- Added A4 landscape PDF export for charts with title and "Exported by" attribution
- Added validators column and measurement outcome filter to the process control measurement table
- Added default 30-day date filter on runs, logs, and API calls pages for faster initial load
- Fixed invisible connecting lines between points on control charts
- Fixed missing procedure, phase, unit, and run outcome on measurement detail page

### Automate test actions with workflows
URL: https://www.tofupilot.com/changelog/automate-test-actions-with-workflows

Trigger automated actions when test events occur. Connect your test data to the tools your team already uses, without writing glue code.

- Added visual workflow editor with drag-and-drop nodes, branching (if/else, switch), filters, and versioning
- Added integrations for Odoo, Linear, InvenTree, Discord, Slack, email, and generic HTTP
- Added execution history with per-step status, error details, and flow visualization

### Smoother GitHub setup on self-hosted
URL: https://www.tofupilot.com/changelog/smoother-github-setup-on-self-hosted

Self-hosted installations can now connect GitHub with clearer feedback at every step.

- Repository picker lists all repositories from GitHub and GitLab
- Connection errors surface clearly for easier troubleshooting

### Improve station connectivity diagnostics
URL: https://www.tofupilot.com/changelog/improve-station-connectivity-diagnostics

Faster troubleshooting when the kiosk page is unreachable or blank, with clearer log output pointing at the actual cause.

- Added `tofupilot service status` loopback probe that reports listening, refused, or timeout, and names the process holding the port.
- Added locale-stable Windows port-holder lookup so non-English Windows hosts (Chinese, Japanese, German, French) report the same diagnostics as macOS and Linux.
- Fixed hardcoded port `7321` in several paths so `TOFUPILOT_LOCAL_UI_PORT` is honored everywhere.
- Improved bind errors with specific remediation hints for port-in-use, permissions, loopback down, and container/network-namespace cases.
- Added kiosk readiness probe and browser exit detection so blank-page situations surface in the logs instead of failing silently.

### Run Robot Framework suites on stations
URL: https://www.tofupilot.com/changelog/run-robot-framework-suites-on-stations

Robot Framework joins pytest and OpenHTF as a first-class test framework on TofuPilot stations.

- Added a native Robot Framework connector: each test case becomes a phase, with measurements from the shipped TofuPilot keyword library
- Detected .robot suites automatically when importing a repo, with a new Robot starter template
- Added end-to-end coverage with eight Robot scenarios alongside the existing pytest agent-protocol tests

### Fix Dashboard Filters
URL: https://www.tofupilot.com/changelog/fix-dashboard-filters

Fixed an issue where some dashboard filters could return incorrect options after an internal infrastructure upgrade.

Our compatibility audit caught the regression, and we've added preventive checks to our release process for future dependency upgrades.

### Cleaner runs across runtimes
URL: https://www.tofupilot.com/changelog/cleaner-runs-across-runtimes

Stations now produce consistent results whether running OpenHTF or pytest, and back-to-back runs no longer carry over leftover state.

- Aligned operator UI behavior across both runtimes for parity in measurements and outcomes
- Fixed cross-run state leaks so each run starts clean
- Hardened protocol handling to prevent stale data between runs

### Upgrade server runtime to Node.js 24
URL: https://www.tofupilot.com/changelog/upgrade-server-runtime-to-nodejs-24

Upgraded our server runtime to the latest stable LTS version for better performance, security patches, and long-term support.

- Upgraded server runtime from Node.js 20 to 22 across all deployments
- Updated self-hosted Docker image to the new runtime

### Faster, more reliable Windows stations
URL: https://www.tofupilot.com/changelog/faster-more-reliable-windows-stations

Setting up and running stations on Windows is now smoother, with fewer prompts and quicker startup.

- Fixed Windows station install so PowerShell execution policy no longer blocks setup
- Pre-seeded kiosk procedures before launch so stations are ready immediately
- Reduced teardown latency between runs on Windows

### Ship test data from C++
URL: https://www.tofupilot.com/changelog/ship-test-data-from-c

Teams running C++ test scripts can now upload runs, measurements, and attachments to TofuPilot directly from their codebase.

- Added full V2 API coverage for runs, units, parts, batches, stations, procedures, and attachments
- Added builder-pattern API for creating runs with phases, measurements, validators, and logs
- Added one-call file upload and download helpers for test reports and artifacts

### Fix part validation blocking unit creation
URL: https://www.tofupilot.com/changelog/fix-part-validation-blocking-unit-creation

Creating a unit from a part with `.`, `:`, or `+` in its number used to fail with a raw validation blob. Validation now matches what part creation allows, and errors are readable.

- Fixed unit creation rejecting valid parts whose numbers contain `.`, `:`, or `+`
- Improved error messages in modals to show a clear one-line reason instead of a raw JSON payload
- Added a Copy action on error toasts so you can share the exact message with support

### Steadier station operation
URL: https://www.tofupilot.com/changelog/steadier-station-operation

Stations recover better from edge cases and surface clearer feedback when things go wrong.

- Fixed dispatcher hangs so stations no longer get stuck between runs
- Improved worker error reporting so failures are easier to diagnose
- Cleaned up kiosk shutdown and update handling for orphaned stations

### Live updates without polling
URL: https://www.tofupilot.com/changelog/live-updates-without-polling

Runs, units, and logs pages now stream updates over a single realtime channel instead of polling every few seconds, so live mode is lighter and more reliable.

- Replaced 3-second polling on list pages with realtime invalidation

### Mark units as golden or failing samples
URL: https://www.tofupilot.com/changelog/mark-units-as-golden-or-failing-samples

Reference units can now be tagged as known-good (golden) or known-faulty (failing) so they're excluded from production analytics and FPY/Cpk stay honest.

- Added a sample classification dropdown on the unit detail page with activity tracking
- Added a sidebar filter and table avatar treatment for golden and failing samples
- Surfaced the sample field in V2 API for unit and run get/list/create/update

### Add custom metadata to runs and units
URL: https://www.tofupilot.com/changelog/add-custom-metadata-to-runs-and-units

Tag runs and units with your own typed key/value fields so traceability data, and qualification flags live next to the test data they describe.

- Added a metadata editor on run and unit detail pages with type-aware icons and inline edit
- Surfaced metadata in the V2 API as a plain `{key: value}` dict; types (string, number, bool) are detected automatically
- Added metadata filters on run and unit list endpoints (`in`, `contains`, `gte`, `lte`, `eq`)
- Tracked every metadata change in a per-entity activity timeline

### Add file attachments to units via Python SDK
URL: https://www.tofupilot.com/changelog/add-file-attachments-to-units-via-python-sdk

You can now attach files directly to units using the Python SDK v2.1.0. Upload photos, test reports, calibration data, or any file and link it to a unit in two lines of code.

- Added `attachments.upload()` to handle file uploads in a single call
- Added `attachments.download()` to download attachments from units or runs
- Added attachments support to the Update Unit endpoint
- Renamed the attachment finalization endpoint for clarity

### Rewrite test and device data model pages
URL: https://www.tofupilot.com/changelog/rewrite-test-and-device-data-model-pages

Both fundamentals data model pages were rewritten into one consistent reference with updated diagrams.

- Rewrote the test and device data model pages with a uniform per-concept format.
- Refreshed the animated diagrams and added a Schema reference section with direct links to REST API endpoints.


### Deploy self-hosted from GitHub workflows
URL: https://www.tofupilot.com/changelog/deploy-self-hosted-from-github-workflows

Self-hosted instances can now enable automatic deployments by installing the TofuPilot GitHub App. The setup process is documented in the [self-hosting guide](https://www.tofupilot.com/docs/dashboard/self-hosting). Cloud users already have this built in.

### Fix deployment commit filters
URL: https://www.tofupilot.com/changelog/fix-deployment-commit-filters

- Fixed commit list filters on the deployments page that weren't connected to commit data, causing confusing empty results.
- Laid the server, worker, and UI foundations for the upcoming Station v2 release.

### Support timezone offsets in datetime fields
URL: https://www.tofupilot.com/changelog/support-timezone-offsets-in-datetime-fields

Timestamps with timezone offsets like `+02:00` or `+05:30` now work across all API endpoints. You no longer need to convert to UTC before uploading.

- Accepted timezone offsets on all datetime input fields: run timestamps, phase and log timestamps, and date filters on runs, units, batches, and procedures
- Responses still return UTC (`Z` suffix), no changes needed on the read side

```python
from datetime import datetime, timezone, timedelta

# UTC (always worked)
client.runs.create(
    started_at=datetime.now(timezone.utc),
    ended_at=datetime.now(timezone.utc),
    ...
)

# Local timezone (now works too)
cet = timezone(timedelta(hours=2))
client.runs.create(
    started_at=datetime.now(cet),
    ended_at=datetime.now(cet),
    ...
)
```

### Deploy OpenHTF procedures from GitHub
URL: https://www.tofupilot.com/changelog/deploy-openhtf-procedures-from-github

OpenHTF is a first-class framework alongside the TofuPilot Framework for stations: clone a starter from /new, push to GitHub, the station picks up your test on the next deploy.

- Added a one-click OpenHTF Starter template in /new -- single procedure showing plugs, prompts, hard + marginal limits, regex validators, voltage-vs-time chart, attachments, and skip flow control
- Streamed live phase events (phase started/ended, measurements, prompts, attachments) so the operator UI mirrors what is happening on the station in real time
- Routed native user_input prompts to the web operator UI, with timeouts and operator-typed responses flowing back to OpenHTF without leaving the dashboard
- Rendered marginal limits, multi-dim charts, and validator expressions natively in the dashboard from the connector's structured payloads

### Fix template clone auto-deploy
URL: https://www.tofupilot.com/changelog/fix-template-clone-auto-deploy

Fixed an issue when cloning a template: auto-push now picks up the seed commit so the procedure deploys to your stations immediately.

### Speed up the measurement control page
URL: https://www.tofupilot.com/changelog/speed-up-the-measurement-control-page

The measurement control page could freeze or take a long time to open for procedures with very large numbers of measurements.

- Made control charts open instantly, even for measurements with millions of data points, while pass rate and Cpk stay accurate across every point
- Made the value filter appear right away instead of waiting on heavy data
- Added loading previews and smoother chart hover and selection

### C++ SDK gets typed metadata
URL: https://www.tofupilot.com/changelog/c-sdk-gets-typed-metadata

C++ client now ships the run + unit metadata feature with a typed `std::variant` for the value instead of a bare `nlohmann::json`.

- Regenerated C++ SDK against the new metadata endpoints
- Metadata field is now `std::map<std::string, std::variant<std::string, double, bool>>` so values keep their type at compile time
- Added Custom Metadata sections to the runs and units guides with snippets for Python, C#, C++, and MATLAB

### Add changelog filters and "What's new" in Orbit
URL: https://www.tofupilot.com/changelog/add-changelog-filters-and-whats-new-in-orbit

The changelog now supports URL-based filtering, and Orbit shows what changed between your current version and the latest.

- Added changelog filtering by product and version range to quickly find relevant updates
- Added "What's new" link in Orbit instance menus that shows releases between your current and latest available version

### Fix station deployment pulls after CLI deploy
URL: https://www.tofupilot.com/changelog/fix-station-deployment-pulls-after-cli-deploy

Stations that had also been used for `tofupilot deploy` could fail to pull deployments with a "Station authentication required" error, even right after logging in. Reinstalling didn't help.

- Fixed `tofupilot pull` so it always uses the station identity, not a leftover user login from a deploy.
- Added a clearer error message when a user login reaches a station-only command, pointing to the exact fix.

### Run pytest test suites on stations
URL: https://www.tofupilot.com/changelog/run-pytest-test-suites-on-stations

Pytest is now a first-class procedure framework alongside OpenHTF and the TofuPilot Framework. Each test function becomes a phase, and recognized assert shapes are promoted to measurements with validators on the dashboard, no SDK imports required.

- Added a pytest connector that streams phase events live and bundles measurements per phase
- Added AST extraction for numeric ranges, single bounds, equality, pytest.approx, string equality and membership, and boolean asserts
- Added support for multiple measurements in one test (distinct identifiers) and stacked validators on one measurement (repeated asserts on the same identifier)
- Added a Pytest Starter template in the New Procedure picker so you can clone and run in one click

### Improve database performance monitoring
URL: https://www.tofupilot.com/changelog/improve-database-performance-monitoring

TofuPilot now tracks database query performance, so slow pages and API responses get found and fixed faster.

- Added query performance tracking to the TofuPilot cloud database
- Updated self-hosted installations to enable the same tracking automatically at the next upgrade

### Add timezone-aware datetime hover cards
URL: https://www.tofupilot.com/changelog/add-timezone-aware-datetime-hover-cards

You can now hover any date to see UTC and local time side by side.

- Added a rich hover card on date cells: live relative age, plus UTC and local timezone rows.
- Replaced absolute/relative toggle on activity feeds with compact relative time; hover reveals full timestamps.

### Fix linking GitLab repos in nested subgroups
URL: https://www.tofupilot.com/changelog/fix-linking-gitlab-repos-in-nested-subgroups

Linking a GitLab project that lived under a subgroup failed with a generic "Failed to fetch metadata" error, even when the connected token had full access. The metadata lookup was double-encoding the project path before sending it to GitLab.

- Fixed GitLab project lookups for any multi-segment path (e.g. `group/subgroup/project`)
- Linking repositories deep in nested GitLab groups now succeeds on the first try
- Branch listing, commit history, file content, and tree fetches across nested GitLab projects also resolve correctly

### Transitional fixes ahead of user-mode CLI
URL: https://www.tofupilot.com/changelog/transitional-fixes-ahead-of-user-mode-cli

Stop-gap fixes for the most common spawn failures we're seeing today, while we finish full user-mode support for the CLI in an upcoming release.

- Fixed local runs on Windows with default antivirus settings
- Surfaced clear error messages when the Python venv isn't usable
- Reported a terminal event to the kiosk UI when a subprocess fails to start

### New Test Lifecycle page with animated diagram
URL: https://www.tofupilot.com/changelog/new-test-lifecycle-page-with-animated-diagram

A new fundamentals page maps the four stages of a hardware test on TofuPilot end to end.

- Added a "Test Lifecycle" page covering author, deploy, run, and analyze with an animated diagram of the cycle
- Listed the three authoring paths (TofuPilot Framework, supported framework, custom test code) at every stage so it is clear how each one moves through the lifecycle

### Documentation 2.0
URL: https://www.tofupilot.com/changelog/documentation-20

A re-architected documentation site with cleaner navigation and dedicated space for the new surface areas TofuPilot ships against.

- Reorganized the sidebar around what you actually use day-to-day. Top groups now flow from "Getting started" and "Fundamentals" into product surfaces: Frameworks (TofuPilot, OpenHTF, Pytest, Robot), API & SDKs (Python, C#, Rust, C++, MATLAB, REST), CLI, AI, Analytics, Deployments, Integrations, Inventory, Station, Access, Security, Pricing, Self-hosting
- Replaced the flat dump of pages with grouped sections, so framework-specific guidance, SDK references, and integration setups each live under their own dedicated group instead of sharing one bucket
- Added per-resource sub-sections so growing surface areas (every CLI command, every REST endpoint, every Station toggle) have their own page instead of being concatenated
- Added a Preview badge and banner so features still rolling out (AI, Slack, InvenTree, Reports, Insights, Failure Analysis, Rolling Rollout, Instant Rollback) are clearly flagged

### Add Run, Log, and Unit explorers
URL: https://www.tofupilot.com/changelog/add-run-log-and-unit-explorers

Three new explorer pages give the table-driven workflows you live in every day their own home in the docs.

- Added Run, Log, and Unit explorer pages under Analytics, each covering columns, filters, saved views, export (CSV and JSON, visible or selected rows), metadata, and role-based permissions
- Renamed the station Operator UI page to Remote UI to avoid confusion with the framework's per-phase operator UI components
- Rewrote the docs landing around the actual journey: build, deploy, operate, observe, collaborate, integrate, then explore
- Added a "Get support" section on the landing with direct links to Discord, support email, trainings, and custom integrations

### Animate test and device data model diagrams
URL: https://www.tofupilot.com/changelog/animate-test-and-device-data-model-diagrams

Two new animated tree diagrams make the test and device data models easy to read at a glance, with example values and pass/fail states streaming in as each entity is highlighted.

- Added an animated tree on the Test Data Model page (Procedure, Run, Phase, Measurement, Validator, Log, Attachment) with live example values and a PASS validator outcome
- Added an animated tree on the Device Data Model page (Part, Revision, Unit, Sub-unit, Batch) with the Batch drawn as a dashed connector to signal that batches are optional and cross-cutting
- Renamed the underlying pages to Test Data Model and Device Data Model and updated every link across the docs

### Keep yield alerts open until quality recovers
URL: https://www.tofupilot.com/changelog/keep-yield-alerts-open-until-quality-recovers

A yield alert could close itself while the line was still failing. The drop that raised it slowly became part of the baseline it was being compared against, so a single quality incident produced a string of alerts opening and closing on their own — one incident on a production line generated nine.

- Fixed yield alerts closing on the very failures that raised them, so one incident now stays one alert until someone resolves it
- Locked each alert to the baseline measured when it was raised, so the bar to close it no longer drifts while the incident is still open
- Applied the same rule to custom yield rules, which recalculated their reference window on every check

### Refresh doc links for docs v2.0
URL: https://www.tofupilot.com/changelog/refresh-doc-links-for-docs-v20

Following the docs v2.0 release, every doc link across the dashboard, marketing site, and docs site has been audited and pointed at the current pages so users land on the right place instead of a stale route.

- Repointed dashboard empty states, deployment pages, and OpenAPI references to the new docs IA.
- Updated landing footer, navbar, connector, and self-hosting links so the site sends visitors to the right docs.
- Fixed internal docs redirects after the data-model rename and refreshed SDK readmes with the new per-language guide links.

### Document the run import API
URL: https://www.tofupilot.com/changelog/document-the-run-import-api

The REST API reference now covers run import, and every import format has a quick-reference spec at a glance.

- Added reference pages for the import and batch import endpoints.
- Added a spec block to each format page (OpenHTF, WATS, ATML, TestStand, STDF, ATDF, CSV, Excel) showing encoding, extension, detection, origin, and what it maps to.
- Replaced the import endpoints table with cards matching the rest of the page.

### Rewrite incremental migration guide
URL: https://www.tofupilot.com/changelog/rewrite-incremental-migration-guide

The `/docs/incremental-migration` page is now structured around vertical and horizontal migration strategies, with an animated diagram that shows each one in action.

- Reframed the page around incremental migration benefits, disadvantages of one-time migrations, and three strategies: vertical (one integration layer at a time across procedures), horizontal (one procedure end-to-end), and hybrid.
- Added an animated grid that plays vertical fill first, pauses, then horizontal fill so the difference is visible at a glance.

### Fix bot author label on template deployments
URL: https://www.tofupilot.com/changelog/fix-bot-author-label-on-template-deployments

Fixed an issue where deployments from a new procedure template showed the TofuPilot bot as an "External Contributor".

- Bot-authored template commits now show a "TofuPilot Bot" badge.

### Import runs from OpenHTF, WATS, and CSV files
URL: https://www.tofupilot.com/changelog/import-runs-from-openhtf-wats-and-csv-files

Bring your existing test reports into TofuPilot without writing any API code. Drag a file into the dashboard and it becomes a run, so you can backfill history from benches and testers you already have.

- Added file import in the dashboard for OpenHTF JSON logs and WATS reports, auto-detected on upload, plus CSV files where you map your columns to fields once and save the mapping as a preset to reuse on every later file. Batch import handles many files at once.
- Added duplicate detection, so re-importing the same file links to the existing run instead of creating a copy.
- Improved the import and list pages with an inline drag-and-drop dropzone at the top of the import table, quick add buttons on the runs and units lists, and a clearer download icon for export.

### Filter runs and units by custom metadata
URL: https://www.tofupilot.com/changelog/filter-runs-and-units-by-custom-metadata

The runs and units pages now let you filter by any custom metadata you attach, and stay responsive even with tens of thousands of records.

- Added per-key metadata filters that auto-discover the keys in use: pick string values from a list, set numeric min/max ranges, or toggle true/false. Combine several keys to narrow on all of them at once.
- Applied the same metadata filter to the timeline chart so its counts match the table below.
- Rebuilt list loading so a page is a fixed handful of queries regardless of row count: metadata filtering resolves in a single query instead of one per key, and revision images and avatars load in one batch instead of one request per row.
- Fixed a bug where sorting units by part number, last run, or last procedure while a filter was active could fail to load results.

### Fix stuck deployment in new procedure wizard
URL: https://www.tofupilot.com/changelog/fix-stuck-deployment-in-new-procedure-wizard

The "Build first deployment" step in the clone-template and import wizards could spin forever even after the build finished successfully, leaving you unable to complete setup.

- Fixed the wizard hanging on the build step by ensuring live build status always reaches the page
- Added a fallback that detects a finished build within a second if a realtime update is missed
- Prevented duplicate procedures and stations from a double-click or retrying after a failed build

### Clearer metadata filters on runs and units
URL: https://www.tofupilot.com/changelog/clearer-metadata-filters-on-runs-and-units

Custom metadata filters on the runs and units pages are now easier to read and stay in sync.

- Added a divider separating standard filters from your custom metadata filters.
- Metadata you add to a run or unit appears as a filter option immediately, no refresh needed.

### Split file import API by format
URL: https://www.tofupilot.com/changelog/split-file-import-api-by-format

The V2 import API now has a dedicated endpoint for each kind of file, so structured datalogs and tabular spreadsheets each get a request shape that fits them.

- Added POST /v2/imports/structured to import 1 to 100 OpenHTF, WATS, ATML, NI TestStand, STDF, or ATDF files in one call, each parsed independently so one bad file never fails the rest.
- Added POST /v2/imports/tabular to import a CSV or Excel file with an inline column mapping or a saved template.
- Updated every SDK client (Python, C#, Rust, C++, MATLAB) with typed methods for both endpoints.

### Rewrite the file import guide
URL: https://www.tofupilot.com/changelog/rewrite-the-file-import-guide

The import documentation now explains every supported format, how to import from the dashboard or API, and how to convert an unsupported format yourself.

- Added a clear breakdown of structured and tabular formats with a brand icon for each.
- Added a section on writing a converter with the REST API, a typed SDK, or a coding agent when your format isn't supported natively.
- Updated the API reference with the new structured and tabular import endpoints.

### Match procedure docs to actual CLI behavior
URL: https://www.tofupilot.com/changelog/match-procedure-docs-to-actual-cli-behavior

The procedure documentation now reflects what the CLI actually enforces, removing constraints that never existed.

- Updated CLI and framework pages to say procedure files can use any `.yaml` name; `procedure.yaml` is only the default lookup when running a directory
- Fixed the procedures page: names are not uniqueness-enforced and semantic versioning is a recommendation, not a requirement
- Added the distinction between the version field you bump yourself and deployments built automatically from every commit

### Real-time and deployments for self-hosted
URL: https://www.tofupilot.com/changelog/real-time-and-deployments-for-self-hosted

Self-hosted instances now get the same live updates and build pipeline as the cloud, from a single prebuilt image on any domain.

- Added a real-time WebSocket service so self-hosted dashboards show live station status, telemetry, and streaming build logs instead of manual refresh.
- Added a build worker that compiles procedure bundles and stores them as deployment artifacts.
- Improved setup with automatic DNS, TLS, secrets, and image checks; documented the new subdomain, requirements, and troubleshooting.

### Rename procedure link file to tofupilot.json
URL: https://www.tofupilot.com/changelog/rename-procedure-link-file-to-tofupilotjson

The file `tofupilot link` writes to bind a local procedure directory to the dashboard is now named after the product, so it is easier to recognize in your repo.

- Renamed the link file from `procedure.json` to `tofupilot.json`; re-run `tofupilot link` in previously linked directories after updating
- Improved error messages to reflect that procedure files can use any `.yaml` or `.yml` name, not only `procedure.yaml`
- Clarified the no-procedure-found error to list everything the CLI looks for in a directory

### Speed up dashboard page loads
URL: https://www.tofupilot.com/changelog/speed-up-dashboard-page-loads

Navigating to the dashboard previously showed a blank page until account and organization data finished loading, and every page carried chart code it did not use.

- Improved perceived load time: pages now paint instantly with loading skeletons while the sidebar streams in
- Added loading skeletons to phase pareto, run analytics, workflows, and reports
- Reduced the script payload shipped on every dashboard page by loading chart-based filters only on the routes that use them

### Fix real-time updates across page navigation
URL: https://www.tofupilot.com/changelog/fix-real-time-updates-across-page-navigation

Fixed an issue where real-time station online/offline status, dashboard updates, and build logs stopped connecting after navigating within the app.

### Fix self-hosted CLI install command in sidebar
URL: https://www.tofupilot.com/changelog/fix-self-hosted-cli-install-command-in-sidebar

Improved the station sidebar install command to include your instance URL on self-hosted deployments.

### Speed up measurement analysis to under 1s
URL: https://www.tofupilot.com/changelog/speed-up-measurement-analysis-to-under-1s

The measurement analysis view on high-volume procedures could take 38 seconds to load and sometimes timed out entirely. It now loads in under a second on the same data.

- Added a daily measurement rollup that pre-aggregates pass/fail counts and Cpk, so the analysis view reads summarized data instead of scanning millions of raw measurements.
- Improved load time on the heaviest procedures from 38 seconds (often a timeout) to under one second.
- Kept results live and exact by combining the rollup with a real-time scan of the most recent runs.

### Fix deployment builds for GitLab repositories
URL: https://www.tofupilot.com/changelog/fix-deployment-builds-for-gitlab-repositories

Deployment builds failed for procedures linked to GitLab repositories, including self-hosted instances.

- Fixed the build worker so it can clone GitLab repositories and produce deployments
- Added a clear failure reason to build errors when a deployment cannot be cloned

### Fix value counts on boolean and string charts
URL: https://www.tofupilot.com/changelog/fix-value-counts-on-boolean-and-string-charts

On measurements with more than 500 records, the boolean and string control charts derived their value distribution from a sampled subset, so the True/False split and per-value counts could be slightly off. They now reflect the exact full dataset.

- Fixed the True/False and string value counts to be computed over all records, not the chart's sampled points.
- Kept the chart fast by computing the exact distribution server-side only when sampling is in effect.

### Control station auto-updates from dashboard
URL: https://www.tofupilot.com/changelog/control-station-auto-updates-from-dashboard

Stations previously installed CLI updates automatically with no way to opt out, which made update timing unpredictable on production lines.

- Added the ability to turn automatic CLI updates on or off per station from the Setup panel
- Fixed stations without a saved setting showing auto-update as off while updates were actually running

### Fix auto-deploy on first push from template
URL: https://www.tofupilot.com/changelog/fix-auto-deploy-on-first-push-from-template

Fixed an issue where auto-deploy did not trigger on self-hosted instances when GitHub omitted commit details from the push event, such as the first push after creating a repository from a template.

- Fixed auto-deploy to fetch the pushed commit when the webhook payload omits it, so deployments trigger reliably.
- Fixed self-hosted GitHub App commits not being recognized as trusted, so template-clone deployments run with the right permissions.

### Simplify organization member management
URL: https://www.tofupilot.com/changelog/simplify-organization-member-management

Managing who has access to your organization is now simpler and safer.

- Replaced account suspension with a clear "Remove from organization" action that revokes access without touching the person's account elsewhere
- Removed members keep their history: runs and records they created stay fully visible for traceability
- Restoring access is now a simple re-invite

### Fix self-hosted builds on cgroup v2 hosts
URL: https://www.tofupilot.com/changelog/fix-self-hosted-builds-on-cgroup-v2-hosts

Fixed an issue where the self-hosted build worker could not start procedure builds on hosts using cgroup v2 (current Ubuntu and Debian releases).

- Fixed the build sandbox so procedure builds run correctly on cgroup v2 hosts.

### Add instant rollback for deployments
URL: https://www.tofupilot.com/changelog/add-instant-rollback-for-deployments

Stations can be re-pinned to a previous deployment without a rebuild.

- Added instant rollback: re-pins all stations to an older production build and pauses auto-push until you push again or resume it
- Added a deployment row menu: push to all stations, redeploy, view source, copy URL
- Added current and rolled back labels in the deployments list

### Lay groundwork for Failure Analysis
URL: https://www.tofupilot.com/changelog/lay-groundwork-for-failure-analysis

We're laying the groundwork for Failure Analysis, a new way to find which test phases fail most and dig into why.

- Added the first building blocks ahead of a wider rollout.

### Guide v1.0 stations to upgrade to the CLI
URL: https://www.tofupilot.com/changelog/guide-v10-stations-to-upgrade-to-the-cli

Stations v2.0 run on the TofuPilot CLI. The dashboard and the desktop app now point v1.0 station users to the upgrade, with step-by-step guides.

- Added a prompt on the desktop station app and the dashboard deploy page that explains the move from the v1.0 desktop station to the CLI.
- Linked the migration guide everywhere, plus a self-hosted guide for enabling realtime and deploys when the prompt detects a self-hosted instance.
- Marked the desktop station app as v1 so it is clear which stations still need to upgrade.

### Speed up page loads with session caching
URL: https://www.tofupilot.com/changelog/speed-up-page-loads-with-session-caching

Every page load and API call previously checked the session against the database before doing anything else.

- Added short-lived signed session caching so pages and API calls skip one database round-trip on every request
- Improved time to first paint on all dashboard pages

### Fix rollback offered when nothing is pushed
URL: https://www.tofupilot.com/changelog/fix-rollback-offered-when-nothing-is-pushed

Instant Rollback could run when no production deployment was pushed to any station, pausing auto-push on what was really a first push.

- Fixed the rollback dialog to disable Rollback and point to Push to All Stations when no production deployment is currently pushed
- Added a server check rejecting rollbacks with nothing to roll back from, so the procedure can no longer end up incorrectly pinned with auto-push paused

### Fix docs search returning no results
URL: https://www.tofupilot.com/changelog/fix-docs-search-returning-no-results

Search in the documentation was not returning results.

- Fixed the docs search endpoint so queries return results again
- Moved search server-side, replacing the full index download with small per-query responses

### Speed up measurement and log pages
URL: https://www.tofupilot.com/changelog/speed-up-measurement-and-log-pages

The measurement analysis and list views could take several seconds to load on high-volume procedures, and the logs page was slow on large log histories. A round of optimizations removes those slow queries so these pages stay fast as your data grows.

- Reworked how the measurement views load data, cutting load times from seconds to milliseconds.
- Sped up the logs page on large log histories.
- Removed redundant queries by deriving sidebar filter counts from data the page already loads.
- Changes apply automatically on both cloud and self-hosted instances, with no manual setup.

### Improve dashboard run links from the CLI
URL: https://www.tofupilot.com/changelog/improve-dashboard-run-links-from-the-cli

Run links shared by the CLI now point directly to the right run page.

- Updated CLI dashboard run links to open the correct run page

### Fix sidebar back button losing procedure
URL: https://www.tofupilot.com/changelog/fix-sidebar-back-button-losing-procedure

Going back from a procedure-scoped page like measurement control no longer resets the selected procedure.

- Fixed the sidebar back button to return to the procedure overview instead of the organization home when a procedure is selected
- Applied the fix across all sidebar pages: runs, logs, units, batches, measurement control, and phase pareto

### Tidy up global search results
URL: https://www.tofupilot.com/changelog/tidy-up-global-search-results

Search results now carry only the links that are actually used, removing stale entries behind the scenes.

- Removed unused run, unit, part, and batch links from search results

### Filter runs and units by metadata via the API
URL: https://www.tofupilot.com/changelog/filter-runs-and-units-by-metadata-via-the-api

Filter runs and units by their custom metadata, directly from the API and SDKs.

- Added metadata filtering to the run and unit list endpoints with per-key operators (`in`, `contains`, `gte`, `lte`, `gt`, `lt`, `eq`)
- Supported metadata filters across all SDKs and the CLI

### Add operator hints to Identify Unit fields
URL: https://www.tofupilot.com/changelog/add-operator-hints-to-identify-unit-fields

Guide operators with custom hints under each field on the Identify Unit screen, so they know exactly what to scan or enter.

- Added an optional `description` to identify-unit fields (serial number, part number, revision, batch, sub-units) in the procedure config
- Rendered the helper text under each field in both the CLI and the Station operator UI

### Patch dependency from routine security audit
URL: https://www.tofupilot.com/changelog/patch-dependency-from-routine-security-audit

Our routine dependency audit flagged a security issue, which we patched right away.

- Updated an affected dependency to its patched release
- No action needed on your end

### Launch Phase Pareto failure analysis
URL: https://www.tofupilot.com/changelog/launch-phase-pareto-failure-analysis

Find the weakest step in a procedure: Phase Pareto ranks phases by failures, retries, and duration so you target the real bottleneck instead of guessing.

- Added a Phase Pareto chart with Failures, Retries, and Duration views, each with previous-period trend badges
- Added phase-level filters for outcome, sample class (production / golden / failing), and full run and unit slicing, kept in sync between the sidebar and command bar
- Added a retry-count breakdown that surfaces phases churning through repeated attempts
- Improved the date filter to window on when each phase ran, not when its run started

### Fix station install command on test stations
URL: https://www.tofupilot.com/changelog/fix-station-install-command-on-test-stations

The station install command now uses an explicit secure URL, avoiding a redirect hop that could fail on older or proxied curl and leave test stations unable to install the CLI.

- Fixed station install commands to use a direct https URL instead of relying on a redirect
- Unified the Linux install command on sh to match the installer script
- Updated all CLI and station docs snippets for consistency

### Rename Analytics to Run Analytics
URL: https://www.tofupilot.com/changelog/rename-analytics-to-run-analytics

The Analytics page is now called Run Analytics, making it clearer that it covers yield, throughput, and duration across your test runs and distinguishing it from Phase Pareto and Process Control.

- Renamed the Analytics page to Run Analytics in the dashboard and docs
- Added automatic redirects so existing bookmarks and links keep working

### Redesign Measurement Control analytics
URL: https://www.tofupilot.com/changelog/redesign-measurement-control-analytics

Measurement Control now covers every measurement type and makes it faster to find which measurement is driving failures.

- Added support for all measurement types (numeric, boolean, string, multidimensional, and more), each listed and counted in the pareto and detail table
- Added a Failures view showing measurement outcomes over time, plus an average Cpk indicator with a period-over-period trend
- Reorganized the sidebar with phase, measurement, type, and outcome filters that scope the chart and table together
- Improved the table to list every measurement when nothing is selected, with stable columns whether or not a measurement is picked

### Polish Measurement Control charts and filters
URL: https://www.tofupilot.com/changelog/polish-measurement-control-charts-and-filters

Refinements to the Measurement Control page make the charts, filters, and detail table clearer and more consistent.

- Improved the Cpk view with an average Cpk indicator and trend, a capability-based bar color, and a clearer 1.33 threshold marker
- Made sidebar filter counts match the chart by using the same default time window, and unified the outcome and value filters with the rest of the sidebar
- Improved the measurement table with sortable phase and measurement columns, sample badges, type icons for multidimensional and JSON values, and selection shortcuts to open the related runs or units
- Fixed bin deselection in the control chart and tidied the resizable divider between charts

### Filter Measurement Control by run details
URL: https://www.tofupilot.com/changelog/filter-measurement-control-by-run-details

New sidebar filters let you narrow Measurement Control to specific runs by where and how they were produced.

- Added Run ID and Duration filters to scope measurements to specific or longer/shorter runs
- Added a Procedure section with Environment (production/preview/development), Deployment (by commit), and Version filters
- Each filter shows live counts and narrows the charts and table together

### Filter Phase Pareto by run details
URL: https://www.tofupilot.com/changelog/filter-phase-pareto-by-run-details

The Phase Pareto page now has the same run-detail filters as Measurement Control.

- Added Run ID and Duration filters to scope phases to specific or longer/shorter runs
- Added a Procedure section with Environment (production/preview/development), Deployment (by commit), and Version filters
- Each filter shows live counts and narrows the chart and table together

### Fix a publish stall after station plugs start
URL: https://www.tofupilot.com/changelog/fix-a-publish-stall-after-station-plugs-start

On a station, the run that started the station-scoped plugs kept the dashboard live feed open after it ended: the plugs' log stream was still attached to that run, so its events only stopped after a 10 second timeout, and later runs' plug logs were written into the first run's log.

- Fixed station plug logs to follow the run currently using the plugs instead of the one that started them
- Fixed the run log and the dashboard feed of the first run closing as soon as it ends

### Fix Windows CLI install access denied error
URL: https://www.tofupilot.com/changelog/fix-windows-cli-install-access-denied-error

Windows users could hit an "Access is denied" error when running the CLI right after installing it, because Windows flagged the downloaded binary as untrusted.

- Fixed the installer to clear the Mark-of-the-Web tag on the CLI binary so it runs immediately after install
- Removed the need to manually unblock tofupilot.exe before first use

### Grouped filters + deployment filters on tables
URL: https://www.tofupilot.com/changelog/grouped-filters-deployment-filters-on-tables

Table filter sidebars are now organized into clear sub-groups, and run-detail filters are available across the run-backed pages.

- Reorganized the Runs, Logs, Units, API calls, and Batches filter sidebars into grouped sections (Run, Unit, Procedure, Metadata) matching the analytics pages
- Added Environment, Deployment, and Version filters to Runs and Logs, plus Run ID and Duration
- Exposed previously hidden filters: started/ended/created date ranges on runs, run count and run-mode toggles on units, and a filter sidebar on batches for the first time

### Tidy Runs and Logs filter sidebars
URL: https://www.tofupilot.com/changelog/tidy-runs-and-logs-filter-sidebars

Small cleanups to the Runs and Logs filter sidebars.

- Removed the Run ID filter from the sidebars (filter by run ID via the command palette or search instead)
- Renamed the Logs "Run Outcome" filter to "Outcome"

### Logs unit filters + consistent filter names
URL: https://www.tofupilot.com/changelog/logs-unit-filters-consistent-filter-names

Logs gain the same unit filters as Runs, and filter names are now consistent across every table and analytics sidebar.

- Added Part, Revision, Batch, and Sample filters to the Logs sidebar
- Standardized date filter labels (Started At, Ended At, Created At, Timestamp) and removed redundant prefixes across Runs, Logs, Units, Batches, Measurement Control, and Phase Pareto

### Connect AI agents to your test data with MCP
URL: https://www.tofupilot.com/changelog/connect-ai-agents-to-your-test-data-with-mcp

Connect Claude, Cursor, and any MCP client to your TofuPilot data so an AI agent can query and manage your test operations directly, with no API key to copy or rotate.

- Added an MCP server you connect by authorizing once in your browser, scoped to a single organization per client
- Added read tools for runs, units, parts, batches, procedures, and stations, plus optional write access to create, update, and delete
- Added per-connection controls: read-only by default, write opt-in, role-aware access, and a connected-clients list with one-click revoke

### Cleaner MCP connection screen
URL: https://www.tofupilot.com/changelog/cleaner-mcp-connection-screen

Refined how you connect AI agents to your test data with MCP.

- Simplified the authorization screen with a clear read or read-and-write choice per connection
- Spelled out exactly what a client can access, including that write includes permanent deletes
- Surfaced MCP in the sidebar so it's easier to find and connect a client

### Develop procedures locally, upload when ready
URL: https://www.tofupilot.com/changelog/develop-procedures-locally-upload-when-ready

Iterate on a procedure in your own working directory and upload runs to the dashboard only when you choose, with no deploy step in the loop.

- Added `tofupilot link` to bind a working directory to a dashboard procedure, picking interactively or with `--procedure` for CI.
- Added `--upload` to `tofupilot run` so a linked local run syncs to the dashboard, while a plain `run` stays local for fast iteration.
- Added `TOFUPILOT_PROCEDURE_ID` to set the target in automation, and `tofupilot unlink` to remove a link.

### Fix Windows install errors on managed devices
URL: https://www.tofupilot.com/changelog/fix-windows-install-errors-on-managed-devices

Installing the CLI on a managed Windows machine could fail with an unhelpful "Access is denied" and no way to tell why.

- Fixed the Windows installer to pinpoint the cause when a blocked launch occurs, with clear next steps and a working alternate install location.
- Added a saved install log and step-by-step status so failures are easy to share and resolve.

### Fix Created At filter placement on units page
URL: https://www.tofupilot.com/changelog/fix-created-at-filter-placement-on-units-page

- Moved the Created At date filter to the top of the Unit filter group
- Kept the run-scoped Started At filter in the Run group for clarity

### Faster CLI startup, offline-friendly
URL: https://www.tofupilot.com/changelog/faster-cli-startup-offline-friendly

The CLI no longer waits on the network at startup when you are already logged in, so commands stay fast offline and on flaky connections.

- Throttled the background update check so a burst of commands makes one network call per window instead of one per command.
- Made `whoami` cache-first, showing your identity instantly and refreshing only once a day.
- Slowed the station update-check cadence to favour stability on production benches.

### Fix headless runs hanging on unit ID
URL: https://www.tofupilot.com/changelog/fix-headless-runs-hanging-on-unit-id

Headless runs needing manual unit identification could hang forever with no interface to answer the prompt.

- Fixed the hang, headless runs now fail fast with clear guidance
- Kept dashboard and station operators able to answer identification on upload and deployment runs

### The TofuPilot CLI is now open source
URL: https://www.tofupilot.com/changelog/the-tofupilot-cli-is-now-open-source

The CLI that runs, deploys, and manages your test procedures is now fully open source under the MIT license, so you can read the code, build it yourself, and contribute.

- Published the CLI at github.com/tofupilot/cli with prebuilt binaries for macOS, Linux, and Windows
- Added contributor docs, an architecture guide, and a security policy
- Unified internal error handling and expanded the test suite for a cleaner, more reliable codebase

### Simplify station deployment status view
URL: https://www.tofupilot.com/changelog/simplify-station-deployment-status-view

The station page now shows each linked procedure's current deployment at a glance instead of a dense history timeline.

- Replaced the per-procedure deployment timeline with a single row showing the latest commit and a Pushed / Not pushed status
- Added clear empty states for stations with no linked procedures and procedures with no deployment yet

### CLI now installs and updates in China
URL: https://www.tofupilot.com/changelog/cli-now-installs-and-updates-in-china

CLI install and update downloads were blocked in mainland China because they were served from GitHub, which is unreachable behind the Great Firewall without a VPN.

- Mirrored every CLI release to a global edge network reachable from China
- Switched all binary downloads to that mirror so every client uses one fast, China-reachable path
- Verified the latest release is reachable in-region so existing installs keep updating without interruption

### Refine station and procedure deployment views
URL: https://www.tofupilot.com/changelog/refine-station-and-procedure-deployment-views

Cleaner station and procedure pages make it easier to see what is deployed where and to manage deployments.

- Improved station deployment items: full-width rows with the commit shown inline and a new "Remove from station" action.
- Restyled the procedure page station list to match the deployment view for a consistent look.
- Streamlined the station setup tab with clearer framework support (TofuPilot Framework, OpenHTF, pytest, Robot Framework) and a simpler API key empty state.

### Redesigned framework documentation
URL: https://www.tofupilot.com/changelog/redesigned-framework-documentation

The TofuPilot Framework docs are reorganized into a clearer, deeper reference with interactive previews you can read at a glance.

- Added dedicated pages for procedures, phases, measurements, operator UI, plugs, unit identification, execution, logs, attachments, reports, and environments.
- Added live previews of measurements and operator UI components, each paired with a text rendering so AI assistants and screen readers get the same information.
- Improved the framework home page with a concise overview and direct links to every concept.

### Simplify station login with setup tokens
URL: https://www.tofupilot.com/changelog/simplify-station-login-with-setup-tokens

Logging in as a station now uses a single, consistent flow. `tofupilot login` authenticates you as a user in the browser, and stations register with a setup token from the dashboard.

- Removed the `--station` login flag in favor of setup tokens for every station.
- Updated the station login errors to point you straight to the dashboard's setup-token page.
- Streamlined the login backend so user and station credentials follow separate, dedicated paths.

### Fix CLI setup for stations in China
URL: https://www.tofupilot.com/changelog/fix-cli-setup-for-stations-in-china

Test stations in China could not complete first-time CLI setup because a required Python tooling download came from a host that is unreachable there.

- Fixed station setup in China by serving the Python environment manager from TofuPilot's own download infrastructure
- Added a configurable download source for self-hosted and air-gapped deployments

### See live run progress in the terminal
URL: https://www.tofupilot.com/changelog/see-live-run-progress-in-the-terminal

Running a procedure with the CLI now shows what's happening as it happens, so a run never finishes silently — and a failure always explains itself.

- Added a live console stream of phases, measurements, logs, and the final outcome to every `tofupilot run`.
- Surfaced the full error and traceback when a phase or run fails, instead of a bare status.
- Defaulted the interactive terminal UI on, and quieted internal plug startup chatter.

### Drill from charts into phases and measurements
URL: https://www.tofupilot.com/changelog/drill-from-charts-into-phases-and-measurements

Analytics charts now let you drill straight from a high-level view down to the runs, phases, and measurements behind any data point, carrying your filters the whole way.

- Added right-click drill-down across the run analytics, phase, and measurement charts so you can jump from runs to phases to individual measurements in a few clicks.
- Added clickable outcome tiles and a "Pass Rate" view to the phase and measurement charts, with a Pareto 80% line to surface the vital few.
- Improved filter carry-over so the date window, outcomes, parts, batches, and other filters follow you across every drill-down.

### Fix CLI login approval in the browser
URL: https://www.tofupilot.com/changelog/fix-cli-login-approval-in-the-browser

Approving `tofupilot login` from the browser was failing with a "Device denied" error, blocking new CLI sign-ins.

- Fixed the device-approval page so the login code is recognized when you approve it.
- `tofupilot login` now completes as expected from both the CLI link and a manually entered code.

### Add full JSON output and clear command feedback
URL: https://www.tofupilot.com/changelog/add-full-json-output-and-clear-command-feedback

Scripts and agents driving the CLI needed every command to speak JSON, and several commands used to succeed without printing anything.

- Added `--json` output to whoami, link, unlink and service status, and made API command errors return structured JSON instead of plain text
- Added success messages to all create, update and delete commands (`Created part <id>`) so they no longer succeed silently
- Fixed environment bootstrap output corrupting the `--json` event stream during runs

### Set up a station with one login command
URL: https://www.tofupilot.com/changelog/set-up-a-station-with-one-login-command

Turning a machine into a station no longer needs a separate install step, so the dashboard's one-line setup command fully provisions the station on its own.

- Added automatic boot-service setup on station login, so the station restarts after a reboot with no extra command.
- Removed the boot service automatically when you log back in as yourself, making login the single command to switch a bench between station and development use.
- Removed the separate install command, which is now redundant.

### Filter procedures as you type in the CLI
URL: https://www.tofupilot.com/changelog/filter-procedures-as-you-type-in-the-cli

The interactive procedure and deployment pickers in `tofupilot link` and `tofupilot run` now filter as you type, so you no longer page through long lists to find one.

- Added fuzzy filtering to the link and run pickers — start typing to narrow the list instantly.
- Showed a short id next to each procedure name so duplicate names are easy to tell apart.
- Fixed `link --procedure <name>` to report an error and list matching ids when a name is ambiguous, instead of silently picking the first match.

### Document JSON measurements
URL: https://www.tofupilot.com/changelog/document-json-measurements

JSON measurements let you capture structured objects and arrays such as device state, config snapshots, and diagnostics, and now have their own documentation page.

- Added a JSON measurement guide covering how values are captured, how the type is detected, and the validators it supports.
- Added a live preview that renders a JSON payload the way it appears in the dashboard.
- Clarified that numeric matrices stay multi-dimensional while objects and other arrays become JSON measurements.

### Use any name for your procedure YAML file
URL: https://www.tofupilot.com/changelog/use-any-name-for-your-procedure-yaml-file

Your procedure file no longer has to be named procedure.yaml. Point your deployment's entry point at any YAML file and TofuPilot runs it.

- Added support for custom-named procedure YAML files (any `.yaml`/`.yml`) via the deployment entry point.
- Improved repository import so a custom-named procedure is detected and offered in the picker instead of being reported as missing.
- Fixed monorepo builds so a sibling procedure with a custom-named YAML ships as source instead of being skipped.

### Add image cards to radio and checklist options
URL: https://www.tofupilot.com/changelog/add-image-cards-to-radio-and-checklist-options

Radio and checklist components now render as selectable image-card grids when their options include images, replacing the separate image choice and image checklist component types.

- Added an `image` field on radio and checklist options that switches rendering to an image-card grid in the operator UI, with `columns`, `aspect`, and `fit` layout controls
- Removed the `image_choice` and `image_checklist` component types; procedures should use `radio` and `checklist` with image options instead
- Merged the image choice and image checklist documentation into the radio and checklist pages, with redirects from the old URLs

### Add queue get, export and single-entry retry
URL: https://www.tofupilot.com/changelog/add-queue-get-export-and-single-entry-retry

Stations that run offline queue their runs locally, and operators needed better tools to inspect, recover, and clean up stuck uploads from the CLI.

- Added `queue get` and `queue export` to inspect a queued run's failure detail and archive its upload payload before removing it
- Added single-entry retry that reports the created run ID or the persisted failure record with `--json`, and exit codes that signal when entries remain queued
- Renamed `queue drop` to `queue rm` and added lifecycle status (parked, backoff, attachments pending) plus attempt counts to `queue ls`

### Faster dashboard startup and navigation
URL: https://www.tofupilot.com/changelog/faster-dashboard-startup-and-navigation

Opening the dashboard previously fired six separate data fetches in sequence just to render the sidebar, and server startup waited on analytics tooling before serving the first request.

- Improved sidebar loading by fetching organization, plan, and profile data together in a single round-trip
- Reduced server cold-start time by deferring analytics initialization and trimming the server bundle
- Reduced the script payload on every page by loading the live-updates client only when it connects

### Export phases and measurements to CSV
URL: https://www.tofupilot.com/changelog/export-phases-and-measurements-to-csv

Phase Pareto and Measurement Control are the two analytics tables that could not be exported, so getting their data into a spreadsheet meant copying rows by hand.

- Added CSV and JSON export to the phase and measurement lists, available from the selection bar once rows are selected.
- Added a copy-to-clipboard option so selected rows can be pasted straight into a spreadsheet.
- Kept numeric, boolean and text measurement values in separate columns so numbers stay sortable in the exported file.

### Catch YAML typos and fix concurrent run crashes
URL: https://www.tofupilot.com/changelog/catch-yaml-typos-and-fix-concurrent-run-crashes

Running several CLI processes on one machine is now reliable, and mistakes in procedure YAML surface immediately instead of being silently ignored.

- Fixed concurrent runs and station commands interfering with each other through the shared state database: the lock is now held only while in use and contention waits politely instead of interrupting other processes
- Added strict procedure YAML validation: unknown fields like `phases:` or `value:` now fail with a clear error naming the field instead of being silently ignored
- Fixed orphaned Python worker processes surviving after runs, `tofupilot run file.yml` ignoring the given file, and timeout phases reporting zero duration

### Expand CLI docs with new command references
URL: https://www.tofupilot.com/changelog/expand-cli-docs-with-new-command-references

Every code example in the CLI and framework docs is now verified to run as written, and the CLI's automation surface is fully documented.

- Added reference pages for the imports, logs, phases, measurements and deployments commands
- Added a complete `run --json` event-stream reference covering the protocol, every event type, stdin commands and headless CI recipes
- Updated every YAML and Python example to the current procedure schema so copy-pasted snippets work out of the box

### Improve scheduled workflow reliability
URL: https://www.tofupilot.com/changelog/improve-scheduled-workflow-reliability

Scheduled workflows and background maintenance now run on dedicated platform schedules instead of inside request servers, removing a class of intermittent failures.

- Fixed intermittent connection errors that could disrupt scheduled workflow runs
- Fixed slow responses on station event ingestion caused by background work competing with requests
- Improved sign-in cookie handling and pinned strict database connection security

### Speed up procedure list initial load
URL: https://www.tofupilot.com/changelog/speed-up-procedure-list-initial-load

The procedure list now appears immediately when opening the dashboard, instead of after a blank loading spinner.

- Improved initial load by rendering procedure cards, yield circles, and run sparklines directly in the first page response, removing a full client roundtrip
- Added skeleton placeholders and kept previous results visible while searching, so the page never flashes empty
- Added a "/" keyboard shortcut to focus the search bar on list pages, and create buttons no longer pop in after loading

### Improve run validation and API error messages
URL: https://www.tofupilot.com/changelog/improve-run-validation-and-api-error-messages

Run uploads now validate timestamps before anything else, and API errors are shorter and clearer.

- Added validation rejecting runs and phases whose end time precedes the start time, with a message naming the field
- Improved database error messages in the MCP interface to short, readable descriptions

### Speed up the dashboard load (1)
URL: https://www.tofupilot.com/changelog/speed-up-the-dashboard-load-1

Data pages now load noticeably faster, starting with the logs page.

- Fixed data being fetched twice on page load across logs, API activity, stations, workflows, batches, and measurement analysis pages
- Added an instant loading skeleton to the logs page so it no longer waits for data before appearing
- Reduced background work during startup so pages become interactive sooner

### Deploy procedures from the CLI
URL: https://www.tofupilot.com/changelog/deploy-procedures-from-the-cli

Ship a procedure's local source straight from your machine, no git push required.

- Added `tofupilot deploy`: packs the linked directory, builds it in the cloud, and streams build logs to your terminal
- Added `--prod` to ship to every linked station; preview by default
- Streamed build status and logs live until the build finishes

### Show CLI deployments in the dashboard
URL: https://www.tofupilot.com/changelog/show-cli-deployments-in-the-dashboard

Deployments pushed with `tofupilot deploy` now appear alongside git deployments.

- Added source deployments to the deployments list and detail, labeled "Uploaded from CLI"
- Showed the source hash in place of a commit for CLI deployments

### Fix run uploads behind a private CA
URL: https://www.tofupilot.com/changelog/fix-run-uploads-behind-a-private-ca

A station logged in to a self-hosted instance with `--ca-cert` showed online, but its run uploads never reached the server: the upload path did not trust the private CA unless it was also installed in the OS trust store.

- Fixed run uploads, the offline upload queue and `tofupilot link` to trust the certificate passed with `--ca-cert`, as login and the live link already did
- Improved CA file handling: the file is read once per process and a missing or malformed file warns once instead of on every upload
- Added a build-time guard so every network client in the CLI carries the configured CA

### Smoother, consistent analytics chart loading
URL: https://www.tofupilot.com/changelog/smoother-consistent-analytics-chart-loading

Analytics charts now load and render more consistently across the run, phase, and measurement views.

- Added uniform loading skeletons that fill the whole chart area, so charts no longer flash a spinner or jump when data arrives.
- Fixed pareto charts overflowing past the card edge, which hid bars behind the scrollbar and left an unusable horizontal scroll.

### Speed up the settings page
URL: https://www.tofupilot.com/changelog/speed-up-the-settings-page

Opening and navigating organization settings is now noticeably lighter and faster.

- Removed dozens of unnecessary background requests triggered on every settings visit
- Improved how organization logos and avatars load, so they no longer re-download on every visit

### Exact histogram for large numeric measurements
URL: https://www.tofupilot.com/changelog/exact-histogram-for-large-numeric-measurements

On numeric measurements with more than 500 points, the control chart histogram was built from a sample of the data, so the distribution was approximate and rare out-of-spec values could be missing from the bars while the OOS counter showed them.

- Improved the numeric histogram to show the exact distribution over all points, matching the boolean and string charts
- Fixed out-of-spec outliers disappearing from the histogram on large datasets; the value range now always covers them
- Fixed the histogram tooltip ratio, which could overstate bin proportions on sampled data

### Fix sidebar filters not updating list pages
URL: https://www.tofupilot.com/changelog/fix-sidebar-filters-not-updating-list-pages

Selecting a sidebar filter on the runs, units, logs, and other list pages updated the URL and the counts but the table kept showing unfiltered rows until a page reload. This was a short-lived regression introduced in the previous daily release.

### Faster list pages and smoother tab switching
URL: https://www.tofupilot.com/changelog/faster-list-pages-and-smoother-tab-switching

Data pages now show their layout instantly while results load, and the dashboard no longer reloads everything when you return to the tab.

- Added instant loading skeletons to units, batches, parts, stations, deployments, commits, and API logs pages so they appear immediately instead of waiting on data
- Stopped the dashboard from refetching every open query when you switch back to its browser tab
- Sped up the measurement table by removing redundant per-row database work

### Speed up the run analytics page
URL: https://www.tofupilot.com/changelog/speed-up-the-run-analytics-page

The run analytics page now loads its charts with the data already in place, instead of fetching it twice after the page appeared.

- Removed a duplicate data fetch on the default view, so charts render with their numbers immediately on load
- Fixed "Open in Table" links so the Runs and Units tables open scoped to the exact window shown in the charts
- Loaded part and revision thumbnails in the filters in a single request, removing avatar flicker when changing filters or dates

### Speed up the phase pareto page (2)
URL: https://www.tofupilot.com/changelog/speed-up-the-phase-pareto-page-2

The phase pareto page now loads its filters from the same recent window as the chart, instead of scanning a wide history range on first open.

- Fixed a cold-load delay where the sidebar filters ran an unbounded query before the page was ready
- Loaded the phase, outcome, and sample-class filter counts from the default last-week window, so they appear with the chart instead of after a long wait

### Speed up the phase pareto page (1)
URL: https://www.tofupilot.com/changelog/speed-up-the-phase-pareto-page-1

The failures table below the pareto chart now loads with the page instead of fetching separately after it appears.

- Loaded the phase failures table from the server so it shows up immediately on first paint
- Scoped the table to the same time window as the chart above it, so both show the same period
- Stopped sidebar filter counts from loading while the sidebar is collapsed

### Speed up the timeline charts
URL: https://www.tofupilot.com/changelog/speed-up-the-timeline-charts

The timeline chart above runs, logs, and units tables used to show a blank band while the page loaded, then pop in with a delay.

- Added chart bars directly into the initial page response, so the timeline is visible the moment the page appears instead of after a blank pause
- Improved loading placeholders to match the chart and table layout exactly, removing visible jumps when content arrives
- Fixed charts disappearing into a loading state on every filter change; the previous chart now stays visible while new data loads

### Streamline member offboarding
URL: https://www.tofupilot.com/changelog/streamline-member-offboarding

Offboarding a team member is now a single action with nothing left to clean up by hand.

- Removing a member now automatically deactivates their organization API keys and MCP connections in one step
- Improved the remove and re-invite flow so returning members get a fresh start while their test history stays intact
- Updated the member management docs to walk through the new offboarding flow

### Speed up the usage page
URL: https://www.tofupilot.com/changelog/speed-up-the-usage-page

The usage page now loads in well under two seconds instead of over four, so billing and consumption analytics appear almost instantly.

- Improved usage analytics query speed by collapsing eight database round trips into two and adding date-range indexes
- Added instant rendering from a server-prefetched cache, removing the wait for the page to become interactive before data loads
- Improved annual billing views to show a clean monthly breakdown instead of 365 daily bars
- Fixed empty monthly and weekly charts for users in time zones ahead of UTC

### Reset filters from empty chart placeholders
URL: https://www.tofupilot.com/changelog/reset-filters-from-empty-chart-placeholders

When an analytics chart comes up empty because your filters excluded everything, you can now clear them right from the placeholder.

- Added a Reset action to the empty state on the run, phase, and measurement analytics charts, so you can recover from an over-filtered view in one click.
- The placeholder now tells you when active filters are the reason a chart is empty, and only offers Reset when there's actually a filter to clear.

### Keep the auto-started run when you click Run
URL: https://www.tofupilot.com/changelog/keep-the-auto-started-run-when-you-click-run

On a station with one procedure, clicking Run in the kiosk while the station was still starting that procedure could stop the run and leave the station stuck until a reboot.

- Fixed a stop or kill that landed while an instrument was still connecting: the run now ends in under a second instead of hanging, and the instrument is released.
- Improved the kiosk at boot: it shows the run that is starting instead of the procedure picker, and a click on Run during start-up no longer restarts it.
- Fixed a kiosk page opened during a run start that could miss the run's first prompt.

### Speed up the phase pareto page (3)
URL: https://www.tofupilot.com/changelog/speed-up-the-phase-pareto-page-3

Heavy pages now paint faster by streaming less work to the browser and scanning less data on the server.

- Rebuilt the timeline chart on list pages (runs, units, logs) to render instantly without a charting library, cutting script work and layout jank
- Made the phase pareto page's data queries scan a bounded set of runs instead of the procedure's full history, removing the multi-second wait before the chart appears

### Speed up the dashboard load (2)
URL: https://www.tofupilot.com/changelog/speed-up-the-dashboard-load-2

Dashboard pages now paint faster on first load by deferring heavy code that isn't needed until you open a detail panel.

- Cut the JavaScript loaded before a page becomes interactive by nearly half
- Moved the code viewer and diagram rendering to load on demand instead of on every page
- Reduced the analytics script footprint on initial load

### Speed up measurement control links and sidebar
URL: https://www.tofupilot.com/changelog/speed-up-measurement-control-links-and-sidebar

Opening a measurement control link that already had a measurement selected made every sidebar filter fetch its options separately, even though the data was already loaded with the page.

- Improved shared links with a measurement selection: sidebar filters now reuse the data delivered with the page instead of firing six extra requests
- Improved internal consistency: measurement control now uses the same first-load data handling as all other list pages, closing the gap that caused the recent sidebar filter bug
- Fixed an edge case where certain link parameters unnecessarily skipped the fast first load

### Show procedure images in the operator view
URL: https://www.tofupilot.com/changelog/show-procedure-images-in-the-operator-view

Reference images defined in your procedure YAML (radio and checklist option images, image components) now load everywhere an operator runs a test, not just in Studio.

- Added image resolution to the dashboard operator view and the CLI kiosk so relative image paths from your procedure load correctly
- Fixed radio and checklist option image grids that had stopped showing images
- Corrected the image component docs to use `default_value`

### Speed up the list pages
URL: https://www.tofupilot.com/changelog/speed-up-the-list-pages

Clicking a filter used to flash loading placeholders across the whole sidebar, lag the checkbox tick, and swap the table to skeleton rows on every change.

- Fixed checkboxes so they respond instantly on click, with options and counts staying visible while updated numbers load in the background
- Improved the table to keep showing results during filter changes instead of clearing to a loading state, and selections now reset when filters change
- Added smooth open and close animations to filter sections, clear retry messaging when options fail to load, and consistent loading placeholders across all filter types

### Speed up the procedure picker
URL: https://www.tofupilot.com/changelog/speed-up-the-procedure-picker

The procedure picker now opens instantly instead of waiting on a load when you land on it or open the switcher in the top bar.

- Removed the ~680ms load when opening the full-page procedure selector by serving its first results with the page itself
- Added background warming of the top-bar procedure switcher so it is ready before you click
- Kept search instant and unchanged

### Speed up the phase pareto page (5)
URL: https://www.tofupilot.com/changelog/speed-up-the-phase-pareto-page-5

The phase pareto page is now consistently fast for every procedure, including large ones, after finishing the move to a faster data path.

- Finished switching the page's queries to the faster lookup so the chart and table load quickly even on procedures with millions of phases
- Rolled out automatically in the background with no downtime

### Speed up the phase pareto page (4)
URL: https://www.tofupilot.com/changelog/speed-up-the-phase-pareto-page-4

The phase pareto page now opens in well under a second instead of several seconds, matching the other analytics pages.

- Reworked how the page loads its data so the chart appears almost instantly
- Rolled out in the background with no downtime, switching to the faster version automatically once ready

### Speed up the filter sidebars
URL: https://www.tofupilot.com/changelog/speed-up-the-filter-sidebars

The filter sidebars on the runs and units pages now load their option counts much faster, even on large organizations.

- Sped up how filter option counts are calculated, several times faster on busy pages
- Fixed revision number filters missing matches when the number was stored with different capitalization

### Speed up the units page filters
URL: https://www.tofupilot.com/changelog/speed-up-the-units-page-filters

The units page filter sidebar now loads quickly even when you combine unit filters with run-based ones like outcome or operator.

- Extended the faster filter counts to the units page's mixed filter combinations
- Kept results identical to before, just faster

### Fix sidebar loading and reset behavior
URL: https://www.tofupilot.com/changelog/fix-sidebar-loading-and-reset-behavior

Two filter-sidebar fixes on the phase pareto and measurement control pages.

- Fixed filters briefly showing "No options available" while loading; they now show a loading skeleton until the options arrive
- Fixed the Reset button so it also clears the selected phase or measurement, returning the page to a clean state

### Speed up the phase pareto page (6)
URL: https://www.tofupilot.com/changelog/speed-up-the-phase-pareto-page-6

The phase pareto page now loads faster by skipping work most views never need.

- Stopped computing the sample-class breakdown on every load; it now loads only when you open the Sample filter, cutting the page's main query time
- Sample counts stay live and accurate, updating immediately when a unit's sample type changes

### Add a page loading bar
URL: https://www.tofupilot.com/changelog/add-a-page-loading-bar

Navigating between pages now shows a thin progress bar at the top, so it's clear when the next page is loading.

- Added a slim loading bar that animates across the top during navigation
- Matches the light and dark themes

### Show a clear error when a page fails to load
URL: https://www.tofupilot.com/changelog/show-a-clear-error-when-a-page-fails-to-load

When a list page failed to load, it used to look empty, as if there was no data. It now shows a clear error with a retry button.

- Added a proper error state with a "Try again" button to the runs, units, logs, batches, and other list pages
- A failed load no longer looks like an empty page

### Fix changelog crash when scrolling for more
URL: https://www.tofupilot.com/changelog/fix-changelog-crash-when-scrolling-for-more

Scrolling to load more entries on the changelog, news, and guides pages could crash the page instead of fetching the next batch.

- Fixed the load-more pagination so infinite scroll reliably loads older entries on changelog, news, and guides.

### Show a real error state on failed list loads
URL: https://www.tofupilot.com/changelog/show-a-real-error-state-on-failed-list-loads

When a list page failed to load, a network or server error looked identical to having no data — now it's clearly an error you can retry.

- Added a dedicated error state with a "Try again" action across runs, units, logs, batches, API activity, and procedures.
- Kept existing rows on screen when a "load more" fails, so a failed fetch no longer wipes the list.

### Speed up dashboard load and rendering
URL: https://www.tofupilot.com/changelog/speed-up-dashboard-load-and-rendering

The dashboard now loads less code up front and re-renders more efficiently, so pages feel faster across the board.

- Trimmed the JavaScript loaded on every page, with the heaviest detail pages cut the most
- Charts, search, and other heavy panels now load on demand instead of up front
- Enabled automatic rendering optimizations for smoother interactions

### Speed up the station page
URL: https://www.tofupilot.com/changelog/speed-up-the-station-page

The station detail page now loads noticeably less code up front, so it opens faster.

- Removed a heavy charting library from the initial load by drawing the telemetry sparklines directly
- Loaded the realtime connection only when it's actually needed
- Cut about 900KB of JavaScript from the page's first load

### Speed up measurement, chat, and report pages
URL: https://www.tofupilot.com/changelog/speed-up-measurement-chat-and-report-pages

Charts on the measurement, chat, and report pages now load on demand instead of up front, so the pages open faster.

- Deferred chart rendering until it's actually shown
- Loaded the code viewer only for measurements that use it
- Trimmed the initial load across all three page types

### Trim JavaScript loaded on every page
URL: https://www.tofupilot.com/changelog/trim-javascript-loaded-on-every-page

Reduced the shared code that every dashboard page loads, so navigation feels lighter across the app.

- Shipped only the English text for validation messages instead of every language
- Loaded search, notifications, and analytics on demand instead of up front
- Cut roughly 300KB from the code shared by every page

### Smoother, more reliable dashboard rendering
URL: https://www.tofupilot.com/changelog/smoother-more-reliable-dashboard-rendering

Tightened the checks behind the dashboard's automatic rendering optimizations, so the UI stays fast and dependable as it grows.

- Strengthened safeguards around how components render and update
- Fixed several edge cases surfaced by the new checks
- Resolved two subtle display glitches in the process

### Fix measurement control page timeouts
URL: https://www.tofupilot.com/changelog/fix-measurement-control-page-timeouts

The measurement control page could time out on procedures with millions of measurements, leaving the Pareto charts unable to load.

- Fixed the measurement control page timing out on large procedures, so the Pareto and Cpk charts load reliably.
- Improved how historical measurement data is aggregated in the background, keeping the page fast as your data grows.
- Fixed the live runs toggle getting stuck on "Reconnecting" after a connection drop.

### Fix uneven gaps in timeline chart bars
URL: https://www.tofupilot.com/changelog/fix-uneven-gaps-in-timeline-chart-bars

The activity timeline above each list now renders with consistent spacing between bars.

- Fixed uneven horizontal gaps between bars in the timeline chart shown on the runs, units, logs, and API activity pages.
- Aligned the chart's spacing with its loading skeleton so the layout no longer shifts as data loads.

### Speed up public content pages
URL: https://www.tofupilot.com/changelog/speed-up-public-content-pages

Changelog, news, guides, templates, and roadmap detail pages now load faster on first view.

- Cached each public detail page behind a single shared, tagged getter, cutting roughly four uncached database round-trips per view down to one.
- Busted the cache automatically on publish so updates still appear immediately.

### Speed up homepage hero image load
URL: https://www.tofupilot.com/changelog/speed-up-homepage-hero-image-load

The homepage hero image was loading at low priority and arriving ~1.4s into page load, hurting perceived speed.

- Fixed the high-priority hint on the theme-aware hero image so it now downloads first.
- Marked the visible hero image as eager so the browser stops deferring the largest content element, dropping load to under 800ms.

### Document Python version selection
URL: https://www.tofupilot.com/changelog/document-python-version-selection

Clarifies how TofuPilot picks the Python version for a procedure so users avoid "No interpreter found" and package-incompatibility errors.

- Added a Python version section explaining selection happens at the minor level (3.12), not the patch (3.12.4), with supported minors 3.10 to 3.14.
- Documented the gotcha for packages limited to older minors and how to cap the upper bound in requires-python.
- Noted that air-gapped stations can only use Python minors already installed locally, since provisioning a new minor requires fetching the interpreter.

### Show attachment images in the operator UI
URL: https://www.tofupilot.com/changelog/show-attachment-images-in-the-operator-ui

The CLI kiosk and the dashboard operator UI now support attachment images. Attach a board photo, probe shot, or failure close-up during a run and it shows inline next to the phase instead of as a bare filename, so the operator can compare it against the unit while the run is live.

- Added inline image rendering for run attachments in the CLI kiosk (served locally) and the dashboard operator UI (served after upload)
- Fixed station runs so attachment images reach the dashboard, matching OpenHTF and Robot runs
- Improved deployment history to show who ran a CLI deploy instead of "unknown"

### Run stations as root with launch on boot
URL: https://www.tofupilot.com/changelog/run-stations-as-root-with-launch-on-boot

Test stations that need root access to their hardware can now start automatically at boot. Previously enabling launch on boot as root failed because the station installed a user-level service.

- Added automatic system-service install when the station runs as root, so launch on boot works without a login session.
- Hid the on-screen kiosk option for root stations, which have no display, and made clear that these stations are controlled from the dashboard.
- Showed a clear, actionable message instead of a raw systemd error when launch on boot cannot be enabled.

### Deploy on any Python from 3.10 to 3.14
URL: https://www.tofupilot.com/changelog/deploy-on-any-python-from-310-to-314

Deployments now build on the Python version your procedure declares, instead of always on 3.12. A procedure pinning `requires-python = ">=3.13"` previously failed to build with "No interpreter found for Python 3.13".

- Added support for building procedures on Python 3.10 through 3.14, selected from your `pyproject.toml` `requires-python`.
- Improved error messages when a version can't be satisfied, pointing you to a supported range instead of failing cryptically.
- Kept the default at 3.12 when `requires-python` is omitted, so existing procedures build unchanged.

### Support pre-baked answers for OpenHTF prompts
URL: https://www.tofupilot.com/changelog/support-pre-baked-answers-for-openhtf-prompts

Headless and automated OpenHTF runs can now answer operator prompts ahead of time, so a prompt no longer stalls an unattended run.

- Fixed operator prompts ignoring pre-baked input values during automated runs, which caused them to wait and eventually time out.
- Added automatic prompt resolution when every required input is supplied up front, matching the behavior of native procedures.

### Fix guide and template pages failing to load
URL: https://www.tofupilot.com/changelog/fix-guide-and-template-pages-failing-to-load

Guide and template pages were briefly returning an error instead of loading.

- Fixed an issue that prevented guide and template pages from opening
- Restored these pages so they load correctly again

### Fix update failures from network blips
URL: https://www.tofupilot.com/changelog/fix-update-failures-from-network-blips

A transient connection drop right after a network switch or wake-from-sleep could abort `tofupilot update` even when the server was healthy, and an immediate re-run would succeed.

- Fixed `tofupilot update` failing on a single transient network error when checking for the latest version
- Added automatic retries with backoff so a brief connection hiccup no longer interrupts the update

### Fix attachment upload from the CLI
URL: https://www.tofupilot.com/changelog/fix-attachment-upload-from-the-cli

Attachments produced during a CLI run now upload reliably and display correctly on the dashboard.

- Fixed `attach.data` attachments being silently dropped on native test runs so they never reached the dashboard.
- Fixed `attach.file` attachments uploading without a size, type, or image preview.
- Improved cleanup so attachment files left by a failed or aborted run no longer pile up on disk.

### Retry interrupted update downloads
URL: https://www.tofupilot.com/changelog/retry-interrupted-update-downloads

A connection drop during the binary download — common after a network switch or wake-from-sleep — could abort `tofupilot update` partway through, even when the server was healthy.

- Fixed `tofupilot update` failing when the connection dropped mid-download
- Added automatic retries with backoff so an interrupted transfer restarts cleanly instead of aborting the update

### Fix kiosk operator UI on root test stations
URL: https://www.tofupilot.com/changelog/fix-kiosk-operator-ui-on-root-test-stations

Headless test jigs that run as root could no longer open the kiosk operator UI, leaving runs stuck on unit identification.

- Fixed the kiosk operator UI failing to start for foreground `run --kiosk` on root stations.
- Restored remote viewing of the kiosk over an SSH port-forward on headless machines.
- Kept the root station service hardened: its command channel still refuses to bind as root.

### Fix station logout when deploying as a user
URL: https://www.tofupilot.com/changelog/fix-station-logout-when-deploying-as-a-user

Running `tofupilot deploy` on a station required a user login, which previously overwrote the station's own credentials and stopped its service from starting.

- Fixed user and station logins so they no longer overwrite each other; a station stays authenticated after you log in as a user to deploy.
- Improved the station service so it keeps running after a user login on the same machine.
- Updated existing stations to migrate automatically on upgrade, with no re-login required.

### Use system Python packages in deployments
URL: https://www.tofupilot.com/changelog/use-system-python-packages-in-deployments

Procedures can now use Python packages installed on the station that pip can't install, like native instrument drivers and vendor SDKs.

- Added a "System packages" toggle on the station Setup page that lets deployment environments reach the machine's system Python packages.
- Kept deployments isolated by default; the setting is opt-in per station and applies on the next deploy.

### Fix operator answers not saving from terminal
URL: https://www.tofupilot.com/changelog/fix-operator-answers-not-saving-from-terminal

Operator answers to radio, dropdown, and text prompts now record correctly no matter how a run is launched, fixing a crash when a later phase read a measurement from an earlier prompt.

- Fixed bound measurements not being saved on terminal (`tofupilot run`) and automated `--json` runs, which previously worked only in the kiosk and Studio.
- Fixed text prompts skipping length and pattern validation on automated runs, so invalid values are now rejected consistently across every surface.
- Improved prompts to record their default value when left unanswered, matching the kiosk.

### Fix duplicate deployments from GitLab pushes
URL: https://www.tofupilot.com/changelog/fix-duplicate-deployments-from-gitlab-pushes

A single GitLab push could appear as several identical deployments when a repository had accumulated more than one webhook.

- Fixed GitLab webhook registration so each repository keeps exactly one hook, removing stale duplicates automatically.
- Added a safeguard that ignores repeated deliveries of the same push, so one commit produces one deployment per procedure.

### Fix desktop shortcut on localized systems
URL: https://www.tofupilot.com/changelog/fix-desktop-shortcut-on-localized-systems

Installing the station desktop launcher on a non-English system put the shortcut in the wrong place, so it never appeared for the operator.

- Fixed the launcher to use the system's localized Desktop folder (such as Bureau on French installs) instead of always creating an English "Desktop" folder.
- Aligned shortcut install, detection, and removal across Linux, Windows, and macOS so they all target the same folder, including OneDrive-redirected desktops on Windows.

### Speed up dashboard with infrastructure upgrade
URL: https://www.tofupilot.com/changelog/speed-up-dashboard-with-infrastructure-upgrade

We've improved our backend infrastructure to make the dashboard faster and more reliable.

- Improved database response times by running compute closer to the data layer
- Fixed a class of transient connection errors that could occasionally cause page-load failures
- Reduced latency for dashboard requests

### Rename Cycle Time metric to Average Duration
URL: https://www.tofupilot.com/changelog/rename-cycle-time-metric-to-average-duration

The analytics metric labeled "Cycle Time" actually measures average per-run test duration, not the manufacturing inter-completion cadence the term implies. Renamed it to remove the ambiguity for test and process engineers.

- Renamed the "Cycle Time" KPI card, duration chart tooltip, and phase analysis headline to "Average Duration"
- Clarified that the value reflects average passing-run execution time, not unit-to-unit cadence

### Speed up list pages and refine the timeline
URL: https://www.tofupilot.com/changelog/speed-up-list-pages-and-refine-the-timeline

The runs, units, logs, and API activity pages now paint their timeline chart instantly while keeping the richer interactive look.

- Improved first paint on list pages by loading the interactive chart layer off the critical path, so the bars appear immediately.
- Restored the smoother timeline look with hover tooltips and drag-to-filter range selection.
- Evened out the table header placeholders so columns line up while a page loads.

### Add MCP icon and group it under Integrations
URL: https://www.tofupilot.com/changelog/add-mcp-icon-and-group-it-under-integrations

The MCP server now lives alongside your other connections for easier discovery.

- Added the MCP logo to the settings sidebar and connected-client list
- Moved MCP into the Integrations section next to GitHub, GitLab, and Linear

### Speed up Run again on the station kiosk
URL: https://www.tofupilot.com/changelog/speed-up-run-again-on-the-station-kiosk

On a station, clicking Run again or New run on the outcome screen waited for the previous run's upload to the dashboard before the next run could start, and Stop and form responses queued behind it.

- Fixed the next run starting only after the previous upload finished: the upload now completes in the background, so the next run starts as soon as the previous one has released its instruments
- Fixed a 10 second stall after each run on an idle outcome screen that delayed the dashboard's live feed and the station exit
- Fixed a late upload of a previous run stamping its dashboard link on the run that followed, and kept the per-run log complete up to the upload result

### Sign and notarize macOS CLI builds
URL: https://www.tofupilot.com/changelog/sign-and-notarize-macos-cli-builds

macOS now runs the TofuPilot CLI without Gatekeeper blocking it, so downloaded builds launch without the "cannot be opened because the developer cannot be verified" warning.

- Signed the macOS CLI binary for both Apple Silicon and Intel with a new Apple Developer ID certificate
- Notarized every build with Apple so it passes Gatekeeper checks on first run

### Make filters harder to break
URL: https://www.tofupilot.com/changelog/make-filters-harder-to-break

Filters that silently matched nothing were hard to notice and easy to reintroduce. Alongside the fix, we tightened how filters are built and tested.

- Rebuilt the filter search syntax so quoted values survive intact, covered by an exhaustive test suite
- Made a misconfigured filter fail immediately instead of quietly rendering nothing
- Fixed long filter names pushing their counts out of column, and shortened them with the full name on hover

### Fix stations dropping offline mid-run
URL: https://www.tofupilot.com/changelog/fix-stations-dropping-offline-mid-run

A single oversized event — a large image in a prompt, a verbose log dump or a long measurement trace — used to cut a station's live link for the rest of the session, so runs kept passing locally while the dashboard showed the station as offline.

- Fixed the live link dropping when a phase sends an unusually large payload: the event is now shrunk to fit instead of breaking the connection.
- Added readable stand-ins for degraded content — an oversized image becomes a short note explaining the operator screen still shows the full picture, and truncated logs keep their start and end with an explicit marker.
- Kept normal runs untouched: only the payloads that would break the connection are degraded, and phase outcomes always arrive.

### Add a red STOP button to abort a run
URL: https://www.tofupilot.com/changelog/add-a-red-stop-button-to-abort-a-run

Operators could not find how to abort a running sequence: the stop control was a small grey icon that read as decoration rather than an emergency control.

- Added a solid red STOP button that morphs into KILL after a stop, with a fixed width so the bottom bar never reflows.
- Added a 500 ms delay before KILL becomes available, so a touchscreen double-tap cannot escalate a graceful stop into a forced kill.
- Fixed the button keeping the previous run's state after a restart or a lost command, and made a kill re-sendable if the run has not ended after 5 seconds.

### Run a specific Python file with tofupilot run
URL: https://www.tofupilot.com/changelog/run-a-specific-python-file-with-tofupilot-run

Projects with several test stages sharing one codebase could not validate a stage locally: local runs always fell back to main.py, so the only workarounds were renaming files or keeping separate checkouts.

- Added support for naming the entry file directly, so `tofupilot run ./stage1_entry.py` runs that stage instead of main.py, with a warning when the framework expects a directory rather than a file.
- Fixed ID columns in `tofupilot ls` being cut to eight characters, so IDs can now be copy-pasted straight into get, update, rm and link.
- Improved the logged-out and "no procedure found" messages to show the direct-file form, annotated as needing no account.

### Continue between phases with the keyboard
URL: https://www.tofupilot.com/changelog/continue-between-phases-with-the-keyboard

Operators work with their hands on a barcode scanner, not a mouse, but only phases starting with a text input kept the keyboard alive — an acknowledge prompt, a switch, a radio group or a bare Continue step forced a reach for the mouse once per phase.

- Added keyboard focus on every prompt shape, so Enter — including the trailing Enter a barcode scanner sends — acknowledges a phase and moves to the next one.
- Added Cmd/Ctrl+Enter to confirm a prompt from anywhere, including inside a multi-line text field where Enter must stay a newline.
- Fixed a held Enter skipping a prompt the operator never saw, and restored focus to the form automatically after a failed send so a retry needs no mouse.
- Added a ⏎ hint on the Continue button, shown only when Enter really confirms.

### Fix Logs sidebar filters not applying
URL: https://www.tofupilot.com/changelog/fix-logs-sidebar-filters-not-applying

Clicking a filter in the Logs sidebar now actually filters the list.

- Fixed the Level, Source, Outcome, Station, User, and Serial Number sidebar filters doing nothing when clicked
- Restored parity so sidebar filters behave the same as the search-bar filters

### Fix long commit text overflowing deploy modal
URL: https://www.tofupilot.com/changelog/fix-long-commit-text-overflowing-deploy-modal

Long commit messages and procedure names no longer break the manual deployment dialog layout.

- Fixed the commit and procedure fields overflowing past the right edge of the New Manual Deployment modal
- Improved long text to truncate cleanly with an ellipsis while keeping the commit hash and icons visible

### Launch a public questions and answers hub
URL: https://www.tofupilot.com/changelog/launch-a-public-questions-and-answers-hub

Common questions about TofuPilot now have a public home: browse direct answers by category or share a link to a single question.

- Added a questions hub at tofupilot.com/questions with category filters and instant search
- Added a dedicated page per question with related questions, so every answer has its own shareable link
- Improved search visibility with structured data and sitemap coverage, in English and French

### Choose which orgs and access each AI agent gets
URL: https://www.tofupilot.com/changelog/choose-which-orgs-and-access-each-ai-agent-gets

Connecting an AI agent through MCP now lets you pick exactly which organizations it can reach and whether each allows write.

- Added a multi-select organization picker to the MCP consent screen, with a per-organization read or write toggle
- Added org switching so an agent moves between your authorized organizations, with write access following your role in each
- Improved revoke to disconnect a client across every organization at once, not just the current one

### Manage teams and clean up data from AI agents
URL: https://www.tofupilot.com/changelog/manage-teams-and-clean-up-data-from-ai-agents

MCP now exposes more of your workspace to AI agents, so they can manage teams and remove test data the same way you do in the dashboard.

- Added team tools: list teams and members, create and rename teams, add or remove members, and delete a team
- Added delete tools for runs and units, plus tools to add or remove units from a batch

### Fix API list crash when called with no filters
URL: https://www.tofupilot.com/changelog/fix-api-list-crash-when-called-with-no-filters

A plain runs or units list call from any SDK returned a 400 error because the optional metadata filter was rejected when empty. Basic list calls now work again.

- Fixed runs.list and units.list returning "expected record, received string" when called without a metadata filter
- Fixed metadata filters not applying over the API when passed as a query parameter

### Fix Microsoft SSO sign-up with profile photo
URL: https://www.tofupilot.com/changelog/fix-microsoft-sso-sign-up-with-profile-photo

New users signing in through Microsoft or Azure AD SSO could fail to be created when their account had a profile photo, because Microsoft sends the photo as inline image data that exceeded an internal limit on the avatar field.

- Fixed account creation for Microsoft and Azure AD SSO users who have a profile photo
- Improved handling of inline profile photos so they no longer block sign-up
- Added a fallback to initials when no uploaded avatar is set

### Fix filters on names containing spaces
URL: https://www.tofupilot.com/changelog/fix-filters-on-names-containing-spaces

Filtering on a name that contains a space — a phase called "Contact Voltage Drop", a batch called "Lot 2024 rev B" — matched only the first word and returned nothing.

- Fixed filters on values containing spaces or commas, everywhere filters are used
- Added export to the Phase Pareto and Measurement Control toolbars, matching runs and units
- Added the search bar to Measurement Control

### Fix operator UI examples the CLI rejected
URL: https://www.tofupilot.com/changelog/fix-operator-ui-examples-the-cli-rejected

The Text and Progress pages documented a `value` field that the CLI does not accept, so every example on those pages failed to load when copied into a procedure.

- Fixed the Text and Progress operator UI pages to use `default_value`, the field the CLI actually reads, in all eleven YAML examples and both property tables.

### Improve API reliability under low-traffic load
URL: https://www.tofupilot.com/changelog/improve-api-reliability-under-low-traffic-load

Intermittent 60-second delays could affect API requests and station status checks during low-traffic periods, caused by background processing competing with database connections as infrastructure scaled down between requests.

- Fixed background processing after run creation so it completes reliably without delaying or blocking subsequent requests
- Improved database connection handling for internal job processing, eliminating connection errors during idle-to-active transitions
- Added a 20-second limit on background database operations so a slow query can no longer stall a request slot

### Run OpenHTF 1.3.0 through 1.6.1
URL: https://www.tofupilot.com/changelog/run-openhtf-130-through-161

Tests running an older pinned OpenHTF could not use the CLI at all. Every release from 1.3.0 onward now works the same way.

- Added support for OpenHTF 1.3.0 through 1.6.1, with the same measurements, validators, attachments and phase outcomes on every release.
- Fixed attachments being dropped on releases below 1.5.2.

### Leave nothing behind when an API call fails
URL: https://www.tofupilot.com/changelog/leave-nothing-behind-when-an-api-call-fails

A call that fails now leaves your workspace exactly as it was before you made it, on both API versions.

- Fixed failed calls so any record already written during the call is removed instead of kept.
- Stopped a fully failed import from leaving an empty entry in your import history.
- Kept the failed call visible in API Logs, so you can still see what went wrong.

### Show CLI deployments without a linked repo
URL: https://www.tofupilot.com/changelog/show-cli-deployments-without-a-linked-repo

Deploying with the CLI created a deployment you could open by link, but the Deployments page still claimed no repository was connected and listed nothing.

- Fixed the Deployments page hiding CLI deployments on procedures with no linked Git repository.
- Fixed the same blank state on the organization-wide Deployments page.
- Updated the empty state to mention `tofupilot deploy` alongside connecting a repository.

### Fix new procedures missing from list
URL: https://www.tofupilot.com/changelog/fix-new-procedures-missing-from-list

Newly created procedures now appear in the procedure list right away, without needing a page refresh.

- Fixed the procedure list not showing procedures you just created until a manual refresh
- Ensured the list and switcher both update immediately after creating a procedure

### Keep procedure lists fresh after changes
URL: https://www.tofupilot.com/changelog/keep-procedure-lists-fresh-after-changes

Creating or renaming a procedure now updates every place it appears, without needing a page refresh.

- Fixed procedures created through the import and clone setup flows not appearing in the list until a refresh
- Fixed a renamed procedure showing its old name on the dashboard cards, the procedure switcher, and the stations table
- Updated the units filter sidebar to reflect newly created and renamed procedures right away

### Use one MCP connection for all your orgs
URL: https://www.tofupilot.com/changelog/use-one-mcp-connection-for-all-your-orgs

Connect your AI agent once and reach every organization you belong to, switching between them mid-session instead of re-authorizing for each one.

- Added `list_orgs` and `set_org` tools so an agent can see your organizations and switch the active one on the fly
- Fixed the connected-clients list leaking connections across organizations
- Fixed a reconnect loop that blocked members of more than one organization
- Improved the sign-in callback so the browser no longer shows a connection error after you authorize

### Accept invitations from the org switcher
URL: https://www.tofupilot.com/changelog/accept-invitations-from-the-org-switcher

Switching between organizations is now easier, and invitations you haven't accepted yet show up where you'd expect them.

- Added pending invitations to the organization switcher, so you can accept an invite without hunting for the email
- Improved the expired-demo experience with a dedicated screen that keeps the switcher available so you can move to your other workspaces

### Invite members from AI agents, safer roles
URL: https://www.tofupilot.com/changelog/invite-members-from-ai-agents-safer-roles

You can now invite teammates to your organization directly from connected MCP, and invites can no longer grant a role above your own.

- Added a tool to invite members by email from connected AI assistants, with the invite sent automatically.
- Prevented inviting or promoting someone to a role higher than yours, closing a privilege-escalation gap across the dashboard and API.
- Filtered the invite role picker to the roles you're allowed to grant.

### Clearer procedure IDs in docs and SDK examples
URL: https://www.tofupilot.com/changelog/clearer-procedure-ids-in-docs-and-sdk-examples

Code examples now use an obviously-fake placeholder procedure ID instead of a name or a real-looking UUID, so a copy-pasted snippet can't accidentally target a real procedure and clearly reads as a value to replace.

- Replaced name-style (`FVT1`) and a real-looking UUID with `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` across the OpenHTF and SDK docs and the V1 API example.
- Clarified that `procedure_id` is the dashboard procedure UUID (a legacy external identifier only resolves if one was set).
- Updated the OpenHTF client docstrings and released python client 2.13.2.

### Deploy and run behind a strict firewall
URL: https://www.tofupilot.com/changelog/deploy-and-run-behind-a-strict-firewall

Stations on locked-down networks could not reach the third-party hosts used to download the Python runtime and to transfer deployment archives, so setup failed where IT could not allowlist them.

- Added a TofuPilot-hosted mirror for Python interpreter downloads, so station networks now only need `*.tofupilot.app`, `tofupilot.sh` and `dl.tofupilot.sh` (plus PyPI).
- Moved deployment uploads and downloads to `artifacts.tofupilot.app`, with no change to how stations work — they follow whatever address the API returns.
- Added a clear error when a project's sources exceed the 95 MB upload limit, instead of an opaque failure part-way through.
- Added overrides for air-gapped setups: point the mirror at your own host, or turn it off to use the upstream source.

### Strengthen user management on self-hosted
URL: https://www.tofupilot.com/changelog/strengthen-user-management-on-self-hosted

User seat limits are now enforced server-side on self-hosted instances, so member management respects your license.

- Enforced the user cap on invitations, matching how station limits already work.
- Kept re-invites and role changes unaffected, since they add no seat.

### Improve self-hosted deployment migration
URL: https://www.tofupilot.com/changelog/improve-self-hosted-deployment-migration

Self-hosted instances migrating to the new deployment architecture now get a smoother, more reliable experience, even on restricted networks.

- Improved resilience when optional deployment services are not yet configured, removing recurring errors from instance logs.
- Added a prebuilt build environment so new deployments start faster without downloading dependencies at boot.
- Fixed sign-in for existing members using single sign-on after upgrading from older versions.

### Fix rare sign-in lockout after security upgrade
URL: https://www.tofupilot.com/changelog/fix-rare-sign-in-lockout-after-security-upgrade

A recent security upgrade hardened how sign-ins from identity providers attach to existing accounts, which could lock out a small set of long-standing accounts on upgraded self-hosted instances.

### Rename Teams to Access Groups + free for all
URL: https://www.tofupilot.com/changelog/rename-teams-to-access-groups-free-for-all

Teams are now called Access Groups and available to every organization, no add-on required.

- Renamed Teams to Access Groups across the dashboard, docs, and pricing
- Removed the add-on requirement so any organization can scope station and member access
- Updated documentation with the new Access Groups guide

### Fix automatic database backups before upgrades
URL: https://www.tofupilot.com/changelog/fix-automatic-database-backups-before-upgrades

Self-hosted instances take a database snapshot before applying upgrade migrations, but a version mismatch had silently prevented it from completing, and snapshots were not kept across upgrades.

- Fixed the pre-upgrade database backup so it completes and persists in a dedicated storage volume across upgrades
- Added clear log output when a backup fails, with an optional strict mode (REQUIRE_BACKUP) that halts the upgrade instead of continuing without one
- Limited backups to one per version with automatic cleanup so they never fill the server disk

### Fix invitation acceptance for SSO users
URL: https://www.tofupilot.com/changelog/fix-invitation-acceptance-for-sso-users

New team members signing in through Microsoft SSO could hit an "Invalid invitation" error when opening their invite link.

- Fixed invitation acceptance for users whose identity provider does not report a verified email
- Improved invitation error pages to show the actual reason instead of a generic message

### Configure plugs from your procedure file
URL: https://www.tofupilot.com/changelog/configure-plugs-from-your-procedure-file

Plugs can now receive settings like instrument addresses directly from your procedure file, so you no longer have to hard-code them in Python or duplicate a plug class per instrument.

- Added an optional config mapping on each plug that is passed to the plug class as keyword arguments
- Added support for running several instruments of the same model by giving each plug its own config
- Documented the new config option and the multi-instrument pattern in the plugs guide

### Run and unit metadata added to the framework
URL: https://www.tofupilot.com/changelog/run-and-unit-metadata-added-to-the-framework

High-value units are often tested while undergoing modifications, and standard identifiers like revision number can't capture that state — custom metadata now records it on every run and unit.

- Added `run.metadata` and `unit.metadata` to the Python phase API, validated at assignment with clear errors and included even when a phase fails
- Added operator-prompted unit metadata fields to the procedure file, rendered in the unit identification form next to the serial number
- Added metadata to uploaded reports and Studio local reports so runs and units can be filtered by any key

### Connect Bitbucket Data Center repositories
URL: https://www.tofupilot.com/changelog/connect-bitbucket-data-center-repositories

Teams hosting their code on a self-managed Bitbucket Data Center instance can now link it to TofuPilot, alongside GitHub and GitLab.

- Added Bitbucket Data Center as a git provider, connected with your instance URL and an HTTP access token
- Added automatic repository sync and webhooks so procedures deploy on push, with commit details shown in deploy views
- Added a Bitbucket settings page to manage the connection and replace tokens

### Debug Python phases with a debugger
URL: https://www.tofupilot.com/changelog/debug-python-phases-with-a-debugger

Debugging a failing phase used to mean adding print statements; you can now attach VS Code or any debugpy client and step through your Python phase code with breakpoints.

- Added a debug mode to the CLI (`tofupilot run --debug`) that starts a debugger listener, runs a single worker, and disables phase timeouts so a breakpoint pause isn't cut off
- Added a Debugging guide covering debug mode and manual debugpy attachment, with a ready-to-use VS Code configuration

### Document procedure entry points and run_if
URL: https://www.tofupilot.com/changelog/document-procedure-entry-points-and-run-if

Teams that run several procedures from one codebase no longer have to rename files to main.py or guess which file a station executes.

- Added a full reference for the procedure entry point and root directory settings: accepted values, defaults for each framework, when a change takes effect, and how imports resolve for entry points in subdirectories.
- Added a monorepo guide for deploying several procedures from one repository, including procedure files with custom names.
- Documented how to run a subset of phases with OpenHTF's run_if option, and how skipped phases appear on the dashboard.

### Fix phases timing out at 60s during plug calls
URL: https://www.tofupilot.com/changelog/fix-phases-timing-out-at-60s-during-plug-calls

Long-running hardware operations — thermal soaks, firmware flashes, slow measurements — were killed after exactly one minute, even when the phase allowed more time.

- Fixed plug method calls dying at 60 seconds: they now run for as long as their phase allows, with no time limit when the phase has none
- Improved timeout errors to name the interrupted plug method instead of a bare "timed out"
- Documented the phase timeout field: duration syntax, bounds, and how the deadline covers plug calls and operator prompts

### Fix self-hosted license creation in Orbit
URL: https://www.tofupilot.com/changelog/fix-self-hosted-license-creation-in-orbit

Requesting a self-hosted license in Orbit failed with an unauthorized error for every account, blocking new self-hosting evaluations.

- Fixed license creation by resolving your organization from your membership instead of a session flag that was never set
- Added clear error messages for each failure case: signed out, no organization yet, missing admin role, or multiple organizations
- Fixed stale changelog links from search results to land on the changelog index instead of a 404 page

### Fix DNS failures on linux stations
URL: https://www.tofupilot.com/changelog/fix-dns-failures-on-linux-stations

On some networks, the linux CLI failed to connect with a DNS "Try again" error while other tools on the same machine resolved fine, because the static linux build shipped a strict built-in resolver.

- Fixed the linux build to use a more robust DNS resolver that tolerates unanswered parallel queries and falls back to TCP
- Kept the system resolver on macOS and Windows, which were not affected

### Invite teammates with reminders and resend
URL: https://www.tofupilot.com/changelog/invite-teammates-with-reminders-and-resend

Invitations no longer get lost: invitees land back on the invitation after signing in, and unanswered invites get a nudge.

- Added a reminder to invitees 3 days after the invitation, and a notice to the inviter when an invitation expires.
- Added Resend and expired invitations to Settings › Members, plus suggestions of colleagues on your company domain who already use TofuPilot.
- Improved the invitation email and page: they now name who invited you, your role and the expiry date, and you can decline an invitation.
- Fixed sign-in losing the invitation link, and added a banner when you have a pending invitation to another organization.
- Improved security of the MCP connection and the Linear integration. Existing MCP clients reconnect once.

### Add custom CA support to the Rust SDK
URL: https://www.tofupilot.com/changelog/add-custom-ca-support-to-the-rust-sdk

The Rust SDK can now connect to self-hosted instances that use a private certificate authority.

- Added `add_root_certificate` and `root_certificate_from_pem_file` to the client configuration.
- Switched to the system trust store by default, so system-installed certificates work without code changes.
- Updated the self-hosting guide with the Rust certificate setup.

### Clarify certificate setup for self-hosted
URL: https://www.tofupilot.com/changelog/clarify-certificate-setup-for-self-hosted

Self-hosted instances often run behind a private CA, and the deploy guide only showed one client's certificate setup while still documenting the retired v1 Python client.

- Removed the legacy Python SDK v1 examples from the self-hosting deploy guide, leaving OpenHTF and the current Python SDK.
- Added certificate guidance for every client: the Python SDK accepts a custom certificate directly, while OpenHTF and the C++ and MATLAB SDKs use the standard OpenSSL environment variable.
- Documented that the Rust SDK cannot connect to an instance using a private certificate authority yet.

### Fix missing charts for imported test history
URL: https://www.tofupilot.com/changelog/fix-missing-charts-for-imported-test-history

Runs uploaded with an older start date, such as imported history or a migration from another system, were missing from the Measurement Control charts and filters even though the measurement table below listed them.

- Fixed backdated runs not appearing in measurement analytics: charts, trends and sidebar filters now include them as soon as the data is uploaded
- Improved how historical data is folded into the pre-computed analytics so charts stay fast once the import settles

### Fix cut-off labels on the measurement chart
URL: https://www.tofupilot.com/changelog/fix-cut-off-labels-on-the-measurement-chart

The Y axis on a measurement control chart could cut the leading digits off its own labels, so the scale read as non-linear and the same number appeared twice at two different heights.

- Fixed axis labels being clipped, which hid the most significant digits of every long value and made limit lines look misplaced
- Rounded the axis values so they stay short and readable whatever the measurement scale
- Kept enough decimals on narrow ranges, where every label used to collapse to 0

### Fix stuck station runs with a clear error
URL: https://www.tofupilot.com/changelog/fix-stuck-station-runs-with-a-clear-error

A station run could get stuck on a spinner with no explanation when the test process was blocked before it could start — often by antivirus or endpoint-protection software holding the newly launched process on a locked-down machine.

- Fixed runs hanging indefinitely: the station now surfaces a clear, actionable error instead of an endless spinner when a run cannot start.
- Added a "still starting" hint in the operator UI that points you to the terminal if a run is slow to begin.
- Improved crash messages so a failed run tells you whether it hit a Python error or was stopped from outside.

### Stop framework runs hanging silently at start
URL: https://www.tofupilot.com/changelog/stop-framework-runs-hanging-silently-at-start

A station run of a framework procedure could sit on the spinner forever with no error when the test worker process was blocked from starting — most often by security software on locked-down machines.

- Fixed framework runs hanging indefinitely at startup: the station now stops waiting after a deadline and shows a clear, actionable error naming the likely cause.
- Added the worker's own error output to the failure message, so a crash during startup points straight at the offending line.

### Export every result matching your filters
URL: https://www.tofupilot.com/changelog/export-every-result-matching-your-filters

Exporting from Phase Pareto or Measurement Control only covered the rows already loaded on screen — filter to 295 phase executions, export, get 50.

- Added "Export all matching" to both export menus — the file now contains every row matching your filters, with a clear warning if the export is very large
- Added selection chips to the Measurement Control search bar, so the selected measurement shows up there and can be cleared in one click
- Fixed removing a filter chip doing nothing, and filters occasionally conflicting when changed quickly from different places
- Made Measurement Control load faster on first visit

### Fail fast when security software blocks a run
URL: https://www.tofupilot.com/changelog/fail-fast-when-security-software-blocks-a-run

On locked-down stations, endpoint-protection software could suspend the Python worker or block the venv interpreter, leaving runs hanging forever with no explanation.

- Added startup watchdogs that detect a stalled Python worker (30s to respond, 300s to finish importing) and fail the phase with clear guidance to allowlist the venv in EDR/antivirus software
- Fixed `tofupilot pull` to name AppLocker/WDAC/EDR blocking as the cause instead of a bare "Access is denied" error, with one automatic retry for transient scans
- Added a per-run event log at `~/.tofupilot/logs/` referenced from stall diagnostics for faster troubleshooting with IT

### Create up to 50 organizations per account
URL: https://www.tofupilot.com/changelog/create-up-to-50-organizations-per-account

Teams that split work across many sites, customers or product lines no longer hit the organization cap at 10.

- Increased the limit on new organizations from 10 to 50 on TofuPilot Cloud: you can create one as long as you belong to fewer than 50.
- Kept self-hosted instances at one organization.

### Fix selections reverting after quick clicks
URL: https://www.tofupilot.com/changelog/fix-selections-reverting-after-quick-clicks

Changing a measurement selection quickly — select, deselect, select again — could end up in the wrong state a few seconds later, showing no results until the page was reloaded.

- Fixed rapid filter and selection changes occasionally reverting to an earlier state (a race in the URL state library, fixed upstream and upgraded)
- Fixed measurement names containing a comma breaking multi-selection entirely
- Selecting a measurement now applies in a single step instead of two, removing a flicker

### Keep instruments connected between runs
URL: https://www.tofupilot.com/changelog/keep-instruments-connected-between-runs

Reconnecting to instruments between units costs time on every run — slow VISA sessions, TCP links, or serial handshakes were re-established for each unit tested on a station.

- Added `scope: station` for plugs: the connection opens once, is shared by every run on the station, and closes when the station stops
- Health-checked the held instance before each run, respawning it if the process died, the plug definition changed, or a new deployment was applied
- Renamed plug and phase scopes to `slot` / `execution` / `station`; the legacy `each` / `all` spellings still parse

### Get alerted to quality incidents early
URL: https://www.tofupilot.com/changelog/get-alerted-to-quality-incidents-early

TofuPilot now watches your production test data and tells you when quality slips — automatically, with no dashboards to babysit.

- Added automatic detection for measurement drift, yield drops, and streaks of failed runs — each judged against the procedure's own history, with nothing to configure.
- Added checks for units that get retested too often and for reference (golden) samples that stop behaving — early signs of a drifting fixture or test.
- Added custom alert rules with your own thresholds and scope, plus an in-app inbox and email notifications when something fires.
- Added full Alerts documentation: alert types, automatic detection, rules, and the alert lifecycle.

### Sign in with your Atlassian account
URL: https://www.tofupilot.com/changelog/sign-in-with-your-atlassian-account

Teams already using Jira or Confluence can now access TofuPilot with their existing Atlassian account.

- Added Atlassian sign-in on the login and sign-up pages, and to the connected accounts settings
- Documented the self-hosted setup with the OAuth app configuration and environment variables

Note: requires an Atlassian account with access to a Jira or Confluence site.

### See run outcomes on the measurement chart
URL: https://www.tofupilot.com/changelog/see-run-outcomes-on-the-measurement-chart

The measurement control chart showed each point's own pass/fail, but seeing how runs behaved meant filtering by run outcome and reloading, losing the full distribution.

- Added a colored ring around every control chart point showing its run's outcome (pass, fail, error, timeout, aborted), so value-to-run-outcome correlation is visible in one view
- Hid the ring only where it would blend into the point's own color, and unified point sizes so ringed points don't read as larger values

### Fix test runs freezing when realtime is blocked
URL: https://www.tofupilot.com/changelog/fix-test-runs-freezing-when-realtime-is-blocked

On locked-down factory networks where the realtime streaming server is unreachable (missing DNS record, firewalled WebSockets), test runs and the station service could freeze forever before starting, with no error and no log.

- Fixed runs and the station service to start immediately in every case: the dashboard live view now connects in the background and can never block a run, the operator UI, or result uploads. If it cannot connect within 10 seconds, a clear warning explains what to check and the station keeps working locally
- Fixed stations where a user login coexisted with the station identity: deployment runs now always use the station identity, restoring the dashboard live view that was silently disabled
- Added self-identifying diagnostics: every run prints its CLI version and writes a per-run log from the very start, so support screenshots and log files always show which binary ran and what happened

### Cut drift alert noise to one alert per event
URL: https://www.tofupilot.com/changelog/cut-drift-alert-noise-to-one-alert-per-event

One production incident — a failing fixture, a drifting environment — used to raise a separate drift alert for every affected measurement, and brief wobbles could alert and resolve within minutes.

- Grouped automatic drift alerts into one event per procedure and part: every drifting measurement attaches to the same alert with its own timeline, and same-day recurrences reopen it instead of opening a duplicate. Custom alert rules are unchanged.
- Added persistence gates: a drift now has to hold across consecutive runs before it alerts — and before it auto-resolves — ending one-off wobble alerts and resolve/reopen flapping. Sudden critical shifts still fire immediately.
- Added an instrument-resolution floor: a shift smaller than the measurement device's own step no longer alarms, silencing last-digit noise on quantized sensors.

### Investigate quality incidents with AI
URL: https://www.tofupilot.com/changelog/investigate-quality-incidents-with-ai

Every alert can now launch an AI root-cause investigation on demand — testing hypotheses against production data and reporting what actually changed.

- Added a root-cause investigation to every alert, started with one click: it builds a live hypothesis tree scoped to the incident window, tests each lead against baseline data, and shows the supporting evidence as charts.
- Added a written conclusion that highlights the most likely cause(s), with recommended next steps split into immediate fixes and longer-term improvements grounded in the findings.
- Added drift attribution that pinpoints whether a measurement shifted in level, spread, or ranking, plus incident-vs-baseline checks for outcome shifts and retest churn.
- Added stop and re-run controls for investigations, with results visible to everyone in the organization.

### Post workflow messages to Slack
URL: https://www.tofupilot.com/changelog/post-workflow-messages-to-slack

Workflows can now post to Slack, so a failed run or a new batch reaches your team's channel without anyone watching the dashboard.

- Added a Send Slack Message action, available to every organization: paste a Slack incoming webhook URL, write the message with `{{variable}}` placeholders, and it posts whenever the workflow fires.
- Improved Slack and Discord messages: run data can no longer trigger channel-wide mentions or disguised links, rate limits are retried automatically, and the step error now names the reason Slack or Discord gave.
- Added a username and avatar override to the Discord action, and webhook URLs are now hidden from members with the Viewer role and from execution logs.

### Keep date range when switching pages
URL: https://www.tofupilot.com/changelog/keep-date-range-when-switching-pages

Switching between analytics pages used to reset your selected date range back to the default, forcing you to re-pick it on every page.

- Added carry-over of the selected date range when navigating between Runs, Run Analytics, Phase Pareto, Measurement Control and API Logs
- Kept default behavior unchanged when no range is selected, so shared links stay relative

### Upload and view attachments behind a firewall
URL: https://www.tofupilot.com/changelog/upload-and-view-attachments-behind-a-firewall

Factory networks behind strict proxies had to whitelist third-party storage domains to use run attachments; every attachment URL now stays on TofuPilot domains.

- Added attachments.tofupilot.app: uploads and downloads from SDKs, stations, and the dashboard no longer touch external storage hosts
- Added a 95 MB per-attachment limit with a clear error message, replacing an undocumented unlimited behavior
- Improved attachment serving with video seeking support and stricter content-type protections

### Clarify procedure and station setup tabs
URL: https://www.tofupilot.com/changelog/clarify-procedure-and-station-setup-tabs

The Get Started tabs on procedure and station pages now say what each integration path actually does, so new users can pick between the CLI and the SDKs at a glance.

- Renamed the setup tabs to "TofuPilot runs your tests" and "Your code sends results" on both pages, with icons matching each path
- Improved the station setup descriptions: supported frameworks and SDK languages are now listed inline in a single readable sentence
- Updated the API key empty state to a shorter, clearer call to action

### Fix attachment size showing 0 on the dashboard
URL: https://www.tofupilot.com/changelog/fix-attachment-size-showing-0-on-the-dashboard

Attachments uploaded through the SDKs showed a size of 0 and no file type on the dashboard, even though the file itself was stored and downloadable.

- Fixed the server to record each attachment's size and content type when an SDK links an uploaded file to a run
- Fixed the C#, Rust, C++ and MATLAB upload helpers to finalize uploads, and the Python client to declare file size and type up front
- Improved upload error reporting: a stored file that fails to link to its run is now reported instead of logged as success

### Add private CA support to live stations
URL: https://www.tofupilot.com/changelog/add-private-ca-support-to-live-stations

The `--ca-cert` certificate from CLI 1.2.0 only covered HTTP, so stations behind a corporate CA uploaded runs but showed as offline.

- Added private CA support to the live connection, so one flag covers every connection.
- Added the system certificate store as a trust source, so a CA installed by IT works without the flag.
- Fixed the "check DNS" message hiding the real certificate error.

### Slim OpenHTF uploads, trust private CAs
URL: https://www.tofupilot.com/changelog/slim-openhtf-uploads-trust-private-cas

Stations that only upload no longer install a full test framework, and self-hosted instances behind a corporate certificate authority work end to end.

- Removed the `TofuPilot` context manager and its streaming stack from the Python client — uploads go through the `upload()` output callback, and the CLI provides the live operator UI.
- Made OpenHTF an optional extra: install `tofupilot[openhtf]` for the callback, plain `tofupilot` for API-only stations.
- Added private CA support to the CLI: pass `--ca-cert` at login or set `TOFUPILOT_CA_CERT`, and the saved certificate is reused by unattended stations across logins.
- Neutralized leftover `upload()` callbacks under `tofupilot run`, so migrated procedures work unedited without crashing on a missing API key or creating duplicate runs.
- Fixed custom CA bundles replacing the trust store instead of extending it, which broke attachment uploads to publicly-signed storage hosts.

### Read full measurement names in pareto charts
URL: https://www.tofupilot.com/changelog/read-full-measurement-names-in-pareto-charts

Long measurement and phase names were cut off after 24 characters in Measurement Control and Phase Pareto, so bars that differed only by frequency band or test temperature all looked identical and there was no way to tell which one to click.

- Added a name column that sizes itself to the longest name on screen, with a drag handle to widen or narrow it and a double-click to return to automatic
- Improved shortening to keep both ends of a name when space runs out, so measurements that differ only by their suffix stay distinguishable
- Added the execution count and Cpk value at the end of each bar, and kept the value scale pinned in view while scrolling through long lists

### Fix MCP client connection after authorization
URL: https://www.tofupilot.com/changelog/fix-mcp-client-connection-after-authorization

Connecting an AI client over MCP could stall after clicking Authorize, leaving the client waiting even though access was granted.

- Fixed the authorization hand-off so the client receives its connection code reliably on the first try
- Removed the confusing "finish connecting manually" step from the authorization screen

### Restore the OpenHTF TofuPilot context manager
URL: https://www.tofupilot.com/changelog/restore-the-openhtf-tofupilot-context-manager

Bench scripts using `with TofuPilot(test):` work again without code changes, while a deprecation warning points to the recommended upload callback and CLI operator UI.

- Added the `TofuPilot` context manager back as a thin wrapper around the `upload` output callback, keeping the historical quiet Ctrl-C exit
- Fixed duplicate uploads when reusing one test object across several runs, and restored console quieting between runs
- Fixed attachment uploads to always verify TLS with the client's CA bundle on self-hosted instances with a private CA

### Compare multi-dimensional curves across runs
URL: https://www.tofupilot.com/changelog/compare-multi-dimensional-curves-across-runs

Measurement Control now charts measurements that record a whole curve per run rather than a single value — a temperature ramp, a current draw over time, a sweep across setpoints.

- Added an Individuals view that overlays one curve per run, colouring golden and failing reference units automatically and keeping the rest as a grey population. Selecting a curve selects its row in the table below, and vice versa.
- Added a Band view that summarises every run in the window as a p05–p95 envelope with its median, so a single unit can be read against the whole fleet at production volume rather than against a wall of overlapping curves.
- Added spec limits drawn from the min and max aggregations already attached to a series, and an optional second series on its own right-hand axis.


### Show who can connect or change integrations
URL: https://www.tofupilot.com/changelog/show-who-can-connect-or-change-integrations

Members without the Admin or Owner role could open an integration, fill in a token, and only then be refused with a message that read like a GitLab or GitHub permission problem.

- Improved the GitHub, GitLab, Bitbucket, Odoo, InvenTree and Linear settings: for Developers and Viewers, the connect, change and remove actions are now disabled up front, with the names of the Admins and Owners to ask.
- Fixed the GitHub buttons outside Settings (new procedure, template cloning, a procedure's Repository tab): members who cannot install the GitHub App now go to the GitHub settings page instead of starting an install that was refused at the end.
- Updated the refusal message to name your TofuPilot role, and rewrote the access roles page with one permission table per area.

### Start runs faster on stations
URL: https://www.tofupilot.com/changelog/start-runs-faster-on-stations

Between two runs on a station, the next run only started after its Python workers had spawned and each plug had started in turn, so an operator clicking Run again waited on work that could have overlapped.

- Improved run start: the worker pool now spawns while the unit is identified and the plugs start, and plugs of the same scope start together instead of one after another
- Measured on a demo station: Run again to the first prompt went from about 500 ms to 300-420 ms, and a cold start with three plugs from about 1.1 s to 0.7 s
- Fixed a worker that cannot start being reported as a phase error instead of a run initialization error

### Pick up deployments after a station reconnects
URL: https://www.tofupilot.com/changelog/pick-up-deployments-after-a-station-reconnects

A station that was offline when a deployment was pushed kept running the previous deployment after it came back online: the dashboard's deployment command is sent once, and a station whose link was down never received it. The dashboard showed the new deployment as ready while runs kept using the old one.

- Added a deployment check every time the station's dashboard link comes up, on start and after every reconnect; when nothing changed it is a single request
- Fixed the station not noticing a dropped dashboard link, so "Connection lost" and "Reconnected" now show on a real drop and telemetry pauses while the link is down
- A check that lands during a run is staged and applied between runs, as for a deployment pushed from the dashboard

### Version your procedures from procedure.yaml
URL: https://www.tofupilot.com/changelog/version-your-procedures-from-procedureyaml

The `version` in your `procedure.yaml` is now the version recorded with every run. It used to be ignored — runs read `pyproject.toml` instead, so monorepo procedures recorded no version at all.

- Changed the recorded version to come from `procedure.yaml`. OpenHTF, pytest and Robot Framework procedures keep using `pyproject.toml`.
- Made `version` optional. Procedures without one run normally and are not linked to a version.

### Fix phase outcomes after stops and retries
URL: https://www.tofupilot.com/changelog/fix-phase-outcomes-after-stops-and-retries

On a real bench, a teardown that ran after a failure showed as stopped, exhausted retries kept the run going, and a phase could read another phase's results only when scheduling happened to allow it. Each case now behaves the same way every run.

- Fixed teardown phases to report their own outcome after a stop, so a power-off that crashed shows ERROR with its message instead of a silent STOP
- Fixed `then: retry` so a phase whose retries run out follows `on_first_failure`, and so a setup phase retries before cancelling the run
- Changed `phase.stop()` after a failed measurement to read FAIL, so the defective unit enters the failure statistics
- Changed phase results to be visible only to phases that declare the dependency with `depends_on` or run in a later stage; procedures that read a sibling phase without declaring it must add the entry
- Updated the framework docs: outcome order, when to fail or raise, data sharing between phases, previous attempt on retry, retry exhaustion, phase start order and workers

### Keep a starting run when any tab clicks Run
URL: https://www.tofupilot.com/changelog/keep-a-starting-run-when-any-tab-clicks-run

On a station, a tab still showing another procedure's outcome screen could restart the run that was already starting through its Run again button, and a plug caught during that restart kept spinning on the ended run's screen.

- Fixed the station to keep a run in its start-up window whatever procedure a Run click names; a click with a unit to reuse still restarts as before
- Fixed plugs caught mid-start to read skipped once the run has ended, instead of a permanent initializing badge
- Added `TOFUPILOT_HOME` to move every CLI file (credentials, state database, pinned CA bundles, logs, attachments) to a chosen directory, the same on every OS
- Verified that Ctrl-C and a hard kill of the CLI both close the kiosk window and the Python workers within seconds

### Track each curve axis on its own control chart
URL: https://www.tofupilot.com/changelog/track-each-curve-axis-on-its-own-control-chart

A multi-dimensional measurement records several quantities across one test, and each of them now gets its own control chart in Measurement Control.

- Added one control chart per axis, so an average, minimum or maximum is tracked against the quantity it actually belongs to.
- Added series names to the chart's axis selectors, so a curve is picked by what it measures rather than by its unit alone.
- Extended the second Y axis to the whole population of runs, so a comparison between two quantities is visible before any unit is selected.

### Pass private CA certificates as PEM bytes
URL: https://www.tofupilot.com/changelog/pass-private-ca-certificates-as-pem-bytes

`add_root_certificate` now takes the PEM bytes and returns a `Result`. It accepts a bundle, so a root and its intermediates can be passed together.

- Changed `add_root_certificate` to take PEM bytes; certificates added through it are now applied to the trust store
- Added the bundled certificate authorities alongside the system trust store, so a container without a system certificate bundle keeps connecting
- Added a connect timeout to attachment transfers

### Filter units that have never been tested
URL: https://www.tofupilot.com/changelog/filter-units-that-have-never-been-tested

The units list could not answer "which units were never tested?" — filtering by run count only ever considered units that already had a run, so asking for zero came back empty on every organization.

- Fixed the run count filter to count units with no runs as zero and return them.
- Added the zero bucket to the run count histogram, so the range covers never-tested units.

### Keep each tab on its own organization
URL: https://www.tofupilot.com/changelog/keep-each-tab-on-its-own-organization

With two organizations open in two browser tabs, the tab opened first could read the other organization's data after loading: the Link repository dialog on a procedure then listed no repositories, and the assistant and live updates could follow the other organization.

- Fixed every page under an organization to act on that organization, whatever another tab has opened since, including the Link repository dialog, the assistant chats and live updates.
- Fixed the import page for an existing procedure: it now keeps the procedure's root directory instead of preselecting the first directory found in a monorepo, and the repository root can be picked.
- Improved the Link repository dialog to show a loading state instead of "No repositories are granted" while the list loads.

### Add Bitbucket support to all five SDKs
URL: https://www.tofupilot.com/changelog/add-bitbucket-support-to-all-five-sdks

All five official clients are generated from one V2 specification, so they stay consistent with each other and with the API.

- Added Bitbucket alongside GitHub and GitLab wherever a repository provider is returned, so procedures and deployments connected through Bitbucket now read correctly.
- Improved aggregation values in the C# client to carry any measured type, instead of text only.
- Documented the order phases are returned in: the attempts of a retried phase always arrive in attempt order.
- Removed the conflict error from the station create and update paths, since station names are deliberately not unique and the API never returns it.

### Fix operator emails rejected in Operated By
URL: https://www.tofupilot.com/changelog/fix-operator-emails-rejected-in-operated-by

Declaring `operated_by` at the procedure root prompts the operator on the identify-unit screen, where the field wrongly inherited the serial-number character check and rejected email addresses.

- Fixed the Operated By field rejecting emails and other free-text operator names on the identify-unit screen
- Fixed custom metadata values on the same screen being limited to the serial-number character set
- Added regression tests locking the identify-form validation to unit-identity fields only

### Rework access and alerts documentation
URL: https://www.tofupilot.com/changelog/rework-access-and-alerts-documentation

The access and alerts docs are rewritten in a shorter overview-and-configure format: corrected access-group visibility rules, five alerts pages collapsed into two with all detection thresholds documented, and Root Cause Analysis promoted to the top level of the AI section, with redirects for every removed page.

### Match declared measurements to recorded data
URL: https://www.tofupilot.com/changelog/match-declared-measurements-to-recorded-data

A run could pass even when some declared checks never ran. A measurement, Y axis or aggregation declared in procedure.yaml but never recorded by the phase disappeared from the run, failed with no clear reason, or was stored as 0.

- Fixed measurement outcomes. A multi-dimensional measurement now shows PASS or FAIL from its axis and aggregation limits instead of always UNSET. A measurement with no limits stays UNSET.
- Fixed declared measurements, axes and aggregations that a phase never records. They are now kept in the run, empty and in YAML order, with a clear message. If they have limits, they FAIL and the phase fails.
- Fixed axis descriptions from procedure.yaml, which were not uploaded with the run.
- Fixed series with all values at zero, which were dropped on upload. Multi-dimensional measurements with no data are now stored as empty measurements.
- Fixed limit edits in Studio, which could change booleans and numbers into text.
- Updated the API description of measurement outcomes and released new SDK versions.

### Choose and sort the columns of Runs and Units
URL: https://www.tofupilot.com/changelog/choose-and-sort-the-columns-of-runs-and-units

The Runs and Units lists showed a fixed set of columns, so checking a part number, a unit's secondary serial or its sub-units meant opening each row.

- Added a column picker at the right end of the table header on the Runs and Units lists, grouped like the filters, with every run and unit metadata key available as a column.
- Added sorting on every column that holds a single value, metadata included. Rows without a value always come last.
- Added a Sub-units column and a Contains Sub-Unit filter on Units, and a Unit Metadata filter on Runs.
- Improved long lists: the table header now stays visible while you scroll.
- Fixed the Units search being ignored when sorting by part number, last run or last procedure.

### Fix Robot Framework run outcomes
URL: https://www.tofupilot.com/changelog/fix-robot-framework-run-outcomes

Robot Framework runs now end PASS, FAIL or ERROR by what actually happened, so a failed check is never mistaken for a broken station and the reverse.

- Fixed passing suites without a Suite Setup ending FAIL.
- Fixed failed BuiltIn checks (`Should Be Equal`, `Should Be True`, `Fail`) ending ERROR: they now end FAIL like a failed `Measure` limit, while crashes and setup failures stay ERROR. A failed check also stands when the test teardown fails afterwards, and errors ignored with `Run Keyword And Ignore Error` no longer decide the outcome.
- Fixed empty suites ending FAIL as a crash: they now end ERROR with Robot Framework's own reason.

### Catch broken Python references before the run
URL: https://www.tofupilot.com/changelog/catch-broken-python-references-before-the-run

A test run that cannot succeed now refuses to start and tells you exactly why, and everything your phase code logs is captured where you expect it.

- Added a pre-flight check that names every unresolvable Python reference — phase or plug — before the run starts, replacing mid-run tracebacks and silently missing plug arguments
- Added capture of Python `logging` output from phase code into run logs, with severity and file:line origin; critical messages render as errors
- Fixed `deploy` packing the `.git` file of worktree checkouts into the source archive
- Fixed printed dashboard links mixing identities on machines holding both a user and a station login, and a false "Not logged in" hint when the identity cache was cold

### Import OpenHTF reports with missing sections
URL: https://www.tofupilot.com/changelog/import-openhtf-reports-with-missing-sections

Test reports in OpenHTF format that were generated by your own tooling rather than by OpenHTF itself could be rejected on upload, and the run was lost.

- Fixed the importer crashing on reports with no `log_records` or `code_info`: they now import with no logs and no docstring instead of failing
- Fixed the importer crashing on phases that declare no measurements
- Added a regression test for reports with these sections omitted

### Fix Python procedures crashing on Windows
URL: https://www.tofupilot.com/changelog/fix-python-procedures-crashing-on-windows

Every OpenHTF, pytest and Robot Framework procedure died at startup on a Windows station, before the first phase could run, showing a Python traceback instead of a test result.

- Fixed two calls in the connector the CLI injects that do not exist on Windows, which stopped the interpreter before any phase started.
- Restored station deployment on Windows for all three frameworks — procedures written on the TofuPilot framework were never affected.
- Added Windows to the automated test suite, so a procedure now runs end to end on Windows before every release.

### Fix run duration filtering at any timescale
URL: https://www.tofupilot.com/changelog/fix-run-duration-filtering-at-any-timescale

The duration histogram in the run filters bucketed every run shorter than ten seconds into a single column, so a fast test was drawn a full second away from its real duration and the slider could not select it.

- Fixed the histogram collapsing short runs into one column — it now spans the real range of the runs you are filtering, down to milliseconds
- Fixed the slider handles not lining up with the bars, and typed values like `0.4s` being rounded away to no bound at all
- Fixed the histogram failing to load when the run list was filtered by sample, procedure version or environment

### Automate InvenTree stock from your workflows
URL: https://www.tofupilot.com/changelog/automate-inventree-stock-from-your-workflows

Workflows can write test results and stock movements straight into InvenTree, so your inventory reflects what happened on the line without anyone retyping it.

- Added workflow actions that create stock items, upload test results, update stock status, transfer stock, complete build outputs and print labels on your InvenTree server, with an existing serial reused rather than rejected so a re-run stays idempotent.
- Added serial-number matching, with an optional part IPN when several parts share a serial, so an action targets the item a run actually produced.
- Added pickers for parts, stock locations, build orders, build outputs and label templates, so a resolved option shows its name rather than a raw ID.

### Attribute runs to operators without accounts
URL: https://www.tofupilot.com/changelog/attribute-runs-to-operators-without-accounts

A run's operator no longer has to be a TofuPilot account, so a line running on a station key can attribute its runs to the operators who ran them.

- Added free-text operator names to `operated_by`: an email matching a member of your organization still links the run to that account, any other value is recorded as a declared name.
- Extended it to every surface that produces a run — the V2 API, the `operated_by:` key in `procedure.yaml`, an Operator UI badge scan, Python phases, OpenHTF, pytest, Robot, and CSV, STDF, ATDF, ATML, TestStand, WATS and WSXF imports.
- Added declared names to the runs list, the Operated By filter and the operator analytics, so a declared operator gets its own row instead of falling into Unknown, and alert concentration can name them.
- Added `operated_by_names` to the run, histogram, measurement and phase list endpoints, combined with `operated_by_ids` as OR.
- Changed file imports to take the operator from the file rather than from whoever clicks Import.

### Lock down the station kiosk window
URL: https://www.tofupilot.com/changelog/lock-down-the-station-kiosk-window

On Windows stations the kiosk ran inside Edge, so operators saw translate prompts, Copilot, image search buttons and could open tabs with Ctrl+T.

- Added a dedicated kiosk window on Windows and macOS: full screen, no tabs, no address bar, no browser shortcuts, no translate prompt, no sidebar; navigation is limited to the station itself and the window closes when the station stops. Windows needs the WebView2 runtime, which current Windows 10 and 11 editions include.
- Updated the Linux kiosk to run on Chromium only, with translate, autofill and spellcheck prompts turned off and a per-launch profile that is cleaned up automatically. Firefox, Edge and Brave are no longer launched on Linux.
- Added the TofuPilot icon to the kiosk window on every platform.

### Trust a private CA from the install command
URL: https://www.tofupilot.com/changelog/trust-a-private-ca-from-the-install-command

Self-hosted instances behind a private certificate authority no longer need the CA file copied onto every station: the instance publishes its CA bundle, the dashboard puts the bundle's fingerprint into the install command, and the CLI refuses the bundle unless the fingerprint matches before it trusts anything.

- Added `SSL_CA_CERT_PATH` to the self-hosting deploy script, which validates the bundle, publishes it on the instance and checks the dashboard, storage and realtime hosts against it
- Added `tofupilot login --ca-cert-sha256`, carried by the station install card and the Studio connect screen, with the CA subject and expiry shown next to the command
- Improved CA handling on stations: a rotated CA is adopted automatically over the existing connection, `tofupilot whoami` shows the CA in effect, certificate errors name the fix, and the CLI's own updates never trust a customer CA

### Fix invitations for Microsoft logins
URL: https://www.tofupilot.com/changelog/fix-invitations-for-microsoft-logins

Self-hosted instances using Microsoft Entra login could leave invited users locked out: Microsoft does not report emails as verified by default, so invitees never saw their pending invitation and, without an email server configured, administrators had to copy invitation links by hand.

- Fixed invited users signing in with a company Microsoft account now landing directly on their pending invitation
- Fixed previously blocked accounts recovering automatically on their next sign-in


### Fix invitation re-sends and seat limit gaps
URL: https://www.tofupilot.com/changelog/fix-invitation-re-sends-and-seat-limit-gaps

Re-inviting a pending user, or inviting through the authentication API rather than the dashboard, did not follow the same rules as a regular invitation.

- Fixed re-inviting an already-invited user: the link already shared stays valid and its validity is extended; changing the role or access group now issues a fresh invitation instead
- Fixed plan seat limits and the role ceiling not applying to invitations created through the authentication API
- Unified invitation validity at 30 days across every invite path


### Fix shared plug calls under many parallel slots
URL: https://www.tofupilot.com/changelog/fix-shared-plug-calls-under-many-parallel-slots

Multi-slot stations that share one instrument plug across slots could see phases fail with "Connection reset by peer" when several slots called the plug at the same instant.

- Fixed the plug service dropping simultaneous calls: every call is now accepted immediately and executed one at a time, verified with 80 slots on a single shared power supply
- Improved plug shutdown and health checks so a long-running or abandoned instrument call no longer blocks them
- Added a safeguard that skips a queued call whose phase already timed out, so it never runs late on the instrument

### Stop lost responses duplicating your runs
URL: https://www.tofupilot.com/changelog/stop-lost-responses-duplicating-your-runs

A run upload whose response never made it back to the station was re-sent and stored a second time, so one test appeared twice on the dashboard.

- Added an idempotency reference that the CLI mints and saves before its first upload attempt, so every retry of the same run returns the run already created instead of storing another one.
- Removed a redundant retry loop inside the CLI upload queue, which had turned a single queued attempt into up to four uploads.
- Extended the reference to the Python, C#, Rust, C++ and MATLAB clients, and to stations already in the field, which pick it up on upgrade with no re-login.

### Fix operators tabbing out of the Windows kiosk
URL: https://www.tofupilot.com/changelog/fix-operators-tabbing-out-of-the-windows-kiosk

On Windows the station kiosk launched Microsoft Edge with generic full-screen flags, which hide the toolbars but leave an ordinary browser session behind them, so pressing Ctrl+T opened a tab on the MSN start page.

- Fixed the kiosk launch on Windows to use Edge's own kiosk mode, which serves a single site in a private session and blocks new tabs and new windows.
- Pinned the kiosk idle timeout to zero, so the window can never be closed and reset under a running test.
- Rewrote the message shown when the local UI cannot bind its port, which blamed privileged ports on every platform and sent Windows operators looking for administrator rights instead of the reserved port ranges that were the real cause.

### Connect internal GitLab hosts on self-hosted
URL: https://www.tofupilot.com/changelog/connect-internal-gitlab-hosts-on-self-hosted

Self-hosted dashboards could not connect a GitLab group or Bitbucket project on an internal hostname, because the instance URL had to resolve to a public IP address, a protection that only makes sense for the cloud service.

- Fixed the GitLab and Bitbucket setup on self-hosted installs to accept instance URLs on internal DNS
- Kept HTTPS mandatory for the instance URL, since the access token travels over it

### Filter and group runs by slot in every SDK
URL: https://www.tofupilot.com/changelog/filter-and-group-runs-by-slot-in-every-sdk

Multi-slot stations now upload one run per slot; the SDKs needed the matching fields to read and query them.

- Added: `execution_id`, `slot_key` and `slot_name` on run creation and run details in the Python, C#, C++ and MATLAB clients
- Added: `execution_ids` and `slot_keys` filters on run listing, so one start's runs or one fixture position's history can be pulled in a single call
- Updated: clients regenerated from the current API specification; no breaking changes

### Add metadata to stations, parts and revisions
URL: https://www.tofupilot.com/changelog/add-metadata-to-stations-parts-and-revisions

Stations, parts and revisions now carry custom metadata, the same typed key/value pairs runs and units already had, so a station's location or a part's supplier lives in TofuPilot instead of a spreadsheet next to it.

- Added a metadata editor to the station, part and revision pages: string, number and boolean values, up to 50 keys per entity, read-only for viewers.
- Added `metadata` to the V2 create, get and update endpoints for stations, parts and revisions. Updates are a patch: keys you don't send are kept, a `null` value deletes a key.
- Added a `metadata` filter and `include_metadata` to the station and part list endpoints, with per-key operators (`in`, `contains`, `gte`/`lte`/`gt`/`lt`/`eq`, boolean `eq`), so you can list every station in one building or every part from one supplier in a single call.
- Added every metadata change to the entity's timeline with who made it and the old and new value. The station Activity section now shows without a CLI install, since these edits come from the dashboard or the API.
- Shipped the new fields and list parameters in the Python, C#, Rust, C++ and MATLAB SDKs.

### Report runs that stop at a failure as Failed
URL: https://www.tofupilot.com/changelog/report-runs-that-stop-at-a-failure-as-failed

When a phase failed under the default stop-on-first-failure setting, the remaining phases were skipped and the whole run was reported as aborted, so a normal failing unit landed in the aborted bucket and a filter on Failed never found it.

- Fixed the outcome of a run stopped by a failed phase, which now reports Failed, or Timeout when the phase timed out, instead of Aborted. The skipped phases after the failure are unchanged.
- Kept Aborted for genuine interruptions: an operator stop or kill, and an explicit stop asked for by a phase. An operator kill during a passing phase used to report Error and now reports Aborted.

### Fix GitLab settings page crashing before setup
URL: https://www.tofupilot.com/changelog/fix-gitlab-settings-page-crashing-before-setup

Opening Settings > GitLab in a workspace that had not connected a GitLab group yet showed "Something went wrong" instead of the connect button, so the integration could not be set up from the dashboard.

- Fixed the GitLab and Bitbucket settings pages so they load correctly before an integration is connected
- Added error reporting from the dashboard's error screens, so crashes like this one are detected by the team instead of relying on user reports

### Add execution and slot identity to runs
URL: https://www.tofupilot.com/changelog/add-execution-and-slot-identity-to-runs

Stations that test several units at once can now record which fixture slot produced each run and which execution the runs belong to, so a batch is traceable as a whole and per slot.

- Added execution_id, slot_key and slot_name to the run API, returned on run get, list, header and metadata
- Added execution and slot filters to the run list, the timeline and the runs page, with the fields in CSV exports
- Added slot and execution details to the run overview, with a link to every run of the same execution

### Stop a failing slot without stopping the rest
URL: https://www.tofupilot.com/changelog/stop-a-failing-slot-without-stopping-the-rest

On a multi-slot station, one unit failing used to stop every slot and mark their runs as aborted.

- Fixed: a phase that fails, times out, errors or calls `phase.stop()` now stops its own slot only; that slot's remaining phases are skipped, its slot teardown runs, and the other slots continue to their own outcome
- Fixed: each slot now gets its own outcome, with shared setup and teardown phases counted in, so a failed shared setup reads FAIL on every slot and a slot finished before an operator stop keeps its result
- Fixed: the last main phase can retry again when a teardown stage follows it; the run no longer reads PASS off a phase that was still asking to retry
- Updated: docs on `phase.stop()`, run outcome and a new "Failure in one slot" section

### Follow every slot of a multi-slot run
URL: https://www.tofupilot.com/changelog/follow-every-slot-of-a-multi-slot-run

Stations that test many units at once now get an operator view built for the fixture instead of a single-unit screen.

- Added a fixture grid with one tile per slot showing its name, serial, current phase and status, sized to fill the screen from 2 to 80 slots
- Added slot pages with each slot's own phases, prompts, outcome and run link, reachable from the grid or a direct link
- Improved the progress bar to show one group per slot with its phases, and a violet state wherever a slot waits on the operator
- Updated the CLI to report slot names, units and outcomes so every slot gets its own run in the dashboard

### Upload one run per slot on multi-slot stations
URL: https://www.tofupilot.com/changelog/upload-one-run-per-slot-on-multi-slot-stations

A station testing several units at once used to upload a single run for the whole fixture, so per-unit results and per-slot yield were invisible.

- Added: one run per slot on every start, each with its own unit, outcome, phases, measurements, logs and attachments; the runs of one start share an execution id and carry their slot key and name (needs a dashboard at 2.42.2 or later; single-slot uploads are unchanged)
- Added: `{slot}` and `{slot_name}` placeholders in unit default values, so `auto_identify` derives each slot's serial number and a burn-in rack starts with no operator input
- Added: `execution_id` on `run_started` and per-slot outcomes on `run_finished` in the agent protocol, per-slot phase rows and prompts in the terminal UI
- Updated: Rust client with the run grouping fields and filters; docs on slot placeholders and one run per slot

### Export charts and pick measurements by name
URL: https://www.tofupilot.com/changelog/export-charts-and-pick-measurements-by-name

Getting a Measurement Control curve into a report meant a screenshot, and finding one measurement among hundreds meant scrolling a narrow list, so the analytics pages now do both from where you already work.

- Added Copy chart as image, Export chart to PNG and Export chart to PDF to the right-click menu of every chart on Measurement Control and Phase Pareto, with the KPI strip included in the picture
- Added measurement picking by name in the filter bar: tick several at once, see the typed characters highlighted in each name, and remove any filter with one click
- Added a shortcut from a measurement's page to Measurement Control, with a date window that already contains the run it came from
- Fixed the date range filter so a date can be typed on the keyboard, keeps its day in every timezone and includes the whole end day, in the run filters and on the usage page
- Fixed the sidebar collapse strip covering the filter list's scrollbar, and made slider handles easier to grab

### Patch 2 dependency security advisories
URL: https://www.tofupilot.com/changelog/patch-2-dependency-security-advisories

Preventive security maintenance. We patched the 2 advisories published on 6 October against dependencies TofuPilot ships, both rated high upstream. We found no way to trigger either flaw through TofuPilot.

- CVE-2026-96889, a memory flaw in the SVG decoder bundled with our image processing library, which could lead to remote code execution on Linux. TofuPilot never decodes SVG files from untrusted sources.
- GHSA-6qxp-vccf-f47h, CVSS 7.5, a flaw in the OAuth client of the Model Context Protocol SDK that could send credentials to an authorization server chosen by the remote server. TofuPilot only uses the server side of this SDK.

TofuPilot Cloud requires no action. Self-hosted instances do not need an urgent update either: the fixes come with your next regular image pull. Taking it also stops vulnerability scanners on your side from reporting these versions.

### Fix a slow first Run again after boot
URL: https://www.tofupilot.com/changelog/fix-a-slow-first-run-again-after-boot

The first Run again after a station booted could take three times longer than the following ones, because the run start reopened the station's local state file four times while the previous run's upload was still writing to disk.

- Fixed the run start to open the local state file once, so Run again takes about 220 ms from the click to the first prompt, first cycle included
- Reduced disk writes at every run start, which also spares the SD card on Raspberry Pi stations

### Fix plugs that receive the dashboard client
URL: https://www.tofupilot.com/changelog/fix-plugs-that-receive-the-dashboard-client

A plug whose constructor takes the `dashboard` client was refused before the run started, although the Framework passes that client in on every uploading run.

- Fixed the procedure check that reported `dashboard` as a missing `config:` value and stopped the run
- Updated the error shown when a procedure asks for `dashboard` on a run that does not upload: it now says to run with `--upload` or on a station, instead of pointing to a procedure setting that no longer exists

### See every slot of a fixture from one run
URL: https://www.tofupilot.com/changelog/see-every-slot-of-a-fixture-from-one-run

Runs from a multi-slot execution now show where they ran and what ran beside them.

- Added a Slot section on the run page with the slot key, name and execution, and a hover card listing every sibling run with its serial and outcome
- Added a Slot filter group to the runs list with key, name and execution, with counts, shown only for procedures that use a fixture
- Improved the runs toolbar with a tooltip on Import and consistent pointer cursors

### Reject bad operator input before the test runs
URL: https://www.tofupilot.com/changelog/reject-bad-operator-input-before-the-test-runs

Operator entries used to be checked differently by the kiosk, the web console and the terminal, and a serial the server would refuse was only caught at upload, after the test had already run.

- Added: every submission is now judged by the station itself; a rejected entry keeps the form open with a plain-language message under the field (“Remove the space (character 3) — allowed: uppercase letters, digits, -”, “Must be exactly 4 digits — you typed 2”), and the raw pattern is never shown to the operator
- Added: `prefix` and `suffix` on unit fields and text inputs, displayed locked around the input and composed into the recorded value, so the operator scans only the variable part and `pattern` never has to repeat the fixed one
- Added: `pattern_message` for patterns that cannot explain themselves; patterns now match the whole entry (`SN-[0-9]+` rejects `SCRAP SN-0042 XX`, anchors optional), and serial, part, revision and batch fields default to the server's allowed characters so a bad scan fails at the form, not at upload
- Improved: one shared form for phase prompts and unit identification with keyboard-first flow, Ctrl/⌘+Enter submit and password managers suppressed; the Studio editor refuses a pattern the station cannot compile while you type, nudges you to add a message when one cannot be derived, and screen previews use the exact verdict a run would give
- Fixed: “Run again” no longer crashes when the unit config changed between two runs; the docs for text inputs and unit under test cover the new fields (Studio desktop keeps the previous client-side checks for now)

### Declare 80 slots in one line
URL: https://www.tofupilot.com/changelog/declare-80-slots-in-one-line

Multi-slot procedures no longer need one YAML entry per slot.

- Added an integer shorthand for `execution.slots`: `slots: 80` expands to slots `slot_01` to `slot_80`, zero-padded so they sort in order everywhere
- Kept the list form for custom slot keys and names
- Improved load errors: a wrong count or type says what was expected, and a bad entry in the list keeps its own message
- Updated Studio to open procedures that use the shorthand

### Stop a burn-in safely, even mid-run
URL: https://www.tofupilot.com/changelog/stop-a-burn-in-safely-even-mid-run

Long multi-slot runs on shared instruments now stop the way an operator expects.

- Added a graceful stop on Ctrl-C, Ctrl-Break, SIGTERM and station stop: running phases are interrupted, the execution-scoped teardown (your power-off) runs, and every slot's run is queued as ABORTED with what it measured
- Added a second-signal escalation to force, ignored inside 3 s of the first so a reflex double Ctrl-C cannot cut a power-off in progress
- Fixed operator stops on multi-slot runs, where the shared plugs were released before the teardown could use them
- Fixed shared plug processes outliving the CLI and holding their instrument session after a crash or kill
- Fixed the operator screen showing idle after a reconnect hours into a run
- Changed phase deadlines to a monotonic clock, so a time sync no longer shortens a timeout

Closing the console window on Windows is best-effort inside the 5 s the OS grants; use Ctrl-C to stop a run, and an instrument-side watchdog for fixture safety.

### See how many filter values are hidden
URL: https://www.tofupilot.com/changelog/see-how-many-filter-values-are-hidden

Filter lists on the runs, phases and measurements pages show at most 30 values, and the cut was silent, so a part number or measurement name past the 30th row looked like it did not exist.

- Added a footer under every cut filter list that stays visible while you scroll and points you to the search box, so you know more values exist and how to reach them
- Added the real total where it is known, for example "Showing 30 of 1,800" on the Measurement Control and Phase Pareto filters and on every environment, deployment and version filter
- Improved row counts in filter lists so digits are grouped the same way as in the command bar

### Install the CLI from tofupilot.app
URL: https://www.tofupilot.com/changelog/install-the-cli-from-tofupilotapp

Some corporate networks block the tofupilot.sh domain, so a station could not install or update the CLI without an IT ticket. Everything a station needs now lives under one domain that stations already reach for the API.

- Moved the CLI installer to `curl -fsSL https://tofupilot.app/install | sh` (and `install.ps1` for Windows), with binaries, updates and the Python runtime mirror served from `dl.tofupilot.app` and `get.tofupilot.app`
- Moved the self-hosted deploy script to `curl -fsSL https://www.tofupilot.com/deploy | bash`, the domain the license check already uses
- Updated the station page, Orbit, docs and FAQ to the new commands; the firewall allowlist for a station is now `*.tofupilot.app`

The tofupilot.sh commands keep working, and installed stations update themselves without any change.

### Keep the CLI working when the disk fails
URL: https://www.tofupilot.com/changelog/keep-the-cli-working-when-the-disk-fails

A single unreadable block on a station's disk used to stop every TofuPilot command, including the update that would have fixed it.

- Fixed a damaged local state file stopping the CLI entirely. It is now set aside automatically and rebuilt, and the station stays logged in because credentials are stored separately.
- Added the operating system's own error name and one concrete action to filesystem failures during a pull, so a full disk or a damaged volume names itself instead of looking like a TofuPilot bug.
- Improved the recovery warning, which now points to the copy of the damaged file kept on disk for support.

### Speed up Measurement Control on large datasets
URL: https://www.tofupilot.com/changelog/speed-up-measurement-control-on-large-datasets

Measurement Control could take more than a minute to answer on procedures with very high run volume, and the page sometimes stopped loading altogether before any chart appeared.

- Fixed the page stalling on procedures with hundreds of thousands of measurements. The heaviest query dropped from 106 seconds to 5 seconds, and opening a measurement now answers in well under a second.
- Improved the cold page load so the sidebar filter counts wait for the main chart instead of competing with it, which is what made every query on the page slow down at once.
- Added a clear message asking you to narrow the date range or the filters when a query is too broad, in place of a page that never finished loading.

### Patch 16 dependency security advisories
URL: https://www.tofupilot.com/changelog/patch-16-dependency-security-advisories

Preventive security maintenance. We patched the 16 advisories published on 8 September against dependencies TofuPilot ships, two of them rated critical upstream.

- CVE-2026-75604, CVSS 9.0, unauthenticated remote code execution on Windows filesystems.
- GHSA-2xp9-vwfh-vxw4, CVSS 9.5, unauthenticated remote code execution through the image optimization path. No CVE assigned.
- Fourteen further advisories rated high, covering XML parsing, YAML parsing and image decoding.

TofuPilot Cloud requires no action. Self-hosted instances pick these updates up on the next image pull, and updating is recommended.

### Read full plug results in the Studio debugger
URL: https://www.tofupilot.com/changelog/read-full-plug-results-in-the-studio-debugger

The Studio plug debugger cut every result at one line and failed with a raw traceback on some return values.

- Added an expandable result under each method: a result too long for its cell opens in full, with a copy button, and a failed call opens its whole error the same way.
- Added image previews: a method returning a PNG, JPEG, GIF or BMP image as base64 text now shows the picture, named with its format and size, in the debugger, the call log and the assistant's chat.
- Improved the error for a return value a phase cannot receive (bytes, a datetime, a numpy number): the debugger and the test run now both say the method ran, which type is the problem and how to convert it, for example `x.item()` for a numpy number or `base64.b64encode(data).decode()` for bytes.

### Add typed timeout errors to the API and SDKs
URL: https://www.tofupilot.com/changelog/add-typed-timeout-errors-to-the-api-and-sdks

A read query that runs past the 20 second limit is stopped and answered with a 408 timeout. Until now that status was missing from the API reference, so the SDKs reported it as an unknown error with nothing to catch.

- Added the 408 timeout to every read endpoint of the V2 API reference, with the message that tells you to narrow the date range or the filters and try again
- Added a typed timeout error to the Python and C# SDKs (ErrorTIMEOUT and RequestTimeoutException) so you can handle it without parsing the error text
- Documented the timeout in the REST API error table and in the SDK READMEs

### Keep branch pushes off your stations
URL: https://www.tofupilot.com/changelog/keep-branch-pushes-off-your-stations

A Git push to a branch other than your production branch used to roll out to every station linked to the procedure, so work in progress could replace the code running on your line.

- Fixed branch pushes so they still build a preview deployment but no longer reach any station. Only pushes to the production branch roll out automatically.
- Kept the manual path: open a preview deployment in the dashboard and push it to the station you choose when you want to test a branch on real hardware.
- Fixed a deployment that could stay pending forever when its build could not be queued. It now shows as failed, and the other procedures fed by the same repository still get their build.
- Updated the Deployments and Environments docs to describe when a push is built, when it is skipped, and how previews reach a station.

### Show when your GitHub App is suspended or gone
URL: https://www.tofupilot.com/changelog/show-when-your-github-app-is-suspended-or-gone

Uninstalling the GitHub App from GitHub left the dashboard showing the account as connected, with a Manage button pointing at a page that no longer existed and no way to install the App again.

- Fixed the GitHub settings page so an App uninstalled from GitHub reads as not installed and the Install button comes back. A suspended App stays visible with a Suspended badge and a pointer to the GitHub App settings, and Manage now opens the right page for organization installs.
- Added a warning on the Repository section of a procedure when the provider of the linked repository is suspended, has lost its token or was disconnected. The repository and its Unlink button stay visible so you can always move on.
- Improved how a missed GitHub notification is caught up: the dashboard re-checks the installation with GitHub when you open the GitHub settings page, and records a suspended or removed App the moment a repository listing fails because of it.

### Run the OpenHTF teardown when a run is stopped
URL: https://www.tofupilot.com/changelog/run-the-openhtf-teardown-when-a-run-is-stopped

Stopping an OpenHTF run from the operator UI, the terminal or a signal used to kill the test outright: no plug tearDown, no PhaseGroup teardown phase, and nothing uploaded.

- Added a clean stop for OpenHTF runs, the way OpenHTF stops on its own Ctrl-C: the running phase is interrupted, the PhaseGroup teardown phases and the plugs' tearDown run, and the run is uploaded as ABORTED with the phases that completed
- Added a Kill that matches OpenHTF's second Ctrl-C: the teardown phase under way is abandoned, the plugs are still released and the run is still uploaded, with a forced kill only when the process has not exited seven seconds later
- Fixed stopped runs being reported as FAIL on both frameworks: they now report ABORTED and exit with code 130, and a run the procedure stops itself exits with code 3
- Added the ui_cancelled event for agents when an open prompt is withdrawn with its interrupted phase

### Fill operator UI options at run time
URL: https://www.tofupilot.com/changelog/fill-operator-ui-options-at-run-time

A phase can now change what an operator-screen component offers, not only its value, so choices and input rules that only exist at run time no longer need one procedure per case.

- Added `ui.set(key, ...)` to change a component's options, label, placeholder, affixes, numeric bounds, text length and pattern while its screen is open. The operator's entry is checked against what the phase sent.
- Added a "Waiting for options" state: a choice declared without options now loads, fills when the phase sends the list, and never pre-selects an entry.
- Improved the agent protocol with a `ui_update` event and the same entry validation as the kiosk, including pre-baked values.

### Pass measurements recorded without limits
URL: https://www.tofupilot.com/changelog/pass-measurements-recorded-without-limits

A measurement that records a value but has no validators now passes, as in OpenHTF and TestStand, instead of showing Unset.

- Fixed the outcome of a measurement with a value and no validators: it is PASS again in the Framework, the station kiosk and the Robot Framework connector. A measurement with no value and no validators stays Unset.
- Fixed TestStand and WATS imports: a step reported as Done now imports as PASS, as ATML imports already did.
- Updated the API reference and the Python, C#, Rust and C++ clients to describe the new rule.

### Run the Windows CLI without the VC++ runtime
URL: https://www.tofupilot.com/changelog/run-the-windows-cli-without-the-vcpp-runtime

On a locked-down Windows station without the Microsoft VC++ redistributable, the CLI failed to start and the installer wrongly pointed at antivirus; the binary is now self-contained and the installer can pin a version or install a preview build.

- Fixed the Windows CLI to start on any machine: it no longer depends on vcruntime140.dll or other runtime DLLs, and every release is checked for stray runtime imports before it ships
- Improved the Windows installer: it now reports the real exit code on first run and explains a missing-DLL failure instead of the antivirus checklist
- Added `--version <x.y.z>` to both installers to pin or roll back one station to a released build, and `--preview <slot>` plus `tofupilot update --preview <slot>` to install a preview build shared by support before it is released

### Update dependencies after security advisories
URL: https://www.tofupilot.com/changelog/update-dependencies-after-security-advisories

The CLI's dependencies were updated following recent security advisories, with a few related fixes to how procedure files are read.

- Fixed four security advisories in the TLS library used for uploads, the dashboard link and the station connection.
- Added a warning, in the console, in Studio and at the top of the run logs, for `.inf` and `.nan` values and for two layouts a future version will refuse (a quoted text continued at its key's column, a comment touching its value). Procedures using them keep running as before.
- Fixed procedure files starting with a byte order mark, as some Windows editors save them, failing to load.
- Fixed `in` and `not in` validators on a list mixing numbers and text, which never matched a measured number.
- Updated plug `config` to refuse a key written twice, at its line, like the rest of the procedure file.

### Speed up Measurement Control on busy procedures
URL: https://www.tofupilot.com/changelog/speed-up-measurement-control-on-busy-procedures

Measurement Control could time out on procedures with very high run volume.

- Improved the main chart, the failure bars and the Exclude retries option. They now load in a few seconds where they used to time out.
- Improved the measurement table for a selected measurement, from about 12 seconds to under one second.
- Updated GET /v2/measurements: started_after and started_before now always filter on when the measurement was taken, not on when its run started.

### Fix kiosk window relaunch and macOS copy-paste
URL: https://www.tofupilot.com/changelog/fix-kiosk-window-relaunch-and-macos-copy-paste

The station kiosk window now comes back after a crash or Dock Quit, and the macOS window accepts the standard keyboard shortcuts.

- Fixed the kiosk watcher on macOS and Linux: a closed or crashed window was never detected, so the station kept running with no window. It now reopens one a few seconds later.
- Added the Edit menu to the macOS kiosk window, so Cmd+C, Cmd+V, Cmd+X, Cmd+A and Cmd+Z work in operator forms. Cmd+Q stays disabled.
- Fixed the Linux kiosk profile sweep to remove the profile left behind by a crashed window.

### Pick organizations faster when connecting AI
URL: https://www.tofupilot.com/changelog/pick-organizations-faster-when-connecting-ai

Connecting an AI assistant to TofuPilot is quicker when you belong to many organizations.

- Added a select-all checkbox and a single switch to allow write access in every selected organization
- Improved the authorization screen with a shorter layout that stays compact however many organizations you have
- Updated the screen to show clearly where the assistant returns after you authorize it

### Call the TofuPilot API from a procedure
URL: https://www.tofupilot.com/changelog/call-the-tofupilot-api-from-a-procedure

A test that needs workspace state as an input, such as reserving a serial number by creating a unit or checking that a unit's previous run passed, no longer needs its own API key stored on the station.

- Added a `dashboard` parameter for phases and plugs: a run that uploads hands the procedure a ready-made TofuPilot API client acting as the identity the run uploads with, the station on a bench, your own login on a development machine, with nothing to declare in procedure.yaml
- Added `GET /v2/whoami`, which names the station or the user behind any API key and its organization, and `GET /v2/users/current`, the mirror of `GET /v2/stations/current`, so a procedure that must act as a station checks for it instead of failing mid-run
- Removed the `current=true` option of `GET /v2/users`, replaced by the two routes above
- Fixed procedures inheriting an API key set on the station: a run that does not upload receives no credentials at all

### Restore self-hosted dashboard updates
URL: https://www.tofupilot.com/changelog/restore-self-hosted-dashboard-updates

Self-hosted instances could not update past dashboard 2.44.5, because the images of the releases that followed failed to build.

- Fixed the self-hosted dashboard image, so version 2.44.12 and later are published again.
- Brought the dashboard changes announced since 2.44.5 to self-hosted instances, including teammate invitation reminders, standard MCP sign-in for AI assistants and waiting for the operator in the middle of a phase.

### Retire Studio desktop, stations move to the CLI
URL: https://www.tofupilot.com/changelog/retire-studio-desktop-stations-move-to-the-cli

Studio desktop gets its final update: it is now an offline procedure editor, and stations run on the TofuPilot CLI.

- Removed Station mode from the desktop app: it now shows how to install the TofuPilot CLI on each test station
- Removed every connection from Studio desktop to TofuPilot and Git: it opens, edits and runs procedures locally only
- Added a migration guide in the app: deploy from Git, install the CLI on stations, and write procedures with Studio in the browser

### Update stations only through Deployments
URL: https://www.tofupilot.com/changelog/update-stations-only-through-deployments

Every station update now goes through the Deployments page, and the API reference describes station access accurately.

- Removed the old Commits page deploy flow: stations now show the version you pushed from Deployments
- Fixed the API reference saying stations cannot use endpoints they can use for their linked procedures
- Updated the station docs to explain how a Git push reaches your stations, with or without auto-push

### See fast-failing runs live on the dashboard
URL: https://www.tofupilot.com/changelog/see-fast-failing-runs-live-on-the-dashboard

When a run failed right at start, the dashboard showed its phases arriving one by one for several seconds and a run time far longer than the station's, while the kiosk showed the error at once.

- Improved live delivery: a burst of phase results now reaches the dashboard in one update instead of one at a time, cutting a ~5 s catch-up to a single round trip.
- Fixed the run and phase durations on the dashboard to use the station's own clock, so they match what the kiosk shows.

### Catch upload rejections before the test runs
URL: https://www.tofupilot.com/changelog/catch-upload-rejections-before-the-test-runs

A run the station had already executed could be refused at upload, stay in the upload queue, and say only "input validation failed". The station now refuses up front what the dashboard would refuse, the dashboard accepts the limits a test can legitimately compare against, and a rejection names the field.

- Added support for object and mixed-list limits on validators (`== {}` for "no deviation from the reference configuration"), so these runs upload instead of being refused
- Changed the identify form and Python assignments to refuse serial, part, revision, batch and sub-unit values longer than 60 characters or outside letters, digits and `_ . : + -` before the test runs, and to refuse measurement names over 100 characters
- Fixed `== [3]` failing against a recorded `[3]`: numbers now compare by value in `==`, `!=` and `in`
- Improved upload errors to name the rejected field, for example `phases[2].measurements[4].validators[0].expected_value`
- Fixed `depends_on`: a dependency on a phase further down the file is honoured, unknown keys and cycles are refused in every stage, and a teardown cut by the shutdown limit is recorded as an error instead of disappearing

### Fix teardown, stops and limits in the Framework
URL: https://www.tofupilot.com/changelog/fix-teardown-stops-and-limits-in-the-framework

A full review of the Framework found teardowns that lost their instruments, runs that could hang, and limits that passed without being checked; this release fixes them and hardens the station.

- Fixed teardown phases losing their instruments after the first one, runs hanging when a phase timed out during a stop, teardown retries dropped under a stop, and teardown prompts closed by the stop that triggered them; a failed unit with an interactive teardown now reads FAIL, not ERROR.
- Fixed limits that could never be checked passing silently: a numeric limit written as text, `in` without a list, an invalid regex or a missing expected value is now refused when the procedure loads, with its line and column, and duplicate slot keys are refused too.
- Fixed shell phases not being stopped on timeout or stop, NaN and numpy measurement values losing the whole phase, and plug and worker errors going missing from the logs.
- Fixed runs with several attachments getting stuck in the upload queue after one network error, and OpenHTF, pytest and Robot runs without `--upload` being uploaded later; a stopped pytest or Robot run now uploads its finished tests as ABORTED.
- Improved station safety and the operator UI: only the operator UI's own commands are accepted locally, a reverted deploy is no longer applied at the end of a run, and a kiosk reload no longer brings back an answered prompt or loses one still open.

### Fix kiosk prompts, reloads and preset UI values
URL: https://www.tofupilot.com/changelog/fix-kiosk-prompts-reloads-and-preset-ui-values

Operators on the station kiosk and the dashboard operator page lost prompts or saw stale screens after a reconnect, and preset UI values only worked in agent mode.

- Fixed the operator screen after a reload or a dropped connection: finished phases stay finished, retries keep their own outcome and prompt, answered prompts no longer come back, and the submit button no longer stays locked.
- Fixed `--ui-values` to answer prompts in every mode (kiosk, terminal UI, plain and agent) and every framework, including unit identification; a run with no way to answer a required prompt now fails at once instead of hanging, and locked unit fields are no longer requested.
- Added a Deployment section to the kiosk's details panel with the procedure's commit, message, pull time and folder.
- Improved the operator screen with tooltips that also work on disabled buttons, coloured plug logs, and a Kill button that a stray Enter key cannot trigger.
- Fixed a run whose kiosk port is already taken: it now explains which process holds the port and stops instead of waiting with no screen.

### Remove and delete deployments you no longer use
URL: https://www.tofupilot.com/changelog/remove-and-delete-deployments-you-no-longer-use

Old deployments stayed in the Deployments list and on stations, and the only way to take one off a station was to unlink the procedure, which also stopped the station from uploading runs.

- Added Remove deployment, which takes a deployment off a station while the procedure stays linked and the station keeps uploading runs. It is on the station's procedure cards and next to each station on a deployment's page, and the confirmation names the build that goes.
- Added Delete on the deployment page and in the Deployments list. A deployment that is still on a station, still building, used by runs or targeted by an instant rollback cannot be deleted, and the dialog says why and what to do.
- Improved the station views: the station page and the procedure's Stations section now show the build each station actually runs, and the action that unlinks a procedure is now called Unlink procedure.
- Fixed runs from a station that still had the code of a deleted deployment: they now upload without the link instead of failing.

### Choose how tofupilot run shows the operator UI
URL: https://www.tofupilot.com/changelog/choose-how-tofupilot-run-shows-the-operator-ui

Running a procedure on a laptop no longer opens the full-screen kiosk window, and the way the operator UI is shown is one flag instead of a flag and an environment variable.

- Added `--ui kiosk|browser|headless` to `tofupilot run`: full screen for a station, a tab in your browser for developing a procedure, or just the printed URL for agents, CI and SSH port-forwards. `--no-ui` serves no UI.
- Removed `--kiosk`, `--no-kiosk` and `TOFUPILOT_NO_BROWSER`; scripts using them must switch to `--ui` and `--no-ui`.
- Changed local-path runs to follow their flags only: they no longer inherit the station's kiosk and terminal UI settings on a machine that is also a station. Deployment runs and the station service still follow the dashboard settings.
- Fixed a station running as a root service opening its own operator UI server for every run.

### Wait for the operator in the middle of a phase
URL: https://www.tofupilot.com/changelog/wait-for-the-operator-in-the-middle-of-a-phase

A phase can now ask the operator for input while it runs and react to the answer, so a calibration loop no longer needs one phase per try.

- Added `ui.wait()` to the TofuPilot Framework: the phase blocks until the operator clicks Continue, then gets the answer with numbers, switches and multiple choices already typed. Each call shows the phase's screen again with the last answer and the values the phase set since.
- Changed Esc in the terminal UI to cancel a phase prompt on every screen: the phase ends in error instead of continuing on an empty answer, and a phase blocked in `ui.wait()` gets `ui.WaitCancelled`.
- Changed `--ui-values` to answer the first prompt of a phase only, so a phase that loops on `ui.wait()` asks the following ones.
- Updated Studio and the dashboard operator view to replace a waiting phase's previous prompt with its new one.

### Apply station updates without a manual restart
URL: https://www.tofupilot.com/changelog/apply-station-updates-without-a-manual-restart

A station that started before its network was up could download a CLI update yet keep running the old version until someone restarted it by hand.

- Fixed station self-updates so the station restarts into the new version on its own, and checks again every few minutes after starting offline instead of waiting 4 hours.
- Fixed Windows stations closing their window, or exiting with "already running", after an automatic update.
- Improved deployment updates: instruments are released before a new bundle is swapped in, a failed swap keeps the previous deployment, and a deployment pushed during a test is applied as soon as the test ends.

### Show which publisher to allow on Windows
URL: https://www.tofupilot.com/changelog/show-which-publisher-to-allow-on-windows

When Windows blocked the CLI, the installer still said it was not code-signed, which sent IT looking for a fix that already exists.

- Updated the Windows installer's error messages to show the file's signature and give IT one line to act on: allow the publisher TofuPilot SA.
- Added guidance for unsigned preview builds, altered downloads, and files a security tool keeps locked.

### Make waiting for the operator mid-phase safer
URL: https://www.tofupilot.com/changelog/make-waiting-for-the-operator-mid-phase-safer

A phase that waits for its operator now behaves predictably when the operator answers late, when two threads wait at once, or when an answer is incomplete.

- Fixed an answer submitted after a phase timed out being recorded as a measurement of that phase
- Fixed a phase hanging when two of its threads waited for the operator at the same time; the second wait now fails with a clear error
- Improved error messages: a wait without a screen or a duplicate wait reads as a procedure error, and an incomplete answer no longer says the operator cancelled
- Fixed the terminal UI moving a refreshed prompt to the back of the queue

### Fix failed first runs after a CLI update
URL: https://www.tofupilot.com/changelog/fix-failed-first-runs-after-a-cli-update

The first test run on a new machine, or right after a CLI update, could fail before any phase ran, and only a second run worked.

- Fixed the first run after installing or updating the CLI skipping all of its phases and ending FAIL with "Worker failed to start".
- Added checks that a local run without `--upload` is never uploaded, for YAML, pytest, OpenHTF and Robot Framework procedures.

### Connect AI assistants with standard MCP sign-in
URL: https://www.tofupilot.com/changelog/connect-ai-assistants-with-standard-mcp-sign-in

TofuPilot's MCP server now follows the standard MCP authorization flow, so AI assistants and coding tools connect to your test data with the sign-in they already support, without custom setup.

- Added support for desktop AI apps that register as native applications, so tools like Cursor connect on the first try.
- Improved connection security: when an app supports it, its token is bound to that app, so a copied token can't be used on its own.
- Updated the MCP documentation with connecting several organizations, switching between them, and checking which account a connection uses.

### Verify who connects GitHub, GitLab or Bitbucket
URL: https://www.tofupilot.com/changelog/verify-who-connects-github-gitlab-or-bitbucket

Connecting a Git provider now confirms that the person connecting it really controls that account, so an organization can only link repositories its own admins own.

- Improved GitHub connections: TofuPilot checks on GitHub that you own the installation, either as the account itself or as an owner of the GitHub organization, and that the connection was started from your own browser.
- Improved GitLab and Bitbucket connections: each group or project is claimed per server, so organizations on different self-managed servers no longer block each other, and GitLab now requires a Maintainer token as documented.
- Improved how account emails are sent, so only TofuPilot itself can trigger them.
- Updated the self-hosted GitHub setup: add the App's client ID and client secret and enable user authorization during installation before connecting GitHub; GitLab tokens below Maintainer need replacing when they next expire.

### Accept any character in self-hosted .env values
URL: https://www.tofupilot.com/changelog/accept-any-character-in-self-hosted-env-values

Self-hosted operators can now put any password or secret in their configuration file, including the characters that used to abort an update.

- Fixed the deploy script failing on a `.env` value containing a backtick, a dollar sign, a space or a quote
- Kept the quotes an operator adds around a value when the script rewrites `.env` on each deployment
- Added a clear error with the line number when a value is badly quoted, instead of a silently shortened value
- Documented how to quote special characters on the self-hosting configuration page

### See the operator UI URL on every local run
URL: https://www.tofupilot.com/changelog/see-the-operator-ui-url-on-every-local-run

A local `tofupilot run` now gives you a browser operator UI by default instead of taking over the terminal, so the run log stays readable and prompts are answered where they render best.

- Updated local runs in a terminal to serve the operator UI and print its URL, with the plain run log in the terminal; pass `--tui` for the terminal UI.
- Added a reminder with the URL each time a prompt or unit identification waits for input.
- Improved runs next to a station or another run: the default UI picks a free port instead of failing, and CI or `--json` runs still fail fast on unanswered prompts.

### Deploy to stations only from Deployments
URL: https://www.tofupilot.com/changelog/deploy-to-stations-only-from-deployments

The old Commits page deployed versions that current stations no longer install, so every station update now goes through Deployments.

- Removed manual deploys from the Commits page; old links now open Deployments
- Updated the Deployments banner to explain where station updates now happen
- Kept stations deployed from the Commits page running their current version

### See which stations run each deployment
URL: https://www.tofupilot.com/changelog/see-which-stations-run-each-deployment

The dashboard used to show a deployment as pushed even when a station never installed it, and a missed update could leave a station on old code without anyone noticing.

- Added a per-station status on deployments (Running, Installing, Install failed, Waiting for station, Wrong platform) and a "3/5 running" count in the Deployments list, updated live.
- Improved stations so they check for deployments every 15 minutes, retry network failures within minutes, wait longer before retrying a deployment that fails to install, and reconnect on their own after losing the dashboard link.
- Fixed production deployments so stations move to a build only once it is ready, never onto a failed build, and never onto a build made for another platform.
- Fixed a second station process on the same machine fighting the first one over installs.

### Fix the OpenHTF docs to match the CLI
URL: https://www.tofupilot.com/changelog/fix-the-openhtf-docs-to-match-the-cli

The OpenHTF integration page listed metadata keys and prompt behavior that did not match what the CLI does.

- Fixed the metadata table: the CLI reads `serial_number` and `revision_number`, not `procedure_id` or `revision`.
- Documented which serial number a run gets under `tofupilot run`, and that a timed-out prompt returns an empty string.
- Corrected the minimum Python version in the SDK overview and renamed the "Streaming" links to "Remote UI".

### Compare every curve in Measurement Control
URL: https://www.tofupilot.com/changelog/compare-every-curve-in-measurement-control

Curves such as frequency sweeps or efficiency curves can be compared across every run, revision and batch in Measurement Control, and the docs now explain how.

- Added a guide to the Waveform, Scatter and Band views for multi-dimensional measurements, with the 100-curve limit and the band computed over every run.
- Documented how limits, reference units and the second axis appear on these charts.

### See live what each station runs
URL: https://www.tofupilot.com/changelog/see-live-what-each-station-runs

The station page now shows what each station actually runs and updates on its own as the station reports, so you no longer reload to see whether a push landed.

- Added the install state to each procedure on the station page: running, installing, install failed, or waiting, with the commit it still runs until the new one is installed.
- Improved live updates on the station page: the activity timeline and procedure cards refresh as soon as the station reports, catch up after a dropped connection, and stay connected through short network or server interruptions.
- Fixed repeated "Deployment updated" notifications in the operator view and long commit names hiding the pushed commit on small screens.

### Remove the legacy operator UI streaming page
URL: https://www.tofupilot.com/changelog/remove-the-legacy-operator-ui-streaming-page

The operator UI that streamed OpenHTF tests from the Python client to a dashboard page has been replaced by the CLI, which runs the same unmodified scripts with a live operator UI on any browser.

- Removed the legacy streaming page and the server endpoint older Python clients called to reach it; those clients now stop retrying at once instead of logging a warning every second.
- Removed the "streaming was removed" warning the Python client logged when a script passed `stream=True`; the argument is still accepted and ignored.
- Planned: the `TofuPilot` OpenHTF context manager, deprecated with a warning since August 11, 2026, will be removed in a future release. Run your tests with `tofupilot run` instead, or register the `upload()` callback if you manage your own pipeline.

### Polish the operator UI cards and pickers
URL: https://www.tofupilot.com/changelog/polish-the-operator-ui-cards-and-pickers

A few minor improvements to the operator UI after its redesign, so operators see the state of the station and its procedure at a glance.

- Added the next run pick to the procedure card, with the deployment of each procedure (ID, commit message, when it was pulled).
- Improved the station card: mode, name, organization and how the station starts, the CLI version and status, and the dashboard connection next to the upload queue.
- Updated the procedure list and form controls to match the operator form, and turned sound off by default.

### Align the operator UI side panels
URL: https://www.tofupilot.com/changelog/align-the-operator-ui-side-panels

The Run and Unit panels of the operator UI now read like the top bar menus, so field names no longer wrap and values line up.

- Updated panel rows to show the field name and its value on one line, with the value aligned right.
- Fixed long field names wrapping onto two lines, such as Serial Number.

### Read clearer CLI errors with the next step
URL: https://www.tofupilot.com/changelog/read-clearer-cli-errors-with-the-next-step

Every message the CLI prints now says what failed, names the file, ID or URL involved, and gives the next step on a `hint:` line, so a failed command tells you how to fix it.

- Improved errors across every command: server errors show the server's message instead of raw responses, `procedure.yaml` errors point to the line and column with "did you mean" suggestions, and the installers name the URL that failed.
- Added `-v`, `-q` and `--color` flags, with `NO_COLOR` support, and moved diagnostics behind `-v` for quieter output.
- Updated output for scripts and agents: `--json` goes after the subcommand on the commands that return data, every error is one JSON object, and exit codes now mean something (2 usage error, 4 not found, 6 not logged in). Scripts that relied on the old exit code 1 or on message text need updating.

### Run tests on a redesigned operator UI
URL: https://www.tofupilot.com/changelog/run-tests-on-a-redesigned-operator-ui

The operator UI is rebuilt so operators move faster at the bench and engineers see everything about a run without leaving the station.

- Added three details levels from the bottom bar: off, a side column of compact panels, or one full tab for the run, plugs, phases, logs or attachments, with search, filters and keyboard shortcuts.
- Improved forms for scanners and keyboards: field-sized controls, short errors, stray scans never answer a question, and every choice has a key.
- Added station and procedure cards in the top bar, a status chip and phase cards in the bottom bar, and recorded sound cues on verdicts and prompts.
- Added the procedure version and description, run metadata and the operator to the run details, and file and line to plug log lines.
- Improved consistency with the dashboard: phase, measurement and log tables now render the same way in both.

### Fill operator forms together, live
URL: https://www.tofupilot.com/changelog/fill-operator-forms-together-live

The station and the dashboard now share one live view of every run, so a supervisor can follow, help and answer from anywhere while the operator works at the bench.

- Added shared operator forms: a value typed at the station, in the terminal or on the dashboard shows up live on every other screen, and nothing is submitted until someone presses Submit
- Added presence on operator screens: who is here (avatar, name and role), their cursor, the field they are on and who is typing, with viewers watching but never editing
- Improved the live view: a dashboard opened halfway through a long run shows it from the start, and a dropped connection catches up on its own without missing a step
- Improved station commands: every Run, Stop, Pull and answer from the dashboard is confirmed by the station, records who sent it, and never lands on a later run by mistake
- Updated security: only the station itself can publish its run, and live access ends as soon as someone's role or team changes

Stations need CLI 2.1.0 for the live view; older stations keep running and uploading until they update.

### Upgrade GitHub connection security
URL: https://www.tofupilot.com/changelog/upgrade-github-connection-security

Connecting a GitHub organization to a workspace now confirms that the person connecting it owns that organization.

- Added a GitHub authorization step when installing the TofuPilot app, so the installation is verified against your GitHub account before it is linked.
- Added a read-only organization membership permission to the TofuPilot GitHub app. It covers members and their roles, with no access to code.
- Updated existing installations to keep working as they are. Organization owners are asked to accept the new permission before their next reconnect.

### Get precise messages on validation errors
URL: https://www.tofupilot.com/changelog/get-precise-messages-on-validation-errors

When a value is rejected, the API and the dashboard now say exactly what is wrong with it instead of a generic message.

- Improved API error responses, which name the problem, for example "Invalid UUID" or "Too small: expected string to have >=1 characters".
- Improved dashboard form errors the same way, such as the API key name field.

### Keep the macOS kiosk on its own desktop
URL: https://www.tofupilot.com/changelog/keep-the-macos-kiosk-on-its-own-desktop

On macOS the kiosk followed you to every desktop, so it flashed on top of other apps in full screen whenever you switched to them.

- Fixed the kiosk flashing over other full-screen apps when switching desktops on macOS
- Updated the kiosk to stay on its own desktop like any other app: swipe away to use another app, then swipe back or click its Dock icon to return
- Kept the kiosk full screen, with the Dock and menu bar hidden while it is in front

### Retry imports without creating duplicate runs
URL: https://www.tofupilot.com/changelog/retry-imports-without-creating-duplicate-runs

Scripts that push test reports on a schedule can now retry safely: a file you already imported is recognized instead of being imported twice.

- Added duplicate detection to structured imports (OpenHTF, WATS, ATML, NI TestStand, STDF, ATDF), through the API and the dashboard. Importing the same file again returns its existing runs and creates nothing.
- Added an `allow_duplicate` option to import a file again on purpose, as new runs next to the existing ones. The dashboard asks first when it recognizes a file.
- Fixed OpenHTF records without a procedure in their metadata, which now import from the dashboard, and WATS WSJF files with unnamed measurements. The file picker now also offers XML, STDF and ATDF files.

### Start the kiosk faster, with no white screen
URL: https://www.tofupilot.com/changelog/start-the-kiosk-faster-with-no-white-screen

Operator stations showed a long white screen while the kiosk started, and an update that could not be installed left the station without its window until a restart.

- Improved kiosk startup: the window now appears only once the operator UI is drawn, in your system's light or dark colour, and the UI is ready up to 7x sooner on macOS (from 4.8 s to under 0.7 s on a Mac with Screen Time web filtering)
- Fixed stations that lost their kiosk window after a failed update; a window that crashes now reopens immediately, and the station no longer waits for the window before it connects
- Improved operator UI loading: 20% less code to start, with prompt controls, menus and run details loaded when first needed, and a recovery screen instead of a blank page if the UI hits an error

### Reuse groups of phases across procedures
URL: https://www.tofupilot.com/changelog/reuse-groups-of-phases-across-procedures

Stations that test several products often repeat the same groups of phases in every procedure. Sequences let you write such a group once and call it wherever a procedure needs it.

- Added sequence files: a group of phases with its own setup, main and teardown, called from a procedure with `uses:`, with typed inputs, plug bindings and nested calls.
- Added an outcome per call. A failing phase stops its sequence and runs its teardown, then the procedure decides with `then:` whether to go on.
- Added sequences to the dashboard: the run page nests phases under each call, the unit page lists the calls of each run, and the phase pareto tells apart the same phase used in two calls, with a Sequence filter.
- Added sequence calls to Studio: add a phase or a sequence from any list, set a call's inputs and plugs, and edit the phases of a sequence file in the Inspector.
- Added the sequence fields to the API, every SDK and the CLI commands for runs and phases.

### Keep your stations on your latest deployment
URL: https://www.tofupilot.com/changelog/keep-your-stations-on-your-latest-deployment

Stations now follow your latest production push more reliably, and you can see and act on each step from the dashboard.

- Improved delivery to stations. A station always ends up on your newest production push, even when several builds overlap or a station is linked mid-build, and a rollback stays in place until a newer build is ready.
- Added the reason a push was skipped to the deployment list and page, and a Pull now action on the station page to retry an install at once.
- Improved builds and station updates. Builds retry on their own after a build service hiccup, stations stay responsive while they install an update, and Windows stations now write a log file.

### See teammates' cursors across the operator UI
URL: https://www.tofupilot.com/changelog/see-teammates-cursors-across-the-operator-ui

When an operator and a remote viewer share a station, each now sees where the other is pointing anywhere on the screen, not only over form fields.

- Added live cursors over the whole operator screen between the top and bottom bars, placed the same way on the kiosk and in the dashboard.
- Improved the rings around a field someone else is filling: more space around the field, and no longer cut off at the edge.
- Removed the repeated procedure name from the procedure card on stations with a single procedure, and the hover tooltips on its deployment details.

### Get clear errors when a deployment build fails
URL: https://www.tofupilot.com/changelog/get-clear-errors-when-a-deployment-build-fails

A build that failed because of a wrong root directory or a package index it could not reach showed an error unrelated to the cause, sometimes after minutes of retries.

- Added a clear error when a procedure's root directory does not exist, listing the folders of the repository that hold a pyproject.toml.
- Improved builds that cannot reach a package index or a git source: they stop after the first attempt instead of several minutes of retries, and explain how to build with a private registry by committing uv.lock.
- Fixed build errors that blamed the Python version when the real cause was a registry refusing credentials, a missing package or a dependency that needs another Python.

### Pick the root directory from your repository
URL: https://www.tofupilot.com/changelog/pick-the-root-directory-from-your-repository

A procedure's root directory was a free-text field: a typo such as a dash instead of an underscore went unnoticed until every build failed, and nothing pointed to the redeploy that applies the fix.

- Added suggestions under the root directory setting: the procedure folders found on the repository's default branch, one click to fill the field.
- Added a warning when no pyproject.toml exists at the typed path. Saving stays possible, for a folder that arrives in the next commit.
- Added a Redeploy action to the confirmation after saving, which rebuilds the latest production commit with the new root directory.

### Viewers and operators are now free
URL: https://www.tofupilot.com/changelog/viewers-and-operators-are-now-free

Viewers and operators no longer take a paid seat, on every plan. Give read-only access to everyone who reviews your results, and an account to every operator at the bench.

- Changed seat billing: only owners, admins and developers take a paid seat. Viewers and operators are free and unlimited.
- Changed the Lab plan: invite as many viewers and operators as you need alongside the owner.
- Changed self-hosted licenses: the user limit counts only owners, admins and developers.
- Added Transfer ownership in Settings › Members, which hands the owner role to another member and changes your own role in one step.

### Share plug libraries with git submodules
URL: https://www.tofupilot.com/changelog/share-plug-libraries-with-git-submodules

Keep shared code such as a plug library in its own repository: deployment builds now fetch the procedure's git submodules.

- Added git submodules to deployment builds, nested ones included, with HTTPS, relative or SSH URLs. SSH URLs need no key.
- Added access to private submodule repositories through your git integration. On GitHub, add the repository to the TofuPilot GitHub App. Public submodules work from any host.
- Added clear build errors when a submodule cannot be fetched, naming the submodule and what to fix.

### Change how first pass yield is calculated
URL: https://www.tofupilot.com/changelog/change-how-first-pass-yield-is-calculated

First pass yield now follows one definition everywhere, so the analytics tab, the procedure list, the AI assistant and yield drop alerts all report the same number.

- Updated first pass yield to count each unit once, on its first attempt on the procedure: a retest is never a first pass, even when it happens on another day or inside the selected date range.
- Updated aborted runs so they no longer count as attempts: they are left out of yield, and a unit's first attempt is its first run that was not aborted.
- Fixed the analytics charts going empty when you filter on reference samples: the golden or failing units you select are now included.
- Expect a lower first pass yield on lines that retest often, since retests no longer count as first passes.

### Reject the compromised eMudhra certificates
URL: https://www.tofupilot.com/changelog/reject-the-compromised-emudhra-certificates

This is a security update: eMudhra disclosed that the private keys of several of its certificates were exposed, and we recommend updating your stations, your Python SDK and your self-hosted installation.

- Updated the CLI dependencies so stations refuse the compromised eMudhra certificates, both for their own connections and for the Python environments they install for your procedures.
- Updated the Python SDK dependencies so scripts that use it refuse the same certificates once the SDK is updated.
- Rebuilt self-hosted installations on updated dependencies, including the image that builds your deployments.

### See CLI updates on the station screen
URL: https://www.tofupilot.com/changelog/see-cli-updates-on-the-station-screen

When a station installed a new CLI version, its operator screen closed and reopened with no explanation, which looked like a crash.

- Added a line in the bottom bar of the operator screen while the station downloads a CLI update, with its progress.
- Added an "Installing, restarting…" message right before the station restarts on the new version, then a confirmation once it is back.
- Added a short error message when an update cannot be installed; the station keeps running its current version.

### Filter run analytics by sample class
URL: https://www.tofupilot.com/changelog/filter-run-analytics-by-sample-class

Run Analytics now lets you choose which units count in its charts: production units, golden samples, failing samples or ignored units.

- Added a Sample filter to the Run Analytics sidebar. It shows run counts for each class, and several classes can be selected at once.
- Fixed Run Analytics failing to load from links that select production units, such as links copied from Phase Pareto or Measurement Control.
- Improved drill-downs from Run Analytics to Phase Pareto so both pages count the same units when reference samples are present.

### Limit viewers to their access group's data
URL: https://www.tofupilot.com/changelog/limit-viewers-to-their-access-groups-data

Give a supplier or a customer a viewer seat without opening the rest of your production.

- Added access group scoping: a viewer in a group sees only its stations, their test data and the members of their groups, without email.
- Added Access for new members in Settings › Members, and a switch to turn off API and MCP access for a viewer.
- Changed roles: billing is for owners and admins, admins can edit organization settings, and buttons a role cannot use are hidden.

## News

### TofuPilot Supported by Innosuisse
URL: https://www.tofupilot.com/news/tofupilot-supported-by-innosuisse
TofuPilot receives coaching support from Innosuisse, Switzerland's innovation agency, to accelerate hardware development.

![TofuPilot Supported by Innosuisse](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-02-22-tofupilot-supported-by-innosuisse-featured-image-D69rRqcL6DnUBUl0IE2dhAndKpkT8H.png)

TofuPilot has been accepted into Innosuisse's startup coaching program. Innosuisse is Switzerland's federal innovation agency. Their support means structured coaching, credibility with institutional partners, and a path toward full R&D funding.

Concretely, we're using the coaching to sharpen our go-to-market and prepare for their next funding stage. The goal: make hardware development faster and more efficient for every team using TofuPilot.

More updates soon.

### Enlightra x TofuPilot: Photonic Testing
URL: https://www.tofupilot.com/news/enlightra-x-tofupilot-scaling-photonic-device-testing-from-lab-to-production
How Enlightra uses TofuPilot to manage photonic device testing from EPFL lab prototypes to Y Combinator-backed production.

![Enlightra's engineering team using TofuPilot for their development process](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/enlightra-s-engineering-team-is-using-tofupilot-everyday-from-the-lab-to-project-follow-up-meetings-r3b1n7H3uGirorbZGDJ8WcwnAaXPLF.png)

Enlightra builds multicolor comb lasers for data center optical interconnects, enabling 100x faster data transfer and 10x less energy. They went from EPFL research lab to Y Combinator-backed production, and they use TofuPilot to manage their test data along the way.

![Enlightra chipscale photonic engine on a test rig.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/enlightra-chipscale-photonic-engine-on-a-test-rig-Lyp16TDisROsVRLEgdBwuoAvwgp2bE.png)

## The Problem

Photonic devices require validation across optical performance, thermal characteristics, and electrical interfaces. Each prototype has dozens of precision components with tight tolerances. Before TofuPilot, the Enlightra team was dealing with:

- Test data scattered across multiple instruments and file formats
- Manual correlation between design specs and measured performance
- Inconsistent protocols making results hard to compare across builds
- Limited traceability from results back to component batches

![Johana Bernasconi, Lead Engineer at Enlightra.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/johana-bernasconi-lead-engineer-at-enlightra-W3pvVRqcfp7OooPu90u7gCYlKwGKvt.png)

> We needed a systematic approach to test data management that could scale with our development velocity. TofuPilot provided the infrastructure we needed to maintain engineering rigor while moving fast.
>
> — **Johana Bernasconi**, Lead Engineer at Enlightra

## What They Built

Within two days the team had:

- **Prototype traceability**: Each device build linked to component batch records, assembly procedures, and design revisions
- **Standardized test protocols**: Consistent validation sequences with automated parameter logging across engineers
- **Automated analysis**: Direct correlation between measured performance and specifications with pass/fail determination

![Lou Kanger, Engineer at Enlightra is using TofuPilot to track prototypes of their devices.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/lou-kanger-engineer-at-enlightra-is-using-tofupilot-to-track-prototypes-of-their-devices-5uWXpuVqLXP7kpAaWrZJeqtBob9Hv7.png)

![The TofuPilot Units feature allows for precise tracking of units and validation results.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/the-tofupilot-assets-feature-allows-for-precise-tracking-of-assets-and-validation-results-tYV7SLMMgUqILDG5WKbD3MNJvBG21O.png)

## The Result

When optical power measurements showed variation across device batches, the team could trace issues to specific manufacturing steps and suppliers. That level of traceability wasn't possible before.

> The visibility into our test data has changed how we approach design iterations. We can now make data-driven decisions about design changes and identify optimization opportunities that weren't visible before.
>
> — **Johana Bernasconi**, Lead Engineer at Enlightra

TofuPilot now manages Enlightra's complete validation pipeline: component characterization, packaged device testing, and system-level integration. Test data flows automatically from instruments into structured records, with no manual transcription.

If your team is scaling from lab to production, we'd love to hear how we can help.

### OpenHTF Integration, Docs, and Discord
URL: https://www.tofupilot.com/news/openhtf-integration-docs-tofupilot-discord
TofuPilot adds native OpenHTF support, launches openhtf.org documentation, and opens a Discord community for test engineers.

Many of you have asked us how to standardise hardware test execution. We know how time-consuming building your own test framework can be. We've been there. Luckily, some hardware folks at Google had the same problem and built OpenHTF.

Today, we're shipping native support for OpenHTF tests in TofuPilot. If you're using your own logic with our Python client, nothing changes. But if you're looking for a test framework or already using OpenHTF, you should give it a try.

## OpenHTF in TofuPilot

With just **one line** added to your OpenHTF scripts, our open-source Python client automatically creates a run on TofuPilot, parsing the test phases, measurements, and attachments, while storing the original OpenHTF JSON log for traceability.

![Python code example showing OpenHTF test script with one-line TofuPilot integration for automatic test result upload](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-10-21-openhtf-integration-docs-and-tofupilot-discord-python-script-ty8KZyG7G3jSvZUVZnJaGpA06KrEiE.png)

Check out the OpenHTF integration section in our docs for details.

## A Great Framework Deserves Great Docs

OpenHTF is robust, built from the ground-up for hardware testing, but it lacked proper documentation.

So, we rolled up our sleeves and created openhtf.org, a complete documentation for the framework, packed with screenshots, code snippets and advanced use cases.

![OpenHTF documentation website homepage showing comprehensive guides code examples and framework tutorials at openhtf.org](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-10-21-openhtf-integration-docs-and-tofupilot-discord-openhtf-documentation-89NUxKSCLF8xzkfb74LPK35UaNPKXF.png)

Of course, we couldn't resist adding some TofuPilot integration examples.

## Don't Build Tests Alone

We're on Discord and would love to have you join us for insights from fellow test engineers and real-time connections with our team. If Discord isn't your thing, no worries, you can always reach us at support@tofupilot.com.

![TofuPilot Discord community invitation banner with join link for hardware test engineers](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-10-21-openhtf-integration-docs-and-tofupilot-discord-discord-community-FKlo8R2RriEiU6zaxRp9jygGAksMTx.png)

### TofuPilot Backed by FIT Innovation Foundation
URL: https://www.tofupilot.com/news/tofupilot-backed-by-fit-innovation-foundation
TofuPilot receives support from the FIT Innovation Foundation, a Swiss deep-tech program, to accelerate product development.

TofuPilot has received support from the FIT Innovation Foundation, a Swiss tech program that backs early-stage deep tech. The funding will go directly into product development. More features, faster.

![TofuPilot and FIT Innovation Foundation logos](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-12-02-pro-plans-self-hosting-and-swiss-innovation-award-fit-foundation-logo-JpAJc3MW6ep8Pj6UQIr1pLJyC1EnUB.png)

Thanks to the FIT team for the support.

### Free and Paid Plans, and Self-Hosting
URL: https://www.tofupilot.com/news/pro-plans-self-hosting-swiss-innovation-award
TofuPilot introduces free and paid plans with a Lab free tier, Pro at $50/month, and Docker-based self-hosting for enterprise teams.

Many of you have asked: "What's your business model?" and "Can we self-host TofuPilot?" This week, we're answering both.

## Free & Paid Plans

Defining pricing while balancing simplicity, accessibility, and sustainability took time. After gathering your feedback, here's what we landed on: one free tier and two paid plans.

The **Lab Plan** is free. It includes all key features and supports small production volumes, for as long as you want.

The **Pro Plan**, starting at $50/month, adds secure collaboration, external test deployments, generous run and attachment volumes, and scales with your growth.

The **Enterprise Plan** covers self-hosting, annual billing, dedicated support, and custom needs.

![TofuPilot pricing page showing three plan tiers Lab free tier Pro plan at $50 per month and Enterprise plan with custom pricing](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-12-02-pro-plans-self-hosting-and-swiss-innovation-award-pricing-page-PlOXpVctwYdqt2i0oLIQkIQt3LZDJD.png)

### Smooth Transition

No action needed. Free accounts will automatically switch to the Lab Plan. Paying organizations will transition based on their current setup.

If there are billing changes, we'll notify you directly.

## Self-Hosting

All plans include secure cloud hosting by default. But some organizations face connectivity constraints or IT policies that require on-premises deployment. TofuPilot Self-Hosting is now available as part of the Enterprise Plan.

![TofuPilot self-hosting documentation showing Docker deployment instructions and on-premises setup configuration guides](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-12-02-pro-plans-self-hosting-and-swiss-innovation-award-self-hosting-docs-OO0r1slZ2wWlYx5lOHcUb7RMxgvtDJ.png)

### Multi-Measurement Support and IMU Calibration
URL: https://www.tofupilot.com/news/multi-measurement-support-and-imu-calibration-template
TofuPilot now supports measurements and multi-measurements for OpenHTF and Python client, plus a new IMU calibration template.

TofuPilot now supports measurements, one of the most requested features. This means better analytics for OpenHTF users and a simpler way to write test scripts with the Python client.

## Measurements & Multi-Measurement

Hardware tests usually follow a common structure: data is collected from the Device Under Test or an instrument, errors are logged, then specific measurements are performed and validated before moving to the next phase.

![TofuPilot measurement analytics dashboard displaying voltage timeseries data with pass fail status and validation ranges](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-12-20-multi-measurement-support-and-imu-calibration-template-measurement-features-vpR1cYjdXbAi63z7eNKuUDnQI7LmrX.png)

For example, we acquire a voltage timeseries from a multimeter, compute the average voltage, and verify it falls within the specified range of 3.1–3.5V.

Frameworks like OpenHTF maintain this structure by separating:

**Phases**: Test steps that perform actions and return a pass, fail, or skip status.

**Measurements**: Specific values captured during phases with optional validators.

TofuPilot now fully supports this data model, making your test scripts shorter and enabling more granular analytics.

Measurements are a major change but support incremental adoption on your side.

**OpenHTF users**: No action needed. Tests with multi-measurements now sync automatically.

**Python client users**: You can use the new *phases* argument in the *create_run* function. Steps will be deprecated but remain accessible for now. Existing steps have been migrated automatically.

## Template: IMU Thermal Calibration

Many of you asked for code samples. Here's our first open-source template: calibrating an IMU for temperature on the production line. With two sensors, three axes each, and 36 measurements across two phases, it shows what multi-measurement can do with OpenHTF and TofuPilot.

It's a bit nerdy, but we're sure you'll find it useful. More templates are coming. Share your ideas with us.

![Industrial thermal calibration chamber for IMU sensor testing with precise temperature control system](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-12-20-multi-measurement-support-and-imu-calibration-template-thermal-chamber-duPE3RVygb7JSGmGgxbIDn5F0pXfbC.png)

## TofuPilot's 2024

TofuPilot is now live in 10 factories, processing thousands of runs per month. We won 3 innovation awards, incorporated the company, shipped 30+ releases, and moved into new offices at the Unlimitrust Campus in Switzerland.

![Modern glass entrance of Unlimitrust Campus innovation center in Prilly Switzerland where TofuPilot offices are located](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-12-20-multi-measurement-support-and-imu-calibration-template-unlimitrust-campus-edpOnWiXkZvYvfTUr8LVSMBhp3H34r.jpg)

See you in 2025.

### Explore TofuPilot Without Writing Any Code
URL: https://www.tofupilot.com/news/explore-tofupilot-without-writing-any-code
Explore TofuPilot features instantly with one-click demo scripts — no coding required. Plus updates on OpenHTF maintainers and open positions.

TofuPilot is built to be easy to integrate into your test scripts. But for users wanting to evaluate it quickly, setting up an IDE, installing a package, and writing a script was too many steps.

## Try TofuPilot with one click

![TofuPilot's Welcome Aboard page, allowing users to try the platform without writing code.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2025-02-18-explore-tofupilot-without-writing-any-code-welcome-aboard-page-WduVqw0M6NPXgxcbCAo9StCq9VVz7L.png)

We added buttons in the web app to run sample OpenHTF and vanilla Python test scripts in a virtual console. You can create mock-up test data and explore TofuPilot's features with a single click from the new Welcome Aboard page.

The scripts running in the background are visible on the page and can be copied as snippets to kickstart your own development.

## We met OpenHTF maintainers

OpenHTF is the most popular open-source hardware testing framework. We used it in our previous lives manufacturing drones and recommend it if you don't have an internal solution. But we weren't sure who maintains it today or what its future holds.

Our team flew to Mountain View to meet OpenHTF's founders, John and Joe, who originally built it at Google while working on the Glass project. We also met Akash, who leads its current maintenance at Waymo. A team of five engineers actively maintains the framework, with no internal forks or breaking changes expected.

![OpenHTF founders John and Joe meeting with TofuPilot founders Charlotte and Julien at Google campus in Mountain View](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2025-02-18-explore-tofupilot-without-writing-any-code-openhtf-founders-photo-24Rh1rvTn4CxqNOVTHEX5CekSEZ5Fo.png)

We're looking forward to working more closely with them, helping improve documentation, assisting users, and contributing to the framework's future.

## We're hiring

The past few months have stretched the team. We're growing with two open positions: **Full-Stack Software Engineer** and **Data Scientist Intern**.

They'll join us in our new offices and help build more features for you. If you know someone looking, send them to our careers page.

See you next month.

### TofuPilot at EPFL Startup Champions Seed Night
URL: https://www.tofupilot.com/news/selected-for-the-epfl-startup-champions-night
TofuPilot pitched at EPFL Startup Champions Seed Night in Lausanne, Switzerland's top deep-tech startup showcase.

![TofuPilot at EPFL Startup Champions Seed Night](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-03-21-selected-for-the-epfl-startup-champions-night-featured-image-gUdPk6pNYP5ikvUGzqsy1pwrVBPN89.png)

TofuPilot was selected to pitch at EPFL Startup Champions Seed Night, a showcase for deep-tech startups from the EPFL ecosystem in Lausanne, Switzerland.

We presented TofuPilot to investors, founders, and engineers building hardware products. The event brought together 12 startups across robotics, medtech, photonics, and space, all solving real engineering problems.

- **Event**: EPFL Startup Champions Seed Night
- **Date**: Thursday, 25 April 2024
- **Time**: 17:30 - 21:30 GMT+2
- **Location**: Forum Rolex (Rolex Learning Center EPFL), 1015 Lausanne

Thanks to the EPFL Innovation Park team for the selection.

### Histograms, Assembly History, and Batches
URL: https://www.tofupilot.com/news/test-histogram-assembly-history-batch-number-support-more
TofuPilot adds test histograms, date filters, sub-unit assembly tracking, string control charts, and batch number support for production teams.

You asked for more analytics after our August release. Here's what shipped in September.

## Visualize Distribution with Histograms

We've added a histogram next to the control chart for numeric steps. This lets you see the distribution of measurements across units and runs, and compare them with test limits and 3-sigma values.

No code changes needed. Histograms appear automatically for any numeric step.

![Screenshot of TofuPilot's histogram feature, showing measurement distribution and Gaussian curve.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-10-03-test-histogram-assembly-history-batch-number-support-and-more-histogram-feature-cohL7FMZExePyq4hqS56wQfGYSSywl.png)

## Date Filters for Time-Based Comparisons

You can now compare test results across different time ranges. Available on all analytics pages, no setup required.

![Screenshot of TofuPilot's date filter, allowing selection of test results across different time ranges.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-10-03-test-histogram-assembly-history-batch-number-support-and-more-date-filter-k6rhN1HSKg6rx8L0mnmrs6qdWkdQbf.png)

## Track Sub-Unit Activity

You can now track any assembly or rework timeline with the new units activity panel. Operations are logged automatically from test runs or through new dedicated API endpoints.

Using the Python client? Pass `sub_units` in your `create_run` call. The activity panel populates automatically.

![Screenshot of TofuPilot's units activity panel, displaying logged operations from test runs and API inputs.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-10-03-test-histogram-assembly-history-batch-number-support-and-more-units-activity-panel-JVFTAgB8BRzGdiQ7JetR1nX8UHmbZg.png)

## Control Charts for String Steps

You can now log and track text-based measurements, such as firmware versions and component IDs, and visualize their distributions using new control charts for string data.

These work the same way as numeric charts. Just pass string values in your measurements.

![Screenshot of TofuPilot's control charts for string data, visualizing distributions of firmware versions.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-10-03-test-histogram-assembly-history-batch-number-support-and-more-string-control-charts-VrrSgVMTzf5Lb9ojVx2ru40PHiGxQ0.png)

## Batch Number Support

You can now add batch numbers alongside unit serial numbers and compare test results between batches. Just pass `batch_number` in your `create_run` call.

![Python code showing TofuPilot API usage for tracking batch numbers with unit serial numbers for production traceability](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-10-03-test-histogram-assembly-history-batch-number-support-and-more-batch-numbers-python-ViiD55FaR0bTipyGPKm2iiauH5Q3C9.png)

We'd love to hear your thoughts on these updates.

### TofuPilot Framework is here
URL: https://www.tofupilot.com/news/tofupilot-framework-is-here
TofuPilot Framework launches as an open-source industrial test orchestrator with TofuPilot Studio desktop IDE for building hardware test procedures.

Developing hardware tests shouldn't require vendor lock-in, extensive custom development, or Windows-exclusive deployment options.

Today's release introduces two complementary products:

## TofuPilot Framework

This open-source industrial test orchestrator enables you to:

- Execute Python test phases sequentially or in parallel
- Create operator interfaces without frontend development expertise
- Integrate hardware resources through reusable plugs
- Record measurements and attach files with automatic report generation
- Test multiple units concurrently

![Framework Overview](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-28c5c5a4-f726-7b93-0fda-92f1d88c632c-mUpfE5WxvU6o1Z3PuRUvW5psraOAeM.png)

## TofuPilot Studio

This free lightweight IDE (available for Windows, Linux, and macOS) accelerates test development by providing:

- Visual operator UI editing
- Test sequence flow configuration
- Phase and plug execution/debugging
- Report synchronization to TofuPilot Dashboard

![Studio Interface](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-27073aba-7265-1358-6d5b-ff82c8a0d961-qaB1Fz89GqzhdY0VqeWGGSiocF1dLo.png)

## Getting Started

Download TofuPilot Studio for your operating system, select a template, and begin constructing test procedures immediately.

![Getting Started Guide](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-31f3a438-31a7-9d9f-bb2a-301cd59a9548-6dcv2azvFVykJVjNEAQW3gXFUEgKWO.png)

## Community

Connect with the community on Discord, explore the source on GitHub, or check the docs. All available from tofupilot.com.

Happy holidays, and happy testing.

### 1D, 2D, and ND Measurement Arrays
URL: https://www.tofupilot.com/news/launch-week-multi-dimensional-measurements-day-2
TofuPilot Launch Week Day 2: Upload and visualize 1D, 2D, and multi-dimensional measurement arrays with graph previews and CSV export.

Yesterday, your test logs found a home. Today, measurements arrays get the same attention.

## Tuesday: Multi-dimensional Measurements

Until now, measurements in TofuPilot were limited to single values. But real-world tests often produce more complex data: sampled over time, across multiple axes, or as full waveforms. TofuPilot now supports 1D, 2D, and ND measurement arrays.

![TofuPilot multi-dimensional measurements interface with graph previews](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/tofupilot-multidim-screenshot-yZgkn5EtQXCQm0x5LIOqYc59ENWpLh.png)

You can explore multi-dimensional measurements directly in the run interface. TofuPilot enables:

- Full graph previews of time-series and array data
- Toggle visibility for individual measurements
- Export data series as CSV files for external analysis

Using OpenHTF? You don't need to change anything. OpenHTF native multi-dimensional measurements are now processed for all new runs.

Using the vanilla Python client? Just pass a list or array in your measurements. Our updated documentation includes a simple snippet to get you started.

Multi-dimensional measurements are automatically included in analytics for duration and yield, but Cpk isn't available yet as it requires custom validators - support for that is coming soon.

## Inside TofuPilot: Backed by Switzerland's Startup Ecosystem

TofuPilot wouldn't exist without the support of Switzerland's top innovation programs. Venture Kick and La FIT provided our first funding to get started. EPFL, Innosuisse, and the Canton of Vaud connected us with expert coaching, financial support, and a strong network of tech founders.

![Summary of TofuPilot's sponsors and supporters in Switzerland's startup ecosystem](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/tofupilot-sponsors-summary-RfwZ6w1PJWAgWPlp77KUFIB3cMRyBI.png)

Thanks to all these programs for their early support. See you tomorrow!

### Operator UI for the Factory Floor
URL: https://www.tofupilot.com/news/launch-week-operator-ui-day-3
TofuPilot Launch Week Day 3: A plug-and-play Operator UI with real-time test streaming and browser-based operator input for the factory floor.

Console logs and prompts work fine for developers, but end users of test scripts, operators and technicians, need clearer interfaces. Building custom UIs from scratch is costly and time-consuming.

## Wednesday: Operator UI

The new TofuPilot Operator UI provides a plug-and-play web interface that eliminates the need for custom UI development.

Key features:

- Real-time streaming of test phases, measurements, logs, and metadata
- Browser-based user input through OpenHTF's native plug
- Secure, permission-controlled real-time connections

![Operator UI Interface](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-ed2630f1-4ca1-5eae-23be-bbfb4275dc13-7IsFI6W1vqYd3RXOvE7HbLadrtRf1Q.png)

The Operator UI works with any OpenHTF script. Just update your tofupilot package, run your test, and real-time updates appear automatically. Prompts work through the existing `user_input` plug without additional integration code.

Scripts using the vanilla Python client aren't yet supported but are planned for a future release.

## Looking Ahead: Role-Based Permissions

The Operator UI fully integrates with TofuPilot Stations for real-time monitoring. Upcoming features include:

- Granular permissions by user role
- Viewer-only accounts with new pricing
- External deployment capabilities with maintained control

![Feature Preview](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-a8d22b3c-a4dc-6c58-b0f7-e2eea8b2fa66-Z75vjtXKoIZEM9Ryreg9Jg8sqnEuUY.jpeg)

See you tomorrow for Day 4!

### API Activity Page and Launch Week Recap
URL: https://www.tofupilot.com/news/launch-week-api-activity-day-5
TofuPilot Launch Week finale: new API Activity page for monitoring usage, errors, and performance, plus a recap of all week's releases.

It's the last day of Launch Week, and we're closing with something every test engineer needs in production: visibility and control over the TofuPilot API.

## Friday: API Activity

We know your test stations run in manufacturing environments where updates aren't always easy, and reliability is critical. As the platform grows with new features and performance improvements, we want to help you stay in control.

That's why we added the new API Activity page for all users. It lets you:

- See who's calling the API, when, and with which client version
- Catch errors quickly, with direct links to the related test runs
- Stay informed about upcoming changes or deprecations
- Track the performance of your API calls over time

![TofuPilot API Activity page showing API usage, errors, and performance metrics](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/tofupilot-api-activity-screenshot-s9G6Jh60kNVdaR8IygXQFPWxSoamxB.png)

No code changes needed. The Activity page is available now in your dashboard.

We'll always do our best to avoid breaking backward compatibility. If we ever have to, you'll get several weeks' notice to prepare.

## Launch Week Recap

Here's everything we shipped this week:

- Support for full test log ingestion
- Multi-dimensional measurements
- A new plug-and-play Operator UI
- Our new Pro Plan to support teams in production

We added a lot, but also made the ingestion under the hood up to 60% faster.

Until next time. Happy testing.

### REST API and Python SDK for Hardware Testing
URL: https://www.tofupilot.com/news/test-hardware-faster-with-new-python-sdk
TofuPilot launches its REST API, Python SDK, redesigned Testing Hub UI, and improved onboarding to help hardware teams test faster.

Uploading test results used to require multiple UI actions. Now it's a single API call.

## API & Python SDK

We've built a REST API and a dedicated Python SDK (available on pip) to automate test result uploads from your scripts. With one call you can:

- Create a new test run from any procedure
- Link runs to units under test, creating them if they're new
- Save test run duration and status
- Generate a full test report programmatically with our new "Smart Fields" feature

![Python code example demonstrating TofuPilotClient SDK usage for creating units uploading test results and attachments via REST API](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-06-18-test-hardware-faster-with-new-python-sdk-python-client-M3AJYDhYrQzlwHiSTmaF6uIE4khEZp.png)

## New Dashboard

With the API enabling significantly more test results, our old UI wasn't cutting it for navigating that volume. We designed a new dashboard from the ground up.

![Screenshot of TofuPilot's new Procedures page displaying weekly runs, tested components, FPY, and average test time per procedure.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-06-18-test-hardware-faster-with-new-python-sdk-procedures-page-qUOovGGRBszQWT131Dh0c1L6T9on18.png)

## Get Started

We've improved onboarding and invite flows, with a new "Get Started" tab accessible from the sidebar.

![TofuPilot Get Started tab in sidebar navigation for quick onboarding and setup guides](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-06-18-test-hardware-faster-with-new-python-sdk-get-started-FE6vefWYsEqMLqWJnOZhQhmihKDalk.png)

We'd love to hear your feedback.

### Test Logs in the Run Page
URL: https://www.tofupilot.com/news/launch-week-logger-day-1
TofuPilot Launch Week Day 1: View, filter, and search test logs directly in the Run page. Works with OpenHTF logger out of the box.

TofuPilot turns one this week. To mark it, we're shipping one feature per day, all from your feedback.

## Monday: Logger View

When you run a test, phases and measurements tell part of the story - but console logs provide essential context, helping you trace execution, catch errors, and debug issues effectively.

![TofuPilot Logger UI screenshot showing log filtering and search capabilities](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/tofupilot-logger-ui-screenshot-eEoI2JuWwHUZS6oOxmeqk5AGA8y7KV.png)

You can now view logs directly on the Run page in TofuPilot. Easily:

- Filter by log level (debug, info, warning, error, critical)
- Search by keyword
- Sort by timestamp
- See source file names and line numbers

Using OpenHTF? You don't need to change anything, logs from the OpenHTF logger are now handled automatically.

Using the vanilla Python client? Just add a small snippet to your scripts. Our new Logger documentation page will show you exactly how.

## Inside TofuPilot: Doubling the Team in April

With more users and growing feature requests, we've expanded our team to move faster. Dehlya and Quentin have joined as full-stack engineers, and Manon as a data engineer focused on ingestion and analytics, all now working from our new offices in Prilly, Switzerland.

![TofuPilot team at their new offices in Prilly, Switzerland](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/tofupilot-team-picture-XhqpDbfsnqKpVz4fe0xcS0qiTW0KLP.png)

See you tomorrow with something new!

### API Docs, Sub-Units, and Attachments
URL: https://www.tofupilot.com/news/new-api-documentation-sub-units-attachments-support
TofuPilot adds sub-unit tracking for assemblies, file attachments for test runs, and launches new developer documentation for its REST API.

TofuPilot's API and Python client now support sub-units, attachments, and we've launched new documentation.

## Sub-Units

The API now accepts optional sub-unit parameters on the test run endpoint. Assembly test stations can automatically assign sub-units as children of the assembly units being tested, giving you full unit traceability across the assembly line.

Sub-units appear in the Units table and on dedicated Unit pages.

![Screenshot of TofuPilot's Unit page showing units, sub-units, serial numbers, and test counts in a tree view.](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-07-02-new-api-documentation-sub-units-and-attachments-support-unit-page-tree-view-or2UVp9T0M0XRCeUgIQx57SqBm8jfc.png)

## Attachments

Test scripts often produce files beyond text and numbers: calibration images, graphs, videos. You can now upload them as attachments directly from the Python client.

Attachments can be viewed, downloaded, or deleted from the dedicated run page.

![TofuPilot test run page showing file attachments panel with uploaded calibration images graphs and JSON data files](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-07-02-new-api-documentation-sub-units-and-attachments-support-run-page-attachments-kL4V1l4xBbjravCqHmIlecq3gwhl1F.png)

## Documentation

Our new documentation is live on tofupilot.com, continuously updated as the platform evolves.

![TofuPilot documentation homepage featuring Getting Started guide with API examples and integration tutorials](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-07-02-new-api-documentation-sub-units-and-attachments-support-documentation-getting-started-rpdPMEcMGYnihexk4P46Qxu67wnPVk.png)

We'd love to hear how you're using these. Let us know.

### Real-Time or Offline Test Sync?
URL: https://www.tofupilot.com/news/real-time-or-offline-test-sync
TofuPilot adds real-time OpenHTF streaming and improved offline upload so hardware teams can sync test data from any environment.

When developing or running tests in connected environments, you want to see results immediately. But testing doesn't always happen online and offline uploads need to be possible, too. Our new release improves both scenarios.

## Online Streaming

In connected environments, you want test results instantly, not after they finish. Our new release introduces real-time streaming for OpenHTF test scripts, powered by a new secure real-time infrastructure. Tests appear on your TofuPilot Runs page as soon as they start, with real-time step status tracking.

![TofuPilot dashboard showing real-time OpenHTF test execution with live status updates for each test phase](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-11-07-real-time-or-offline-test-sync-openhtf-real-time-NDC5oqXLahxYi0WWXnwX4JbSt7BHQI.png)

No code changes are needed on your side. Just update the TofuPilot Python client, and live streaming kicks in automatically.

## Offline Sync

TofuPilot now offers improved support for offline uploads when test stations aren't directly connected, allowing offline uploads without losing accuracy. With the optional "started_at" field, analytics reflect the actual test date instead of the upload date, and upload capacity is now up to 1,000 runs per minute.

Check out the offline upload section in our docs for more on this use case.

## Updated Docs

We've revamped the docs with insights from our work on openhtf.org. You'll find new guides, feature and code examples to help you get the most out of TofuPilot.

![TofuPilot documentation quickstart section featuring step-by-step guides code examples and integration tutorials](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-11-07-real-time-or-offline-test-sync-documentation-quickstart-xWP3BC4TMP5aKNb6PNWD9iUh9wrvQY.png)

We'd love to hear how these work for your setup.

### Introducing TofuPilot: Hardware Test Analytics
URL: https://www.tofupilot.com/news/introducing-tofupilot-accelerating-hardware-innovation
TofuPilot launches as a plug-and-play analytics platform for hardware test data, with OpenHTF and Python SDK support out of the box.

![TofuPilot - Logo Reveal](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-01-28-introducing-tofupilot-accelerating-hardware-innovation-featured-image-uqBTEAUCMxQmNaPS0TTDoXIWjaCOap.png)

Over the last few months, we've met over a hundred hardware teams working on complex multi-disciplinary products like autonomous robots, spacecraft, and medical devices. We've heard their feedback: traditional industrial solutions don't match their speed, lack flexibility, and don't fit modern development workflows.

On the other hand, generic collaboration tools create unstructured data that's hard to manage, analyse, and connect, slowing down progress as projects grow.

This inspired us to build TofuPilot, a plug-and-play analytics platform for hardware test data. You connect it to your test scripts (OpenHTF, pytest, or our Python SDK), and it gives you runs, measurements, FPY tracking, and Cpk analysis out of the box.

We picked a name that reflects a departure from traditional tooling: something short, memorable, and distinctive. Our new name comes with a mascot, for which the community is already voting on a name.

Welcome aboard TofuPilot.

### TofuPilot Selected by Venture Kick
URL: https://www.tofupilot.com/news/tofupilot-selected-by-venture-kick
Learn how TofuPilot was selected by Venture Kick, Switzerland's leading startup program, and what it means for product development.

![TofuPilot Selected by Venture Kick](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-01-31-tofupilot-selected-by-venture-kick-featured-image-Pg5CIMMudsf0mjbUDwxHTKzq8dfbXP.png)

TofuPilot has been selected by Venture Kick, Switzerland's leading startup funding program. The program provides CHF 150K in equity-free funding across three stages, along with mentoring and access to investor networks.

For us, this means more resources to ship features faster and grow the team. For you, it means a more stable product with a clearer path forward.

Thank you to everyone who's been part of the journey so far. We'll keep building.

### Rust SDK for Hardware Test Data
URL: https://www.tofupilot.com/news/rust-sdk-for-hardware-test-data
TofuPilot ships an official Rust client covering the full V2 API with async builders, typed errors, retries, and file upload helpers.

![Rust SDK for TofuPilot hardware test data](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news-1775071248267-HxAN3Cf41kVAiOo1Y4H46mOihWvcPl.png)

Not every test station runs Python. Some teams need bare-metal performance, memory safety, or just prefer `cargo test` over `openhtf` or `pytest`. TofuPilot now has an official Rust client.

## Create a run

The SDK uses async builders. Required fields are enforced at compile time, optional ones chain before `.send()`.

```rust
use tofupilot::TofuPilot;
use tofupilot::types::*;

let client = TofuPilot::new(std::env::var("TOFUPILOT_API_KEY").unwrap());

let run = client.runs().create()
    .procedure_id("FVT-001")
    .serial_number("SN-042")
    .part_number("PCB-V2")
    .outcome(Outcome::Pass)
    .started_at(chrono::Utc::now() - chrono::TimeDelta::minutes(5))
    .ended_at(chrono::Utc::now())
    .send()
    .await?;
```

## Phases and measurements

You can push structured test data in one call. Builders keep the nested types readable.

```rust
let run = client.runs().create()
    .procedure_id("FVT-001")
    .serial_number("SN-042")
    .part_number("PCB-V2")
    .outcome(Outcome::Pass)
    .started_at(now - chrono::TimeDelta::minutes(5))
    .ended_at(now)
    .phases(vec![RunCreatePhases::builder()
        .name("voltage_check")
        .outcome(PhasesOutcome::Pass)
        .started_at(now - chrono::TimeDelta::minutes(5))
        .ended_at(now - chrono::TimeDelta::minutes(3))
        .measurements(vec![RunCreateMeasurements::builder()
            .name("output_voltage")
            .outcome(ValidatorsOutcome::Pass)
            .measured_value(serde_json::json!(3.3))
            .units("V")
            .build()
            .unwrap()
        ])
        .build()
        .unwrap()
    ])
    .send()
    .await?;
```

## Upload files in one call

Attach files directly to runs or units through the `attachments()` sub-resource.

```rust
// Upload a file to a run
let id = client.runs().attachments().upload(&run.id, "report.pdf").await?;

// Upload a file to a unit
let id = client.units().attachments().upload("SN-0001", "calibration.pdf").await?;

// Download an attachment
client.runs().attachments().download(&url, "local-report.pdf").await?;

// Delete a unit attachment
client.units().attachments().delete("SN-0001", vec![id]).await?;
```

## Typed errors

Every API error maps to a Rust enum. No string matching.

```rust
use tofupilot::Error;

match client.runs().get().id("nonexistent").send().await {
    Ok(run) => println!("Found: {}", run.id),
    Err(Error::NotFound(e)) => println!("Not found: {}", e.message),
    Err(Error::Unauthorized(e)) => println!("Bad key: {}", e.message),
    Err(e) => println!("Other: {e}"),
}
```

## Retries built in

The client retries on 429 and 5xx with exponential backoff and jitter. You can tune it or leave the defaults.

## Every V2 endpoint

Runs, units, parts, revisions, procedures, batches, stations, versions, users. 132 integration tests run on every deployment.

The SDK is open source under MIT. Source on [GitHub](https://github.com/tofupilot/rust), package on [crates.io](https://crates.io/crates/tofupilot).

```bash
cargo add tofupilot
```

### Workflows: Tests to Odoo, Linear & More
URL: https://www.tofupilot.com/news/workflow-automation-for-test-data
TofuPilot now has a visual workflow engine. Trigger actions when test runs complete, route data to Odoo or Discord, and build logic with filters and branches.

![TofuPilot workflow editor showing a visual automation flow triggered by a test run](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/news-workflows-N0V9X18PQdDe1AXj9em8kBrRpmPxeL.png)

You can now build workflows that react to your test data. A run comes in, TofuPilot checks conditions, and fires actions: emails, Discord messages, Odoo records, Linear tickets, or any HTTP endpoint. All configured from a visual editor with a settings sidebar for each node.

## Triggers

![Trigger selection panel](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/news-workflows-triggers-RYxA3CBa21FBdIfKcVo0h39zTXbqpF.png)

You can start a workflow from any of these events:

- **Run created**, **Unit created**, **Part created**, **Revision created**, **Batch created**
- **Recurring schedule** with frequency (daily, weekly, monthly), time, and timezone picker
- **Manual trigger**

## Actions

![Action palette by category](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/news-workflows-actions-02yAiGpbhEJBtTBBdIYUVEqxc7kWob.png)

You can send notifications, push data to external tools, or run utilities. Each action has its own settings panel in the sidebar.

**Email**: Pick recipients from your team (with avatar search), write a subject and body. All text fields have a variable explorer that lets you browse and insert fields from the trigger and upstream nodes (serial number, outcome, procedure, part number...).

**Discord**: Paste a webhook URL, compose a message with the same variable explorer.

**Odoo**: Select a connection, pick a model from a live dropdown (fetched from your Odoo instance), and map fields with a dynamic builder. Three actions available: Create Record, Update Record, Post Message.

**Linear**: Select a connection, pick team, project, state, priority, labels, and assignee from live dropdowns. Create Issue or Add Comment.

**HTTP Request**: Choose method (GET, POST, PUT, PATCH, DELETE), enter a URL, and write a body with variables.

**Utilities**: Delay (configurable duration), Formula (math on variable values), Celebrate (confetti on completion).

## Conditions

![If/Else condition builder](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/news-workflows-filters-m25Vhcjn3FBQMYyho4wU4sQOlOj5Ql.png)

You can route test data down different paths based on any field. The condition builder lets you pick a variable, choose an operator, and set a value.

- **Filter**: stops the flow unless conditions are met. Supports multiple rules with AND/OR logic.
- **If / Else**: splits into two paths based on a condition.
- **Switch**: fans out into multiple named branches based on a field value. Each branch gets its own downstream actions.

Operators include equals, not equals, contains, starts with, greater than, less than, is empty, and relative date comparisons.

## Example: quality alert on failed run

1. **Trigger**: Run created
2. **Filter**: outcome equals FAIL
3. **Switch** on procedure:
   - `FVT-001`: email quality team + Linear issue
   - `ICT-001`: Discord alert to electronics channel
   - Default: email to test engineering lead

Failed tests go to the right people with full context. No glue scripts.

## Example: sync serial numbers to Odoo

1. **Trigger**: Run created
2. **Filter**: outcome equals PASS
3. **Odoo Update Record**: write the serial number to the corresponding production lot in Odoo using the part number to find the record

Your serial numbers stay in sync between TofuPilot and Odoo without manual entry.

## Execution tracking

![Execution view with step status and timing](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/news-workflows-executions-Z7PetWsf0lxWlaO9NzWkFMlOnItZkn.png)

You can inspect every workflow run step by step. Each execution shows status, duration, errors, and the data that flowed through each node. The view follows the same layout as the editor.

Running executions refresh live. Failed steps show the error inline. Filter by status and drill into any execution.

## The editor

You can build everything from a visual canvas. Add nodes from the palette, wire them up, and configure each one in the sidebar. Drag to pan, scroll to zoom. Auto-saves.

## What's next

InvenTree and Slack integrations are in progress. Webhook triggers (so external systems can start workflows) are next. If you need a specific integration, tell us on Discord.

### Test Steps and Open-Source Python Client
URL: https://www.tofupilot.com/news/test-steps-oss-python-client
TofuPilot introduces structured test Steps for pass/fail and numeric measurements, plus open-sources the Python client under MIT license.

Two updates shipping today: structured test steps and our open-source Python client.

## Introducing Steps

Steps allow you to log structured test results, from simple pass/fail checks to advanced numeric tests with limits.

![Steps visualization](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-e5cb85c8-697d-f865-ad6d-7eccac807647-ZSRhx8OsR84kNPoBA5hjWZp0DW5Iy5.png)

The Run page now displays step-by-step execution details, while the Procedure page aggregates insights across all runs, including average durations and pass rates, with more analytics coming soon.

![Analytics dashboard](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-b5f7623e-93fa-887c-d5be-cc43a96acc65-iPYqqCFpwK4ZQUwCtv6PZzNRA1iJ6p.png)

You can incrementally adopt Steps in existing scripts without disrupting current workflows.

## Open Source Python Client

Our Python client is now open source under the MIT license. Review the implementation before production deployment and contribute through GitHub.

The repository is available on GitHub for community contributions and feature requests.

### Test Step Insights and First Pass Yield
URL: https://www.tofupilot.com/news/test-steps-insights-first-pass-yield-limits-accuracy-and-more
TofuPilot launches Test Step Insights with first-pass yield tracking, limits accuracy analysis, and test duration breakdown.

Building in-house test analytics is resource-intensive. TofuPilot's new **Test Step Insights** module gives you manufacturing test analytics out of the box: plug-and-play, no dashboard development needed.

The module helps you identify real issues and separate them from false failures caused by test errors. Here's what it covers.

## First-Pass Yield at Step Level

First-pass yield is the key metric for test system quality. TofuPilot now provides it in real-time, broken down by step, so you can spot which steps cause the most failures.

![TofuPilot first pass yield chart showing FPY percentage trends over time with step-by-step breakdown for failure analysis](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-08-28-test-steps-insights-first-pass-yield-limits-accuracy-and-more-fpy-over-time-CZfUqgg4wBx3Hd9hyNlD7hMX82sGZJ.png)

The module also ranks steps by priority so you know where to focus first.

![TofuPilot test steps analysis with color-coded prioritization showing critical failing steps in red and improvement opportunities](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-08-28-test-steps-insights-first-pass-yield-limits-accuracy-and-more-test-steps-prioritization-7CmI2kt2JidgNUiB7owNASb5x6bm4h.png)

## Limits Accuracy & Deviations

Many tests require comparing measured values to precise limits. TofuPilot now automatically computes how well each step meets its limits, giving you a clear view of your procedure's process capability.

![TofuPilot process capability analysis showing Cpk scores with color indicators for test limit accuracy and deviation detection](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-08-28-test-steps-insights-first-pass-yield-limits-accuracy-and-more-process-capability-1Gf3J2mAq8h8XNVezMMV5KwZKqd8Op.png)

This makes it straightforward to spot incorrectly set limits or sudden deviations.

## Test Duration Analysis

A breakdown of each step's duration helps you identify the time-consuming steps slowing down your test benches.

![TofuPilot test duration breakdown showing execution time analysis for each test step to identify bottlenecks and optimization opportunities](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/2024-08-28-test-steps-insights-first-pass-yield-limits-accuracy-and-more-test-duration-analysis-b4AmsoygOkuGD2qOfxYpPZ0myhTAF3.png)

No code changes needed. These analytics appear automatically for any procedure with numeric steps.

We'd love to hear your thoughts.

### Stations, Procedure Versions, and Roadmap
URL: https://www.tofupilot.com/news/stations-procedure-versions-public-roadmap
TofuPilot introduces Stations for secure production deployment, procedure version tracking, and a public development roadmap.

Three major features ship today: secure production deployment stations, procedure version tracking, and a public roadmap.

## Stations for Production Deployment

![Stations feature](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-93d04979-81cb-1bc2-8183-4d0db282d315-NNfp9EtleRsTdxnsitnVzclGzG0dnv.png)

Stations let you deploy test procedures to production securely. Each Station can be associated with one or multiple procedures, providing isolation and security through unique API access with limited permissions.

Existing API keys continue to work. Stations add a new, scoped option alongside them.

## Procedure Version Tracking

![Version tracking interface](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-be558a0c-6bcc-0d86-f277-b3181e940179-d4jNyheoQ3EsecknztayrNqU1DD9UM.png)

The platform now supports logging and filtering by procedure version, making it simpler to pinpoint issues and determine when they first appeared.

Using the Python client? Pass `procedure_version` in your `create_run` call. Using OpenHTF? The version is picked up automatically from your script metadata.

## Public Roadmap

![Roadmap announcement](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/newsletter-2699778d-5da3-631c-f550-a151c222a9db-eVsh3ioSWz5fDuiwmO388wL6OOJ7Bv.jpeg)

We've published our development roadmap on tofupilot.com. Your feedback shapes what we build next.

### Pro Plan: Stations, Orgs, and Billing
URL: https://www.tofupilot.com/news/launch-week-pro-plan-day-4
TofuPilot Launch Week Day 4: The new Pro Plan adds Stations for secure deployment, multi-user orgs, and self-serve subscription management.

Today, we're making it easier to scale your testing to production and shape what's next for TofuPilot.

## Thursday: Upgrade to Pro

Or stick with Lab - it's free forever. But if you're ready for production, Pro unlocks everything you need. Here's what each plan includes:

**Lab:**
- 100 test runs /month
- 10 GB /month attachment storage
- Access to the Analytics and the new Operator UI

![TofuPilot Operator UI web interface screenshot](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/tofupilot-operator-ui-screenshot-EggH1Oxtrhc1paJJaTrGxXLMn6Hfoj.png)

**Pro unlocks:**
- Stations for secure deployment (scoped API keys tied to procedures)
- Multi-user organization support
- 1,000 runs /month and 100 GB /month attachments before usage-based billing

You can check full plan details on our pricing page.

**What's ahead:**
- Starting May 16, Pro features will require an active subscription
- Existing Pro subscribers will be upgraded automatically

We know engineers love nothing more than a good sales call - but with the new Subscription page, you can check your usage and upgrade without one.

![TofuPilot subscription UI showing usage and upgrade options](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/tofupilot-subscription-ui-screenshot-Fndb6TraX8TCRlaMNoJVhNguUwGZxq.png)

That said, we'd still love to meet. Our calendar (or inbox) is always open to hear about your challenges, assist you with on-boarding, or chat about what's next.

![TofuPilot mascot at airport gate, metaphor for user signup journey](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/tofupilot-mascot-airport-signup-QvzMyyjVxsCuW4BezeF3QxJPQ2Oych.jpeg)

See you tomorrow for the final day of our first anniversary Launch Week!

### TofuPilot now speaks C#
URL: https://www.tofupilot.com/news/csharp-sdk-dotnet-hardware-testing
TofuPilot now has an official C# client built on community work by @Hylaean, covering the full V2 API with typed responses and errors.

![TofuPilot C# SDK code example showing how to create a test run with measurements from .NET](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news/news-csharp-yQynqzRgcZX1u2qrQ5KuDWzgHb3v8h.png)

Python isn't the only language on the factory floor. Many hardware teams run .NET for TestStand sequences, LabVIEW integrations, or custom test executors. Until now, they had to wrap raw HTTP calls to use TofuPilot.

Our community member **@Hylaean** changed that. He built the **first C# client from scratch**, covering core endpoints with clean async patterns. It was great work, and teams started using it in production.

We took it one step further. Starting from his foundation, we auto-generated **full V2 coverage** from our OpenAPI spec, added typed models for every request and response, and wired up **131 xUnit tests** that run on each deployment. The SDK is open source under [MIT license on GitHub](https://github.com/tofupilot/csharp).

## Install

You can install the SDK from NuGet:

```bash
dotnet add package TofuPilot
```

## Create a run with measurements

You can create a run with phases, measurements, and limits in one call:

```csharp
using TofuPilot;
using TofuPilot.Models.Requests;

var client = new TofuPilot(apiKey: Environment.GetEnvironmentVariable("TOFUPILOT_API_KEY")!);

var run = await client.Runs.CreateAsync(new RunCreateRequest
{
    ProcedureId = "FVT-001",
    SerialNumber = "SN-042",
    PartNumber = "PCB-V2",
    Outcome = RunCreateOutcome.Pass,
    StartedAt = DateTime.UtcNow.AddMinutes(-5),
    EndedAt = DateTime.UtcNow,
    Phases = new List<RunCreatePhases>
    {
        new()
        {
            Name = "Voltage Test",
            Outcome = RunCreatePhasesOutcome.Pass,
            StartedAt = DateTime.UtcNow.AddMinutes(-5),
            EndedAt = DateTime.UtcNow,
            Measurements = new List<RunCreateMeasurements>
            {
                new()
                {
                    Name = "Output Voltage",
                    Outcome = RunCreateMeasurementsOutcome.Pass,
                    MeasuredValue = 3.3,
                    Validators = new List<RunCreateValidators>
                    {
                        new() { Operator = ">=", ExpectedValue = RunCreateExpectedValue.CreateNumber(3.0) },
                        new() { Operator = "<=", ExpectedValue = RunCreateExpectedValue.CreateNumber(3.6) },
                    },
                },
            },
        },
    },
});
```

## Upload files in one call

Attach files directly to runs or units through the `Attachments()` sub-resource:

```csharp
// Upload a file to a run
await client.Runs.Attachments().UploadAsync(run.Id, "report.pdf");

// Upload a file to a unit
await client.Units.Attachments().UploadAsync("SN-0001", "calibration.pdf");

// Download an attachment
await client.Runs.Attachments().DownloadAsync(downloadUrl, "local-copy.pdf");

// Delete a unit attachment
await client.Units.Attachments().DeleteAsync("SN-0001", new List<string> { attachmentId });
```

## Every V2 endpoint, fully typed

Runs, units, parts, revisions, procedures, batches, stations, users. Every resource has full CRUD with typed requests and responses.

## Updated docs with C# examples

All [dashboard documentation](https://www.tofupilot.com/docs) pages and every [API reference](https://www.tofupilot.com/docs/api/rest-api) endpoint now include C# code examples alongside Python. Both are generated from the OpenAPI spec, so they stay in sync as we ship new features.

## What's next

We're working on a **NI TestStand integration** so .NET teams running TestStand sequences can push results to TofuPilot without custom scripts. If you're using TestStand or LabVIEW, we'd like to hear how you export test data today.

The SDK targets **net10.0 LTS** and is available on [NuGet](https://www.nuget.org/packages/TofuPilot).

### MCP: Connect TofuPilot to Claude, Cursor
URL: https://www.tofupilot.com/news/connect-ai-agents-to-your-test-data
TofuPilot now has an MCP server. Connect Claude, Cursor, or any MCP client to your test data, authorized in your browser with no API key.

![TofuPilot MCP server connecting Claude, Cursor, and other AI clients to test data](https://bfusqmdwknceg6ih.public.blob.vercel-storage.com/news-mcp-tofu-connection-HWISz7sq3htG6jQnjC0pU7cNdoZjKa.jpg)

TofuPilot now has an MCP server. Connect Claude, Cursor, or any Model Context Protocol client to your test data and it can read your runs, phases, measurements, logs, units, parts, batches, procedures, stations, deployments, and organization members. With write access, agents can also create, update, and delete runs, units, parts, batches, procedures, and stations, including revisions and versions.

No API key to copy or rotate. You authorize once in your browser.

## Connect

Your organization settings have a dedicated MCP page with copy-ready snippets for each client. Grab the one you need, authorize in the browser that opens, pick an organization, approve. A token is issued to the client. That's it.

**Claude**: `claude mcp add --transport http tofupilot https://www.tofupilot.app/mcp`, then run `/mcp` to authorize.

**Cursor**: add the URL to `.cursor/mcp.json`. It opens the browser on first use.

**Any other client**: use the endpoint with Streamable HTTP. The client discovers the OAuth flow from the URL.

## Read

Every connection can read: runs, phases, measurements, logs, units, parts, batches, procedures, stations, deployments, and organization members. Same filters the API exposes. Ask for the failing units on `FVT-001` this week, the out-of-limit measurements in a run, or the log lines around a failed phase, and the data comes back.

## Write

Write is opt-in. When you connect a client, you choose read-only or read and write. With write, the connection can create and update runs, units, parts, batches, procedures, and stations (including revisions and versions). Deletes are there too, behind an explicit confirmation, because a delete can cascade and there's no undo.

A read-only connection can never write.

## Access stays scoped

Every request runs against your organization only, enforced at the database. The connection acts with your role: operators get no access, viewers stay read-only, the rest read and write. Member emails are never exposed.

The same settings page lists every connected client, with its organization and access level, and lets you revoke any of them in one click. Revoking cuts it off immediately.

## What's next

Procedure-level scoping (limiting a connection to specific procedures) is on the list. If your agent needs data the server doesn't expose yet, tell us on Discord.

## Roadmap

### Build Artifacts
URL: https://www.tofupilot.com/roadmap/build-artifacts
Builds run once in isolation per commit, producing reproducible artifacts. Stations install fast, no local venv or dependency management.

### Phase & Measurement API
URL: https://www.tofupilot.com/roadmap/phase-measurement-api
Dedicated API v2 endpoints to create, read, and manage phases and measurements independently from runs. Enables advanced integrations and bulk data operations.

### GitHub Integration
URL: https://www.tofupilot.com/roadmap/github-integration
Connect GitHub repositories to TofuPilot for automated deployments, test script version control, and CI/CD pipeline integration.

### Workflow trigger vars
URL: https://www.tofupilot.com/roadmap/workflow-trigger-vars
Send emails on failed runs with a direct link to the report, and filter triggers on multiple outcomes at once with the "is one of" operator.

### C# SDK
URL: https://www.tofupilot.com/roadmap/csharp-sdk
Official C# SDK for TofuPilot API v2: upload runs, measurements and attachments from .NET test code or OpenTAP, with typed responses and NuGet publishing.

### Self-Service Self-Hosting
URL: https://www.tofupilot.com/roadmap/self-service-self-hosting
Deploy TofuPilot on your own infrastructure with a free Lab license. Sign in to Orbit, grab your license key, and run the deploy script. No sales call needed.

### Station v2
URL: https://www.tofupilot.com/roadmap/station-v2
Lightweight evolution of the Station app: faster deploys, broader platform support, and first-class integration with coding AI agents.

### Process Control (Insight)
URL: https://www.tofupilot.com/roadmap/process-control-insight
Rebuilt Insights page with interactive control charts, spec and control limits, histograms, capability indices (Cp, Cpk, Pp, Ppk), and drill-down filters.

### Python 3.13 & 3.14 builds
URL: https://www.tofupilot.com/roadmap/python-313-314-builds
Pick Python 3.12, 3.13, or 3.14 for your test stations from your pyproject.toml. Use the latest language features with no extra setup needed.

### Slack Integration
URL: https://www.tofupilot.com/roadmap/slack-integration
Post to a Slack channel from any workflow through an incoming webhook: test failures, new units, batch events, with run data in the message.

### MATLAB SDK
URL: https://www.tofupilot.com/roadmap/matlab-sdk
Upload MATLAB test results to TofuPilot with a native connector. Bridge simulation and hardware validation data in a single platform.

### SPC Multi-Select
URL: https://www.tofupilot.com/roadmap/spc-multi-select
Compare multiple measurements on one control chart. Shift+Click to select measurements with matching type, units, and limits. Shared stats and phase filtering.

### Auto-Push Flow
URL: https://www.tofupilot.com/roadmap/auto-push-flow
Push to your main branch and TofuPilot automatically deploys the new test script to all linked stations. No manual step.

### Measurements Table
URL: https://www.tofupilot.com/roadmap/measurements-table
New measurement table in the run page with improved filtering, sorting, and inline limit visualization for faster hardware test analysis.

### NI TestStand Plugin
URL: https://www.tofupilot.com/roadmap/ni-teststand-plugin
Native NI TestStand plugin to upload test runs, measurements, and attachments to TofuPilot directly from your TestStand sequences without custom code.

### Access Groups
URL: https://www.tofupilot.com/roadmap/access-groups
Segment data visibility across teams, facilities, or external suppliers. Control who sees what with team-based access scoping.

### Dashboard UI v2
URL: https://www.tofupilot.com/roadmap/dashboard-ui-v2
Redesigned dashboard with 10x more filters, a new table UI, global search, and improved navigation to find test data faster.

### Unit Attachments
URL: https://www.tofupilot.com/roadmap/unit-attachments
Attach files, images, and documents directly to units and test runs. Store evidence, photos, and reports alongside test data.

### Roles & Permissions
URL: https://www.tofupilot.com/roadmap/roles-permissions
Assign admin, operator, or viewer roles to control who can access, modify, or export hardware test data in your organization.

### InvenTree Integration
URL: https://www.tofupilot.com/roadmap/inventree-integration
Sync test data with InvenTree inventory. Upload test results to stock items, update statuses, transfer stock, complete builds, and print labels from workflows.

### Guides
URL: https://www.tofupilot.com/roadmap/guides
Step-by-step tutorials and best practices to help you build, automate, and scale hardware tests with TofuPilot and OpenHTF.

### Phase Pareto
URL: https://www.tofupilot.com/roadmap/phase-pareto
Rank phases by failures, retries, and wall-clock duration to find the weakest step, then drill into the measurements behind each failure.

### Discord Integration
URL: https://www.tofupilot.com/roadmap/discord-integration
Send test result notifications to Discord channels via webhooks. Trigger alerts on test failures, unit creation, or batch events to keep your team informed.

### SSO
URL: https://www.tofupilot.com/roadmap/sso
Single sign-on with your identity provider — Okta, Azure AD, OneLogin, and more — for secure, centralized team authentication.

### ARM64 Station Support
URL: https://www.tofupilot.com/roadmap/arm64-station-support
Run TofuPilot stations on Raspberry Pi 4/5 and other ARM64 single-board computers for cost-effective production line testing.

### Get Started Videos
URL: https://www.tofupilot.com/roadmap/get-started-videos
Video tutorials to get up and running with TofuPilot quickly — from first test upload to full production line deployment and beyond.

### Rust SDK
URL: https://www.tofupilot.com/roadmap/rust-sdk
Official Rust client for TofuPilot API v2. Generated from OpenAPI spec with full endpoint coverage, typed responses, integration tests, and crates.io release.

### C++ SDK
URL: https://www.tofupilot.com/roadmap/cpp-sdk
Official C++ client for TofuPilot API v2. Generated from OpenAPI spec with full endpoint coverage, typed responses, and integration tests for native workflows.

### Changelog
URL: https://www.tofupilot.com/roadmap/changelog
Stay up to date with every new feature, improvement, bug fix, and product release shipped by the TofuPilot engineering team.

### Instant Rollback
URL: https://www.tofupilot.com/roadmap/instant-rollback
Roll back every station to a previous test script build in one click. No rebuild, and auto-push pauses until you push again or resume it.

### Alerts
URL: https://www.tofupilot.com/roadmap/alerts
Define rules on metrics like FPY, failure rate, Cpk, or station heartbeat. Route triggered alerts to email, Slack, or webhook channels.

### Golden & Black Samples
URL: https://www.tofupilot.com/roadmap/golden-black-samples
Mark units as golden or black samples for calibration, validation, and SPC reference tracking across your test stations.

### MCP Server
URL: https://www.tofupilot.com/roadmap/mcp-server
Official MCP server for TofuPilot. Lets AI assistants like Claude query test data, manage runs, units, and stations through the Model Context Protocol.

### GitLab Integration
URL: https://www.tofupilot.com/roadmap/gitlab-integration
Connect GitLab repositories with real-time webhook sync, soft delete for branches and commits, and support for both Group and Personal Access Tokens.

### Linear Integration
URL: https://www.tofupilot.com/roadmap/linear-integration
Connect TofuPilot to Linear to automatically create issues and add comments from test results. Trigger workflows on failures to open bugs or track tasks.

### Station
URL: https://www.tofupilot.com/roadmap/station
Headless production test runner with offline mode, local caching, and automatic sync — built to run tests on the manufacturing line.

### Workflows
URL: https://www.tofupilot.com/roadmap/workflows
Automate actions when test events occur. Connect to Odoo, Linear, InvenTree, Discord, Slack, or any HTTP API. Filter, branch, and chain steps visually.

### Metrology
URL: https://www.tofupilot.com/roadmap/metrology
Track every instrument by serial with its calibration date and certificate, link each measurement to it, and block runs when a calibration has expired.

### SCIM Provisioning
URL: https://www.tofupilot.com/roadmap/scim-provisioning
Automatically sync users and groups from your identity provider to TofuPilot. Onboard and offboard team members instantly.

### Operator UI 2.0
URL: https://www.tofupilot.com/roadmap/technician-view
Redesigned operator screen in kiosk and dashboard: station and procedure cards, scanner-safe forms, Off, Side or Full run details, shared with the run page.

### OAuth2 SMTP Support
URL: https://www.tofupilot.com/roadmap/oauth2-smtp-support
Add OAuth2 (XOAUTH2) authentication for SMTP email in self-hosted deployments. Required for Microsoft 365 compatibility after Basic Auth deprecation.

### Dynamic measurements
URL: https://www.tofupilot.com/roadmap/dynamic-measurements
Declare measurements at record time inside a phase, not only up front in YAML, so tests can adapt their checks to live conditions.

### Templates
URL: https://www.tofupilot.com/roadmap/templates
Ready-to-use Python test templates for PCB, battery, motor, and sensor testing. Copy, customize, and deploy to your stations in minutes.

### ARMv7 & 32-bit stations
URL: https://www.tofupilot.com/roadmap/armv7-32-bit-stations
Run stations on ARMv7 boards like 32-bit Raspberry Pi OS and on 32-bit Windows (x86), with the same CLI, operator UI and automatic updates.

### Kubernetes deployment
URL: https://www.tofupilot.com/roadmap/kubernetes-deployment
Deploy TofuPilot on your Kubernetes cluster with a Helm chart, ready for high availability, scaling, and enterprise-grade operations.

### Offline upload queue
URL: https://www.tofupilot.com/roadmap/offline-upload-queue
Stations keep testing when the network drops: runs are queued locally, retried automatically, and visible in the operator UI until uploaded.

### Windows stations
URL: https://www.tofupilot.com/roadmap/windows-stations
Run TofuPilot stations on Windows PCs alongside Linux and macOS, with app shortcuts, automatic updates and the same operator UI on every OS.

### Monorepo procedures
URL: https://www.tofupilot.com/roadmap/monorepo-procedures
Keep many procedures in one repository: point each one at its package directory, share code through uv workspaces and deploy them separately.

### Operator role
URL: https://www.tofupilot.com/roadmap/operator-role
A dedicated operator role for line technicians: run tests and see their results without access to settings, and every run is attributed to them.

### Failure analysis
URL: https://www.tofupilot.com/roadmap/failure-analysis
Find why units fail: a pareto of failing phases, drill-down to the runs behind each bar, and side-by-side comparison of phases across runs.

### China network support
URL: https://www.tofupilot.com/roadmap/china-network-support
Stations in China install and update reliably: the CLI, Python runtime and packages are served from mirrors reachable behind the Great Firewall.

### Docs 2.0
URL: https://www.tofupilot.com/roadmap/docs-20
Revamped documentation with restructured pages, interactive guides, updated API reference, and improved search across all content.

### Unit flow
URL: https://www.tofupilot.com/roadmap/unit-flow
Define the path a unit follows across stations: each test checks that earlier steps passed, chains the next one, and gives one verdict per unit.

### SAP Integration
URL: https://www.tofupilot.com/roadmap/sap-integration
Sync hardware test data and quality metrics with SAP ERP and QM modules. Bridge manufacturing execution and quality management systems.

### Instrument library docs
URL: https://www.tofupilot.com/roadmap/instrument-library-docs
Standalone reference docs for PyVISA, pySerial and pymodbus, rebuilt with runnable examples for driving lab instruments from Python tests.

### One-line installer
URL: https://www.tofupilot.com/roadmap/one-line-installer
Install the TofuPilot CLI on any station with a single command from tofupilot.app/install, on Linux, macOS and Windows, with no extra setup.

### Persistent equipment
URL: https://www.tofupilot.com/roadmap/persistent-equipment
Keep an instrument connection warm across runs with a station-level plug scope, so equipment is set up once instead of every run.

### Run metadata
URL: https://www.tofupilot.com/roadmap/run-metadata
Attach custom key-value metadata to runs beyond built-in fields, for traceability, filtering, and export across procedures and stations.

### Phase parameters
URL: https://www.tofupilot.com/roadmap/phase-parameters
Pass parameters to a phase with with: in procedure.yaml, so one Python phase function is reused at several steps with different values.

### Stop on failure
URL: https://www.tofupilot.com/roadmap/abort-on-bad-reading
Stop a run as soon as a phase fails with on_first_failure, or end it from the operator UI with the Stop button, which runs teardown before exiting.

### Teams Integration
URL: https://www.tofupilot.com/roadmap/teams-integration
Get real-time alerts in Microsoft Teams on test failures, threshold breaches, and key production events. Stay informed without switching tools.

### Unit metadata
URL: https://www.tofupilot.com/roadmap/unit-metadata
Attach custom key-value metadata to units beyond built-in fields (serial, part, revision, batch), queryable and exportable across runs.

### Import test files
URL: https://www.tofupilot.com/roadmap/import-test-files
Import existing test reports into TofuPilot from OpenHTF, WATS, ATML, NI TestStand, STDF, CSV, and Excel files, with downloadable sample files for every format.

### Bitbucket Integration
URL: https://www.tofupilot.com/roadmap/bitbucket-integration
Connect Bitbucket repositories to TofuPilot for automated deployments, test script version control, and CI/CD pipeline integration.

### Multi-slot testing
URL: https://www.tofupilot.com/roadmap/multi-slot-testing
Test several units in parallel on one fixture: per-slot runs grouped by execution, live slot status in the operator UI, and per-slot results in the dashboard.

### pytest Connector
URL: https://www.tofupilot.com/roadmap/pytest-connector
Upload pytest test results to TofuPilot with a native connector. Analyze hardware validation tests alongside production data in one view.

### Gage R&R
URL: https://www.tofupilot.com/roadmap/gage-rr
Measurement system analysis with Gage R&R studies to quantify repeatability and reproducibility of your test stations and instruments.

### OpenHTF in Station v2.0
URL: https://www.tofupilot.com/roadmap/openhtf-in-station-v20
Run, debug, and monitor OpenHTF tests directly from TofuPilot Studio with live logs, breakpoints, and result inspection.

### Sequences
URL: https://www.tofupilot.com/roadmap/sequences
Write steps and benches once as sequence files, called with uses: and with: inside a procedure, or used around it for bench setup and teardown.

### Custom Validators
URL: https://www.tofupilot.com/roadmap/custom-validators
Write pass/fail logic beyond fixed operators, including limits computed from an earlier measurement, such as twice the reference voltage, in the YAML.

### Station touch mode
URL: https://www.tofupilot.com/roadmap/station-touch-mode
Touch-optimized Station UI for production screens with touch input, enabling operators to run procedures without mouse or keyboard.

### USB deployment transfer
URL: https://www.tofupilot.com/roadmap/usb-deployment-transfer
Deploy procedures and collect run data on fully air-gapped stations by transferring them on a USB key, with no network link to the dashboard.

### Dev stations
URL: https://www.tofupilot.com/roadmap/dev-stations
Flag stations as dev or production: run preview builds on a dev bench safely, and keep development runs out of production KPIs and alerts.

### Plug actions
URL: https://www.tofupilot.com/roadmap/plug-actions
Trigger plug methods from UI component actions so operators can drive plugs from a switch or button, with results captured in phase_results.

### OpenHTF Docs v2.0
URL: https://www.tofupilot.com/roadmap/openhtf-docs-v20
Revamped OpenHTF documentation with clearer step-by-step guides, real-world examples, and best practices for hardware testing.

### Reports
URL: https://www.tofupilot.com/roadmap/reports
Build custom reports combining multiple insights, charts, and metrics to visualize hardware test data across your organization.

### Repair
URL: https://www.tofupilot.com/roadmap/repair
Log failures and repair actions on failing units, with AI-suggested failure and repair codes plus rework yield and repair-effectiveness analytics.

### Odoo Integration
URL: https://www.tofupilot.com/roadmap/odoo-integration
Sync hardware test data and quality metrics with your Odoo ERP instance. Bridge manufacturing execution and quality management systems.

### Live run streaming
URL: https://www.tofupilot.com/roadmap/live-run-streaming
Watch test runs live from the dashboard as they execute on the station: phases, measurements and operator prompts stream in real time.

### Waveform Persistence
URL: https://www.tofupilot.com/roadmap/waveform-persistence
Overlay waveforms from all runs as an intensity map, eye-diagram style, with the current run drawn on top to spot degradation trends at a glance.

### Sticky Operator Inputs
URL: https://www.tofupilot.com/roadmap/sticky-operator-inputs
Mark operator input fields as sticky so their values persist across runs when using Run Again, reducing repetitive entry for recurring tests.

### Robot Framework
URL: https://www.tofupilot.com/roadmap/robot-framework
Upload Robot Framework test results to TofuPilot with a native connector. Track hardware test trends and quality metrics over time.

### Parametrized procedures
URL: https://www.tofupilot.com/roadmap/parametrized-procedures
Define per-channel measurements, swept phases or many slots in one block with matrix:, instead of copy-pasting YAML blocks for each value.

### Jira Integration
URL: https://www.tofupilot.com/roadmap/jira-integration
Create Jira issues automatically from test failures and sync status updates between TofuPilot and your project management workflow.

### Environment Variables
URL: https://www.tofupilot.com/roadmap/environment-variables
Manage environment variables per station for deployments, secrets, and configuration. Keep credentials and machine-specific configuration out of your test code.

### News
URL: https://www.tofupilot.com/roadmap/news
Blog posts, product updates, engineering deep-dives, and company announcements — all in one place to keep your team informed.

### Station health
URL: https://www.tofupilot.com/roadmap/station-health
See every station's hardware, CLI version, connectivity and activity timeline in the dashboard, so you spot an offline or outdated bench fast.

### Builder mode
URL: https://www.tofupilot.com/roadmap/builder-mode
Run a single phase or a range of phases on its own while building a procedure, so you iterate on one step without replaying the whole test.

### Measurement Control v2
URL: https://www.tofupilot.com/roadmap/measurement-control-v2
Control charts for all six measurement types, with a Failures tab, filters by environment, deployment, version and duration, and full-dataset histograms.

### Curve overlay
URL: https://www.tofupilot.com/roadmap/curve-overlay
Overlay multi-dimensional measurements such as sweeps and IV curves across runs in Measurement Control to spot drift in the whole curve shape.

### Python debug mode
URL: https://www.tofupilot.com/roadmap/python-debug-mode
Attach VS Code or any debugpy client to Python phases on a station, set breakpoints and step through a live test run line by line.

### Data export
URL: https://www.tofupilot.com/roadmap/data-export
Export selected phases and measurements to CSV or JSON straight from the dashboard selection bar, ready for Excel, Python or your own reports.

### Private CA support
URL: https://www.tofupilot.com/roadmap/private-ca-support
Run stations behind a corporate TLS proxy or private certificate authority: uploads, the realtime link and the OpenHTF client all trust your CA.

### Custom metadata
URL: https://www.tofupilot.com/roadmap/custom-metadata
Attach your own key-value fields to stations, parts and revisions, alongside run and unit metadata, to filter and trace with your own vocabulary.

### Operator tracking
URL: https://www.tofupilot.com/roadmap/operator-tracking
Record who ran each test, from a logged-in account or a free-text operator name typed at the station, and filter runs by operator.

### Atlassian sign-in
URL: https://www.tofupilot.com/roadmap/atlassian-sign-in
Sign in to TofuPilot with your Atlassian account, so teams already working in Jira, Confluence and Bitbucket log in with a single click.

### Duplicate-safe uploads
URL: https://www.tofupilot.com/roadmap/duplicate-safe-uploads
Run creation is idempotent: when a station loses the server response on a flaky network and retries, the run is stored once, never duplicated.

### Chart image export
URL: https://www.tofupilot.com/roadmap/chart-image-export
Download any analytics chart as an image for reports and reviews, plus keyboard shortcuts and a date filter you can type into directly.

### Launch on boot
URL: https://www.tofupilot.com/roadmap/launch-on-boot
Install the station as a system service that starts on boot and restarts on failure, so production benches come back without anyone logging in.

### Studio v2
URL: https://www.tofupilot.com/roadmap/studio-v2
Write, run and debug procedures from a web workbench paired with your local station, with an AI assistant, diff review and live run output.

### TofuPilot CLI
URL: https://www.tofupilot.com/roadmap/tofupilot-cli
One command-line tool to log in, pull procedures, run tests and upload results, with device login for stations and the full API from your terminal.

### Preconditions
URL: https://www.tofupilot.com/roadmap/preconditions
Run a phase, sequence or measurement only if a condition holds with if: (revision, run mode, earlier result) or the run status: always(), failure().

### Live presence
URL: https://www.tofupilot.com/roadmap/live-presence
See who is looking at a station's operator UI and watch fields being typed live, so engineers can follow and help an operator remotely.

### Signed macOS builds
URL: https://www.tofupilot.com/roadmap/signed-macos-builds
The TofuPilot CLI for macOS is signed and notarized with an Apple Developer ID, so it installs and runs on station Macs without security warnings.

### Git submodules
URL: https://www.tofupilot.com/roadmap/git-submodules
Deploy repositories that use git submodules: shared plugs and phases in their own repository, fetched at build time through the same Git connection.

### Plug parameters
URL: https://www.tofupilot.com/roadmap/plug-parameters
Configure plugs with with: in procedure.yaml and per-station environment variables, so one plug class serves every bench without editing YAML.

### Post-conditions
URL: https://www.tofupilot.com/roadmap/post-conditions
Decide what happens after each phase result with then: continue, stop, retry, or ask the operator, per outcome and as a default for the whole run.

### Measurement signals
URL: https://www.tofupilot.com/roadmap/measurement-signals
Name product signals in the procedure and wire them to instruments in the bench sequence, so a sequence never changes when the bench or an instrument does.

### Variants
URL: https://www.tofupilot.com/roadmap/variants
Declare per-revision and per-part limits and skipped phases in procedure.yaml, so one procedure covers every board revision without Python logic.

### Diagnostics
URL: https://www.tofupilot.com/roadmap/diagnostics
Guide operators when a step fails: per-phase hints and images, decision trees and guided checks, so they find the cause before retesting or escalating.

### Input validation
URL: https://www.tofupilot.com/roadmap/input-validation
Validate serial numbers and operator inputs at the station before the run starts, with patterns, prefixes, suffixes and clear rejection messages.

### Plug restart
URL: https://www.tofupilot.com/roadmap/plug-restart
Restart a plug whose process crashed or lost its instrument, then let the phase retry, so a dropped connection does not fail the unit under test.

### Identify
URL: https://www.tofupilot.com/roadmap/identify
Scan a label once: every field is parsed, the right procedure is picked from the part number, options load at runtime and the unit history shows up.

### Azure DevOps integration
URL: https://www.tofupilot.com/roadmap/azure-devops-integration
Connect Azure DevOps repositories, Services or Server, so procedures deploy from your Azure Repos on every push, like GitHub, GitLab and Bitbucket.

### Python SDK
URL: https://www.tofupilot.com/roadmap/python-sdk
Official Python SDK for TofuPilot API v2: create runs, upload measurements and attachments, and query units from any Python test script, published on PyPI.

### Procedure API access
URL: https://www.tofupilot.com/roadmap/procedure-api-access
Call the TofuPilot API from test procedures as the identity the run uploads with, with no API key to create or store on the station.

### Kiosk browser lockdown
URL: https://www.tofupilot.com/roadmap/kiosk-browser-lockdown
Station kiosk opens without translate prompts, AI sidebars, image search or autofill bubbles on Edge, Chrome and Firefox, with policy files for a full lockdown.

### Interactive phase loops
URL: https://www.tofupilot.com/roadmap/interactive-phase-loops
Pause a phase on its operator screen with ui.wait(), apply the answer, show the new reading and ask again, for calibration and adjustment loops.

### Operator UI v1 sunset
URL: https://www.tofupilot.com/roadmap/operator-ui-v1-sunset
Retire the original operator UI once the technician view ships. Stations move to the new interface automatically, with no change to procedures.

### Studio v1 sunset
URL: https://www.tofupilot.com/roadmap/studio-v1-sunset
Retire the original Studio desktop app. Procedures, plugs and stations keep working, and editing moves to Studio v2 in the dashboard and to the CLI.

### Robust plug lifecycle
URL: https://www.tofupilot.com/roadmap/robust-plug-lifecycle
Predictable plug setup and teardown: teardown on stop and on crash, a fixed teardown order, timeouts on hung calls, and every lifecycle step logged.

### Headless runs
URL: https://www.tofupilot.com/roadmap/headless-runs
Start and follow runs from your own program or CI: pass the serial, procedure and options, then read live progress and results as a JSON stream.

### Panels
URL: https://www.tofupilot.com/roadmap/panels
Customize the operator screen per line: your branding and colors, layout with sections and widths, and custom widgets such as a bench synoptic view.

### Custom list columns
URL: https://www.tofupilot.com/roadmap/custom-list-columns
Choose which columns the Runs and Units lists show, including run and unit metadata keys, with a sticky header that stays visible while scrolling.

### Vision SDK
URL: https://www.tofupilot.com/roadmap/vision-sdk
Python SDK for visual inspection in Framework procedures: capture camera images, detect defects with AI and record each check as a measurement.

## Questions & Answers

### Is TofuPilot station licensing tied to hardware?
URL: https://www.tofupilot.com/questions/is-tofupilot-station-licensing-tied-to-hardware
A station counts against your plan when it is registered, not by hardware fingerprint. You can remove a station and register another in its place, so replacing a bench PC or reflashing a machine does not permanently consume a station slot.

### When am I billed for TofuPilot?
URL: https://www.tofupilot.com/questions/when-am-i-billed-for-tofupilot
Pro plans bill monthly through Stripe based on selected options and the previous month's usage. Usage is visible under Settings > Usage with notifications at 75% and 100% of included quotas. Enterprise plans offer monthly or annual invoicing.

### How much does TofuPilot cost?
URL: https://www.tofupilot.com/questions/how-much-does-tofupilot-cost
Three plans. Lab is free: 1 seat, 1 station, 100 runs per month, 1 GB storage per month. Pro is $50 per user per month with 1,000 runs per month included (then $0.01 per run), 100 GB storage per month (then $0.10 per GB), and 1 station included (then $50 per station). Enterprise adds SSO/SCIM at scale, self-hosting, air-gapped operation, custom limits, and SLA-backed support.

### Is TofuPilot subject to the US Patriot Act or CLOUD Act?
URL: https://www.tofupilot.com/questions/is-tofupilot-subject-to-the-us-patriot-act-or-cloud-act
No. TofuPilot SA is a Swiss-incorporated company with no US legal entity. US surveillance laws, including the CLOUD Act, FISA Section 702, and National Security Letters, cannot compel TofuPilot to disclose customer data. For customers requiring full infrastructure sovereignty, TofuPilot offers self-hosted deployment on your own servers.

### How long does integration take?
URL: https://www.tofupilot.com/questions/how-long-does-integration-take
Most teams are up and running within 30 minutes. Integration requires a single line of Python — no migration or infrastructure changes needed.

### Which test frameworks are supported?
URL: https://www.tofupilot.com/questions/which-test-frameworks-are-supported
TofuPilot integrates natively with OpenHTF via an open-source plugin, and supports pytest, Robot Framework, and custom scripts through the Python client.

### How is test data protected?
URL: https://www.tofupilot.com/questions/how-is-test-data-protected
All data is encrypted at rest and in transit. TofuPilot SA is Swiss-incorporated, so your data is protected by Swiss data protection law and is outside the reach of US surveillance legislation such as the CLOUD Act. Organizations requiring full control can self-host TofuPilot on-premise or in air-gapped environments.

### Does TofuPilot scale for high-volume manufacturing?
URL: https://www.tofupilot.com/questions/does-tofupilot-scale-for-high-volume-manufacturing
TofuPilot processes thousands of test runs daily and supports real-time queries across millions of data points, backed by a 99.9% uptime SLA.

### Can TofuPilot be self-hosted?
URL: https://www.tofupilot.com/questions/landing-can-tofupilot-be-self-hosted
Yes. Self-hosted deployment is available on the Enterprise plan, providing complete data sovereignty for regulated and air-gapped environments.

### Do you support air-gapped or offline environments?
URL: https://www.tofupilot.com/questions/landing-what-about-air-gapped-or-offline-environments
The self-hosted deployment operates fully offline with no external dependencies. Data synchronizes automatically when connectivity is restored.

### Can instrument connections stay open across an entire unit or station session?
URL: https://www.tofupilot.com/questions/can-instrument-connections-stay-open-across-an-entire-unit-or-station-session
Yes. Plugs have a lifecycle scope: phase, slot, run, or station. With scope: station, the plug class is constructed once in a long-lived Python process, its connections stay open across units until the station stops, and only the first unit after startup pays the connection cost. Every test method reuses the same live object, so instrument sessions are never reopened between phases or between units.

### What is TofuPilot's relationship with OpenHTF?
URL: https://www.tofupilot.com/questions/what-is-tofupilots-relationship-with-openhtf
We've used TofuPilot for years on drone manufacturing floors and maintain openhtf.org, the open-source OpenHTF documentation site. We're also close to the OpenHTF founders. TofuPilot provides native OpenHTF integration that requires just one line of code to start streaming test results with full metadata preservation.

### Can I sync TofuPilot test failures to Linear?
URL: https://www.tofupilot.com/questions/can-i-sync-tofupilot-test-failures-to-linear
Yes. The Linear integration lets failing runs and phases open or update tickets from the dashboard. It offers two actions: Create issue (with optional assignee, labels, priority, and estimate in story points) and Add comment. Connect via OAuth under Settings > Integrations > Linear; no API keys are required, and you can connect multiple teams by repeating the flow. Text fields support {{variable}} interpolation, and the created issue's ID and URL are exposed to downstream workflow nodes.

### How do I migrate from NI TestStand to TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-i-migrate-from-ni-teststand-to-tofupilot
Export an NI TestStand XML report (.xml) and create a procedure in the dashboard. Drag the .xml onto the import page and click Import; TofuPilot auto-detects the file by its <Reports> root element. Each <Report> becomes a run with its UUTResult as the outcome, ResultList steps become phases, and numeric or pass-fail limit steps become measurements with limits. Via API, upload then import with importer TESTSTAND (up to 100 files per request).

### How do I install the TofuPilot Python SDK?
URL: https://www.tofupilot.com/questions/how-do-i-install-the-tofupilot-python-sdk
Install from PyPI: pip install tofupilot. Import it as "from tofupilot.v2 import TofuPilot". Pass your API key to the constructor or set the TOFUPILOT_API_KEY environment variable. The type-safe SDK wraps the REST API and supports creating runs with measurements, phases, and validators, cursor-paginated run listing, attachment upload and download, and async methods.

### Can TofuPilot sync test results to InvenTree?
URL: https://www.tofupilot.com/questions/can-tofupilot-sync-test-results-to-inventree
Yes. The InvenTree integration writes TofuPilot test data into your InvenTree instance so build outputs, stock items, and test reports stay in sync. Five actions are available: upload a test result, update stock status, transfer stock between locations, complete a build output, and print a label. Connect with an API token that inherits the generating user's permissions, plus the base URL, configured once per instance.

### What is TofuPilot Agent?
URL: https://www.tofupilot.com/questions/what-is-tofupilot-agent
TofuPilot Agent runs autonomous root cause analysis on failing units inside the chat interface. It frames the failure in plain language, hypothesizes candidate causes across phase, measurement, station, revision, and operator, gathers evidence by querying run and measurement data, scores hypotheses with Wilson 95% confidence intervals against baselines, then ranks and reports the highest-confidence causes with inline evidence. Use it to confirm whether a cause is the part, the station, or the test itself; for simple lookups use Chat. Results can be saved as shareable links or attached to tickets.

### Can TofuPilot alert me when a measurement drifts or a unit fails?
URL: https://www.tofupilot.com/questions/can-tofupilot-alert-me-when-a-measurement-drifts-or-a-unit-fails
Yes. Workflows trigger on run events and send notifications through Slack, Discord, email, or HTTP calls, and AI alerts watch for anomalies such as measurement drift or yield drops.

### Can I query my TofuPilot test data in natural language?
URL: https://www.tofupilot.com/questions/can-i-query-my-tofupilot-test-data-in-natural-language
Yes, with TofuPilot Chat. It answers questions across every run, unit, part, batch, and station in your organization. Chat calls typed tools backed by the REST API, so every answer is grounded in real data with links to the dashboard. It handles aggregations ("How many runs failed yesterday?"), drill-downs into failing phases or measurements, comparisons across revisions, stations, or batches, and statistics like the Cpk of a measurement over the last 1000 runs. You only see data your role and team scope allow.

### Can I attach files to a test run in TofuPilot?
URL: https://www.tofupilot.com/questions/can-i-attach-files-to-a-test-run-in-tofupilot
Yes. An attachment is a binary file persisted alongside a run, such as a photo, a scope capture, a diagnostic dump, or a PDF report. Each attachment belongs to one run and can optionally be linked to the phase that produced it. Upload attachments through the REST API or the Python SDK.

### How do I run a test procedure with the TofuPilot CLI?
URL: https://www.tofupilot.com/questions/how-do-i-run-a-test-procedure-with-the-tofupilot-cli
Run tofupilot run [PATH], where PATH is an optional procedure .yaml file, directory, or Python entry point. Without a path it runs locally and, without --upload, never contacts the dashboard. Use --deployment <ID> to run a pulled deployment, --upload to sync a local run to the dashboard, --tui/--no-tui for the terminal operator UI, and --kiosk/--no-kiosk for the browser kiosk UI.

### What is a sub-unit in TofuPilot?
URL: https://www.tofupilot.com/questions/what-is-a-sub-unit-in-tofupilot
A sub-unit is a child unit assembled into a parent unit, such as a PCB inside a finished product. A parent unit can reference multiple sub-units by their child serial numbers, enabling full genealogy tracking of nested components like PCBs, batteries, and enclosures. This lets you trace which components went into which assembly, aligning with MES genealogy records and IPC-1782 traceability.

### What's the difference between a part, revision, and unit?
URL: https://www.tofupilot.com/questions/whats-the-difference-between-a-part-revision-and-unit
A part is the product or component family that units belong to (e.g. PCB-V1). A revision is a version of a part; it defaults to A. A unit is the individual physical item under test, identified by a serial number and belonging to one part-and-revision combination. Batches are an optional layer grouping units made in the same lot. The model maps to PLM and ERP item masters.

### How do I install TofuPilot on Linux?
URL: https://www.tofupilot.com/questions/how-do-i-install-tofupilot-on-linux
Run the install script in a terminal: curl -fsSL https://tofupilot.app/install | sh. This installs the tofupilot CLI. To register the machine as a station, copy the command from the station page in the dashboard, which carries a one-hour setup token: curl -fsSL https://tofupilot.app/install | sh -s -- --token YOUR_SETUP_TOKEN. The station daemon runs under systemd; manage it with tofupilot service start|stop|status.

### How do I uninstall TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-i-uninstall-tofupilot
Run tofupilot uninstall. It removes the CLI binary and, by default, every local data file including queued uploads, pulled deployments, and cached config. Use --keep-data to preserve run data, queued uploads, and pulled deployments before reinstalling, and --yes to skip the confirmation prompt in CI.

### What can I do with the TofuPilot CLI?
URL: https://www.tofupilot.com/questions/what-can-i-do-with-the-tofupilot-cli
The CLI has 26+ subcommands. Common ones: tofupilot login/logout/whoami (auth), tofupilot run (execute a procedure locally with the operator UI), tofupilot deploy (build and deploy procedures to the cloud), tofupilot pull (pull deployments), tofupilot runs/batches/units/measurements (manage test data), tofupilot stations (manage factory stations), tofupilot service (run as a system daemon), and tofupilot update/uninstall. Every command accepts the global --json flag for machine-readable output.

### Can TofuPilot connect to my ERP (Odoo)?
URL: https://www.tofupilot.com/questions/can-tofupilot-connect-to-my-erp-odoo
Yes. The Odoo integration pushes test data into your Odoo instance so production records, quality checks, and chatter stay in sync. Three actions are available: Create record on any Odoo model (e.g. product.product, quality.check, mrp.production), Update record by ID, and Post message to a record's chatter thread. Setup requires an Odoo user with model access, an API key, plus the Odoo URL, database name, and login.

### What is a batch in TofuPilot?
URL: https://www.tofupilot.com/questions/what-is-a-batch-in-tofupilot
A batch groups units you produced together, such as a manufacturing lot, a build day, or a contract manufacturer shipment. Its identity is its number; all other information is derived from its linked units. Units link to a batch at creation via the batch_number parameter or afterward by manual assignment. Batch numbers are 1-60 characters matching ^[a-zA-Z0-9_.:+-]+$, unique per organization (case-insensitive). Deleting a batch keeps unit data but removes the assignment, and empty batches are cleaned up automatically.

### How do I deploy a procedure with the TofuPilot CLI?
URL: https://www.tofupilot.com/questions/how-do-i-deploy-a-procedure-with-the-tofupilot-cli
Run tofupilot deploy from a directory linked to a procedure that contains a pyproject.toml. It uploads the local source tree, builds it in the cloud, and streams build logs. By default it creates a preview deployment; use tofupilot deploy --prod to target production, after which every linked station picks it up between runs. Deploy requires a user login; station setup tokens cannot deploy code.

### How do I import OpenHTF results into TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-i-import-openhtf-results-into-tofupilot
Save an OpenHTF test record as JSON via the OutputToJSON callback and create a procedure in the dashboard. Drag the .json file to the dropzone (auto-detected as OpenHTF) and click Import; no column mapping is needed. The record becomes a run, dut_id becomes the unit serial number, phases become phases, measurements with validators become measurements with limits, and outcome sets the run outcome. Via API, upload then import by upload_id with the OPENHTF importer at POST /v2/import.

### Can TofuPilot call any external API over HTTP?
URL: https://www.tofupilot.com/questions/can-tofupilot-call-any-external-api-over-http
Yes. The HTTP integration lets a workflow send a request to any URL, so TofuPilot events can trigger systems without a native integration. It supports GET, POST, PUT, PATCH, and DELETE, with {{variable}} interpolation in URLs, headers, and bodies. Downstream nodes can read the response status, body, and headers. Limits: timeout 1-60 seconds (default 10), request and response bodies max 1 MB each, and up to three retries with exponential backoff.

### Does TofuPilot have an MCP server for AI agents?
URL: https://www.tofupilot.com/questions/does-tofupilot-have-an-mcp-server-for-ai-agents
Yes. The TofuPilot MCP server lets AI agents (Claude, Cursor, and any Model Context Protocol client) work with your test data. It exposes read tools by default (list and retrieve runs, units, parts, batches, procedures, stations, users, logs, phases, measurements, deployments) and optional write tools. Connect Claude with: claude mcp add --transport http tofupilot https://www.tofupilot.app/mcp, then run /mcp to authorize in your browser via OAuth. Each client binds to one organization with scoped read or write access.

### What happens if a long test run crashes near the end?
URL: https://www.tofupilot.com/questions/what-happens-if-a-long-test-run-crashes-near-the-end
Phases are isolated on worker processes: if one dies, the engine replaces it and execution continues instead of taking the whole run down, and retries are configurable per phase. Plug processes are health-checked between units and restarted automatically if they died, so a crash never carries into the next unit. If the station loses the network, results are stored locally and uploaded when connectivity returns. This matters most on runs measured in hours, where losing the run means losing a full shift of bench time.

### Can an MES block a unit before the test runs?
URL: https://www.tofupilot.com/questions/can-an-mes-block-a-unit-before-the-test-runs
Yes, in both directions. The MES check runs as the first step of the execution, before any test phase: if the MES refuses the unit, the run stops there and the dashboard shows a dedicated status, so a unit blocked by the MES is never counted as a unit that failed testing. At the end of the run the results are sent back to the MES the same way. The check is a normal API call from your own code, so it works with whichever system you run, and dashboard-side MES and ERP integrations are available for the synchronization that does not need to gate execution.

### Does the Rust engine run each Python test in a separate process?
URL: https://www.tofupilot.com/questions/does-the-rust-engine-run-each-python-test-in-a-separate-process
Two mechanisms. Plugs run as persistent processes, one per plug instance, addressed over a local TCP socket, so instrument objects and their state stay alive across phases. Test phases run on a pool of worker processes, which is what isolates a crashing phase from the rest of the run. Both are standard Python subprocesses, not an embedded interpreter, and the environment and dependencies are managed automatically with uv.

### How many units can a station test in parallel?
URL: https://www.tofupilot.com/questions/how-many-units-can-a-station-test-in-parallel
Parallel slots are native to the execution engine, which is why it is written in Rust. Configurations of a few slots up to several dozen are normal operation, including burn-in and aging testers where units simply stay powered for hours. Each slot keeps its own serial number, part number and revision. Slots do not require a live instrument connection, so a large oven load can be identified and recorded without a per-unit script. Very large configurations, in the hundreds of slots, have no architectural limit but are worth validating on your actual bench.

### How do I integrate OpenHTF with TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-i-integrate-openhtf-with-tofupilot
Deployed through the TofuPilot CLI, OpenHTF scripts stream live test data and the operator UI with zero configuration. Standalone, it takes one wrapper: import TofuPilot from tofupilot.openhtf, then wrap your test with "with TofuPilot(test): test.execute()". All phases, measurements, and attachments upload automatically.

### How do I record the operator of a test in TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-i-record-the-operator-of-a-test-in-tofupilot
Pass the operated_by field (operator email) when creating the run through the API or SDK. The run then shows the operator identity in the dashboard, separate from the station that uploaded it.

### Do test stations need permanent internet access?
URL: https://www.tofupilot.com/questions/do-test-stations-need-permanent-internet-access
No. Every completed run is persisted to a local queue before upload; a background loop drains it every few seconds and retries with backoff from 15 seconds up to a 1 hour cap, so network outages lose no data. Inspect or replay the queue with commands like "tofupilot list-queue", "tofupilot queue-retry", and "tofupilot queue-export". Fully offline factories run the self-hosted deployment instead.

### Does TofuPilot run on Windows?
URL: https://www.tofupilot.com/questions/does-tofupilot-run-on-windows
Yes. The CLI, Station, and Studio run on Windows through a PowerShell installer. Linux and macOS are also fully supported.

### How do I migrate an existing test suite to TofuPilot without downtime?
URL: https://www.tofupilot.com/questions/how-do-i-migrate-an-existing-test-suite-to-tofupilot-without-downtime
TofuPilot supports incremental migration: migrate one product line at a time (vertical), one test stage at a time (horizontal), or a mix (hybrid). Your legacy system keeps running while migrated stations stream to TofuPilot, so production never stops.

### How do I install TofuPilot on macOS?
URL: https://www.tofupilot.com/questions/how-do-i-install-tofupilot-on-macos
Run curl -fsSL https://tofupilot.app/install | sh in Terminal to install the tofupilot CLI (same command as Linux). To register the machine as a station, copy the command from the station page in the dashboard, which carries a one-hour setup token: curl -fsSL https://tofupilot.app/install | sh -s -- --token YOUR_SETUP_TOKEN. The daemon runs under launchd, managed via tofupilot service start|stop|status.

### Does TofuPilot work with OpenHTF, pytest, and Robot Framework?
URL: https://www.tofupilot.com/questions/does-tofupilot-work-with-openhtf-pytest-and-robot-framework
Yes. TofuPilot supports OpenHTF, pytest, Robot Framework, and its own open-source TofuPilot framework. Any custom framework can integrate through the REST API or one of the SDKs (Python, C#, Rust, C++, MATLAB).

### How do I install the TofuPilot CLI?
URL: https://www.tofupilot.com/questions/how-do-i-install-the-tofupilot-cli
On Linux or macOS, run: curl -fsSL https://tofupilot.app/install | sh. On Windows, run in PowerShell: $p = "$env:TEMP\tp-install.ps1"; irm https://tofupilot.app/install.ps1 -OutFile $p; powershell -ExecutionPolicy Bypass -File $p; ri $p -EA 0. Nothing else is needed beforehand: the CLI ships its own pinned uv and Python interpreter. After installing, run "tofupilot login", then "tofupilot run" in your test project to execute and stream your first test.

### Can I upload OpenHTF results that were recorded offline?
URL: https://www.tofupilot.com/questions/can-i-upload-openhtf-results-that-were-recorded-offline
Yes. Write the OpenHTF JSON report to disk, then upload it later with client.create_run_from_openhtf_report("./report.json"). Original timestamps are preserved. Stations running the CLI also queue completed runs locally and upload automatically when connectivity returns.

### How do I add a TofuPilot desktop shortcut to a station?
URL: https://www.tofupilot.com/questions/how-do-i-add-a-tofupilot-desktop-shortcut-to-a-station
In the dashboard, go to Stations, select the station, open Setup, and toggle "Desktop shortcut" on. The CLI writes a launcher to the station's desktop equivalent to running tofupilot run in the station's working directory, launching whichever operator UI the station uses. The file is TofuPilot.desktop on Linux, TofuPilot.command on macOS, and TofuPilot.lnk on Windows. Toggling the setting off deletes the shortcut.

### Can I use pytest for hardware testing with TofuPilot?
URL: https://www.tofupilot.com/questions/can-i-use-pytest-for-hardware-testing-with-tofupilot
Yes. The CLI runs pytest suites natively under an embedded plugin: each test_* function becomes a phase, and assert statements like "assert lo <= x <= hi" become numeric measurements with limits. Run identity is read from [tool.tofupilot] in pyproject.toml.

### Can I create TofuPilot test phases dynamically at runtime?
URL: https://www.tofupilot.com/questions/can-i-create-tofupilot-test-phases-dynamically-at-runtime
In the TofuPilot framework, phases are declared in a YAML procedure and each phase runs a Python function. If you need programmatic phase generation at runtime, use OpenHTF, where phases are plain Python callables you can build dynamically.

### Can I share data between test phases in TofuPilot?
URL: https://www.tofupilot.com/questions/can-i-share-data-between-test-phases-in-tofupilot
Yes. A phase can read an earlier phase's results by declaring a function parameter named after that phase's key; TofuPilot injects a read-only result object exposing its measurements, outcome, and duration. Use depends_on to guarantee the earlier phase runs first.

### Are TofuPilot measurements scoped to a test phase?
URL: https://www.tofupilot.com/questions/are-tofupilot-measurements-scoped-to-a-test-phase
Measurements belong to the phase that records them. Each measurement is a typed value (number, string, boolean, JSON, or multi-dimensional series) validated against limits; a failing validator fails the measurement, its phase, and the run.

### Can I control lab instruments over SCPI, VISA, or LabVIEW?
URL: https://www.tofupilot.com/questions/can-i-control-lab-instruments-over-scpi-visa-or-labview
Python is the primary scripting language, so instrument control works through standard libraries like PyVISA, pyserial, or pymodbus inside plugs, which hold persistent hardware connections. Phases can also call external binaries in any language through an executable phase. LabVIEW and TestStand environments integrate through the SDKs.

### Which operating systems does TofuPilot support?
URL: https://www.tofupilot.com/questions/which-operating-systems-does-tofupilot-support
The TofuPilot CLI supports Linux, macOS, and Windows. On macOS and Linux, install with curl -fsSL https://tofupilot.app/install | sh; on Windows, run the PowerShell installer from https://tofupilot.app/install.ps1. The station daemon runs under systemd on Linux and launchd on macOS.

### How do I create a test run with the TofuPilot API?
URL: https://www.tofupilot.com/questions/how-do-i-create-a-test-run-with-the-tofupilot-api
Use the Python SDK (client.runs.create with procedure_id, serial_number, part_number, outcome, and phases) or the REST endpoint POST https://www.tofupilot.app/api/v2/runs. Authenticate with an API key passed as a bearer token or the TOFUPILOT_API_KEY environment variable.

### How do I update the TofuPilot CLI?
URL: https://www.tofupilot.com/questions/how-do-i-update-the-tofupilot-cli
Run tofupilot update. It checks for a newer CLI release and installs it in place. The CLI also checks in the background and prompts you when an update is available.

### How do I create and manage API keys?
URL: https://www.tofupilot.com/questions/how-do-i-create-and-manage-api-keys
Open Settings > API keys, click Create key, name it, and pick an expiry. The key is shown once. There are two types: user keys, which inherit your role and team scope, and station keys, which are issued at station registration and scoped to that station. Revocation is immediate.

### Does TofuPilot integrate with Slack?
URL: https://www.tofupilot.com/questions/does-tofupilot-integrate-with-slack
Yes. TofuPilot posts a message to a Slack channel whenever a workflow fires, using a Slack incoming webhook (no OAuth app or token management). Create an incoming webhook in Slack, copy the URL, then add a "Send Slack message" action to a workflow. The message is plain text and supports {{variable}} interpolation; you can override the bot name and icon, and chain multiple Slack actions to reach several channels.

### Can two units share a serial number if they have different part numbers?
URL: https://www.tofupilot.com/questions/can-two-units-share-a-serial-number-if-they-have-different-part-numbers
No. Serial numbers are unique per organization (case-insensitive), so a unit's full history stays unambiguous. If your lines reuse short serials across products, send the part number explicitly alongside the serial, or embed it in the serial (for example PCB01-000123).

### Can I edit TofuPilot run metadata after upload?
URL: https://www.tofupilot.com/questions/can-i-edit-tofupilot-run-metadata-after-upload
Yes. The v2 endpoint PATCH /v2/runs/{id}/metadata upserts custom metadata key-value pairs on a run after ingestion. To correct a unit's serial number or part number, edit the unit inline rather than using this endpoint.

### How do I fetch a unit's latest test run via the TofuPilot API?
URL: https://www.tofupilot.com/questions/how-do-i-fetch-a-units-latest-test-run-via-the-tofupilot-api
Query runs.list filtered by serial number, sorted by start time descending with limit=1. For per-unit lifetime views in the dashboard, use the Unit Explorer.

### Does TofuPilot have a REST API without the SDK?
URL: https://www.tofupilot.com/questions/does-tofupilot-have-a-rest-api-without-the-sdk
Yes. The full platform is exposed as a REST API at https://www.tofupilot.app/api/v2 with API-key authentication, pagination, and typed error codes. Every resource (runs, units, parts, procedures, stations, measurements, phases, logs, attachments, imports) has documented endpoints.

### Can one test station run multiple procedures?
URL: https://www.tofupilot.com/questions/can-one-test-station-run-multiple-procedures
Yes. Deploy each procedure to the station; the operator then selects which procedure to run from the station UI. A station can hold as many procedures as your workflow needs.

### Can stations run headless, for example in CI or a rack with no display?
URL: https://www.tofupilot.com/questions/can-stations-run-headless-for-example-in-ci-or-a-rack-with-no-display
Yes. Stations authenticate with long-lived station keys instead of interactive login. The install command embeds a single-use setup token (valid one hour, scoped to one station) that the CLI exchanges for a station key on first boot, so no display or manual sign-in is needed.

### How do I install system packages (non-Python dependencies) on a station?
URL: https://www.tofupilot.com/questions/how-do-i-install-system-packages-non-python-dependencies-on-a-station
Install native system dependencies on the machine's system Python, then enable the "System packages" toggle in the station's Setup. On every deployment the CLI rebuilds the virtual environment with uv using --system-site-packages, so the venv can see those machine-level packages instead of losing them on each pull.

### How do I launch a TofuPilot station on boot?
URL: https://www.tofupilot.com/questions/how-do-i-launch-a-tofupilot-station-on-boot
Use the launch-on-boot setting; the CLI installs a system service (systemd on Linux) so the station starts with the machine, including root-level system services for kiosk deployments.

### Can I roll back a bad deployment in TofuPilot?
URL: https://www.tofupilot.com/questions/can-i-roll-back-a-bad-deployment-in-tofupilot
Yes. Open the deployment's menu in the dashboard and select Instant Rollback to revert stations to any previous deployment immediately. Rollback is dashboard-only for now; API and SDK support is planned. Rolling rollouts let you push a new build one station at a time before rolling it out fleet-wide.

### How do I import test results from a CSV file?
URL: https://www.tofupilot.com/questions/how-do-i-import-test-results-from-a-csv-file
You need a CSV with a header row and a procedure created in the dashboard. Drop the .csv on the procedure's import page, click the row to open the mapping sidebar, map columns or apply a saved preset, then click Import. Four fields are required: serial number, outcome, started at, and ended at. Duplicate files are flagged by content hash, so re-importing the same CSV links to the existing run.

### Does TofuPilot integrate with Discord?
URL: https://www.tofupilot.com/questions/does-tofupilot-integrate-with-discord
Yes. TofuPilot posts a message to a Discord channel whenever a workflow fires, via a Discord webhook, so you do not need to host a bot. In Discord, create a webhook under channel Settings > Integrations > Webhooks, copy the URL, then add a "Send Discord message" action to your workflow. The message is plain text with markdown and {{variable}} support, and you can set a custom username and avatar.

### What file formats can I import into TofuPilot?
URL: https://www.tofupilot.com/questions/what-file-formats-can-i-import-into-tofupilot
TofuPilot imports nine formats in two categories. Tabular (require field mapping): CSV and Excel (.xlsx). Structured (auto-detected): STDF, ATDF, ATML, NI TestStand XML, OpenHTF JSON, WATS WSXF, and WSJF. Each file is parsed into one or more runs through the same path a live test uses, so imported runs are indistinguishable from station-produced ones.

### How do I import an Excel file into TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-i-import-an-excel-file-into-tofupilot
You need an Excel workbook (.xlsx) with a header row and a procedure created in the dashboard. Drop the .xlsx on the import page, click the row to open the mapping sidebar, apply a saved preset or map columns by hand, then click Import. Only serial number, outcome, started at, and ended at are required; optional mappings include part number, revision number, batch number, operated by, and description. Save a finished mapping as a preset so the next workbook maps itself.

### What analytics does TofuPilot provide out of the box?
URL: https://www.tofupilot.com/questions/what-analytics-does-tofupilot-provide-out-of-the-box
First pass yield, Cpk and control charts (measurement control), phase Pareto by failure rate and duration, run and unit explorers with filtering, and log search across runs. Charts work on any uploaded measurement without extra configuration.

### Why does my Cpk drop when tests correctly catch failing units?
URL: https://www.tofupilot.com/questions/why-does-my-cpk-drop-when-tests-correctly-catch-failing-units
Cpk measures process capability, so far-out-of-limit readings from a known upstream failure distort it. Use yield to track pass/fail performance and Cpk to track drift on healthy processes. For measurements that are invalid because an earlier step failed, record no value (None) instead of a garbage reading so it is excluded from capability statistics.

### Can I export test data to CSV or JSON?
URL: https://www.tofupilot.com/questions/can-i-export-test-data-to-csv-or-json
Yes. The Run Explorer's Export button writes the current filtered run set to CSV (one row per run) or JSON (nested), for all visible rows or a selection. Exports honor team scope and role permissions. For automated pipelines, use the REST API or SDKs.

### How do I install TofuPilot behind a corporate proxy or firewall?
URL: https://www.tofupilot.com/questions/how-do-i-install-tofupilot-behind-a-corporate-proxy-or-firewall
Allowlist *.tofupilot.app on your proxy. The API, the install script (tofupilot.app/install), the CLI version endpoint (get.tofupilot.app) and the binaries and Python runtime mirror (dl.tofupilot.app) all live on that domain, so one entry covers a station. Stations only need outbound HTTPS to TofuPilot; no inbound ports are required. A self-hosted server additionally needs www.tofupilot.com (deploy script and license) and ghcr.io (container images). If the proxy enforces per-user session limits, use per-user proxy credentials rather than shared ones so the install and upload traffic is not throttled.

### Do I need GitHub to deploy test procedures?
URL: https://www.tofupilot.com/questions/do-i-need-github-to-deploy-test-procedures
No. Git integration with GitHub or GitLab enables auto-deploy on every push, but you can also deploy from any machine without a connected repo using "tofupilot link" and "tofupilot deploy" from your local working tree.

### How do I import STDF files into TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-i-import-stdf-files-into-tofupilot
You need an STDF v4 datalog (.stdf) and a procedure in the dashboard. STDF is structured binary and requires no column mapping. Drag the .stdf file into the dropzone (shows "Auto-detected") and click Import; TofuPilot detects STDF by its binary FAR record header, not the filename. A multi-part STDF file imports as one run per part, so a full wafer or lot lands as a batch of runs. Via API, call POST /v2/import with importer STDF (up to 100 files per call).

### Does TofuPilot integrate with my MES or ERP?
URL: https://www.tofupilot.com/questions/does-tofupilot-integrate-with-my-mes-or-erp
Yes, in both directions. Dashboard-side workflows push run results, serial numbers and batch data into your system, with a native Odoo integration, a generic HTTP action that calls any REST API, and custom integrations such as SAP for Enterprise customers. First Resonance and Epsilon3 integrations are in progress. Execution-side, a phase can query your MES before any test runs and stop the unit if it is refused, with a dedicated status so a blocked unit is never counted as a test failure.

### Is it safe to connect my factory stations to TofuPilot?
URL: https://www.tofupilot.com/questions/is-it-safe-to-connect-my-factory-stations-to-tofupilot
Yes, connecting your test stations to TofuPilot is safe. With our Pro plan, you can generate a secure API key for each station, with permissions restricted to specific actions such as uploading test results on specific test procedures.

### What if my stations can't connect to the internet?
URL: https://www.tofupilot.com/questions/what-if-my-stations-cant-connect-to-the-internet
If your stations can't connect publicly, you can either only allow access to TofuPilot URLs or collect test data on your side and upload it later. The original timestamps are preserved.

### What if my test infrastructure is fully offline?
URL: https://www.tofupilot.com/questions/what-if-my-test-infrastructure-is-fully-offline
We recommend our Enterprise plan for self-hosting TofuPilot within your infrastructure, keeping everything offline while maintaining all features.

### How secure is TofuPilot Cloud?
URL: https://www.tofupilot.com/questions/how-secure-is-tofupilot-cloud
All TofuPilot plans include secure cloud hosting managed by our team, and powered by AWS. Our team manages infrastructure with end-to-end communication encryption, strict internal role-based access, daily backups and regular audits to protect your sensitive test data.

### Can TofuPilot be self-hosted?
URL: https://www.tofupilot.com/questions/can-tofupilot-be-self-hosted
Yes, on the Enterprise plan. The dashboard runs on your own servers so no test data leaves your network, and we work with your IT team on the deployment. The self-hosted instance runs fully offline, including licence activation, and stations queue results locally when connectivity drops.

### Can I export my data from TofuPilot?
URL: https://www.tofupilot.com/questions/can-i-export-my-data-from-tofupilot
Yes, you can export data using our Python client or REST API or request a full extract from our team.

### What support does TofuPilot provide?
URL: https://www.tofupilot.com/questions/what-support-does-tofupilot-provide
Support levels differ by plan: Lab users can access support through Discord, Pro users via email, and Enterprise users with a dedicated contact in our team.

### Who is behind TofuPilot?
URL: https://www.tofupilot.com/questions/who-is-behind-tofupilot
TofuPilot is developed by TofuPilot SA, an award-winning Swiss startup founded by robotics test engineers.

### What is the TofuPilot product roadmap?
URL: https://www.tofupilot.com/questions/what-is-the-tofupilot-product-roadmap
Our roadmap is dedicated to streamlining your test development processes and improving analytics and insights. You can view our full roadmap to see what we're working on.

### When am I billed?
URL: https://www.tofupilot.com/questions/when-am-i-billed
Pro plans are billed monthly based on selected options and last month's usage. Enterprise plans offer monthly or annual billing to suit your needs.

### What payment methods do you accept?
URL: https://www.tofupilot.com/questions/what-payment-methods-do-you-accept
We accept payments from all major card providers for Pro plans, processed securely by our partner Stripe. Enterprise clients can request custom billing options.

### Does TofuPilot support two-factor authentication (2FA)?
URL: https://www.tofupilot.com/questions/does-tofupilot-support-two-factor-authentication-2fa
Yes. TofuPilot supports two methods: passkeys (WebAuthn backed by biometrics, phone, or a hardware key like YubiKey) and authenticator apps (TOTP from 1Password, Google Authenticator, or Authy). Enable it under Settings > Authentication and save your one-time recovery codes; you can register multiple passkeys. Admins can require 2FA for all members under Settings > Organization > Security, and SSO members satisfy the requirement when their IdP enforces 2FA upstream.

### How do I back up a self-hosted TofuPilot instance?
URL: https://www.tofupilot.com/questions/how-do-i-back-up-a-self-hosted-tofupilot-instance
Your data lives in two Docker volumes: tofupilot-pg-data for the PostgreSQL database and tofupilot-seaweed-data for file storage. Back up both volumes on a regular schedule so a restore recovers your full instance.

### What are the server requirements for self-hosting TofuPilot?
URL: https://www.tofupilot.com/questions/what-are-the-server-requirements-for-self-hosting-tofupilot
Ubuntu 20.04+ or Debian on x86_64, 2+ CPU cores, 4 GB RAM, 40 GB+ disk, standard root Docker (not rootless), ports 80/443 open, and three subdomains for the dashboard, storage, and deployer.

### Does TofuPilot support SSO with Entra ID, Okta, or Google Workspace?
URL: https://www.tofupilot.com/questions/does-tofupilot-support-sso-with-entra-id-okta-or-google-workspace
Yes. Single sign-on via SAML 2.0 and OIDC works with Okta, Microsoft Entra, Azure AD, Auth0, OneLogin, and Google Workspace. SSO is available on Enterprise plans, and SCIM provisioning syncs organization members and roles from your directory.

### Does TofuPilot depend on the EU-US Data Privacy Framework?
URL: https://www.tofupilot.com/questions/does-tofupilot-depend-on-the-eu-us-data-privacy-framework
No. The EU-US Data Privacy Framework is required for US companies transferring EU personal data to the US. Since TofuPilot is Swiss-based with EU-hosted infrastructure, we do not rely on transfer frameworks that could be invalidated, as happened with Safe Harbor in 2015 and Privacy Shield in 2020.

### What user roles does TofuPilot have?
URL: https://www.tofupilot.com/questions/what-user-roles-does-tofupilot-have
Five roles: Owner, Admin, Developer, Viewer, and Operator. Teams additionally scope stations and data by department, supplier, or production line.

### How do I remove a team member in TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-i-remove-a-team-member-in-tofupilot
Open Settings > Members. Ban blocks a user's sign-in without deleting their data and is reversible; Remove revokes access permanently. Both end access immediately.

### Does TofuPilot support SCIM provisioning?
URL: https://www.tofupilot.com/questions/does-tofupilot-support-scim-provisioning
Yes. Your identity provider pushes member changes to TofuPilot, so adding a user in Okta creates a TofuPilot account and removing them from the group revokes access. SCIM mirrors user creation, removal, attribute updates, suspension, and group changes. SCIM runs on top of SSO, so it requires SSO. Configure it under Settings > Organization > SCIM Provisioning to generate a bearer token and base URL, then map groups to roles in your IdP. The Owner role cannot be assigned via SCIM.

### How do teams scope data in TofuPilot?
URL: https://www.tofupilot.com/questions/how-do-teams-scope-data-in-tofupilot
Teams scope which stations and runs a member sees. A station belongs to one team, but a member can belong to many. Owners, Admins, and Developers see every team's stations and runs. Viewers with no team see all data read-only; Viewers assigned to teams see only those. Operators see only stations in their assigned teams. A station with no team is visible to every Viewer and no Operator. Deleting a team keeps its stations and members, removing only the scoping link.

### Where is TofuPilot cloud data hosted?
URL: https://www.tofupilot.com/questions/where-is-tofupilot-cloud-data-hosted
Your core data (database and file storage) is hosted in Europe, operated by TofuPilot SA, a Swiss company not subject to the US CLOUD Act. Compute runs on a serverless architecture that routes each request to the nearest data center, so the platform stays fast for teams worldwide while data at rest remains in Europe. Customers needing strict EU-only data residency can get a managed dedicated instance hosted entirely in the EU, or self-host.

### How does TofuPilot secure my test data and access?
URL: https://www.tofupilot.com/questions/how-does-tofupilot-secure-my-test-data-and-access
Data is encrypted in transit and at rest. Access is controlled by role-based permissions and scoped API keys, and two-factor authentication supports passkeys (WebAuthn) and authenticator apps (TOTP). TofuPilot SA is Swiss-incorporated, so your data sits outside the reach of US surveillance law such as the CLOUD Act. Teams needing full control can self-host.

### Does TofuPilot work in air-gapped environments?
URL: https://www.tofupilot.com/questions/does-tofupilot-work-in-air-gapped-environments
Yes. The self-hosted deployment runs fully offline. License activation without internet works by copying a license token from TofuPilot Orbit and pasting it under Settings > Subscription on your instance.

### Can TofuPilot be self-hosted on my own servers?
URL: https://www.tofupilot.com/questions/can-tofupilot-be-self-hosted-on-my-own-servers
Yes, on the Enterprise plan. You run the dashboard, deployer, and PostgreSQL in your own infrastructure with Docker Compose, one organization per instance. Create the instance on tofupilot.com/orbit to get a license key, then install with: curl -fsSL https://www.tofupilot.com/deploy | bash -s -- --license YOUR_LICENSE_KEY. The same script updates the instance (| bash), lists available releases (--list-versions), pins one (--version), and removes the deployment (--uninstall). The server needs outbound access to www.tofupilot.com and ghcr.io.

## Documentation

- [Getting started with TofuPilot](https://www.tofupilot.com/docs/getting-started-with-tofupilot)
- [Incremental Migration](https://www.tofupilot.com/docs/incremental-migration)
- [TofuPilot Documentation](https://www.tofupilot.com/docs)
- [Access Roles](https://www.tofupilot.com/docs/access/access-roles)
- [API Keys](https://www.tofupilot.com/docs/access/api-keys)
- [Access Groups](https://www.tofupilot.com/docs/access/groups)
- [Access](https://www.tofupilot.com/docs/access)
- [Managing Members](https://www.tofupilot.com/docs/access/managing-members)
- [SCIM Provisioning](https://www.tofupilot.com/docs/access/scim)
- [Setup Tokens](https://www.tofupilot.com/docs/access/setup-tokens)
- [Single Sign-On](https://www.tofupilot.com/docs/access/sso)
- [Passkeys and MFA](https://www.tofupilot.com/docs/access/two-factor-authentication)
- [TofuPilot Chat](https://www.tofupilot.com/docs/ai/chat)
- [TofuPilot AI](https://www.tofupilot.com/docs/ai)
- [MCP Server](https://www.tofupilot.com/docs/ai/mcp)
- [Root Cause Analysis](https://www.tofupilot.com/docs/ai/root-cause-analysis)
- [Configure Alerts](https://www.tofupilot.com/docs/alerts/configure-alerts)
- [Alerts](https://www.tofupilot.com/docs/alerts)
- [API & SDKs](https://www.tofupilot.com/docs/api)
- [Environments](https://www.tofupilot.com/docs/deployments/environments)
- [Git](https://www.tofupilot.com/docs/deployments/git)
- [Deployments](https://www.tofupilot.com/docs/deployments)
- [Instant Rollback](https://www.tofupilot.com/docs/deployments/instant-rollback)
- [Rolling Rollout](https://www.tofupilot.com/docs/deployments/rolling-rollout)
- [tofupilot attachments](https://www.tofupilot.com/docs/cli/attachments)
- [tofupilot batches](https://www.tofupilot.com/docs/cli/batches)
- [tofupilot config](https://www.tofupilot.com/docs/cli/config)
- [tofupilot deploy](https://www.tofupilot.com/docs/cli/deploy)
- [tofupilot deployments](https://www.tofupilot.com/docs/cli/deployments)
- [tofupilot env](https://www.tofupilot.com/docs/cli/env)
- [tofupilot imports](https://www.tofupilot.com/docs/cli/imports)
- [CLI](https://www.tofupilot.com/docs/cli)
- [run --json Event Stream in TofuPilot](https://www.tofupilot.com/docs/cli/json-events)
- [tofupilot link](https://www.tofupilot.com/docs/cli/link)
- [tofupilot login](https://www.tofupilot.com/docs/cli/login)
- [tofupilot logout](https://www.tofupilot.com/docs/cli/logout)
- [tofupilot logs](https://www.tofupilot.com/docs/cli/logs)
- [tofupilot measurements](https://www.tofupilot.com/docs/cli/measurements)
- [tofupilot parts](https://www.tofupilot.com/docs/cli/parts)
- [tofupilot phases](https://www.tofupilot.com/docs/cli/phases)
- [tofupilot procedures](https://www.tofupilot.com/docs/cli/procedures)
- [tofupilot pull](https://www.tofupilot.com/docs/cli/pull)
- [tofupilot queue](https://www.tofupilot.com/docs/cli/queue)
- [tofupilot revisions](https://www.tofupilot.com/docs/cli/revisions)
- [tofupilot rollback](https://www.tofupilot.com/docs/cli/rollback)
- [tofupilot run](https://www.tofupilot.com/docs/cli/run)
- [tofupilot runs](https://www.tofupilot.com/docs/cli/runs)
- [tofupilot service](https://www.tofupilot.com/docs/cli/service)
- [tofupilot stations](https://www.tofupilot.com/docs/cli/stations)
- [tofupilot uninstall](https://www.tofupilot.com/docs/cli/uninstall)
- [tofupilot units](https://www.tofupilot.com/docs/cli/units)
- [tofupilot unlink](https://www.tofupilot.com/docs/cli/unlink)
- [tofupilot update](https://www.tofupilot.com/docs/cli/update)
- [tofupilot users](https://www.tofupilot.com/docs/cli/users)
- [tofupilot versions](https://www.tofupilot.com/docs/cli/versions)
- [tofupilot whoami](https://www.tofupilot.com/docs/cli/whoami)
- [Frameworks on TofuPilot](https://www.tofupilot.com/docs/frameworks)
- [OpenHTF on TofuPilot](https://www.tofupilot.com/docs/frameworks/openhtf)
- [Pytest on TofuPilot](https://www.tofupilot.com/docs/frameworks/pytest)
- [Robot Framework on TofuPilot](https://www.tofupilot.com/docs/frameworks/robot-framework)
- [Device Data Model](https://www.tofupilot.com/docs/fundamentals/device-data-model)
- [TofuPilot fundamental concepts](https://www.tofupilot.com/docs/fundamentals)
- [Test Lifecycle](https://www.tofupilot.com/docs/fundamentals/lifecycle)
- [Test Data Model](https://www.tofupilot.com/docs/fundamentals/test-data-model)
- [Import ATDF Test Data](https://www.tofupilot.com/docs/import/atdf)
- [Import ATML Test Data](https://www.tofupilot.com/docs/import/atml)
- [Import CSV Test Data](https://www.tofupilot.com/docs/import/csv)
- [Import Excel Test Data](https://www.tofupilot.com/docs/import/excel)
- [Import](https://www.tofupilot.com/docs/import)
- [Import OpenHTF Logs](https://www.tofupilot.com/docs/import/openhtf)
- [Import STDF Test Data](https://www.tofupilot.com/docs/import/stdf)
- [Import NI TestStand Reports](https://www.tofupilot.com/docs/import/teststand)
- [Import WATS WSJF Reports](https://www.tofupilot.com/docs/import/wsjf)
- [Import WATS WSXF Reports](https://www.tofupilot.com/docs/import/wsxf)
- [Discord](https://www.tofupilot.com/docs/integrations/discord)
- [Email](https://www.tofupilot.com/docs/integrations/email)
- [HTTP](https://www.tofupilot.com/docs/integrations/http)
- [Integrations](https://www.tofupilot.com/docs/integrations)
- [InvenTree](https://www.tofupilot.com/docs/integrations/inventree)
- [Linear](https://www.tofupilot.com/docs/integrations/linear)
- [Odoo](https://www.tofupilot.com/docs/integrations/odoo)
- [Slack](https://www.tofupilot.com/docs/integrations/slack)
- [Batches](https://www.tofupilot.com/docs/inventory/batches)
- [Inventory](https://www.tofupilot.com/docs/inventory)
- [Parts and Revisions](https://www.tofupilot.com/docs/inventory/parts)
- [Serial Numbers](https://www.tofupilot.com/docs/inventory/serial-numbers)
- [Units](https://www.tofupilot.com/docs/inventory/units)
- [Billing](https://www.tofupilot.com/docs/pricing/billing)
- [Enterprise Plan](https://www.tofupilot.com/docs/pricing/enterprise)
- [Pricing](https://www.tofupilot.com/docs/pricing)
- [Invoices](https://www.tofupilot.com/docs/pricing/invoices)
- [Lab Plan](https://www.tofupilot.com/docs/pricing/lab)
- [License](https://www.tofupilot.com/docs/pricing/license)
- [Pro Plan](https://www.tofupilot.com/docs/pricing/pro)
- [Usage](https://www.tofupilot.com/docs/pricing/usage)
- [Analytics](https://www.tofupilot.com/docs/run-analytics)
- [Log Explorer](https://www.tofupilot.com/docs/run-analytics/log-explorer)
- [Measurement Control](https://www.tofupilot.com/docs/run-analytics/measurement-control)
- [Phase Pareto](https://www.tofupilot.com/docs/run-analytics/phase-pareto)
- [Run Analytics](https://www.tofupilot.com/docs/run-analytics/run-analytics)
- [Run Explorer](https://www.tofupilot.com/docs/run-analytics/run-explorer)
- [Unit Explorer](https://www.tofupilot.com/docs/run-analytics/unit-explorer)
- [Using the TofuPilot C++ SDK](https://www.tofupilot.com/docs/sdk/cpp)
- [Using the TofuPilot C# SDK](https://www.tofupilot.com/docs/sdk/csharp)
- [SDKs](https://www.tofupilot.com/docs/sdk)
- [Using the TofuPilot MATLAB SDK](https://www.tofupilot.com/docs/sdk/matlab)
- [Using the TofuPilot Python SDK](https://www.tofupilot.com/docs/sdk/python)
- [Using the TofuPilot Rust SDK](https://www.tofupilot.com/docs/sdk/rust)
- [API Audit Log](https://www.tofupilot.com/docs/security/api-audit-log)
- [Environment Variables Security](https://www.tofupilot.com/docs/security/environment-variables)
- [Security](https://www.tofupilot.com/docs/security)
- [Configure](https://www.tofupilot.com/docs/self-hosting/configure)
- [Deploy](https://www.tofupilot.com/docs/self-hosting/deploy)
- [Self-Hosting](https://www.tofupilot.com/docs/self-hosting)
- [Operations](https://www.tofupilot.com/docs/self-hosting/operations)
- [Private CA](https://www.tofupilot.com/docs/self-hosting/private-ca)
- [Upgrade](https://www.tofupilot.com/docs/self-hosting/upgrade)
- [Auto-update](https://www.tofupilot.com/docs/station/auto-update)
- [How to Configure Station Variables with TofuPilot](https://www.tofupilot.com/docs/station/configure-station-variables)
- [Desktop shortcut](https://www.tofupilot.com/docs/station/desktop-shortcut)
- [Station](https://www.tofupilot.com/docs/station)
- [Kiosk UI](https://www.tofupilot.com/docs/station/kiosk-ui)
- [Launch on boot](https://www.tofupilot.com/docs/station/launch-on-boot)
- [Offline Upload Queue](https://www.tofupilot.com/docs/station/offline-upload)
- [Remote UI](https://www.tofupilot.com/docs/station/remote-ui)
- [System packages](https://www.tofupilot.com/docs/station/system-packages)
- [Terminal UI](https://www.tofupilot.com/docs/station/terminal-ui)
- [Studio](https://www.tofupilot.com/docs/studio)
- [REST API](https://www.tofupilot.com/docs/api/rest-api)
- [Attachments](https://www.tofupilot.com/docs/frameworks/tofupilot/attachments)
- [Dashboard Access](https://www.tofupilot.com/docs/frameworks/tofupilot/dashboard-access)
- [Environment Variables](https://www.tofupilot.com/docs/frameworks/tofupilot/environment-variables)
- [TofuPilot Framework](https://www.tofupilot.com/docs/frameworks/tofupilot)
- [Logs](https://www.tofupilot.com/docs/frameworks/tofupilot/logs)
- [Monorepos](https://www.tofupilot.com/docs/frameworks/tofupilot/monorepos)
- [Plugs](https://www.tofupilot.com/docs/frameworks/tofupilot/plugs)
- [Procedures](https://www.tofupilot.com/docs/frameworks/tofupilot/procedures)
- [Reports](https://www.tofupilot.com/docs/frameworks/tofupilot/reports)
- [Run Context](https://www.tofupilot.com/docs/frameworks/tofupilot/run-context)
- [Sequences](https://www.tofupilot.com/docs/frameworks/tofupilot/sequences)
- [Unit Under Test (UUT)](https://www.tofupilot.com/docs/frameworks/tofupilot/unit-under-test)
- [Executions](https://www.tofupilot.com/docs/integrations/workflows/executions)
- [Workflows](https://www.tofupilot.com/docs/integrations/workflows)
- [Lifecycle](https://www.tofupilot.com/docs/integrations/workflows/lifecycle)
- [Versions](https://www.tofupilot.com/docs/integrations/workflows/versions)
- [Create batch](https://www.tofupilot.com/docs/api/rest-api/batches/create)
- [Delete batch](https://www.tofupilot.com/docs/api/rest-api/batches/delete)
- [Get batch](https://www.tofupilot.com/docs/api/rest-api/batches/get)
- [List and filter batches](https://www.tofupilot.com/docs/api/rest-api/batches/list)
- [Update batch](https://www.tofupilot.com/docs/api/rest-api/batches/update)
- [Get deployment](https://www.tofupilot.com/docs/api/rest-api/deployments/get)
- [List and filter deployments](https://www.tofupilot.com/docs/api/rest-api/deployments/list)
- [Who am I](https://www.tofupilot.com/docs/api/rest-api/identity/list)
- [Import runs from structured files](https://www.tofupilot.com/docs/api/rest-api/imports/create_structured)
- [Import a run from a tabular file](https://www.tofupilot.com/docs/api/rest-api/imports/create_tabular)
- [Initialize upload](https://www.tofupilot.com/docs/api/rest-api/attachments/create)
- [Finalize upload](https://www.tofupilot.com/docs/api/rest-api/attachments/create_finalize)
- [Get log](https://www.tofupilot.com/docs/api/rest-api/logs/get)
- [List and filter logs](https://www.tofupilot.com/docs/api/rest-api/logs/list)
- [Get measurement](https://www.tofupilot.com/docs/api/rest-api/measurements/get)
- [List and filter measurements](https://www.tofupilot.com/docs/api/rest-api/measurements/list)
- [Create part](https://www.tofupilot.com/docs/api/rest-api/parts/create)
- [Delete part](https://www.tofupilot.com/docs/api/rest-api/parts/delete)
- [Get part](https://www.tofupilot.com/docs/api/rest-api/parts/get)
- [List and filter parts](https://www.tofupilot.com/docs/api/rest-api/parts/list)
- [Update part](https://www.tofupilot.com/docs/api/rest-api/parts/update)
- [Get phase](https://www.tofupilot.com/docs/api/rest-api/phases/get)
- [List and filter phases](https://www.tofupilot.com/docs/api/rest-api/phases/list)
- [Create procedure](https://www.tofupilot.com/docs/api/rest-api/procedures/create)
- [Delete procedure](https://www.tofupilot.com/docs/api/rest-api/procedures/delete)
- [Get procedure](https://www.tofupilot.com/docs/api/rest-api/procedures/get)
- [List and filter procedures](https://www.tofupilot.com/docs/api/rest-api/procedures/list)
- [Update procedure](https://www.tofupilot.com/docs/api/rest-api/procedures/update)
- [Attach file to run](https://www.tofupilot.com/docs/api/rest-api/runs/attach_create)
- [Create run](https://www.tofupilot.com/docs/api/rest-api/runs/create)
- [Delete runs](https://www.tofupilot.com/docs/api/rest-api/runs/delete)
- [Get run](https://www.tofupilot.com/docs/api/rest-api/runs/get)
- [List and filter runs](https://www.tofupilot.com/docs/api/rest-api/runs/list)
- [Update run](https://www.tofupilot.com/docs/api/rest-api/runs/update)
- [Update run metadata](https://www.tofupilot.com/docs/api/rest-api/runs/update_metadata)
- [Create station](https://www.tofupilot.com/docs/api/rest-api/stations/create)
- [Remove station](https://www.tofupilot.com/docs/api/rest-api/stations/delete)
- [Get station](https://www.tofupilot.com/docs/api/rest-api/stations/get)
- [Get current station](https://www.tofupilot.com/docs/api/rest-api/stations/get_current)
- [List and filter stations](https://www.tofupilot.com/docs/api/rest-api/stations/list)
- [Update station](https://www.tofupilot.com/docs/api/rest-api/stations/update)
- [Get current user](https://www.tofupilot.com/docs/api/rest-api/users/get_current)
- [List users](https://www.tofupilot.com/docs/api/rest-api/users/list)
- [Attach file to unit](https://www.tofupilot.com/docs/api/rest-api/units/attach_create)
- [Delete unit attachments](https://www.tofupilot.com/docs/api/rest-api/units/attach_delete)
- [Create unit](https://www.tofupilot.com/docs/api/rest-api/units/create)
- [Delete units](https://www.tofupilot.com/docs/api/rest-api/units/delete)
- [Get unit](https://www.tofupilot.com/docs/api/rest-api/units/get)
- [List and filter units](https://www.tofupilot.com/docs/api/rest-api/units/list)
- [Update unit](https://www.tofupilot.com/docs/api/rest-api/units/update)
- [Development](https://www.tofupilot.com/docs/frameworks/tofupilot/environments/development)
- [Environments](https://www.tofupilot.com/docs/frameworks/tofupilot/environments)
- [Station](https://www.tofupilot.com/docs/frameworks/tofupilot/environments/station)
- [On Fail](https://www.tofupilot.com/docs/frameworks/tofupilot/execution/callbacks)
- [Debugging](https://www.tofupilot.com/docs/frameworks/tofupilot/execution/debugging)
- [Execution](https://www.tofupilot.com/docs/frameworks/tofupilot/execution)
- [Slots](https://www.tofupilot.com/docs/frameworks/tofupilot/execution/slots)
- [Strategy](https://www.tofupilot.com/docs/frameworks/tofupilot/execution/strategy)
- [Workers](https://www.tofupilot.com/docs/frameworks/tofupilot/execution/workers)
- [Aggregations](https://www.tofupilot.com/docs/frameworks/tofupilot/measurements/aggregations)
- [Boolean](https://www.tofupilot.com/docs/frameworks/tofupilot/measurements/boolean)
- [Examples](https://www.tofupilot.com/docs/frameworks/tofupilot/measurements/examples)
- [Measurements](https://www.tofupilot.com/docs/frameworks/tofupilot/measurements)
- [JSON](https://www.tofupilot.com/docs/frameworks/tofupilot/measurements/json)
- [Multi-Dimensional](https://www.tofupilot.com/docs/frameworks/tofupilot/measurements/multi-dimensional)
- [Numeric](https://www.tofupilot.com/docs/frameworks/tofupilot/measurements/numeric)
- [String](https://www.tofupilot.com/docs/frameworks/tofupilot/measurements/string)
- [Checklist](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/checklist)
- [Image](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/image)
- [Operator UI](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui)
- [Multi-Select](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/multiselect)
- [Number Input](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/number-input)
- [Panels](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/panels)
- [Progress](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/progress)
- [Radio](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/radio)
- [Select](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/select)
- [Slider](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/slider)
- [Switch](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/switch)
- [Text Input](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/text-input)
- [Text](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/text)
- [Textarea](https://www.tofupilot.com/docs/frameworks/tofupilot/operator-ui/textarea)
- [Depends On](https://www.tofupilot.com/docs/frameworks/tofupilot/phases/depends-on)
- [Enable / Disable](https://www.tofupilot.com/docs/frameworks/tofupilot/phases/enable-disable)
- [Executable](https://www.tofupilot.com/docs/frameworks/tofupilot/phases/executable)
- [Phases](https://www.tofupilot.com/docs/frameworks/tofupilot/phases)
- [Python](https://www.tofupilot.com/docs/frameworks/tofupilot/phases/python)
- [Retry](https://www.tofupilot.com/docs/frameworks/tofupilot/phases/retry)
- [Setup / Teardown](https://www.tofupilot.com/docs/frameworks/tofupilot/phases/setup-teardown)
- [Timeout](https://www.tofupilot.com/docs/frameworks/tofupilot/phases/timeout)
- [Create part revision](https://www.tofupilot.com/docs/api/rest-api/parts/revisions/part_number_revisions_create)
- [Delete part revision](https://www.tofupilot.com/docs/api/rest-api/parts/revisions/part_number_revisions_revision_number_delete)
- [Get part revision](https://www.tofupilot.com/docs/api/rest-api/parts/revisions/part_number_revisions_revision_number_get)
- [Update part revision](https://www.tofupilot.com/docs/api/rest-api/parts/revisions/part_number_revisions_revision_number_update)
- [Create procedure version](https://www.tofupilot.com/docs/api/rest-api/procedures/versions/create)
- [Delete procedure version](https://www.tofupilot.com/docs/api/rest-api/procedures/versions/delete)
- [Get procedure version](https://www.tofupilot.com/docs/api/rest-api/procedures/versions/get)
- [Add sub-unit](https://www.tofupilot.com/docs/api/rest-api/units/sub-units/add_child)
- [Remove sub-unit](https://www.tofupilot.com/docs/api/rest-api/units/sub-units/remove_child)