Test Sequencing and Orchestration

Test Sequencing and Orchestration with TofuPilot

A hardware test isn't one check. It's a sequence: power up, wait for boot, measure voltages, run calibration, check communication, stress test, power down. TofuPilot captures each step with its measurements and timing, giving you a structured record of the full test flow.

Why Test Sequencing Matters

Running tests manually from a bench instrument works for prototypes. It doesn't work for production. Production testing needs:

Repeatability: Every unit goes through the exact same steps in the exact same order
Speed: No waiting for an operator to click "next"
Data capture: Every measurement recorded automatically with limits and pass/fail status
Traceability: A record of exactly what was tested, in what order, with what results

Test orchestration means defining that sequence once and running it the same way every time.

Test Sequence Architecture

A well-structured test sequence follows a pattern:

┌─────────────┐
│  Setup       │  Power on, initialize instruments, identify DUT
├─────────────┤
│  Step 1      │  Measurement or action with pass/fail criteria
├─────────────┤
│  Step 2      │  Next measurement or action
├─────────────┤
│  ...         │  Additional steps as needed
├─────────────┤
│  Teardown    │  Power off, release instruments, upload results
└─────────────┘

Each step produces measurements. Each measurement has limits. The sequence stops on critical failures or continues through all steps depending on your strategy.

Building a Test Sequence with OpenHTF

OpenHTF is a Python framework designed for hardware test sequencing. TofuPilot integrates directly as an output callback.

production_test_sequence.py

import openhtf as htf
from openhtf.plugs import BasePlug
from tofupilot.openhtf import TofuPilotClient

class PowerSupplyPlug(BasePlug):
    """Controls the bench power supply."""
    def setup(self):
        self.psu = connect_power_supply()

    def set_voltage(self, voltage):
        self.psu.write(f"VOLT {voltage}")

    def enable_output(self):
        self.psu.write("OUTP ON")

    def disable_output(self):
        self.psu.write("OUTP OFF")

    def teardown(self):
        self.disable_output()

class DMMPlug(BasePlug):
    """Reads from the digital multimeter."""
    def setup(self):
        self.dmm = connect_dmm()

    def measure_voltage(self):
        return float(self.dmm.query("MEAS:VOLT:DC?"))

    def measure_current(self):
        return float(self.dmm.query("MEAS:CURR:DC?"))

# Step 1: Power rail verification
@htf.measures(
    htf.Measurement("vcc_3v3").in_range(3.25, 3.35).with_units("V"),
    htf.Measurement("vcc_1v8").in_range(1.75, 1.85).with_units("V"),
    htf.Measurement("vcc_5v0").in_range(4.90, 5.10).with_units("V"),
)
@htf.PhaseOptions(name="Power Rail Verification")
def power_rail_check(test, psu: PowerSupplyPlug, dmm: DMMPlug):
    psu.set_voltage(12.0)
    psu.enable_output()
    time.sleep(0.5)  # Wait for rails to stabilize

    test.measurements.vcc_3v3 = dmm.measure_voltage()
    # Switch DMM channel and measure other rails
    test.measurements.vcc_1v8 = dmm.measure_voltage()
    test.measurements.vcc_5v0 = dmm.measure_voltage()

# Step 2: Current consumption
@htf.measures(
    htf.Measurement("idle_current_ma").in_range(30, 60).with_units("mA"),
    htf.Measurement("active_current_ma").in_range(80, 150).with_units("mA"),
)
@htf.PhaseOptions(name="Current Consumption")
def current_check(test, psu: PowerSupplyPlug, dmm: DMMPlug):
    test.measurements.idle_current_ma = dmm.measure_current() * 1000
    trigger_active_mode()
    time.sleep(0.2)
    test.measurements.active_current_ma = dmm.measure_current() * 1000

# Step 3: Communication check
@htf.measures(
    htf.Measurement("uart_loopback").equals(True),
    htf.Measurement("spi_whoami").equals(0x68),
)
@htf.PhaseOptions(name="Communication Interfaces")
def comm_check(test):
    test.measurements.uart_loopback = verify_uart_loopback()
    test.measurements.spi_whoami = read_spi_register(0x75)

def main():
    test = htf.Test(
        power_rail_check,
        current_check,
        comm_check,
    )
    test.add_output_callbacks(TofuPilotClient())
    test.execute(lambda: input("Scan DUT serial: "))

Every phase runs in order. Every measurement is captured. The full sequence uploads to TofuPilot when the test completes.

Building a Test Sequence with the Python Client

If you're not using OpenHTF, the TofuPilot Python client handles test sequencing directly.

sequence_with_client.py

from tofupilot import TofuPilotClient
import time

client = TofuPilotClient()

serial = input("Scan DUT serial: ")
steps = []

# Step 1: Power rails
psu.enable(12.0)
time.sleep(0.5)
vcc_3v3 = dmm.measure_voltage(channel=1)
vcc_1v8 = dmm.measure_voltage(channel=2)

steps.append({
    "name": "Power Rail Verification",
    "step_type": "measurement",
    "status": 3.25 <= vcc_3v3 <= 3.35 and 1.75 <= vcc_1v8 <= 1.85,
    "measurements": [
        {"name": "vcc_3v3", "value": vcc_3v3, "unit": "V", "limit_low": 3.25, "limit_high": 3.35},
        {"name": "vcc_1v8", "value": vcc_1v8, "unit": "V", "limit_low": 1.75, "limit_high": 1.85},
    ],
})

# Step 2: Functional check
boot_ok = wait_for_boot(timeout=5)
steps.append({
    "name": "Boot Sequence",
    "step_type": "measurement",
    "status": boot_ok,
    "measurements": [
        {"name": "boot_success", "value": boot_ok, "unit": "bool"},
    ],
})

# Upload the complete sequence
run_passed = all(s["status"] for s in steps)
client.create_run(
    procedure_id="BOARD-FUNCTIONAL-V3",
    unit_under_test={"serial_number": serial},
    run_passed=run_passed,
    steps=steps,
)

Sequence Design Best Practices

Practice	Why
Test cheap things first	If a power rail is shorted, don't waste time running communication checks
Group related measurements	Keep all voltage checks in one step, all current checks in another
Use consistent step names	"Power Rail Verification" across all procedures, not "Voltage Check" in one and "Rail Test" in another
Include setup/teardown	Always power down the DUT at the end, even if the test fails
Set limits on every measurement	A measurement without limits can't be trended or analyzed

Handling Sequence Failures

Two strategies for what happens when a step fails:

Fail-fast: Stop the sequence immediately on the first failure. Use this for safety-critical tests or when a failure in step 1 makes later steps meaningless (e.g., power rail failure means no point testing communication).

Run-all: Continue through all steps even if one fails. Use this when you want complete diagnostic data (e.g., knowing which 3 out of 20 measurements failed helps root cause analysis).

OpenHTF supports both via phase options. The TofuPilot Python client lets you implement either pattern in your test logic.

What TofuPilot Stores for Each Sequence

Every test run in TofuPilot captures:

Data	Purpose
Procedure ID	Which test sequence was run
Serial number	Which unit was tested
Overall pass/fail	Did the sequence pass?
Steps with measurements	Every step, every measurement, every limit
Timestamps	When the test started and ended
Station ID	Which station ran the test
Duration	How long the sequence took

This structured data is what enables trending, comparison, and analytics across your entire production history.

Test Sequencing and Orchestration