HIL tests generate a lot of data: measurements, waveform captures, firmware versions, environmental conditions. Without structured logging, spotting regressions across firmware releases turns into a spreadsheet archaeology project. This guide shows how to tag, attach, and query HIL results so regressions surface automatically.
Prerequisites
- Python 3.9+
- OpenHTF installed (
pip install openhtf) - TofuPilot Python client (
pip install "tofupilot[openhtf]") - A working HIL test (see How to Set Up HIL Testing for Embedded Systems with TofuPilot)
Step 1: Tag Runs with Firmware Version and Hardware Revision
Every HIL test run should carry the firmware version and hardware revision of the DUT. Record them as measurements so you can filter and query them later.
Query the firmware version from the DUT at the start of the test.
hil_regression/main.py27 lines
import openhtf as htffrom tofupilot.openhtf import uploadfrom hil_regression.phases import ( read_firmware_info, test_adc_accuracy, test_pwm_output, test_sleep_current,)def main(): test = htf.Test( read_firmware_info, test_adc_accuracy, test_pwm_output, test_sleep_current, procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", # procedure UUID from the dashboard part_number="DUT-100", ) test.add_output_callbacks(upload()) test.execute(lambda: input("Scan DUT serial number: "))if __name__ == "__main__": main()The firmware version and hardware revision get recorded as measurements, making them searchable across all runs.
import openhtf as htffrom hil_regression.serial_plug import SerialCommandPlug@htf.plug(serial=SerialCommandPlug)@htf.measures( htf.Measurement("firmware_version"), htf.Measurement("hardware_revision"), htf.Measurement("bootloader_version"),)def read_firmware_info(test, serial): """Record firmware and hardware identifiers for traceability.""" test.measurements.firmware_version = serial.send_command("VERSION") test.measurements.hardware_revision = serial.send_command("HWREV") test.measurements.bootloader_version = serial.send_command("BLVER")This gives you three filterable fields on every run. When first pass yield drops after a firmware update, filter by firmware_version to isolate the change.
Put this phase first. If it fails, you have a run whose results cannot be attributed to a firmware version, which is worse than no run at all for regression work.
Step 2: Attach Waveform and Log Files
HIL tests often capture oscilloscope waveforms, logic analyzer traces, or DUT console logs. Attach these to the test run so they're available for post-mortem analysis without digging through file shares.
hil_regression/capture.py32 lines
import csvimport timeimport openhtf as htffrom openhtf.plugs import BasePlugclass WaveformCapturePlug(BasePlug): """Captures analog samples and saves to CSV for attachment.""" def setUp(self): pass def capture_waveform(self, adc, channel: int, duration_s: float, sample_rate_hz: int) -> str: """Sample an ADC channel and write results to a CSV file.""" filepath = f"/tmp/waveform_ch{channel}_{int(time.time())}.csv" samples = [] interval = 1.0 / sample_rate_hz for i in range(int(duration_s * sample_rate_hz)): voltage = adc.read_voltage(channel) samples.append((i * interval, voltage)) time.sleep(interval) with open(filepath, "w", newline="") as f: writer = csv.writer(f) writer.writerow(["time_s", "voltage_v"]) writer.writerows(samples) return filepath def tearDown(self): passThen attach the file in your test phase:
hil_regression/phases.py27 lines
import openhtf as htffrom openhtf.util import unitsfrom hil_regression.capture import WaveformCapturePlugfrom hil_regression.analog_plug import AnalogInputPlugfrom hil_regression.serial_plug import SerialCommandPlug@htf.plug(serial=SerialCommandPlug, adc=AnalogInputPlug, capture=WaveformCapturePlug)@htf.measures( htf.Measurement("pwm_voltage_mean").in_range(minimum=1.6, maximum=1.7).with_units(units.VOLT),)def test_pwm_output(test, serial, adc, capture): """Command 50% PWM and capture the output waveform.""" serial.send_command("PWM SET 50") # Capture 1 second of data at 1 kHz waveform_path = capture.capture_waveform(adc, channel=3, duration_s=1.0, sample_rate_hz=1000) # Attach the waveform CSV to the test run test.attach("pwm_waveform", waveform_path, "text/csv") # Also record the mean voltage as a measurement import csv with open(waveform_path) as f: reader = csv.DictReader(f) voltages = [float(row["voltage_v"]) for row in reader] test.measurements.pwm_voltage_mean = sum(voltages) / len(voltages)Attachments show up in the run detail view. You can download them later for offline analysis or comparison across firmware versions.
Note that the mean is the measurement and the waveform is the attachment. That split matters: the number is what trends and alerts, while the raw capture is what you open once a trend has already told you where to look. Attaching a full capture on every unit at volume will outgrow its usefulness quickly, so consider attaching only on failure or on a sample.
Step 3: Use Sub-Units for Multi-Board HIL Setups
Many products contain multiple boards (main CPU board, power board, sensor board) that get tested together in a HIL fixture. Sub-units let you track each board independently while keeping them linked to the parent assembly.
sub_units is a field on htf.Test(...), alongside procedure_id and part_number:
hil_regression/multi_board.py30 lines
import openhtf as htffrom tofupilot.openhtf import uploadfrom hil_regression.phases import ( test_cpu_board, test_power_board, test_sensor_board,)def main(): test = htf.Test( test_cpu_board, test_power_board, test_sensor_board, procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", part_number="ASSY-100", sub_units=[ {"serial_number": "CPU-BRD-0042", "part_number": "PCB-CPU-R3"}, {"serial_number": "PWR-BRD-0019", "part_number": "PCB-PWR-R2"}, {"serial_number": "SNS-BRD-0088", "part_number": "PCB-SNS-R1"}, ], ) test.add_output_callbacks(upload()) test.execute(lambda: input("Scan assembly serial number: "))if __name__ == "__main__": main()Each sub-unit gets its own traceability record. If the sensor board starts failing after a hardware revision change, you can filter by PCB-SNS-R1 and see exactly when failures started.
The serial numbers are hardcoded here to keep the example short. On a real fixture they come from barcode scans collected before the test is built, since sub_units is a constructor argument.
hil_regression/board_phases.py40 lines
import openhtf as htffrom openhtf.util import unitsfrom hil_regression.analog_plug import AnalogInputPlugfrom hil_regression.serial_plug import SerialCommandPlug@htf.plug(serial=SerialCommandPlug)@htf.measures( htf.Measurement("cpu_clock_mhz").in_range(minimum=79, maximum=81).with_units(units.HERTZ), htf.Measurement("cpu_temp").in_range(maximum=85).with_units(units.DEGREE_CELSIUS),)def test_cpu_board(test, serial): """Verify CPU board clock and temperature.""" test.measurements.cpu_clock_mhz = float(serial.send_command("CLOCK?")) test.measurements.cpu_temp = float(serial.send_command("TEMP?"))@htf.plug(adc=AnalogInputPlug)@htf.measures( htf.Measurement("pwr_12v_rail").in_range(minimum=11.4, maximum=12.6).with_units(units.VOLT), htf.Measurement("pwr_efficiency_pct").in_range(minimum=85),)def test_power_board(test, adc): """Verify power board output rails and efficiency.""" test.measurements.pwr_12v_rail = adc.read_voltage(channel=0) vin = adc.read_voltage(channel=4) vout = adc.read_voltage(channel=5) test.measurements.pwr_efficiency_pct = (vout / vin) * 100 if vin > 0 else 0@htf.plug(serial=SerialCommandPlug, adc=AnalogInputPlug)@htf.measures( htf.Measurement("sensor_offset_mv").in_range(minimum=-5, maximum=5), htf.Measurement("sensor_gain_error_pct").in_range(minimum=-1, maximum=1),)def test_sensor_board(test, serial, adc): """Verify sensor board calibration.""" serial.send_command("SENSOR CAL_CHECK") test.measurements.sensor_offset_mv = float(serial.send_command("SENSOR OFFSET?")) test.measurements.sensor_gain_error_pct = float(serial.send_command("SENSOR GAIN_ERR?"))Detecting Regressions
Every measurement value is tracked across all runs. To detect firmware regressions:
- Filter by firmware version. Filter runs by the
firmware_versionmeasurement and compare pass rates and measurement distributions between versions. - Check measurement trends. Trend charts show measurement values over time. A sudden shift after a firmware update is a clear regression signal.
- Compare Cpk by version. If a measurement's Cpk drops after a firmware change, the process capability has degraded.
One caveat on comparing versions: firmware releases rarely land on a clean boundary. A version often overlaps with a fixture change, a new component lot, or a different shift pattern. Before attributing a shift to firmware, check whether the other dimensions moved at the same time.
Manual Tracking vs Structured Logging
| Aspect | Spreadsheet / Manual | Structured logging |
|---|---|---|
| Firmware version tagging | Copy-paste into a column | Recorded as a measurement per run |
| Waveform storage | Shared drive with naming conventions | Attached to the run, always findable |
| Multi-board traceability | Separate sheets or tabs | Sub-units linked to parent assembly |
| Regression detection | Manual chart inspection | Filter by firmware version, compare trends |
| Cross-station comparison | Merge files from different PCs | All stations upload to one workspace |
| Historical lookup | "Which folder was that in?" | Search by serial number, part number, or date |
| Audit trail | Hope nobody deleted a row | Immutable records with timestamps |
