Manufacturing Test Frameworks Compared

Four frameworks dominate manufacturing test automation: OpenHTF, pytest, OpenTAP, and NI TestStand. Each makes different tradeoffs between structure, flexibility, cost, and ecosystem. This guide compares them with real code, concrete metrics, and decision criteria so you can pick the right one.

Framework Overview

Feature Comparison Matrix

The Same Test in Each Framework

A simple functional test: measure a 3.3V rail voltage, check it's within 3.2V to 3.4V.

OpenHTF

import openhtf as htf
from openhtf.util import units
from tofupilot.openhtf import TofuPilot


class DutPlug(htf.plugs.BasePlug):
    def setUp(self):
        self.voltage = 3.31  # Replace with instrument read

    def tearDown(self):
        pass


@htf.measures(
    htf.Measurement("rail_3v3")
    .in_range(3.2, 3.4)
    .with_units(units.VOLT),
)
@htf.plug(dut=DutPlug)
def test_power(test, dut):
    test.measurements.rail_3v3 = dut.voltage


def main():
    test = htf.Test(test_power, procedure_id="FCT-001", part_number="PCBA-100")
    with TofuPilot(test):
        test.execute(test_start=lambda: input("Serial: "))


if __name__ == "__main__":
    main()

Strengths: Measurements are structured data (name, value, limits, units). One line for TofuPilot integration. Operator gets a serial number prompt automatically.

pytest

import pytest
from tofupilot import TofuPilotClient


@pytest.fixture
def dut():
    connection = {"voltage": 3.31}  # Replace with real connection
    yield connection


def test_power_rail(dut):
    voltage = dut["voltage"]
    assert 3.2 <= voltage <= 3.4, f"3.3V rail: {voltage}V"

Strengths: Familiar to every Python developer. Huge plugin ecosystem. Flexible fixture system. Great for R&D and validation.

Weakness: Measurements are implicit (assert statements). No structured data for analytics without extra code.

OpenTAP

// OpenTAP C# test step
using OpenTap;

[Display("Power Rail Test", Group: "FCT")]
public class PowerRailStep : TestStep
{
    [Display("Lower Limit")]
    public double LowerLimit { get; set; } = 3.2;

    [Display("Upper Limit")]
    public double UpperLimit { get; set; } = 3.4;

    public override void Run()
    {
        double voltage = 3.31; // Replace with instrument read
        Results.Publish("rail_3v3", new { Voltage = voltage });

        if (voltage < LowerLimit || voltage > UpperLimit)
            UpgradeVerdict(Verdict.Fail);
    }
}

Strengths: GUI step editor for non-programmers. Strong Keysight instrument integration. Plugin architecture for test plans.

Weakness: C# primary language (Python plugin exists but is secondary). Smaller community.

NI TestStand

TestStand uses a visual sequence editor. The equivalent test is a sequence file (.seq) with a "Numeric Limit Test" step configured via GUI: test value = voltage reading, low limit = 3.2, high limit = 3.4.

Strengths: Mature, enterprise-grade. Deep NI hardware integration. Built-in report generation, database logging, parallel execution.

Weakness: $4,310/seat/year. Windows only. Binary sequence files don't version-control well. Tied to NI ecosystem.

Cost Analysis

When to Use Each Framework

Migration Paths

From TestStand to OpenHTF

From pytest to OpenHTF

TofuPilot Integration

All four frameworks work with TofuPilot:

Decision Flowchart

1. Do you have existing TestStand infrastructure? Yes, and it works well: stay with TestStand. Yes, but you want to migrate: move to OpenHTF.
2. Are you building production tests? Yes: OpenHTF. It was built for this.
3. Are you doing R&D or firmware validation? Yes: pytest. More flexible, better CI/CD.
4. Do you need a GUI for non-programmers? Yes: OpenTAP. Visual step editor.
5. Unsure? Start with pytest. It's the easiest to learn and you can always add OpenHTF for production later.