NI TestStand costs $3-5K per seat and runs only on Windows. Engineers are searching for open source alternatives that run on Linux, integrate with Git, and don't require proprietary licenses. This guide compares the main open source test executives for manufacturing: the TofuPilot Framework, OpenHTF, OpenTAP, pytest (with hardware plugins), and HardPy.
What a Test Executive Does
A test executive manages the execution of test sequences. It handles:
| Function | What It Does |
|---|---|
| Sequencing | Runs test steps in order with branching and looping |
| Measurement collection | Records values with units and limits |
| Pass/fail decisions | Compares measurements to limits, determines result |
| Operator interaction | Prompts for input, displays status |
| Data logging | Stores results for traceability and analytics |
| Plug/driver management | Initializes and tears down instrument connections |
TestStand does all of this in a proprietary Windows application. The open source alternatives do it in code.
Comparison Table
| Feature | NI TestStand | TofuPilot Framework | OpenHTF | OpenTAP | pytest + plugins | HardPy |
|---|---|---|---|---|---|---|
| Language | LabVIEW/C#/VB | Python (Rust engine) | Python | C#/.NET | Python | Python |
| License | Proprietary ($3-5K) | MIT | Apache 2.0 | MPL 2.0 | MIT | GPL-3.0 |
| Platform | Windows only | Windows/Linux/macOS | Windows/Linux/macOS | Windows/Linux/macOS | Windows/Linux/macOS | Windows/Linux/macOS |
| Origin | National Instruments | TofuPilot | Keysight | Community | Everypin | |
| Measurements with limits | Yes | Yes (YAML validators) | Yes | Yes | Via plugins | Yes |
| Phase/step sequencing | Visual editor | YAML procedure | Python decorators | C# classes | Test functions | Test functions |
| Operator UI | Built-in OIs | Declarative, from the procedure | Station Server | Editor or custom panel | None built-in | Browser panel |
| Parallel phases | Yes | Yes (Rust engine) | No | Yes | Via xdist | No |
| Multi-slot fixtures | Yes | Yes | Limited | Yes | Manual | One instance per stand |
| Instrument plugins | NI drivers | Plugs (plain classes) | Plugs system | DUT/Instrument plugins | Fixtures | Fixtures |
| Deploy to stations | Manual | Git push, instant rollback | Manual | Deployment license | Manual | Manual |
| Data export | ATML, custom | JSON, dashboard | JSON, protobuf | CSV, SQLite, PostgreSQL listeners | JUnit XML | CouchDB, JSON |
| Version control | Difficult (binary) | Git-native | Git-native | Git-native | Git-native | Git-native |
| Community | Large (NI forums) | Newer, smaller | Small (GitHub) | Small (forum) | Large (general pytest) | Small |
TofuPilot Framework
The TofuPilot Framework is an open source test framework purpose-built for hardware, released under the MIT license. A procedure is a YAML file that declares the sequence and the limits; phases are plain Python functions and plugs are plain Python classes.
Strengths
- Declarative procedure. The test plan lives in YAML as data, readable and diffable separately from the logic.
- Parallel execution. A Rust engine runs independent phases and multiple fixture slots concurrently while your code stays plain Python. This is the one that cuts cycle time on a throughput-bound line.
- Operator UI from the procedure. Text inputs, checklists, images, sliders and progress bars are declared in the YAML, with no frontend work.
- Rich measurements. Aggregations (mean, min, max, std over arrays) and per-axis validators that check the shape of a curve, not just its endpoints.
- Deployment built in. Procedures deploy to stations on a Git push with instant rollback, and runs queue offline and sync when the network returns.
Weaknesses
- Newer project. Smaller community and fewer public examples than OpenHTF.
- Two files, not one. The YAML layer is an extra concept to learn if you want everything in Python.
Example
procedure.yaml24 lines
name: Supply Voltage Checkversion: 1.0.0unit: serial_number: default_value: "PCBA000001" part_number: default_value: "PCBA-100"plugs: - name: dmm python: plugs.dmm:Multimetermain: - name: Voltage Check python: phases.voltage_check measurements: - name: Supply Voltage unit: V validators: - operator: ">=" expected_value: 4.9 - operator: "<=" expected_value: 5.1def voltage_check(measurements, dmm): measurements.supply_voltage = dmm.measure_voltage()class Multimeter: def measure_voltage(self) -> float: return 5.01 # Replace with a real instrument readRun it with tofupilot run ./procedure.yaml, which executes on your machine without an account.
OpenHTF
OpenHTF is Google's open source test framework for manufacturing. It was built for testing consumer electronics at scale.
Strengths
- Python-native. Tests are Python functions decorated with measurements. No visual editor, no XML, no binary files.
- Plug system. Instruments are injected into test phases automatically. Plugs handle initialization and teardown.
- Measurement validators. Built-in support for ranges, tolerances, regex, and exact match.
- Station Server. Web-based operator interface at
localhost:12000. - Proven at scale. Years of production use, and a larger body of public examples than newer options.
Weaknesses
- Documentation. Official docs from Google don't exist. The community (via openhtf.com) fills the gap.
- Community size. ~640 GitHub stars, low Stack Overflow presence.
- Sequential execution. Phases run in order against one unit. Parallel phases and multi-slot fixtures are orchestrated outside the framework.
- Built-in UI. The Station Server is functional but basic.
Example
import openhtf as htffrom openhtf.util import units@htf.measures( htf.Measurement("voltage_V") .in_range(minimum=4.9, maximum=5.1) .with_units(units.VOLT),)def phase_voltage_check(test): """Measure and validate supply voltage.""" test.measurements.voltage_V = 5.01test = htf.Test(phase_voltage_check)test.execute(test_start=lambda: input("Scan serial: "))Existing OpenHTF suites run natively on the TofuPilot CLI with no rewrite, so choosing a platform does not force a framework migration.
OpenTAP
OpenTAP is Keysight's open source test automation framework. It's written in C# and runs on .NET.
Strengths
- Keysight backing. Active development, commercial support available through PathWave Test Automation.
- Plugin ecosystem. DUT, instrument, and result listener plugins.
- Test plan editor. GUI for building test sequences without code.
- Cross-platform. Runs on Windows, Linux and macOS via .NET.
Weaknesses
- C#/.NET. Most test engineers prefer Python. Python steps go through a Python.NET plugin.
- Editor and support terms. Keysight's GUI editor is free under a Community License that requires an account, an internet connection and opt-in telemetry. Support and the analysis tools come with the paid PathWave edition.
- Learning curve. Plugin architecture is powerful but complex for simple tests.
Example
[Display("Voltage Check", Description = "Measure supply voltage")]public class VoltageCheck : TestStep{ [Display("Voltage")] [Unit("V")] public double Voltage { get; set; } public override void Run() { Voltage = 5.01; if (Voltage < 4.9 || Voltage > 5.1) UpgradeVerdict(Verdict.Fail); else UpgradeVerdict(Verdict.Pass); }}pytest with Hardware Plugins
pytest is the standard Python testing framework. With the right plugins, it can work for hardware testing.
Strengths
- Massive community. Most Python developers already know pytest.
- Fixture system. Handles instrument setup/teardown cleanly.
- Plugin ecosystem. Thousands of plugins for reporting, parallel execution, and CI integration.
- Low learning curve. If you know Python, you know pytest.
Weaknesses
- Not built for manufacturing. No built-in measurement system with limits, units, or validators.
- No operator UI. No prompts, no serial number input, no pass/fail display.
- No native data logging. Requires custom code or third-party tools to store results.
- Designed for software testing. Concepts like fixtures, marks, and parametrize don't map cleanly to hardware test sequences.
Example
def test_voltage_check(dmm_fixture): """Measure and validate supply voltage.""" voltage = dmm_fixture.measure_voltage() assert 4.9 <= voltage <= 5.1, f"Voltage {voltage}V out of range"HardPy
HardPy, from Everypin, adds a browser-based operator panel and result storage to pytest for hardware testing.
Strengths
- pytest-native. Tests are regular pytest functions with hardware-specific additions.
- Measurements with limits.
NumericMeasurementrecords a value with its unit, comparison operator and limits. - Browser UI. Operator panel with test hierarchy, dialogs, data entry and images, usable from a tablet.
- Measurement storage. Built-in CouchDB integration for test results, or JSON files.
Weaknesses
- No analytics in the library. Yield, traceability and fleet monitoring require StandCloud, Everypin's paid hosted service.
- CouchDB dependency. The default storage runs in Docker, which adds infrastructure on each station.
- GPL-3.0. The copyleft license may matter to companies that embed it.
Example
import pytestfrom hardpy import ComparisonOperation as CompOp, NumericMeasurement, set_case_measurement@pytest.mark.case_name("Supply voltage")def test_voltage_check(dmm): meas = NumericMeasurement( value=dmm.measure_voltage(), unit="V", operation=CompOp.GELE, lower_limit=4.9, upper_limit=5.1, ) set_case_measurement(meas) assert meas.resultWhen to Use Each
| Situation | Recommendation |
|---|---|
| Starting a new production test system in Python | TofuPilot Framework |
| Throughput-bound line, several boards per fixture | TofuPilot Framework (multi-slot) |
| Operators need guided workflows on the station | TofuPilot Framework (declarative UI) |
| Validating waveform or sweep shape | TofuPilot Framework (per-axis validators) |
| Already running OpenHTF in production | Keep OpenHTF; it runs natively on the TofuPilot CLI |
| Want one pure-Python paradigm, no YAML layer | OpenHTF |
| Already using pytest for hardware tests | Run the suite on the TofuPilot CLI for data logging and operator UI |
| .NET shop with Keysight instruments | OpenTAP |
| Need a simple operator panel on top of pytest | HardPy |
| Migrating from TestStand, want minimal change | OpenTAP (closest architecture) |
| Migrating from TestStand, want Python | TofuPilot Framework |
| Budget is zero, need it working today | TofuPilot Framework (tofupilot run, no account needed) |
| Need analytics, traceability, and operator UI | Any framework + TofuPilot |
Cost Comparison
| Solution | Software License | Per-Station Cost | Operator UI | Data Analytics |
|---|---|---|---|---|
| NI TestStand | $3-5K/seat | $3-5K | Included | NI InsightCM (extra) |
| TofuPilot Framework | Free (MIT) | $0 | Included, declarative | Included |
| OpenHTF + TofuPilot | Free | $0 | Included (all tiers) | Included |
| OpenTAP | Free (core), Community License for the editor | $0, or PathWave license | Editor, custom panel | Desktop tools in PathWave |
| pytest + TofuPilot | Free | $0 | Via TofuPilot | Included |
| HardPy | Free (GPL-3.0) | $0 | Included | StandCloud, per device |
The open source alternatives eliminate per-seat licensing. The total cost of ownership depends on engineering time to set up and maintain the system, which is where the operator UI and the deployment story matter as much as the test logic.
Detailed Comparisons
Each tool compared with TofuPilot in depth:
| Tool | Guide |
|---|---|
| OpenHTF | TofuPilot Framework vs OpenHTF |
| HardPy | HardPy vs TofuPilot |
| OpenTAP | OpenTAP vs TofuPilot |
| Robot Framework | Robot Framework vs TofuPilot for Hardware Test |
| Exclave | Exclave vs TofuPilot |
| Flojoy Studio | Flojoy Studio Alternatives |
| yieldHUB | yieldHUB vs TofuPilot |
| NI TestStand | TestStand Alternatives |
| WATS | WATS Alternatives for Test Data Management |
