Deploy Test Scripts to Stations with Git

Getting a test script onto production stations reliably is one of the least discussed problems in manufacturing test. The common approaches, copying files over SSH, a network share everyone reads from, or a USB stick, all share the same weakness: nobody can say with certainty which station is running which version.

The advice engineers usually converge on is to keep a stable configuration in Git and have stations check it out. That instinct is correct, and it is what the deployment model below automates.

Why manual deployment goes wrong

The failure modes are consistent across teams:

• Version drift. Station 3 was offline during the last rollout and is still running last month's limits. Nobody notices until its yield diverges.
• Partial rollouts. A copy loop fails halfway. Some stations updated, some did not, and the script does not report which.
• No rollback. A bad release reaches every station simultaneously and reverting means repeating the whole process under pressure.
• Untracked local edits. Someone fixed something directly on a station. That fix exists nowhere else and disappears at the next deployment.
• Python version mismatch. The script works on the development machine and fails on a station with a different interpreter.

The underlying problem is that the deployed artifact is not identifiable. If you cannot name exactly what is running, you cannot reason about it.

Deploying from a Git push

Connect a repository to a procedure, and every push to a tracked branch builds a deployment. Pushes to the production branch roll out to every linked station; pushes to other branches produce preview deployments you can pin to a single station for testing.

GitHub, GitLab and Bitbucket Data Center are supported.

A procedure is a .yaml file declaring the sequence, the measurements and their limits. Phases are plain Python functions, and plugs connect instruments as Python classes.

name: PCBA Functional Testunit:  serial_number:    default_value: "PCBA000001"  part_number:    default_value: "PCBA-200"plugs:  - name: Multimeter    python: plugs.dmm:Multimetermain:  - name: Power Rail Test    python: phases.power_rail    measurements:      - name: Rail 3V3        unit: V        validators:          - operator: ">="            expected_value: 3.2          - operator: "<="            expected_value: 3.4

def power_rail(measurements, multimeter):    measurements.rail_3v3 = multimeter.read_voltage()

Because the limits live in the YAML rather than in Python, tightening a limit is a one-line diff that reviews cleanly and is visible in the deployment history.

Commit, push, and the stations pick it up:

git add procedure.yaml phases/power_rail.pygit commit -m "Tighten 3V3 rail limits to 3.2-3.4V"git push origin main

The build produces an immutable artifact carrying the executable, the resolved runtime, and every input the station needs. A station running dep_abc123 always runs exactly that code. There is no ambiguity about what is on which station.

Rolling out gradually

For a change you want to validate before it reaches the whole floor, pin a preview deployment to one station:

git checkout -b tighter-limitsgit push origin tighter-limits# Pin the resulting preview deployment to one station from the dashboard

Run a shift on that station, compare the yield against the others, then merge to production when you are satisfied.

Rolling back

Because artifacts are immutable and retained, rollback is selecting a previous deployment rather than rebuilding anything. The station picks it up on its next check. This matters most at the moment you least want to be running a build.

Other ways to create a deployment

Git push is the usual path, but not the only one:

• Manual trigger from the procedure's Deployments page, by commit SHA or branch.
• CLI, which builds from your local working tree with or without a connected repo. Useful for a fix you have not committed yet.

Deploying without the platform

If you are not using TofuPilot's deployment, the same principles apply and you can build them yourself. The rest of this guide covers that path.

Pin your dependencies

Unpinned dependencies are the most common source of "works on my machine" failures on a station.

openhtf==1.6.1tofupilot[openhtf]==2.16.0pyserial==3.5numpy==1.26.4pyinstaller==6.10.0

#!/bin/bashpip install -r requirements.txtpip freeze > requirements.lock.txt  # capture transitive deps for audit

Use requirements.lock.txt for auditing, requirements.txt for installs. Never pin transitive dependencies manually.

Bundle into a single executable

PyInstaller packages the script and its dependencies into one file, which removes the station Python version from the equation.

OpenHTF uses dynamic imports that PyInstaller cannot detect, so the spec file needs explicit hidden imports:

# -*- mode: python ; coding: utf-8 -*-from PyInstaller.utils.hooks import collect_data_files, collect_submodulesblock_cipher = Noneopenhtf_datas = collect_data_files("openhtf")openhtf_hiddenimports = collect_submodules("openhtf")a = Analysis(    ["test_main.py"],    pathex=["."],    binaries=[],    datas=openhtf_datas + [        ("plugs/", "plugs/"),        ("config/", "config/"),        ("VERSION", "."),    ],    hiddenimports=openhtf_hiddenimports + [        "openhtf.output.callbacks",        "openhtf.output.callbacks.json_factory",        "openhtf.util.logs",        "tofupilot",        "tofupilot.openhtf",    ],    hookspath=[],    hooksconfig={},    runtime_hooks=[],    excludes=[],    cipher=block_cipher,    noarchive=False,)pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)exe = EXE(    pyz,    a.scripts,    a.binaries,    a.zipfiles,    a.datas,    [],    name="station_tests",    debug=False,    strip=False,    upx=False,    runtime_tmpdir=None,    console=True,)

#!/bin/bashset -eVERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")echo "$VERSION" > VERSIONecho "Building version $VERSION"pyinstaller station_tests.spec --clean --noconfirmmv dist/station_tests "dist/station_tests_v${VERSION}"rm VERSIONecho "Built: dist/station_tests_v${VERSION}"

PyInstaller troubleshooting:

Keep station identity out of the code

Each station needs its own identity and credentials. Never hardcode either.

#!/bin/bash# Place in /etc/profile.d/tofupilot.sh for persistent configurationexport TOFUPILOT_API_KEY="tp_live_xxxxxxxxxxxxxxxxxxxx"export TOFUPILOT_STATION_ID="station-floor-1-cell-3"export STATION_SERIAL_PORT="/dev/ttyUSB0"export STATION_BAUD_RATE="115200"

For stations where environment variables are impractical, fall back to a config file:

import jsonimport osfrom pathlib import Pathdef load_station_config() -> dict:    if os.environ.get("TOFUPILOT_API_KEY"):        return {            "api_key": os.environ["TOFUPILOT_API_KEY"],            "station_id": os.environ.get("TOFUPILOT_STATION_ID", "unknown"),            "serial_port": os.environ.get("STATION_SERIAL_PORT", "/dev/ttyUSB0"),            "baud_rate": int(os.environ.get("STATION_BAUD_RATE", "115200")),        }    config_path = Path("/etc/tofupilot/station.json")    if not config_path.exists():        raise FileNotFoundError(            f"Station config not found at {config_path}. "            "Set TOFUPILOT_API_KEY or create the config file."        )    with config_path.open() as f:        return json.load(f)

Push to stations

#!/bin/bashset -eVERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")EXECUTABLE="dist/station_tests_v${VERSION}"DEPLOY_PATH="/opt/tofupilot/station_tests"STATIONS=(    "operator@station-01.local"    "operator@station-02.local"    "operator@station-03.local")for STATION in "${STATIONS[@]}"; do    echo "Deploying v${VERSION} to ${STATION}..."    ssh "$STATION" "[ -f ${DEPLOY_PATH} ] && cp ${DEPLOY_PATH} ${DEPLOY_PATH}.bak || true"    scp "$EXECUTABLE" "${STATION}:${DEPLOY_PATH}"    ssh "$STATION" "chmod +x ${DEPLOY_PATH}"    ssh "$STATION" "${DEPLOY_PATH} --version 2>&1 | head -1"    echo "  Done: ${STATION}"doneecho "Deployed v${VERSION} to ${#STATIONS[@]} stations."

Note the .bak copy before overwriting. That is your rollback, and it only goes back one version.

Record the version on every run

Whichever method you use, record which version produced each result. Without it you cannot correlate a yield change with a code release.

The version is a property of the run, so record it where the rest of your run metadata lives. With OpenHTF, the test record's metadata carries it:

import importlib.metadataimport openhtf as htffrom tofupilot.openhtf import uploaddef get_version() -> str:    try:        return importlib.metadata.version("station-tests")    except importlib.metadata.PackageNotFoundError:        import os, sys        base = getattr(sys, "_MEIPASS", os.path.dirname(__file__))        with open(os.path.join(base, "VERSION")) as f:            return f.read().strip()def record_version(test):    test.metadata["software_version"] = get_version()def main():    test = htf.Test(        record_version,        phase_power_on,        phase_voltage_check,        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",        part_number="PCBA-200",    )    test.add_output_callbacks(upload())    test.execute(lambda: input("Enter serial number: ").strip())

With Git-push deployment you get this for free: the deployment identifier already names the exact artifact, so there is nothing to record by hand.

Comparing the options

The network share row is worth avoiding. It looks simplest and it is the only method where a station can silently pick up a half-written file mid-shift.

What to check after any rollout

1. Every station reports the expected version. If one does not, it did not update.
2. The first runs after rollout pass. A broken deployment is most visible in the first few units.
3. Yield after matches yield before. If limits changed deliberately, expect a step; if they did not, a step means something went wrong.

Deployment is one of the few places in production test where being able to undo quickly matters more than getting it right first time.