Scale from 1 to 100 Test Stations

Run one station or a hundred: the architecture decisions you make at 10 stations determine whether you succeed at 100. This guide walks through the changes needed at each scale threshold.

Scaling Challenges by Stage

Architecture at Each Scale

1 Station

A single machine running your test script. No special infrastructure needed.

10 Stations

At 10 stations, the bottleneck is configuration. You need a shared config source, consistent naming, and a way to push script updates.

100 Stations

At 100, you need a deployment pipeline, centralized monitoring, and staggered upload scheduling.

Station Naming and Registration

Stations are identified by name. A collision corrupts per-station analytics.

Set the station name as an environment variable:

TOFUPILOT_STATION_ID=taipei-line1-001TOFUPILOT_API_KEY=tp_live_xxxxxxxxxxxx

Pick the scheme before the second station exists. Renaming later splits one station's history into two identities, and the old runs keep the old name forever.

Test Script Distribution

Option 1: Git Pull (1-20 stations)

0 6 * * * testuser cd /opt/testscripts && git pull origin main

Pros: Simple. Cons: Requires network access to Git host. Fails silently.

That silent failure is the real cost. A station that missed the pull keeps testing with old limits and reports nothing unusual, so you find out from diverging yield rather than from an error.

Option 2: PyInstaller Binary (20-50 stations)

#!/bin/bashpyinstaller --onefile test_main.py --name test_runnerfor station in taipei-line1-{001..020}; do  rsync -az dist/test_runner "${station}:/opt/testscripts/test_runner_new"  ssh "${station}" "mv /opt/testscripts/test_runner_new /opt/testscripts/test_runner"done

Pros: No Python on stations. Cons: Slower iteration, larger binary.

Note the write-then-rename: copying directly onto the running path can leave a station with a half-written binary. The loop also has no error handling, so a station that is offline is skipped without comment. Check the exit status per station if you use this.

Option 3: Docker (50-100 stations)

FROM python:3.11-slimWORKDIR /appCOPY requirements.txt .RUN pip install --no-cache-dir -r requirements.txtCOPY test_main.py .CMD ["python", "test_main.py"]

version: "3.9"services:  test_runner:    image: your-registry/test-runner:latest    restart: unless-stopped    environment:      - TOFUPILOT_STATION_ID      - TOFUPILOT_API_KEY    devices:      - /dev/ttyUSB0:/dev/ttyUSB0

Update all stations:

#!/bin/bashparallel-ssh -h stations.txt "docker compose pull && docker compose up -d"

Pin an explicit image tag rather than latest once you are past a handful of stations. With latest, two stations that pulled on different days run different code and nothing records which.

Distribution Method Comparison

If you deploy procedures from a Git push, this table is mostly moot: the build produces an immutable artifact per commit, every linked station picks it up, and rollback is selecting a previous deployment.

Centralized Configuration

Never hardcode values that vary per station or product.

voltage_rail_3v3:  min: 3.2  max: 3.4voltage_rail_5v:  min: 4.8  max: 5.2boot_time_ms:  min: 0  max: 3000

import yamlimport openhtf as htffrom openhtf.util import unitsfrom tofupilot.openhtf import uploadwith open("config/product_v2.yaml") as f:    cfg = yaml.safe_load(f)@htf.measures(    htf.Measurement("voltage_3v3").in_range(        cfg["voltage_rail_3v3"]["min"],        cfg["voltage_rail_3v3"]["max"]    ).with_units(units.VOLT))def phase_power_check(test):    voltage = read_voltage_rail("3v3")    test.measurements.voltage_3v3 = voltagedef main():    test = htf.Test(        phase_power_check,        procedure_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",  # procedure UUID from the dashboard        part_number="PCBA-100",    )    test.add_output_callbacks(upload())    test.execute(lambda: input("Scan serial number: "))if __name__ == "__main__":    main()

Version the config file alongside the script. Limits that change without a version marker make results from before and after the change silently incomparable.

Network Topology

Stations communicate only outbound. No inbound ports required.

At 100 stations, stagger uploads to avoid thundering herd:

import timeimport randomdef upload_with_jitter(upload_fn, max_jitter_seconds=10):    time.sleep(random.uniform(0, max_jitter_seconds))    upload_fn()

Jitter costs cycle time on every unit, so reach for it only if you actually see contention. Runs queue locally and sync when the network returns, which absorbs most bursts without help.

Monitoring Station Health

Per-station yield, throughput, and test duration are tracked automatically. Filter analytics by station to spot degradation.

The single most useful query at scale is failure rate grouped by station. If one station sits at 12% while the rest are at 2%, the problem is that station's fixture or instrument, not your product.

Station Health Checklist

Station Bootstrap Script

#!/bin/bashset -eSTATION_ID=$1API_KEY=$2if [ -z "$STATION_ID" ] || [ -z "$API_KEY" ]; then  echo "Usage: bootstrap.sh <station-id> <api-key>"  exit 1fiecho "TOFUPILOT_STATION_ID=${STATION_ID}" >> /etc/environmentecho "TOFUPILOT_API_KEY=${API_KEY}" >> /etc/environmentcurl -fsSL https://get.docker.com | shmkdir -p /opt/testrunnercat > /opt/testrunner/docker-compose.yml << EOFversion: "3.9"services:  test_runner:    image: your-registry/test-runner:latest    restart: unless-stopped    env_file: /etc/environmentEOFdocker compose -f /opt/testrunner/docker-compose.yml up -decho "Station ${STATION_ID} is running."

sudo bash bootstrap.sh taipei-line1-042 tp_live_xxxxxxxxxxxx

Two things to tighten before using this on a real floor. The API key is passed as a command-line argument, which puts it in shell history and in the process list; read it from a file or a prompt instead. And appending to /etc/environment on a re-run duplicates the entries rather than replacing them.