run --json Event Stream in TofuPilot
Last updated on October 28, 2026
tofupilot run --json emits a newline-delimited JSON (NDJSON) event stream on stdout. Each line is one event object with a type field and a seq counter. The stream is the integration surface for agents, MES bridges, and CI automation: everything the operator UI shows is also on the wire.
tofupilot run ./procedures/pcb-fvt --json{"type":"run_started","procedure_id":"pcb-fvt","protocol_version":"1.2","seq":0}
{"type":"plan","phases":[{"key":"flash","name":"Flash firmware"}],"seq":1}
{"type":"phase_started","phase_key":"flash","attempt":1,"slot_id":"default","started_at":"2026-06-10T08:00:00Z","seq":2}
{"type":"phase_finished","phase_key":"flash","outcome":"PASS","duration_ms":1240,"seq":3}
{"type":"run_finished","outcome":"PASS","exit_code":0,"seq":4}Protocol guarantees
- Versioned:
run_startedcarriesprotocol_version(currently1.2). New optional fields can appear without a version bump; ignore unknown fields. Removals and renames bump the version. - Ordering:
seqstarts at 0 and increments by 1 per line, across every run of a--repeatloop too. It is the only reliable ordering primitive; timestamps are advisory for display. - Terminal invariant:
run_finishedis always the last event of a run. With--repeat, each run is a stream of its own (run_startedandrun_finishedcarry its 1-basediteration), andrepeat_summaryfollows the last one. If the run could not start or the Python subprocess died,run_crashedimmediately precedes it. A stop while the run is set up ends it early: before the procedure is prepared (a venv build), asrun_crashedwithkind: stopped; after that and before theplan, asrun_finished(ABORTED) right afterrun_started, with nothing run. Stop sending stdin commands afterrun_finished. - stderr stays human: progress notes and warnings go to stderr; stdout carries only event lines.
Event reference
type | Purpose |
|---|---|
run_started | First event. Carries procedure_id and protocol_version. |
plan | Ordered phase list as {key, name, stage}, stage the YAML list it is in: setup, main or teardown (main for pytest, Robot Framework and OpenHTF). |
phase_started | Phase entered execution: phase_key, attempt, slot_id, started_at. |
phase_finished | Outcome (PASS, FAIL, ERROR, TIMEOUT, SKIP, RETRY for an attempt that asked phase.retry() (an attempt a then: … retry retries keeps its FAIL, ERROR or TIMEOUT; either way the next phase_started has the next attempt), or STOP for a phase a stop ended: its own phase.stop(), or a stop that cut it short; a phase that finishes its body while a stop is under way keeps its own outcome), forced (PASS or FAIL, only when --force gave the result and the phase never ran), timestamps, duration_ms, optional error (a traceback starts from the phase's own code) with, when it is a traceback, error_type (the exception class, OSError) and error_message (its message), reason on a SKIP that started but did not run its body (Not run: plug psu did not open: …, a teardown phase whose plug never opened), and run_metadata when the phase set any (run.metadata[...], values as text; later phases overwrite a key). With --locals, when the phase raised, locals lists the variables of its own function at that point ([{name, value}], values shortened, secret-looking values masked, never uploaded). |
phase_skipped | Phase never ran, once per phase, with its reason: a plug that could not start, a stop, on_first_failure: stop, left out with --skip, or outside a --phase run. |
sequence_finished | A sequence call settled, once per call and slot, after its phases' phase_finished or phase_skipped: key (the call's path, which prefixes its phases' phase_key: calibration, then calibration/zero_cal for a call inside it), name, source (the uses: as written), outcome (PASS, FAIL, ERROR, TIMEOUT, ABORTED when a stop cut it, SKIP when none of its phases ran), slot_id, started_at and ended_at (started_at and ended_at absent when none of its phases ran). |
phase_log | Live log line from a running phase. level is upper case (INFO, WARNING, ERROR…); DEBUG lines come only with -v, as on the console (the run log and the dashboard keep them). |
ui_request | Operator prompt with the component specs, and the slot_id of the slot asking on a multi-slot run (the same prompt opens once per slot, each with its own request_id). Reply with ui_response. |
ui_auto_continue | Prompt auto-resolved (display-only components or pre-baked --ui-values), with its slot_id on a multi-slot run. |
ui_update | A phase changed an open prompt: action is set_value (data: {id, value}, from ui.<key> = value) or set_properties (data: {id, properties}, from ui.set). Fold data into the ui_request with that request_id. data.properties carries only the fields a ui_request component shows you (label, description, placeholder, options as value and label, min, max, step), null for a cleared one. Options set at run time are the list your ui_response is checked against. request_id is absent on a display-only prompt, which has nothing to answer, and once the prompt it belonged to has been answered (a phase may keep updating its screen after your ui_response). |
ui_timeout | Prompt expired without a response (--ui-timeout). |
ui_cancelled | The prompt's phase was interrupted (a stop, or OpenHTF's own phase timeout) before anyone answered. The request is withdrawn, a late ui_response to it is refused as unknown. OpenHTF only. |
ui_error | A stdin command was rejected; reason enumerates the cause. |
identify_request | Unit identification prompt (serial number, part number). |
identify_resolved | Identification accepted. |
identify_timeout | Identification expired (--ui-timeout). |
plug_status | Plug lifecycle transition: initializing, active or error when it opens, destructing then idle when it closes. skipped with stage: setup: it never opened (a sibling plug failed first, or the run was stopped while it opened); with stage: teardown: a forced stop killed it without its cleanup (its __del__ did not run). Carries scope (slot, execution, or station) and slot_id (absent for execution- and station-scope plugs, which are shared across slots). With --repeat, a station plug stays open from one run to the next and closes after the loop's last run_finished, with no plug_status for its closing. |
plug_log | Live log line from a plug's Python service, with the same level rules as phase_log. A line logged through logging carries file and line, as a phase's does. |
measurement_recorded | Live measurement write, from the phase's code or from an operator answer a bind: measurements.<key> records (sent when the phase ends). value is the value as recorded (5.0, "ok", a list). outcome is its verdict at record time against the declared limits (PASS, FAIL or UNSET), with low and high when the limits are numeric, and low_exclusive / high_exclusive set to true when that end is strict (> / <: a value equal to it fails). YAML procedures are judged live; pytest, Robot Framework and OpenHTF send UNSET and judge at phase end. The final verdicts are in phase_finished and run_summary. |
attachment_added | Phase attached a file. |
state_snapshot | Reply to get_state: run_status, phase history, any in-flight active_ui_request. |
run_upload_queued | Run persisted to the local queue for deferred upload. |
run_upload_started / run_upload_succeeded / run_upload_failed / run_upload_dropped | Upload lifecycle for queued runs. run_upload_succeeded carries the final run_id and dashboard_url. |
internal_warning | Non-fatal CLI anomaly, for example payload truncation: kind, message, optional detail. Kind troubleshooting for what --skip or --force changes and the run still allows (a phase reading a forced result); ignored_flag for a flag this procedure type does not use (--locals on pytest). |
run_crashed | The run failed to start or its subprocess died; immediately precedes run_finished. Carries kind, message, exit_code, and stderr_tail. |
run_summary | YAML procedures, just before run_finished (absent after run_crashed): runs, one entry per slot run with its outcome, failed_phases (phase keys of FAIL, ERROR and TIMEOUT phases; a phase whose asked-for retry was cancelled reads RETRY and is not one) and stopped_phases (the phases that asked for the stop that ended the run: phase.stop(), then: pass: stop), each phase's key, name, outcome, forced when --force set it, interrupted: true when a stop it did not ask for cut it short (the record reads FAIL, it is in neither list), and measurements with their value, outcome, validators, low/high and unit (the names measurement_recorded uses), and skipped_phases (key, reason) for what never ran. Built from the run record the upload sends. |
run_finished | Last event: outcome and exit_code, and stopped_by (SIGTERM, SIGHUP, Ctrl-C, abort_run, TUI or operator UI) when something asked the run to stop. |
repeat_summary | With --repeat only, after the last run's run_finished, with its own seq: run_count, outcomes (count per outcome), pass_rate and first_non_pass (1-based run number; both leave out a run cut short by a stop, as the measurement statistics do, and pass_rate is null when no run finished), stopped (ended by Ctrl-C, a signal or the operator; a procedure stopping itself is one run's ABORTED and the loop goes on), iterations (each run's outcome, the last 1000; iterations_omitted counts the rest) and per-measurement slot, phase_key, measurement_key, a display measurement label, count, min, max, mean, stdev. |
Large payloads are capped: measurement_recorded.value truncates above 1 MB (replaced by {"truncated": true, "original_size_bytes": N} plus an internal_warning).
Crashes and pre-run failures
run_crashed.kind is a stable key for the reason, message the one-line summary a person would read, stderr_tail the end of the subprocess's stderr. A run that fails before the procedure starts (path not found, not linked, nothing pulled) still writes run_started, run_crashed, and run_finished, so stdout is never empty. So does a stop at any time after the process has set up its signal handling (its first few milliseconds); with --repeat, a stop before the first run writes only repeat_summary (run_count: 0).
{"type":"run_started","procedure_id":"pcb-fvt","protocol_version":"1.2","execution_id":"…","seq":0}
{"type":"run_crashed","exit_code":5,"kind":"not_linked","message":"Not linked: ./pcb-fvt; run `tofupilot link ./pcb-fvt` or set TOFUPILOT_PROCEDURE_ID","stderr_tail":"…","seq":1}
{"type":"run_finished","outcome":"ERROR","exit_code":5,"seq":2}kind | Cause |
|---|---|
load_error | Path not found, no procedure in the directory, or invalid procedure.yaml |
env_error | The venv cannot be created or used |
not_linked | --upload on a directory with no linked procedure |
no_deployment | --deployment names nothing pulled |
ambiguous_deployment | Several deployments are pulled and none is named, with no terminal to pick on |
agent_init_error | The --ui-values file stopped loading after the run's start check (removed or changed between two runs of a --repeat loop). A bad file on the command line is a usage error (exit 2) |
spawn_failed | The Python process could not start |
subprocess_crash | The Python process exited before the run completed |
startup_stall | No phase started in time |
identify_unit_failed | Unit identification failed |
init_error | A procedure.yaml run could not start its workers or plugs |
submit_error | A procedure.yaml run could not schedule its phases |
execution_error | A procedure.yaml run failed outside any phase |
stopped | Stopped before the procedure reported a result |
not_configured | Required config or secrets have no value. Nothing ran; exit code 5. The event also carries missing, a list of {"kind": "config"|"secret", "key": ...} |
New kinds can appear; treat an unknown one as a generic crash.
Exit codes
run's exit code is the run's outcome. Errors outside a run stream (a usage error, missing credentials before the run starts, and every other command that takes --json) print one error object on stdout under --json. code is the stable key, action and status are optional.
{"type":"error","code":"not_found","message":"No queued run q_123","action":"run `tofupilot queue ls`"}| Code | Meaning |
|---|---|
| 0 | Success; for run, outcome PASS |
| 1 | General error; for run, outcome FAIL or TIMEOUT (run_finished.outcome tells them apart) |
| 2 | Usage: invalid flag value, or a choice that needs a terminal (--deployment, --yes); for run, a refused --phase, --skip, --force or --plug, or a --ui-values file that is missing or not valid JSON, reported before the run starts as an error object |
| 3 | run only: the procedure stopped itself |
| 4 | Not found (queue ID, deployment ID, HTTP 404, a PATH validate or plug cannot find); never from run |
| 5 | run only: outcome ERROR, or a run that could not start (nothing was tested) |
| 6 | Not logged in or credentials rejected; never from run |
| 126, 127 | env run only: the command could not be run (126) or was not found (127), as in a shell; otherwise env run exits with the command's own code |
| 130 | Stopped by Ctrl-C, abort_run, the operator (TUI, kiosk) or a console close; 143 for SIGTERM and 129 for SIGHUP (128 + the signal, as a shell reports it). A run that had already ended FAIL, TIMEOUT or ERROR when the stop landed (its teardown still running) keeps that code; one whose main phases all passed reads ABORTED (130), even when its teardown then completes: the stop landed before the run finished. A stop while main phases still run ends the run ABORTED (130) even after a FAIL: the failed phase keeps its FAIL in phase_finished and the record, but the run did not finish. The exit code of a single run is always the one run_finished carries; with --repeat, a stopped loop exits as the stop (130, 143, 129) whatever its last run's run_finished says, and otherwise as its first run that did not pass |
Stdin commands
The CLI reads NDJSON commands on stdin while the run is active:
| Command | Purpose |
|---|---|
ui_response | Answer a ui_request by request_id with component values. Answers are checked by the same validator as the kiosk (required fields, bounds, step, pattern, selections against the current option list) and refused with ui_error (reason: "invalid_value", the field and the message) while the prompt stays open. |
abort_run | Stop the run like a first Ctrl-C: the running phases end STOP and the teardown still runs. A second abort_run forces the stop at once, teardown included: the teardown phases are cut and each plug gets up to 4 s to run its __del__ (plug_status destructing then idle) before its process is killed (skipped, stage teardown). A second Ctrl-C or signal does the same, but one within 3 s of the first is ignored (a reflex double press must not cut a power-off); without --json the console says when the next one counts. A third one, or 10 s after the forced stop, exits the process at once: no plug cleanup and no run_finished, the way out of a run that is stuck even past the forced stop. Rejected with ui_error (reason: "invalid_state") before run_started or after run_finished. |
get_state | Request a state_snapshot to recover after reconnecting mid-run. |
{"type":"ui_response","request_id":"<id from ui_request>","values":{"operator_note":"ok"}}Headless runs
Prompts block the stream until answered. For unattended runs:
- Set
unit.auto_identify: truewithserial_number.default_valueandpart_number.default_valuein the procedure.yamlfile, or give every unit field adefault_valuewithlocked: true, so the identify prompt resolves itself. - Pass
--ui-values <file>to pre-bake answers for UI phases. An empty map{}resolves display-only and acknowledgement prompts. A phase that callsui.wait()gets the file values for its first prompt only. - Pass
--ui-timeout <seconds>so an unanswerable prompt fails the phase instead of hanging forever.
tofupilot run ./procedures/pcb-fvt --json --ui-values ui.json --ui-timeout 300How is this guide?