Offline Upload Queue

The offline upload queue is the local buffer the CLI uses to keep runs flowing when the network or the dashboard is unreachable. Every completed run is enqueued first and then uploaded, so a Station or CLI session never loses a run to a transient outage.

How it works

Every run produced by tofupilot run, the OpenHTF connector, or CLI is persisted to the local queue before upload. A background drain loop wakes every 5 seconds, picks up due entries, and calls runs.create plus any attachment uploads, and because the original execution timestamp is part of the queued payload, analytics stay accurate no matter how late the upload lands.

Where the queue lives

The queue lives in an redb database at ~/.tofupilot/state.redb, and attachments are written to a sibling directory that the queue row references by path. The CLI holds an exclusive file lock on the database, so two CLI processes cannot corrupt the queue.

Removing the directory wipes every queued run that has not uploaded, so be careful with cleanup. tofupilot uninstall --keep-data preserves it, and tofupilot uninstall deletes it.

Operator commands

When you need to inspect or steer the queue from the CLI, three commands cover the lifecycle.

tofupilot queue

tofupilot queue retry

The retry command un-parks every entry (including 4xx-parked rows) and drains immediately, so use it after fixing a payload or credential issue that parked the entry.

tofupilot queue drop <queue-id>
tofupilot queue drop --all

The drop command hard-deletes the entry and its on-disk attachments, and there is no recovery, so only drop entries you have inspected.

Claims and duplicate prevention

The drain loop and manual retries can race on the same entry, and the CLI guards against this with a process-wide claim set. An upload acquires the claim before calling runs.create and releases it on success, failure, or panic, so the loser no-ops silently and the same queued run never mints two dashboard rows.

Once runs.create succeeds, the server-issued run_id is persisted on the queue row, so a later retry (for example, when an attachment failed) skips the create call and resumes from the attachment step. Retrying never produces a duplicate run.

Station vs CLI behavior

The drain loop runs in both modes, but each surfaces queue progress differently.

Station mode runs the drain loop for the daemon's lifetime, and live progress events emit over the local WebSocket: RunUploadQueued, RunUploadStarted, RunUploadSucceeded, and RunUploadFailed. The Operator UI renders these in the Uploads panel.

CLI mode runs the same loop and also exposes the queue through /api/v1/queue/list, so the CLI reports page can correlate each queue row with the on-disk report.json that produced it. Double-clicking Upload on the same report returns the existing queue_id, so there is no duplicate enqueue.