Kubernetes
Last updated on November 20, 2026
The tofupilot Helm chart runs a self-hosted TofuPilot instance on your Kubernetes cluster. It runs the same services and images as the Docker Compose install, with your cluster's ingress controller in place of the bundled reverse proxy. One release is one instance, and one instance serves one organization.
What the chart runs
| Component | Kubernetes object | Role |
|---|---|---|
| Dashboard | Deployment, 1 replica | TofuPilot application and API. Backs up and migrates the database when a new version starts. |
| PostgreSQL | StatefulSet | Database. Optional: you can use your own. |
| File storage | StatefulSet | SeaweedFS, S3-compatible, signed requests only. Attachments, uploaded files and build artifacts. |
| Realtime | Deployment | Centrifugo WebSocket server: live station status, telemetry, build logs. |
| Build worker | Deployment, privileged | Builds Deployments in its own Docker daemon. Optional. |
| Bucket job | Job, after each install and upgrade | Creates the two storage buckets. |
Resource names start with <fullname>: the release name, followed by -tofupilot unless the release name already contains it. With the release name tofupilot used on this page, <fullname> is tofupilot: the dashboard Deployment is tofupilot-dashboard and the database pod is tofupilot-pg-0.
Requirements
| Requirement | Details |
|---|---|
| Kubernetes | 1.27 or later, linux/amd64 nodes |
| Helm | 3.14 or later |
| Ingress controller | Any. Tested with the official Traefik chart. See Ingress. |
| TLS | A certificate for three hostnames, or cert-manager to issue them |
| Storage | A default StorageClass with ReadWriteOnce volumes: 20 GB for the database, 50 GB for files, 10 GB for backups, 20 GB for the build cache (all configurable) |
| CNI | One that enforces NetworkPolicy (Cilium, Calico, ...). Otherwise set networkPolicy.enabled: false. |
| Pod Security | The build worker needs a privileged pod. If your policy forbids it, disable the build worker. |
| Sign-in | SMTP or an OAuth provider, or nobody can create the first account |
| License | A license key from TofuPilot Orbit |
You also need three hostnames pointing at your ingress controller: one for the app, one for file storage and one for realtime, for example tofupilot.example.com, storage.tofupilot.example.com and realtime.tofupilot.example.com. Browsers and stations reach all three.
Install
Get registry credentials
The chart and its images are in TofuPilot's registry, ghcr.io/tofupilot. Exchange your license key for read credentials:
curl -s -X POST https://www.tofupilot.com/api/docker/token \-H 'Content-Type: application/json' \-d '{"licenseKey":"lk_..."}'The response holds a username and a token. Log Helm in with them:
helm registry login ghcr.io --username <username> --password <token>Create the namespace and its secrets
Keep credentials out of your values file. Create the namespace, the image pull secret, and a secret for your sign-in credentials:
kubectl create namespace tofupilotkubectl -n tofupilot create secret docker-registry tofupilot-registry \--docker-server=ghcr.io --docker-username=<username> --docker-password=<token>kubectl -n tofupilot create secret generic tofupilot-auth \--from-literal=EMAIL_SMTP_PASSWORD=<smtp-password>tofupilot-auth holds the secret half of each sign-in method you use: EMAIL_SMTP_PASSWORD, AUTH_GOOGLE_SECRET, AUTH_MICROSOFT_ENTRA_ID_SECRET, AUTH_GITHUB_SECRET, AUTH_GITLAB_SECRET or AUTH_ATLASSIAN_SECRET. The variables are the same as in the Docker Compose configuration.
Write your values file
This example uses Traefik and cert-manager. Adapt the hostnames, the ingress class, the issuer and the sign-in settings:
domains:app: tofupilot.example.comstorage: storage.tofupilot.example.comrealtime: realtime.tofupilot.example.comimagePullSecrets:- name: tofupilot-registryingress:className: traefikannotations: cert-manager.io/cluster-issuer: letsencryptnetworkPolicy:ingressNamespaceSelector: matchLabels: kubernetes.io/metadata.name: traefik# Sign-in: SMTP here. Its password is in the tofupilot-auth secret.env:EMAIL_SMTP_HOST: smtp.example.comEMAIL_SMTP_PORT: "587"EMAIL_SMTP_USER: noreply@example.comEMAIL_FROM_AUTH: noreply@example.comextraEnvFrom:- secretRef: name: tofupilot-authSave it as values.yaml. Every option is documented in the chart's own values:
helm show values oci://ghcr.io/tofupilot/charts/tofupilotInstall the chart
helm install tofupilot oci://ghcr.io/tofupilot/charts/tofupilot \--namespace tofupilot -f values.yaml \--set licenseKey=lk_... \--wait --timeout 65mThe first start applies every database migration, which takes a few minutes. Follow it with:
kubectl -n tofupilot rollout status deployment/tofupilot-dashboardPin a chart version with --version. Each chart version defaults to one TofuPilot release; set image.tag to run another one. The build worker needs 2.52.9 or later on Kubernetes: earlier images cannot start build containers there.
Do not pass --atomic (or --rollback-on-failure). If the migrations outlast the timeout, Helm would roll back to the previous version on a database that is already migrated. With --wait, give --timeout at least dashboard.startupSeconds, which defaults to one hour.
Create the first account
Open https://tofupilot.example.com and sign in with the method you configured. The first account becomes the organization admin.
If the sign-in page only offers single sign-on, no sign-in method is configured: check the env and extraEnvFrom settings. The install notes warn about it when env sets none.
Point your stations at the instance
Stations, scripts and the CLI use the app hostname, the same way as a Docker Compose instance. See Upload runs.
Ingress
The chart creates one Ingress per hostname. ingress.annotations applies to all three, and ingress.app.annotations, ingress.storage.annotations and ingress.realtime.annotations to one each. On the same key, the per-host value wins.
Two kinds of traffic need more than the usual defaults:
- Uploads go straight from browsers and stations to the storage hostname, through presigned links: attachments up to 95 MB, and deployment bundles. The request body size limit and the request read timeout must allow them over your slowest link.
- Realtime keeps a WebSocket open for as long as the dashboard or a station is connected. The idle timeout must allow it.
| Controller | What to set |
|---|---|
| Traefik (v3) | No body size limit. Each request has 60 s to arrive, so a slow upload fails with 502 Bad Gateway. Raise it on the Traefik chart: --set ports.websecure.transport.respondingTimeouts.readTimeout=600s. Measured: a 95 MB upload at 500 KB/s got a 502 at 60 s, and completed in 195 s with that setting. Idle WebSockets are not affected. |
| ingress-nginx | The chart's defaults lift the 1 MB body limit and the 60 s WebSocket timeout. ingress-nginx is retired upstream: prefer another controller for a new cluster. |
| GKE Ingress | Attach a BackendConfig with timeoutSec: 3600 to the realtime Service. |
| AWS Load Balancer Controller | Set alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=3600. |
TLS
Each Ingress reads its certificate from a secret named <fullname>-tls-app, -storage and -realtime. A cert-manager issuer annotation, as in the example values, creates the three.
To use one wildcard or multi-domain certificate instead, create its secret, set ingress.tls.existingSecret to its name, and remove the issuer annotation.
If the certificate comes from a private CA, stations must trust it. See Private CA, and publish the bundle with the chart's privateCa values.
Network policies
The chart's NetworkPolicies are on by default. They give the cluster the isolation the Docker Compose install gets from its private network:
- PostgreSQL accepts connections from the dashboard and the build worker only.
- The build worker, which runs your procedure code, reaches the stack's own services, DNS and the internet. It cannot reach link-local addresses, which covers every major cloud's metadata endpoint, nor
100.100.100.200. It has no IPv6 egress. - The dashboard, storage and realtime accept traffic from your ingress controller's namespace and from the components that call them, once
networkPolicy.ingressNamespaceSelectoris set: the dashboard from realtime and the build worker, storage from the dashboard, the build worker and the bucket job, realtime from the dashboard and the build worker.
Set networkPolicy.ingressNamespaceSelector to your ingress controller's namespace, as in the example values. The build worker reaches its own dashboard through the public hostname, and many clusters route it straight to the controller's pods. Without the selector, builds can fail on those clusters. networkPolicy.ingressControllerPorts lists the ports those pods listen on; the default covers ingress-nginx (80, 443) and Traefik (8000, 8443).
The policies are tested on Cilium, with Traefik in its own namespace. On other setups:
| Setup | What to change |
|---|---|
| Calico | Add your pod CIDR to networkPolicy.deployerEgressExcept so builds cannot reach other pods. |
| OpenShift | Set networkPolicy.dns to the dns-default pods on port 5353. |
| NodeLocal DNSCache | Allowed by default (169.254.20.10). |
| Host-network controllers, AWS ALB in IP mode, GKE NEG | Leave ingressNamespaceSelector empty: their traffic does not come from a namespace. |
| Git server inside the cluster | Allow it in networkPolicy.deployerExtraEgress: builds clone from it. |
Database and secrets
Bring your own PostgreSQL
Set postgresql.enabled: false and give the connection URL, either directly in postgresql.externalUrl or in a secret named by postgresql.externalUrlSecret, under the key DATABASE_URL. The database must run PostgreSQL 17 with pg_stat_statements in shared_preload_libraries, and its user needs the CREATE privilege on the database. If it runs inside the cluster, allow the build worker to reach it in networkPolicy.deployerExtraEgress: the network policies only open the bundled database.
Generated secrets
On the first install, the chart generates the instance secrets (session, database, storage, realtime and encryption keys) into the secret <fullname>-secrets, and reads them back on every upgrade. The secret is kept when you uninstall the release.
Back up <fullname>-secrets with the database. SECRETS_MASTER_KEY encrypts every stored config value and secret: without it they are unreadable.
Argo CD and other tools that render with helm template cannot read the secret back, and would generate new values on every sync. Use existingSecret with them: create the secret yourself, with the keys listed in the chart's README (helm show readme oci://ghcr.io/tofupilot/charts/tofupilot). Flux runs a real Helm install and needs nothing special.
Upgrade
Upgrade with your values file:
helm upgrade tofupilot oci://ghcr.io/tofupilot/charts/tofupilot \--namespace tofupilot -f values.yaml --version <chart-version>Or reuse the values stored in the release, with the new chart's defaults:
helm upgrade tofupilot oci://ghcr.io/tofupilot/charts/tofupilot \--namespace tofupilot --reset-then-reuse-values --version <chart-version>Do not use --reuse-values. It keeps the previous chart's defaults, security fixes included: an upgrade that way once kept an older, wider network rule for the build worker.
When a new version starts, the dashboard dumps the database, then migrates it. Downgrades are refused, because an older version cannot run on a migrated database. To go back, restore the dump instead (see Back up and restore). If an upgrade went to a version that never started (wrong tag, image pull error), go back with helm rollback.
A change to a secret listed in extraEnvFrom does not restart the dashboard. Restart it yourself:
kubectl -n tofupilot rollout restart deployment/tofupilot-dashboardBack up and restore
Back up on a regular schedule:
| What | Where |
|---|---|
| Database | The volume data-<fullname>-pg-0, or your own PostgreSQL |
| Files | The volume data-<fullname>-storage-0 |
| Instance secrets | The secret <fullname>-secrets |
Each time a new version starts, the dashboard also dumps the database into the <fullname>-backups volume, mounted at /backups in its pod. The naming, the 7-day retention and REQUIRE_BACKUP are the same as on Docker Compose: see Automatic pre-migration dumps.
To restore a dump named <date>_<time>_pre-v<new>_from-v<old>.sql.gz, run the version it came from, <old>. For a from-unknown or older-style dump, find <old> in the APP VERSION column of helm history tofupilot -n tofupilot. With your own PostgreSQL, run steps 2 and 3 against your server.
Copy the dump out of the dashboard pod:
kubectl -n tofupilot exec deploy/tofupilot-dashboard -- ls -l /backupsPOD=$(kubectl -n tofupilot get pod -l app.kubernetes.io/component=dashboard -o name | cut -d/ -f2)kubectl -n tofupilot cp "$POD:/backups/<file>.sql.gz" ./<file>.sql.gzStop the dashboard and the build worker (leave the build worker out if you disabled it), then replace the database with an empty one:
kubectl -n tofupilot scale deployment/tofupilot-dashboard deployment/tofupilot-deployer --replicas=0kubectl -n tofupilot exec tofupilot-pg-0 -- dropdb -U tofupilot tofupilotkubectl -n tofupilot exec tofupilot-pg-0 -- createdb -U tofupilot tofupilotLoad the dump:
gunzip -c <file>.sql.gz | kubectl -n tofupilot exec -i tofupilot-pg-0 -- psql -U tofupilot -d tofupilot -v ON_ERROR_STOP=1Start the version the dump came from. image.allowDowngrade lets this one upgrade go back, and Helm scales both deployments back up:
helm upgrade tofupilot oci://ghcr.io/tofupilot/charts/tofupilot \--namespace tofupilot --reset-then-reuse-values \--set image.tag=<old> --set image.allowDowngrade=true--reset-then-reuse-values keeps both values on every later upgrade. Clear them on the next one, or it stays on <old> with the downgrade guard off:
helm upgrade tofupilot oci://ghcr.io/tofupilot/charts/tofupilot \--namespace tofupilot --reset-then-reuse-values --version <chart-version> \--set image.tag=null --set image.allowDowngrade=falseWith a values file, run the restore with -f values.yaml instead: the next -f values.yaml upgrade drops both.
Build worker
The build worker builds deployment bundles in its own Docker daemon, so its pod is privileged. A container escape from it is root on its node: give it a dedicated node pool with deployer.nodeSelector and deployer.tolerations.
If your Pod Security policy forbids privileged pods, set deployer.enabled: false. Everything else keeps working; only the Deployments build pipeline is unavailable.
Uninstall
helm uninstall tofupilot --namespace tofupilotThe secret <fullname>-secrets, the backups volume and the database and file volumes are kept, so a new install with the same release name finds its data again. To remove everything, delete the namespace after the uninstall:
kubectl delete namespace tofupilotDeleting the namespace deletes the database, the uploaded files and the instance secrets. It cannot be undone.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
Pods stuck in ImagePullBackOff | The pull secret is missing or the token is wrong. Check imagePullSecrets and get a new token from your license. |
| The sign-in page only offers single sign-on | No sign-in method is configured. Set SMTP or an OAuth provider in env and extraEnvFrom. |
Uploads fail with 502 after about 60 s | The ingress read timeout. See Ingress. |
| Live views do not update | The realtime hostname does not reach the realtime pod, or its WebSocket is cut by an idle timeout. Check the realtime Ingress and its certificate. |
| Builds fail to clone or to reach the dashboard | NetworkPolicy. Set networkPolicy.ingressNamespaceSelector, and allow your Git server in deployerExtraEgress. |
helm upgrade refuses: "older than the deployed" | The downgrade guard. Restore a dump instead; use helm rollback only for a version that never started. |
How is this guide?
Upgrade
Learn how to upgrade a self-hosted TofuPilot instance, back up your data first, and enable realtime views when moving from an older version.
Station
Learn how to install and run the TofuPilot CLI as a station so a physical test bench can pull deployments and stream every run back to the dashboard.