Documentation
tow reads one file, the cluster file, and builds the cluster it describes: the machines, their preparation, Kubernetes, and every add-on it names.
1 - Installing tow
tow runs on your machine (Linux or macOS, amd64 or arm64) and reaches the cluster’s machines over SSH and their providers’ APIs.
Debian, Ubuntu and their derivatives (apt)
sudo mkdir -p /etc/apt/keyrings
sudo curl -fsSLo /etc/apt/keyrings/tow.asc https://towproj.github.io/tow/apt/gpg.key
echo "deb [signed-by=/etc/apt/keyrings/tow.asc] https://towproj.github.io/tow/apt stable main" | sudo tee /etc/apt/sources.list.d/tow.list
sudo apt update && sudo apt install tow
sudo apt upgrade brings later releases.
RHEL, Rocky Linux, AlmaLinux, Fedora (dnf)
sudo curl -fsSLo /etc/yum.repos.d/tow.repo https://towproj.github.io/tow/rpm/tow.repo
sudo dnf install tow
dnf checks the repository’s signature and each package’s. sudo dnf upgrade brings later releases.
macOS (Homebrew)
brew install --cask towproj/tap/tow
Any Linux or macOS: the binary
Download the archive for your system from the releases, check it, and put tow on your PATH:
tar -xzf tow_0.1.0_linux_amd64.tar.gz tow
sudo install tow /usr/local/bin/
How to check a download’s signature and provenance: releasing.md.
With Go
go install github.com/towproj/tow@latest
Shell completions
The packages and Homebrew install them for bash, zsh and fish. Otherwise: tow completion --help.
What tow needs besides
- OpenTofu for machines on AWS, GCP, Azure or Proxmox, and helm and kubectl for add-ons: when one is missing, tow asks before installing the release it was tested with into
~/.local/bin. - Docker and k3d for a local cluster.
- The provider’s credentials: platforms.md.
2 - The cluster file
tow.yaml describes your whole cluster: the machines, where they come from, and the Kubernetes on them, down to the network, storage, load balancers and ingress. tow installs exactly what the file asks for, and nothing else. Every field has a default, and a flavor is an optional ready-made starting point: anything it sets can be overridden.
apiVersion: towproj.github.io/v1alpha1
kind: Cluster
metadata:
name: lab
spec:
flavor: homelab # optional preset; fills only what this file leaves out
kubernetes:
distribution: k3s # k3s | kubeadm | k3d (local only)
version: v1.37.2 # left out: v1.37.2, the release tow tests; "stable": the newest
apiEndpoint: 192.168.1.100:6443 # with several control planes (AWS creates a load balancer instead)
networking:
cni: cilium # flannel | calico | cilium | none
storage: {csi: [longhorn], defaultClass: longhorn}
serviceLoadBalancer: {provider: metallb, metallb: {addresses: [192.168.1.200-192.168.1.220]}}
ingress: {controllers: [traefik]}
metricsServer: {enabled: true}
apiServer: {extraArgs: {audit-log-maxage: 30}} # flags passed as written to the component
addons:
helmCharts: [{name: cert-manager, repo: https://charts.jetstack.io, chart: cert-manager, version: v1.19.1}]
infrastructure:
provider: byoh # aws | gcp | azure | proxmox | byoh (your own machines) | local (k3d)
access:
ssh: {user: pi}
controlPlane:
machines: [{host: 192.168.1.10}, {host: 192.168.1.11}, {host: 192.168.1.12}]
workers:
- name: pis
labels: {disk: ssd}
machines: [{host: 192.168.1.13}, {host: 192.168.1.14, ssh: {port: 2222}}]
Flavors:
| Flavor | Kubernetes | Network | Storage | Ingress | Service load balancer | metrics-server |
|---|
local | k3d, as it ships | flannel | local-path | traefik | servicelb | ✓ |
homelab | k3s, as it ships | flannel | local-path | traefik | servicelb | ✓ |
dev-test | kubeadm | calico | local-path | traefik | none | ✓ |
production | kubeadm | cilium | longhorn | traefik | none | ✓ |
Without a flavor you get Kubernetes with nothing optional, not even a network (CNI), except on local clusters, which get flannel unless you write cni: none. A cluster without a CNI works but its nodes stay NotReady; tow says so.
Machines: byoh lists them under machines; aws, gcp, azure, proxmox and local create count of them. Sizes use instanceType on clouds (the cloud’s own name for it: t3.large, e2-standard-4, Standard_B2ms) or resources: {cpu, memory} on Proxmox, plus disk (40Gi). Worker pools can differ in size, and each pool can have its own labels, taints and kubelet flags.
Machines are prepared as the official docs ask (Kubernetes, k3s, and each component you pick): packages, kernel modules and settings, swap off for kubeadm, and the firewall’s rules when ufw or firewalld is on. Each item is checked first and changed only if missing. On your own machines tow lists the changes and asks before making them (apply --yes skips the question); every change is recorded on the machine in /etc/tow/prepared.json, with how to undo it.
Systems: Ubuntu and Debian (and their derivatives), prepared with apt, and RHEL 9 and 10 and their rebuilds (Rocky, AlmaLinux), prepared with dnf. tow reads each machine’s /etc/os-release, so one cluster may mix them. On RHEL-family machines, k3s keeps SELinux enforcing (its installer adds k3s’s policy, and tow turns it on in k3s); a custom k3s data-dir isn’t supported under SELinux and is refused there. kubeadm sets SELinux to permissive, as its install page does (recorded, and enforcing again after delete --reset). On AWS, spec.infrastructure.os picks ubuntu-24.04 (the default), almalinux-9 or almalinux-10 (AlmaLinux’s own free images, login user ec2-user; examples/homelab-aws-almalinux.yaml; a kubeadm control plane there needs more than 2 GiB, since AlmaLinux keeps memory aside for kdump, and tow refuses a smaller one before creating anything: examples/dev-test-aws-almalinux.yaml). GCP offers the same three, from Canonical’s and AlmaLinux’s free images, with the login user ubuntu or almalinux (examples/dev-test-gcp-almalinux.yaml); so does Azure, from Canonical’s images and AlmaLinux’s Marketplace images (free, no terms to accept; examples/disk-azure.yaml). Proxmox VMs run their template’s system (examples/homelab-proxmox-rocky.yaml).
firewalld, when it runs, lets the cluster through with two objects of tow’s own: the zone tow-nodes accepts the other nodes and the pod and Service ranges (k3s’s docs use the trusted zone for those ranges; tow leaves yours alone), and the policy tow-admin lets the admin addresses reach the API, NodePorts and the cluster’s open ports. A source firewalld already puts in another zone is refused, with the command that frees it. Both go with delete --reset.
Kubernetes release: the one the file pins; left out, v1.37.2, the release tow is tested with; stable for the newest upstream release at apply time. A newer tow that tests a newer release doesn’t count as a change to a cluster that left it out.
k3s goes on the machines with its official installer, at that release (its first k3s build: v1.37.2+k3s1), and the installer checks the download’s sha256 sum. Each machine’s settings are written to /etc/rancher/k3s/config.yaml, readable by root only. The first control plane starts the cluster (embedded etcd, so it can grow to several control planes), the others join it one at a time, then the workers.
- Joining is checked both ways: a joining machine verifies the cluster’s certificate authority before it sends the join password, and workers get a password that can’t join a control plane.
- Join passwords are never kept by tow; they stay on the control planes.
- CoreDNS gets two replicas with several control planes, on different machines (kubeadm runs two too). k3s ships one: when its machine goes away, Kubernetes waits five minutes before starting it elsewhere, and the cluster has no DNS meanwhile. k3s’s manifest sets no replica count, so k3s keeps the two when it applies the manifest again.
- The kubeconfig is kept in
~/.tow/clusters/<name>/kubeconfig and added to yours as context <name>. - apply finishes only when the API answers from your machine and every node has joined (and is Ready when there’s a network).
- An apply that stops partway picks up where it stopped on the next run.
kubeadm builds the cluster as the kubeadm docs describe: each machine gets a container runtime and the Kubernetes packages, then kubeadm init runs on the control plane and the workers join it.
- Several control planes, as kubeadm’s HA page sets them up: kubeadm init on the first uploads the certificates the others need, encrypted with a key tow makes (never saved; deleted from the cluster once they joined), then the others join one at a time, then the workers. The address in front of them (the cloud’s load balancer, yours, or kube-vip’s virtual IP) is the cluster’s
controlPlaneEndpoint, and the kubeconfig uses it. Your own load balancer is tested from the first control plane before init, as kubeadm’s page does.
kube-vip serves apiEndpoint on Proxmox and your own machines (the default there; apiLoadBalancer: external names your own load balancer instead): a virtual IP that one control plane holds and announces on the local network (ARP), moved to another by leader election when that one goes away. Only the API: Services get their addresses elsewhere. kube-vip v1.3.1, pinned by its image digest, set up as its docs say:
- kubeadm: a static pod on each control plane, on the first before
kubeadm init (reading super-admin.conf while init runs, then admin.conf), on the others once they joined. - k3s: a DaemonSet on the control planes, put in k3s’s manifests folder before k3s starts. It reads the k3s admin kubeconfig on each control plane, not a service account as kube-vip’s k3s page does: the service account reaches the API through the Service network, which a CNI that replaces kube-proxy only brings once the API answers on the virtual IP. One DaemonSet means one interface name: control planes whose interfaces are named differently are refused.
- The address must be an IPv4 address on the control planes’ network, port 6443; tow finds the interface that holds each control plane’s address and refuses a virtual IP outside its network before anything changes.
- Each machine’s kubeadm config is written from the file (release, pod and service ranges, cluster domain,
certSANs, each component’s extraArgs, node labels, taints, kubelet flags) and checked with kubeadm config validate on every machine before anything starts. It goes to the machine in a file only root can read, removed right after. - Joining is checked both ways: each worker checks the cluster’s certificate authority (by its hash) before it trusts the control plane. The join token is made by tow, lives an hour at most, never leaves the machines, and is deleted once the workers joined.
- kube-proxy is left out when the CNI replaces it (Cilium’s option, or Calico’s eBPF).
- An apply that stops partway picks up where it stopped: machines where kubeadm finished are left alone; one where it failed is reset (
kubeadm reset) and tried again. A failure shows kubeadm’s output, the kubelet’s last log lines and the machine’s containers. - The kubeconfig is kubeadm’s admin one, kept and merged like k3s’s.
- The runtime:
containerRuntime: containerd (the default) or cri-o. k3s brings its own containerd. Docker Engine (it needs cri-dockerd since Kubernetes 1.24), Podman (no CRI) and rkt (archived) are refused, with the reason. - containerd: a release from containerd’s GitHub releases, 2.3.7 unless the file pins another (
containerd: {version: ...}), with runc 1.5.2, each checked against a sha256 sum. Its config.toml comes from containerd config default, with systemd cgroups and the pause image kubeadm asks for. The release must be one containerd recommends for the Kubernetes minor (2.3 or 2.4 for 1.37); tow says what to pin otherwise. - CRI-O: from the CRI-O project’s packages for the Kubernetes minor (download.opensuse.org, deb or rpm), the newest release unless the file pins one, with kubeadm’s pause image in a drop-in.
- kubelet, kubeadm and kubectl at the cluster’s exact release, from pkgs.k8s.io, held there (
apt-mark hold; on RHEL-family machines, the repository’s exclude line, as kubeadm’s install page writes it, with dnf), plus crictl. Each kubelet restarts every few seconds until kubeadm sets it up; that’s expected. - A runtime already on your machine is used as it is when it goes with the Kubernetes release (for containerd: a recommended release, with its Kubernetes plugin on). Otherwise apply stops before changing anything and says why: tow never replaces or reconfigures a runtime it didn’t install. The same goes for kubelet, kubeadm or kubectl of another release, k3s, or a machine already in a cluster tow didn’t build.
The network (CNI): k3s and k3d bring flannel. On machines, Calico and Cilium go on right after Kubernetes (k3s or kubeadm), from your machine, as their docs describe, at a tested release unless the file pins one. apply then waits until every node is Ready.
- flannel on kubeadm (which brings no CNI) comes from flannel’s own Helm chart, 0.28.10 unless the file pins one, with the cluster’s pod range and the file’s
backend (vxlan by default, host-gw or wireguard). - Calico: VXLAN and the iptables dataplane by default;
encapsulation and dataplane: ebpf are supported. See examples/calico-proxmox.yaml and calico-ebpf-proxmox.yaml. - Cilium:
kubeProxyReplacement and WireGuard encryption are supported. See examples/cilium-proxmox.yaml. - Without kube-proxy: when the CNI replaces kube-proxy (Cilium’s option, or Calico’s eBPF), k3s runs without it, kubeadm skips it, and the CNI is given the API’s real address.
- Tools: Calico, and flannel on kubeadm, also need
kubectl of the cluster’s minor release (1.37 for a 1.37 cluster, any patch), which is checked and offered the same way as helm: tow offers the newest patch of that minor, the one dl.k8s.io names, checked against the sha256 dl.k8s.io publishes beside it.
Storage (storage.csi, several at once; examples/storage-proxmox.yaml) is installed right after the network, each driver from its official source at a pinned release, waited for, and the file’s defaultClass made the only default StorageClass:
- local-path 0.0.37, from its release’s manifest (k3s brings its own).
- OpenEBS LocalPV hostpath 4.6.2, class
openebs-hostpath. Its chart turns on five storage engines and a log stack by default; tow keeps hostpath only (LVM arrives with spare disks, in a later version). - Longhorn 1.13.0,
replicas copies of each volume. Its machines get open-iscsi and nfs-common (iscsi-initiator-utils and nfs-utils on RHEL) and the iscsi_tcp module, and multipathd is kept off its volumes with Longhorn’s own fix (sd* disks blacklisted in /etc/multipath/conf.d/longhorn.conf, recorded with its undo). A machine where multipath manages devices (a SAN) is refused, since that fix would cut their paths. - NFS (csi-driver-nfs 4.13.6) and SMB (csi-driver-smb 1.20.4): the driver, then a StorageClass
nfs or smb for the file’s share. SMB’s user and password come from the environment variables the file names, into a Secret in kube-system. - AWS EBS (
aws-ebs, examples/ebs-aws.yaml), GCE persistent disks (gcp-pd, examples/pd-gcp.yaml), Azure managed disks (azure-disk, examples/disk-azure.yaml) and Proxmox disks (proxmox, examples/proxmox-csi.yaml): each provider’s own disks as volumes. The cloud drivers’ permissions come from an IAM role, a service account or a managed identity on the control planes only; the Proxmox plugin takes a token of its own from PROXMOX_CSI_TOKEN. Setup and details: storage.md. - A version in the file is the chart’s (or, for local-path and gcp-pd, the release’s).
tow delete keeps your data safe: EBS volumes, persistent disks, Azure disks and Proxmox disks outlive the machines, and on your own machines volumes keep their data on the machines’ disks (local-path, OpenEBS, Longhorn), so a cluster that has some isn’t deleted. delete lists them and changes nothing. tow delete -v deletes them with the cluster, through Kubernetes, after naming them.
A service load balancer (serviceLoadBalancer.provider) gives LoadBalancer Services addresses from your pool (addresses: ranges, CIDRs or single addresses), announced on the local network (mode: l2, the default) or to your BGP routers (mode: bgp, with bgp: {localASN, peers}). Your machine (the admin addresses) may reach the pool’s addresses, any port, through the Proxmox firewall and ufw; on Proxmox the pool must sit on the VMs’ network. Layer 2 can’t work on AWS, GCP or Azure (their networks ignore it), so it’s refused there.
- MetalLB 0.16.2 (
examples/metallb-proxmox.yaml): its chart, its namespace allowed privileged pods as its docs ask, then an address pool and an L2 or BGP announcement. - kube-vip (
examples/kube-vip-lb-proxmox.yaml): kube-vip-cloud-provider 0.0.13 hands out the pool’s addresses. The kube-vip in front of the API announces them when there is one; otherwise a kube-vip DaemonSet of its own does, on each node’s default-route interface. BGP isn’t supported for it yet (kube-vip needs a router ID per node); use MetalLB or Cilium for BGP. - Cilium (
examples/cilium-lb-proxmox.yaml): LB IPAM with L2 announcements or its BGP control plane, turned on with Cilium’s own install. It works through Cilium’s kube-proxy replacement, which must be on. - servicelb is k3s’s own.
Ingress (ingress.controllers): tow builds in only what needs it. Traefik (k3s’s own, or its chart on kubeadm: on a pool address, or on every node’s ports 80 and 443 without a load balancer) and Cilium’s Ingress and Gateway API (with Cilium’s install; Gateway API’s definitions go in first). Any other controller (F5 NGINX, HAProxy, Envoy Gateway, Contour…) is a chart or manifest under addons; ingress.md has tested entries to copy.
Firewalls: ports 80 and 443 for ingress, and any TCP ports you list in spec.infrastructure.access.ports (for what you install yourself), are opened to the admin addresses in the AWS security group, the GCP firewall rules, the Azure network security group, the Proxmox firewall, ufw and firewalld. The admin addresses are your machine by default; access.adminCIDRs widens them, on your own machines too.
metrics-server (chart 3.14.0, metrics-server 0.9.0) on kubeadm; k3s brings its own. kubeadm’s kubelets then get serving certificates the cluster signs (serverTLSBootstrap), as kubeadm’s certificate page says, so metrics-server can check them; kubelet-csr-approver 1.2.15 approves them, for the cluster’s own node names and addresses only.
Manifests (addons.manifests: https URLs, or files and folders next to the cluster file) are applied last with kubectl apply --server-side. Objects that name no namespace go into one named after the file (app.yaml → app), created by tow; write {path: app.yaml, namespace: shop} to pick it. Objects that name their own keep it.
etcd backups (backups: {schedule, keep}, backups.md): snapshots on each control plane on a schedule, k3s’s own or a systemd timer on kubeadm. tow backup --name <cluster> takes one now and keeps a copy in ~/.tow/backups/<cluster>/, checked against its sha256. The copies outlive the cluster. tow restore --name <cluster> puts one back, on the same machines (lifecycle.md). A snapshot holds the cluster’s Secrets: keep it safe.
Nothing goes into default: a chart without a namespace gets one named after its release.
Helm charts (addons.helmCharts) are installed once the cluster runs, in the file’s order, with your own helm (3.8 or newer, or 4.x), from your machine (examples/local-helm.yaml). The file’s values win over what a controller changed in a chart’s objects, as Helm 3 always does: on Helm 4, which applies server-side, tow adds --force-conflicts to every helm upgrade it runs.
- Each chart is installed with
helm upgrade --install --wait, at the exact version the file pins. - Values go to helm in a file only you can read, deleted right after. A secret value comes from an environment variable (
valuesFromEnv), never the file. - A failed chart keeps the cluster; the next apply installs the charts again.
- When helm is missing or too old, tow says so and asks before installing the release it was tested with into
~/.local/bin, its download checked against a sha256 built into tow. A helm you already have is never replaced without your yes. Unattended runs need --install-tools.
Pass-through settings (extraArgs, chart values, manifests) go as written to the tool that uses them (kubeadm, k3s, Helm, the API server), and that tool’s own error is shown if it rejects one. A flag a field of the file already sets is refused as a conflict.
Secrets never go in the file. A flag or chart value that needs one names an environment variable instead (etcd-s3-secret-key: {env: S3_SECRET_KEY}, or valuesFromEnv for a chart). tow reads it when it runs and hides it from all its output; a key or password written in the file is refused.
Only real limits are enforced. A field the provider doesn’t use is refused rather than ignored, and every refusal says why; risky but working choices get a warning. Anything this version can’t install yet is listed by name, never silently skipped.
More examples in examples/: custom-bare.yaml is the smallest file, custom-full.yaml shows every field, and the other custom-*.yaml files use no flavor.
3 - Platforms
Where the machines come from: spec.infrastructure.provider in the cluster file.
| Platform | Provider | Tested end to end |
|---|
| Your laptop (k3d) | local | yes |
| AWS | aws | one control plane, a cluster’s whole life, EBS volumes, conformance (469/469); several control planes not yet (NLBs aren’t in the free plan) |
| GCP | gcp | one and three control planes, a cluster’s whole life, persistent disks, conformance (469/469) |
| Azure | azure | one control plane, managed disks; several control planes and a whole life not yet |
| Proxmox VE | proxmox | one and three control planes (kube-vip), a cluster’s whole life, every network, storage and load balancer option |
| Your own machines | byoh | k3s and kubeadm on AWS servers (Ubuntu and AlmaLinux), including delete --reset |
Systems: Ubuntu and Debian and their derivatives, RHEL 9 and 10 and their rebuilds. Tested on Ubuntu 24.04, Rocky Linux 9 and 10, and AlmaLinux 9 and 10, x86-64; the clouds offer Ubuntu 24.04 and AlmaLinux 9 and 10 (spec.infrastructure.os).
Servers on AWS
Requires OpenTofu (tofu 1.12 or a newer 1.x; when it’s missing, tow asks before installing the release it was tested with into ~/.local/bin, sha256 checked) and AWS credentials from any standard source: aws sso login or aws configure on your machine, AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY or an OIDC role for service accounts and CI, or an instance role. Credentials never go in tow.yaml; tow refuses a file that contains them.
tow apply -f examples/dev-test-aws.yaml # two t3.small servers; SSH, the API and NodePorts open to your IP only
tow status --name dev-aws
tow delete --name dev-aws # destroys everything and checks nothing is left
OpenTofu’s files and state live in ~/.tow/clusters/<name>/tofu/, never in your working directory. Its providers come from registry.opentofu.org, pinned to the releases tow is tested with, checked against their checksums, and downloaded once into ~/.tow/plugin-cache for every cluster (AWS 6.72.0, Google 8.11.0, Azure 5.14.0, Proxmox 0.120.0); several applies can run at once. If an apply fails partway, fix the cause and run it again; servers already created are reused. If it’s killed while OpenTofu runs, tow delete still finds what it created.
With several control planes (examples/production-aws.yaml), tow puts two network load balancers (NLBs) in front of them:
- A public one, for you: your kubeconfig points at it, and as with one control plane only your IP reaches the API.
- An internal one, for the nodes: kubeadm’s
controlPlaneEndpoint, which the kubelets, kube-proxy, the CNI and joins use. Its address is private, so the nodes’ API calls stay on the VPC’s private network like the rest of their traffic. It doesn’t keep the caller’s address, since AWS doesn’t support a machine behind an NLB reaching that NLB otherwise, and control planes do.
The API certificate names both. NLBs aren’t in AWS’s free tier.
Machines on GCP
Requires OpenTofu (as for AWS), a project with the Compute Engine and IAM APIs on (gcloud services enable compute.googleapis.com iam.googleapis.com --project <id>), and Google’s Application Default Credentials: gcloud auth application-default login on your machine, GOOGLE_APPLICATION_CREDENTIALS (a key file, or a workload identity federation configuration) for service accounts and CI, or a VM’s own service account. The file names the project and region: gcp: {project: my-project-123456, region: europe-west1}.
tow apply -f examples/dev-test-gcp.yaml # two e2-medium machines; SSH, the API and NodePorts open to your IP only
tow delete --name dev-gcp # destroys everything and checks nothing is left
Each cluster gets its own VPC and subnet, firewall rules, and machines spread over up to three of the region’s zones, each with a public address (no Cloud NAT). Google’s guest agent creates the login user from tow’s SSH key; the project’s own SSH keys and OS Login stay out. With several control planes (examples/homelab-gcp-ha.yaml), tow puts two passthrough network load balancers in front of them, a public one for you and an internal one for the nodes, as on AWS. A machine behind a GCP load balancer can’t reach it (Google routes the address back to the machine itself), so the nodes reach the API by a name, api.<cluster>.internal, which each machine resolves in its /etc/hosts: a worker to the internal load balancer, a control plane to itself (to the first control plane while it joins). The API certificate names it; delete takes the line away. A free trial allows 4 public addresses in all: one per machine, one per HA cluster’s load balancer.
VMs on Azure
Requires OpenTofu (as for AWS) and Azure credentials: az login on your machine, a service principal in ARM_CLIENT_ID, ARM_TENANT_ID and ARM_CLIENT_SECRET (as OpenTofu’s azurerm provider reads them) or Azure’s AZURE_ variables and workload identity for CI, or a VM’s managed identity. The file names the subscription and location: azure: {subscription: <its ID>, location: swedencentral} (new subscriptions may find a location closed to them, or a VM size not offered: az vm list-skus --size Standard_B2s_v2 --query "[?length(restrictions)==\0`].locationInfo[].location"` lists where yours may use it).
tow apply -f examples/dev-test-azure.yaml # two Standard_B2s_v2 VMs; SSH, the API and NodePorts open to your IP only
tow delete --name dev-azure # deletes the resource group and checks nothing is left
Everything of a cluster’s goes in a resource group of its own, tow-<cluster>: a VNet and its subnet, a network security group, and VMs spread over up to three of the location’s availability zones (none where the location has no zones), each with a static public address (no NAT gateway). Azure creates the login user from tow’s SSH key, with no password. With several control planes (examples/homelab-azure-ha.yaml), tow puts two Standard load balancers in front of them, a public one for you and an internal one for the nodes; as on GCP, the nodes reach the API by a name each machine resolves in its /etc/hosts, since a machine behind an internal load balancer can’t always reach it. OS disks are 30 GB at least: Azure’s images are that big. Several control planes and a cluster’s whole life are written but not run on Azure yet: they need more public addresses and vCPUs than a free subscription allows.
VMs on Proxmox VE
Requires OpenTofu (as for AWS) and an API token in PROXMOX_VE_API_TOKEN (user@realm!tokenid=secret); set PROXMOX_VE_INSECURE=true only for a self-signed lab certificate. VMs are cloned from a cloud-init template and get fixed addresses from the range in the file. A template of RHEL 10 or one of its rebuilds needs proxmox.cpuType: x86-64-v3 (or host): their kernel doesn’t boot on Proxmox’s default CPU model, x86-64-v2-AES.
tow apply -f examples/homelab-proxmox.yaml # two VMs running k3s; kubectl switches to context "homelab-pve"
tow delete --name homelab-pve # removes the VMs and the context, and checks nothing is left
With several control planes, set spec.kubernetes.apiEndpoint to a free address on the VMs’ network, outside their range: kube-vip holds it as the API’s virtual IP, and the kubeconfig uses it (examples/homelab-proxmox-ha.yaml: three control planes, no workers). An existing load balancer of your own goes with your own machines (provider: byoh). Setting up a test server: testing/proxmox.md.
Your own machines
List them under machines with provider: byoh (see examples/homelab.yaml). tow checks them, asks before preparing them, and installs k3s or kubeadm. A machine where Kubernetes was installed by someone else, or for another cluster, is refused before anything changes. With ufw or firewalld on, the checks can’t see past it before preparation opens the ports in it: tow warns instead, and if the install then stalls or fails with an unclear error, something else blocks those ports too (a cloud security group, another firewall).
Every change tow makes on a machine is recorded in /etc/tow/prepared.json there (root only), with how to undo it. tow delete undoes from that record:
- Kubernetes goes: k3s’s own uninstaller, or
kubeadm reset and what tow installed for it (the Kubernetes packages, containerd). - The preparation stays (packages, firewall rules, swap, kernel settings) unless you pass
--reset, or answer yes when delete asks. Only what tow recorded is undone, never what was there before; --yes alone keeps it. - A machine that’s gone for good:
tow delete --forget forgets the cluster without touching any machine.
What stays until a restart, and how to undo it all by hand: reset-machines.md.
4 - A running cluster's life
The cluster’s file stays its source of truth: change it, run tow apply, and apply shows a plan (Infrastructure first, then Cluster), asks, and makes it. --dry-run only shows the plan; --yes doesn’t ask. A change apply can’t make is refused before anything changes, with the reason and the way to get there. A run that stops says what it had done and the command to run again; running it again plans the rest from what’s really there.
What can change, and what can’t
| In the file | On a running cluster |
|---|
machines: count, machines, pools added or removed | ✓ joined, or drained and removed, one at a time (machines) |
labels, taints, controlPlane.schedulable | ✓ through the API |
infrastructure.access.adminCIDRs, access.ports | ✓ firewalls follow (AWS, Proxmox, ufw) |
kubernetes.version | ✓ an upgrade, alone in its apply (below) |
component settings: extraArgs, certSANs, kubelet flags, k3s flags | ✓ one machine at a time (settings); k3s’s critical ones refused |
backups | ✓ (backups) |
built-in add-ons on or off, their settings; helmCharts, manifests | ✓ at the version that runs; versions move with an upgrade |
kubeconfig | ✓ renamed, merged or not |
a machine’s size (instanceType, resources, disk) | ✗ add a pool with the new size, apply, remove the old pool |
metadata.name, distribution, containerRuntime, the provider and region | ✗ build a new cluster and move your workloads |
networking: ranges, clusterDomain, cni, flannel’s backend, kube-proxy replacement, Calico’s encapsulation | ✗ build a new cluster and move your workloads |
apiEndpoint, apiLoadBalancer | ✗ every node and certificate uses the API’s address |
how tow reaches the machines (ssh) | ✗ |
| containerd’s or CRI-O’s release | with an upgrade only |
A local (k3d) cluster takes only its own charts and manifests; anything else: delete it and apply again.
Adding and removing machines
New machines are created (OpenTofu) or checked (your own), prepared, and joined; the plan names them. A machine leaving goes last, one at a time: drained (once the workloads with a pod on it have their other pods Ready, as for upgrades; PodDisruptionBudgets respected, at most 10 minutes, apply --drain-timeout; nothing is forced), a control plane out of etcd while it still runs, Kubernetes removed, its Node deleted, then the machine destroyed (on your own machines, the cluster’s firewall rules undone; all of tow’s preparation too with apply --reset). The first control plane stays; an even number of control planes is warned about. A VM deleted outside tow is made again under its name and address.
Upgrading Kubernetes
Raise spec.kubernetes.version in the cluster’s file, then run tow apply. Nothing else upgrades a cluster: an empty or stable version never does. apply and tow status say when a newer patch of the release you run is out:
Upgrade: Kubernetes v1.37.4+k3s1 is out (v1.37.2+k3s1 runs): set spec.kubernetes.version: v1.37.4+k3s1 and apply to upgrade
On k3s, v1.37.4 means k3s’s first build of it (v1.37.4+k3s1); write +k3s2 for a later build.
What’s refused, before anything changes
- Skipping a minor release: 1.36 to 1.38 is refused; upgrade to 1.37 first. Kubernetes and k3s upgrade one minor release at a time.
- A downgrade: Kubernetes has no supported way back to an older release.
- A release that doesn’t exist: checked on dl.k8s.io (kubeadm) or k3s’s releases before the plan is shown.
- Other changes in the same apply: an upgrade goes alone. Apply the file’s other changes first, with the version that runs, then the new version. The versions of what tow installs (the CNI, the built-in add-ons, containerd, CRI-O) may change with it.
- A container runtime that doesn’t go with the new release: containerd or CRI-O pinned to a release that doesn’t support it; change its version in the same apply. One tow didn’t install is never changed: the upgrade is refused.
- An add-on moving too far: Cilium and Longhorn move one minor release at a time, Calico from its two previous releases (their upgrade docs); never backwards. Pin the release in between with the upgrade.
- The file going back while an upgrade is half done: nodes already upgraded can’t go back. Set the version they run, and apply to finish the upgrade.
How it goes
apply shows the plan and asks first (--yes doesn’t ask, --dry-run only shows it):
- An etcd snapshot, copied to
~/.tow/backups/<cluster>/ like tow backup does (backups). Its .json says beforeUpgradeTo: the release it was taken before. - The nodes, one at a time: control planes first, then workers, as k3s’s and kubeadm’s upgrade docs say. Each node is:
- waited for, before it’s touched: every workload with a pod on it must have its other pods Ready (so a second pod of theirs isn’t taken away before the first came back), for at most the drain timeout;
- drained: its pods move elsewhere, PodDisruptionBudgets respected, for at most 10 minutes (
apply --drain-timeout); nothing is forced; - upgraded (below);
- waited for: Ready at the new release;
- after a control plane, checked: the API answers
/readyz (etcd included) on every control plane; on kubeadm, its control-plane containers run and every etcd member is healthy too; - uncordoned.
- The CNI and the built-in add-ons move to the releases this tow installs, or those the file pins, in install order: Cilium after its pre-flight check (which pulls the new images on every node), keeping the defaults of the release it was first installed at (
upgradeCompatibility); Calico’s CRDs before its operator. Your own helmCharts and manifests aren’t touched. - The new release is recorded once every node runs it.
k3s upgrades a node with its own installer, at the new release, with the settings the node already has.
kubeadm follows its upgrade page, on each node:
- the Kubernetes package repository moves to the new minor release, and the kubeadm package is upgraded;
kubeadm upgrade apply on the first control plane (after kubeadm upgrade plan checks it), kubeadm upgrade node on the others and on workers. kubeadm renews the certificates it manages, and upgrades CoreDNS (and kube-proxy, when the cluster has it) once the last control plane is done; tow spreads CoreDNS over the nodes again after;- the container runtime, when it doesn’t go with the new release: containerd to the release tow installs (or the file’s), CRI-O from its new minor release’s repository;
- kubelet and kubectl, held at the new release, and the kubelet restarted. The kubelet is never newer than the API server.
kubeadm keeps a copy of each control plane’s manifests and etcd data from before the upgrade in /etc/kubernetes/tmp; delete it once the cluster runs well. kubeadm’s docs also describe stopping the API server for a few seconds before an etcd upgrade, so requests finish first; tow doesn’t.
A cluster of one machine stops its workloads while that machine is drained and upgraded. So does a cluster whose pods can run on one node only (one worker, control planes that take no pods): its pods wait while it’s drained, and a PodDisruptionBudget that needs one of them running stops the drain. The plan says so in both cases; add a worker, or make the control planes schedulable, first.
Databases and other replicated workloads: before draining a node, tow waits until every workload with a pod there (a ReplicaSet, a StatefulSet, an operator’s object) has its other pods Ready, as if each had a PodDisruptionBudget allowing one pod away at a time; your own budgets apply on top. That wait is only as good as the pods’ readiness: a database replica on a local volume comes back on its own node and needs time to catch up, so its readiness should say “caught up”, not just “accepts connections”. With CloudNativePG, set the readiness probe to type: streaming with a maximumLag (.spec.probes.readiness): a replica is then Ready only once it streams from the primary within that lag, and the primary’s node isn’t drained before. Replication can stay asynchronous: the primary’s orderly shutdown hands its last writes to a replica that streams. The lifecycle test does exactly this, with Postgres written to throughout the upgrade; before tow waited, the lab lost committed rows when the primary’s worker was drained right after the replica’s worker came back. A workload whose other pods never become Ready stops the upgrade after the drain timeout, named, with nothing changed on that node.
When it stops partway
The error names the node and the step that failed, says which nodes are already upgraded, and ends with the command to run again. Running it again:
- skips the nodes already upgraded, read from each node’s own release;
- finishes a node it left cordoned: waits for it, checks it, then uncordons it;
- takes no second snapshot: the first one is from before any node changed.
A drain that doesn’t finish leaves the node cordoned and names the pods in the way (a PodDisruptionBudget that allows no disruption, a pod nothing manages). tow status shows which nodes run which release while an upgrade is half done.
Going back
There’s no downgrade. The snapshot taken before the upgrade holds the cluster’s data from just before, and tow restore can put it back on the upgraded release (below): the data goes back, the release stays.
Changing settings
Change a component’s settings in the file, then run tow apply. The plan names each machine and what changes on it, and asks first.
What can change: the API server’s, controller manager’s, scheduler’s and etcd’s extraArgs, apiServer.certSANs, kubelet flags (cluster-wide, the control planes’, each pool’s), metrics-server on or off, and kube-vip announcing LoadBalancer Services. On k3s also k3s.server and k3s.agent extraArgs, the backups schedule, and k3s’s bundled Traefik and local-path on or off.
- One machine at a time, control planes first, only those whose settings change.
- k3s: the machine gets its new
config.yaml, and k3s restarts with it. - kubeadm, as its reconfigure page says: the cluster’s stored configuration is updated first (
kubeadm init phase upload-config, so a later upgrade keeps it); each control plane’s static pods are written again for the components whose settings changed (kubeadm init phase control-plane …, init phase etcd local), with a new API server certificate for new certSANs (the old one set aside), and kube-vip’s static pod for its Services; each kubelet’s flags are changed in /var/lib/kubelet/kubeadm-flags.env (kubeadm’s own flags stay), its configuration pulled again when metrics-server goes on or off (serving certificates signed by the cluster), and the kubelet restarts. metrics-server and its certificate approver are then installed or removed.
- No drain: restarting k3s, a static pod or the kubelet keeps the machine’s pods running.
- The API stays up, except for a few seconds while the control plane holding kube-vip’s address restarts: kube-vip keeps the address there, and that machine’s API server is starting again (about 10 seconds in the lab).
- Checked before the next one: the node is Ready, and after a control plane, the API answers
/readyz on every control plane (on kubeadm, its components run and every etcd member is healthy too). - Turning a bundled add-on off uses k3s’s own
disable, which removes it. local-path is refused while it still has volumes; the plan names the Ingresses Traefik serves. - A run that stops names the machine and those already done; running it again skips them.
Refused, with the reason:
- k3s’s critical settings, which k3s requires to be the same on every server (servers whose value differs fail to join), so they can’t change one server at a time:
agent-token, cluster-cidr, cluster-dns, cluster-domain, disable-cloud-controller, disable-helm-controller, disable-network-policy, disable=servicelb (servicelb on or off), egress-selector-mode, embedded-registry, flannel-backend, flannel-external-ip, flannel-ipv6-masq, secrets-encryption, secrets-encryption-provider, service-cidr; and data-dir, which holds each machine’s data. - flannel’s backend, on either distribution: nodes on different backends can’t reach each other.
- containerd’s and CRI-O’s release: they change with a Kubernetes upgrade.
Certificates
Kubernetes’ certificates, and the client certificate in the kubeconfig tow gave you, last a year (k3s’s and kubeadm’s defaults).
tow status shows when the API server’s and your kubeconfig’s expire.tow apply renews them when either expires within 30 days; apply --renew-certificates renews them now. The plan says so first.- k3s: on each control plane in turn, k3s stops,
k3s certificate rotate renews its certificates, and k3s starts and is checked. Then each worker’s k3s restarts; it renews its own certificates within 120 days of their expiry. - kubeadm: on each control plane in turn,
kubeadm certs renew all, then its control plane restarts (its static pods, which read their certificates when they start) and is checked. The kubelets renew their own certificates. - Then your kubeconfig is fetched again, with its renewed client certificate, and stored as before: in
~/.tow/clusters/<cluster>/ and, when it was, in your own kubeconfig.
- An upgrade renews them too: a cluster upgraded at least once a year never needs this.
Restoring a snapshot
tow restore --name prod # the newest backup on this machine
tow restore --name prod --snapshot prod-20261113T021206Z --dry-run
restore puts a running cluster’s etcd back to one of its backups in ~/.tow/backups/<cluster>/ (backups), on the same machines. Everything the cluster’s API knew comes back as it was then: objects, Secrets, Helm releases. Every change since is lost. Volumes’ data isn’t in etcd and stays as it is.
- Which snapshot: the newest, unless
--snapshot names one (tow backup --name <cluster> --list shows them). - Only these can be restored: a snapshot of the release the cluster runs, or the one an upgrade took right before upgrading to it (the release stays the upgraded one). A newer release’s snapshot, an older one, another cluster’s, or a copy whose sha256 no longer matches is refused before anything changes. On kubeadm, etcd restores only snapshots of its own minor release: a pre-upgrade snapshot is refused when the upgrade moved etcd to another one.
- A safety backup first, of the cluster as it is now, so the restore itself can be undone;
--no-backup skips it. When no control plane can take one (etcd is down, often the reason to restore), the restore goes on without it. - The plan is shown and asked about first;
--yes doesn’t ask, --dry-run only shows it. The cluster is locked while it runs.
How it goes:
- k3s (its docs): k3s stops on every control plane; the first one is reset to the snapshot (
k3s server --cluster-reset --cluster-reset-restore-path, with --etcd-s3=false when the file has k3s upload to S3) and starts; then each other control plane, one at a time, has its etcd data moved aside and joins the restored cluster. Workers keep running. - kubeadm (Kubernetes’ and etcd’s docs):
etcdutl is read from each control plane’s own etcd, and the snapshot’s etcd release checked against it, before anything stops. Then etcd and the API servers stop on every control plane (their static pods set aside; kube-vip’s stays), every etcd member is restored from the same snapshot into its own data directory (with --bump-revision and --mark-compacted, so nothing watching the API sees revisions go backwards), and they start again. The control planes are checked (the API ready, every etcd member healthy), and the kubelet restarts on every node, as the docs advise.
After a restore:
- The old etcd data stays, moved aside (
/var/lib/etcd.before-restore-<time>, or <data-dir>/server/db.before-restore-<time> on k3s), and the output names each one: delete them once the cluster runs well. - Nodes the snapshot has that the cluster no longer has (machines removed since) come back NotReady; they’re named, for you to delete with
kubectl delete node. tow doesn’t delete them. - Run
tow apply with the cluster’s file: the snapshot holds the add-ons and charts as they were, and apply puts back what the file says. - A run that stops says where, and the command to run again; running it again restores the same snapshot from the start.
Restoring onto new machines (when the old ones are gone for good) isn’t possible yet.
5 - Storage on the provider's own disks
AWS: EBS volumes (storage.csi: [aws-ebs])
tow installs the AWS EBS CSI driver (1.68.0, chart 2.68.0) with a StorageClass aws-ebs: gp3, encrypted, created in the zone of the node its pod runs on.
- Permissions: the AWS module puts an IAM role with AWS’s own
AmazonEBSCSIDriverPolicy on the control planes, and only there. The driver’s controller runs on them. No keys anywhere. - Instance metadata: the driver’s pods read it (the instance, its zone) through one more network hop. The machines allow two hops instead of one, as the driver’s docs ask for clusters AWS doesn’t manage.
- Cleanup: every volume is tagged with the cluster, so the check after
tow delete finds any left behind.
GCP: persistent disks (storage.csi: [gcp-pd])
tow installs the GCE PD CSI driver (1.28.1) with a StorageClass gcp-pd: pd-balanced, created in the zone of the node its pod runs on. The driver ships no Helm chart; tow applies its release’s own manifests (the noauth overlay, which uses the machine’s service account instead of a key), with a few changes, each marked in gcp-pd.yaml.tmpl: no Windows parts, the controller on the control planes, the node plugin on every node (a tow cluster may be control planes only), and the cluster’s label on every disk. A version in the file picks the driver’s image; the manifests stay the tested release’s.
- Permissions: the GCP module creates a service account that only the control planes run as, and the driver’s controller runs there. No keys anywhere. The driver’s own setup grants it
roles/editor on the project; tow grants less: Compute Storage Admin (disks and snapshots), a custom role of the cluster’s own to get machines and attach and detach disks (the driver’s own custom role, plus reading operations), and Service Account User on that one account (attaching a disk to a machine that runs as an account asks for it). - Machine types: pd-balanced works with the E2, N1, N2, N2D, C2 and T2 families; the newest ones (C3, C4, N4) take Hyperdisk only.
- Cleanup: every disk is labeled with the cluster, so the check after
tow delete finds any left behind. A deleted custom role stays, marked deleted, for 7 days before Google purges it; a cluster made again under the same name takes it back.
Azure: managed disks (storage.csi: [azure-disk])
tow installs the Azure Disk CSI driver (chart 1.36.0) with a StorageClass azure-disk: StandardSSD, created in the zone of the node its pod runs on, in the cluster’s resource group, tagged with the cluster.
- Permissions: the Azure module creates a managed identity that only the control planes carry, with Contributor on the cluster’s resource group and nothing else. The driver’s controller runs on the control planes and reads it from the instance metadata. No keys anywhere.
- Its cloud config is a Secret,
azure-cloud-provider in kube-system, as the driver’s docs set it up: the tenant, subscription, resource group, location and the identity’s client ID. IDs only, nothing secret; tow takes them from OpenTofu once the machines exist. - Node names are the VMs’ names: the driver finds a node’s VM by it.
- Cleanup: the driver’s disks live in the cluster’s resource group, which
tow delete removes; Azure refuses to delete the group while a disk tow didn’t make is still in it, so nothing goes unnoticed.
Proxmox: disks as volumes (storage.csi: [proxmox])
tow installs the Proxmox CSI plugin (0.22.0, chart 0.5.15) with a StorageClass proxmox on the file’s storage (spec.infrastructure.proxmox.storage, local-lvm by default). A volume is a disk attached to the VM its pod runs on, and it moves with the pod to another VM on the same server. tow labels each node with its region (the cluster’s name) and zone (the Proxmox node), as the plugin requires.
The plugin needs two things from you:
Its own token, with only its privileges. Never use the admin token tow creates VMs with: every pod that can read the plugin’s Secret would hold it. On the Proxmox server, as root (from the plugin’s docs):
pveum role add CSI -privs "VM.Audit VM.Config.Disk Datastore.Allocate Datastore.AllocateSpace Datastore.Audit"
pveum user add kubernetes-csi@pve
pveum aclmod / -user kubernetes-csi@pve -role CSI
pveum user token add kubernetes-csi@pve csi -privsep 0
Export the result as PROXMOX_CSI_TOKEN=kubernetes-csi@pve!csi=<secret> before tow apply. tow reads it at run time, hides it from its output, and never writes it to the file or its state.
Proxmox in a cluster. One server can be a cluster of its own: pvecm create <a name>, as root. apply checks this before creating anything.
Deleting a cluster that has such volumes
EBS volumes, persistent disks, Azure disks and Proxmox disks outlive the machines: a cloud volume keeps costing money, and a Proxmox disk keeps taking space. So tow delete refuses a cluster that has some, lists them, and changes nothing. tow delete -v deletes them with the cluster, after naming them in the confirmation. It does this through Kubernetes, before the machines go, so each driver deletes its own disks. Back up what you need first.
6 - Ingress and Gateway
tow installs two ingress controllers itself, because each needs something only tow can do:
- Traefik (
ingress.controllers: [traefik]). k3s brings its own; on kubeadm, tow installs Traefik’s chart (42.3.2, Traefik 3.7.17).- With a service load balancer, it gets an address from your pool.
- Without one, it runs on every node and listens on their ports 80 and 443.
- Its IngressClass is the default only when
defaultClass says so.
- Cilium (
ingress.controllers: [cilium], with cni: cilium). Its Ingress and Gateway API are settings of Cilium’s own install. tow installs Gateway API’s definitions (v1.6.3, standard channel) before Cilium, then turns both on. They work through Cilium’s kube-proxy replacement and get their addresses from its load balancer, so both are required. You write your own Gateways, with gatewayClassName: cilium.
Ports 80 and 443 are opened to the admin addresses (spec.infrastructure.access.adminCIDRs, by default your machine) in the AWS security group, the Proxmox firewall and ufw.
Any other controller
Install it as what it is, a chart or a manifest, under spec.kubernetes.addons. tow installs it in order, waits for it, and shows the tool’s own error if it fails. If it listens on node ports (no service load balancer), list them in spec.infrastructure.access.ports so the firewalls let you in.
These entries are tested together in examples/ingress-proxmox.yaml, each on its own MetalLB address. They’re ready to copy:
spec:
kubernetes:
addons:
helmCharts:
- name: nginx-ingress # F5 NGINX Ingress Controller, class "nginx"
repo: oci://ghcr.io/nginx/charts
chart: nginx-ingress
version: 2.8.1
- name: haproxy # HAProxy Kubernetes Ingress, class "haproxy"
repo: https://haproxytech.github.io/helm-charts
chart: kubernetes-ingress
version: 1.54.4
values: {controller: {service: {type: LoadBalancer}}} # its default is NodePort
- name: envoy-gateway # Gateway API only; brings Gateway API's definitions
repo: oci://docker.io/envoyproxy
chart: gateway-helm
version: v1.10.0
namespace: envoy-gateway-system
manifests:
# Contour has no maintained Helm chart; its own rendered manifest, pinned to a release:
- {path: "https://raw.githubusercontent.com/projectcontour/contour/v1.33.8/examples/render/contour.yaml", namespace: projectcontour}
- manifests/ingress-classes.yaml # IngressClass contour, GatewayClass envoy-gateway
Notes:
- Contour’s manifest always takes ports 80 and 443 on every node (its Envoy DaemonSet), on top of its LoadBalancer Service. No other controller in the cluster can listen on those node ports.
- Envoy Gateway’s chart creates no GatewayClass.
examples/manifests/ingress-classes.yaml adds one, and an IngressClass for Contour. - Envoy Gateway and Cilium both bring Gateway API’s definitions, so pick one of them for the Gateway API.
- ingress-nginx (the Kubernetes community controller) is retired upstream. If you still need it, it’s a chart like the others.
7 - Backups: etcd snapshots
etcd holds everything the cluster’s API knows: every object, every Secret, every setting. A snapshot of it brings that back. It doesn’t hold your volumes’ data. For that, see App data below.
Snapshots on a schedule (spec.kubernetes.backups)
spec:
kubernetes:
backups:
schedule: "0 */6 * * *" # as cron: minute hour day-of-month month weekday
keep: 10 # the newest kept on each control plane; default 5
The schedule runs in the machines’ time zone (UTC on cloud images). Without backups, tow adds no schedule of its own.
- k3s takes etcd snapshots itself: every 12 hours, keeping 5, in
/var/lib/rancher/k3s/server/db/snapshots on each control plane. backups sets k3s’s own etcd-snapshot-schedule-cron and etcd-snapshot-retention. - kubeadm has no schedule of its own.
backups puts a systemd timer (tow-etcd-snapshot.timer) on each control plane. It runs the etcdctl in the machine’s own etcd container (through crictl), so nothing else is downloaded. The snapshots go to /var/lib/tow/snapshots, readable by root only. The timer is recorded in /etc/tow/prepared.json with its undo, and it goes when Kubernetes is removed from the machine. The snapshots stay.- systemd and cron read some schedules differently, so these are refused: a day of the month and a weekday both set (cron runs when either matches, systemd only when both do), and a step on a range (
0-12/2). List the values instead. systemctl list-timers tow-etcd-snapshot.timer shows the next run; journalctl -u tow-etcd-snapshot shows past ones.
On a running cluster
- kubeadm:
apply turns the timer on, changes it, or takes it off on every control plane. Taking it off leaves the snapshots already taken. A control plane added later gets the timer when it joins. - k3s: a change is refused for now. k3s reads its schedule only when it starts, and restarting the control planes one at a time comes in a later version.
tow backup takes a snapshot whenever you need one.
A backup now: tow backup
tow backup --name prod # a snapshot now, copied to this machine
tow backup --name prod --list # the copies on this machine
backup takes a snapshot on the first control plane that answers (k3s etcd-snapshot save on k3s, the same etcdctl as the timer on kubeadm). The snapshot stays next to the scheduled ones and is pruned with them. A copy comes to this machine and is checked against the snapshot’s sha256:
- Where:
~/.tow/backups/<cluster>/, in a folder only you can read; each file is 0600. - What it is: a
.json beside each copy says the cluster, its Kubernetes release and distribution, the machine and time it was taken on, and its sha256. - It outlives the cluster:
tow delete leaves the copies and says where they are. On your own kubeadm machines, the timer’s snapshots also stay in /var/lib/tow/snapshots, and delete names those machines.
A local (k3d) cluster has no snapshots.
A snapshot holds your Secrets
Every Secret in the cluster is in its snapshots: in plain text, unless Secrets are encrypted at rest. Keep the snapshots as safe as the cluster’s credentials. backup says so every time.
- k3s:
secrets-encryption: true in spec.kubernetes.k3s.server.extraArgs, when the cluster is built. - kubeadm: an
EncryptionConfiguration for the API server. tow doesn’t set one up yet.
Copies somewhere else
The copy on this machine is your off-machine copy. tow uploads nothing anywhere itself.
k3s can upload its own snapshots to S3 or any S3-compatible store. Its settings go through as written, with the keys read from environment variables when you run tow, never from the file:
spec:
kubernetes:
k3s:
server:
extraArgs:
etcd-s3: true
etcd-s3-endpoint: s3.eu-west-3.amazonaws.com
etcd-s3-region: eu-west-3
etcd-s3-bucket: my-cluster-snapshots
etcd-s3-folder: prod
etcd-s3-access-key: {env: S3_ACCESS_KEY}
etcd-s3-secret-key: {env: S3_SECRET_KEY}
They’re k3s’s own flags (https://docs.k3s.io/cli/etcd-snapshot), set when the cluster is built. tow backup uploads too, since k3s etcd-snapshot save reads the same settings.
kubeadm: copy ~/.tow/backups/<cluster>/, or the machines’ /var/lib/tow/snapshots, with your own backup tool.
Restoring
tow restore --name <cluster> puts the cluster’s etcd back to its newest copy on this machine, or the one --snapshot names, on the same machines: lifecycle says how, and what can be restored.
The machines’ own scheduled snapshots have no .json beside them, so restore doesn’t offer them: restore one with k3s’s own --cluster-reset --cluster-reset-restore-path (https://docs.k3s.io/cli/etcd-snapshot), or etcd’s etcdutl snapshot restore for kubeadm (https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#restoring-an-etcd-cluster).
App data
etcd snapshots don’t hold your volumes’ data. Use your applications’ own tools, installed through addons.helmCharts:
8 - What tow changes on your own machines, and how it's undone
On machines you list yourself (provider: byoh), tow records every change it makes in /etc/tow/prepared.json on each machine, readable by root only. Each entry says which cluster made the change, what it did, how to undo it (undo, for you), and the exact script tow runs to undo it (revert):
sudo cat /etc/tow/prepared.json
A machine’s packages or settings that were there before tow are never recorded, so they’re never undone.
What tow delete does
| Command | Kubernetes | The preparation |
|---|
tow delete --name lab | removed | stays; in a terminal, delete asks first (default: no) |
tow delete --name lab --reset | removed | undone |
tow delete --name lab --yes | removed | stays: undoing it unattended takes --reset |
tow delete --name lab --forget | stays | stays: nothing is touched |
- Before anything changes, delete reads each machine’s record and lists, machine by machine, what goes and what stays.
- A machine it can’t reach stops it before anything changes.
--forget is for a machine that’s gone for good. - When an undo fails, the cluster is kept. Run
delete again: what was undone left the record and isn’t done twice. - Volumes on the machines’ disks (local-path, OpenEBS, Longhorn) hold your data, so delete refuses a cluster that has some, and lists them.
delete -v deletes them first, through Kubernetes, so each storage driver deletes its own data.
What’s recorded, and how each is undone
Kubernetes (removed by every delete but --forget):
| Change | Undone by |
|---|
| k3s | k3s’s own uninstaller: k3s-uninstall.sh on control planes, k3s-agent-uninstall.sh on workers. It also removes k3s’s network interfaces and rules, and its data, /var/lib/rancher/k3s. |
| kubeadm’s init or join | kubeadm reset -f, then /etc/cni/net.d (kubeadm leaves it, its docs say) |
| kubelet, kubeadm, kubectl | purged with cri-tools and kubernetes-cni, then their apt repository and key (on RHEL-family machines: dnf remove, then /etc/yum.repos.d/kubernetes.repo) |
| containerd and runc | stopped, binaries and service removed, /var/lib/containerd deleted; a config.toml that was there before tow comes back |
| CRI-O | stopped, purged, its repository and settings removed |
The preparation (undone by --reset):
| Change | Undone by |
|---|
| packages (conntrack, open-iscsi…) | apt-get remove (on RHEL-family machines, dnf remove) of those packages only: no purge, no autoremove |
| kernel modules | tow’s lines removed from /etc/modules-load.d/k8s.conf; the file goes when it’s empty |
| kernel settings | tow’s lines removed from /etc/sysctl.d/k8s.conf; the file goes when it’s empty |
| swap | the lines tow commented out of /etc/fstab (# tow: ) come back, swap units are unmasked, dphys-swapfile is enabled again, swapon -a |
| SELinux (kubeadm, RHEL family) | enforcing again, now (setenforce 1) and in /etc/selinux/config, as far as it enforced before |
| ufw rules | the rules commented tow:<cluster> are deleted. ufw stays on or off as it was: tow never turns it on. |
| firewalld | its zone tow-nodes and policy tow-admin are deleted (with firewall-offline-cmd when firewalld isn’t running). firewalld stays on or off as it was. |
| NetworkManager (Calico) | /etc/NetworkManager/conf.d/calico.conf removed, NetworkManager reloaded |
| multipathd (Longhorn) | /etc/multipath/conf.d/longhorn.conf removed, multipathd restarted |
Each undone change leaves the record; once nothing is left in it, the file and /etc/tow go too.
What stays until the machine restarts
- Kernel modules and kernel settings: they stay loaded and set; the next boot no longer applies them.
- On kubeadm, kube-proxy’s iptables rules and the network’s interfaces (
cni0, flannel.1, vxlan.calico…). kubeadm’s docs say the rules are harmless left as they are. k3s’s uninstaller removes its own. - kube-vip’s virtual IP, if kube-vip was stopped before it could let the address go.
Doing it by hand
For a machine tow can no longer reach, or after --forget, use the undo of each entry in the machine’s record, newest first, Kubernetes first. As root:
# k3s
/usr/local/bin/k3s-uninstall.sh # control plane
/usr/local/bin/k3s-agent-uninstall.sh # worker
# kubeadm
kubeadm reset -f && rm -rf /etc/cni/net.d /etc/tow/kubeadm-done
apt-mark unhold kubelet kubeadm kubectl
apt-get purge kubelet kubeadm kubectl cri-tools kubernetes-cni
rm /etc/apt/sources.list.d/kubernetes.list /etc/apt/keyrings/kubernetes-apt-keyring.gpg
# (RHEL family instead: dnf remove kubelet kubeadm kubectl cri-tools kubernetes-cni && rm /etc/yum.repos.d/kubernetes.repo)
systemctl disable --now containerd
rm -f /usr/local/bin/containerd* /usr/local/bin/ctr /usr/local/sbin/runc /usr/local/lib/systemd/system/containerd.service
rm -rf /etc/containerd /var/lib/containerd && systemctl daemon-reload
# the preparation: see each entry's "undo"; for the firewall
ufw status numbered # then 'ufw delete <number>' for each rule marked tow:<cluster>, highest number first
firewall-cmd --permanent --delete-zone=tow-nodes && firewall-cmd --permanent --delete-policy=tow-admin && firewall-cmd --reload
# last
rm -rf /etc/tow
Volume data folders are left for you to delete once you’re sure you don’t need them: /opt/local-path-provisioner (local-path on kubeadm), /var/openebs/local (OpenEBS), /var/lib/longhorn (Longhorn).
9 - Testing tow
Every way to test tow, from the cheapest to the most expensive. Run the commands from the repository root, in Linux, macOS or WSL.
| Test | Command | Needs | Time | Cost |
|---|
| Unit tests | make test | Go | ~10 s | free |
| Local k3d clusters | make test-integration | Docker, k3d | ~6 min | free |
| A kubeadm cluster on AWS | make test-aws | AWS credentials | ~10 min | ~$0.03 |
| An HA cluster on AWS (opt-in) | make test-aws-ha | AWS credentials; two NLBs, outside the free tier | ~25 min | ~$0.07 |
Your own machines (byoh) with k3s then kubeadm, each deleted with --reset, on AWS | make test-byoh-aws | AWS credentials | ~30 min | ~$0.05 |
| Proxmox VMs | make test-proxmox | a Proxmox server and token | ~6 min with k3s, more with kubeadm | free |
| A cluster’s whole life on Proxmox | make test-lifecycle DIST=k3s (or kubeadm) | the same | about an hour, unattended | free |
| A kubeadm cluster’s life on AWS | make test-aws-lifecycle | AWS credentials | ~45 min | ~$0.04 |
| A kubeadm cluster on GCP | make test-gcp | a GCP project (gcp.md) | ~15 min | ~$0.02 |
| An HA cluster on GCP (three control planes, no worker) | make test-gcp-ha | the same | ~30 min | ~$0.06 |
| A kubeadm cluster’s life on GCP | make test-gcp-lifecycle | the same | ~45 min | ~$0.05 |
| A kubeadm cluster on Azure | make test-azure | an Azure subscription (azure.md) | ~15 min | ~$0.03 |
| An HA cluster on Azure | make test-azure-ha | the same | ~30 min | ~$0.08 |
| A kubeadm cluster’s life on Azure | make test-azure-lifecycle | the same | ~45 min | ~$0.07 |
| Kubernetes conformance on an upgraded cluster (opt-in) | make test-lifecycle-conformance | a Proxmox server and token | 3 to 4 h, unattended | free |
| Every Proxmox case, one at a time (the matrix) | make test-proxmox-all | the same | 5 to 6 h, unattended | free |
| Kubernetes conformance (opt-in, what CNCF’s certification uses) | make test-conformance | the same, plus a cluster with two workers | ~2 h | free |
| Secrets scan | make secrets | gitleaks | seconds | free |
When to run them:
make test runs before every commit. CI also runs it, with Go’s race detector, together with the k3d test, the OpenTofu module checks and the secrets scan, on every push.- The race detector needs a C compiler (
sudo apt install gcc). On a machine with little memory, leave out internal/core and internal/providers/tofu: under -race they rebuild the AWS SDK, which takes about 3 GB. For example: go test -race -p 1 $(go list ./internal/... | grep -v -e internal/core -e providers/tofu). - The tests that create machines (AWS, GCP, Azure and Proxmox) use your
tofu (1.12 or a newer 1.x) when you have one; otherwise they download the release tow was tested with (sha256 checked) into a throwaway folder. Tests never ask, so they never install anything into ~/.local/bin. - The AWS, GCP, Azure and Proxmox tests create real machines, so CI never runs them. Run them yourself when you change provisioning, byoh or
internal/remote, and before merging a phase. - Several can run at once (each from its own test binary, not two
make targets in one folder, since they share integration/integration.test). A new AWS account allows 8 vCPUs of running instances, so two AWS tests at a time; ask for more under Service Quotas → EC2 → “Running On-Demand Standard instances”.
Every test that creates machines also removes them:
delete runs even after a failure, on Ctrl+C, or when the test hits its own deadline.delete then checks that nothing tagged for the cluster is left, and fails if something is.- On AWS, GCP and Azure, the test then waits up to 5 minutes for the cloud to report every machine gone.
Exit codes of the cloud targets (make test-aws, test-aws-ha, test-aws-lifecycle, test-byoh-aws, and the test-gcp* and test-azure* ones):
| Code | Meaning |
|---|
| 0 | the checks passed and every machine is gone |
| 1 | a check failed; every machine is gone |
3 (make prints Error 3) | the cloud may still have machines from the run up, or delete failed, whether the checks passed or not. Not a test failure: the last lines name each machine and the command to shut it down (aws ec2 terminate-instances ..., gcloud compute instances delete ..., az group delete ...). They also go away by themselves within the hour (AWS: they shut down; GCP: Compute Engine deletes them, from their maximum run duration; Azure: auto-shutdown deallocates them, but their disks and addresses stay until the resource group is deleted). |
go test turns every failing exit code into 1, so these targets build the test binary and run it themselves.
If a test dies anyway, see If a test dies midway.
Setup, once per clone
go version # 1.26 or newer
tofu version # optional: 1.12 or a newer 1.x (the tests that create machines download it otherwise)
k3d version # local k3d test only, with Docker running
make hooks # scan every commit for secrets (needs gitleaks)
ls ~/.ssh/id_ed25519.pub || ssh-keygen -t ed25519 # the clouds and Proxmox install this key on their machines
1. Unit tests
make test # everything, no network or cloud needed
go test -short ./internal/config/... # one package
go test -short -run TestProxmox ./... # tests whose name matches
What they cover:
internal/config:- every file in
examples/ parses, including the custom-* ones without a flavor - the order defaults apply in: the file, then the flavor, then the distribution
- each flavor’s components, and nothing optional without one (except flannel on local clusters)
- a file with every field is accepted, without warnings
- custom architectures are accepted (2 control planes with Cilium on k3s, kubeadm with flannel, several worker pools)
- fields a provider doesn’t use are refused, per provider
- each validation error, for byoh, AWS, Proxmox and local, and the warnings: components per distribution, provider and CNI, options for a component that isn’t selected, address pools that overlap anything, labels and taints, Helm charts and manifests
extraArgs: plain values, lists that repeat a flag, {env: VARIABLE}, and flags that conflict with a field or with what tow sets- credentials are refused in the file (by key in tow’s own fields, by flag in
extraArgs, Secret references allowed in chart values, and any value shaped like a secret anywhere) - the fingerprint used for drift detection: pools, machines, taints, charts and manifests keyed by name, sets in any order, lists whose order matters kept in order
- what state keeps of each example file reads back as the same config
internal/k3s:- the file becomes k3s’s own flags: none for k3s as it ships,
--disable for every bundled extra not asked for, per-pool labels, taints and kubelet flags, role labels left to the API - the install, against fake machines:
- order: the first control plane, then the other control planes one by one, then the workers
- each machine’s
config.yaml: the plain password on the first control plane, the K10… tokens with the CA’s hash on joining machines, the same worker password on every control plane, tls-san, each pool’s flags, YAML lists and quoting - the installer from the release’s tag
- the record in
/etc/tow/prepared.json - a second run changes nothing, nor does a file whose labels or taints changed (k3s reads them only when a node registers)
- k3s installed by someone else, for another cluster, or in another role is refused before anything changes
- a failure shows the end of the k3s log
- the user’s own
{env} tokens - the kubeconfig points at the API address
- kube-vip: its DaemonSet goes on each control plane before k3s starts there, never on workers, with the interface holding the machine’s address; the virtual IP is in
tls-san and the kubeconfig; control planes whose interfaces are named differently, and a virtual IP off their network, are refused before anything changes
- the dry run shows no secret, and its lines survive the output’s redaction
- the scripts parse with
sh -n, and the read-only check runs in a local shell
internal/kubeapi: waiting for nodes to join and be Ready, and role labels, against a fake API; labels and taints set and removed one by one, the others kept.internal/tools: finding helm against fake helms on PATH and in ~/.local/bin:- supported releases are used as they are
- a missing or unsupported helm is installed only after a yes, or with
--install-tools; with nobody to ask, tow stops - the user’s own helm stays as it was, and replacing the one in
~/.local/bin names both releases - a download whose sha256 doesn’t match installs nothing; an unknown platform is refused
- a dry run installs nothing
INTEGRATION=1 also downloads the real pinned helm and tofu into throwaway folders (part of make test-integration)
internal/tools (tofu): 1.12 or a newer 1.x fits; the pinned release is one of them, with a sum for linux and darwin, amd64 and arm64 (OpenTofu’s SHA256SUMS, its GPG signature checked when pinned); the download URL; tofu version’s output read.internal/addons (CNIs):- Calico: CRDs rendered by
helm template and applied with kubectl apply --server-side, with the CRD API version that matches the cluster’s release; the operator with the pod range as its IP pool; VXLAN without BGP by default, the other encapsulations with BGP; eBPF adds the API-address ConfigMap and the BPF dataplane. - Cilium: the pod range, kube-proxy replacement with the API address, WireGuard, and one operator replica on a one-node cluster.
- Both: failures show the tool’s error, and the dry run’s commands.
internal/tools (kubectl): only the cluster’s minor release fits (any patch: one already on PATH is used); otherwise the newest patch of that minor, as dl.k8s.io names it, checked against the sha256 published beside it. A dry run fetches nothing; a named release of another minor, a sum that doesn’t match, or a file that holds none, installs nothing. With INTEGRATION=1, real downloads of the newest 1.37 and 1.34.internal/addons: the helm command line for https and OCI charts; values and each secret in a file only the user can read, deleted afterwards; secrets never on the command line; the first failure stops the rest, with helm’s own error; the dry run’s commands.internal/addons (running cluster): the objects a manifest makes, read from kubectl’s JSON (a list or a single object); a manifest is compared with what runs by kubectl diff and a server-side dry run, which change nothing (diff’s exit 1 is a difference, 2 an error); applying it creates its namespace first and returns what it made; objects are deleted by name, one already gone being fine; Helm releases listed in every status with flags Helm 3 and 4 both take.internal/addons (built-in add-ons on a running cluster): each built-in add-on as a unit: what it is (storage, the default StorageClass, a service load balancer, an ingress controller), where its version is read (its Helm release, or its Deployment’s image for those installed from a manifest), and what turning it off runs: Longhorn’s deletion confirmation first, the objects it applied (never its namespace that way), its release or manifest, its own namespace (never kube-system), the Proxmox CSI plugin’s node labels; the default class’s annotation removed. A unit keeps the version that runs unless the file pins one. Two units are the same when they run the same commands with the same input; a StorageClass whose settings changed is made again; what a unit no longer applies is known (MetalLB from L2 to BGP drops its L2Advertisement). Cilium’s settings compare the same way, its version kept. Versions read from helm list and image tags, and compared.internal/addons (catalog): every add-on of the first batch on kubeadm, in order (storage, the default StorageClass, kubelet-csr-approver then metrics-server, the charts, the manifests), each command checked: local-path’s manifest at its tag and its rollout; OpenEBS with only hostpath on; Longhorn’s replica count and no default class of its own; the NFS and SMB StorageClasses as their examples, the SMB credentials in a Secret and never in the dry run; kubelet-csr-approver allowing the nodes’ names and addresses only; each manifest’s namespace created, then the manifest applied with a kubeconfig copy whose context names it, a folder recursively; no values or kubeconfig copy left behind. On k3s, its bundled local-path and metrics-server install nothing. A failure names the add-on with the tool’s output.internal/addons (service load balancers): MetalLB’s privileged namespace, chart and objects (the pool, then an L2 announcement, or BGP peers in v1beta2 and a BGP announcement); kube-vip-cloud-provider at its tag with the pool as ranges, plus kube-vip’s own DaemonSet only when no kube-vip serves the API; Cilium’s pool (cidr and start/stop blocks) and L2 policy or BGP objects, and its values (L2 announcements or the BGP control plane). Every manifest decodes as YAML.internal/kubevip (Services): the API’s kube-vip with Services on (its own Lease); the Services-only DaemonSet with its account and role, no interface given.internal/providers: a pool entry as the fewest CIDRs (ranges, CIDRs, single addresses, the address space’s edges).internal/prepare (pool): ufw lets this machine reach the pool’s addresses, as CIDRs.internal/addons (ingress, provider storage):- Traefik on kubeadm: on a pool address with a load balancer, or as a DaemonSet on every node’s ports 80 and 443 without one; default class only when the file says so; k3s’s own left alone.
- Cilium’s Ingress and Gateway values, with Gateway API’s definitions (v1.6.3) applied before Cilium.
- EBS: controller on the control planes, volumes tagged with the cluster, the gp3 class.
- Proxmox CSI: nodes labeled with region and zone, privileged namespace, its token and class in the chart’s values, the token never shown; a malformed token refused.
internal/config (ingress, ports): only traefik and cilium in ingress.controllers; the others refused with a pointer to addons.helmCharts; Cilium’s ingress needs its kube-proxy replacement and a load balancer; access.ports checked (range, duplicates; refused locally); byoh takes adminCIDRs.internal/prepare (ports): 80 and 443 for ingress and access.ports open to the admin addresses: this machine ($ADMIN), or each of the file’s adminCIDRs.internal/providers (Proxmox): with the Proxmox CSI plugin, a server that isn’t in a cluster is refused, with pvecm create; the module takes open_ports. OpenTofu: both modules validate with open_ports, the pool’s CIDRs, and (AWS) the EBS role on control planes only and the metadata hop limit.internal/kubeapi (volumes): cloud volumes found by their driver; deleting them takes the pods using their claims first, then the claims, and waits until every volume is gone (one that stays is named).internal/kubeapi (machines leaving): a Node deleted (one already gone is fine), the local volumes tied to a node.internal/prepare (machines leaving): undoing Kubernetes and the cluster’s firewall rules, the rest of the preparation kept.internal/providers/tofu (machines): the guard lets a run create and destroy exactly the machines it names (a destroyed one with its firewall and load balancer attachments), and create again a machine of the cluster’s that’s missing; nothing else; the machines a plan creates are named; a new machine’s address loses any old host key.internal/kubeapi (add-ons in use): LoadBalancer Services, the Ingresses a class serves (and those naming none when it’s the default), a Deployment’s image (none when it’s missing).internal/config (backups): cron schedules become systemd’s OnCalendar, each checked with systemd-analyze calendar where it’s installed (a weekday range from Sunday split, as systemd’s week starts on Monday); refused for kubeadm’s timer: a day of the month and a weekday both set, a step on a range or on weekdays; refused for both: a field out of range, a backwards range, a step of 0, a schedule without 5 fields; keep defaults to 5, from 1 to 100; none on a local cluster; k3s’s own snapshot flags in extraArgs conflict with the field.internal/k3s (backups): the file’s schedule and keep become etcd-snapshot-schedule-cron and etcd-snapshot-retention; without one, no snapshot flag.internal/kubeadm (backups): the snapshot timer goes on every control plane (never a worker), for containerd’s or CRI-O’s socket, checked by sh -n, recorded once in prepared.json with its undo; a new schedule is written over it without a second record; taking it off leaves the snapshots. The snapshot script runs in a real shell with a stand-in crictl: each run adds a root-only snapshot and keeps the newest keep (all of them with no schedule), nothing left in etcd’s folder.internal/core (backups): backup takes a snapshot on the first control plane that answers, copies it to ~/.tow/backups/<cluster>/ (folder 0700, files 0600) with its .json, and lists it after the cluster’s state is gone; a copy that doesn’t match its sha256 isn’t kept; no control plane answering names each one’s error; a local cluster has none. Output sudo prints before the script is kept out of the copy. On a running cluster, a changed schedule is refused on k3s, and on kubeadm puts the timer on, rewrites it or takes it off, once per change.internal/versions (upgrades): the newest patch of a minor release, from dl.k8s.io’s stable-1.N.txt and k3s’s v1.N channel; whether a release exists (HEAD, 404 means no).internal/kubeapi (upgrades): each node’s kubelet release, Ready and cordoned.internal/k3s (upgrades): a rolling install changes one machine at a time inside the caller’s step (drain before, checks after), control planes first; a machine already on the release is skipped (a resumed upgrade); a failure stops the roll.internal/kubeadm (upgrades): a node goes as kubeadm’s upgrade page says, against a fake machine: the kubeadm package (its repository moved to the new minor) and nothing else first, kubeadm upgrade plan then apply --yes on the first control plane, kubeadm upgrade node elsewhere, then containerd when it doesn’t go with the new release, then kubelet and kubectl held and the kubelet restarted; run again, what’s done is skipped (kubeadm upgrade runs again: it’s idempotent); each machine’s runtime change is named beforehand, and a runtime tow didn’t install that doesn’t go with the release stops it; the control-plane check (/readyz on the machine’s own address, its containers, etcdctl endpoint health --cluster) passes sh -n.internal/addons (upgrades): Cilium moved to another release runs its pre-flight check (applied, its DaemonSet and Deployment waited for, deleted) before its chart, with upgradeCompatibility the release it was first installed at; Calico’s CRDs are applied with --force-conflicts.internal/core (upgrades): an upgrade only when the file names a newer release (empty, stable, the same release in k3s’s two spellings: none); refused: a downgrade, a skipped minor (naming those between), other changes in the same apply (the versions of what tow installs aside), a release that doesn’t exist, an add-on moving backwards or further than its docs allow (naming the release to pin); kubeadm: the runtime changes in the plan, the first control plane with upgrade apply, CoreDNS spread again after; the CNI and built-in add-ons move to this tow’s releases after the nodes, Cilium’s first release recorded; the plan: a snapshot first (marked beforeUpgradeTo), then control planes, then workers, each drained, upgraded, Ready at the release, the API checked on every control plane after a control plane, uncordoned; a dry run changes nothing; a run that stops names the node and those already upgraded; the file going back while nodes run the newer release is refused; the next run finishes the node it left cordoned, takes no second snapshot, records the release; apply and status note a newer patch (silent when the channel doesn’t answer); status shows nodes on different releases.internal/core (settings): k3s’s new settings are planned per role and pool (flags added and removed, a secret named by its variable), then roll: each machine restarts k3s with them, a control plane checked before the next, nothing drained; refused: k3s’s critical settings (and data-dir), on servers or agents; turning k3s’s local-path off with volumes is refused, Traefik off names its Ingresses. kubeadm: the first control plane uploads the configuration and writes its changed static pods (a new certificate for new certSANs), each kubelet gets its new flags and configuration (metrics-server on: serving certificates), node by node, each checked; metrics-server and its CSR approver follow.internal/kubeadm (settings): a control plane’s config for kubeadm’s phases is init’s without any token (kube-proxy left out as at init); the phases follow what changed, in order (upload, certificate set aside and made again, static pods), passing sh -n; kubeadm-flags.env keeps kubeadm’s own flags while tow’s change; kubelet flags carry secrets’ values.internal/core (certificates): the API server’s certificate is read from its TLS handshake and the kubeconfig’s from its data; status shows both, and when one is near; within 30 days (or --renew-certificates) apply renews: k3s rotates on each control plane then restarts its workers, kubeadm renews on each control plane and restarts its static pods (the script passes sh -n), each checked; the kubeconfig is fetched again, keeping its context name and address; with an upgrade, only a note.internal/k3s (restore): k3s stops on every control plane, the first one is reset to the snapshot (its old reset-flag removed first, --etcd-s3=false when the file uploads to S3) and starts, then each other one moves its etcd data aside and joins, one at a time; one that can’t stops it there; the scripts pass sh -n.internal/kubeadm (restore): etcdutl is read from each control plane’s etcd and the snapshot’s etcd release checked against it (another minor refused); then etcd and the API servers stop on every control plane before any member is restored, each with its own name and address, the whole cluster, a new token, --bump-revision and --mark-compacted, then they start again; kube-vip’s static pod stays; the scripts pass sh -n.internal/core (restore): the newest backup by default, or the one named (with or without .db, or its path); only a snapshot of the running release or the pre-upgrade one, of this cluster and distribution; a dry run, or nobody to say yes, changes nothing; a safety backup first unless --no-backup; a copy whose sha256 changed is refused; k3s gets the file’s data-dir and S3 setting; kubeadm: the snapshot to every control plane, checked, restored, the control planes checked, every kubelet restarted, cleaned up; two backups in the same second never share a file.internal/core (delete -v): a cluster with cloud volumes isn’t deleted without -v, the volumes named and nothing changed (also when they can’t be listed); with -v they’re named in the confirmation and deleted before the machines. On your own machines, local-path, OpenEBS and Longhorn volumes count too.internal/prepare (undo): each revert script runs in a real shell on a temporary /etc, with stand-ins for ufw, apt and systemctl: swap’s /etc/fstab lines come back, only tow’s lines leave k8s.conf (the file goes once empty), only the cluster’s ufw rules go (not lab2’s, IPv6 ones too), only packages still installed are removed. Undo removes Kubernetes first, then the preparation when asked, newest first; what’s undone leaves the record even after a failure, so a second run picks up where it stopped; entries without a script and other clusters’ stay; an empty record is removed. A record that can’t be read, or a machine that can’t be reached, is an error naming it.internal/core (delete on your machines): Kubernetes removed and the preparation kept, both listed machine by machine with what must be undone by hand; --reset; the question asked only without --reset or --yes, and only when there’s preparation to undo; a machine that can’t be read stops everything before any change, a failed undo keeps the state, both pointing to --forget; --forget touches no machine; --reset and --forget refused on clusters tow created the machines of.internal/core and internal/kubeapi (status): healthy only when the machines run, the API answers, every node is Ready and no pod is stuck; a node down, a stuck pod, an API that doesn’t answer are each shown; a long list of stuck pods is cut. Stuck means crashing, an image that won’t pull, unschedulable, failed; a failed pod a controller replaced, or of a Job that succeeded since, isn’t.internal/prepare (Longhorn): multipathd kept off Longhorn’s volumes with Longhorn’s blacklist, written once, with its undo; refused where multipath manages devices.internal/kubecfg: a kubeconfig is renamed to the file’s context, merged into $KUBECONFIG next to other clusters, and removed again; files are owner-only; a context added without moving the user off their current one, unless it’s the same cluster’s old name.internal/state: a second run on a cluster stops at once (the lock), another cluster isn’t held up, and a lock alone isn’t a cluster.internal/secretenv: secret variables never reach the programs tow starts; a value read is hidden from all output.internal/providers/k3d: the k3d command for each setting; a dry run shows $VARIABLE, never a secret.internal/core:- apply, status and delete with fake providers
- a failed cluster is retried
- a failed cluster whose machines may run Kubernetes keeps them (the guard), even after a refused run
- every field of the file has a change class (a walk over the config’s types), and every class names a real field
- a running cluster: changes it can’t follow are refused first, with the reason (a new or renamed pool counts once, a local cluster refuses all but its own charts and manifests); the plan shows the file’s changes, then infrastructure, then cluster
- the file’s own Helm charts on a running cluster: one added, changed, or whose release is gone is installed (
helm upgrade --install), one that left the file uninstalled after everything else, an unchanged one left alone; a release moving namespace is refused - machines added and removed on a running cluster: OpenTofu’s first run names the new ones (and keeps the leaving ones), every machine is prepared for them, then k3s or kubeadm joins them and they’re waited for; the leaving ones go last, one at a time: drained (with
--drain-timeout), a kubeadm control plane out of etcd, Kubernetes removed, the Node deleted, then OpenTofu’s second run names them; on the user’s own machines only the new ones are checked, and one leaving keeps its preparation but not the cluster’s firewall rules (all of it with --reset); state follows each machine; a drain that doesn’t finish stops the run with the machine kept - a machine deleted outside tow is made again under its name: its Node (and a kubeadm control plane’s etcd member) cleared first, then it joins; never the first control plane; refused before anything changes: the first control plane leaving, a machine changing role or pool, control planes added where the API’s address would change (kubeadm without an address in front, AWS’s single control plane)
- the built-in add-ons on a running cluster, compared at the version that runs: a load balancer the file replaces goes first, Cilium’s new settings next, then what’s turned on or changed in install order (a StorageClass whose settings changed made again), the file’s own charts and manifests, and what’s turned off last; an unchanged file changes nothing; a changed add-on keeps the version that runs, one whose release is gone is installed again; refused before anything changes: another version of one that runs, storage that still has volumes, k3s’s bundled add-ons, kube-vip’s Services while it serves the API, Calico’s encapsulation; the plan names the LoadBalancer Services and Ingresses a removal affects; a dry run changes nothing;
apply and status name the add-ons this tow installs a newer release of, unless the file pins theirs - the file’s own manifests on a running cluster: each compared with what runs and applied again when it differs; the objects it dropped, and those of a manifest that left the file, deleted before any chart is uninstalled; state keeps what each one made; one whose namespace was deleted by hand is applied again, and only what it no longer has is deleted, never what it just made; its objects moving namespace is refused, writing the default namespace out isn’t a move
- labels and taints follow the file through the API, only those the file has or had, the control-plane taint with
schedulable; a node tow didn’t make is reported, never touched - a new kubeconfig context renames the cluster’s kubeconfig here and in yours
- an unchanged cluster’s infrastructure is planned again: up to date, or put back after a yes (
--yes, never in a dry run) - every apply checks the machines as preparation does; what they need is planned and made under the plan’s yes, preparation not asking again
- when this machine can’t reach the machines while OpenTofu has changes (its address moved), they’re planned and changed right after OpenTofu, in the same run; with nothing for OpenTofu to do, that’s an error
- the tofu a provider runs is found, or installed after a yes, before its PreFlight: when apply creates a cluster, checks a running one, and when delete destroys one;
--install-tools installs without asking, a dry run only says what it would install, and no tofu stops everything before PreFlight - a run that stops says what it had done, and doesn’t record the file; two runs on one cluster: the second stops at once
- status names nodes tow didn’t make, and machines that aren’t nodes
- mode switches are refused
- a machine can only be in one cluster
--dry-run creates nothing- what this version doesn’t install yet is listed by name
internal/providers/tofu:- every target’s module has the shared inputs and outputs, and a lock file from registry.opentofu.org with checksums for linux and darwin, amd64 and arm64
- without a tofu tow works with, PreFlight stops before any cloud call and says how to get one
- module files are written with mode 600; every machine reaches the module with every field, named as always, control planes first
- the plan summary
- the guard: a replaced or deleted resource (a tainted machine, an attribute that can’t change in place, anything no longer asked for) is refused once Kubernetes may run on the machines, and a missing machine on a running cluster
- apply runs the plan the guard checked, and nothing after a refusal
- machines keep their place from the last inputs; a running cluster keeps its test expiry, and this machine’s address is detected every time: when it moved, the plan says so first
- the guard lets a run that changes the admin addresses or the ports replace or delete the admin firewall rules, and nothing else (never the rule between nodes, never a machine)
- delete rechecks leftovers
- the target checks stop apply before OpenTofu runs
- only a target’s own package may import its cloud SDK
internal/providers/tofu/aws:- a new cluster is spread round robin over its zones (every shape of up to three control planes and two pools)
- adding machines moves none of the others; a new one goes where its pool has the fewest
- instance types: only zones that offer them, a misspelled one, types with no zone in common
- a cluster whose machines all have a zone asks AWS nothing
internal/providers/tofu/proxmox:- against a fake Proxmox API: token problems, an unknown node, a missing template, a self-signed certificate
- only the cluster’s own VMs count for status and leftovers
- a new cluster gets the range in order; later a VM keeps its address, and a new one gets the lowest free
- addresses already in use are refused
internal/providers/byoh: the checks each machine must pass (Ubuntu or Debian, or RHEL 9 or 10 and their rebuilds; CPU, memory, sudo, ports), run against local scripts; a port ufw or firewalld blocks before preparation is a warning naming it.internal/prepare (RHEL family): a system’s family from /etc/os-release (RHEL, Rocky, AlmaLinux, any that says it’s like RHEL; 8 and older, Fedora and machines without the file refused); packages by their name on each family, asked of dpkg or rpm, installed with dnf, and removed by their record against stand-ins for rpm and dnf; kernel-modules-extra for k3s on RHEL 10, built for the running kernel.internal/prepare (firewalld): with firewalld on, the zone tow-nodes (target ACCEPT) takes the other nodes and the pod and Service ranges, and the policy tow-admin (from any zone to the machine) the admin rules as rich rules, $ADMIN filled in on the machine; once there, nothing more; a source that left and an old admin address are removed, and no admin rule when this machine’s address isn’t known; refused: a source firewalld already has in another zone, firewalld older than 0.9; Calico warns. Rich rules are compared as firewalld writes them (a single address with or without /32). The scripts run against a stand-in firewall-cmd (and firewall-offline-cmd for the undo with firewalld stopped), and the facts script reads a stand-in firewalld.internal/kubeadm (RHEL family): against fake Rocky machines: pkgs.k8s.io’s rpm repository written as kubeadm’s install page writes it (its exclude line holding the packages), dnf installing the release with the exclude lifted (and without weak dependencies on RHEL 10), containerd as on Ubuntu, CRI-O from its rpm repository; records that undo with dnf; a second run changes nothing; packages of another release refused with dnf’s command; an upgrade moves the repository file to the new minor and takes kubeadm, then kubelet and kubectl, to the release.internal/prepare (SELinux): kubeadm sets SELinux to permissive now (setenforce 0) and at boot (/etc/selinux/config), only what enforced, and the undo enforces again as it did; k3s leaves it alone.internal/k3s (SELinux): a machine whose SELinux enforces gets selinux: true in its config.yaml (once, even when the file sets it), the others don’t; a custom data-dir is refused there before anything changes.internal/config (systems): spec.infrastructure.os defaults to ubuntu-24.04 on aws, with the image’s login user (ubuntu, ec2-user) unless the file names one; an unknown system, and the field on proxmox, byoh or local, are refused; proxmox.cpuType defaults to x86-64-v2-AES, and a value that isn’t a CPU model’s name is refused.internal/providers/tofu (systems): the AWS module gets the file’s system (Ubuntu by default) and picks the newest image of it for each CPU architecture; Proxmox VMs get the file’s CPU model, or the module’s default.internal/prepare: what each distribution and component needs (from their docs), compared with what a machine has: packages, kernel modules and settings, swap, NetworkManager for Calico, ufw and firewalld rules; refusals (other systems, old kernels for Calico and Cilium, no memory cgroup with the Raspberry Pi fix, cloned MAC or product_uuid); the question before changing your machines; a failed change stops with the tool’s output; the record on each machine. The generated scripts are checked with sh -n, and the read-only check runs in a local shell. ufw rules are compared as ufw reads them (any word order, a single address with or without /32, the short form), and the cluster’s rules nothing needs any more are closed: never another cluster’s, the user’s own, or a rule in a form tow doesn’t add, and nothing when this machine’s address isn’t known.internal/runtimes: containerd’s support table (which releases go with which Kubernetes minor, 2.0 or newer only), CRI-O’s minor matching Kubernetes’, the pinned containerd and runc sums, and the embedded containerd unit being containerd’s own, byte for byte.internal/kubeadm: against fake machines that answer its scripts: kubelet, kubeadm and kubectl from pkgs.k8s.io at the release and held; containerd from its release with the pinned sum (or the published one for another release), systemd cgroups and kubeadm’s pause image; CRI-O from its repository for the minor, its drop-in written first; a second run changes nothing; a runtime already there is used when it goes with the release; refusals before any change on any machine (Docker’s containerd.io, a release containerd doesn’t recommend, a release other than the file’s, a stopped runtime, another cluster’s, CRI-O of another minor, k3s, a machine already in a cluster, packages of another release, an unsupported architecture); a failure names the machine with apt’s output. The cluster itself: the init and join configs, compared in full (CRI socket, node IP, labels, taints and the control plane’s own, kubelet flags, extraArgs with secrets filled in, certSANs, kube-proxy skipped when replaced); every config validated before anything starts; the token in kubeadm’s form, made per run and deleted after the joins (also after a failed one); the CA hash workers check; the kubeconfig pointed at the API address; role labels; a second run changes nothing; a failed init or join (even one that left kubeadm’s files and a running kubelet) is reset and retried on the next run, with a new token, leaving finished machines alone; a failure shows kubeadm’s output, the kubelet’s log and the containers; refusals (another cluster’s node, an unknown pool). Several control planes: init uploads the certificates with a key tow makes, the others join one at a time with it (their config compared in full), then the workers; the certificates and the token are deleted after the joins, also after a failed one; a control plane that failed is reset and joins on the next run with the certificates uploaded again; the API’s certificate names every control plane; your own load balancer is tested from the first control plane before init, and a timeout stops everything. kube-vip: its static pod goes on the first control plane after any reset and before kubeadm init, reading super-admin.conf, then admin.conf once init is done, and on the others once they joined, never on workers; the virtual IP is the controlPlaneEndpoint, in the certificate and the kubeconfig; one off the control planes’ network is refused before anything changes. Every script is checked with sh -n.internal/kubevip: the static pod and the DaemonSet decode as Kubernetes objects and carry what kube-vip manifest pod|daemonset prints at the pinned release (environment, capabilities, host alias, kubeconfig mount, control planes only); the image is pinned by digest; the interface is found from ip -o -4 addr show (VLANs too, the virtual IP already held too) and a virtual IP off its network is refused; the write script runs in a local shell, writes root-only, and leaves an unchanged manifest alone.internal/core: also:- machines are prepared after they’re created or checked, with a question only for your own; a failed preparation keeps the machines and the next apply resumes without recreating them
- k3s goes on prepared machines, and the kubeconfig is saved (owner-only) and merged
- a failed install resumes on the same release, even if “stable” moved
- an existing kubeconfig context is refused before anything starts
- secret variables are read first and never saved
- kubeadm machines get their runtime and packages after preparation (whose own check now includes the runtime), then the cluster, the kubeconfig and the CNI; a failed setup or install resumes; “stable” is checked against the runtime once resolved, before anything is created; several kubeadm control planes are refused before anything happens;
StopAfter (tests only) leaves machines prepared with nothing on them - delete removes the context, and on your own machines removes Kubernetes (see “delete on your machines” above)
- Calico and Cilium go on right after Kubernetes, and flannel too on kubeadm (kubectl for Calico and flannel, matched to the cluster’s release), then every node must be Ready; a failed CNI keeps the machines and resumes; the address the CNI reaches the API on (the private one with one control plane)
- Helm charts go on every cluster, after helm is found; no helm means nothing is created; a failed chart keeps the cluster and the next apply installs the charts again (a k3d cluster the file changed is built again); chart secrets are read first and never saved
internal/remote: SSH, with host keys pinned per cluster.internal/ui: secrets are redacted from every output, including streamed OpenTofu logs.
2. Local k3d cluster
Three tests; every cluster they create is deleted at the end, whatever happens. KUBECONFIG points to a temporary file, so your own kubeconfig is never touched.
TestK3dLifecycle checks that Docker and k3d are installed, creates a bare k3d cluster named test-<number>, and waits until it reports healthy.
TestK3dFromFiles runs apply on four files under throwaway names, then runs the cluster checks through the API’s service proxy (k3d’s nodes live in Docker). When this machine has no helm, the test downloads the pinned release into a throwaway folder first, never into ~/.local/bin.
| File | Must run | Must not exist |
|---|
local-helm.yaml (flavor local + the podinfo chart) | the Helm release deployed; podinfo with 2 ready replicas, answering through its Service | |
custom-bare.yaml (no flavor) | CoreDNS; nodes Ready on flannel | traefik, servicelb, local-path, metrics-server, Gateway API CRDs |
local.yaml (flavor local) | CoreDNS, traefik, servicelb, local-path, metrics-server, Gateway API CRDs | |
| a server and a worker with a role label and a taint | CoreDNS; the label (set through the API) and the taint on the worker | traefik, metrics-server |
Each cluster’s context must be in the kubeconfig after apply, and delete must leave no cluster, context or state.
TestK3dAddonChanges changes a running cluster’s own add-ons through apply, one file after another, checked through the API and in state: the podinfo chart (1 replica) and a manifest of two ConfigMaps added; the same file again changes nothing; 2 replicas and one ConfigMap dropped from the manifest (it’s deleted, the other stays); the manifest’s namespace deleted by hand (the same file puts it back, the ConfigMap with it); both removed from the file (the release uninstalled, the ConfigMap deleted, both namespaces kept). It also downloads the kubectl of the cluster’s minor release into the throwaway folder.
Cluster checks
Every test that builds a cluster ends with the same checks (checkCluster in integration/). They run through the kubeconfig tow handed out, from this machine, so they prove it works:
- What the file asks for, and nothing else:
- every node registered under its name, with its role and the release asked for
- each pool’s labels (role labels included) and taints
- the control planes tainted or not, as
schedulable says - k3s and k3d: each bundled component asked for is running, and the others don’t exist
- kubeadm: etcd, the API server, the controller manager, the scheduler and CoreDNS run; kube-proxy runs unless the CNI replaces it; none of k3s’s extras exist
- each Helm chart the file asks for is deployed
- Calico’s, Cilium’s or (on kubeadm) flannel’s DaemonSet Ready on every node. flannel’s config has the cluster’s pod range and the file’s backend. When asked for: kube-proxy’s job taken over (Cilium’s
kube-proxy-replacement, Calico’s BPF dataplane) and WireGuard keys published by every node.
- Storage (
checkStorage): each driver’s StorageClass exists and the file’s default is the only default. In a throwaway namespace, a pod writes into a new 1 GiB volume of each class and another pod reads it back, on another node for networked storage (Longhorn, NFS, SMB). An NFS or SMB server the file names as an in-cluster Service (<name>.<namespace>.svc.cluster.local) is started first from the drivers’ own example servers, so the lab needs no NAS. - Ingress (
checkIngress): every IngressClass in the cluster (built in, or from a chart the file installs) serves an Ingress from this machine, at the address its controller publishes in the Ingress’s status; built-in Traefik without a load balancer on a node’s port 80. - Gateway (
checkGateway): every GatewayClass serves a Gateway and an HTTPRoute from this machine, at the Gateway’s address. - Firewall ports (
checkAccessPorts): a pod listening on each of the file’s access.ports (a node’s port) answers at that node’s address. - Cloud volumes and delete (
keepCloudVolume, with EBS or Proxmox disks): a volume left in use makes a delete without -v refuse, naming it; the test’s own delete then uses -v, and on AWS the leftover check proves the volume is gone. - Service load balancer (
checkServiceLB), when tow installs one: a LoadBalancer Service gets an address from the file’s pool, and answers HTTP there from this machine. - metrics-server (
checkMetrics): the metrics API reports every node. On kubeadm this also proves the kubelets’ serving certificates were signed and approved. - Manifests (
checkManifests): each manifest’s namespace exists, marked as tow’s, with the ConfigMaps a local manifest names in it. - A hello world, in a throwaway namespace that’s always deleted. It uses the Kubernetes project’s test image,
agnhost.- Two replicas run, on different nodes when two can take them.
- The test calls one replica from this machine: through a NodePort on a machine’s address, which proves the firewall rules, or through the API’s service proxy on k3d.
- That replica calls the ClusterIP Service by its DNS name until both replicas have answered. This proves pod networking across nodes, cluster DNS and Service routing.
- It then calls each replica directly by its pod IP: pod-to-pod traffic across nodes through the CNI alone.
- Status (
checkStatus): tow status says healthy once the checks’ own pods are gone. - Failover, for clusters of several control planes (
checkFailover): a control plane is powered off through the cloud’s API: on AWS the first one, on Proxmox the one holding kube-vip’s virtual IP (the holder of its Lease, kube-system/plndr-cp-lock). The API must still answer through the address in front of the control planes, the rest of the control plane must see the missing one NotReady, and a new hello world must run and answer (scheduling, DNS, Services, a NodePort on a machine that stays up). While it’s off, tow status must say not healthy, with that control plane NotReady. Then it’s powered on and must be Ready again. With kube-vip, the kubeconfig must point at the virtual IP, and another control plane must hold it afterwards. First, CoreDNS’s replicas must run on different nodes. - CoreDNS on k3s with several control planes (
checkK3sCoreDNS, Proxmox): two replicas Ready on different machines; then k3s is restarted on the last control plane, which makes it apply its own CoreDNS manifest again (seen by the Addon’s AppliedManifest event), and CoreDNS must still have two replicas.
3. AWS
Use credentials from any standard source: aws sso login, aws configure (pick one with AWS_PROFILE=<name>), or AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Check them first:
aws sts get-caller-identity
Optional settings for both tests:
TOW_TEST_REGION, default eu-west-3.
TOW_TEST_AWS_CONFIG=../examples/ebs-aws.yaml: EBS volumes, Traefik on the nodes’ ports 80/443 and access.ports through the security group, then delete with -v (~$0.03).
TOW_TEST_AWS_CONFIG, make test-aws only: a cluster file to build instead of the small dev-test one, under a throwaway name, in its own region. ../examples/storage-aws.yaml installs every storage driver, metrics-server and a manifest on three t3.small (~25 minutes, about $0.04), with the NFS and SMB test servers the checks start. ../examples/cnpg-aws.yaml runs Postgres with CloudNativePG on three workers in three zones (~12 minutes, about $0.05): the test writes a row, powers off the primary’s machine, waits for a replica in another zone to take over, reads the row back through the Service db-rw, then powers the machine on again.
TOW_TEST_INSTANCE_TYPE, default t3.small, the smallest size kubeadm accepts.
TOW_TEST_KUBERNETES, make test-aws (and make test-proxmox): YAML merged into the cluster’s spec.kubernetes, to try other choices. Its keys win; the rest of the file stays. For example:
TOW_TEST_KUBERNETES='{containerRuntime: cri-o, networking: {cni: flannel}}' make test-aws
TOW_TEST_KUBERNETES='{networking: {cni: cilium, cilium: {kubeProxyReplacement: true, encryption: wireguard}}}' make test-aws
TOW_TEST_KUBERNETES='{networking: {cni: calico, calico: {dataplane: ebpf}}}' make test-aws
Cleanup has a third layer on AWS: the test machines shut down and terminate themselves after one hour (TOW_RESOURCE_TTL=1h), even if the test process is killed.
make test-aws
TestAWSInfra runs tow apply on a dev-test cluster: kubeadm with Calico, one control plane and one worker, in a new VPC. It then logs in to each server and checks:
| Check | What it proves |
|---|
OS is Ubuntu, cloud-init finished, sudo -n true works | the server is ready for Phase 2 |
| at least 2 CPUs and ~1.7 GB of memory | kubeadm will accept it |
DNS, HTTPS out, apt-get update | it can download what Phase 2 installs |
| the metadata service refuses requests without a token (401) | IMDSv2 only, which blocks credential theft |
| a shutdown is scheduled | the one-hour self-destruct is armed |
curl and ca-certificates installed, swap off, net.ipv4.ip_forward = 1 now and in /etc/sysctl.d/k8s.conf, /etc/tow/prepared.json lists each change with its undo | the machine was prepared as the kubeadm docs ask, and it’s recorded |
kubelet, kubeadm and kubectl at the cluster’s release and held, the kubelet enabled; containerd 2.3.7 running, its config (as containerd config dump reads it) with SystemdCgroup = true and kubeadm’s pause image; crictl info answers on containerd’s socket; both recorded with their undo | the machine has what kubeadm needs, as its docs ask |
/etc/tow/kubeadm-done names the cluster, the kubelet runs, the kubeadm run is recorded with its undo; on the control plane, tow’s join token is gone | kubeadm init or join finished here, for this cluster |
| ping and TCP to the other node on its private IP | nodes can reach each other |
Then the cluster checks run, with the API and a NodePort reached on the control plane’s public address. When this machine has no helm or kubectl, the test downloads the pinned releases into a throwaway folder.
Last, it runs tow delete, which fails if AWS still lists anything tagged tow-cluster=<name>, and waits for AWS to report both machines terminated (see the exit codes above).
make test-aws-ha (opt-in)
TestAWSHA does what make test-aws does with three control planes and a worker, behind the two NLBs tow creates (a public one for you, an internal one for the nodes): kubeadm by default, k3s with TOW_TEST_KUBERNETES='{distribution: k3s}'. On top:
- kubeadm’s control planes join one at a time with the certificates init uploaded, behind the internal NLB; the cluster checks reach the API through the public NLB, and a NodePort on the worker.
- The failover check stops the first control plane through AWS, then starts it again (its self-destruct, armed at first boot only, is armed again).
The load balancers don’t destroy themselves after the hour like the machines do: if the test process dies, run the tow delete its last lines name. About $0.07 a run (two NLBs: a public one the test reaches the API through, an internal one the nodes use).
make test-byoh-aws
TestBYOHOnAWS creates two throwaway servers with the AWS module (whose apply stops once they exist: nothing prepared, no Kubernetes), then hands them to byoh as if they were your own machines, with --yes standing in for your answer. It runs three cases, in this order, on the same two servers, and ends like make test-aws: the servers deleted, then confirmed terminated. About 30 minutes, ~$0.05.
| Case | What it proves |
|---|
| blocked node-to-node port is caught | after the worker’s firewall drops the control plane’s traffic to port 10250, byoh fails with “node-to-node: can’t reach … port 10250”, and only for that port (6443 still works). The rule is removed afterwards. Then ufw is turned on on both servers (SSH allowed), as on many servers. |
| k3s, then delete –reset | with the homelab flavor, byoh finds each machine’s private IP and a node name by itself, prepares the machines (its ufw rules among the changes) and installs k3s. The cluster checks pass, with the API and a NodePort reached on the control plane’s public address. Each machine’s /etc/tow/prepared.json lists Kubernetes and the firewall with revert scripts. delete --reset -v then leaves nothing tow added: no k3s, no tow ufw rule (ufw still on), none of the packages it installed, no k8s.conf files, no /etc/tow; the kubeconfig context is gone. |
| kubeadm, then delete –reset | on the machines the k3s case handed back, with swap turned on first: the dev-test flavor (kubeadm, Calico), whose preparation turns swap off. Same checks, and after delete --reset -v no kubelet, kubeadm, kubectl, containerd or runc, no /etc/cni/net.d or /etc/containerd, swap on again and its /etc/fstab line back. |
TOW_TEST_OS=almalinux-9 make test-byoh-aws runs the same cases on AlmaLinux 9 servers, with RHEL’s tools: firewalld instead of ufw (its zone tow-nodes and policy tow-admin gone after --reset, firewalld still on), packages removed with dnf, /etc/yum.repos.d/kubernetes.repo gone, and SELinux, which kubeadm’s preparation set to permissive, enforcing again now and at boot.
4. Proxmox VE
You need a Proxmox server with an Ubuntu cloud-init template and an API token. proxmox.md covers the setup, including a Hyper-V lab on a Windows PC.
export PROXMOX_VE_API_TOKEN='tow@pve!tow=<secret>' # type or paste it in your own terminal only
export PROXMOX_VE_INSECURE=true # self-signed lab certificate only
make test-proxmox
TestProxmoxInfra builds the cluster in TOW_TEST_PROXMOX_CONFIG (default examples/dev-test-proxmox.yaml: two VMs of 2 GB) under a throwaway name.
Before creating anything, tow checks the token, the node, the template, and that no VM address is already in use.
On each VM, the test checks:
- Ubuntu, cloud-init finished, passwordless sudo
- the hostname is the VM name
- at least 2 CPUs and ~1.7 GB of memory
- DNS, HTTPS and the package mirrors are reachable
- ping and TCP to the other node
- prepared as the distribution’s docs ask (packages; for kubeadm swap off and IPv4 forwarding), with each change in
/etc/tow/prepared.json - with kubeadm: kubelet, kubeadm and kubectl at the cluster’s release and held, and the file’s container runtime (containerd or CRI-O) at its release, set up for Kubernetes, answering
crictl info, and recorded; kubeadm finished there for this cluster, as on AWS
The cluster checks run, with a NodePort reached on the first VM’s address. Files for the lab:
- kubeadm:
dev-test-proxmox.yaml (Calico), the default; other choices through TOW_TEST_KUBERNETES (AWS has examples), like '{networking: {cni: flannel}}' or '{containerRuntime: cri-o}' - k3s:
homelab-proxmox.yaml (flannel), calico-proxmox.yaml, calico-ebpf-proxmox.yaml, cilium-proxmox.yaml (kube-proxy replacement + WireGuard) - several control planes behind kube-vip:
homelab-proxmox-ha.yaml (three VMs of 2 GB, kubeadm + flannel; k3s with TOW_TEST_KUBERNETES='{distribution: k3s}'), then the failover check - storage, metrics-server and manifests:
storage-proxmox.yaml (three VMs of 2 GB, kubeadm + flannel; k3s the same way), with the NFS and SMB test servers the checks start - ingress:
ingress-proxmox.yaml (kubeadm + MetalLB: Traefik built in, F5 NGINX, HAProxy and Envoy Gateway from their charts, Contour from its manifest), cilium-ingress-proxmox.yaml (k3s + Cilium’s Ingress and Gateway), and homelab-proxmox.yaml (k3s’s own Traefik through servicelb) - Proxmox disks as volumes:
proxmox-csi.yaml, with PROXMOX_CSI_TOKEN set and Proxmox in a cluster (docs/storage.md) - service load balancers:
metallb-proxmox.yaml (kubeadm), kube-vip-lb-proxmox.yaml (k3s, kube-vip’s own DaemonSet), cilium-lb-proxmox.yaml (k3s, Cilium L2); and kube-vip in front of the API announcing Services too: homelab-proxmox-ha.yaml with TOW_TEST_KUBERNETES='{serviceLoadBalancer: {provider: kube-vip, kube-vip: {addresses: [192.168.100.200-192.168.100.210]}}}'. BGP needs a router the lab doesn’t have: unit-tested only.
When this machine has no helm or kubectl, the test downloads the pinned releases into a throwaway folder.
delete must leave no VM tagged tow-cluster.<name>, and no kubeconfig context.
Expected in a lab: a warning that the certificate check is off, and another if the Datacenter firewall is off (turn it on: the tests then prove tow’s firewall rules too). cloud-init also reports “recoverable errors”: the configuration Proxmox generates uses a deprecated user: key. The test logs this instead of failing.
production-proxmox.yaml isn’t run: its six VMs (3 × 4 GB and 3 × 8 GB, about 36 GB) need more than the lab described in proxmox.md. Run it on a real server.
make test-lifecycle: a cluster’s whole life
TestProxmoxLifecycle drives one cluster through every change a running cluster takes, the cluster checked after each stage, stopping at the first that fails (each builds on the one before). DIST=k3s: three control planes behind kube-vip (schedulable) and a worker; DIST=kubeadm: one control plane behind kube-vip and a worker, Cilium replacing kube-proxy. Both start at the previous minor release, with CloudNativePG’s operator from the file.
| Stage | What it proves |
|---|
| create | the cluster checks, as for make test-proxmox |
| re-apply unchanged | “nothing to do” |
| a worker added, then removed | the dry run shows Infrastructure, then Cluster, and replaces nothing; the worker joins, then is drained and removed |
| the control planes change | k3s: one removed and added back; kubeadm: a second added (the lab holds one more VM), then removed; an even number is warned about |
| add-ons on and off | a Helm chart (podinfo) installed, then uninstalled |
| the admin CIDRs | an extra CIDR added, then removed: the firewalls follow, this machine keeps its access |
| new settings | an API server flag and the worker’s max-pods (150): no drain |
| a marker, then a backup | tow backup |
| the upgrade | a pod pinned to the worker under a budget allowing no disruption: the drain times out (2 min), the error names the budget and the nodes already upgraded; the budget removed, the next apply skips them and finishes; every node on the new release; Postgres (CloudNativePG, two instances, asynchronous replication, the readiness probe streaming with a maximum lag, enablePDB: false as its docs advise for drains) written to every few seconds throughout: every committed row is there after, and the failed writes are counted; the snapshot taken first is on this machine, whole (0600, the size its .json records) |
| the pre-upgrade snapshot restored | the marker from before the upgrade is back, the one after it gone (kubeadm: skipped when the upgrade moved etcd’s minor release, which restore refuses) |
| certificates renewed | --renew-certificates: the kubeconfig is fetched again |
| delete | nothing left |
On kubeadm, the stages k3s’s run already covers (a worker, add-ons, the admin CIDRs, settings) are skipped, so it takes about 35 minutes.
A failed run keeps its cluster and says how to carry on: TOW_TEST_LIFECYCLE_NAME=<its name> TOW_TEST_LIFECYCLE_FROM="<a stage's title>" make test-lifecycle DIST=… reuses it and starts at that stage (the file edits of the stages before are still made, so the file matches the cluster; a stage this run never runs leaves none). Fix the test that way, on the running cluster; the full run is the last proof.
TOW_TEST_LIFECYCLE_WORKERS=2 starts the cluster with two workers: Postgres’s instances then sit on two workers drained one after the other, the case tow’s wait before each drain is for (kubeadm on the lab: three VMs, so the control-plane stage is left out).
make test-lifecycle-conformance runs the kubeadm lifecycle with two workers (the conformance tests want them) and without the control-plane stage (the lab holds no fourth VM), then, after the certificates are renewed, Kubernetes’ conformance tests as make test-conformance does: on a cluster that was upgraded and restored. The results stay in integration/conformance/.
make test-aws-lifecycle (TestAWSLifecycle) runs the stages only AWS can show, on a kubeadm cluster of two t3.small (Cilium replacing kube-proxy): create, re-apply unchanged, a worker added and removed, new settings, the upgrade, certificates renewed, delete. Adding the worker also proves OpenTofu’s part: each machine keeps its server and zone (read from its own metadata), and the new one goes to a zone none of them uses. The cluster is deleted even after a failure, unlike on the lab (TOW_TEST_LIFECYCLE_KEEP=1 keeps it; its machines shut themselves down after 2 hours anyway). TOW_TEST_REGION and TOW_TEST_INSTANCE_TYPE work as for make test-aws.
proxmox-matrix.txt has a lifecycle:k3s and a lifecycle:kubeadm line: make test-proxmox-all runs them too.
make test-proxmox-all: every case, one at a time
integration/proxmox-matrix.txt lists every case, one per line: an example file, and optionally YAML merged into its spec.kubernetes. It covers the distributions and CRI-O, several control planes (kubeadm, and k3s with kube-vip announcing Services), every network and its options (Calico’s encapsulations, Cilium as it ships, flannel’s backends, cni: none), storage with a manifest folder, Proxmox disks, the service load balancers and ingress.
export PROXMOX_CSI_TOKEN='kubernetes-csi@pve!csi=<secret>' # for proxmox-csi.yaml
make test-proxmox-all # 3 to 4 hours; nothing asks anything, so it can run overnight
make test-proxmox-all ONLY=calico # only the lines that contain "calico": to run again what failed
Each line runs TestProxmoxInfra from one test binary, built once, and a failed line doesn’t stop the others. Each line’s output goes to integration/logs/<date>/, with summary.txt: PASS or FAIL and the minutes, line by line. The exit code is 1 if any line failed. make test checks that every line is a valid cluster file, so a typo fails in seconds rather than hours later.
Runs Kubernetes’ conformance tests with Sonobuoy 0.57.5 (pinned, its archive checked against its sha256), in the certified-conformance mode CNCF’s Certified Kubernetes program uses, on a cluster TestProxmoxInfra builds: custom-proxmox.yaml by default (kubeadm, two workers: some tests need two nodes that take workloads). About two hours. Any failed test fails it. The results stay in integration/conformance/<cluster>-<release>/: Sonobuoy’s archive and results.txt, what a submission to CNCF is made from.
The cluster file, for writing your own test files
Every field, in one place. Only apiVersion, kind, metadata.name and spec.infrastructure.provider are always required; everything else has a default. flavor is optional and only fills in what the file leaves out.
This is a reference, not a file to apply: it shows the fields of every provider at once, so tow would refuse it as written (the table below says which fields go with which provider). For files that work, start from examples/.
apiVersion: towproj.github.io/v1alpha1
kind: Cluster
metadata:
name: lab # DNS label: names the machines, the state and the cloud tags
spec:
flavor: homelab # optional preset: local | homelab | dev-test | production
kubernetes: # what gets installed: exactly this, nothing else (every field: examples/custom-full.yaml)
distribution: k3s # k3s | kubeadm | k3d (local only); required when there's no flavor
version: v1.37.2 # left out: v1.37.2, the release tow tests; "stable": the newest; k3s: v1.37.2+k3s2 for a later k3s build
# containerRuntime: containerd # kubeadm only: containerd (default) | cri-o; k3s brings its own containerd
# containerd: {version: 2.3.7} # left out: 2.3.7; a release containerd recommends for the Kubernetes minor
apiEndpoint: 192.168.1.100:6443 # required with >1 control plane on byoh and proxmox; refused on aws (it makes a load balancer)
apiLoadBalancer: kube-vip # what serves apiEndpoint: kube-vip (default) | external (byoh only)
networking: # defaults: k3s/k3d 10.42.0.0/16 + 10.43.0.0/16, kubeadm 10.244.0.0/16 + 10.96.0.0/12
podCIDR: 10.42.0.0/16
serviceCIDR: 10.43.0.0/16
clusterDomain: cluster.local
cni: cilium # flannel | calico | cilium | none; default none (local: flannel); decides the ports between nodes
storage: {csi: [local-path]} # several drivers; defaultClass names the default
serviceLoadBalancer: {provider: servicelb} # exactly one; pools need addresses
ingress: {controllers: [traefik]}
metricsServer: {enabled: true}
apiServer: {extraArgs: {audit-log-maxage: 30}} # also controllerManager, scheduler, etcd, kubelet; k3s.server / k3s.agent on k3s
addons: {helmCharts: [], manifests: []}
kubeconfig: {context: lab, merge: true} # defaults: metadata.name, merged into ~/.kube/config
infrastructure:
provider: proxmox # aws | gcp | azure | proxmox | byoh | local
os: ubuntu-24.04 # clouds only: ubuntu-24.04 (default) | almalinux-9 | almalinux-10
aws: # provider aws only
region: eu-west-3
gcp: {project: my-project-123456, region: europe-west1} # provider gcp only; Application Default Credentials
azure: {subscription: 00000000-0000-0000-0000-000000000000, location: swedencentral} # provider azure only; az login or ARM_* variables
proxmox: # provider proxmox only; the token comes from PROXMOX_VE_API_TOKEN
url: https://192.168.100.10:8006
node: pve
template: 9000
storage: local-lvm # default
bridge: vmbr0 # default
cpuType: x86-64-v2-AES # default; x86-64-v3 or host for a RHEL 10-based template
network: # proxmox: all four fields; clouds: subnet only (the VPC/VNet, default 10.250.0.0/16)
subnet: 192.168.100.0/24
addresses: 192.168.100.20-192.168.100.29 # VMs take them in order, control planes first
gateway: 192.168.100.1
dns: [1.1.1.1]
access:
ssh: {user: ubuntu, keyPath: ~/.ssh/id_ed25519, port: 22} # port: byoh only
adminCIDRs: [203.0.113.7/32] # clouds/proxmox: who reaches SSH and 6443; default: this machine
ports: [8080] # more TCP ports open to adminCIDRs, for what you install yourself
controlPlane:
count: 1 # clouds, proxmox, local (k3d servers)
instanceType: t3.small # clouds: the cloud's own size names (e2-medium, Standard_B2s_v2)
resources: {cpu: 2, memory: 2Gi} # proxmox
disk: 20Gi # clouds, proxmox
machines: # byoh only, instead of count
- host: 192.168.1.10
ssh: {user: admin} # per-machine override of access.ssh
schedulable: auto # control plane only: auto = workloads here only without workers
labels: {zone: rack-1} # every pool: labels, taints, kubelet: {extraArgs}
taints: [{key: dedicated, value: control, effect: NoSchedule}]
workers: # pools; a single pool may omit its name (it becomes "workers")
- name: general
count: 2 # local: one pool only, its count is k3d's agents
instanceType: t3.large
disk: 50Gi
What each provider accepts. A field outside its column is refused, with the reason:
| Field | aws | proxmox | byoh | local |
|---|
kubernetes.distribution | k3s, kubeadm | k3s, kubeadm | k3s, kubeadm | k3d (implied) |
networking.cni | ✓ | ✓ | ✓ | flannel or none |
networking ranges, clusterDomain | ✓ | ✓ | ✓ | refused (k3d brings its own) |
| storage, ingress, service load balancer | all but proxmox storage | all but aws-* | all but aws-*, proxmox storage | k3s’s own: local-path, traefik, servicelb |
kubernetes.apiEndpoint | refused (load balancer) | ✓, required with >1 control plane | ✓, required with >1 control plane | refused |
kubernetes.apiLoadBalancer | refused | kube-vip | kube-vip, external | refused |
apiServer.certSANs | ✓ | ✓ | ✓ | refused (k3d sets them) |
infrastructure.network | subnet only | ✓ required | refused | refused |
access.ssh | user, keyPath | user, keyPath | user, keyPath, port | refused |
access.adminCIDRs | ✓ | ✓ | refused (not yet) | refused |
count | ✓ | ✓ | refused | ✓ |
instanceType | ✓ | refused | refused | refused |
resources | refused (not yet) | ✓ | refused | refused |
disk | ✓ | ✓ | refused | refused |
machines | refused | refused | ✓ required | refused |
Limits that apply everywhere:
- Sizes:
- At least 1 control plane.
- A listed worker pool needs at least 1 machine.
- Sizes are written as quantities (
512Mi, 4Gi, 1Ti); a bare number like 4 is refused. - Provisioned machines can’t be smaller than the distribution’s minimum for their role. A kubeadm control plane needs 2 CPUs, 2 GB of memory and a 14 GB disk. On AWS, instance types are checked through the API before anything is created.
- Address ranges: the pod and service ranges must not overlap each other or the machines’ addresses. Load balancer pools must not overlap them either, nor the API address or each other; on Proxmox a layer-2 pool must be inside the subnet and outside the VM range. Layer-2 pools are refused on AWS.
- Components: each one only where it works (servicelb on k3s and k3d, Cilium’s load balancer and ingress with
cni: cilium, aws-* on AWS, proxmox storage on Proxmox). Options for a component that isn’t selected are refused. Components k3s bundles take no version on k3s and k3d. - Pass-through:
extraArgs flags are checked for shape only, plus conflicts with fields of the file or flags tow sets. Helm charts need repo, chart and an exact version; local manifests must exist. - Warnings, not refusals:
- an even number of control planes
- no workers
- no CNI (nodes stay NotReady)
- several storage drivers or ingress controllers and no default class
- a secret flag whose value can still be seen somewhere (kubeadm’s process list, k3s’s log,
docker inspect on local clusters) - what this version doesn’t install yet
5. By hand, with the CLI
Build first with go build -o tow. Every cluster follows the same four steps:
./tow apply -f <file> --dry-run # shows what would happen; creates nothing, writes no state
./tow apply -f <file> # does it; state goes to ~/.tow/clusters/<name>/
./tow status --name <name>
./tow delete --name <name> # asks for confirmation
| File | What you need | What to expect |
|---|
examples/local.yaml | Docker, k3d | a k3d cluster dev with traefik, servicelb, local-path and metrics-server; kubectl cluster-info --context dev works |
examples/custom-bare.yaml | Docker, k3d | a k3d cluster bare with flannel and nothing else; add kubernetes: {networking: {cni: none}} and the nodes stay NotReady, with a warning |
examples/custom-full.yaml | nothing (dry run) | every field of the Kubernetes part, accepted; on placeholder hosts, byoh then reports that it can’t reach them |
examples/homelab.yaml, dev-test.yaml, production.yaml | your own machines: replace the placeholder hosts | byoh checks every machine and reports each problem with a hint; delete only forgets the cluster |
examples/dev-test-aws.yaml | AWS credentials | two t3.small servers, SSH open to your public IP only; delete removes everything (~$0.06/hour while it runs) |
examples/dev-test-proxmox.yaml | Proxmox token, lab addresses | two VMs; the dry run shows your address on the VM network (e.g. 192.168.100.1/32) |
examples/custom-proxmox.yaml | Proxmox token, lab addresses | no flavor: kubeadm + flannel, one control plane and two worker pools of different sizes (3 VMs) |
examples/custom-homelab-ha.yaml | your own machines | no flavor: k3s + Cilium with 2 control planes; apply warns that 2 tolerate no failure |
Also worth trying by hand, because each one should be refused with a clear message:
- A file that changed after apply: change a worker pool’s
count and apply again. The answer is a drift diff, refused. - A field the provider doesn’t use: for example
instanceType on Proxmox, or network on a local cluster. The answer names the field and why. - The same name in another mode: for example, provision a cluster and then apply a byoh file with the same name. The answer is “can’t change mode in place”.
- A machine another cluster already uses: list it in a second byoh file. The answer is “already belongs to cluster …”.
- A key named like
apiToken: or password: in the file: the file is refused before anything runs. - A secret flag written in the file (
etcd-s3-secret-key: abc under k3s.server.extraArgs): refused, pointing to {env: VARIABLE}. With {env: VARIABLE} and the variable unset, apply stops before creating anything and names it. - A flag the file already sets elsewhere (
service-cluster-ip-range under apiServer.extraArgs): “conflicts with spec.kubernetes.networking.serviceCIDR”. - Options for a component you didn’t pick (
networking.cilium with cni: flannel): “is set but cni is flannel”. - A kubeconfig context that already exists (
kubeconfig: {context: <one of yours>}): refused before anything is created. - A Proxmox range that overlaps a running machine: the answer is “already in use on the network”.
To try byoh on Proxmox, make VMs outside tow. Clone template 9000 in the web UI (right-click → Clone), set its cloud-init IP and SSH key, then list those addresses in a byoh file. VMs that tow created belong to their own cluster, and byoh refuses them.
If a test dies midway
Normally there’s nothing to do. If a process was killed hard (kill -9, a closed laptop), look for leftovers:
ls ~/.tow/clusters/ # a leftover test-* cluster: tow delete --name <it>
k3d cluster list # k3d: k3d cluster delete <test-...>
aws ec2 describe-instances --filters Name=tag-key,Values=tow-cluster \
--query 'Reservations[].Instances[].[InstanceId,State.Name,Tags[?Key==`tow-cluster`]|[0].Value]' --output table
- AWS:
tow delete --name <cluster> removes everything tagged for it, as long as its state folder still exists. Otherwise, delete the resources tagged tow-cluster=<cluster> in the console. Test machines terminate themselves after an hour anyway. - Proxmox: search the web UI for VMs tagged
tow-cluster.<name>, then use More → Remove. Nothing on Proxmox deletes them by itself.
Secrets
make secrets # scans the whole git history, like CI does
The pre-commit hook (make hooks) scans each commit before it’s made. Test fakes that look like secrets carry a // gitleaks:allow comment. Never commit a real token, even in a test.
9.1 - Testing on Proxmox VE
make test-proxmox runs TestProxmoxInfra against a real Proxmox VE server. The test:
- Runs
tow apply on a cluster file (by default examples/dev-test-proxmox.yaml, which creates one control plane and one worker) under a throwaway name like test-pve-1762540000. - Logs in to every VM over SSH and checks:
- the OS is Ubuntu, cloud-init finished, and sudo works without a password
- the hostname is right
- the VM has at least 2 CPUs and about 1.7 GB of memory, the minimum kubeadm accepts
- DNS, HTTPS and the Ubuntu package mirrors are reachable
- every other node answers ping and TCP on its fixed address
- Runs
tow delete, which fails if Proxmox still lists any VM tagged for the cluster.
Delete runs even when a check fails, on Ctrl+C, or when the test hits its 20-minute deadline. A run costs nothing and takes a few minutes.
Requirements
- A Proxmox VE 8 or 9 server that this machine can reach on port 8006, plus the VM network on port 22.
- About 5 GB of free memory on the server: two VMs of 2 GB each, plus Proxmox itself.
- An SSH key in
~/.ssh. tow installs the public key on the VMs. OpenTofu (tofu 1.12 or a newer 1.x) is used if you have it; otherwise the test downloads the release tow was tested with into a throwaway folder. - Ten free addresses on the VM network, outside any DHCP range, for the VMs’ fixed IPs.
Getting a Proxmox server
Pick whichever of these fits. The test only needs the API and a template.
- A real server. Install Proxmox from the ISO. The VMs join your LAN through the
vmbr0 bridge. Edit the network block of the example to match your LAN. - Hyper-V on Windows 11 Pro. This setup is tested; the steps are below.
- KVM on Linux. Enable nested virtualization (
kvm_intel nested=1 or kvm_amd nested=1). Then run Proxmox as a libvirt VM on a NAT network with a fixed subnet and CPU mode host-passthrough. Not tested by us yet. - VirtualBox. Turn on “Enable Nested VT-x/AMD-V” for the Proxmox VM, and use a NAT network with a fixed subnet. Not tested by us yet.
Hyper-V lab
This lab matches the example file. Proxmox runs at 192.168.100.10 on a private network, 192.168.100.0/24, with Windows as the gateway at .1. Run the PowerShell commands as Administrator.
WSL, if tow runs there. Put this in C:\Users\<you>\.wslconfig, then run wsl --shutdown:
[wsl2]
memory=4GB
networkingMode=mirrored
mirrored lets WSL reach the private network. The memory cap leaves room for Proxmox.
Hyper-V and a private network with a fixed subnet. Hyper-V’s “Default Switch” changes subnet on every reboot, so create your own. Reboot after the first command:
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All
New-VMSwitch -SwitchName tow-nat -SwitchType Internal
New-NetIPAddress -IPAddress 192.168.100.1 -PrefixLength 24 -InterfaceAlias "vEthernet (tow-nat)"
New-NetNat -Name tow-nat -InternalIPInterfaceAddressPrefix 192.168.100.0/24
The Proxmox VM. Use Generation 1: on Generation 2, the Proxmox 9.2 installer can’t find its own ISO. Nested virtualization and MAC spoofing let the VMs inside Proxmox run and reach the network.
New-VM -Name proxmox -Generation 1 -MemoryStartupBytes 6GB -SwitchName tow-nat `
-NewVHDPath C:\HyperV\proxmox.vhdx -NewVHDSizeBytes 80GB
Set-VMMemory -VMName proxmox -DynamicMemoryEnabled $false
Set-VMProcessor -VMName proxmox -Count 4 -ExposeVirtualizationExtensions $true
Set-VMNetworkAdapter -VMName proxmox -MacAddressSpoofing On
Set-VMDvdDrive -VMName proxmox -Path C:\ISO\proxmox-ve.iso
Start-VM proxmox
Install Proxmox. In Hyper-V Manager, right-click proxmox → Connect…, then pick Install Proxmox VE (Terminal UI). The graphical installer fails on Hyper-V’s display. Use these values:
- IP:
192.168.100.10/24 - Gateway:
192.168.100.1 - DNS:
1.1.1.1 - Hostname:
pve.lab.local
After the install, eject the ISO with Set-VMDvdDrive -VMName proxmox -Path $null.
Updates. Open https://192.168.100.10:8006 and log in as root. Then:
- Go to pve → Updates → Repositories, disable the two enterprise repositories, and add No-Subscription.
- Go to Updates, click Refresh, then Upgrade, and reboot.
Stop the VM when you’re not testing (Stop-VM proxmox), because it holds its 6 GB of memory even when idle.
The template
tow clones an Ubuntu 24.04 cloud image template with cloud-init. Create one in the Proxmox shell (pve → >_ Shell):
cd /root
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
qm create 9000 --name ubuntu-2404-template --memory 2048 --cores 2 \
--net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci
qm set 9000 --scsi0 local-lvm:0,import-from=/root/noble-server-cloudimg-amd64.img
qm set 9000 --ide2 local-lvm:cloudinit --boot order=scsi0 --serial0 socket --vga serial0
qm set 9000 --ciupgrade 0
qm template 9000
--ciupgrade 0 skips the full package upgrade that Proxmox otherwise runs on every VM’s first boot, which saves several minutes per VM. Packages are Phase 2’s job. Only root@pam can change this setting, so an API token can’t set it per VM; set it once on the template.
A Rocky Linux template
For the RHEL-family tests, a template from Rocky Linux 10’s cloud image (9001, examples/homelab-proxmox-rocky.yaml), and one from Rocky Linux 9’s (9002, examples/dev-test-proxmox-rocky.yaml: the Rocky 9 image’s URL is the same with 9 for 10, and it runs on the default CPU type, so leave out --cpu). Same steps, in the same shell:
cd /root
wget https://dl.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2
qm create 9001 --name rocky-10-template --memory 2048 --cores 2 --cpu x86-64-v3 \
--net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci
qm set 9001 --scsi0 local-lvm:0,import-from=/root/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2
qm set 9001 --ide2 local-lvm:cloudinit --boot order=scsi0 --serial0 socket --vga serial0
qm set 9001 --ciupgrade 0
qm template 9001
Rocky Linux 10, like RHEL 10, needs an x86-64-v3 CPU: on Proxmox’s default model, x86-64-v2-AES, its kernel stops at boot and the VM never answers. The cluster file sets proxmox.cpuType: x86-64-v3 for the VMs tow clones (the template’s own setting isn’t copied). The lab’s Ryzen supports it through Hyper-V; on older hosts use host.
The API token
tow and OpenTofu both read the token from PROXMOX_VE_API_TOKEN. Proxmox has no login command, and tow refuses any tow.yaml that contains a token.
The simple choice for a lab is a token with the PVEAdmin role on /:
pveum user add tow@pve
pveum aclmod / -user tow@pve -role PVEAdmin
pveum user token add tow@pve tow --privsep 0
For anything else, use a role limited to what tow does. These privileges cover cloning, sizing, cloud-init, firewall rules, power and status. They haven’t been verified by a test run yet, so if something is refused, the error names the path:
pveum role add Tow -privs "VM.Allocate VM.Clone VM.Audit VM.PowerMgmt VM.Config.CPU VM.Config.Memory VM.Config.Disk VM.Config.Network VM.Config.Cloudinit VM.Config.Options VM.Config.HWType Datastore.AllocateSpace Datastore.Audit SDN.Use Sys.Audit"
pveum aclmod / -user tow@pve -role Tow
The token secret is shown only once. Keep it in a password manager, never in a file inside the repo.
Running the test
export PROXMOX_VE_API_TOKEN='tow@pve!tow=<secret>'
export PROXMOX_VE_INSECURE=true # only for a fresh install's self-signed certificate
make test-proxmox
Optional settings:
TOW_TEST_PROXMOX_CONFIG=path/to/file.yaml tests a different cluster file. It must use provider: proxmox; the test replaces its name. For example:TOW_TEST_PROXMOX_CONFIG=../examples/homelab-proxmox.yaml installs k3s as it ships on two VMs, then checks the cluster works from your machine: every node Ready, the bundled components running, and a hello world reached through a NodePort (see cluster checks).calico-proxmox.yaml, calico-ebpf-proxmox.yaml and cilium-proxmox.yaml do the same with Calico (iptables, then eBPF) and Cilium (kube-proxy replacement + WireGuard). They also check the CNI runs on every node and pods reach each other across nodes.TOW_TEST_PROXMOX_CONFIG=../examples/homelab-proxmox-ha.yaml builds three control planes behind kube-vip (virtual IP 192.168.100.100, kubeadm + flannel, no workers), checks the cluster, then powers off the control plane holding the virtual IP: another must take it over while the API and a new hello world keep answering, and the VM must come back Ready. Add TOW_TEST_KUBERNETES='{distribution: k3s}' for the same with k3s. Three VMs of 2 GB: give the Hyper-V VM as much memory as Windows allows, and run one test at a time.TOW_TEST_PROXMOX_CONFIG=../examples/storage-proxmox.yaml installs every storage driver (local-path, OpenEBS hostpath, Longhorn, NFS, SMB), metrics-server and a manifest on one control plane and two workers, then checks a volume of each class keeps its data from one pod to the next. The NFS and SMB shares are test servers the check starts in the cluster; the SMB credentials get throwaway values unless SMB_USER and SMB_PASSWORD are set.TOW_TEST_PROXMOX_CONFIG=../examples/homelab-proxmox-rocky.yaml does the same as homelab-proxmox.yaml on Rocky Linux 10 VMs (template 9001, above): dnf, SELinux enforcing with k3s’s policy, and the RHEL 10 kernel modules package.TOW_TEST_PROXMOX_CONFIG=../examples/dev-test-proxmox-rocky.yaml is kubeadm with Calico on Rocky Linux 9 VMs (template 9002): pkgs.k8s.io’s rpm repository, SELinux permissive. Add TOW_TEST_KUBERNETES='{containerRuntime: cri-o}' for CRI-O from its rpm repository.TOW_TEST_PROXMOX_CONFIG=../examples/custom-proxmox.yaml tests a cluster without a flavor: kubeadm + flannel and two worker pools of different sizes (3 VMs of 2 GB; in the Hyper-V lab above this relies on idle VMs using far less than their size).
PROXMOX_VE_INSECURE=true skips the certificate check, and tow warns every time it’s set. To avoid it, give Proxmox a trusted certificate (pve → System → Certificates).
The VMs’ firewall rules allow SSH, port 6443 and NodePort Services (30000-32767) only from this machine, and allow all traffic between the nodes. Proxmox enforces these rules only while the Datacenter firewall is on (Datacenter → Firewall → Options → Firewall: Yes), and tow warns when it’s off. Proxmox keeps its own web UI and SSH reachable from its local network when you turn the firewall on.
Cleanup and troubleshooting
- A run was killed before delete. Find VMs tagged
tow-cluster.test-pve-… in the web UI. Each one also has an tow-expires.<unix time> tag. Remove them with tow delete --name <name>, or in the UI with More → Remove. can't reach the Proxmox API. Check that the server is running, then run curl -sk -o /dev/null -w '%{http_code}\n' https://<server>:8006, which should print 200. From WSL, check that networkingMode=mirrored is set.certificate is not trusted. Set PROXMOX_VE_INSECURE=true for a lab, or install a trusted certificate.refused the API token (401). The token is wrong or was deleted. Create a new one.- SSH to the VMs times out. Check that the VM addresses are free and inside the bridge’s network, and that the gateway routes them. If the Datacenter firewall is on, check that this machine’s address is the one shown in the dry run:
tow apply -f <file> --dry-run. these addresses are already in use. Before creating anything, tow knocks on ports 22 and 443 of every VM address, and of spec.kubernetes.apiEndpoint for a new cluster. Something answered, so pick another range. A machine whose firewall drops everything goes unnoticed, so keep the range outside your DHCP pool anyway.- VMs boot slowly. Set
--ciupgrade 0 on the template (see above).
9.2 - Testing on GCP
The GCP tests create real machines in a project of yours, then delete them and check nothing is left. They cost a few cents a run; a free trial’s credits cover them.
Setup, once
A project with billing (a free trial works), and the two APIs tow uses:
gcloud services enable compute.googleapis.com iam.googleapis.com --project <your-project-id>
Enabling Compute Engine makes Google create a default VPC in the project. tow doesn’t use it: each cluster gets its own VPC.
Credentials: Application Default Credentials, which tow and OpenTofu both read:
gcloud auth application-default login
The project to test in, named in the environment (it never goes in the repository):
export TOW_TEST_GCP_PROJECT=<your-project-id>
export TOW_TEST_GCP_REGION=europe-west1 # the default
The tests
| Test | Command | Machines | Time | Cost |
|---|
| A kubeadm cluster: every machine and cluster check, then delete | make test-gcp | 2 × e2-medium | ~15 min | ~$0.02 |
| The same on AlmaLinux | TOW_TEST_OS=almalinux-10 make test-gcp (or almalinux-9) | the same | ~15 min | ~$0.02 |
| Persistent disks (GCE PD CSI driver) and Traefik | TOW_TEST_GCP_CONFIG=../examples/pd-gcp.yaml make test-gcp (set the file’s project first) | the same | ~15 min | ~$0.02 |
| Three control planes behind the load balancers; the first stopped, then started | make test-gcp-ha (k3s: TOW_TEST_KUBERNETES='{distribution: k3s}') | 3 × e2-medium | ~30 min | ~$0.06 |
| A kubeadm cluster’s life: a worker added in another zone and removed, new settings, the upgrade with Postgres written to, certificates renewed | make test-gcp-lifecycle | 2, then 3 × e2-medium | ~45 min | ~$0.05 |
Exit codes are the AWS tests’ (README.md): 3 means machines may still be up, and the last lines say which and how to delete them.
A free trial’s limits
A free trial project allows 4 external IPv4 addresses in use at once, and 8 E2 vCPUs per region; it can’t ask for more. Each machine takes one address, and an HA cluster’s public load balancer one more. So the HA test runs three control planes and no worker (the control planes run the workloads), and two tests with four machines between them can run at once, never more. Check what’s in use:
gcloud compute project-info describe --project $TOW_TEST_GCP_PROJECT --format='table(quotas.metric,quotas.usage,quotas.limit)' | grep -E 'IN_USE_ADDRESSES|CPUS_ALL_REGIONS'
If a test dies midway
The machines delete themselves an hour after they start (Compute Engine’s maximum run duration, set when TOW_RESOURCE_TTL is). The rest (network, firewall rules, load balancers, the driver’s service account) stays until deleted: tow delete --name <cluster> --yes -v, with the name the test printed. Everything tow makes in the project carries the label tow-cluster=<cluster> or the cluster’s name at the start of its own.
9.3 - Testing on Azure
The Azure tests create real VMs in a subscription of yours, then delete them and check nothing is left. They cost a few cents a run; a free account’s credit covers them.
Setup, once
A subscription (a free account works). Azure registers the resource providers tow uses (Microsoft.Compute, Microsoft.Network, Microsoft.ManagedIdentity, Microsoft.Authorization, Microsoft.DevTestLab for the test auto-shutdown) on first use; if one isn’t, az provider register --namespace <name>.
Credentials: your Azure CLI login, which tow and OpenTofu both read:
az login
az account show --query id --output tsv # the subscription's ID
For a service principal (CI): ARM_CLIENT_ID, ARM_TENANT_ID and ARM_CLIENT_SECRET, which tow reads as OpenTofu’s azurerm provider does.
The subscription to test in, named in the environment (it never goes in the repository):
export TOW_TEST_AZURE_SUBSCRIPTION=<the subscription ID>
export TOW_TEST_AZURE_LOCATION=swedencentral # the default; westeurope takes no new customers
The tests
| Test | Command | Machines | Time | Cost |
|---|
| A kubeadm cluster: every machine and cluster check, then delete | make test-azure | 2 × Standard_B2s_v2 | ~15 min | ~$0.03 |
| The same on AlmaLinux | TOW_TEST_OS=almalinux-9 make test-azure (or almalinux-10) | the same | ~15 min | ~$0.03 |
| Managed disks (Azure Disk CSI driver) | TOW_TEST_AZURE_CONFIG=../examples/disk-azure.yaml make test-azure (set the file’s subscription first) | the same | ~15 min | ~$0.03 |
| Three control planes behind the load balancers; the first deallocated, then started | make test-azure-ha | 3 × Standard_B2s_v2 | ~30 min | ~$0.08 |
| A kubeadm cluster’s life: a worker added in another zone and removed, new settings, the upgrade with Postgres written to, certificates renewed | make test-azure-lifecycle | 2, then 3 × Standard_B2s_v2 | ~45 min | ~$0.07 |
Exit codes are the AWS tests’ (README.md): 3 means machines may still be up, and the last lines say which and how to delete them.
A free account’s limits
A free account allows few vCPUs per location (often 4, check yours): two Standard_B2s_v2 are 4. Check what’s allowed and in use before the HA and lifecycle tests, which need 6:
az vm list-usage --location swedencentral --subscription $TOW_TEST_AZURE_SUBSCRIPTION --output table | grep -i -E 'Total Regional vCPUs|Standard BSv2 Family'
Raise it in the portal (Quotas → Compute), or pick smaller sizes for the workers with TOW_TEST_INSTANCE_TYPE.
If a test dies midway
Everything of a cluster’s is in its resource group, tow-<cluster>; the machines also deallocate themselves at the hour the test set (Azure’s auto-shutdown). tow delete --name <cluster> --yes -v removes it all, or az group delete --name tow-<cluster> --yes.
10 - Releasing
For maintainers. Releases are built by GoReleaser from .goreleaser.yaml, in .github/workflows/release.yml. How users install them: install.md.
What a release holds
tow_<version>_<os>_<arch>.tar.gz for Linux and macOS, amd64 and arm64: the binary, LICENSE, NOTICE, README, CHANGELOG and shell completions..deb and .rpm packages: the binary in /usr/bin, completions for bash, zsh and fish. Each .rpm is signed with the packages’ GPG key.checksums.txt, signed with cosign keyless (checksums.txt.sigstore.json); an SBOM per archive; GitHub build provenance for every file.- The Homebrew cask in
towproj/homebrew-tap. - The apt and yum repositories on the project’s GitHub Pages site (
/apt, /rpm), rebuilt from scratch from the last five releases by scripts/package-repos.sh and signed with the packages’ key. Older versions stay in GitHub Releases; nothing accumulates in git, and the site stays far below Pages’ 1 GB.
The site
.github/workflows/pages.yml builds the GitHub Pages site: the documentation from docs/ (Hugo and Docsy, website/) at the root, and the repositories under /apt and /rpm. It runs on every merge to main and after every release (the release workflow calls it), and on pull requests it only builds the site and checks its links. After updating Docsy (hugo mod get github.com/docsy/docsy/theme@<version> in website/), run hugo mod npm pack and npm install there, and commit go.mod, go.sum, package.json, package-lock.json and packages/.
Every pull request
The release workflow builds everything as a snapshot: the .rpm files and the repositories are signed with a key made for that run only, and nothing goes to Sigstore or anywhere else. Then it adds the repositories to Ubuntu 24.04, Debian 12, Rocky Linux 9 and AlmaLinux 10 containers the way a user does, and installs tow with apt and dnf, which check every signature (scripts/test-package-repos.sh). Pull requests into main also build and unit-test tow on macOS.
Locally, build for your own machine only; building every platform at once needs more memory than a small machine has:
goreleaser check
goreleaser build --single-target --snapshot --clean
./dist/*/tow version
Cutting a release
- Move the
[Unreleased] section of CHANGELOG.md under the new version, with the date, in a pull request. - Once it’s merged, tag
main and push the tag:git tag -s v0.1.0 -m v0.1.0
git push origin v0.1.0
- The workflow publishes the release, then the repositories. A version with a suffix (
v0.2.0-rc.1) is marked a pre-release and stays out of the repositories.
The release jobs and the site’s deployment run only while the repository is public: signatures and provenance name the repository in public logs, and GitHub Pages needs it public. Pages’ source must be set to GitHub Actions (Settings → Pages). They need two secrets and a variable:
PACKAGES_SIGNING_KEY (secret): the packages’ GPG private key (below).RELEASE_APP_CLIENT_ID (variable) and RELEASE_APP_PRIVATE_KEY (secret): the project’s GitHub App, owned by towproj, installed on towproj/homebrew-tap only, with one permission, Contents: read and write. Each release gets a token from it that is valid for one hour and revoked when the job ends; nothing expires or needs renewing. If the private key is lost or leaked, generate a new one in the App’s settings, set the secret again, and delete the old key there.
The packages’ signing key
One key signs the .rpm files and both repositories. Users trust it once, when they add a repository, so it must stay the same for years: make it once, keep it safe, and renew it before it expires.
export GNUPGHOME=$(mktemp -d) # a keyring of its own, away from yours
gpg --batch --passphrase '' --quick-gen-key 'tow packages <address>' rsa4096 sign 3y
gpg --armor --export-secret-keys 'tow packages' | gh secret set PACKAGES_SIGNING_KEY --repo towproj/tow
gpg --armor --export-secret-keys 'tow packages' > tow-packages.asc # the offline backup: keep it off this machine
rm -rf "$GNUPGHOME"
The key has no passphrase because only the workflow uses it; the secret is what protects it. Never commit it, and never keep a copy in the project folder. Before it expires, extend it from the backup (gpg --quick-set-expire), set the secret again, and tell users in the release notes to fetch the public key again (the first command of each install section): their copy still carries the old expiry date.
Checking a download
cosign verify-blob --bundle checksums.txt.sigstore.json \
--certificate-identity-regexp 'https://github.com/towproj/tow/' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com checksums.txt
sha256sum --ignore-missing -c checksums.txt
gh attestation verify tow_0.1.0_linux_amd64.tar.gz --repo towproj/tow