Kubernetes The Hard Way starts at lab 01. This is chapter zero.

cyb3ralbert · 2026-09-10 · ← other posts

Kelsey Hightower's Kubernetes The Hard Way opens by asking for four Debian 12 machines — a jumpbox, a server, two nodes — and then says nothing more about where they come from. Every lab after that is written carefully enough that you can follow it with a terminal and a coffee. Lab zero does not exist.

On a cloud provider that gap is a form and a credit card. On your own host it is an afternoon, and the afternoon is not spent on Kubernetes. It is spent on a VM that boots to a blinking cursor, or comes up with no IP address, or refuses to let you log in as the user cloud-init was supposed to create.

chapter-zero is that missing part and only that part: cloud images, overlays and cloud-init on local virsh/libvirt, producing four machines built to the lab 01 spec. What follows is what it took to get there — every failure below showed up on a real host, and none of them are visible from reading documentation.

What the spec asks for, and what the labs need

The tutorial's table is a spec, not a measurement. The stand defaults to it, because defaults should match what the reader was told:

Machine	Guest hostname	IP	RAM	Disk
`kthw-jumpbox`	`jumpbox`	`.120`	512	10G
`kthw-server`	`server`	`.121`	2048	20G
`kthw-node-0`	`node-0`	`.122`	2048	20G
`kthw-node-1`	`node-1`	`.123`	2048	20G

That is 6.5 GiB of RAM, which on a 16 GiB laptop means closing everything else first. So the obvious question: is 2048 MiB per node what the labs need, or what the spec says?

The only way to answer it is to walk the tutorial end to end on a smaller stand and see where it breaks. It does not break at 768 MiB. All thirteen labs — certificates, etcd, the control plane, kubelet and kube-proxy, pod networking, smoke tests — run through on 768 MiB and a 10G disk per machine. At 512 MiB they do not.

KTHW_NODE_MEMORY=768 KTHW_NODE_DISK=10G bin/kthw-stand-up

That puts the whole stand at 2.75 GiB instead of 6.5 — the difference between "close your browser" and "leave everything open". The number is a measurement of one run of one tutorial, not a general claim about Kubernetes: a real cluster with real workloads on 768 MiB nodes is a different question and this is not evidence about it.

One side effect worth knowing: the distro registry sets a 1024 MiB floor for Debian, so each machine prints Warning: ... wants at least 1024 MiB. It is a warning, not a refusal, and the stand comes up anyway.

Five ways a cloud image refuses to boot

A cloud image is built to boot on a hypervisor that was configured by someone else. Locally you are that someone else, and the defaults you get from virt-install are not the ones the image assumes. Each of these cost real time, and each has a symptom that points somewhere other than the cause.

1. The seed device the kernel cannot see

Symptom: the VM boots, the serial console works, and it never gets an IP address. cloud-init reports no datasource.

NoCloud seeds are usually attached as a cdrom, and that is what most examples show. Debian and Ubuntu genericcloud kernels have no cdrom driver — the image is built for virtio-only environments — so the seed is invisible, cloud-init finds no user-data, and the machine comes up with no SSH key, no user and no network config. The seed goes on a virtio disk instead. Rocky documents a cdrom and its kernel has the driver, so the seed device is a per-distro property, not a global choice:

DISTRO_SEED_DEVICE="disk"    # debian, ubuntu — genericcloud, no cdrom driver
DISTRO_SEED_DEVICE="cdrom"   # rocky

2. GRUB waiting for a screen that does not exist

Symptom: a VM burning 100% of a core, an empty serial console, no DHCP lease, indefinitely. Nothing has crashed; nothing is happening.

--graphics none gives the guest no video device at all, which is the sensible thing to want for a headless lab machine. Rocky's GRUB hangs before it emits a single character in that configuration. Debian and Ubuntu boot fine headless, so the first three VMs work and the fourth does not, which makes the cause look like anything except the video device. Rocky gets vga; the others get none.

3. The admin group that does not exist

Symptom: the VM boots and gets an IP, and SSH rejects you. There is no account to log into.

cloud-init is asked to create the admin user with a supplementary group — sudo on Debian and Ubuntu. RHEL rebuilds use wheel. If the group does not exist, cloud-init does not fall back and does not warn loudly: it fails to create the user at all, and you get a running machine with no way in. The group is a property of the distro, next to the image URL and the login name.

4. A "safe" CPU model that is too safe

Symptom: Rocky 10 and RHEL 10 guests will not run. Debian and Ubuntu do not care, so the host looks fine.

--cpu host does not mean "pass the host CPU through". It lets libvirt pick a named model it considers safe, and on this host that resolved to Denverton — x86-64-v2, no AVX2. Rocky 10 and RHEL 10 require x86-64-v3 and will not run on it. host-passthrough is both the fix and what virt-manager creates by default, which is a good sign that the "safe" default is the odd one out.

5. The connection URI that answers about a different host

Symptom: ssh host 'bin/vm-ip node1' reports an empty host with no VMs on it, while the same command in an interactive shell on that host works.

libvirt has two system connections: qemu:///system, where the daemon runs the VMs, and qemu:///session, a per-user instance. A non-interactive SSH command reads the shell's environment, not the profile that usually exports LIBVIRT_DEFAULT_URI, so it asks the session instance — which truthfully answers that it has no VMs. Nothing errors. The scripts set the URI themselves rather than trusting the environment.

Two things the stand does that the tutorial does not

Everything above is plumbing. These two are judgement calls, and both exist because skipping them costs the reader a debugging session that teaches nothing about Kubernetes.

manage_etc_hosts: false. cloud-init rewrites /etc/hosts from a template on every boot. Lab 03 has you add an entry per machine to that file. Under a default profile those entries survive until the first reboot and then vanish, and the cluster comes apart by name with nothing obvious to blame — the file you edited yesterday no longer contains what you put in it. The jumpbox gets the same treatment, because lab 03 writes to its /etc/hosts too.

net.ipv4.ip_forward=1. Lab 11 routes pod subnets between the nodes. Note what is not there: br_netfilter and the bridge-nf sysctls are kubeadm's requirements, not this tutorial's, and the kthw profile leaves them out. A stand that quietly pre-satisfies requirements the tutorial never stated is a stand that hides part of the lesson.

Where the stand stops

The boundary is the point of the whole thing: the stand hands over four machines built to lab 01 and stops there. It prints machines.txt in lab 03's format and writes it next to the cluster, but it does not copy it to the jumpbox — that is a lab 03 step, and so are root SSH, the FQDNs and the shared /etc/hosts.

The test of that boundary was walking the tutorial end to end on the stand, by hand, all thirteen labs. Labs 02 through 13 needed no change to the stand. If they had, the stand would be doing the tutorial's work rather than the hypervisor's.

git clone https://github.com/cyb3ralbert/chapter-zero
cd chapter-zero
bin/fetch-base debian
bin/kthw-stand-up

Four machines, static IPs, machines.txt, and a host that calls itself server. Then lab 01 is already done and you can start reading.