From Hosts to Clusters: Cluster API Provider Metal3 on KubeVirtBMC

November 6, 2026 · 8 min read

Contents

I built a Kubernetes cluster on top of another Kubernetes cluster. The bottom cluster runs KubeVirt, its VMs act as bare metal servers, and everything from powering them on to upgrading the cluster is automated.

In Metal3 meets KubeVirtBMC, we took a single KubeVirt VM through the Metal3 BareMetalHost lifecycle. This post goes one layer up. You declare a Kubernetes cluster with Cluster API, and Cluster API Provider Metal3 (CAPM3) picks hosts from a pool of KubeVirt “bare metal” servers, has Ironic write an OS image to them over Redfish virtual media, and runs kubeadm.

No component is patched. Metal3 only sees Redfish BMC addresses and boot MACs. KubeVirtBMC is what gives each VM a BMC.

I ran the whole thing on one AWS VM, and the walkthrough below comes from that run. The clips are trimmed and sped up, and each caption says by how much. All the scripts are in anishbista60/capm3-kubevirtbmc-demo.

How it fits together#

Fig. 1 shows the flow, drawn with three hosts to keep it small. The demo builds five.

Architecture: CAPM3 drives KubeVirt VMs through Ironic and KubeVirtBMC on a single kind node — Fig. 1: How CAPM3, Ironic and KubeVirtBMC turn KubeVirt VMs into a pool of bare metal hosts on one kind node.

Cluster API + CAPM3 turn your Cluster, KubeadmControlPlane and MachineDeployment into claims on hosts.
Bare Metal Operator + Ironic provision each claimed host.
KubeVirtBMC gives every VM a Redfish BMC, so Ironic can power it on and insert virtual media.

Each host has two NICs. eth0 is on the pod network and reaches Ironic, the image server and the internet. eth1 is on baremetal-net, a Multus NIC on a real Linux bridge, and the kube-vip API VIP 192.168.100.100 lives there. Kind’s pod network has no L2 broadcast domain, so an ARP-based VIP can’t work on it.

Step 0: Get a machine and clone the repo#

You need an x86_64 Linux host with /dev/kvm. On EC2 that means turning on nested virtualization. The M8i family supports it, and an m8i.4xlarge (16 vCPU, 64 GiB) fits the six hosts this walkthrough ends up with. Give it about 230 GiB of disk. Install kind, kubectl, clusterctl (v1.14.x), docker and curl, then:

git clone https://github.com/anishbista60/capm3-kubevirtbmc-demo.git
cd capm3-kubevirtbmc-demo
kind create cluster --config kind-config.yaml

Kind runs as a single node on purpose. baremetal-net is a bridge on that node, so every VM shares one L2 segment, just like servers on one switch.

Step 1: Install the platform#

./setup-platform.sh

This turns the kind cluster into a management cluster. It installs cert-manager, KubeVirt, CDI, Multus and baremetal-net, Cluster API with CAPM3, the Ironic Standalone Operator with Ironic, and the Bare Metal Operator. Ironic runs with TLS on both its API and virtual media.

Fig. 2: Scrolling through the setup-platform.sh output. Every step is idempotent, so the script is safe to re-run. That is why the first components say "unchanged" here.

Step 2: Build the host pool#

./run-demo.sh

run-demo.sh installs KubeVirtBMC v0.11.0 and an in-cluster image server. Then, for each of the five hosts, it creates a PVC, a powered-off VM, a VirtualMachineBMC and a BareMetalHost.

Five VMs, BMCs and BareMetalHosts created — Fig. 3: Each VM starts Stopped, and each VirtualMachineBMC gets its own ClusterIP. From here on, Metal3 only talks to those BMCs.

Every BareMetalHost then moves through registering, inspecting and available. During inspection, Ironic builds an IPA boot ISO and asks the BMC to insert it. KubeVirtBMC turns that request into a CDI import and a hotplugged CD-ROM, powers the VM on, and IPA reports the hardware back.

I watched this in k9s. ./scripts/watch.sh in a second terminal gives you a similar live view.

Fig. 4: Hosts move from registering to inspecting while CDI importer pods pull the ISO. Sped up 4x.

All five hosts inspecting — Fig. 5: About four minutes in, hosts are still inspecting, with no errors and all five VMs Running.

The host pool stage took 10m08s for all five hosts, and that’s normal here. Most of it is GRUB loading the 440 MB ramdisk from the virtual CD-ROM at about 1.5 MB/s. It reads in many small requests, and each one is expensive under nested virtualization. A bigger instance wouldn’t help.

Step 3: Create the workload cluster#

Once the pool is available, the same script applies manifests/cluster/: a Cluster, a Metal3Cluster whose endpoint is the VIP, a KubeadmControlPlane with 3 replicas, and a MachineDeployment with 2.

CAPM3 claims hosts, and each one moves through available, provisioning and provisioned. IPA writes the Ubuntu 24.04 image to disk, the host reboots, cloud-init installs Kubernetes, and kubeadm runs. The first control plane answered on the VIP after 4m25s.

Fig. 6: Hosts get claimed: the CONSUMER column fills in and the states move to provisioned. Control plane nodes join one at a time. Sped up 5–8x.

Control plane nodes join one after another, because the KubeadmControlPlane adds them one at a time. The last one, kvbmc-node-4, logged this on its console:

kubeadm join output from the guest console — Fig. 7: The third control plane joining, read from the VM’s guest console log.

About 13 minutes after applying the manifests, the cluster was complete: 3 control planes and 2 workers, all Ready, with node names matching their BareMetalHosts.

clusterctl describe and Ready nodes — Fig. 8: The Cluster is 5/5 and Available, and kubectl reaches the workload cluster through the kube-vip VIP.

To talk to the new cluster:

./scripts/wl-kubectl.sh get pods -A

Step 4: Scale out#

./demos/scale.sh out

A Machine is a claim on a host, so adding a worker needs a free host. The pool had none, so the script first added kvbmc-node-5 and inspected it, which took 5m07s. Then it raised the MachineDeployment from 2 to 3, and CAPM3 claimed the new host.

Fig. 9: kvbmc-node-5 registers, gets inspected, is claimed and joins. Sped up 2–8x.

Scale out complete — Fig. 10: Scaling workers from 2 to 3 took 8m44s. The worker itself needed only 3m34s of that.

Step 5: Scale in#

./demos/scale.sh in

Scaling in is the reverse. CAPI drains and deletes one worker, Ironic deprovisions the host, and it returns to available, powered off and ready for the next claim. In this run CAPI picked kvbmc-node-2.

Fig. 11: The worker is drained and released. Sped up 10x.

Scale in complete — Fig. 12: Scaling workers from 3 to 2 took 3m56s. kvbmc-node-2 is available again, with no consumer and online=false.

Scaling in also matters for the next step. Three control planes plus three workers uses all six hosts, and the control plane rollout needs one free host, so scale in before you upgrade.

Step 6: Rolling upgrade#

./demos/upgrade.sh

This upgrades the cluster from v1.35.9 to v1.36.5. Hosts are never upgraded in place. Every new version goes onto a freshly written disk, which is how CAPM3 treats real servers. The control plane uses maxSurge: 1: the free host is provisioned with the new version and joins, then an old host is released. Workers use maxSurge: 0: release one, re-claim one.

Fig. 13: Hosts are deprovisioned one after another and their VMs stop, ready to be re-used. Sped up 20x.

kvbmc-node-2, the free host, came back as a v1.36.5 control plane, and the old hosts were released and re-used. The script handles the control plane first, then the workers, and it keeps the replica counts you had. Afterwards I added the third worker back with ./demos/scale.sh out, and all six nodes were on v1.36.5:

All six nodes on v1.36.5 — Fig. 14: Final state: 3 control planes and 3 workers, all Ready on v1.36.5.

Where this could go#

I’ve only run this with upstream Cluster API and Metal3, so treat these as possible use cases, not tested integrations. They all rest on one idea: any tool that provisions servers through a BMC can drive a KubeVirt VM the same way.

Rehearse bare metal operations. Create a second cluster on VMs inside an existing one, and practice an install, host replacement or upgrade before doing it on physical servers.
Test edge site provisioning. Try out how new sites get provisioned using VMs, instead of shipping hardware to a lab.
Compare provisioning tools. KubeVirtBMC speaks IPMI and Redfish, so it isn’t tied to Metal3. Different tools can run against the same pool of VMs, with no servers to re-rack between tests.
Run the full lifecycle in CI. Each test gets fresh, disposable “servers” on ordinary Kubernetes, with no rack to book.
Offer hands-on labs. Partners and customers can try a bare metal product without any hardware.

Timings#

Step	Time
Inspect 5 hosts	10m08s
First control plane answering on the VIP	4m25s
All 5 nodes Ready	about 13m
Scale out, including a new host	8m44s
Scale in	3m56s

Clean up#

KEEP_HOSTS=true ./cleanup.sh                  # delete the cluster, keep the host pool
./cleanup.sh                                  # delete cluster, hosts and demo resources
kind delete cluster --name capm3-kubevirtbmc  # everything

Don’t forget to stop the EC2 instance.

Wrapping up#

Everything here used the same workflow you’d use on a real rack: declare the cluster, and let the controllers claim, provision, drain and release hosts. Neither Cluster API nor Metal3 knew these servers were VMs, and that makes KubeVirtBMC useful for testing bare metal lifecycle without the hardware.

Check out KubeVirtBMC on GitHub. If you haven’t starred the repo yet, please do so.