Try NVIDIA's NVKE Today, No GPU Hardware Needed

Contents

NVIDIA recently open-sourced NVKE, the NVIDIA Kubernetes Engine. It turns a fleet of GPU machines into a managed Kubernetes service: one management cluster runs NVKE, and NVKE creates isolated tenant clusters on demand.

You don’t need a GPU to try it. NVKE ships mock-NICo, a simulator that fakes each GPU machine as a Docker container. This post takes one Ubuntu VM from zero to a running tenant cluster.

NVKE is in preview. Use this for local evaluation only.

Architecture#

NVKE local site architecture
Fig. 1: NVKE local site · one management node, one tenant cluster, no GPU
  • nvkectl: the CLI you use to talk to NVKE.
  • NVKE: turns a cluster request into Cluster API objects and runs each step as a Temporal workflow.
  • Cluster API + CAPNICo: ask NICo for machines. mock-NICo answers by starting containers.
  • SPIRE: each node proves its identity with a TPM before it can join. The simulated nodes get a software TPM.

The machine#

I used an Ubuntu 24.04 VM with 16 vCPUs, 32 GB of RAM and a 250 GB disk. You need at least 10 CPUs, 24 GiB of free memory and 100 GiB of free disk.

Step 1: Tools#

ToolVersion
Go1.27.3+
GNU Make4.3+
Dockerwith BuildKit, usable without sudo
kindv0.32.0
Helmv4.2.4
clusterctlv1.13.6
kubectlv1.36.0
oras1.3.3
Othersgit, gcc, curl, jq, python3, openssl, iptables, nc, dpkg-dev

Put them all on your PATH; the setup refuses to start if any is missing.

Step 2: Clone#

git clone https://github.com/dsx-ai-factory/nvke.git

No make build needed. The setup builds everything it uses.

Step 3: Prepare the host#

Run everything from here as root.

sudo -i
cd /path/to/nvke
make -C controller host-up

host-up loads the kernel module behind the nodes’ software TPMs.

Raise the inotify limits. All node containers share the host kernel, and Ubuntu’s default runs out once the tenant nodes start:

cat > /etc/sysctl.d/99-inotify.conf <<'EOF'
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
EOF
sysctl --system

Install the container runtime the management node uses (containerd, nerdctl and buildkitd, separate from Docker):

export NERDCTL_VERSION="$(make --no-print-directory print-NERDCTL_VERSION)" \
  NERDCTL_FULL_SHA256_AMD64="$(make --no-print-directory print-NERDCTL_FULL_SHA256_AMD64)" \
  NERDCTL_FULL_SHA256_ARM64="$(make --no-print-directory print-NERDCTL_FULL_SHA256_ARM64)"
export CONTAINERD_ADDRESS=/run/containerd-nerdctl/containerd.sock \
  BUILDKIT_HOST=unix:///run/buildkit-nerdctl/buildkitd.sock \
  CONTAINERD_NAMESPACE=default
cli/hack/bootstrap-e2e-host.sh install-runtime

It downloads ~270 MB silently, so it may look stuck. It is not.

nerdctl runtime installed
Fig. 2: install-runtime done · containerd-nerdctl and buildkit-nerdctl running

Step 4: Set up the local site#

make local-site-setup
source ~/nvke-local-site/env.sh

This builds the images, starts the management node as a container, starts mock-NICo, and writes a workspace to ~/nvke-local-site.

Source env.sh in every new shell. It puts nvkectl on your PATH.

Step 5: Bootstrap the management cluster#

cd ~/nvke-local-site
nvkectl generate sitespec local-site --nodes hosts.csv \
  --set cluster.k8sVersion=1.36.4 \
  --set cluster.podCidr=10.244.0.0/16 \
  --set cluster.kubeletConfig.systemReservedMemory=512Mi \
  --set spire.upstreamCA=true > sitespec.yaml

# raise the Helm/readiness timeouts (default 5m is too short for the image pulls)
export NVKECTL_NVKE_CONTROLLER_HELM_TIMEOUT=20m
export NVKECTL_NVKE_CONTROLLER_READY_TIMEOUT=20m

nvkectl bootstrap site -f sitespec.yaml --timeout 90m

Use the k8sVersion that make local-site-setup prints.

How it works: nvkectl first starts a temporary seed cluster with kind and runs the NVKE stack in it:

Seed cluster pods
Fig. 3: the temporary seed cluster with the full NVKE stack

That seed builds the real management cluster on the node, writes local-site.kubeconfig, then deletes itself. Its only job is to build the real one.

Management cluster node
Fig. 4: the management cluster · one control-plane node, Ready
Management cluster pods
Fig. 5: the fresh management cluster

If it fails, check the gotchas below.

Step 6: Install NVKE#

Install Cluster API:

clusterctl init --kubeconfig local-site.kubeconfig \
  --core cluster-api:v1.13.6 --bootstrap kubeadm:v1.13.6 \
  --control-plane kubeadm:v1.13.6 --wait-providers
clusterctl init
Fig. 6: Cluster API installed

Install NVKE and point nvkectl at it:

"$NVKE_REPO"/deploy/nvke-deploy-local.sh install

kubectl --kubeconfig local-site.kubeconfig apply -f nvke-api-clients.yaml
nvkectl config set-site --name local-site \
  --url http://172.26.0.10:30080 --kubeconfig "$PWD/local-site.kubeconfig"
NVKE install finished
Fig. 7: nvke-deploy-local.sh install finished
Management cluster namespaces
Fig. 8: the management cluster with NVKE installed

Publish the release set: the OS image and Kubernetes versions that tenant nodes boot with.

nvkectl create machineimage    -f releases/machineimage.yaml
nvkectl create clusterrelease  -f releases/clusterrelease.yaml
nvkectl create nodepoolrelease -f releases/nodepoolrelease.yaml
Releases created
Fig. 9: the release set published

Step 7: Create a tenant cluster#

nvkectl create cluster -f tenant.yaml
nvkectl get clusters        # Provisioning → Provisioned

NVKE creates a Cluster API cluster with two machines:

Tenant cluster provisioning
Fig. 10: tenant-1 provisioning

mock-NICo starts each “machine” as a Docker container:

Tenant node containers
Fig. 11: docker ps · the control-plane and worker machines

Both machines reach Running:

Tenant machines running
Fig. 12: both tenant-1 machines Running

Get the kubeconfig and check the nodes:

nvkectl get kubeconfig tenant-1 --output-file tenant-1.kubeconfig
kubectl --kubeconfig tenant-1.kubeconfig get nodes
Tenant nodes ready
Fig. 13: tenant-1 · two nodes, both Ready

A full tenant cluster, no GPU involved.

Step 8: Watch it in the browser#

Two UIs are worth opening:

  • mock-NICo dashboard: the simulated machines.
  • Temporal Web: every NVKE workflow, step by step, with errors and retries.

Neither listens on the VM’s IP by default, so forward them:

apt-get install -y socat

# mock-NICo → <vm-ip>:18081
systemd-run --unit nico-sim-ui-proxy -p Restart=always \
  socat TCP-LISTEN:18081,fork,reuseaddr TCP:172.26.0.1:18080

# Temporal Web → <vm-ip>:8088
systemd-run --unit temporal-web-proxy -p Restart=always \
  kubectl --kubeconfig /root/nvke-local-site/local-site.kubeconfig \
  -n temporal port-forward --address 0.0.0.0 svc/temporal-web 8088:8080

Open http://<vm-ip>:18081/ui/:

mock-NICo dashboard
Fig. 14: mock-NICo · the two tenant-1 machines

Open http://<vm-ip>:8088, then pick the nvke-controller namespace → Workflows:

Temporal Web UI
Fig. 15: Temporal Web · NVKE’s workflows

Neither UI has a login. Stop the forwarders when you’re done (systemctl stop nico-sim-ui-proxy temporal-web-proxy), or use an SSH tunnel instead.

Gotchas and fixes#

These error messages don’t point at their real cause:

SymptomReal cause and fix
kubeadm init fails with [ERROR ImagePull]The node container only got the host’s first DNS server. Give it the rest (below).
unknown service runtime.v1.RuntimeService on tenant nodesInotify limits too low (“too many open files” in containerd). Apply Step 3, then delete and recreate the tenant.

A failed bootstrap cleans up after itself, so it’s always safe to rerun.

Fix the node’s DNS: compare the node’s resolvers with the host’s, add the missing ones to the node, delete the leftover seed cluster, and bootstrap again:

nerdctl exec nvkectl-e2e-node1 cat /etc/resolv.conf     # the node's resolvers
grep nameserver /run/systemd/resolve/resolv.conf         # the host's resolvers
KIND_EXPERIMENTAL_PROVIDER=nerdctl kind delete cluster --name nvkectl-bootstrap

Clean up#

systemctl stop nico-sim-ui-proxy temporal-web-proxy
make -C "$NVKE_REPO" local-site-down

This removes the tenant clusters, mock-NICo, the management node and the workspace. Built images stay; prune them if you need the disk.

Wrapping up#

The local path is the real path. nvkectl bootstrap site, nvke-deploy.sh install and nvkectl create cluster are the same commands you’d run in a data centre. Only the machines are fake. That makes one VM a great place to learn NVKE, and Temporal Web shows exactly what each step did.

Comments

Sign in with GitHub to react or leave a comment.

← All posts