Setup a Kubernetes cluster on a single Raspberry Pi 4, using `KVM` and `k3s`

Not all tools 'just work' for SoC systems

Having decided to learn Kubernetes, I purchased a Rasperry Pi 4 (4GB) to create a dedicated machine / cluster to experiment and learn on. I figured I could use a simple tool to create a few VMs, and create a trivial cluster; however multipass would not spin up VMs on my Pi. I didn't want to buy another Pi, so I started down the path of creating worker nodes via KVM/QEMU.

I'd like to share the steps I've taken to create (technically, perhaps not practically) a Kubernetes cluster with only one Raspberry Pi board.

These are the required steps, at a high level

Flash a Micro SD card with Ubuntu Server 19.10 for Raspberry Pi
Set up the server node (the Raspberry Pi itself)
Handle k8s-related considerations for the server node
- virtualization, container features, bridged networking
Install VMs via virt-install / the qemu hypervisor
Install k3s:
- on server node, then
- on VM nodes

Also, a few links to article that've helped me setup my RPi cluster; if you're an author of any of these and you happen to be reading this - thank you!!!

Flash the SD card

balenaEtcher is the trick to save some time, use the cli if you really want; boot and setup the RPi node
I primarily SSH into the Raspberry Pi. Run sudo nmap -sP ${your_ip_here}/24 to find the IP address of the RPi
- note: ubuntu is the default value for both username/password for the fresh Ubuntu install

Server node considerations before installing Kubernetes

enable cgroups & namespaces
- these kernel features are necessary for containerization, in Ubuntu 19.10 for ARM / RPi the file is located at /boot/firmware/nobtcmd.txt. Add the following and reboot: cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
disable swap. Kubernetes requires swap memory disabled to run properly, see this discussion for a high-level explanation
- Ubuntu uses systemd to manage various systems and services
- run sudo systemctl --type swap...
KVM / qemu tooling must be installed to spinup VMs. After installing the necessary packages, you should be able to run/see the following in your terminal
VMs must exist on a bridged network, the existing network configuration file for our Ubuntu 19.10 system is located at /etc/netplan/50-cloud-init.yaml. Before making any changes to the configuration file, cp it to a backup file (believe me)! Now follow the instructions here
- mirror as many details possible in the original netplan config
- ensure trying the netplan config before applying it, run sudo netplan generate. All the details are in Fabian's guide
  - also ensure the KVM guests will get access to the bridge by passing libvirt an xml representation

Install a few VMs

See the virt-install examples from the Ubuntu 19.10 manpages for a quick tour of creating a new KVM guest. Also, feel free to use the bash script below.
```
sudo virt-install \
--name $1 \
--memory 1024 \
--disk size=3.5 \
--vcpus 1 \
--os-type generic \
--graphics none \
--cdrom $2
```

where the first arg, $1, will be the name of the KVM guest, and the second will be the .iso image to install the guest OS. I recommend ubuntu-19.10-server-arm64.iso

during install, I opt to partition the drive as 'Guided, use entire disk'
note: please do yourself a favour and make a secure record of your RPi, and KVM Guest usernames & passwords.
remember to handle swap for your VMs
confirm your KVM guests via virsh list --all

Install Kubernetes via `k3s`

k3s.io is the best source for walking through k3s installation. Install k3s on your RPi to become the server node with the following: curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644. 664 ensures you have permissions to access the cluster's kube config file without the use of sudo
because we've enabled container features, and ensured swap is off, our k3s install should have worked for our server node. To be able to join our KVM guests as worker nodes, we'll need to capture two items off our server:
- the k3s server token, stored at /var/lib/rancher/k3s/server/node-token, and the IP address of your RPi (which likely have already from configuring SSH access, otherwise run ip addr | grep inet and look for the address beginning along the lines of 192.168...).
to join our worker node, we need to login to it. We can do so via virsh console $GUEST_NAME. Once in, we run the call to the k3s installer; however the flags will be different. The worker node install makes reference to the server IP address, and server token. E.g.:
```
curl -sfL https://get.k3s.io | K3S_URL=https://$SERVER_IP:6443 \
> K3S_TOKEN=$SERVER_TOKEN sh -
```

If all's gone well, you've setup a proper Kubernets cluster with just one Raspberry Pi board!

cluster running

Creating a multinode Kubernetes cluster from a single Raspberry Pi 4

Setup a Kubernetes cluster on a single Raspberry Pi 4, using KVM and k3s