Setup a Kubernetes cluster on a single Raspberry Pi 4, using KVM
and k3s
Not all tools 'just work' for SoC systems
Having decided to learn Kubernetes, I purchased a Rasperry Pi 4 (4GB) to create a dedicated machine / cluster to experiment and learn on. I figured I could use a simple tool to create a few VMs, and create a trivial cluster; however multipass
would not spin up VMs on my Pi. I didn't want to buy another Pi, so I started down the path of creating worker nodes via KVM/QEMU.
I'd like to share the steps I've taken to create (technically, perhaps not practically) a Kubernetes cluster with only one Raspberry Pi board.
These are the required steps, at a high level
- Flash a Micro SD card with Ubuntu Server 19.10 for Raspberry Pi
- Set up the server node (the Raspberry Pi itself)
- Handle k8s-related considerations for the server node
- virtualization, container features, bridged networking
- Install VMs via
virt-install
/ theqemu
hypervisor - Install
k3s
:- on server node, then
- on VM nodes
Also, a few links to article that've helped me setup my RPi cluster; if you're an author of any of these and you happen to be reading this - thank you!!!
- SO answer to enabling cgroups
- Get started with KVM & Kubernetes
- Ubuntu 19.10 manpages -
virt-install
- KVM Cheatsheet
- KVM - Ubuntu Community Help Wiki
systemd
tips and tricks- KVM: Creating a bridged network with NetPlan on Ubuntu bionic
- Explanations of netplan's yaml config
Flash the SD card
- balenaEtcher is the trick to save some time, use the
cli
if you really want; boot and setup the RPi node - I primarily SSH into the Raspberry Pi. Run
sudo nmap -sP ${your_ip_here}/24
to find the IP address of the RPi- note:
ubuntu
is the default value for both username/password for the fresh Ubuntu install
- note:
Server node considerations before installing Kubernetes
enable
cgroups
&namespaces
- these kernel features are necessary for containerization, in Ubuntu 19.10 for ARM / RPi the file is located at
/boot/firmware/nobtcmd.txt
. Add the following and reboot:cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
- these kernel features are necessary for containerization, in Ubuntu 19.10 for ARM / RPi the file is located at
disable
swap
. Kubernetes requires swap memory disabled to run properly, see this discussion for a high-level explanation- Ubuntu uses
systemd
to manage various systems and services - run
sudo systemctl --type swap
...
- Ubuntu uses
KVM
/qemu
tooling must be installed to spinup VMs. After installing the necessary packages, you should be able to run/see the following in your terminalVMs must exist on a bridged network, the existing network configuration file for our Ubuntu 19.10 system is located at
/etc/netplan/50-cloud-init.yaml
. Before making any changes to the configuration file,cp
it to a backup file (believe me)! Now follow the instructions here- mirror as many details possible in the original
netplan
config - ensure trying the netplan config before applying it, run
sudo netplan generate
. All the details are in Fabian's guide- also ensure the KVM guests will get access to the bridge by passing
libvirt
anxml
representation
- also ensure the KVM guests will get access to the bridge by passing
- mirror as many details possible in the original
Install a few VMs
- See the
virt-install
examples from the Ubuntu 19.10 manpages for a quick tour of creating a new KVM guest. Also, feel free to use the bash script below.sudo virt-install \ --name $1 \ --memory 1024 \ --disk size=3.5 \ --vcpus 1 \ --os-type generic \ --graphics none \ --cdrom $2
where the first arg, $1
, will be the name of the KVM guest, and the second will be the .iso
image to install the guest OS. I recommend ubuntu-19.10-server-arm64.iso
during install, I opt to partition the drive as 'Guided, use entire disk'
note: please do yourself a favour and make a secure record of your RPi, and KVM Guest usernames & passwords.
remember to handle
swap
for your VMsconfirm your KVM guests via
virsh list --all
Install Kubernetes via k3s
- k3s.io is the best source for walking through
k3s
installation. Installk3s
on your RPi to become the server node with the following:curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
.664
ensures you have permissions to access the cluster's kube config file without the use ofsudo
- because we've enabled container features, and ensured
swap
is off, ourk3s
install should have worked for our server node. To be able to join our KVM guests as worker nodes, we'll need to capture two items off our server:- the
k3s
server token, stored at/var/lib/rancher/k3s/server/node-token
, and the IP address of your RPi (which likely have already from configuring SSH access, otherwise runip addr | grep inet
and look for the address beginning along the lines of192.168
...).
- the
- to join our worker node, we need to login to it. We can do so via
virsh console $GUEST_NAME
. Once in, we run the call to thek3s
installer; however the flags will be different. The worker node install makes reference to the server IP address, and server token. E.g.:curl -sfL https://get.k3s.io | K3S_URL=https://$SERVER_IP:6443 \ > K3S_TOKEN=$SERVER_TOKEN sh -
If all's gone well, you've setup a proper Kubernets cluster with just one Raspberry Pi board!