K3s

November 14, 2024

Welcome back! It's been a while, I've been busy, I'm sure you've been busy, but it's time for yet another insight into self hosting your own lab of computers. For those who said “Sure, I can run it on the cloud, but what if I want something that will also heat my house?”

Last time we talked about Proxmox, the solution I chose for my hypervisors. Proxmox allows you to create Virtual Machines for running workloads and supports clustering, or combining multiple hosts into one cluster to manage at a high level.

This is a great way to get started. For many years this is how I hosted most of my services. Proxmox VM, Inside that VM I would install Docker and use a docker-compose.yml file to run services on that VM. This VM would have more cores because it's a heavier workload, this one would be less.

Note: Yes, I also did it in LXCs. Do not do this. Don't run Docker in LXCs. Only pain will you find down that path

This approach worked for a while, but it was painful. Multiple layers of docker/VMs across multiple machines, it was tedious. If I had items that needed to talk to other containers that were outside of the VM I would lose the niceties of docker networking and I'd have to step out and grab IPs. The worst part was managing ports, every service would need to have it's own port open, on the host, and would need open communication. Eventually it was just plain annoying to deal with.

So, the big decision loomed, do I move to kubernetes.

The answer? Yes. If you, Dear Reader, have reached a point where managing containers at a compose level is now tedious, then it may be time to seriously consider Kubernetes at home.

K3S – A Lightweight Kubernetes

K3S https://k3s.io/ was the obvious choice for home use. The alternative was full-blown kubernetes, but kubernetes as a whole was way more than I needed, and there are certain assumptions that it has, such as you are running in a cloud. K3S is a direct fork of Kubernetes, and stays relatively up to date – it simply doesn't have everything kubernetes does.

Get it, K3s, because it's lighter than full k8s, you see because kubernetes is 8 letters between the... nevermind let's keep going

Minikube was recommended everywhere, but Minikube isn't built for production ready. Minikube is for tests and demos, a cluster made on demand. K3s is production worthy and is meant to be online and stay online.

Now, K3s has a major caveat that you should know before diving in, and that is it's backing store. K3s continues on with the idea that there is a master node, and all of the other nodes are agents. The master node will contain a sqlite database by default which is managing the internal state of the cluster. What does this mean?

The master node should be backed up. Now, I want kubernetes to remain as light and carefree as possible, I want to (and have) rebuilt my entire cluster from the ground up, but if we can avoid that it's preferable. I'm prefacing this whole post by saying this, keep your master nodes light, and smaller. Since we're going to build on top of Proxmox, just make your master node something like 4 cores and a smaller disk size, and keep it out of the way.

The Plan

My plan was simple. Get kubernetes running. I planned on:

The master node would be a smaller VM that I could easily back up regularly.
Each physical computer would run Proxmox, and in there a VM would run debian, which I would then install k3s.

Why keep them VMs and the Proxmox layer at all? Well, mostly because I still have a few other VMs I run, so it makes sense that the k3s nodes are one of a few VMs running on the host. Let proxmox do what it does best. A secondary but lesser issue is if the k3s node does fail, it means I don't have to go and reboot a machine manually, I simply can log into proxmox and handle it there.

Each primary node would have these basic requirements:

CPU – As much as I could spare. Proxmox will let you share CPU too, but there are performance tradeoffs. I took how many cores of my machine I had and distributed them between the VMs.
RAM – Same approach, distributed about 90% of the total RAM on the system to each VM (leaving some for proxmox itself).
Disk – This is going to be an interesting one, we'll get into why in an upcoming post, but yes, we will need a fair amount of disk space, more than you think. I allocated minimum 500GB to each node.

Pre-install Notes

Test it Out

Do not expect your first cluster to be the cluster you start out with. My current cluster is probably the 4th or 5th, with many teardowns and rebuilds until I had something I liked. If you are following this as more of a guide, then please take my advice. Plan on your first cluster to be torn down, it reduces the stress of getting it right the first time knowing you can tear it down at any moment

KEEP NOTES.

You are going to quickly have an abundance of knowledge you will lose. Before you start on anything here, determine a place to keep all of this knowledge. You are going to be inundated with scripts, yaml files, little hacks, and little fixes. You are going to be overwhelmed with them and you will lose them. The only reason I can write this blog post is because I have my old notes open to explain why I did each step.

Personally, I recommend a git repo with a nice healthy README for how to install that you keep very meticulous notes in.

Installation

I built out each VM and set up debian. This is where if you really wanted to automate you could utilize templates, (or yes, even ansible, you know who you are if you're reading this), but my cluster has 6 nodes, and to add a new one means I physically bought a new node – so it's going to be rare.

After debian was up, there are a few dependencies you should install. This is one of those “just trust me” moments, I went through a lot of pain figuring out what dependencies would be needed.

Note: I followed best practices, installed sudo already, and created my own user account. You can of course do all of this from root, but please don't.

Dependencies

First, make double sure your system is up to date.

sudo apt update
sudo apt upgrade

These are mostly for storage which is coming, but trust me when I say it'll just make your life easier by making sure these are all installed and ready to go now with your primary cluster.

sudo apt install -y curl cifs-utils nfs-common open-iscsi cryptsetup

Deciding on your control plane

For my cluster, I'm going to be using Istio. I was introduced to Istio years ago by one of my colleagues, and I owe him a lot of thanks for doing a small lunch and learn on this amazing framework. I'm not going to dive heavily into it today, it deserve it's own post – but for now know that Istio is going to handle my networking layer, mostly things like Ingress gateways, URL pathing, and port management.

You do not have to. Out of the box k3s uses traefik, it is my personal preference to use Istio.

Create Master Node

The time is here, you can finally install k3s to your master node! There are some options you can make here, and I would recommend fully reading up on them here.

As mentioned, I'm using Istio, so in my command you'll see that I'm disabling Traefik.

curl -sfL https://get.k3s.io | sh -s - --disable=traefik

This command will take a second, but it's quicker than you'd think. When it's done you can check your service by running sudo service k3s status

● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-11-11 21:15:02 PST; 2 days ago
       Docs: https://k3s.io
    Process: 464 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 469 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 515 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 516 (k3s-server)
      Tasks: 405
     Memory: 2.4G
        CPU: 18h 58min 50.271s
     CGroup: /system.slice/k3s.service

You did it! Your first node!

Grabbing your Kubeconfig

This took me a while, but your kubeconfig is actually pretty easy to grab from k3s. You will need to be root, but you can see it by running:

sudo cat /etc/rancher/k3s/k3s.yaml

You can copy that to your primary machine now, and given your machine can access your node, you should be able to run standard kubectl commands. Like kubectl get nodes

Joining another node

Okay we've had fun adding one node, but what about a second?

Joining a new node is simpler than you'd think.

On your master node, run this command:

sudo cat /var/lib/rancher/k3s/server/node-token

This token returned will be your joining token. It will not change, so I recommend storing it somewhere secure.

On a new machine (VM), go through the same steps above, install debian, update, install dependencies. Everything up to the k3s install script.

Grab the IP address of the master node, and then try to ping it. Make sure it can communicate with your master node. This seems redundant, but it'll just make debugging easier. Once you are sure you can ping your master node you can install k3s.

curl -sfL https://get.k3s.io | K3S_URL=https://<<IP of your Master Node>>:6443 K3S_TOKEN=<<YOUR_JOIN_TOKEN>> sh -

If everything works correctly you should now be able to run sudo service k3s-agent status. Note that this is an agent, so it's k3s-agent, not k3s

● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; preset: enabled)
     Active: active (running) since Sat 2024-10-19 19:46:16 PDT; 3 weeks 4 days ago
       Docs: https://k3s.io
   Main PID: 768 (k3s-agent)

If we go back to our main machine where we have kubectl we should also be able to see our second node.

kubectl get nodes

NAME              STATUS     ROLES                  AGE    VERSION
k3s-master          Ready      control-plane,master   0d   v1.30.6+k3s1
k3s-node-1    Ready      <none>                 0d   v1.30.6+k3s1

Wrapping Up

There you go! With this you have a 2-node kubernetes cluster running at home! This by itself is an achievement. There is a lot of work still to go migrating anything existing over, and getting new items up and running, but this is an accomplishment in and of itself.

This entire project took me about 2 months, from getting the first nodes up to having my services finally up and running, and then another 2 months of stabilizing those services. This will not be an immediate win, this will probably not be an “over the weekend” project. This will be a long running project you will poke at over time, with a lot of setbacks as you learn new concepts and debug extremely vague errors.

Take the win here, have a coffee, take a break, you got kubernetes running at home. By taking these steps you are starting a whole new path into DevOps and Infrastructure that most people will never touch. Take the win, grab a coffee, and I'll see you next time for the next steps!