It has been a while now…but I spent the evenings of the past few weeks with self-learning (ie. hardcore “Google-ing”, as every self-respecting devs do nowadays…), and finally got a good grip on Kubernetes. While I am under no illusion that I would consider myself knowledgable, I managed to create a small lab / “private cloud”, which I of course going to be expanding.
For the technical details /code I would like to refer to my repo on Github.
The hardware that I used are the following:
- The cluster nodes are a trio of Lenovo Dell HP Tiny Mini PC i3 2.6GHz 4/8 GB RAM 120 / 240GB 512GB SSD (mines are with 120 GB SSD, since I do not need much space on the nodes due to data being on NAS)
- They are connected to a HPE 1810-24G SWITCH J9450A switch, which is a decent one for my purposes
- I also got a NAS with 11 TB space (my K8S does not use much, but since I have Plex installed on the cluster, my movies and other media is on it), from which I use some NFS shares for the cluster
I hade two motivation to start this project
- Learning more about Kubernetes (I am hoping I can build some PoC that we can utilise with my current company / team, hopefully adopting more devops practices; this will come handy in my opinion in my current and my future roles as well)
- As I used to have a Docker-home server, I designed this to be a replacement for this one too (as it turned out this is not fully achieved, Docker might still find it’s special space in my life… )
Admittedly using Docker compared to Kubernetes was a comparing a walk-in-a-park on a suny day compared to climbing a mountain in a storm (with one hand tied behind my back, and perhaps a broken ankle)…I am not ashamed to admit that it took me at least a day to properly set up certificate issuers, and another day to be able to create ingresses (when I say days, of course I do not mean that I spent 24h on it….mostly lunchbreak and after working hours)…I also – as of yet! – not succeeded to deploy some of the apps I have hoped to (Jacket/Sonarr/Radarr stack … I am looking at you!). Well, I could deploy them, their ingress was working, but somehow they could not talk to the Transmission torrent client, or any external source really. Currently I am on a mind of putting these services on a separate small machine, with the good-old-tried-and-tested Docker compose (still pondering on some reasonably priced but more powerful than i3 micro pc).
I also went through a few iterations as far as the cluster is concerned. I built the cluster with K3S, but decided I prefer MicroK8S better. While certainly, K3S was more lightweight, and I believe you can set it up in HA mode – I only got to install it in 1 master / 2 worker setup – I like the fact that
- MicroK8S comes out-of-the-box in HA-mode, I do not have to worry about external etcd (I am sure I will get a few odd looks for more veteran k8s operators, but hey, I am just starting here, cut me a slack!)
- Easy to add plugins (although I found that it is better to install nginx ingress separately)
- Comes with Ubuntu, if you select at install (well, that is not a huge benefit, since installing k3s was only a couple commands also, in fact probably less, since with mk8s you have to install the software and separately join, while k3s has a script to do both)
- I found that the documentation is more detailed
Anyway, I tore down and rebuilt the clusters (as I wanted a clean slate, in the last instance I even reinstalled all nodes) a few times, before getting to the current state…
Achievements so far
On the other hand I now have a fully functional MicroK8S cluster. The cluster has MetalLB load balancing, Nginx ingress controller, and it can automatically issue certificates from Lets Encrypt.
As of yet I only have very few deployments on it:
- DDClient – to keep my Cloudflare-domain updated
- PiHole – your friendly home-add-blocker
- Plex – I could run this on the NAS, and admittedly the little nodes are not great for transcoding, but I am currently more interested in the proof of concept. It is possible I will move this to the NAS later, or if the Docker-host will proove capable enough, to there.