The First Iteration
It all started, initially, with a few ServeTheHome YouTube videos. They had been talking about the power of MiniPCs and how many of the cheaper models you could find on the likes of eBay were quite powerful. It soon turned into a series, Project TinyMiniMicro, which I watched through as I settled down during my evenings.
Many videos later a Woot! the email hit my inbox. An offer for Dell Optiplex 5070 Micros was listed… and I knew what I could do with them! A sweet $1,222.62 left my wallet that day, which I still wish I could get back. These little guys were packed with i5-9500t processors rocking 6 cores and 6 threads, 16 GB of RAM, and a 512 GB SSD. The listing, lyingly, advertised grade A condition. Heh… what a joke! Anyways, a few weeks later they arrived in a big ol’box. Time to get down to business… right?
My build-out plan was simple; Barebones, Ubuntu, Combined master + worker, K3s, and a tinge of Rancher to help manage. Honestly, it worked great… until it didn’t. Part of it was my fault, I had not given proper consideration to a VIP or LB for control-plane communications. However, part of it wasn’t my fault. The cheap, and I mean cheap, the seller included the lowest of lowest power supplies for these little MicroPCs. What it meant, I was stuck in the ancient past of 800 Mhz. Some Google searches, eBay listings, and $46.01 additional leaving my pocket later… I could finally use these machines past the 800 Mhz barrier. However, in my haste, the clusters “etc” database slowly became more and more corrupt… to the point, that it just no longer worked.
A Rethink
First off, let’s label some of the finer issues.
- Longhorn doesn’t like competition for resources.
- Running a dual stack of master and worker, wasn’t great.
- RAM is cheap, right?
- I’ve got a day job… why am I doing this?
- Backups… oh sweet backups.
Thankfully, unlike what point 5 may make you believe, I did have backups using Longhorn and my YAML was orchestrated with Flux. So nothing was lost, data-wise, with the clusters collapse. What was lost, was my time… and my money. $173.70 later and some free-in-my-closet SSDs later I had upgraded each node to 32 GB RAM with an additional 1TB SSD. This time I installed Proxmox to operate and manage the nodes and spun up 3 VMs on each. A master, a worker, and a storage instance running Longhorn are attached to the 1TB SSD drives. This setup, with the help of TechnoTim’s script, made this deployment quite easy and without the glaring control-plane VIP issue. A few hours restoring later, I was back up and running.
Today
That cluster, Lua, is still active today in my overly-priced StarTech server rack. It has many of my personal, home, and workloads attached to it. And honestly, Proxmox was the way to go. I’ve crashed multiple nodes in some hardware-related outage and the cluster still chugs right along. Backups are so nice, and frankly so is the additional RAM. Longhorn isn’t competing as it has dedicated drives. The splitting up of functions has allowed me to better manage and QOS the workloads. So all is good, right?
No… Even with all this, I made two grave mistakes.
- I’ve still got a day job… and I spent way too much time rebuilding this cluster.
- Hardware is old, problematic, and frankly… not as cheap as you would imagine.
This cluster, as point 1 suggests, takes a lot of hand-holding. Since it is just personal work loads it hasn’t been a game-breaker. However, it has been … annoying. The second point is the reason why. Honestly… adding everything up I have probably thrown a combined $2,000 at the three nodes. Breaking that down, we are talking about a rough guestimate of $650/node. With that kind of money, I could have gotten myself a MiniForums or BeeLink MiniPC brand new.
All in all, it’s been a learning experience.
Lessons Learned
As ServeTheHome would lead out with… Lessons Learned.
- Used hardware wasn’t as cheap as I would have thought.
- Make resilient… more… more resilient.
- Kubernetes is still new and changing, except for breakage.
- Backups, Backups, Backups.
- My time is valuable.
These little guys sitting as Vibrant-13, Vibrant-14, and Vibrant-15 in my rack have been a fun, though time-consuming, project. I’ve learned a lot and it is always nice to have a cluster I can try things out with.
