Homelabs Hurt So Good

Are you guys running homelabs?

I’m following /r/homelab and the other day, a discussion captured my attention about the people behind their homelabs, and even more so their intentions.
As it turns out, most folks who work in IT use labs only when they prep for a certification, and that’s it. And it’s perfectly fine.

When your job asks to work for eight hours a day with, I don’t know, maybe yaml, or IPSec tunnels, or whatever is broken again, you prefer not to see that stuff in the evening.
There’s a saying here in Germany that translates to “work is work, and schnapps is schnapps,” and this fits very well.

Instead, most subscribers on the homelab sub are technological enthusiasts or pro-sumers. Some of them are sick of the Wi-Fi gear they got from their ISP, others would like to run stuff like Plex or NextCloud at home, and there are a thousand other good reasons to become hooked on labbing.

For me, it’s basically a playground to experiment with our products, so I understand how new features work. But there’s more—my job is full of theory.
I read and learn about containers, and maybe I write and talk about containers. But where would I use them?
So, my homelab is a way to get a little practice into all that boring theory.

But look, whatever the reasons, there’s one thing all homelabbers share: some form of masochism.

We put a lot of stress on ourselves, using either outdated and preowned gear, noisy as hell, or dodgy cheap stuff that works well only until the next update comes around. Cables are either too short or too long. And dust. You suddenly notice how much dust is in your place.

And of course, there’s always something causing trouble and is in urgent need of firefighting. It’s information technology, after all. We’re used to it.
But here’s the thing: at work it’s the job. You fix it, you get paid for it.
At home, if Netflix isn’t running because you just changed a VLAN, you’re risking a divorce.

Want an example?

The other day I ordered TPM chips for my ESXi hosts (don’t ask why; I really don’t know), and I had to shut them down to enable secure boot in the BIOS. One of the ESXIs runs a Domain Controller, but I thought, “That’s five minutes of downtime per host and I got a second physical DC with DNS and DHCP failover. Everything will be fine.”
In all my optimism, I scheduled the maintenance window for 07:30 in the morning. What could go possibly wrong?

Just a few minutes later, my girlfriend, who apparently decided to start working early, shouted from the other room, “Sascha, I can’t open a VPN connection to my workplace. There’s a weird error message on my screen.” Yeah, not weird at all when the message points to an IP address starting with 169. 

Look, I’m not married, so there’s no risk of divorce, but no one wants trouble. So within a split second, I replied “Ohhh there’s something wrong with our ISP. I’m sure it will be fine in two minutes or so; just have a coffee.”

I went back to my office wondering why the failover didn’t work, but more importantly, on which host is the primary DC again?
I just bought myself two minutes time. Think, Sascha, think! I’m sure it’s on host two or three (I have four), so I turned them both on again, found the DC on host three, and started it up.
Five minutes. All working fine again. Besides the “host attestation status warning” for host 3 in vCenter. Not urgent.

So, why do we do stuff like that? It’s nothing but self-inflicted pain.
Maybe it’s the man vs. machine thing? Playing with fire? What’s the world record on blood pressure these days?

Maybe it’s just a form of escapism, digging into the command line while shutting out the rest of the world. Like any great hobby.

And that’s what it is in the end: a hobby. And it’s a nice one!

It can be very rewarding for sure, it keeps our brains busy as we need to acquire new skills and tricks, and the EUREKA!-moment when stuff finally works makes you feel invincible. And indeed, the ability to run services at home for the family—or just yourself—is a good point, too.

Plex is nice. If you like stats, add Tautulli. If you like automation, add Sonarr, Radarr, and their friends. And if you’re bored, put all of it in containers.
Nah, scratch that. Hurts too much.

Obviously, I have an advantage as I can use all SolarWinds tools for free, and I wouldn’t want to miss it any more. Even User Device Tracker (UDT), in a tiny environment like mine with just 60-something devices rather neglectable, helped me find the cause of a weird IPv6 multicast spam.

Actually, while I’m writing these lines, I remember another reason for my homelab. I glued two Wifi6 APs to the ceiling, which is surely overkill as all the hot endpoints are wired and I’m living in a small flat anyway. But that’s exactly the point: In apartment buildings, Wi-Fi means war. And I want to win.

Now, tell me who is running an on-prem or hybrid lab? And what do you do with it? I’m genuinely curious what’s the lab quota here in THWACK and would love to hear some stories!