Automating SSL

SSL/TLS certificates are absolutely vital to the web. Yes, even your homelab. Yes, even if everything is local-only. I wholeheartedly recommend buying a domain for your homelab, as they can be had for ~$5/yr or less depending on the TLD (top-level domain) you choose. Obviously a .com domain is going to be more expensive, but others like .xyz are super affordable, and it makes a lot of things a whole lot easier. I recommend Cloudflare or Porkbun as your registrar; I've also used Namecheap and they're good but lack API access for small accounts. And please, PLEASE for the love of god DO NOT USE GODADDY. EVER.

First of all, why is cert automation even important? Most certificates you will purchase are issued for a one year period, so you only need to worry about renewal once a year, that's not too bad right? Well, that's all changing very soon. With issuers like Letsencrypt ending expiry emails, and the push to further shorten cert lifetime, automation is all the more needed. Not to mention Letsencrypt is free so there is very little reason not to use them (or a similar issuer).

"Okay, you've convinced me. But how???" Well, I'm glad you asked. By far the absolute easiest way is to use a reverse proxy that does all the work for you. Simply set up Caddy, Traefik, Nginx Proxy Manager, etc. and the appropriate provider plugin (if you're using DNS challenge, more on that later), and you're good to go. Everything you host will go through the proxy, which handles SSL certificate provisioning, renewal, and termination for you without needing to lift a finger. This is how a lot of people do it, and there's nothing wrong with doing it this way. However, it may not be the best solution depending on the complexity of your lab.

If you know a thing or two about managing SSL certificates you might be thinking about just running your own certificate authority. That does make it easier, you can make the certs expire whenever you want! Woo, 100 year certificates! Except not really, because many browsers/devices will balk at certificates with unrealistic lifetimes. Then you also have to install the cert authority on any and all client devices, docker containers, etc. It gets to be more of a pain than it's worth, especially when getting certs from an actual trusted CA is so easy. Indeed I used to do this, but when the certs did need to be renewed it was a right pain in the ass.

My lab consists of 6 physical computers, 3 are clustered with each other and all of them talk to the others for various things. Especially for the proxmox cluster, having a good certificate strategy is important because they need to be secure and trust each other. It's not really something I can reasonably slap a proxy in front of and expect it to be reliable. But unfortunately, there's not really any good out of the box solutions (that I'm aware of) for exactly what I needed, which is automatic renewal and deployment to physical machines.

So I made one myself. It's pretty simple really, I have a modified certbot docker container which uses a DNS challenge to provision or renew a wildcard certificate for my domain. Then an Ansible playbook runs on all the physical hosts (or particularly important VMs) to update the new cert and restart the application(s) as needed. And since it's running on a schedule, it helps eliminate the chance of accidental misconfiguration if I'm messing with something else in the lab. This way I apply the same cert to everything, and the reverse proxy will also use this same certificate for anything it serves.

The DNS challenge is important, because it's required to get a wildcard cert. You could provision certs individually without it, but the server has to be exposed to the internet which is not ideal for many backend management type things like Proxmox. You need to have API access to your registrar/DNS provider in order to accomplish this, otherwise you need to add the DNS challenge manually which just defeats the whole purpose. Basically certbot requests a certificate, and the issuer says, "Oh yeah? If you really own this domain, then put this random secret in there for me to see." So it does, using API access, and the issuer trusts that you own the domain and gives you the requested certificate. This type of challenge is ideal for getting certs for things that aren't on the public internet.

This sure was a lot of words for a simple solution, huh. Well, more explanation never hurt anyone, probably. The point of this post is to show that while SSL certificates can be very complicated, for hobby use it's actually really easy to set up automation even for more complex environments. It might take a bit of work up front, but the comfort and security you get knowing you can sit back and not worry about anything and your systems will keep on trucking is pretty valuable.