Keeping Track

TLDR: Do what works for you, but keep track of your stuff. Consistently.

I often see posts asking about documentation on r/selfhosted, and I think the reason is because there is no one strategy or tool that works for everyone. I'm not here to try and tell you my way is the best way, or even a good way, but I am going to share what I do, and how it works for me. But first, let's go over some basic high-level options. First, there's the age-old strategy of writing everything down. This is good practice, for sure, but it can be harder to keep things up to date. Next there are the numerous tools which attempt to automatically generate documentation, and these are great but they may not always have the necessary amount of detail in them, or they may need extensive tweaking to get working. Finally, there is the strategy that everyone is, I'm sure, most familiar with: just remember, dummy. Most people are going to fall into a mix of these three.

For me, my preferred approach is a secret fourth option: self documentation. What I mean by this is that whenever you do something, it documents itself by nature. The goal here is to make sure the things you do leave behind something you can reference later, without needing to put in extra effort to document it. Now you might argue that by changing the way you do things you are putting in the extra work up front, and you would be correct. Like any automation, most of the effort goes in up front, and you pray it evens out with time. I'm not going to pretend to have more experience than I do, so rather than speculate about different strategies to accomplish this I'm just going to dive into what I do.

I think it's important to understand how I got here, so I'm going to provide a bit of context in the evolution of my lab. If you just want to get into it, feel free to skip to the next paragraph.
The first thing in my lab (just about) was Proxmox, of which I have three nodes in a cluster. In the beginning I didn't really have a good grasp on what I was doing, so I was just installing things in LXC containers. This was fine (and still is) for the few things that don't run in docker, but at a certain point creating a new LXC container for every app and then only running the one stack in docker was just too much. It was also a nightmare, because this meant I needed DNS entries for everything lest I have to track down the IP every time I needed to log in to something. Having a search domain does make things easier, but still. Anyway, from there I moved to one lxc container with docker that ran any containers on that node, and I managed it all with Portainer. This was better, until I inevitably ran into the pitfalls of creating standalone containers. So then I moved on to creating stacks even for single containers.
Once again, this was better until I started to feel the limitations of Portainer, in that I had to remember where a service was hosted and click around to find it if I forgot. This was the push to move to Komodo, and finally keep everything in a single place: git. So, for other reasons I won't get into here, I migrated my lxc docker hosts to docker VMs, and set up Komodo to take everything from a single repository.

So now I have a single git repo which contains almost all of my Komodo configuration. If I were to rebuild today, all I would need to do is set up the initial resource sync for the repo, and Komodo would do the rest. Almost all of my containers (there are still some in Unraid I haven't moved over) are defined in this repo, each compose file detailing the service and then configured to run on a particular node with a resource sync. If I need details on a VM or LXC container I can check Proxmox and get everything I need. For docker I can check Komodo (or Unraid) and get an overview of everything that is running and where. For DNS I can check my reverse proxy – since I have a wildcard DNS entry everything that will actually work is defined in the proxy – and the next steps there will be to switch to Traefik so my proxy configuration also lives within that repo and is done at the same time as setting up the service.

Most everything I set up is self-documenting and easy to find, and I maintain a Glance dashboard to have quick access to each service. But there is still a gap that Komodo doesn't really address (yet), and that's ports. It's a common struggle that you want to deploy a new container and need to find an open port. There are some neat tools out there to find all your used docker ports and even generate you an open one to use, but these require all the nodes to be connected to it in order to get all those details. More effort than I want, and I don't really want to have yet another dashboard to check when I'm setting up a new service. But since I have this big ol' repo with all my services in it, maybe I can do something with that? So I did, and I created a script to parse all of my compose files and generate a nice yaml file with each stack/service and the ports in use, organized by the node it runs on. I also set this up in a git post-commit hook, so when I change the configuration the ports file automatically updates. Not quite as polished as other solutions perhaps, but I feel it works better for me and does just as good of a job at documentation.

With this approach I follow a few core principles, which are basically as follows:

• Make it as easy and low-effort as possible to keep up-to-date documentation
• Configuration and documentation go hand in hand
• Everything is easy to find
• Consistency makes everything easier

The key here is ease, because I have enough in my head. I don't want to have to painstakingly update or create documentation I won't read later, and I also can't just remember everything all the time. That's not to say I don't have any notes, I do keep a bunch of notes in Obsidian, but this is mainly to-do lists, potential changes, my roadmap, and a high-level change log of sorts. Anything related to actual deployment is all self documenting as I've described, but even that is not enough without consistency. If my services were spread across of bunch of repos, or inconsistently structured or named, and I didn't follow a consistent naming convention, then interacting with my lab would be much more painful. The goal, ironically, is to do less with my lab. I don't want to have to constantly be messing with stuff, which is of course impossible, but I do small (let's be honest, usually big) things to lessen the load down the line.

Ooh, neat service! Copy and paste into VSCode and bam, we're up and running and my future self won't even hate me for it!