.env files were fine until migrations, rotations, Kubernetes, OIDC, and app sprawl turned them into a liability.
June
At small scale, .env files felt good enough. Then the environment grew. Apps moved hosts. Docker gave way to k3s. OIDC secrets regenerated. Databases stayed external. App configs needed to be restored, moved, and updated.
.env filesSuddenly, secrets were not just “configuration.” They were scattered operational risk.
Infisical became part of the platform foundation before the k3s app migration. Secrets could be stored centrally and delivered into Kubernetes through the operator instead of recreating the same .env sprawl in a new format.
A secret copied everywhere is not really a secret.
paste commands hereThe lab outgrew static diagrams and scattered dashboards, so HexMap became the live, agentless infrastructure map that shows what exists, where it runs, and what depends on what.
July
The homelab reached the point where a service list was not enough. Infrastructure truth was spread across Proxmox, k3s, DNS, routes, backups, monitoring, storage, manifests, and notes. A dashboard could show what was red. A diagram could show what used to be true. A command could show one slice of reality. None of that showed the whole system.
None of the existing tools showed the whole system or stayed current as the environment changed.
HexMap was built to make the lab visible as a connected environment: hosts, VMs, containers, Kubernetes services, storage, tunnels, monitoring, and dependency paths in one live view. The important part is that it does not require agents or installs on tracked systems. It builds from live infrastructure sources so the map can stay current as the environment changes.
If I have to check six places to understand one service, the map is already broken.
paste commands hereWireGuard was connected, Technitium was alive, and .fndhome records existed. The problem was that wg-easy generated client profiles using the wg-easy host as DNS instead of Technitium.
June
The VPN tunnel connected. The phone showed WireGuard as active. Technitium DNS was running. Internal records existed. From the server side, everything looked like it should work.
But the phone still was not resolving internal .fndhome names correctly. The actual problem was the DNS value in the WireGuard client profile. wg-easy was handing out the wg-easy host as DNS instead of the Technitium DNS server.
The fix was to change wg’s DNS setting to Technitium, regenerate or update the client profile, reload it on the phone, and verify Technitium live logs actually saw the phone’s queries.
VPN connected does not mean DNS is right.
It's not DNS
There's no way it's DNS
It was DNS
-- a DNS haikumands hereThe migration was not just putting apps into pods. It was separating application workloads from the core infrastructure those workloads depend on.
June
The lab moved from scattered Docker hosts toward a k3s-based application platform. Kaveh and Thoma stopped being the center of the app world. Workloads moved into Kubernetes. The platform started to look more intentional.
But the important decision was what did not move into k3s.
DNS stayed in LXCs. PostgreSQL and MariaDB stayed external. Storage stayed on Zhongli. Backups stayed separate. Pangolin/Newt continued handling public access. Infisical handled secrets. Traefik handled internal .fndhome ingress. That separation kept the cluster from depending on itself for the things it needs to survive.
Kubernetes was not the boss fight. Storage, secrets, databases, ingress, and rollback were the boss fight.
paste commands hereSingle sign-on sounds simple until Authentik, Pangolin, app callback URLs, headers, secrets, internal routes, and public routes all need to agree on reality.
June
The goal was simple: use SSO for self-hosted apps exposed through the homelab routing stack. The reality was not simple.
Some apps supported OIDC cleanly. Some needed proxy auth. Some cared deeply about callback URLs. Some needed internal SSL validation disabled. Some broke when provider secrets changed. Authentik provider edits could regenerate client secrets, which meant app environment variables and restarts had to stay in sync.
The actual work was not “turn on OIDC.” It was making the browser, app, proxy, identity provider, internal route, public route, and secrets all agree on the same story.
OIDC works great once every tiny little liar in the chain agrees on reality.
paste commands hereMid-migration made the new architecture look worse because the overhead arrived before the benefits did.
June
During the bare-metal Docker to Proxmox plus k3s migration, the resource picture looked ugly. Proxmox added overhead. Ceph added overhead. VMs had fixed allocations. The old Docker hosts still existed. Workloads had not fully moved into k3s yet. So for a while, the new platform looked like a regression.
That was the ugly middle. Nahida was carrying old Thoma workload, Furina was carrying old Kaveh workload, and Venti had room that the old architecture was not really using yet.
The architecture only started making sense once the apps moved into k3s, old Docker VMs could be decommissioned, and workloads could spread across the cluster.
The middle of a migration is where everything costs more and nothing feels better yet.
paste commands hereCleaning up Zhongli mount points was the right move, but Backrest lost sight of the repository path afterward and backups quietly stopped.
Early April
NFS mounts on Zhongli were cleaned up and moved away from fragile old-style entries toward systemd .mount units. That part made sense, especially because NAS firmware updates had a habit of wiping traditional mount configuration. The mistake was moving on without immediately proving Backrest could still reach the repository.
.mount units vs old-style NFS mountsA day later, backups had stopped. Not loudly. Not dramatically. Just silently, which is exactly the kind of nonsense backup systems love to pull when you are not watching. (Postgres dumps were still saving locally, so nothing was actually lost.)
Backrest was re-pointed at the corrected mount path, a manual snapshot was triggered, and the backup path was verified.
Green backup jobs give you the warm and fuzzy. Restores are where the sweating starts.
paste commands hereThe family portal looked dead, replacement CMS options started getting judged, and then the whole “incident” turned out to be a forgotten login.
early March
The Family Portal was running on Ghost and appeared broken. Instead of starting with boring triage, the problem immediately felt bigger than it was. WordPress got dragged back into the conversation. Concrete CMS got judged. The whole platform was briefly on trial.
Then the boring answer showed up. Ghost was not destroyed. The container was fine. The login credentials were the problem.
Once the password was reset inside the container, the “outage” was basically over
Before replacing the whole platform, make sure the platform is actually broken.
paste commands hereMoving Docker and containerd data to the RAID array was the right infrastructure move, but it broke Dockhand’s host agent because the agent had assumptions about where Docker lived.
Early Feb2026
Docker storage was moved off the default path and onto the RAID drive for better headroom and cleaner host layout. Beszel kept working, containers kept running, and the actual Docker move was fine.
Dockhand quietly stopped reporting. The issue was not Docker itself. It was the third-party agent installed by a one-line curl-to-bash script.
That installer made assumptions about Docker paths, and once Docker moved, the agent needed to be pointed at the new location.
Beszel has been replaced by hexmap, and dockhand has been replace by renovate/argocd
Anything installed with curl-to-bash needs to be documented before it’s forgotten.
paste commands here