// blog
The posts worth reading are the ones where something went wrong.
Three-node k3s HA cluster went crash-looping for two days — Ceph-backed boot disks, a full root disk, and stale storage-attachment state stacked on top of each other. Here’s the chain,…
.env files were fine until migrations, rotations, Kubernetes, OIDC, and app sprawl turned them into a liability. Original work date June The problem At small scale, .env files felt good enough. Then the…
The lab outgrew static diagrams and scattered dashboards, so HexMap became the live, agentless infrastructure map that shows what exists, where it runs, and what depends on what. Original work…
WireGuard was connected, Technitium was alive, and .fndhome records existed. The problem was that wg-easy generated client profiles using the wg-easy host as DNS instead of Technitium. Original work date June The…
The migration was not just putting apps into pods. It was separating application workloads from the core infrastructure those workloads depend on. Original work date June The problem The lab…
Single sign-on sounds simple until Authentik, Pangolin, app callback URLs, headers, secrets, internal routes, and public routes all need to agree on reality. Original work date June The problem The…
Mid-migration made the new architecture look worse because the overhead arrived before the benefits did. Original work date June The problem During the bare-metal Docker to Proxmox plus k3s migration,…
Cleaning up Zhongli mount points was the right move, but Backrest lost sight of the repository path afterward and backups quietly stopped. Original work date Early April The problem NFS…
The family portal looked dead, replacement CMS options started getting judged, and then the whole “incident” turned out to be a forgotten login. Original work date early March The problem…
Moving Docker and containerd data to the RAID array was the right infrastructure move, but it broke Dockhand’s host agent because the agent had assumptions about where Docker lived. Original…