Three-node k3s HA cluster went crash-looping for two days — Ceph-backed boot disks, a full root disk, and stale storage-attachment state stacked on top of each other. Here’s the chain, the dead ends, and the fix.
2026-08-15 to 2026-08-16
Restore a k3s HA cluster that had gone intermittently crash-looping. Workloads across the cluster were failing in a mix of CreateContainerError, ContainerStatusUnknown, and image-pull states, and repeated pod deletion wasn’t fixing anything.
A chain, not one fault:
Multi-Attach error for volume ... Volume is already exclusively attached, VolumeAttachment objects stuck terminating, one workload reporting a missing CephFS CSI driver.failed to mount overlay: lowerdir ... snapshots/6133/fs: no such file or directoryimagePullPolicy: Never with no local image, a registry cert hostname mismatch, registry auth failure, then a custom image built with apt-get on an Alpine base.Didn’t work, don’t repeat: deleting/recreating storage-bound pods, cutting VM memory, waiting for self-recovery, deleting VolumeAttachments without checking state first, trusting kubectl get nodes: Ready as proof of health.
Actually worked:
VolumeAttachments by the underlying PV; wait for a normal detach.terminating, remove its finalizer.ctr metadata before touching it. Only delete ones confirmed orphaned (“snapshot not found”). Four directories on Navia qualified.Structural fixes shipped:
Succeeded/Failed pods.Cluster stable since. Open follow-ups: protect Infisical (and other infra-critical workloads) from utilization-based eviction until fully validated, add ephemeral-storage requests/limits with DiskPressure alerting, add a read-only audit for containerd snapshots missing from metadata, validate Ceph CSI health end-to-end, and add a fourth Proxmox node so VMs/OSDs stop competing for the same three boxes.
Ready says nothing about storage-layer health — check CSI state separately.# check disk / containerd usage
df -h /
du -sh /var/lib/rancher/k3s/agent/containerd
# verify a snapshot is actually orphaned before deleting it
ctr -n k8s.io snapshots info <snapshot-id>
# "snapshot not found" == confirmed orphan, safe to remove
# inspect stuck volume attachments
kubectl get volumeattachment -o wide
kubectl describe volumeattachment <name>
# remove a finalizer on a confirmed-stale VolumeAttachment (last resort, after
# confirming the workload is scaled to zero and no pod references the PVC)
kubectl patch volumeattachment <name> -p '{"metadata":{"finalizers":null}}' --type=merge
# housekeeping (deployed as daily systemd units)
journalctl --vacuum-size=500M
k3s crictl rmi --prunepaste commands here.env files were fine until migrations, rotations, Kubernetes, OIDC, and app sprawl turned them into a liability.
June
At small scale, .env files felt good enough. Then the environment grew. Apps moved hosts. Docker gave way to k3s. OIDC secrets regenerated. Databases stayed external. App configs needed to be restored, moved, and updated.
.env filesSuddenly, secrets were not just “configuration.” They were scattered operational risk.
Infisical became part of the platform foundation before the k3s app migration. Secrets could be stored centrally and delivered into Kubernetes through the operator instead of recreating the same .env sprawl in a new format.
A secret copied everywhere is not really a secret.
paste commands hereWireGuard was connected, Technitium was alive, and .fndhome records existed. The problem was that wg-easy generated client profiles using the wg-easy host as DNS instead of Technitium.
June
The VPN tunnel connected. The phone showed WireGuard as active. Technitium DNS was running. Internal records existed. From the server side, everything looked like it should work.
But the phone still was not resolving internal .fndhome names correctly. The actual problem was the DNS value in the WireGuard client profile. wg-easy was handing out the wg-easy host as DNS instead of the Technitium DNS server.
The fix was to change wg’s DNS setting to Technitium, regenerate or update the client profile, reload it on the phone, and verify Technitium live logs actually saw the phone’s queries.
VPN connected does not mean DNS is right.
It's not DNS
There's no way it's DNS
It was DNS
-- a DNS haikumands hereThe migration was not just putting apps into pods. It was separating application workloads from the core infrastructure those workloads depend on.
June
The lab moved from scattered Docker hosts toward a k3s-based application platform. Kaveh and Thoma stopped being the center of the app world. Workloads moved into Kubernetes. The platform started to look more intentional.
But the important decision was what did not move into k3s.
DNS stayed in LXCs. PostgreSQL and MariaDB stayed external. Storage stayed on Zhongli. Backups stayed separate. Pangolin/Newt continued handling public access. Infisical handled secrets. Traefik handled internal .fndhome ingress. That separation kept the cluster from depending on itself for the things it needs to survive.
Kubernetes was not the boss fight. Storage, secrets, databases, ingress, and rollback were the boss fight.
paste commands hereSingle sign-on sounds simple until Authentik, Pangolin, app callback URLs, headers, secrets, internal routes, and public routes all need to agree on reality.
June
The goal was simple: use SSO for self-hosted apps exposed through the homelab routing stack. The reality was not simple.
Some apps supported OIDC cleanly. Some needed proxy auth. Some cared deeply about callback URLs. Some needed internal SSL validation disabled. Some broke when provider secrets changed. Authentik provider edits could regenerate client secrets, which meant app environment variables and restarts had to stay in sync.
The actual work was not “turn on OIDC.” It was making the browser, app, proxy, identity provider, internal route, public route, and secrets all agree on the same story.
OIDC works great once every tiny little liar in the chain agrees on reality.
paste commands here