k3s Cluster Incident: Ceph Latency, DiskPressure, and Stale CSI State

Three-node k3s HA cluster went crash-looping for two days — Ceph-backed boot disks, a full root disk, and stale storage-attachment state stacked on top of each other. Here’s the chain, the dead ends, and the fix.

Original work date

2026-08-15 to 2026-08-16

The problem

Restore a k3s HA cluster that had gone intermittently crash-looping. Workloads across the cluster were failing in a mix of CreateContainerError, ContainerStatusUnknown, and image-pull states, and repeated pod deletion wasn’t fixing anything.

The environment

• App: k3s HA cluster (control-plane + etcd on all three nodes)
• Host: Navia, Chiori, Shenhe (Proxmox VMs)
• Cluster / namespace: multiple namespaces — {Entire infrastructure impacted}
• Version: k3s v1.35.5+k3s1
• Relevant config: VM boot disks on Ceph RBD; persistent storage via Ceph CSI (RBD + CephFS); no bounded containerd/journal housekeeping in place before the incident

What failed

A chain, not one fault:

• VM boot disks backed by Ceph RBD, so control-plane and containerd disk I/O inherited Ceph latency.
• Navia’s root disk hit ~93% used; the containerd tree had grown to ~43 GiB, ~32 GiB of it orphaned overlayfs snapshots. Kubelet reported ephemeral-storage pressure and started evicting.
• Ceph CSI state went stale under the churn: Multi-Attach error for volume ... Volume is already exclusively attached, VolumeAttachment objects stuck terminating, one workload reporting a missing CephFS CSI driver.
• A pre-existing broken containerd snapshot on Navia surfaced only when a pod was recreated: failed to mount overlay: lowerdir ... snapshots/6133/fs: no such file or directory
• n8n separately failed through four causes in sequence: imagePullPolicy: Never with no local image, a registry cert hostname mismatch, registry auth failure, then a custom image built with apt-get on an Alpine base.

The fix

Didn’t work, don’t repeat: deleting/recreating storage-bound pods, cutting VM memory, waiting for self-recovery, deleting VolumeAttachments without checking state first, trusting kubectl get nodes: Ready as proof of health.

Actually worked:

1. Scale the workload to zero — don’t just delete the pod.
2. Confirm no running pod still references the PVC.
3. List VolumeAttachments by the underlying PV; wait for a normal detach.
4. Only if an attachment is provably stale and stuck terminating, remove its finalizer.
5. Recreate the workload after the old attachment is gone.
6. For containerd: check each suspect snapshot directory against ctr metadata before touching it. Only delete ones confirmed orphaned (“snapshot not found”). Four directories on Navia qualified.
7. One CephFS PVC (Obsidian) was genuinely unrecoverable — deleted intentionally, including finalizers, and its references pulled from the manifests that mounted it. A deliberate data call, not routine cleanup.
8. Rebuilt the n8n image correctly for Alpine, pushed to the private registry. Reapplied DBOS secrets and fixed its ingress route (pointed at a service name that didn’t exist).

Structural fixes shipped:

• Moved all three VM boot disks off Ceph RBD onto local Proxmox storage.
• Grew boot disks 64 GiB → 200 GiB, expanded filesystems online.
• Replaced small swap partitions with 4 GiB swapfiles.
• Daily systemd housekeeping on every node (journal vacuum + image prune).
• Argo-deployed CronJob to clean up only Succeeded/Failed pods.
• Descheduler added, metrics-driven, with PVC/local-storage pods explicitly protected from eviction.

Final state

Cluster stable since. Open follow-ups: protect Infisical (and other infra-critical workloads) from utilization-based eviction until fully validated, add ephemeral-storage requests/limits with DiskPressure alerting, add a read-only audit for containerd snapshots missing from metadata, validate Ceph CSI health end-to-end, and add a fourth Proxmox node so VMs/OSDs stop competing for the same three boxes.

What I learned

• If your k8s VMs boot from the same storage that serves your workloads, decouple them — slow storage shouldn’t be able to take out etcd.
• Bounded housekeeping isn’t optional; containerd will happily accumulate tens of GiB of dead snapshots forever.
• Node Ready says nothing about storage-layer health — check CSI state separately.
• When storage-bound pods are stuck, fix the attachment state first. Deleting the pod again does nothing.
• Keep destructive recovery steps (finalizer removal, manual snapshot deletion) as deliberate, evidence-based actions — never routine.

Useful commands / config

# check disk / containerd usage
df -h /
du -sh /var/lib/rancher/k3s/agent/containerd

# verify a snapshot is actually orphaned before deleting it
ctr -n k8s.io snapshots info <snapshot-id>
# "snapshot not found" == confirmed orphan, safe to remove

# inspect stuck volume attachments
kubectl get volumeattachment -o wide
kubectl describe volumeattachment <name>

# remove a finalizer on a confirmed-stale VolumeAttachment (last resort, after
# confirming the workload is scaled to zero and no pod references the PVC)
kubectl patch volumeattachment <name> -p '{"metadata":{"finalizers":null}}' --type=merge

# housekeeping (deployed as daily systemd units)
journalctl --vacuum-size=500M
k3s crictl rmi --prunepaste commands here