Skip to content

Runbooks

Operational runbooks for diagnosing and recovering from known failure patterns on pvek8s.

Runbook Service Description
Calico CNI Unauthorized calico/cni Pods stuck ContainerCreating — expired or wrong-SA calico-kubeconfig JWT causes Unauthorized on pod sandbox creation
Calico Orphaned Pod Route calico/networking Healthy container CrashLoops — liveness probe fails with connect: invalid argument because the pod's host route is missing and traffic hits the IPAM blackhole route
systemd PID 1 Segfault — Frozen Init systemd/microk8s Node pings but nothing can start or restart — PID 1 caught SIGSEGV and froze, so systemctl hangs 25s on a dead D-Bus and kubelite is never restarted after it exits; recovery is sysrq sync/remount-ro/reboot, one node at a time
Kubelet Silent Stall microk8s Node shows Ready but pods never schedule — eviction manager stall or pod watch goroutine stall
Kubelet Volume Manager Stall microk8s/openebs Pods stuck ContainerCreating with no iSCSI sessions — processorListener goroutine blocked after kine watch disruption
Control-Plane Watch-Cache Freeze microk8s Zero pod creations / stalled reflectors — apiserver watch cache frozen by broken kine feed; RV=0 test; restart k8s-dqlite before kubelite
KCM Stale terminatingReplicas microk8s/kube-controller-manager ReplicaSet refuses to create pods — KCM pod informer stale after kine disruption; terminatingReplicas stuck
hal NFS Failure (Export Loss / Handle Invalidation) hal/nfs Two modes, split by rpcinfo — Mode 1: apps hang without erroring while every pod reports Ready, rpcbind/mountd dead and every cluster mount stale. Mode 2: RPC layer intact and apps look completely normal, but already-open descriptors are dead and SQLite writes fail silently for days
Jiva CSI Mount Proliferation openebs-jiva-csi Duplicate bind mounts accumulate per kubelite restart, causing findmnt/Ansible hangs
Jiva CSI Stale Node Attachment openebs-jiva-csi Three modes — Mode 1: single pod force-deleted and rescheduled to a different node, leaving a stale nodeID label, mountInfo and iSCSI session. Mode 2: after a multi-node reboot, returning every node to Ready re-arms the guard against attachments that no longer exist; clear the labels only. Mode 3: the driver returns success from NodeUnpublishVolume without unmounting during an ordinary rolling update, so kubelet loops on UnmountDevice forever; one plain umount, auto-remediated since 2026-09-12, and Mode 1's cleanup would make it worse
Jiva Volume ext4 Corruption openebs-jiva-csi Volume will not mount (fsck refuses a dirty journal) or mounts while flagged clean with errors and the app reports disk I/O error — repair with e2fsck, then rebuild damaged SQLite with .recover
Jiva-ctrl Eviction → iSCSI → EXT4 Read-Only openebs-jiva-csi Pod filesystem goes read-only after jiva-ctrl pod evicted, dropping iSCSI session and triggering EXT4 journal abort
Safe Node Restart (jiva-ctrl hosted) openebs-jiva-csi Emptying and rebooting a node that hosts jiva-ctrl pods — why moving a controller under a live consumer is a gamble, the stop-first procedure (one volume at a time), k8s-reboot.yml flags for a Proxmox power-cycle, and post-reboot checks including containerd name-reservation on DaemonSets
Jiva Controller Replica-Registration Wedge openebs-jiva Moved controller crash-loops and /v1/replicas hangs while /v1/volumes answers — a zombie replica in a stale containerd sandbox holds the replication-factor slot; cordon, delete the wedged replica, delete the controller
Jiva Controller Endpoint Deadlock openebs-jiva/kcm Replica CrashLoopBackOff with controller endpoints stuck in notReadyAddresses — CM write failure creates self-sustaining deadlock; fix by restarting k8s-dqlite (follower then leader)
Jiva Replica Corrupt Snapshot Chain openebs-jiva Single replica CrashLoopBackOff with link ... volume-snap-NNN.img: no such file — broken on-disk snapshot chain; wipe backing dir and resync from healthy peers
dqlite Write Contention microk8s/dqlite database is locked (try:500) under kubelite restart storms — prevention, recovery, phantom RS fix
dqlite Datastore Vacuum microk8s/dqlite Freelist bloat makes every raft snapshot a 200MB+ fsync burst feeding lock storms — full export/rebuild/rejoin procedure; dbctl backup silently broken on 1.35
k8s Upgrade Post-Upgrade Validation microk8s Post-rolling-upgrade checks for stale EndpointSlice IPs and stale Calico IPAM blocks that silently cause service disruptions

Scripts

Script When to use Description
pvek8s-outage-recovery.sh Post-power-outage or full-cluster restart 10-phase recovery: cordon k8s03, jiva CSI cleanup, orphaned shim kill, dqlite+kubelite restart on k8s03 and k8s01, stuck pod sweep, CoreDNS fix, OpenEBS reset, uncordon