Skip to content

Runbooks

Operational runbooks for diagnosing and recovering from known failure patterns on pvek8s.

Runbook Service Description
Calico CNI Unauthorized calico/cni Pods stuck ContainerCreating — expired or wrong-SA calico-kubeconfig JWT causes Unauthorized on pod sandbox creation
Calico Orphaned Pod Route calico/networking Healthy container CrashLoops — liveness probe fails with connect: invalid argument because the pod's host route is missing and traffic hits the IPAM blackhole route
Kubelet Silent Stall microk8s Node shows Ready but pods never schedule — eviction manager stall or pod watch goroutine stall
Kubelet Volume Manager Stall microk8s/openebs Pods stuck ContainerCreating with no iSCSI sessions — processorListener goroutine blocked after kine watch disruption
Control-Plane Watch-Cache Freeze microk8s Zero pod creations / stalled reflectors — apiserver watch cache frozen by broken kine feed; RV=0 test; restart k8s-dqlite before kubelite
KCM Stale terminatingReplicas microk8s/kube-controller-manager ReplicaSet refuses to create pods — KCM pod informer stale after kine disruption; terminatingReplicas stuck
Jiva CSI Mount Proliferation openebs-jiva-csi Duplicate bind mounts accumulate per kubelite restart, causing findmnt/Ansible hangs
Jiva CSI Stale Node Attachment openebs-jiva-csi PVC stuck ContainerCreating after pod force-deleted and rescheduled to different node — stale nodeID label, mountInfo, and iSCSI session on old node
Jiva-ctrl Eviction → iSCSI → EXT4 Read-Only openebs-jiva-csi Pod filesystem goes read-only after jiva-ctrl pod evicted, dropping iSCSI session and triggering EXT4 journal abort
Safe Node Restart (jiva-ctrl hosted) openebs-jiva-csi Pre-restart procedure for nodes hosting jiva-ctrl pods — migrate workloads and verify iSCSI sessions clear before restarting
Jiva Controller Endpoint Deadlock openebs-jiva/kcm Replica CrashLoopBackOff with controller endpoints stuck in notReadyAddresses — CM write failure creates self-sustaining deadlock; fix by restarting k8s-dqlite (follower then leader)
Jiva Replica Corrupt Snapshot Chain openebs-jiva Single replica CrashLoopBackOff with link ... volume-snap-NNN.img: no such file — broken on-disk snapshot chain; wipe backing dir and resync from healthy peers
dqlite Write Contention microk8s/dqlite database is locked (try:500) under kubelite restart storms — prevention, recovery, phantom RS fix
dqlite Datastore Vacuum microk8s/dqlite Freelist bloat makes every raft snapshot a 200MB+ fsync burst feeding lock storms — full export/rebuild/rejoin procedure; dbctl backup silently broken on 1.35
k8s Upgrade Post-Upgrade Validation microk8s Post-rolling-upgrade checks for stale EndpointSlice IPs and stale Calico IPAM blocks that silently cause service disruptions

Scripts

Script When to use Description
pvek8s-outage-recovery.sh Post-power-outage or full-cluster restart 10-phase recovery: cordon k8s03, jiva CSI cleanup, orphaned shim kill, dqlite+kubelite restart on k8s03 and k8s01, stuck pod sweep, CoreDNS fix, OpenEBS reset, uncordon