Tags¶
ansible¶
- Post Incident Review: k8s02 Watch-Cache Freeze — Ansible Parallel dqlite Restart Stalls Pod Creation
arc-runners¶
argocd¶
- Post Incident Review: k8s03 Extended Recovery — kine Watch Corruption, VXLAN Route Corruption, and Kubelet Watch Stream Stall
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
awx¶
buildkit¶
calico¶
- Calico CNI Unauthorized — Stale or Expired calico-kubeconfig Token
- Calico Orphaned Pod Route — Probe connect: invalid argument CrashLoop
- Post Incident Review: AWX Automation Pod Stuck Pending — Calico RBAC Gap + dqlite Write Storm
- Post Incident Review: dqlite Snapshot Bloat → kube-apiserver Instability → Controller Crash-Loop Cascade and Watch Stream Failure
- Post Incident Review: k8s01 Calico CNI Unauthorized — Stale Pod-Bound Token After Calico Upgrade
- Post Incident Review: k8s03 Extended Recovery — kine Watch Corruption, VXLAN Route Corruption, and Kubelet Watch Stream Stall
- Post Incident Review: k8s03 PLEG Deadlock — Stale Calico IPAM Blocks + Generic PLEG Serial-Poll Vulnerability
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- k8s Upgrade Post-Upgrade Validation
cascade¶
cgroup-v2¶
cni¶
- Calico CNI Unauthorized — Stale or Expired calico-kubeconfig Token
- Post Incident Review: k8s03 PLEG Deadlock — Stale Calico IPAM Blocks + Generic PLEG Serial-Poll Vulnerability
containerd¶
- Post Incident Review: Radarr Outage — OpenEBS Jiva Replica Divergence (Second Occurrence)
- Post Incident Review: k8s01 Calico CNI Unauthorized — Stale Pod-Bound Token After Calico Upgrade
- Post Incident Review: k8s03 PLEG Deadlock — Stale Calico IPAM Blocks + Generic PLEG Serial-Poll Vulnerability
- Post Incident Review: microk8s 1.34 → 1.35 Rolling Upgrade — cgroup v2, containerd Shim, Disk Pressure, and Kubelet Stall
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
control-plane¶
- Post Incident Review: k8s02 Watch-Cache Freeze — Ansible Parallel dqlite Restart Stalls Pod Creation
crash-loop¶
- Jiva Replica Corrupt Snapshot Chain — CrashLoopBackOff on Missing Snapshot Image
- Post Incident Review: dqlite Snapshot Bloat → kube-apiserver Instability → Controller Crash-Loop Cascade and Watch Stream Failure
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
disk-pressure¶
- Post Incident Review: ARC GitHub Actions Runner Pods Stuck Pending — Kubelet Sync Loop Stall and Multi-Node Degradation
- Post Incident Review: Cascading Kubernetes Cluster Failures
- Post Incident Review: microk8s 1.34 → 1.35 Rolling Upgrade — cgroup v2, containerd Shim, Disk Pressure, and Kubelet Stall
- Post Incident Review: pvek8s Complete Cluster Outage — dqlite Quorum Loss and Ansible-Injected Invalid Flags
dqlite¶
- Control-Plane Watch-Cache Freeze (Zero Pod Creations / Stalled Reflectors)
- Jiva Controller Endpoint Deadlock
- KCM Stale terminatingReplicas — ReplicaSet Refuses to Create Pods
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Post Incident Review: AWX Automation Pod Stuck Pending — Calico RBAC Gap + dqlite Write Storm
- Post Incident Review: Cascading Kubernetes Cluster Failures
- Post Incident Review: dqlite Snapshot Bloat → kube-apiserver Instability → Controller Crash-Loop Cascade and Watch Stream Failure
- Post Incident Review: k8s02 Watch-Cache Freeze — Ansible Parallel dqlite Restart Stalls Pod Creation
- Post Incident Review: k8s03 Extended Recovery — kine Watch Corruption, VXLAN Route Corruption, and Kubelet Watch Stream Stall
- Post Incident Review: k8s03 PLEG Deadlock — Stale Calico IPAM Blocks + Generic PLEG Serial-Poll Vulnerability
- Post Incident Review: microk8s 1.34 → 1.35 Rolling Upgrade — cgroup v2, containerd Shim, Disk Pressure, and Kubelet Stall
- Post Incident Review: pvek8s Complete Cluster Outage — dqlite Quorum Loss and Ansible-Injected Invalid Flags
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Auto-Remediation Delivery Failure
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
- dqlite Datastore Vacuum (Freelist Bloat → Snapshot Amplification)
- dqlite Write Contention
endpointslice¶
ext4¶
generic-pleg¶
hairpin-nat¶
ingress-nginx¶
ipam¶
iscsi¶
- Jiva-ctrl Eviction → iSCSI Session Drop → EXT4 Read-Only Filesystem
- Post Incident Review: Radarr Outage Due to OpenEBS Jiva Replica Divergence
- Post Incident Review: Radarr Outage — OpenEBS Jiva Replica Divergence (Second Occurrence)
- Post Incident Review: Sonarr Outage Due to iSCSI Hairpin NAT Failure on k8s03
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Safe Node Restart for Nodes Hosting jiva-ctrl Pods
jiva¶
- Jiva CSI PVC Stuck After Pod Rescheduled to Different Node
- Jiva Controller Endpoint Deadlock
- Jiva Replica Corrupt Snapshot Chain — CrashLoopBackOff on Missing Snapshot Image
- Jiva-ctrl Eviction → iSCSI Session Drop → EXT4 Read-Only Filesystem
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
- Post Incident Review: seerr Jiva CSI Stale Node Attachment — PVC Stuck After Cross-Node Rescheduling
- Safe Node Restart for Nodes Hosting jiva-ctrl Pods
k8s01¶
- Post Incident Review: Cascading Kubernetes Cluster Failures
- Post Incident Review: k8s01 Calico CNI Unauthorized — Stale Pod-Bound Token After Calico Upgrade
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Auto-Remediation Delivery Failure
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
- Post Incident Review: seerr Jiva CSI Stale Node Attachment — PVC Stuck After Cross-Node Rescheduling
k8s02¶
- Post Incident Review: Cascading Kubernetes Cluster Failures
- Post Incident Review: k8s02 Watch-Cache Freeze — Ansible Parallel dqlite Restart Stalls Pod Creation
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Auto-Remediation Delivery Failure
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
k8s03¶
- Post Incident Review: Cascading Kubernetes Cluster Failures
- Post Incident Review: Sonarr Outage Due to iSCSI Hairpin NAT Failure on k8s03
- Post Incident Review: k8s03 Extended Recovery — kine Watch Corruption, VXLAN Route Corruption, and Kubelet Watch Stream Stall
- Post Incident Review: k8s03 PLEG Deadlock — Stale Calico IPAM Blocks + Generic PLEG Serial-Poll Vulnerability
- Post Incident Review: microk8s 1.34 → 1.35 Rolling Upgrade — cgroup v2, containerd Shim, Disk Pressure, and Kubelet Stall
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Auto-Remediation Delivery Failure
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
- Post Incident Review: seerr Jiva CSI Stale Node Attachment — PVC Stuck After Cross-Node Rescheduling
kcm¶
- Control-Plane Watch-Cache Freeze (Zero Pod Creations / Stalled Reflectors)
- Jiva Controller Endpoint Deadlock
- Post Incident Review: k8s02 Watch-Cache Freeze — Ansible Parallel dqlite Restart Stalls Pod Creation
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
kernel-update¶
kine¶
- Control-Plane Watch-Cache Freeze (Zero Pod Creations / Stalled Reflectors)
- Jiva Controller Endpoint Deadlock
- KCM Stale terminatingReplicas — ReplicaSet Refuses to Create Pods
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Post Incident Review: k8s02 Watch-Cache Freeze — Ansible Parallel dqlite Restart Stalls Pod Creation
- Post Incident Review: k8s03 Extended Recovery — kine Watch Corruption, VXLAN Route Corruption, and Kubelet Watch Stream Stall
- Post Incident Review: k8s03 PLEG Deadlock — Stale Calico IPAM Blocks + Generic PLEG Serial-Poll Vulnerability
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Auto-Remediation Delivery Failure
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
- dqlite Datastore Vacuum (Freelist Bloat → Snapshot Amplification)
kube-apiserver¶
kubelet¶
- Control-Plane Watch-Cache Freeze (Zero Pod Creations / Stalled Reflectors)
- KCM Stale terminatingReplicas — ReplicaSet Refuses to Create Pods
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Post Incident Review: ARC GitHub Actions Runner Pods Stuck Pending — Kubelet Sync Loop Stall and Multi-Node Degradation
- Post Incident Review: Cascading Kubernetes Cluster Failures
- Post Incident Review: k8s01 Calico CNI Unauthorized — Stale Pod-Bound Token After Calico Upgrade
- Post Incident Review: k8s03 Extended Recovery — kine Watch Corruption, VXLAN Route Corruption, and Kubelet Watch Stream Stall
- Post Incident Review: microk8s 1.34 → 1.35 Rolling Upgrade — cgroup v2, containerd Shim, Disk Pressure, and Kubelet Stall
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
maintenance¶
microk8s¶
- Calico Orphaned Pod Route — Probe connect: invalid argument CrashLoop
- Control-Plane Watch-Cache Freeze (Zero Pod Creations / Stalled Reflectors)
- Jiva CSI Mount Proliferation
- Jiva CSI PVC Stuck After Pod Rescheduled to Different Node
- Jiva Controller Endpoint Deadlock
- Jiva Replica Corrupt Snapshot Chain — CrashLoopBackOff on Missing Snapshot Image
- Jiva-ctrl Eviction → iSCSI Session Drop → EXT4 Read-Only Filesystem
- KCM Stale terminatingReplicas — ReplicaSet Refuses to Create Pods
- Kubelet Silent Stall — Node Ready, Pods Never Schedule
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Post Incident Review: k8s02 Watch-Cache Freeze — Ansible Parallel dqlite Restart Stalls Pod Creation
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
- Safe Node Restart for Nodes Hosting jiva-ctrl Pods
- dqlite Datastore Vacuum (Freelist Bloat → Snapshot Amplification)
- dqlite Write Contention
- k8s Upgrade Post-Upgrade Validation
nagios¶
network¶
networking¶
- Calico CNI Unauthorized — Stale or Expired calico-kubeconfig Token
- Calico Orphaned Pod Route — Probe connect: invalid argument CrashLoop
- Post Incident Review: k8s01 Calico CNI Unauthorized — Stale Pod-Bound Token After Calico Upgrade
node-restart¶
openebs¶
- Jiva CSI Mount Proliferation
- Jiva CSI PVC Stuck After Pod Rescheduled to Different Node
- Jiva Controller Endpoint Deadlock
- Jiva Replica Corrupt Snapshot Chain — CrashLoopBackOff on Missing Snapshot Image
- Jiva-ctrl Eviction → iSCSI Session Drop → EXT4 Read-Only Filesystem
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Post Incident Review: Cascading Kubernetes Cluster Failures
- Post Incident Review: Radarr Outage Due to OpenEBS Jiva Replica Divergence
- Post Incident Review: Radarr Outage — OpenEBS Jiva Replica Divergence (Second Occurrence)
- Post Incident Review: k8s01 Calico CNI Unauthorized — Stale Pod-Bound Token After Calico Upgrade
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
- Post Incident Review: seerr Jiva CSI Stale Node Attachment — PVC Stuck After Cross-Node Rescheduling
- Safe Node Restart for Nodes Hosting jiva-ctrl Pods
outage¶
pleg¶
pleg-stall¶
quorum-loss¶
radarr¶
- Post Incident Review: Radarr Outage Due to OpenEBS Jiva Replica Divergence
- Post Incident Review: Radarr Outage — OpenEBS Jiva Replica Divergence (Second Occurrence)
rbac¶
replica-divergence¶
- Post Incident Review: Radarr Outage Due to OpenEBS Jiva Replica Divergence
- Post Incident Review: Radarr Outage — OpenEBS Jiva Replica Divergence (Second Occurrence)
ro-filesystem¶
- Jiva-ctrl Eviction → iSCSI Session Drop → EXT4 Read-Only Filesystem
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
runbook¶
- Calico CNI Unauthorized — Stale or Expired calico-kubeconfig Token
- Calico Orphaned Pod Route — Probe connect: invalid argument CrashLoop
- Control-Plane Watch-Cache Freeze (Zero Pod Creations / Stalled Reflectors)
- Jiva CSI Mount Proliferation
- Jiva CSI PVC Stuck After Pod Rescheduled to Different Node
- Jiva Controller Endpoint Deadlock
- Jiva Replica Corrupt Snapshot Chain — CrashLoopBackOff on Missing Snapshot Image
- Jiva-ctrl Eviction → iSCSI Session Drop → EXT4 Read-Only Filesystem
- KCM Stale terminatingReplicas — ReplicaSet Refuses to Create Pods
- Kubelet Silent Stall — Node Ready, Pods Never Schedule
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Runbooks
- Safe Node Restart for Nodes Hosting jiva-ctrl Pods
- dqlite Datastore Vacuum (Freelist Bloat → Snapshot Amplification)
- dqlite Write Contention
- k8s Upgrade Post-Upgrade Validation
scheduler¶
scheduling¶
- KCM Stale terminatingReplicas — ReplicaSet Refuses to Create Pods
- Post Incident Review: pvek8s Kernel Update — Simultaneous 3-Node Reboot Cascade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Auto-Remediation Delivery Failure
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: seerr Jiva CSI Stale Node Attachment — PVC Stuck After Cross-Node Rescheduling
simultaneous-reboot¶
snapshot-bloat¶
sonarr¶
storage¶
- Jiva CSI Mount Proliferation
- Jiva CSI PVC Stuck After Pod Rescheduled to Different Node
- Jiva Controller Endpoint Deadlock
- Jiva Replica Corrupt Snapshot Chain — CrashLoopBackOff on Missing Snapshot Image
- Jiva-ctrl Eviction → iSCSI Session Drop → EXT4 Read-Only Filesystem
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Post Incident Review: Radarr Outage Due to OpenEBS Jiva Replica Divergence
- Post Incident Review: k8s01 Calico CNI Unauthorized — Stale Pod-Bound Token After Calico Upgrade
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Auto-Remediation Delivery Failure
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes
- Post Incident Review: pvek8s dqlite WAL Lock Storm — Jiva Controller Endpoint Deadlock
- Post Incident Review: seerr Jiva CSI Stale Node Attachment — PVC Stuck After Cross-Node Rescheduling
- Safe Node Restart for Nodes Hosting jiva-ctrl Pods
upgrade¶
- Post Incident Review: microk8s 1.34 → 1.35 Rolling Upgrade — cgroup v2, containerd Shim, Disk Pressure, and Kubelet Stall
- k8s Upgrade Post-Upgrade Validation
vxlan¶
watch-cache¶
- Control-Plane Watch-Cache Freeze (Zero Pod Creations / Stalled Reflectors)
- Post Incident Review: k8s02 Watch-Cache Freeze — Ansible Parallel dqlite Restart Stalls Pod Creation
watch-stream¶
- KCM Stale terminatingReplicas — ReplicaSet Refuses to Create Pods
- Kubelet Volume Manager Stall — iSCSI WaitForAttachAndMount Hang
- Post Incident Review: dqlite Snapshot Bloat → kube-apiserver Instability → Controller Crash-Loop Cascade and Watch Stream Failure
- Post Incident Review: k8s03 Extended Recovery — kine Watch Corruption, VXLAN Route Corruption, and Kubelet Watch Stream Stall
- Post Incident Review: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Auto-Remediation Delivery Failure
- Post Incident Review: pvek8s Scheduling Outage — k8s03 Watch-Cache Freeze and Stale-Unit Watchdog Lockout
- Post Incident Review: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes