Skip to content

Runbooks

Jiva Replica Corrupt Snapshot Chain — CrashLoopBackOff on Missing Snapshot Image

Service: OpenEBS Jiva (pvek8s) First observed: 2026-06-28 (diagnosed and fixed 2026-07-13) PIR: pvek8s Storage Cascade — ArgoCD Sync Burst, Watch-Cache Freeze, and jiva iSCSI Read-Only Volumes

Symptom

A single jiva replica pod is in CrashLoopBackOff with a steadily climbing restart count (tens of restarts over days), while the volume's other replicas and its controller are Running and the workload using the volume is unaffected:

openebs  pvc-<vol-id>-jiva-rep-1   0/1   CrashLoopBackOff   82 (70s ago)   14d

The volume is silently running on degraded redundancy (2/3 replicas). Because the workload keeps working, this state can persist for days without attention — check restart count × age, not just pod status.

This is distinct from jiva-ctrl-endpoint-deadlock (replicas CrashLoop because controller endpoints are stuck notReadyAddresses — there the controller connection is the problem and multiple replicas are usually affected). Here exactly one replica crashes, and its log shows a local file error.

Root Cause

The replica's on-disk snapshot chain is inconsistent: volume.meta (and the .img.meta parent pointers) reference a snapshot or head image file that does not exist in the data directory. On startup, the replica opens the volume by walking this chain and hard-linking the head image; a missing file makes the open fail, the controller drops the connection, and the replica process exits fatally. Jiva has no self-repair for a broken chain — the pod restarts and fails identically forever.

The inconsistency is left behind by an ill-timed kill during snapshot-chain updates (metadata and image files are not updated atomically). Observed after dqlite storm / mass-restart windows. In the 2026-07-13 case the directory held 11 orphaned snapshot images and no head image at all.

Detection / Diagnosis

Step 1: Confirm the error signature

kubectl logs <rep-pod> -n openebs --previous --tail=20

Key signature:

Error link openebs/volume-head-017.img openebs/volume-snap-000.img: no such file or directory during open
...
Failed to handle connection, err: EOF, shutdown replica...

The named volume-snap-*.img (or the head image) is missing from the replica's data directory.

Second variant — head file present but non-empty (2026-09-15):

Error Can't remove head file volume-head-029.img as it contains some data during open
Error in request: Can't remove head file volume-head-029.img as it contains some data
...
Failed to handle connection, err: EOF, shutdown replica...

Same broken chain, opposite presentation: the replica wants to discard and re-link a head image it does not consider authoritative, and refuses because the file holds data. Recovery is identical (wipe and rebuild).

Step 2: Confirm quorum — MANDATORY before any destructive step

The rebuild wipes this replica's data. Only safe when at least 2 other replicas are RW:

CTRL=$(kubectl get pods -n openebs -o name | grep '<vol-id-prefix>.*ctrl')
kubectl exec -n openebs ${CTRL#pod/} -c jiva-controller -- \
  curl -s http://localhost:9501/v1/replicas | jq -r '.data[] | "\(.address) \(.mode)"'
# → need ≥2 lines showing RW (the crashing replica won't be listed or shows ERR)

If fewer than 2 replicas are RW: STOP. Wiping this replica risks data loss — investigate the other replicas first.

When two replicas of the same volume are broken, the gate sets the repair order (2026-09-15). With one RW replica left, no wipe is permitted at all: repair whichever other replica has a non-destructive fix first — e.g. a blackholed pod route, fixed by deleting the pod — to get back to 2 RW, and only then wipe the corrupt one. Repairing in symptom order instead of quorum order can leave the volume with a single copy mid-wipe.

Step 3: Identify the backing data directory and node

Replica data lives on a local PV bound to a per-replica PVC in the openebs namespace:

# Find the replica's own PVC → PV → hostPath and node
kubectl get pv -o custom-columns='PV:.metadata.name,NS:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name' | grep '<vol-id-prefix>'
kubectl get pv <rep-pv> -o jsonpath='{.spec.local.path} {.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}{"\n"}'
# → /var/openebs/local/<rep-pv>  k8s0N

Optionally inspect the directory to confirm the broken chain (missing head/snap images, orphaned .img files):

ssh <node> "sudo ls -la /var/openebs/local/<rep-pv>/"

Recovery — Wipe and Rebuild

The fix is to empty the replica's data directory so it re-registers with the controller as a fresh replica (WO mode) and performs a full resync from the healthy peers. ~10 min for a 5 GB volume.

Step 1: Wipe the data directory contents

Via a one-shot hostPath pod pinned to the replica's node (keeps the operation inside kubectl; delete the pod afterwards):

apiVersion: v1
kind: Pod
metadata:
  name: jiva-replica-wipe
  namespace: openebs
  labels:
    purpose: debug-cleanup
spec:
  restartPolicy: Never
  nodeName: <node>
  containers:
    - name: wipe
      image: docker.io/library/busybox:1.36
      command: ["sh", "-c", "rm -rf /data/* && echo WIPED && ls -la /data/"]
      volumeMounts:
        - name: replica-data
          mountPath: /data
  volumes:
    - name: replica-data
      hostPath:
        path: /var/openebs/local/<rep-pv>
        type: Directory

kubectl apply -f wipe-pod.yaml
kubectl logs jiva-replica-wipe -n openebs   # → WIPED, empty listing
kubectl delete pod jiva-replica-wipe -n openebs

(Equivalent: ssh <node> "sudo find /var/openebs/local/<rep-pv>/ -mindepth 1 -delete".)

Wipe contents only — the directory itself is the local-PV mount point and must remain.

Step 2: Restart the replica pod

kubectl delete pod <rep-pod> -n openebs

The StatefulSet recreates it; with an empty data dir it registers with the controller in WO (write-only/rebuilding) mode and resyncs.

Verification

# Replica pod stable — no new restarts
kubectl get pod <rep-pod> -n openebs
# → 1/1 Running, RESTARTS 0

# All replicas RW after resync completes (~10 min for 5G)
kubectl exec -n openebs ${CTRL#pod/} -c jiva-controller -- \
  curl -s http://localhost:9501/v1/replicas | jq -r '.data[] | "\(.address) \(.mode)"'
# → 3 lines, all RW

Prevention

  • Alert on prolonged CrashLoopBackOff in the openebs namespace — the 2026-07-13 case ran 92 restarts over 14 days unnoticed because the volume kept serving from 2/3 replicas (see PIR action items).
  • Note both jiva pod naming patterns when scoping checks: bare pvc-<id>-jiva-rep-N StatefulSet pods and pvc-<id>-rep-N-<hash> Deployment pods.

Recurrence — 2026-09-15 (4th and 5th occurrences): two replicas of different volumes corrupted in the same window — pvc-1908508d-...-jiva-rep-2 on k8s03 (14 orphaned 5 GB images, no head image, ~70 GB) and pvc-4aea2a19-...-jiva-rep-2 on k8s02 (13 orphaned images, no head image). Trigger identified for the first time: a Proxmox vzdump guest-agent fs-freeze suspended the node's filesystems mid-snapshot-chain update; both volume.meta mtimes fall inside the backup window. Do not run snapshot-mode backups of pvek8s node VMs with fs-freeze enabled — see PIR 2026-09-15. The orphaned images are also a silent disk-space leak: this pair held ~83 GB.

Recurrence — 2026-09-05 (3rd occurrence): two replicas hit this simultaneously (pvc-4e42a513-...-jiva-rep-2, 421 restarts/12d; pvc-8eccb718-...-jiva-rep-0, 141 restarts/10h), identical signature, fixed with this runbook's wipe-and-rebuild (homelabia#174). Detection worked as intended — microk8s-jiva-pod-health (added after the 2026-07-13 case) caught both — but detection alone hasn't stopped the underlying corruption trigger from recurring. Root-causing why it recurs is tracked separately under the storm-side lineage (#137/#140), not here.

References