Skip to content

Runbooks

Jiva-ctrl Eviction → iSCSI Session Drop → EXT4 Read-Only Filesystem

Service: OpenEBS Jiva iSCSI (pvek8s) First observed: 2026-05-28 PIR: pvek8s Post-Power-Outage Recovery — kubelet Volume Manager Stall and KCM Stale terminatingReplicas Linear: PGM-224

Symptom

A pod running an OpenEBS Jiva-backed PVC enters a read-only or error state. The pod may:

  • Log Read-only file system errors
  • Enter CrashLoopBackOff or Error state
  • Become stuck in Failed or Terminating on a cordoned node (jiva-csi cannot run chmod during teardown because the filesystem is read-only)

In dmesg on the node running the pod: 

EXT4-fs (sdX): Remounting filesystem read-only

This is distinct from the kubelet-volume-manager-stall scenario (where pods are stuck in ContainerCreating and iSCSI never attached). Here, iSCSI was attached and the pod was running — then lost its storage mid-flight.

Root Cause

There are two distinct paths to the same ext4 read-only outcome. Both end in session recovery timed out after 120 secs, but they differ in what killed the session — and therefore in what you should check first.

Mode A: target killed Mode B: initiator starved
Trigger jiva-ctrl pod evicted/killed dqlite write storm starves the node
First kernel signature conn error (1020) (TCP RST / refused) ping timeout of 5 secs expired → conn error (1022)
jiva-ctrl state Restarted/evicted before the remount Healthy throughout; any restart comes after
JivaVolume CR May show stale mountInfo Ready / RW throughout
Blast radius Volumes served by that one ctrl Every volume on the affected node(s), often several nodes at once
First observed 2026-05-28 2026-08-06

Distinguish them first — a ping timeout line before any 1020 means Mode B, and the entire "find the evicted ctrl" branch below is a dead end:

ssh <node> "sudo journalctl -k --since '<incident window>' | grep -E 'ping timeout|conn error|session recovery'" | head

Mode A: jiva-ctrl evicted (target killed)

The jiva-ctrl pod (iSCSI target) running on some node was evicted or killed while an iSCSI initiator on another node had an active session to it.

Full cascade:

  1. A node hosting jiva-ctrl pod(s) receives a NoExecute taint (NotReady, maintenance, or recovery rolling restart)
  2. The taint-eviction-controller deletes the jiva-ctrl pods — the iSCSI target process exits
  3. The iSCSI initiator on the workload node detects conn error (1020) (TCP RST or connection refused)
  4. iSCSI session recovery runs for 120 seconds — if the target does not reappear, the session is declared dead
  5. The kernel marks the SCSI block device offline; in-flight I/O returns -EIO
  6. JBD2 (ext4 journal) aborts on the first failed write, setting JBD2_ABORT flag
  7. EXT4 detects the aborted journal on the next write attempt and remounts the filesystem read-only

Batched eviction amplifier: During cluster recovery, if the kube-controller-manager was temporarily disconnected from dqlite, pending taint evictions queue up. On reconnect, all queued jiva-ctrl pods are evicted simultaneously — dropping all iSCSI sessions at once, leaving no time for individual session recovery.

Earlier link-flap comparison: A brief physical network event (eth0 down <30s) will also trigger error 1020, but the sessions recover once connectivity returns because the iSCSI target is still alive. The critical difference here is that the target process itself was killed.

Mode B: dqlite storm starves the initiator (target healthy)

Added 2026-08-06. The Jiva target never dies — the initiator misses its keepalive.

Full cascade:

  1. A dqlite write-contention storm begins, retrying hundreds of times per key per second: 
    level=error msg="error in txn: update transaction failed for key
/registry/leases/kube-system/kube-controller-manager: exec (try: 500): database is locked"
  2. The resulting CPU/scheduler pressure delays the kernel iSCSI initiator's keepalive processing past its 5-second node.session.timeo.noop_out_timeout deadline
  3. ping timeout of 5 secs expired → detected conn error (1022) — note 1022, not 1020
  4. Session recovery runs 120s and fails; the block device goes offline; JBD2 aborts; ext4 remounts ro

Because the trigger is node-wide rather than volume-specific, this mode hits every Jiva volume on the affected node, and typically several nodes simultaneously — on 2026-08-06 it took six volumes across k8s02 and k8s03 within seven minutes.

Do not go looking for an evicted jiva-ctrl in this mode. The JivaVolume CRs stay Ready/RW throughout, and any jiva-ctrl restart you find in the logs will post-date the remount (on 2026-08-06 the ctrl pods restarted only when the watch-cache auto-remediation restarted kubelite, 12 minutes after the filesystems went read-only).

Structural cause: on a hyperconverged 3-node cluster the iSCSI initiator, dqlite, kubelite and every workload contend for the same CPUs. The default 5s ping / 120s replacement timeouts assume a dedicated storage network, so a control-plane stall is sufficient to kill a storage session.

Detection

Automated (since 2026-07-11, homelabia#140)

Two nagios checks page on this failure mode — check these first, they usually identify the volume and node before any manual digging:

  • microk8s-ro-pvc-mounts — CRITICAL while any PVC volume is mounted read-only on the node, naming each PVC in the alert output. State-based: stays CRITICAL until the volume is remounted rw, so it also catches long-standing silent cases (readarr sat ro for ~8 days before this existed).
  • microk8s-storage-kernel-errors — CRITICAL when hard failure signatures (Aborting journal, Remounting filesystem read-only, session recovery timed out, rejecting I/O to offline device) appear in the node's kernel log within 30 min — fires during the cascade itself, before applications notice. Plain iSCSI conn error lines are perfdata-only (conn_errors=): they occur on benign jiva-ctrl restarts and orphaned-session retries and do not alert.

After any dqlite storm or watch-cache freeze, survey all nodes even if the checks are green (a remount can predate the check window):

for n in k8s01 k8s02 k8s03; do ssh $n "grep -E 'pvc-.* ext4 ro' /proc/mounts"; done

Step 1: Confirm filesystem is read-only on the affected pod

# Check pod events
kubectl describe pod <pod-name> -n <namespace> | grep -iE 'read.only|error|io error'

# Check pod logs for ro filesystem errors
kubectl logs <pod-name> -n <namespace> | grep -iE 'read.only file system|EROFS|I/O error'

Step 2: Confirm EXT4 ro remount in node dmesg

Find which node the pod is (or was) on, then check dmesg:

NODE=$(kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeName}')

# Via the jiva-csi-node DaemonSet pod on that node
JIVA_POD=$(kubectl get pods -n openebs -l app=openebs-jiva-csi-node \
  -o jsonpath="{.items[?(@.spec.nodeName=='$NODE')].metadata.name}")

kubectl exec -n openebs $JIVA_POD -c jiva-csi-plugin -- dmesg | \
  grep -E 'EXT4.*Remounting|conn error|session recovery timed out|I/O error.*sd[a-z]|Aborting journal'

Key signatures: 

iscsid: connection1:0: detected conn error (1020)
iscsid: session1: session recovery timed out after 120 secs
EXT4-fs (sdX): Remounting filesystem read-only

Step 3: Identify the affected PVC and jiva-ctrl ClusterIP

# Get the PVC name from the pod spec
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}'

# Get the jiva-ctrl service ClusterIP (this is what iSCSI connects to)
PVC_NAME=<pvc-name>
PV_NAME=$(kubectl get pvc $PVC_NAME -n <namespace> -o jsonpath='{.spec.volumeName}')
kubectl get svc -n openebs | grep "${PV_NAME:0:25}"

Step 4: Check JivaVolume CR state

kubectl get jivavolume -n openebs | grep "<pvc-partial-name>"
kubectl get jivavolume <pvc-name> -n openebs -o jsonpath='{.spec.mountInfo}'
# Stale if nodeID does not match the node you're trying to mount on

Recovery

The operative step is the iSCSI logout, not the pod recreation

Everything below works for one reason: kubelet tears down the global device mount and logs out the iSCSI session only once no pod on the node references the volume. That logout is what lets the next mount do a fresh login and replay the ext4 journal rw. Any procedure that leaves the volume referenced — even for a few seconds — reuses the stale ro mount and silently does nothing.

A plain kubectl delete pod does not work (verified 2026-08-06, readarr): the ReplicaSet recreates the pod within seconds and it re-binds the stale ro global mount before kubelet can drop it. The volume stays ro,relatime and the pod comes back just as broken.

Fast path A — scale to zero (preferred; proven 2026-08-06, six volumes)

Works whether or not another node is available, and does not depend on winning a race with the scheduler:

kubectl --context pvek8s scale deploy -n <ns> <name> --replicas=0
# StatefulSets: kubectl scale statefulset -n <ns> <name> --replicas=0

# Confirm FULL unreference before scaling back up — both must be true:
ssh <node> "mount | grep <pvc>"      # → empty
ssh <node> "ls -la /dev/sdX"         # → No such file or directory (session logged out)

kubectl --context pvek8s scale deploy -n <ns> <name> --replicas=1

If /dev/sdX is still present after the mounts clear, the session did not drop — go to "Persisted superblock error state" below, because a remount alone will not restore rw.

ArgoCD will fight you. On apps with auto-sync enabled, ArgoCD reverts --replicas=0 within roughly 40 seconds. The unmount usually still completes inside that window, but the procedure becomes racy — prefer disabling auto-sync for the app first:

argocd app set <app> --sync-policy none    # re-enable with --sync-policy automated

Fast path B — cordon then delete (2026-07-11/13; use when another node is free)

If the jiva-ctrl for the volume is healthy (2/2 Running), cordon the node with the ro mount, then kubectl delete pod: kubelet detaches (unmount + iSCSI logout), the replacement pod lands on another node, triggers a fresh login and mount, and ext4 journal replay remounts rw. Uncordon once the replacement is Running elsewhere.

Cordon-first is mandatory, not optional (confirmed 2026-07-13, radarr): the scheduler has no memory of the failure — a plain delete relanded the replacement on the same node, where it silently bind-mounted the stale ro global mount.

This path fails if no other node can take the pod (node affinity, capacity, or all nodes affected — common in Mode B). Use Fast path A instead.

Persisted superblock error state (fsck required)

Added 2026-08-06. If the volume comes back still ro after a clean unmount and remount, the ext4 error flag is persisted in the superblock and no amount of remounting will clear it:

ssh <node> "sudo dumpe2fs -h /dev/sdX 2>&1 | grep -Ei 'filesystem state|FS Error count|First error'"

Filesystem state:         clean with errors      <-- remount will NOT restore rw
Filesystem features:      ... needs_recovery ...
FS Error count:           2
First error function:     ext4_journal_check_start

With the workload scaled to 0 and the device confirmed unmounted:

ssh <node> "sudo e2fsck -f -y /dev/sdX"
ssh <node> "sudo dumpe2fs -h /dev/sdX 2>&1 | grep -i 'filesystem state'"
# → Filesystem state:         clean

On 2026-08-06 this recovered the journal on borked-craft's volume with only minor bitmap/inode corrections and all 789 files intact. e2fsck may warn /dev/sdX is mounted even when mount shows nothing — microk8s kubelet uses its own mount namespace. Verify with grep /dev/sdX /proc/mounts before overriding.

Verifying inside the pod, not on the host

microk8s kubelet mounts in its own namespace, so a volume can be correctly mounted and writable inside the pod while mount on the host shows nothing at all. Do not read an empty host mount table as "the volume failed to attach" — check from inside:

kubectl --context pvek8s exec -n <ns> <pod> -- sh -c 'touch /data/.rwtest && rm /data/.rwtest && echo RW-OK'

The full phases below are only needed when stale prior state gets in the way:

  • Pod wedged in Terminating (jiva-csi NodeUnpublish chmod fails EROFS; the driver attempts unmount once and never retries) → a single manual umount of the pod mount path on the node is the complete fix (Phase 2 step 1 only — confirmed 2026-07-13, seerr). Globalmount teardown and iSCSI logout then complete on their own, and a transient already mounted at more than one place FailedMount on the new node self-clears on jiva-csi's next retry — no JivaVolume CR patch needed.
  • New node's login rejected with target already connected (jiva is single-initiator; a stale session elsewhere holds the slot) → restart the jiva-ctrl pod to drop all sessions instead of hunting the stale one, or use Phase 3 below.
  • Never --force --grace-period=0 a pod with proliferated CSI bind mounts: kubelet unmounts most but any stranded pod-path bind mount wedges UnmountDevice forever (GetDeviceMountRefs check failed, 2-minute retry loop) — the only fix is a manual umount of the leftover path. Let slow teardowns finish. (Re-confirmed the hard way on 2026-08-06: the minecraft pods were force-deleted and produced exactly this wedge, costing four manual unmounts.)

The wedge looks like this in the kubelite journal, retrying forever: 

Error: GetDeviceMountRefs check failed for volume "pvc-746b2837-..." on node "k8s02" :
the device mount path ".../globalmount" is still mounted by other references
[.../pods/<uid>/volumes/kubernetes.io~csi/pvc-746b2837-.../mount]
Unmount the path named in the brackets and kubelet completes teardown on its next retry: 
ssh <node> "sudo umount /var/snap/microk8s/common/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pvc>/mount"
If you manually unmount a path for a pod that still exists, that pod's mount is now broken and it will start with an empty d--------- /data. Delete it so kubelet rebuilds the mount cleanly.

Phase 1: Assess and stabilise

  1. Determine whether the affected node is cordoned: 

    kubectl get node <node> -o jsonpath='{.spec.unschedulable}'
# → "true" = cordoned

  2. If the node is not cordoned and the pod might recover, check whether the jiva-ctrl has restarted and the iSCSI session can re-establish: 

    # Check jiva-ctrl pod status
kubectl get pods -n openebs | grep "<pvc-partial-name>.*ctrl"

# Check iSCSI session state on the affected node
kubectl exec -n openebs $JIVA_POD -c jiva-csi-plugin -- iscsiadm -m session
    If the session is re-established and the filesystem is still ro, proceed to Phase 2. If the jiva-ctrl is running but the session is absent, the 120s timeout already expired — proceed to Phase 2.

Phase 2: Unstick the pod if stuck in Failed/Terminating

A pod on a read-only filesystem will not complete deletion because jiva-csi cannot chmod the mount directory. The deletion finalizer is never cleared.

  1. Unmount the stale CSI mounts from inside the jiva-csi-node container on the affected node: 

    # Get the pod UID
POD_UID=$(kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.metadata.uid}')
PVC_NAME=<pvc-name>

# Unmount pod volume bind mount
kubectl exec -n openebs $JIVA_POD -c jiva-csi-plugin -- \
  umount /var/snap/microk8s/common/var/lib/kubelet/pods/${POD_UID}/volumes/kubernetes.io~csi/${PVC_NAME}/mount

# Unmount CSI globalmount (staging)
# Get the vol-id from the path
VOL_ID=$(kubectl exec -n openebs $JIVA_POD -c jiva-csi-plugin -- \
  sh -c 'ls /var/snap/microk8s/common/var/lib/kubelet/plugins/jiva.csi.openebs.io/')
kubectl exec -n openebs $JIVA_POD -c jiva-csi-plugin -- \
  umount /var/snap/microk8s/common/var/lib/kubelet/plugins/jiva.csi.openebs.io/${VOL_ID}/globalmount

  2. Delete the stuck pod (now that mounts are clear, the finalizer can complete): 

    kubectl delete pod <pod-name> -n <namespace>

Phase 3: Log out stale iSCSI session from old node

If the pod was on node A and you want to reschedule to node B, node A's iSCSI initiator may still have the session registered (even if the session is dead). This will cause "already mounted at more than one place" errors on node B.

# Get jiva-ctrl IQN and portal IP
IQN="iqn.2016-09.com.openebs.jiva:<pvc-name>"
PORTAL_IP=$(kubectl get svc -n openebs -o jsonpath="{.items[?(@.metadata.name contains '<pvc-partial-name>')].spec.clusterIP}")

# Log out from old node via jiva-csi-node container
kubectl exec -n openebs $JIVA_POD -c jiva-csi-plugin -- \
  iscsiadm -m node -T $IQN -p ${PORTAL_IP}:3260 --logout

# Verify session is gone
kubectl exec -n openebs $JIVA_POD -c jiva-csi-plugin -- iscsiadm -m session

Phase 4: Clear stale JivaVolume CR mountInfo

If the JivaVolume CR still has spec.mountInfo from the old node, jiva-csi on the new node will refuse to mount, reporting "already mounted":

kubectl patch jivavolume <pvc-name> -n openebs --type=merge \
  -p '{"spec":{"mountInfo":{"devicePath":"","stagingPath":""}}}'

Note: jiva-operator may repopulate this field quickly during active NodeStageVolume attempts. If the new pod is already trying to mount, the patch may be overwritten. If so: 1. Cordon the new node first to stop scheduling 2. Apply the patch 3. Uncordon to allow the pod to reschedule to a clean node

Phase 5: Reschedule the pod to a healthy node

If the original node is cordoned, the pod will not reschedule automatically (for StatefulSets in particular). Force-delete the pod after Phases 2–4 are complete:

kubectl delete pod <pod-name> -n <namespace> --force --grace-period=0

The StatefulSet controller will recreate the pod. It will schedule to a node where jiva-csi can successfully login iSCSI and mount the volume rw.

Phase 6: Verify

# Pod is Running with rw filesystem
kubectl get pod <pod-name> -n <namespace>
# → 1/1 Running

# Verify mount is rw on the new node
NEW_NODE=$(kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeName}')
NEW_JIVA_POD=$(kubectl get pods -n openebs -l app=openebs-jiva-csi-node \
  -o jsonpath="{.items[?(@.spec.nodeName=='$NEW_NODE')].metadata.name}")
kubectl exec -n openebs $NEW_JIVA_POD -c jiva-csi-plugin -- \
  grep "<pvc-name>" /proc/mounts
# Should show rw (not ro) in the mount options

# iSCSI session active on new node
kubectl exec -n openebs $NEW_JIVA_POD -c jiva-csi-plugin -- iscsiadm -m session
# → tcp: [...] iqn.2016-09.com.openebs.jiva:<pvc-name> (non-flash)

# No stale iSCSI sessions on old node
kubectl exec -n openebs $OLD_JIVA_POD -c jiva-csi-plugin -- iscsiadm -m session
# → (empty or no matching session)

Prevention

During cluster recovery / rolling node restarts

Before applying a NoExecute taint to or draining a node:

  1. Identify jiva-ctrl pods on that node: 

    kubectl get pods -n openebs -o wide | grep "<node-name>" | grep "ctrl"

  2. For each jiva-ctrl pod, find which nodes have active iSCSI sessions to it: 

    # Get the controller service ClusterIP
kubectl get svc -n openebs | grep "<pvc-partial>"

# Check all jiva-csi-node containers for active sessions to that ClusterIP
for pod in $(kubectl get pods -n openebs -l app=openebs-jiva-csi-node -o name); do
  echo "=== $pod ==="; \
  kubectl exec -n openebs $pod -c jiva-csi-plugin -- \
    iscsiadm -m session 2>/dev/null | grep "<clusterIP>"
done

  3. If sessions exist on other nodes: First delete the workload pods that use those PVCs, allow them to reschedule to a node NOT hosting the jiva-ctrl, and verify iSCSI re-attaches to a different controller. Then proceed with the node restart.

  4. If no sessions exist: Safe to proceed directly.

See jiva-ctrl-node-rolling-restart.md for the full step-by-step procedure including commands to identify sessions, migrate workloads, and verify logout before restarting.

Against Mode B (initiator starvation)

Node-level prevention, since there is no jiva-ctrl to migrate:

  1. Watch for the storm before it reaches storage. A dqlite storm precedes the first ping timeout by only ~2 minutes — see dqlite-write-contention.md. Once try: 500 retry depths appear, storage is already at risk.
  2. After any dqlite storm or watch-cache freeze, survey every node, even if checks are green — a remount can predate the check window: 
    for n in k8s01 k8s02 k8s03; do ssh $n "grep -E 'pvc-.* ext4 ro' /proc/mounts"; done
  3. Beware the remediation itself. Auto-remediation restarts kubelite, which duplicates CSI bind mounts (see jiva-csi-mount-proliferation.md). On 2026-08-06 the fix attempt produced 16x stacked mounts on both CSI volumes. Check for duplication after any remediation run.

Structural mitigations (not yet implemented)

  • Extended NoExecute toleration on jiva-ctrl pods (tolerationSeconds=600) — gives more time for transient NotReady to resolve before eviction fires. Tracked: PGM-222.
  • iSCSI session recovery timeout — increase node.session.timeo.replacement_timeout from 120s to 300s+ to give jiva-ctrl pods more time to restart and re-register. Configure via iscsiadm on each node or in the jiva-csi DaemonSet. Raised in priority by the 2026-08-06 incident: in Mode B this timeout (together with the 5s noop_out_timeout) is the entire failure mechanism — a control-plane CPU stall of ~2 minutes was sufficient to destroy six volumes' sessions.
  • Monitoring — log-based alerts on EXT4 ro remount and iSCSI session failure patterns in kern.log. Tracked: PGM-221.

References