How We Fixed a
Ceph Kernel Crash on 8 K8s Nodes
A Ceph RBD kernel incompatibility cascaded into a multi-node NotReady crisis during an OS upgrade. Here's how Database Mart engineers stabilized the cluster, found the right kernel path, and completed a full Kubernetes v1.34.5 upgrade — without data loss.
* 14 days from first PVC failure to full cluster upgrade — kernel 5.4 → 6.17, Ubuntu 20 → 24, Kubernetes v1.33 → v1.34.5.
Ceph RBD PVC Mount Failures — Kernel 5.4 Too Old for Ceph Squid
The <CUSTOMER_TEAM> NOC team reported persistent PVC mount failures across their development Kubernetes cluster after a recent Ceph upgrade to Ceph 19.2.3 (Squid). The error pointed directly at the kernel RBD driver:
rbd: map failed: (22) Invalid argument rbd: sysfs write failed MountVolume.MountDevice failed ... --device-type krbd --options noudev
The cluster was running kernel 5.4.0-216-generic on the master nodes (Ubuntu 20.04). This kernel predates several Ceph RBD features and security fixes introduced in Ceph Squid's requirements. The storage layer was functional but PVC mounts on pods were consistently failing.
The request: upgrade node kernels to a version compatible with Ceph/RBD requirements, and include a Kubernetes upgrade to v1.34.5 in the same maintenance window.
Engineer Robert proposed a rolling upgrade approach — test on node01 first, validate Ceph OSD health, then proceed to remaining nodes.
node01 Upgraded — But OSDs Went Down and fstab Sent It to Maintenance Mode
Engineer Jonathan upgraded node01 to Ubuntu 22.04 with the HWE kernel 6.8.0-106-generic. Two problems surfaced immediately:
- fstab conflict: The node's
/etc/fstabhad entries mounting/dev/sdcand/dev/sdd— the disks used by Ceph OSDs — as regular filesystems. On reboot, the OS couldn't find these devices (Ceph manages them exclusively), causing the node to drop into maintenance/emergency mode and fail to rejoin the cluster. - Kernel 6.8 + Ceph Squid: After fixing the boot issue, the two OSDs on
node01(osd.0andosd.1) wentdown. Kernel 6.8 had known compatibility issues with Ceph RBD block device mapping in this environment.
Resolution: Bruce rolled back node01 to kernel 5.15.0-173-generic (Ubuntu 22.04 GA kernel). With the customer's permission, the two downed OSDs (osd.0, osd.1) were rebuilt. All 10 OSDs returned to up status. The lesson: skip the 6.8 HWE kernel entirely for Ceph Squid environments.
# OSD state after kernel rollback + OSD rebuild: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT -4 2.75089 host node01 0 hdd 1.81940 osd.0 up 1.00000 ✓ 1 ssd 0.93149 osd.1 up 1.00000 ✓ # All 10 OSDs: up ✓
Ubuntu 22 GA Kernel Works — But Can Ubuntu 24 + Kernel 6.17 Do Better?
With node01 stable on kernel 5.15.x, Bruce proceeded with a rolling upgrade of node02 through node05 to Ubuntu 22.04 with the same GA kernel — one node at a time, with Ceph OSD verification between each. All worker nodes reached Ready status with all 10 OSDs healthy.
At this point, the customer requested a test upgrade of node05 to Ubuntu 24.04 LTS — but specifically with the HWE kernel 6.17 (not 6.8), to evaluate whether the latest LTS + latest HWE kernel resolved the Ceph compatibility issue that 6.8 had caused. Ubuntu 24.04 is the current LTS with better long-term support horizon.
Bruce upgraded node05 to Ubuntu 24.04.4 LTS with kernel 6.17.0-20-generic. Result:
Kernel 6.17 was confirmed compatible with Ceph Squid. The path forward was clear: Ubuntu 24.04 + kernel 6.17 for all nodes, skipping 6.8 entirely. The customer approved rollout to all remaining nodes.
master02 Upgrade Triggers Multi-Node Outage — 6 Nodes Go NotReady
On April 3, Bruce upgraded master02 to Ubuntu 24. Within hours, the cluster entered a critical state: only master01 and master03 were reporting as online. The remaining 6 nodes — including all worker nodes — were NotReady.
The <CUSTOMER_TEAM> NOC team escalated urgency on April 6 after no resolution for several days:
Bruce's investigation identified two compounding problems:
- Worker nodes (node01–05): The master02 control plane disruption caused all worker nodes to lose cluster synchronization. These were restored first by reintegrating them from master01/master03. All 5 worker nodes returned to
Readyby April 7. - master02 specifically: The
kube-flannelCNI network plugin was not functioning correctly on master02 after the Ubuntu 24 upgrade with kernel 6.8.0-107 — the wrong kernel version. master02 remainedNotReady.
The pattern was now unmistakable: kernel 6.8 was the recurring source of instability, whether on worker nodes (OSD failures) or control plane nodes (CNI failures). The fix was to align master02 to the confirmed-working kernel 6.17.0-20-generic.
| Node | Role | Status (Apr 7) | Kernel |
|---|---|---|---|
| master01 | control-plane | Ready | 5.4.0-216-generic |
| master02 | control-plane | NotReady | 6.8.0-107 ⚠️ |
| master03 | control-plane | Ready | 5.4.0-216-generic |
| node01–05 | worker | Ready ✓ | 6.17.0-20-generic |
Bruce upgraded master02 to kernel 6.17, fixed the kube-flannel conflict, and rejoined it to the control plane. master02 returned to Ready by April 8.
All 8 Nodes on Ubuntu 24.04 + Kernel 6.17 → Kubernetes v1.34.5 Complete
With the kernel compatibility path validated across all node types, Bruce completed the Ubuntu 24.04 + kernel 6.17 rollout for all remaining nodes (master01, master03). All 8 nodes were now aligned on the same OS and kernel baseline:
| Node | Role | OS | Kernel | Container Runtime |
|---|---|---|---|---|
| master01 | control-plane | Ubuntu 24.04.4 LTS | 6.17.0-20-generic | containerd://2.2.2 |
| master02 | control-plane | Ubuntu 24.04.4 LTS | 6.17.0-20-generic | containerd://2.2.2 |
| master03 | control-plane | Ubuntu 24.04.4 LTS | 6.17.0-20-generic | containerd://2.2.2 |
| node01–05 | worker | Ubuntu 24.04.4 LTS | 6.17.0-20-generic | containerd://2.2.x |
With the environment fully stabilized, Bruce proceeded with the Kubernetes upgrade from v1.33.10 to v1.34.5 across all 8 nodes. The following components were upgraded:
# Final cluster state — April 10, 2026: NAME STATUS ROLES VERSION master01 Ready control-plane v1.34.5 master02 Ready control-plane v1.34.5 master03 Ready control-plane v1.34.5 node01 Ready <none> v1.34.5 node02 Ready <none> v1.34.5 node03 Ready <none> v1.34.5 node04 Ready <none> v1.34.5 node05 Ready <none> v1.34.5 # Post-upgrade checks: ✅ Control plane healthy # ✅ DNS resolution OK # ✅ All 10 Ceph OSDs up # ✅ Node connectivity verified
RESOLVED K8S v1.34.5 UBUNTU 24.04 CEPH ALL OSDs UP
Not all kernels work equally with Ceph RBD block device mapping. Based on this case, the confirmed behavior with Ceph 19.2.x (Squid) is:
- ❌ Kernel 5.4.x (Ubuntu 20.04 default) — Too old, RBD
krbdfailures with--device-type krbd --options noudev - ⚠️ Kernel 6.8.x (Ubuntu 24.04 default HWE) — OSD mounting failures, CNI instability on control plane nodes
- ✅ Kernel 5.15.x (Ubuntu 22.04 GA) — Stable with Ceph Squid, safe baseline
- ✅ Kernel 6.17.x (Ubuntu 24.04 latest HWE) — Fully compatible, recommended for new deployments
💡 When upgrading Ubuntu 24.04, always install the HWE kernel explicitly: apt install linux-image-6.17.0-20-generic — don't rely on the default 6.8 that ships with the base ISO.
If a node has Ceph OSD disks (/dev/sdc, /dev/sdd, etc.), those disks must not appear in /etc/fstab. Ceph manages them exclusively via its own bluestore format. On reboot after an OS upgrade, the kernel will try to mount fstab entries before Ceph initializes — and fail, dropping the node into emergency/maintenance mode.
# Before any OS upgrade on a Ceph OSD node, check: cat /etc/fstab | grep -E "sdc|sdd|sde" # Any Ceph OSD disk appearing here → remove the line # Ceph bluestore disks have no mountable filesystem # After upgrade, verify OSDs came back up: kubectl -n rook-ceph exec -it deploy/rook-ceph-tools \ -- ceph osd tree | grep -E "up|down"
The order of operations matters enormously. Worker nodes are safer to upgrade first — a failed worker affects workloads but not the control plane. A failed master node during upgrade can cascade to NotReady on all workers.
ceph osd tree and ceph health detail. Wait at least 10 minutes for Ceph rebalancing to settle.kubectl drain <node> --ignore-daemonsets --delete-emptydir-data to evict workloads safely.up before proceeding to the next.kubeadm upgrade apply. Mixed-version OS environments increase risk during the k8s upgrade.When a node goes NotReady after an OS/kernel upgrade, work through these checks in order:
# 1. Check kubelet status and logs systemctl status kubelet journalctl -u kubelet -n 50 --no-pager # 2. Check container runtime systemctl status containerd crictl info # 3. Check CNI plugin health (kube-flannel, calico, etc.) kubectl -n kube-system get pods | grep -E "flannel|calico|cni" kubectl -n kube-system logs <cni-pod> --tail=50 # 4. Check if kernel modules are loaded (critical for Ceph RBD) lsmod | grep rbd modprobe rbd # 5. For control plane nodes — check etcd and apiserver kubectl get componentstatuses kubectl -n kube-system get pods | grep etcd
🔍 In this case, master02's NotReady was caused by kube-flannel failing under kernel 6.8 — visible in the CNI pod logs but not in kubelet logs alone.
"Kernel 6.8 broke the OSDs. Kernel 6.17 fixed everything.
Sometimes the right version number is the entire solution —
the hard part is knowing which one to trust."