Alert rule: CephNodeRootFilesystemFull

Please consider opening a PR to improve this runbook if you gain new information about causes of the alert, or how to debug or resolve the alert. Click "Edit this Page" in the top right corner to create a PR directly on GitHub.

Overview

Root volume (OSD and MON store) is dangerously full. You should increase the space available to the node. The default location for the store sits under /var/lib/rook/ as a hostPath volume.

Steps for debugging

Check node low on disk space

$ ceph_cluster_ns=syn-rook-ceph-cluster
$ ceph_mon_name=a (1)
$ kubectl -n ${ceph_cluster_ns} get deploy -lapp=rook-ceph-mon,mon=${ceph_mon_name}  -ojson | jq '.items[].spec.template.spec.nodeSelector'
{
  "kubernetes.io/hostname": "storage-XXXX" (2)
}

1	The name of the monitor that’s alerting.
2	The node that the monitor is running on.

Increase the size of the node’s disk

Increase the size of the node’s disk according to your cloud provider’s documentation.