Alert rule: CephNodeDiskspaceWarning

Please consider opening a PR to improve this runbook if you gain new information about causes of the alert, or how to debug or resolve the alert. Click "Edit this Page" in the top right corner to create a PR directly on GitHub.

Overview

This alert is triggered when the disk space used by a storage node will be full in less than 5 days assuming the average fill-up rate of the past 48 hours. You should increase the space available to the node. The default location for the store sits under /var/lib/rook/ as a hostPath volume.

Steps for debugging

Check node low on disk space

$ ceph_cluster_ns=syn-rook-ceph-cluster
$ ceph_mon_name=a (1)
$ kubectl -n ${ceph_cluster_ns} get deploy -lapp=rook-ceph-mon,mon=${ceph_mon_name}  -ojson | jq '.items[].spec.template.spec.nodeSelector'
{
  "kubernetes.io/hostname": "storage-XXXX" (2)
}
1 The name of the monitor that’s alerting.
2 The node that the monitor is running on.

Increase the size of the node’s disk

Increase the size of the node’s disk according to your cloud provider’s documentation.