Alert rule: APPUiOCloudReportingDatabaseTenantMappingFailed

Please consider opening a PR to improve this runbook if you gain new information about causes of the alert, or how to debug or resolve the alert. Click "Edit this Page" in the top right corner to create a PR directly on GitHub.

Overview

APPUiO Cloud Reporting failed while trying to add a tenant target mapping from billing metrics to the reporting database. This can lead to problems during the monthly invoice generation.

Steps for debugging

  • Check job logs.

  • Check for missing migrations:

    NAMESPACE=appuio-cloud-reporting
FAILED_JOB=job/XXXXXX-XXXXXX

oc --as=cluster-admin -n "${NAMESPACE}" debug "${FAILED_JOB}" \
  --keep-init-containers=false --image=ghcr.io/appuio/appuio-cloud-reporting:latest -- \
  appuio-cloud-reporting migrate --show-pending

# If pending migrations are shown
oc --as=cluster-admin -n "${NAMESPACE}" debug "${FAILED_JOB}" \
  --keep-init-containers=false --image=ghcr.io/appuio/appuio-cloud-reporting:latest -- \
  appuio-cloud-reporting migrate
oc --as=cluster-admin -n "${NAMESPACE}" debug "${FAILED_JOB}" \
  --keep-init-containers=false --image=ghcr.io/appuio/appuio-cloud-reporting:latest -- \
  appuio-cloud-reporting migrate --seed

  • Check Prometheus/Thanos connectivity and logs.

After fixing the underlying issue, the next run of the cronjob should fix the problem.