Alert rule: APPUiOCloudInvoiceGenerationFailed

Please consider opening a PR to improve this runbook if you gain new information about causes of the alert, or how to debug or resolve the alert. Click "Edit this Page" in the top right corner to create a PR directly on GitHub.

Overview

APPUiO Cloud Reporting failed while trying to generate invoices. Invoices might be incomplete or missing. Remove any obstacles and rerun the job.

The job isn’t idempotent. Rerunning the job, can leave duplicates which need to be cleaned up.

Steps for debugging

  1. Identify source of failure

    • Check for missing data indicated by the alert rule APPUiOCloudReportingMissingData.

    • Check logs for other issues. Look out for connectivity issues (for example between the job and the ERP).

      kubectl -n appuio-cloud-reporting logs $(kubectl -n appuio-cloud-reporting get job -l cron-job-name=generate-invoices --field-selector status.successful=0 -o name | tail -n1) --all-containers

  2. Remove any obstacles

    Use the findings from the previous step to remove any obstacles causing the job to fail again.

  3. Rerun job

    name="generate-invoices-manual-$($(command -v gdate || command -v date) +%s)"
kubectl --as cluster-admin -n appuio-cloud-reporting create job --from=cronjob/generate-invoices "${name}"
kubectl -n appuio-cloud-reporting logs "job/${name}"

  4. Check ERP for duplicates

    In the ERP, search for the generated invoices. Remove all duplicates.

    Invoices with status Draft can be removed without issues. For other states, check with finance. They might redo some modifications.

    Use the advanced search to search "APPUiO Cloud <month> <year>" in the field Reference/Description. Then sort by Customer. This makes spotting duplicates easier.

  5. Clean up

    Remove all failed check-missing jobs to clear the alert.

    kubectl -n appuio-cloud-reporting delete $(kubectl -n appuio-cloud-reporting get job -l cron-job-name=generate-invoices --field-selector status.successful=0 -o name)