Alert rule: KyvernoAdmissionLatencyReachedWebhookTimeout

Unresolved include directive in modules/ROOT/pages/runbooks/KyvernoAdmissionLatencyReachedWebhookTimeout.adoc - include::partial$runbooks/contribution_note.adoc[]

Overview

Kyverno average admission latency is higher than the webhook timeout. Creating or updating Kubernetes object failed because of the admission latency.

Steps for debugging

  • Check Kyverno memory and CPU usage / limits.

  • Check Kyverno logs.

    • Check logs for client side throttling messages, such as:

      I0211 08:07:25.200825       1 request.go:668] Waited for 1.197259946s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/namespaces/test-project-request/rolebindings/namespace-owner

      If on Kyverno 1.6.0 or newer, increase -clientRateLimitQPS and -clientRateLimitBurst command line flags. The default QPS limit in the client is 5, 10 burst.

  • VSHN: Notify Tarazed