0

Your cart

TOTAL excl.
TOTAL incl.
Pay
Go back

The deployment that caused 92% of requests to fail — and the gate that could have prevented it

REXAutomatisationCI/CDMonitoring

2026-12-10 | 11:40 AM - 12:00 PM

Information

On a saturated Kubernetes cluster, a simple rolling update caused 92% errors and 13 seconds of p95 latency. This was measured, reproduced, and then blocked with three Prometheus queries in Jenkins. Encrypted feedback, public artifacts.

We had a "well-configured" Kubernetes cluster: HPA enabled, rolling updates with maxSurge=1, and readiness probes in place. And yet, under real-world load conditions, a single deployment was enough to cause the service to collapse, with 92% HTTP errors and 13 seconds of p95 latency.

The problem, in reality
When the HPA reaches maxReplicas, it can't do anything more. A rolling update then degrades the available capacity at the precise moment when the system has the least headroom. For Magento (45 seconds to pass the readiness probe), the vulnerability window lasts long enough for queues to become saturated and requests to time out en masse.

What surprised us: maxSurge=1, the setting everyone considers the safeguard against this, made no difference. 92.4% errors without maxSurge, 93.2% with it. At full capacity, the newly launched pod doesn't actually carry any traffic during its initial 45-second startup time—during which the previous pod is already starting to empty. Net effect on effective capacity: negative.

What we measured
  • Baseline scenario (saturated HPA, no deployment): < 1% errors — the cluster holds
  • Deployment under saturation, maxSurge=0: 92.4% errors, p95 at 13.05s
  • Deployment under saturation, maxSurge=1: 93.2% errors, p95 at 11.97s
  • Same cluster, same saturation, with Nginx (startup in 1.8s): 0.03% errors
The only parameter that changes between collapse and stability is the application startup time.

The gate: 3 requests, 1 decision
We added a step to the Jenkins pipeline before the kubectl rollout. It queries Prometheus on three metrics: aggregate CPU usage of the namespace, 5xx error rate on the ingress, and the ratio of current replicas to maxReplicas. If any of the three thresholds is exceeded, the deployment is blocked.

No additional dependencies: curl + jq, that's all.

Live validation on the same cluster:
  • CPU at 1.73 cores → gate blocked, kubectl not running, zero service interruptions
  • CPU at 0.09 cores → gate passed, deployment complete, zero errors
What you will leave with
  • The 3 PromQL queries to copy into your pipeline today
  • Why doesn't maxSurge protect you in this specific case?
  • How the HPA (current/max replicas) ratio is the most underutilized signal in monitoring

Speakers