The over-provisioning problem
The client's cluster had grown organically for three years — every team requested CPU/memory headroom "generous enough to never page anyone," and nobody ever went back to right-size a deployment once it was running smoothly.
Where the waste was hiding
- Pod resource requests set 3-4x higher than actual observed usage, blocking the cluster autoscaler from ever scaling nodes down
- Several node pools left running 24/7 for batch jobs that only executed a few hours a week
- Persistent volumes from decommissioned services nobody had gone back to delete
- Deployments with no HorizontalPodAutoscaler at all, permanently sized for their own peak traffic
What we changed
We ran two weeks of resource-usage profiling before touching a single limit, then right-sized requests to the observed p95 usage plus a safety margin — not a guess. Idle node pools moved to scheduled scale-to-zero, and every service handling variable traffic got a proper HorizontalPodAutoscaler tuned to its own real load curve instead of a copy-pasted default.
Results
The changes shipped over four weeks with zero downtime, and the client's monthly AWS bill dropped by 35% within the first full billing cycle after rollout — with no reduction in the alerting thresholds that were protecting production reliability in the first place.