Why sizing goes wrong
Most teams set requests and limits once, at deploy time, by guessing. Six months later the cluster is either half empty (requests too high, autoscaler buys nodes you never use) or one spike away from OOMKills (limits too low for real traffic). Both failure modes come from the same root cause: sizing by intuition instead of measurement.
Step 1 — measure before you set anything
Run the workload for at least one full business cycle (a week usually covers daily and weekend patterns) and pull the real usage percentiles:
> kubectl top pod -n prod --containers
# better: query the metrics you already have
# p50, p95, p99 of container_memory_working_set_bytes
# p50, p95, p99 of rate(container_cpu_usage_seconds_total[5m])You are looking for two numbers per container: the steady state (p50) and the realistic peak (p99, not the all-time max — a single restart loop can poison a max).
Step 2 — the sizing rules that survive production
- Memory request = p99 working set + 10–15% headroom. Memory is not compressible; if a node runs out, something gets killed.
- Memory limit = memory request. Setting them equal gives Guaranteed-class behaviour for memory and makes OOM events a signal about the app, not the neighbours.
- CPU request = p95 usage. CPU is compressible; a throttled pod is slow, not dead, so you can be less conservative.
- CPU limit: usually don't set one. CPU limits cause throttling even when the node has idle cores. Unless you run untrusted multi-tenant workloads, let bursty containers burst.
Step 3 — understand what QoS class you just created
| QoS class | How you get it | Eviction priority |
|---|---|---|
| Guaranteed | requests = limits for every container | Evicted last |
| Burstable | requests < limits (or partial) | Middle |
| BestEffort | nothing set | Evicted first |
Databases, queues and anything stateful should be Guaranteed. Stateless web tiers are fine as Burstable — that's the whole point of having replicas.
The mistakes that keep paying my invoices
- Copy-pasted manifests. The sidecar that needed 512Mi in one service gets 512Mi in forty services. Audit with
kubectl get pods -o custom-columnsonce a quarter. - Limits from the worst day ever. One incident with a memory leak becomes a permanent 8Gi limit on a service that uses 600Mi.
- No VPA in recommendation mode. Vertical Pod Autoscaler can just tell you the right numbers without acting. Free measurement, zero risk.
Rightsizing is not a one-time task. Put a quarterly reminder on the calendar; workloads drift.
