The FinOps Cost Incident Runbook for Kubernetes

When costs spike faster than your alerts, you need an incident response muscle—not a spreadsheet. Here is a lean runbook you can run in under 30 minutes.

1) Confirm the signal

• Validate the metric: Compare billing data with Prometheus usage to ensure the spike is real, not a delayed invoice.
• Scope the blast radius: Identify the top three namespaces or services contributing to the jump.
• Check deploy history: Correlate spend inflection with the last 5 deploys or HPA changes.

2) Stop the bleeding

• Throttle scale-out: Temporarily cap replicas or HPA max to stop runaway autoscaling.
• Pause expensive jobs: Suspend non-critical CronJobs or data exports.
• Swap to cheaper capacity: Shift bursty workloads to spot where interruption risk is acceptable.

3) Find the root cause

• Utilization regression: Requests jumped but usage stayed flat → mis-sized containers or removed limits.
• Traffic shock: Load or batch size increased; confirm with ingress and queue metrics.
• Storage or data transfer creep: PV expansion, cross-AZ traffic, or new egress paths to SaaS.
• Third-party sidecars: Logging or APM agents bumped their own resources after an update.

4) Fix and prevent

• Rightsize: Reset requests/limits to match p95 usage plus headroom; re-enable autoscaling slowly.
• Guardrails: Add admission policies for owner labels, request ceilings, and egress annotations.
• Budgets: Set namespace budget alerts tied to burn rate, not month-end totals.
• Postmortem: Keep it short—owner, trigger, dollar impact, and the permanent control you added.

5) Communication template

Incident: Spend spike in checkout namespace
Impact: +$1,200/day vs baseline; no user impact
Trigger: HPA max raised from 10 -> 80 after deploy abc123
Action: Capped at 20, rightsized worker to 300m/512Mi, added guardrail to block >50 replicas
Follow-up: Burn rate monitor in CI, PV size alerts, review in next platform sync

Maturity checkpoints

• Page engineering for cost spikes the same way you page for error rate.
• You can remediate within a deploy (not a finance cycle) because the fixes live in code.
• Every incident leaves behind a new guardrail that prevents the same class of spike.

Cost incidents will keep happening. The difference between chaos and control is a practiced runbook that ships fixes as code, not as a PDF.***