Namespace Chargeback Cookbook for Kubernetes

Chargeback fails when the numbers are delayed, disputed, or easy to ignore. The fix is a lightweight, opinionated process that turns namespaces into invoices engineers actually believe.

Required ingredients

• Label contract: owner, team, env, service, and cost-center are mandatory on every namespace and workload.
• Metrics source: A mix of Prometheus usage (CPU/memory/storage) and cloud billing data for node, EBS, and data transfer costs.
• Allocator: A simple model that maps usage → cost based on price sheets and amortized commitments.
• Delivery channel: Weekly Slack message plus a monthly CSV sent to finance.

5-step recipe

1. Harden labels: Add a validating webhook that rejects namespaces and Deployments without the required labels. Include a default owner fallback to catch legacy workloads.
2. Normalize usage: Export p95 CPU/Memory requests and usage per namespace, plus PV size and egress. Normalize into vCPU hours, GB hours, and GB transferred.
3. Price application: Apply on-demand or reserved rates; add a shared-pool tax (e.g., 8–12%) for cluster overhead and control plane.
4. Allocate shared services: Spread ingress controllers, service mesh, and logging based on traffic or pod count so teams cannot externalize those costs.
5. Publish early: Ship a weekly preview report. Let teams dispute within 72 hours; otherwise the month closes with those numbers.

Example Slack summary

Team: Checkout
Period: Nov 17–23
Total: $4,120 (+8% WoW)
Top drivers: api-server (+$220 from higher HPA floor), payments-worker (+$180 from PV growth)
Waste: 24% (requests vs usage)
Action: Reduce payments-worker request to 500m CPU and 1Gi memory; PV cleanup scheduled for Nov 29.

Governance without bureaucracy

• Budgets at namespace level: Assign a monthly cap per env. If burn rate exceeds 1.2x plan, the platform team files a ticket automatically.
• Exceptions with expiry: Temporary request bumps expire in 14 days. When they expire, the cost allocation drops and the service must re-request.
• Product-aligned rollups: Roll costs up to product lines, not just teams, to match revenue reporting and highlight unit economics.

What good looks like

• Reports land within 24 hours of usage, not after month-end.
• Disputes trend down because teams see the same numbers in Grafana that finance sees in CSV.
• Engineering adopts cost SLOs (waste < 20%, no unlabeled workloads) because the pipeline enforces them.

Chargeback is not a billing exercise; it is a feedback loop. Keep it fast, opinionated, and visible where engineers work every day.***