Fleet
acme-ai — production fleet
4 clusters · 1,728 GPUs · $2.8M/mo
meridian-prod360training-east768inference-west384research216
This month
Recoverable spend
$151K/mo
idle allocated spend at a conservative 55% realization.
Recoverable capacity
164
GPUs back through defragmentation, power-cap fixes and lemon remediation.
Critical exposures
3
correlated-failure findings where one event stops business-critical work.
Loop collisions
2
windows in the last 24h where automation systems fought each other.
Awaiting approval
$146K
simulated savings queued for a human yes — nothing moves without it.
Optimization inbox
What to do Monday morning, ranked. Every action is priced, risk-scored, and reversible — approve it, or simulate it against the governance floor first. Expand a row for the reasoning.
Finding
Value
Risk
Confidence
Action
Reclaim idle H100 reservations
training-east
$86K/mo
Low
96%
Review
61 reserved H100s under 15% utilization for 3+ weeks; right-size the reservation and time-box the holders. Rollback: re-expand the node group.
Redistribute CDU-A workloads
meridian-prod
78% → 57% exposure
Critical
94%
Review
78% of business-critical GPUs share one cooling loop. Move 40 GPUs of standing training onto healthy capacity in other loops (+2.8 kW idle, 5 node moves).
Defragment training pool
training-east
+16 usable GPUs
Low
91%
Review
Free GPUs scattered in sub-gang fragments across 9 nodes; two migration batches recover two 8-GPU gang slots without hardware.
Consolidate inference pool
inference-west
$31K/mo
Medium
88%
Simulate
Off-peak consolidation of the serving fleet saves energy + reservations, but raises single-domain exposure 9 points — simulate against the blast-radius floor before approving.
Cordon 3 lemon nodes
meridian-prod
−54% job-death rate
Low
97%
Review
n23, n35, n29 carry repeat ECC/XID history and keep killing jobs. Cordon, burn-in, re-admit on a clean pass.
Cap rebalancer during the 02:00 window
meridian-prod
$9.4K/mo
Low
93%
Review
The cron rebalancer and cluster-autoscaler fight nightly (see conflict below). One shared migration budget ends the churn.
Control-loop intelligence
Which automation systems are fighting — and what the fight costs.
02:00–03:00meridian-prodConflict detected
$312/window
est. churn cost · ~$9,360/mo if nightly
Event timeline- 02:03Karpenterscaled node pool +2 (underutilization threshold)
- 02:07custom rebalancermigrated 11 workloads onto the new nodes
- 02:11HPAincreased replicas on chat-serving (queue depth)
- 02:13kube-scheduleropened another node for the new replicas
- 02:18custom rebalancerconsolidated the same node back down
- 02:24Karpenterterminated a node the rebalancer had just filled
- 02:31HPAscaled chat-serving back down (queue drained)
Conflicting policiesKarpenter consolidation (underutilized-node TTL 5m) × rebalancer cron (nightly repack)
HPA scale-up (queue depth) × rebalancer consolidation (same pods)
Governed fixPut both loops under one migration budget with a shared cooldown; the governance weight decides which optimization wins when they disagree.
13:00–14:00inference-westConflict detected
$178/window
est. churn cost · ~$5,340/mo if nightly
Event timeline- 13:04spot replacerrotated 4 spot nodes (price event)
- 13:09HPAscaled serving +6 replicas during rotation
- 13:16Cluster Autoscaleradded on-demand node for the surge
- 13:22spot replacerreclaimed capacity; replicas rescheduled again
Conflicting policiesspot replacement (price threshold) × HPA scale-up (latency SLO)
Governed fixGate spot rotation on serving-fleet headroom; price the rotation's churn against the spot discount before acting.
See this for your fleet
Read-only agent, one Helm command. No write access, ever — every action above ships as a reviewed change, not a cluster mutation.
or see a full exposure report