Sample exposure report

meridian-ai / prod

Snapshot 20 Aug 2026, 22:00 UTC · 360 installed GPUs · 48 nodes · 7 racks · dollar figures assume $2.10/GPU-hour

Sample estate — the real report runs read-only against your cluster

Critical findings

Single-event exposures that stop business-critical work.

Safely usable capacity

of 119 free GPUs are safely usable — the rest are fragmented, power-capped, or on flagged nodes.

Worst-domain concentration

78%

of critical GPU capacity stands on CDU loop A alone.

Idle spend

$86,184

per month on 57 allocated GPUs running under 15% utilization.

Findings

What we would tell you first, ordered by how much of your business stops when the underlying assumption breaks. Each finding carries the numbers behind it and the governed action that fixes it — with its cost stated, not hidden.

Critical

78% of business-critical GPU capacity stands on one cooling loop

144 critical GPUs (atlas-70b-ft, rec-train-batch) share CDU loop A. A single cooling event stops them together — redundancy at the node level does not cover a shared loop.

144 of 184 critical GPUs on CDU loop A; est. $685 of training progress lost per failure event (half checkpoint interval + 90 min restart overhead).

Governed action

Redistribute 40 GPUs of the standing training work onto healthy free capacity in other cooling domains.

critical share on worst loop: 78% → 57%idle power added: +2.8 kWnode moves: 5

Critical

One power domain (PDU 1) carries 78% of critical capacity

144 critical GPUs sit behind PDU 1. A breaker or shelf event is a full stop for atlas-70b-ft.

144 critical GPUs behind one PDU; $567 progress at risk per event.

Governed action

Split the run across power domains during the next scheduled restart; make power-domain spread a placement constraint.

single-PDU exposure: 144 → 72 GPUs

Warning

119 GPUs look free — 48 are safely usable

Free GPU count overstates usable capacity: fragmentation blocks 8-GPU gangs, a power-capped rack cannot energize, and lemon nodes fail the jobs placed on them.

raw free 119 = fragmented 15 + power-capped 32 + on lemon nodes 24 + safe 48.

Governed action

Defragment partial nodes, rebalance the rack power budget, and cordon lemon nodes — then re-admit queued gangs against real capacity.

schedulable 8-GPU gangs: 6 today → 7 after defrag

Warning

3 lemon nodes are quietly failing the jobs placed on them

n23, n35, n29 carry repeat failure history. Meta's production study: excluding nodes like these cut large-job failure rates from 14% to 4%.

n23: 41 ECC error events in 90d; 7 jobs died on this node in 90d; 12 driver XID events in 90d · n35: 28 ECC error events in 90d; 5 jobs died on this node in 90d; 6 driver XID events in 90d · n29: 19 ECC error events in 90d; 4 jobs died on this node in 90d; 5 driver XID events in 90d

Governed action

Cordon, burn-in test, and re-admit only after a clean pass; placement should price node failure history until then.

free GPUs behind lemons: 24

Warning

Two control loops fought between 02:00 and 03:00

cluster-autoscaler and custom rebalancer (cron) acted on the same workloads in the same hour — 59 placement actions, each undoing the other's work. No shared objective arbitrates them.

cluster-autoscaler: 21, custom-rebalancer: 38, spot-replacer: 0 actions in hour 2.

Governed action

Put both loops under one migration budget with a shared cooldown; a governance weight decides which optimization wins when they disagree.

actions in collision hour: 59 → ~12

Info

57 allocated GPUs run under 15% utilization

Allocated is not utilized: these GPUs are held by workloads but doing almost no work — invisible in allocation dashboards, fully visible on the bill.

57 GPUs × $2.10/GPU-h × 720 h ≈ $86200/month.

Governed action

Right-size or time-box the holders; reclaim into the safe-usable pool with risk-priced consolidation (not blind bin-packing).

idle spend: $86200/mo

Blast-radius exposure

Every shared failure domain your critical GPUs stand in — cooling loops, power shelves, network spines, NVLink domains. One event in an amber-heavy domain stops that many critical GPUs at once, no matter how redundant the nodes are.

CDU loop Acooling

144 critical GPUs · 78% of critical · $685 at risk

Spine switch 1network

144 critical GPUs · 78% of critical · $685 at risk

PDU 1power

144 critical GPUs · 78% of critical · $567 at risk

NVLink domain — rack 1nvlink

72 critical GPUs · 39% of critical · $284 at risk

NVLink domain — rack 2nvlink

72 critical GPUs · 39% of critical · $284 at risk

CDU loop Bcooling

40 critical GPUs · 22% of critical

Spine switch 2network

40 critical GPUs · 22% of critical

PDU 3power

24 critical GPUs · 13% of critical

critical GPUs standing in the domain (stop together if it fails)critical GPUs elsewhere (survive the event) — track = 184 critical GPUs total

Capacity ledger

Free is not usable. Of the GPUs your scheduler counts as available, some are scattered in fragments too small for 8-GPU gangs, some sit behind an exhausted rack power budget, and some are on nodes that keep killing jobs.

0360 installed GPUs

Allocated 241Safe usable 48Fragmented 15Power-capped 32On lemon nodes 24

Inside the allocated segment: 57 allocated GPUs run under 15% utilization.

Control-loop activity

Placement actions per hour over the last day, attributed to the loop that made them. When two loops are busy in the same hour, they are usually moving the same workloads against each other.

collision

00:0006:0012:0018:0024:00

cluster-autoscaler· node groupscustom rebalancer (cron)· pod placementspot replacer· spot nodespeak 59 actions/h · outlined = collision hour

Flagged nodes

Nodes whose 90-day failure history crosses the lemon threshold. Work placed here fails at a multiple of the fleet rate — cordon them until a burn-in pass comes back clean.

Node	Rack	GPU type	Free GPUs	Score	Why it is flagged
n23	Rack 3	H100	8	145	41 ECC error events in 90d · 7 jobs died on this node in 90d · 12 driver XID events in 90d
n35	Rack 5	A100	8	92	28 ECC error events in 90d · 5 jobs died on this node in 90d · 6 driver XID events in 90d
n29	Rack 4	H100	8	71	19 ECC error events in 90d · 4 jobs died on this node in 90d · 5 driver XID events in 90d

Run this against your estate

Read-only agent, one Helm command, first report within a day. No write access, ever.

or play with the governance dial