Methodology

How every number in the report is computed

The Exposure Report is arithmetic over a snapshot of your estate — not a model's opinion. This page states exactly what the collector reads, what leaves your cluster (in v1: nothing), and the formula behind every metric, with thresholds named. If a number on the report cannot be traced to a definition on this page, that is a bug.

What the collector reads

Collector v1 is a single-file, stdlib-only Python script that shells out to kubectl — auditable in one sitting, no image required. It is read-only by construction: the only verbs in its ClusterRole are get and list.

Resources read

Nodes — capacity and allocatable (including nvidia.com/gpu), labels, and conditions.
Pods — resource requests, owner references, restart counts.
Workloads — Deployments, StatefulSets, DaemonSets, Jobs; PodDisruptionBudgets and priority classes feed the criticality heuristic (a label override always wins).
Events — control-loop churn attributed by actor: HPA, autoscaler, Karpenter, scheduler, evictions.
Metrics — metrics.k8s.io utilization when the API is present; otherwise requests-based estimates, labeled as such in the report.

What it does not do: no Secrets, no ConfigMap bodies, no pod/exec, no writes of any kind. GPU-less clusters run the same pipeline over CPU cores via --unit cpu, and the report labels itself accordingly.

The ClusterRole, verbatim

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
rules:
  - apiGroups: [""]
    resources: [nodes, pods, events]
    verbs: [get, list]
  - apiGroups: [apps]
    resources: [replicasets, deployments, statefulsets, daemonsets]
    verbs: [get, list]
  - apiGroups: [batch]
    resources: [jobs, cronjobs]
    verbs: [get, list]
  - apiGroups: [policy]
    resources: [poddisruptionbudgets]
    verbs: [get, list]
  - apiGroups: [metrics.k8s.io]
    resources: [nodes, pods]
    verbs: [get, list]
  # DCGM scrape (optional feature): discover the dcgm-exporter service and
  # read its /metrics through the API-server proxy. Still read-only —
  # services/proxy get cannot exec, attach, or write.
  - apiGroups: [""]
    resources: [services]
    verbs: [get, list]
  - apiGroups: [""]
    resources: [services/proxy]
    verbs: [get]
  - nonResourceURLs: ["/version"]
    verbs: [get]
# Deliberately absent: secrets, configmaps, pods/exec, pods/attach,
# and every write verb. The role cannot read a credential or touch
# a workload — by construction, not by policy promise.

Scoping note: services/proxy get technically permits GET against any service; the collector only calls exporters whose name matches dcgm (or the one you pass with --dcgm ns/svc:port) — and you can audit that in the file, or run --dcgm off to drop the need for this rule entirely. The Helm Job itself runs as non-root (uid 10001), read-only root filesystem, no privilege escalation, CPU/memory-limited, and self-deletes after an hour.

What leaves your cluster

In v1: nothing.

The collector writes a single estate.json file to local disk — topology, allocation, utilization, failure history, and a provenance block recording which signal sources were reachable and which were not. You render it yourself at /report/live: the file is parsed and scored entirely client-side, in your browser. No upload, no telemetry, no account. The provenance block's coverage caveats are printed verbatim on the report, so the report says out loud what the collector could not see.

Definitions and formulas

Every threshold is a named constant in the scoring code; nothing in the UI hard-codes a number. The formulas below are transcribed from that code, not paraphrased.

Lemon score

lemon_score = ecc_errors_90d × 1 + job_failures_90d × 8 + xid_events_90d × 4

flagged as a lemon when lemon_score ≥ 40

A composite of a node's 90-day failure history. Job deaths weigh 8× a correctable ECC event and driver XID events weigh 4×, because a job that actually died on the node is the strongest evidence that the next one will. Flagged nodes should be cordoned, burn-in tested, and re-admitted only on a clean pass.

Idle allocation

idle when gpus_allocated > 0 and utilization < 0.15

idle_spend = idle_allocated_gpus × $/GPU-hour × 720 h/month

Allocated is not utilized. GPUs held by a workload but measuring under 15% SM utilization are invisible in allocation dashboards and fully visible on the bill. 720 is hours per month.

Capacity ledger

raw_free    = installed − allocated
safe_usable = raw_free − fragmented − power_capped − on_lemon_nodes

per node, free GPUs are classified by strict priority:
  1. power-capped   (rack power headroom exhausted)
  2. on lemon node  (node is above the lemon threshold)
  3. fragmented     (node partially allocated — free GPUs
                     cannot form a full 8-GPU gang)
Each free GPU lands in exactly one bucket; the ledger always
reconciles: raw_free = fragmented + power_capped + on_lemon + safe.

The free-GPU count on a dashboard overstates what you can schedule. A power-capped rack cannot energize new work, a lemon node fails the jobs placed on it, and scattered single free GPUs cannot host a gang-scheduled job. The classification priority means a power-capped lemon counts as power-capped — no double counting, and the decomposition always sums back to raw free.

Progress at risk

lost_minutes = checkpoint_interval / 2 + restart_overhead

progress_at_risk_$ = Σ over training workloads standing in the domain:
  gpus_standing × (lost_minutes / 60) × $/GPU-hour

If a failure domain fails now, each training workload standing in it loses on average half a checkpoint interval of progress, plus a restart overhead (detect + requeue + warmup) — self-reported where you have annotated it, assumed where you have not, with the basis printed (see the next definition). That lost GPU-time is priced at your own $/GPU-hour — the one printed on the report masthead — so the dollar figure is your number, not ours. Applies only to training workloads that report a checkpoint interval; workloads without one are counted in the caveats and their loss is not priced — never guessed.

Checkpoint & restart cadence (self-reported beats assumed)

checkpoint_interval: pod annotation iverson.gpu/checkpoint-minutes
                     owner-annotated — self-reported, not observed

restart_overhead precedence (basis printed beside every $ figure):
  1. iverson.gpu/restart-minutes annotation   owner-annotated (self-reported)
  2. estate defaultRestartMinutes             your incident history
  3. 90 min                                   stated assumption

The restart constant is no longer fixed: annotate a pod with its measured figures and the report prices from your history instead of our assumption. Every dollar built on these inputs states which basis it used, and the 90-minute default survives only where nothing better was supplied. One honesty note the report itself repeats: annotations are self-reported — owner-annotated, not observed by the collector — and they drive shed order and priced dollars, which makes gaming them a real risk. Validate them against actual checkpoint I/O before acting on a shed or failover plan priced from them.

Curtailment drain latency

inference / batch:          1 min from pause order to power shed
training, known cadence:    expected  checkpoint_interval / 2
                            worst     checkpoint_interval (the FULL
                            interval — the run may have checkpointed a
                            moment before the order) — full power until
                            the checkpoint completes, either way
training, unknown cadence:  time-to-target reported as unknowable

time_to_target       = slowest expected drain in the plan
time_to_target_worst = slowest worst-case drain (orders in parallel)

A pause is not an instant shed. A training run keeps drawing full power until its next checkpoint completes, so a plan's time-to-target is its slowest drain — reported both ways, expected and worst-case. The worst case is the compliance clock: grid curtailment is a deadline, not an expectation, so promise the worst-case number and hope for the expected one. Where a run's cadence is unmeasured, the planner reports the time as unknowable rather than inventing one — a shed you cannot time is a shed you cannot promise.

Rated-feed demotion (graph vs. electrical redundancy)

a dual-fed domain that keeps ≥ 1 live feed is still DEMOTED
to lost when:
  load_kw(domain) > Σ ratedKw(surviving feeds)

load_kw = allocated units standing on the domain × wattsPerUnitKw
survivors with ANY unrated feed: labeled "electrically unverified"
demotions cascade through the closure like any other loss

Graph redundancy is not electrical redundancy. A feeds: [a, b] domain survives on paper as long as one feed is up — but if the surviving feeds' combined ratedKw cannot carry the domain's measured load, the closure demotes it to lost anyway. Proven on our lab cluster: with feeds rated 8 kW and 4 kW under a 6 kW load, losing the weak feed is survivable and losing the strong one is a real outage — one a pure graph model would have called "redundant". Where survivors carry no rating, the report labels them electrically unverified rather than silently promoting them to safe.

Thermal ride-through race (cooling failures)

ride_through   = domain.rideThroughMinutes
                 (operator-supplied, from CDU/chiller specs)
drain_expected = max over training in the blast: checkpoint / 2
drain_worst    = max over training in the blast: checkpoint (full)

verdict:
  checkpointable      when drain_worst    ≤ ride_through
  not-checkpointable  when drain_expected > ride_through
  tight               otherwise
  unknown             when ride-through or any cadence is unmeasured

A cooling failure with training in the blast is a race: can everything behind the dying loop checkpoint and drain before thermal ride-through runs out? The report prices both sides — ride-through minutes you supply from equipment specs against checkpoint-and-drain minutes, expected and worst-case — and states a verdict. Where either side is unmeasured, the verdict is unknown, never guessed.

Exposure and critical share

critical_share(domain) =
  critical_gpus_standing_in_domain / max(1, total_critical_gpus)

domains sorted by critical_gpus_standing, then gpus_standing

For each failure domain — cooling loop, PDU, network, NVLink, rack — the report counts the allocated GPUs standing in it and the subset belonging to business-critical workloads. Critical share is the fraction of all critical GPU capacity that a single domain event takes down together. Node-level redundancy does not cover a shared loop; this metric is where that shows.

Criticality — how `critical` is derived

critical when ANY of:
  pod label iverson.gpu/critical = "true"    explicit — always wins
  a PodDisruptionBudget selects the workload
  priority class contains system-cluster-critical

A heuristic, stated as one: PDB protection and cluster-critical priority are the two signals a team has already used to say "this must not go down", and the label is the explicit override for everything the heuristic misses in either direction. Review and re-label — every collected report carries this caveat verbatim, because a wrong criticality flag skews every critical-share figure downstream of it.

Control-loop collision

a loop is "active" in an hour when it takes ≥ 8 actions
a collision window is an hour with ≥ 2 active loops

Autoscalers, rebalancers, HPA, and spot replacers each optimize their own objective. When two of them act heavily on the same hour, they are usually undoing each other's work — churn with no owner. The report shows the 24-hour action histogram per loop and flags the collision windows.

Allocation entropy

for each domain kind (cooling, power, network, nvlink, rack):
  p_i  = allocated GPUs in domain i / total allocated in kind
  H    = −Σ p_i ln p_i
  H_norm = H / ln(max(2, domain_count))    ∈ [0, 1]

A 0-to-1 reading of how spread the allocation is across the domains of each kind. Low entropy plus a high top-domain share is concentration — the quiet precursor of the exposure findings above, visible before any single finding fires.

Facility-coverage and staleness stamps

facility_coverage =
  nodes standing in ≥ 1 facility domain (power, ups,
  transformer, cooling) / all nodes

snapshot_age = now − provenance.collectedAt (fallback: snapshotAt)
warn loudly when snapshot_age > 24 h

topology_file_age = now − provenance.topologyFileModifiedAt
  (stamped when a topology file was supplied)

Stamps every report carries so its own limits are visible. Coverage says how much of the estate the facility findings can actually see — below full coverage they degrade toward zone/host level, and the stamp is how you watch topology authoring progress during a pilot. Staleness says how old the snapshot is; consequence surfaces (curtailment planning especially) warn past 24 hours, because a shed plan computed from yesterday's estate is a plan for yesterday's estate. Re-collecting is one command. The topology file carries its own age stamp besides — a facility map ages on re-cabling, independent of how fresh the snapshot is, so the two clocks are reported separately.

Fleet reclaim realization

recoverable_spend/month =
  Σ over clusters: idle_allocated × $/GPU-hour × 720 × 0.55

Fleet-level recoverable spend deliberately does not claim 100% of idle spend. A fixed realization factor of 0.55 discounts for holders who legitimately need headroom, reclaim friction, and re-expansion — the conservative share we are willing to put in a headline. The per-finding numbers underneath remain undiscounted and itemized, so you can audit the haircut.

Confidence

What the confidence percentages currently mean — stated plainly.

The confidence readings on the sample fleet surface are ordinal badges — High / Med / Low — deliberately not percentages. They convey relative conviction between findings, not a calibrated probability, and the surface says so in its own fine print ("rank, not rate"). They stay ordinal until per-signal calibration ships — confidence derived from signal coverage (was the metrics API reachable, was DCGM present, was a facility map supplied) and from how directly a finding follows from raw counts versus estimates. Findings built on direct counts — a node's 90-day failure history, the capacity ledger — are structurally more certain than findings built on estimated utilization. Until the calibration work lands, treat a badge as a rank, not a rate.

Topology inputs

The cluster API knows nodes and zones. It does not know which rack shares a PDU or which racks share a cooling loop — that map has to come from you, and the report is honest when it is missing.

Node labels the collector understands

iverson.gpu/rack:     rack-1
iverson.gpu/pdu:      pdu-a
iverson.gpu/cooling:  cdu-loop-a

# standard zone labels are also read:
topology.kubernetes.io/zone
topology.kubernetes.io/region

Label your nodes once and every subsequent snapshot carries the physical domains automatically. If labeling nodes is not practical, supply the equivalent map as a file instead.

Or a topology.yaml, supplied at collect time

# topology.yaml — the facility map the cluster API cannot see.
# Supplied by the operator (usually from DCIM or the rack elevation doc).
nodes:
  gpu-node-01: {rack: rack-1, pdu: pdu-a, cooling: cdu-loop-a}
  gpu-node-02: {rack: rack-1, pdu: pdu-a, cooling: cdu-loop-a}
  gpu-node-05: {rack: rack-2, pdu: pdu-a, cooling: cdu-loop-b}
  gpu-node-06: {rack: rack-2, pdu: pdu-a, cooling: cdu-loop-b}
domains:   # optional (v2): facility parent chains
  pdu-a: {kind: power, parent: ups-a}
  ups-a: {kind: ups, parent: xfmr-t1, ratedKw: 40}
  xfmr-t1: {kind: transformer}
  cdu-loop-a: {kind: cooling, rideThroughMinutes: 20}
  # dual-corded (A/B) equipment declares redundant feeds — lost only
  # when ALL feeds are down:
  # pdu-b: {kind: power, feeds: [ups-a, ups-b]}
  # ratedKw (kW) rates a feed electrically: a dual-fed domain whose
  # surviving feeds' combined rating is under its measured load is
  # demoted to lost. rideThroughMinutes (cooling, from CDU/chiller
  # specs) times the checkpoint race on a cooling failure.
# Alternative: label nodes directly —
#   iverson.gpu/rack, iverson.gpu/pdu, iverson.gpu/cooling

Passed as --topology topology.yaml. The optional domains: section (v2) chains facility parents — PDU to UPS to transformer — so a shared upstream failure domain is scored as one domain, not three independent PDUs. feeds: [a, b] declares redundancy (dual-corded A/B power, N+1 cooling): a domain with a feeds list is lost only when all of its feeds are lost, while single-parent chains are scored as the worst-credible case and labeled as such on every closure number. Two optional ratings tighten the model further: ratedKw rates a feed electrically, letting the closure demote paper-redundancy that cannot carry the surviving load (see the rated-feed demotion definition above), and rideThroughMinutes on a cooling domain times the checkpoint race a cooling failure runs. A cycle in the chain or a feed with no domains: entry is a validation error, stated loudly — a broken chain would silently understate every blast-radius figure, so the validator refuses it instead. Without a facility map, physical-domain findings degrade to zone/host level — and the report says so, in the provenance caveats printed on the report itself.

Multi-cluster merge

Transformers and CDU loops serve buildings, not clusters. A per-cluster report understates facility risk whenever two clusters stand on the same feed — merging estates against one shared topology restores the true closure.

Multiple estate.json files merge against one shared topology map. Domain ids are the join key: the same domain id in two files is treated as the same physical domain — that is the whole point — while node, workload, and loop ids are namespaced per cluster, so name collisions cannot silently alias. The merged report then computes facility blast radius across everything standing on a shared feed, which is exactly the number the per-cluster reports each understate.

The merge refuses rather than fudges: one id declared as a power domain in one file and a ups in another is an error, as are conflicting upstream feeds for the same id and mixing a GPU estate with a CPU estate in one ledger. The same-cluster defense is part of the refusal: two estates sharing ≥ 80% of their node ids are rejected as the same cluster uploaded twice — which would double-count its load into every shared domain's blast radius — and overlap of ≥ 20% draws a written note instead of silence. Where files disagree on prices or the power assumption, the first file's figure applies and the merge says so in a note. The merged report is stamped as only as fresh as its stalest input — the staleness clock keys on the oldest snapshot, because warnings must reflect the weakest data in the room. And when no domain ids are shared at all, the merge tells you it is a side-by-side ledger, not a shared-facility analysis.

Simulation vs. measurement

Two kinds of numbers appear across this site. They are never mixed, and each is labeled at the point of use.

Paper results — trace-driven simulation

The sandbox and the published benchmark numbers come from a trace-driven simulator replaying a real production workload trace. The in-browser engine is pinned to the paper's reference implementation: 39 golden parity tests hold the TypeScript port to it within 1e-6 relative tolerance (integer job counts exact), part of a 160+-test suite across the product. These numbers demonstrate the mechanism; they are claims about the simulated benchmark, stated as paired comparisons, and they are labeled as simulation wherever they appear. Where the two headline violation figures appear together: 52% is the mean across the paper's ten Philly-trace windows; 76.5% is the single reproducible window /frontier replays live.

Report numbers — computed from your snapshot

Every number on an Exposure Report is computed from the snapshot the collector took of your estate, using exactly the formulas on this page, priced at your $/GPU-hour. No simulation output feeds a report figure, and no report figure is quoted as a benchmark result. When a report renders from sample data, it carries a visible sample badge; a real snapshot carries its provenance block instead.