Measure logical tasks, execution attempts, node health and financial events separately. Retrying a task must not inflate customer demand, and subsidized prices must not be presented as the full cost of delivering a workload.
Swipe or scroll to view all columns →
| Metric | Definition | Required context |
|---|---|---|
| Task acceptance rate | Passed tasks divided by admitted tasks closed in the measurement set | Include failed, expired, cancelled and disputed outcomes; list pre-admission rejection separately |
| Delivery latency | Time from submission to eligible delivery | p50 and p95, sample count, failures and incomplete tasks |
| Queue and execution time | Waiting and running intervals recorded separately | Clock source, skew and missing events |
| Inference performance | Time to first token and output token rate | Model, precision, engine, input and output lengths, concurrency |
| Observed node health | Successful probes divided by expected probes | Probe definition, interval, observation window and missing observations |
| Cost per task | Allocated service cost and customer charges reported separately | Retries, bandwidth, model fees, subsidies and currency |
Avoid biased reports#
METRIC-01 Report a fixed submission cohort alongside completed-task statistics, including still-running and pending-acceptance tasks. Results based only on completed work can hide slow or stalled tasks. Do not merge different models, hardware or acceptance profiles into an unexplained ranking.
Closed resolution is immutable; later disputes should be reported through separate correction or dispute metrics. Publish the time window, inclusion rules, denominator and exclusions for every rate. A task failure caused by a retry limit still belongs to the original logical task.
Benchmark record#
Include date, artifact digest, hardware, driver, runtime, model or dataset digest, warm-up policy, concurrency, run count, error categories and raw per-run records. Report unsuccessful runs alongside latency percentiles.
The reference bundle's benchmark-template.csv contains headers only. Its 9 tests and 80 cross-runtime comparisons are functional checks, not measurements of throughput, GPU capacity, cost savings or production availability.