Observatory Panel · Field Returns
Send us your benchmarks
Our first-party probe network is in development. Until it reports, the distribution panels in our dossiers carry seed placeholders — illustrative, and marked as such. If you run real traffic on a provider we cover, your measured numbers can sharpen the record.
The gap between a provider's headline figures and its behaviour under production load is precisely the territory this publication exists to chart. Nobody sees that gap more clearly than the engineers paying for it. This page is a standing invitation: send us the distributions you actually observe, and we will fold them into the public record — credited as what they are.
The Return
What to send
A useful field return names its conditions. Copy the template below into an email; every line matters, because a p90 without a concurrency figure is a number without a meaning.
Provider: (one of the 20 we cover, or one we should)
Model: (exact model id as served)
Region / route: (client region → provider region, if known)
Window: (dates and rough hours of measurement)
Method: (client library, streaming on/off, prompt shape,
tokens in / tokens out, concurrency)
Sample size: (number of requests)
TTFT ms: mean= / p50= / p90=
Throughput t/s: mean= / p50= / p90=
May we credit you? (name / handle / anonymous)[email protected] — subject line beginning “Field return:”. Raw CSVs or notebook exports are welcome alongside the summary; we would rather recompute your percentiles than trust them.
Our Undertaking
How a return is treated
- Credited as community data. Field returns are published as exactly that — never re-badged as Datum Index first-party measurement, and never blended silently into a score.
- Cross-checked before use. A single return is a data point, not a verdict. We look for corroboration across submitters and against the public record before a dossier moves.
- Marked for confidence. Community figures carry their own provenance note and confidence mark, under the same legend as everything else we publish.
- No provider hands on the scale. Returns that arrive from a provider about its own service are welcome — and are labelled as self-reported, which is a different thing from independent.
We do not publish submitter identities without consent, and we discard returns we cannot at least plausibly verify. The only reputation we protect is our own.