Reading a Capability Announcement

1The structure of the genre

Capability announcements follow a stable form. A headline claim with a superlative. Two or three benchmark comparisons against named competitors. A selection of demonstrations, usually video, usually edited. A safety section. A note about availability. The form is stable because it works, and it works because most readers stop after the headline.

Almost none of the information content is in the first four sections. The benchmarks are chosen after the results are known, which is not fraud but is selection. The demonstrations are best cases, and the number of attempts behind each one is never stated.

2The two sentences that matter

The first is the availability sentence. General availability today, at a stated price, with a stated rate limit, is a strong claim: the organisation is committing capacity and taking liability. A waitlist, a limited preview or a research demonstration is a much weaker one, and the gap between the two has historically run to many months and occasionally to never.

The second is the pricing sentence, when there is one. Price per unit tells you roughly what the thing costs to serve, subject to whatever the provider is willing to subsidise, and a capability priced at ten times its predecessor is telling you something about compute per request that no benchmark table will.

3What to discount entirely

Comparisons against a competitor's older release. Aggregate scores where the component tasks are not listed. Any percentage improvement stated without the baseline. Demonstrations of tasks with no cheap verifier, since those are the ones where failure is least visible. And safety sections that describe process rather than result — an evaluation was conducted is not a finding.

None of this implies bad faith on the part of the authors. It is what marketing documents are for. The error is reading them as if they were papers.