The Arithmetic of Inference

1Two very different cost structures

Training is a capital expenditure: a large, discrete, forecastable outlay that produces an asset. It is reported, compared, and used as a proxy for seriousness, and it is the number that appears in headlines because it is a single figure with a currency symbol in front of it.

Inference is an operating expenditure that scales with use. Every request costs something, and the cost recurs for as long as the product exists. For any system with meaningful adoption, cumulative inference spend passes training spend and keeps going, which means the economics of the industry are governed by a number that is rarely disclosed and hard to estimate from outside.

This distinction determines what gets built. A capability that is impressive but costs materially more per request than the value it produces does not ship, regardless of how good it is, and a great deal of published capability research sits permanently in that category.

2Where the money goes

Per-token cost decomposes into roughly four parts. Amortised hardware, which is the accelerator cost spread over its useful life and which depends heavily on utilisation. Energy, both for compute and for the cooling it requires. Memory bandwidth, which is frequently the binding constraint rather than raw arithmetic throughput. And the serving overhead: batching, routing, the capacity you hold idle to absorb peaks.

Utilisation is the underappreciated term. An accelerator earns nothing while idle, and demand for interactive services is extremely peaky, following the working day across time zones. Providers either overprovision for the peak and eat the idle cost, or underprovision and degrade at peak. Batch workloads that can fill the troughs are worth far more than their headline price suggests, and this is why batch pricing is so much cheaper than interactive pricing.

Memory bandwidth deserves its own note because it constrains architecture. Generating text is sequential and each step must read the model's parameters. That read is bounded by bandwidth rather than by arithmetic, which is why hardware improvements in raw compute have translated less directly into cost reductions than one might expect, and why so much recent engineering effort has gone into avoiding reads rather than doing arithmetic faster.

3The cost collapse is real

Cost per token for a given level of capability has fallen dramatically, by any reasonable accounting, and it has fallen faster than most people forecast. The drivers are well understood and none of them are mysterious: quantisation, better serving software, speculative decoding, sparse architectures that activate a fraction of parameters per token, and distillation of large models into small ones that retain most of the capability on the tasks that matter.

Some of these have further to run and some are approaching physical limits. Quantisation below current levels degrades quality in ways that are hard to detect with standard benchmarks and obvious to users, which is its own small illustration of the benchmark problem. Distillation depends on having a strong teacher, so it lowers the cost of following the frontier without lowering the cost of establishing it.

So the trend is real and I would expect it to continue for some years. What I would not expect is for it to continue to translate into falling total spend, and the reason is a two-hundred-year-old observation about coal.

4Jevons, and why cheaper does not mean less

When the efficiency of a resource's use improves, consumption of that resource often rises rather than falls, because the efficiency gain expands the set of economically viable uses. Jevons observed it for coal and steam engines in 1865, and the pattern has recurred in lighting, in transport, and now fairly clearly here.

The mechanism in this case is specific and worth naming. Falling per-token cost has made viable a class of techniques that consume enormous numbers of tokens per user-visible answer: extended reasoning traces, sampling many candidates and selecting among them, multi-step agent loops that call themselves dozens of times, retrieval pipelines that read far more than they quote. These were always possible. They were not affordable.

The result is that a request which cost one unit two years ago may cost a tenth of a unit per token today while consuming fifty times as many tokens. Total cost went up. Total energy went up. What improved was the answer, and whether that trade is worth it depends entirely on the application — which is a real answer, but it is not the answer implied by 'costs are collapsing'.

5Who this concentrates power around

Inference at scale requires capital, land, power connections and hardware supply, and all four are constrained. Grid connection queues in major markets now run to multiple years. Advanced packaging capacity is concentrated in a very small number of facilities. These are not software problems and they do not respond to clever engineering on the timescales that software does.

This produces a structural asymmetry that I think is underdiscussed relative to its importance. Model weights can in principle be released to anyone; the capacity to serve them at scale and low latency cannot. An organisation can hold state-of-the-art weights and still be unable to operate them for a large user base, which means openness at the weight layer does not deliver openness at the service layer.

The counterargument is that small models running on ordinary hardware are improving fast, and that a great deal of real work does not need frontier capability. I think this is correct and genuinely important — it is the strongest reason for optimism about decentralisation. It also does not touch the frontier, and the frontier is where the strategic questions are.

6What I would watch

Three indicators tell you more than any announcement. First, the ratio of tokens generated to tokens shown to a user: if it keeps climbing, efficiency gains are being spent on quality rather than banked as savings, and total consumption will keep rising. Second, capital expenditure guidance from the large providers, which is a costly and therefore relatively honest signal about expected demand.

Third, and most legible, electricity. Data centre load growth in the major markets is a physical measurement made by people with no stake in the narrative, and it is available in public filings. Whatever else is uncertain, the power draw is not a matter of opinion.

None of this tells you whether the spending will prove justified. That depends on whether the deployments described in the essay on demos and deployments actually stick, which remains, several years in, an open question.