The Distance Between a Demo and a Deployment

1Two different questions

A demonstration answers an existence question: is there a path through this problem that this system can walk? To answer yes, you need one success. You may curate the input, retry the generation, choose the example, and stop the clock when it looks good. None of this is dishonest if the claim is 'this is possible', and possibility is a genuinely useful thing to establish.

A deployment answers a distributional question: across the full population of real inputs, arriving in the real order, under real time pressure, what is the shape of the failure distribution and who absorbs its tail? Answering that requires knowing the tail, and tails are expensive to characterise because by construction you see very few samples from them.

The gap between these two questions is not a matter of engineering polish. It is a category difference, and it explains why a system that produces astonishing demonstrations can be genuinely unusable in production, and why the people saying so are not being cynical.

2The parts nobody demonstrates

Watch what a demo omits. It omits the input that arrives malformed because a scanner in a regional office produces a PDF with the pages in reverse. It omits the case where the correct answer is that there is no answer and the request should be escalated to a person. It omits the customer who phrases the question in a second language, in a hurry, using an internal acronym that has meant two different things since a merger in 2019.

It also omits the surrounding apparatus. Every deployed system carries a shadow structure of monitoring, fallback, rate limiting, audit logging, escalation routing, versioning and rollback. In conventional software this apparatus is often larger than the feature it protects. In probabilistic systems it is larger still, because the failure modes are not exceptions that throw; they are plausible outputs that are wrong.

The most expensive omission is the human process. A deployed system changes who does what. Work that used to be done by one person in one step becomes review of machine output, which is a different skill, generally less pleasant, and frequently worse paid. Organisations discover this months in, after the procurement is signed.

3Silent failure is the specific problem

Traditional software fails loudly. A null pointer produces a stack trace; a failed request produces a status code. The failure announces itself, and the entire discipline of operational engineering is built on that announcement. Alerting, paging, dashboards, incident review: all of it assumes the system knows when it has broken.

A generative system does not know. It produces a fluent, well-formatted, confidently phrased answer that happens to be false, and it produces it through exactly the same code path as a true one. There is no exception to catch and no status code to alert on. Detection requires either a second system that checks the first, or a person who knows the answer already, which raises the obvious question of why the first system was needed.

This is why the useful deployments cluster in places with a cheap verifier. Code that compiles and passes tests. A database query whose results can be sanity-checked. A draft that a professional will read anyway. Where verification is expensive or impossible, the economics collapse, and no amount of capability improvement fixes it, because the constraint is on the checking side rather than the generating side.

4The pilot that never ends

A pattern has become common enough to have a shape. An organisation runs a pilot. The pilot succeeds by its own measures. It does not become a deployment, and it does not get cancelled either; it persists in a semi-permanent state of nearly-rolled-out for several quarters and then quietly stops being mentioned.

The usual explanation offered is organisational resistance, and there is some of that. But when I have looked closely, the more common cause is that the pilot measured throughput while the deployment decision depends on liability. The pilot showed that the system produced acceptable output eighty-something per cent of the time. Nobody was ever able to answer who is accountable for the rest, and in a regulated industry that question is not a formality.

The organisations that get past this are usually the ones that redesigned the job rather than inserting the system into an existing one. That is slow, expensive, politically difficult work with no vendor to sell it to you, which is precisely why it is underrepresented in case studies and overrepresented in the deployments that actually stuck.

5What a serious evaluation looks like

If you want to know whether something will work in your organisation, the demonstration is close to worthless and the benchmark is not much better. What predicts deployment outcomes, in my experience of reading post-mortems, is a shadow run: the system processes real traffic for several weeks, its outputs go nowhere, and a person scores a stratified sample including deliberately oversampled edge cases.

This is unglamorous and it takes a quarter. It also produces the only number that matters, which is not accuracy but cost of error: how often does it fail, how bad is a failure, how long before anyone notices, and what does the remediation cost? Multiply those out and you have a business case or you do not.

I would add one further test, which almost nobody runs. Take the system away for a week after six months and see what happens. If the surrounding staff have lost the skill and the fallback process has atrophied, you have not deployed a tool, you have acquired a dependency, and its price is whatever the vendor decides it is at renewal.

6Why the gap is not closing on its own

Capability improvements do narrow the gap somewhat: a stronger model fails less often on the easy tail. But the components of the gap that are organisational, legal, and verificational do not respond to model quality at all. You cannot improve your way out of a question about who signs off, and the fraction of deployment cost that sits in that category has been growing, not shrinking.

There is a reasonable counterargument, which is that this describes every general purpose technology in its first decade. Electrification took forty years to show up in factory productivity because factories had to be rebuilt around it, not merely fitted with motors. On that reading the current pilot graveyard is an ordinary early-adoption phenomenon and the eventual reorganisation is coming.

I find that argument persuasive about direction and useless about timing, which is the only thing anyone actually needs to know. It is entirely consistent both with a transformation arriving in three years and with one arriving in twenty. Anybody telling you which, with confidence, is telling you about their portfolio rather than about the world.