When a Benchmark Stops Measuring
Eye for AI · 4 September 2026
Every good test of machine ability eventually becomes a bad one.
Volume III · Number 6 · September 2026
Independent and reader funded. No advertising, no sponsored posts, no affiliate links. editor@eyeforai.blog
Contents/Notes/On the Word ‘Agent’
Note
A term that has meant four things in three years, and what is lost each time it is redefined.
Abstract
The word has meant four things in three years and is now load-bearing in procurement and draft regulation. This note traces the drift, identifies what the original sense was protecting — autonomy under uncertainty — and proposes replacing the adjective with a statement of the action space.
In 2023 an agent was a loop: a model, a scratchpad, and a list of tools, calling itself until a stopping condition. In 2024 it acquired a browser. By 2025 the word had been attached to any product that performed more than one step without asking permission, and by now it is applied to workflows that a decade ago would have been called a cron job with a natural-language front end.
This is ordinary linguistic drift and it would not matter, except that procurement documents and draft regulations are now written using the term. A definition that covers both a multi-step research tool and a scheduled report generator is not going to produce coherent rules.
The useful content of the original sense was about autonomy under uncertainty: the system decides what to do next, and the decision is not enumerable in advance by the person who deployed it. That property is what makes agents hard to test, hard to audit, and interesting.
Almost everything now marketed as an agent lacks that property, which is generally a good thing for reliability and a bad thing for the clarity of the word. When the label stops distinguishing the systems with unbounded action spaces from the ones without, we lose the ability to talk about the specific risk that motivated the term.
Say what the action space is. 'It can call these six functions against this account, with a spend cap and a human confirmation on writes' communicates more than any adjective, and it is the sentence a buyer, an auditor and a regulator all actually need. If a vendor cannot produce that sentence about their product, the word agent is not the thing to worry about.
Longer, on the same themesAll essays
Eye for AI · 4 September 2026
Every good test of machine ability eventually becomes a bad one.
Eye for AI · 21 August 2026
A demonstration proves that a system can succeed.
Eye for AI · 7 August 2026
The clearest historical evidence on automation and employment is not about looms or robots.