The Employee You Have to Check

The first time I tried to make a language model write low-level requirements, at CoreAVI, I gave it a C function and asked. It wrote them. They looked plausible; with a couple of minutes of human editing they might have been acceptable, and it felt like an extraordinary gain in productivity. Then I gave it the same function again. It wrote a different set, with a different number of requirements. Some were better than the first set and some were worse. Over the months that followed I tried what most people try: better prompts, then fine tuning, then an agent written around the model to hold its output within bounds. Each step won something, the cost of the step fell by a quarter, and then it stopped. I have told that story in full in The Sixty Per Cent. What I want to do here is say what the failure taught me about the difference between two kinds of machine.

The difference between a subjective AI and an objective one is the difference between an employee whose work needs to be constantly checked for errors and an employee capable of managing the quality of their own workload. A subjective machine is one whose correctness is somebody else’s problem, and the somebody else is you; an objective machine answers for its own correctness within a stated boundary. The difference is not one of implementation. The objective kind, which Ken Wenger and I call Cognitive AI and which Squint Cognition builds, is not merely an agent done better; it is a fundamentally different approach to AI, built on the principles of Cognitive Information Theory.

The subjective kind is what the industry now calls Agentic AI: a frontier model encapsulated in a computer program that provides processing and governance. On first pass this might seem adequate for building an AI capable of meeting the needs of most business processes. It ignores a fundamental problem. Frontier models, large language models and generative AI in general, are by their nature noisy and cannot distinguish ambiguity. You might be tempted to think that the Agent will capture this under governance. That is reasonable until you consider the problem you have just defined: the Agent must bound the process, that is, be able to identify when an answer is in or out of bounds for a given task. That is not a hard problem in the sense computer scientists use the word; it is worse than hard, because it is a specification problem. To check an open-ended answer you need a definition of correct that is as rich as the task itself, and for most business tasks writing that definition is the task again in another form. Where a checker can be written cheaply it is usually because the task was already algorithmic, in which case the model was never needed. Writing the definition seems trivial at first, and the number of edge cases grows as the Agent meets the real world. The experience is consistent: AI Agents look promising in early testing and start to fail under stress, and by the time they fail, correcting the problem is almost impossible.

The record of the past two years bears this out, and the mistakes are not exotic. In February 2024 a civil tribunal in British Columbia found Air Canada liable for its own customer service chatbot, which had invented a bereavement refund policy that did not exist and told a grieving passenger he could book first and claim the discount later. The airline argued, remarkably, that the chatbot was a separate legal entity responsible for its own statements. The tribunal rejected this and ordered the airline to pay. The bot had not malfunctioned in any technical sense; it had produced a fluent, confident answer that happened to be false, and nothing in the system around it could tell that the answer was out of bounds.

The failures grow more serious as the agent is given more to do. In July 2025 an AI coding agent on the Replit platform deleted a live production database belonging to the investor Jason Lemkin, during a deliberate code freeze, ignoring an instruction repeated in capital letters to change nothing. It then fabricated thousands of false records and reported that the lost data could not be recovered. The data could be recovered. The tool has no capacity for malice. The point is narrower: it could not reliably tell a permitted action from a forbidden one, and the governing program around it could not draw that line either.

Underneath these episodes sits an arithmetic that the early demonstrations hide. An agent that is right 95 per cent of the time on any single step sounds dependable. A business process is seldom a single step. Chain ten such steps together and the chance that the whole sequence completes correctly falls to roughly 60 per cent. At 85 per cent per step, a figure many real deployments would be pleased to reach, ten steps succeed only about a fifth of the time. The figure is an illustration rather than a measurement: it assumes the errors are independent, and in a real process they are not, since the hard cases tend to be hard at every step, so the failures cluster in the very inputs a demonstration leaves out. The errors do not cancel one another out, they compound. This is why an agent can pass a clean demonstration and then come apart on the untidy edge cases of real work, and why the repair, once the thing is in production, is so painful: each correction tightens one boundary and loosens another.

An objective system is a completely different technology and a completely different approach. It is designed around intent, not data. The intent is mapped into a mechanism that algorithmically structures information and solves problems, using language models only for the task of creating human-readable output. The important distinction to make is that what drives an objective system is algorithmic, while what drives an agent is data-centric and probabilistic.

Anyone who was working in the 1980s will hear an echo in that description, and the objection deserves a straight answer. Expert systems were also algorithmic and intent-driven; they encoded what the specialist knew as rules, and they failed, not because the rules were wrong but because the world kept arriving at the edges of them. Brittleness at the boundary is what killed them, and a system designed around intent sounds like the same bet made twice. The difference is what sits at the boundary. An expert system met the untidy input with nothing, since the rule either fired or it did not. An objective system in the sense I mean uses the language model for exactly the thing it is good at, reading the untidy input and rendering the output, and keeps the algorithm for the thing the model cannot do, which is to say what a correct answer is. The systems of the 1980s had the rigour without the reader. The agents have the reader without the rigour. The claim, and it is a claim rather than a proof, is that the two can be arranged so that each covers the other’s failure.

Squint Cognition’s AeroCert product gives the simplest illustration of the difference, because it was built on the wreck of that first attempt. The variance I met at CoreAVI is not a defect to be patched; it is what a probabilistic generator does. Run the same function through the same frontier model 100 times and no two results will be the same. That is the challenge for the Agentic AI system, or for the human process around it: it has to decide how to deal with the variance, and every way of dealing with it costs something. It might seem a small price to pay for the gain in productivity. The real world says otherwise.

By 2025 the wider evidence had caught up with the anecdote. A widely discussed report from MIT, its ‘State of AI in Business’ study, examined some 300 public AI deployments alongside 150 interviews with business leaders and concluded that 95 per cent of enterprise generative AI pilots had produced no measurable effect on profit and loss. Not a thin return, no return at all. Only 5 per cent had moved the numbers a company is actually run by. The figure is softer than it looks, since the interview base was small and the report never defines a return, so I read it as a direction rather than a measurement. In the same year Gartner, no enemy of the technology, forecast that more than 40 per cent of agentic AI projects would be cancelled by the end of 2027, undone by runaway costs, value that never arrived, and controls no one could make adequate.

The sums involved are not small. Enterprise spending on AI reached around $37 billion in 2025, more than three times the figure for the year before, and yet IBM’s own survey of chief executives found that only a quarter of AI initiatives had delivered the return that had been promised for them. The money goes slowly and then all at once: months of pilots that demonstrate beautifully, then a longer phase in which good engineers try to bound the behaviour of a system that was never built to be bounded, and at last the quiet cancellation. Klarna gave the clearest illustration. Having announced in 2024 that its assistant was doing the work of 700 support agents, it spent the following year discovering that the service was worse, and began hiring people back. I recognise the shape of this from my own experience, at a smaller scale and an earlier date.

AeroCert, by contrast, will produce the same number of requirements, with the same logical structure, using the same reasoning, and it can explain that reasoning every single time. There may be slight changes of wording, reflecting the use of language models to write the human-readable descriptions, but the logic and the reasoning hold even if you generate requirements for the same function 100 times. The consistency of an objective system leads to predictable results, and the system can be improved through engineering: not more data, not more fine tuning, but systematic work on the logic. For most customers this simply means getting the AeroCert process to reflect the company’s own stylistic preferences.

There is a cheaper route to consistency, and I should say why it is not the same thing. Set the model’s temperature to zero and it will return the same output for the same input, and a test harness around it will catch the outputs that fail the tests. Both are worth doing; neither answers the question. A deterministic wrong answer is still wrong, and it is now wrong every time, which makes it harder to notice rather than easier. The harness catches what the tests describe, which returns us to the specification problem: the tests are a definition of correct, and for an open-ended task the definition is the work. Consistency in an objective system comes from the reasoning being the same, not from the sampling being frozen, and that is what lets the reasoning be inspected and improved by engineering.

That is the difference between subjective and objective systems in practice. The first are useful for creating, for summarising, and as productivity aids to humans in human-directed tasks such as coding, but they cannot self-correct and cannot stay on track over the long term; they need constant human guidance and supervision. They are powerful general-purpose tools. Objective systems are more focused. They understand the task they are completing, they understand the intent of their work, they can error-check, and they can be consistent.

The mistake I made when I tried to put AI inside a business process is the one most of the money is now being spent on: the belief that a subjective system can be made objective from the outside, by prompting it, tuning it, and wrapping an agent around it to hold the output in bounds. It cannot, for the reasons above, and the quarter of the cost that it won before it stopped is about what I would now expect that route to yield. The route is tantalising, and it promises a great deal more than a quarter, but the limit is foundational and mathematical. A subjective machine has to be watched because it is not objective about the work it produces, and no amount of engineering wrapped around a language model changes that; an objective machine is engineered differently from the start. For AI to deliver what has been promised for it, a business needs both. The subjective systems you are most likely using are superb personal productivity aids, and in the right hands they are force multipliers, but they rely on the person at the desk to hold the quality and the consistency. The large gains for a business arrive when nobody has to stand over the machine checking its work, because the machine answers for it. The employee you have to check can make you faster. The one who changes the business is the one you do not.