Assessing the Suitability of a Problem for AI

Imagine a ten-by-ten grid. Scattered across it are ten squares of gold, 20 traps and 70 empty squares. You start at a known position and move one square at a time, up, down, left or right, and after each move you report two things: what happened, gold or trap or nothing or a bump against the edge, and where you now are. That is the whole game. It has no luck in it and no judgement. Given the board and the moves, there is exactly one correct report at every step, and a few lines of ordinary code will play it perfectly forever. A patient child could play it perfectly for an afternoon.

Ken Wenger and I built this game, which we call Treasure Hunt, not to be hard but to be impossible to get wrong for the wrong reasons, and in early 2026 we ran 29 of the leading language models from eight providers through it. The results were not what scale would predict. One frontier model with something like 670 billion parameters was losing track of the board after about six moves. A compact model built for structured reasoning was still playing correctly after more than 100, and the best sustained around 140. When we tightened the scoring so that both the report and the coordinates had to be right, no model in the cohort stayed fully coherent, and two of them could not manage two consecutive moves. These are systems that write fluent essays and pass professional examinations, failing a game a child can play, and failing it at wildly different points that have nothing to do with how big they are.

The game is a measuring instrument, and what it measures is the thing that decides whether artificial intelligence can do a particular job. This essay is the instrument made light enough to carry into a meeting. It is the most practical thing I have written about these systems, and its purpose is a single question asked before any code is written: what does this task demand, and can the machine in front of me meet it?

Difficulty Is Not One Thing

Anyone who has spent real time with these systems has had the following experience, and it is maddening. You give a model a task and it does it beautifully. You give it another that seems no harder and it fails, and no amount of rephrasing rescues it. A third, which looked hopeless, comes right the moment you word the request differently. There is no obvious pattern, and a great deal of what passes for skill with these tools is an accumulated, inarticulate sense of where the invisible line runs. People call the craft prompt engineering, and much of it is real, but at bottom it is map-making by trial and error, because nobody handed the practitioner a map.

The instinct everyone reaches for is a single scale of difficulty, and it fails immediately, because difficulty is not one thing. A task can be hard because it asks for near-perfect accuracy on something with a little natural slack. It can be hard because the problem is structurally intricate, with many parts and constraints. Or it can be hard because it must stay correct across a long chain of steps, where small errors compound. These are three different kinds of hard, they have three different cures, and collapsing them into one word is why prompting feels like guesswork. Pull them apart, treat them separately, and the line starts to become visible, and even, within limits, calculable.

A colleague once spent the better part of a week trying to get a model to extract figures from scanned invoices. On clean invoices it was flawless. Where the layout varied it slipped, and he did what everyone does: rewrote the prompt, again and again, adding instructions, examples, threats and pleas, each version working better on some invoices and worse on others. He was fighting three problems at once and treating them as one. Some invoices were structurally messier. Some demanded a precision the smudged scans could not support at all. The longer documents accumulated small errors down the page. No single prompt could fix all three, because they had nothing in common except that each showed up as the model being wrong, and being wrong is a symptom, not a diagnosis.

Three Dials

The framework has three parts, and the whole of its usefulness comes from insisting that they never bleed into one another.

The first dial is precision: how good the answer has to be. This is not how hard the task is and not how well the model performs. It is only the threshold the task imposes, the success rate below which the job is not done. Reading a novel for pleasure demands low precision, since the odd misread word costs nothing. Reading a drug dosage demands very high precision, since one misread digit can kill. Same act of reading, wildly different demand, and the demand is a property of the use rather than of the difficulty.

The second dial is complexity, and it carries everything about the structure of the problem itself: how many parts it has, how many rules and constraints must be satisfied at once, how much must be held in mind, how tangled the relationships are among the pieces, and how much irreducible ambiguity the data contains. Everything the model must understand, compute over or reconcile lives here and nowhere else.

The third dial is the time horizon: not what each step involves, only how many steps there are and how their outcomes must hold together. A task that is one shot, right or wrong in a single stroke, sits at one end. A task that requires 200 consecutive correct decisions, each depending on the last, sits at the other, and it inherits every one of the compounding troubles that long chains bring, because steps do not average, they multiply.

People constantly confuse a demanding standard with a hard problem, and they are not the same. Demanding that a coin come up heads 99 times in 100 is not a hard problem in any structural sense. It is a trivial problem with an impossible standard attached. Demanding the rough gist of a 1,000-page report is a substantial problem with a forgiving standard. The two dials move independently, and almost every muddled argument about whether a model is good enough turns out to be two people who have silently fixed different points on that grid and are talking past each other.

Three examples fix the dials in mind. Transcribing a doctor’s handwritten dose is structurally simple and a single line long, but the precision demand is brutal, because one error in 1,000 is a patient harmed every 1,000 prescriptions. That is a precision-bound task. Building a valid duty roster for a hospital, satisfying dozens of interlocking rules about qualifications, rest periods and cover all at once, has an ordinary precision demand and a short horizon, but the density of simultaneous constraints makes it complexity-bound. An autonomous agent asked to carry out a 50-step research-and-booking task without supervision has easy steps and an unremarkable per-step precision, but the length of the dependent chain makes it horizon-bound, and it will fail for compounding reasons however clever each step is. Treat all three as generic hardness and you will apply the wrong fix to at least two of them.

The Ceiling Nobody Can Raise

Before asking whether a model is good enough for a task, you have to ask a stranger question, and the framework forces it into the open. What is the best that any possible system could do on this task, model or mind or oracle? For some tasks the answer is a perfect score. Treasure Hunt is one, and so are arithmetic and checking a document against a schema. We call such tasks algorithmic, and their natural ceiling is perfection. A great many real tasks are not like this. Ask whether a faint shadow on a scan is a tumour, or which of two digits a smudged handwritten figure was meant to be, or what a customer will do next, and there is no perfect score to be had, because the evidence itself does not determine the answer. That is the irreducible ambiguity I have set out in the primer The Three Faces of Information, and it sets a ceiling below perfection that belongs to the task and the world, not to any machine.

The natural ceiling is the most clarifying idea in the whole framework, because it separates two failures that look identical and are opposites. If a task has a ceiling of 70 per cent and you are demanding 99, no model will ever satisfy you, and the fault is not in the model. You are asking for a certainty the world does not offer. If the ceiling is 100 and your model delivers 70, the shortfall is the model’s, and better engineering may close it. The same disappointing number means a wall built into reality in one case and a bridgeable gap in the other. Without the distinction you cannot tell whether to replace your model or abandon your specification. With it, the first question about any task becomes not whether the model is good enough but whether anyone is allowed to be, and that question has saved us from a great deal of wasted effort chasing accuracy that was never on offer.

There are three precisions, then, not one: what the task demands, what the world permits, and what the model manages. The last can never exceed the second, since no system reads what the evidence does not contain, but it can fall well short of it, and the space between what the world allows and what the model delivers is exactly the room that engineering has to work in. Most disputes about whether a system is good enough are a failure to hold these three apart.

Capacity Meets Demand

Put the three dials together and each task can be assigned a single quantity, the cognitive capacity it demands, and each model a matching quantity, the capacity it possesses. The task is feasible when the model’s capacity meets or exceeds the demand and infeasible when it does not. That much is bookkeeping. What makes it bite is how the dials combine. They do not add. They multiply. Raising the precision required, deepening the complexity or lengthening the horizon each drives the demand up as a power, so that their effects compound. A task with twice the complexity and twice the horizon is not four times as demanding but a good deal more, because each dial amplifies the others.

Treasure Hunt let us measure this rather than assert it. The base game is one point on the dials. Add a single rule, for instance that landing on two empty squares in a row must be reported with a different word, and the complexity moves up a notch. Tighten the scoring so that both the report and the coordinates must be right, and the precision moves up a notch. Lengthen the run, and the horizon moves. Across the cohort, the horizon a model could sustain shrank far more sharply when a rule was added than when the scoring was tightened, by something like an order of magnitude in sensitivity. Complexity, in other words, erodes a model’s endurance much faster than precision does, and both slopes behaved as properties of the architecture rather than of the particular game. A model trained with explicit support for structured reasoning did not merely start with a longer horizon. Its envelope was flatter, degrading gracefully as the dials turned, where a model of the same size without that training fell off a cliff.

The idea that does the practical work is the margin, the distance between what a model can supply and what a task demands. When the margin is wide, the task is easy in the only sense that matters: the model has capacity to spare, and small perturbations, an added rule, a longer input, a stricter standard, do not threaten it. When the margin is thin, the task is precarious. It works today and fails tomorrow when the input grows or the request tightens, and it behaves inconsistently in ways that feel random but are not. When the margin is negative, the task is out of reach, and it will fail however many times you try. Most of the frustration of working with these systems comes from operating near a thin or negative margin without knowing it, experiencing as caprice what is a system sitting at the edge of its capacity, tipping back and forth across the line as conditions wobble.

The margin also explains the most misleading phenomenon in the whole industry, the demonstration that works perfectly followed by the deployment that fails on what looks like the same task. A demonstration is chosen, consciously or not, to sit comfortably inside the model’s capacity: clean inputs, moderate length, a forgiving standard. The deployment meets the ragged world, where inputs are messier, chains longer and standards stricter, and every one of those pushes the demand up until the margin, wide in the demo, goes thin or negative in the field. The model did not get worse. The task got heavier, along dials the demonstration was keeping light.

An objection belongs here, before the framework is put to work: capacity is not the fixed scalar the bookkeeping implies. Give a model a scratchpad and its horizon lengthens; give it retrieval and the complexity it must hold shrinks; give it a calculator and a whole class of precision failures disappears. Scaffolding of that kind is now standard, and a model measured bare is not the model that ships. I accept the point, and it fits the framework rather than breaking it. Every tool either moves the task down the demand scale or lends the model capacity it did not have, and either way the arithmetic is the same: the margin is what matters, and the margin is a property of the pair, the model with its scaffolding against the task. What the objection does show is that the number has to be measured for the system as deployed, not for the model in the catalogue, and that a change of scaffolding is a change of capacity that must be measured again. Treasure Hunt was run bare on purpose, to see the architecture. The deployment number is a different number.

When Words Help and When They Do Not

Now the framework earns its keep, because it says what prompt engineering can and cannot do. Rewording a request never adds capacity to a model. The machine has whatever capacity it has, and no sentence enlarges it. What good prompting does is lower the demand. A clear prompt strips needless complexity out of a task, spelling out structure the model would otherwise have to infer, breaking a tangled request into cleaner parts, removing ambiguity you did not need to impose. It clarifies the precision required, so the model is not straining for a standard the task never demanded. It breaks a long horizon into checkpointed stages, so that the compounding of errors is interrupted before it runs away. Each of these moves the same task to a lower point on the demand scale, and if the task was sitting just outside the model’s capacity, the reduction can carry it inside. That is why prompt engineering is not snake oil.

Notice what none of those moves can do. They cannot lower the natural ceiling, because it belongs to the world. They cannot shorten a horizon that is irreducibly long. They cannot manufacture structure the model lacks the capacity to hold, however cleanly it is presented. When the demand exceeds the capacity for these deeper reasons, prompting is rearranging deckchairs. This gives the practitioner a rule worth more than any list of prompt tricks. Before you iterate, diagnose which dial is binding. If complexity is the problem, decompose the task. If the horizon is the problem, insert checkpoints. If the precision you need exceeds the precision the task can ever yield, stop, because you are not facing a prompting problem or even a model problem. You are facing a fact about the world.

The clearest case I have of a binding dial is my own. At CoreAVI, when I first put a language model to work writing low-level requirements from source code, the model wrote plausible requirements at once, and I spent months treating what followed as a complexity problem: better prompts, more examples, then fine tuning, then an agent around the model to hold it in bounds. Complexity was never the dial that bound. A C function is not structurally intricate. What bound was precision, because certification tolerates no wrong requirement and the model produced a different set each time it was asked, and horizon, because the requirement was one step in an eight-step chain and the other seven had to hold together around it. I was lowering the wrong dial, and the cost of the step fell by a quarter and stopped. The diagnosis, and what it took to move the two dials that mattered, is in The Sixty Per Cent.

Two Kinds of Task

There is a last distinction the dials make visible, and it is the one that decides most real deployments. A great deal of what people ask these systems to do has a generous target. Draft an email. Summarise this report. Suggest a name. For such tasks thousands or millions of different responses would all count as acceptable, the precision demand is low by construction, and a fluent generalist with a person editing lightly behind it is superb. This is the mode in which the machines have earned their reputation, and the reputation is deserved. The precision dial is set so low that the ceiling and the margin scarcely matter.

The other kind of task has one right answer, or a narrow band of them, and the consequences of leaving the band are real. Certification documents. Dosages. Accounts. The control of a vehicle. Here the precision dial is high, often the horizon is long, and the natural ceiling has to be known before anything else. What such tasks need is what I have called elsewhere objective AI, a system that answers for its own correctness within a stated boundary: it performs a task consistently within a predefined range of outcomes and can be shown to stay there. That is a different kind of object from the generalist, and the difference is not cleverness. It is that the task’s demand has been measured, the model’s capacity has been measured, and the margin is known to be wide.

The two kinds also explain a pattern that used to puzzle me about AI and human competence. Plot a machine’s competence against the entropy and ambiguity in a task and it falls off a cliff at the point where the learnable structure runs out, because a single-objective system sees only structure. Plot a person’s and the curve is lower at the left, people are worse at dense, well-structured, repetitive work than the machines now are, and far flatter to the right, because a person can notice that the evidence forks, hold two readings open, ask a question, and decline to answer. The old examples make sense in this light. Chess and defect inspection suit the machine because their ceilings are perfect and their structure is dense. Predicting a market does not, because its ceiling is low and the machine cannot tell that it is low. Writing a book is suited to the machine as a draft and unsuited as a finished thing, because the target is generous until the moment it is not, and the moment it is not is the moment nobody can specify in advance.

A Map Before You Build

The deepest value of this way of thinking is that it moves the moment of judgement from after deployment to before it. The usual way a team discovers that a model cannot do a job is to build the whole system around it, ship it, and watch it fail in the field, expensively and sometimes dangerously. The framework answers the question earlier, at the design table, by estimating what a task demands along the three dials and comparing it with what a candidate model can supply. It does not give a precise verdict. Measuring the demand of a messy real task is approximate work, and I would not trust a decimal place from it. The framework is an engineering model rather than a law of nature; how sharply each dial drives demand is estimated from experiments and carries real uncertainty. What it gives is the right shape of the question and a first estimate of the answer: whether you are comfortably inside a model’s capacity, dangerously near its margin, or asking for something no model can provide, and, when there is a shortfall, which dial is responsible and therefore what might be done. That is the difference between engineering and hope.

It also tells you where governance is the only remedy and where it is unnecessary. When a task sits comfortably inside a model’s capacity, you do not need a watchdog; the model is reliable on its own and a second system only adds cost. When a task runs near or beyond the margin, especially along the horizon dial where drift lives, no amount of scale makes the bare model trustworthy, and the governed architecture is not a luxury but the only thing standing between you and the failures. Use the dials to see where the walls are. Use governance to operate safely against the walls you cannot move.

The next time a model release posts a higher score on some benchmark, ask one question before anything else. Better along which dial, and does that dial bind my task? A model that has become better at holding structure will help if complexity was your constraint, and it will do nothing for you if your problem was a long horizon or an impossible precision demand. Progress is uneven across the dials, and the question whether this model is better, which is nearly meaningless, becomes the question whether it is better at the thing my task demands, which can be answered. That is what the grid game is for.

I watched the method work at the start of 2026, when Anthropic released a product called Cowork and people who had shrugged at earlier releases found it useful in their daily work. I count myself among the converts, and what struck me was why it worked, as far as I can infer from the outside. The orchestration built into the product was lowering the demand of every task along all three dials. Tasks were broken into small chunks, which shortened the horizon. The chunks were handed where possible to small conventional programs, which raised precision, because a program that assembles a document gets the formatting right every time. The model no longer wrestled with the innards of a word processor. It wrote the text and passed the rest to code that could not get it wrong, and with that burden gone the complexity of what remained fell too. The model underneath had not leapt forward. The builders had moved the task down the demand scale until it sat inside the machine’s margin. That is the whole of the method.