Two Kinds of Not Knowing

In May 2018 I sat through a talk at the Embedded Vision Summit in Santa Clara by Burkhard Huhnke, who had led Volkswagen’s side of the Stanford autonomous car programme that grew out of the DARPA challenges and had by then moved to Synopsys. The talk was called Making Cars That See: Failure Is Not an Option, and its most memorable line was that he was surprised we were not all being driven by our cars already. I had lunch with him afterwards, and what impressed me was his candour. As the cars had moved from a track across the desert to city streets, he said, the problem had become more complex in a way the demonstrations did not show, and each small gain in accuracy had cost almost double what the last one had. In aircraft that curve is acceptable. In a car it is not, because the industry lives on cost, and that difference, in his telling, was a large part of why the cars had not arrived.

I carried that lunch around for years without knowing what it was for. For the better part of a decade I had been looking at how AI might help my avionics company build better products, give our customers new capabilities, and make the business itself more efficient. The environment we worked in, high-reliability systems in which the cost of failure can be measured in lives, meant that my colleague Ken Wenger and I came at the technology from an unusual direction: precision, predictability and the ability to bound a system’s behaviour came before everything else, including raw capability. When we tested vision systems and tried to put generative models to work, we got 80 per cent of the engineering problem solved almost immediately, and then, as we pressed for higher precision, progress slowed, and after six months it stopped. Huhnke’s curve was sitting in front of us. My experience with the requirements and his lunch-table arithmetic made me realise we were looking at something more fundamental than a need for better engineering, something in the nature of the problem itself, and it called for a root cause analysis. Cognitive Information Theory is the result of that analysis, carried out not on AI but on information theory itself.

Like most people in the field, I had assumed the unreliability was a temporary condition, an engineering problem, and that more data and larger models would close the gap. I was wrong about that, and the way in which I was wrong is the subject of this essay. What began as an attempt to explain why intelligent systems fail in the particular ways they do grew into something I did not expect: a way of looking at uncertainty that seems to say as much about minds in general as about the machines that prompted it. The mathematics stays out of sight and the ideas in plain view. The central idea is not difficult. It is only unfamiliar.

Let me give that failure a face. Any company that builds certified software has to produce a great quantity of supporting artefacts, the documents and analyses that prove, or at least demonstrate, that the software does exactly what it was intended to do, and nothing else. It is expensive, slow, and intensely manual work, and on the face of it precisely the sort of thing generative AI ought to be able to take off our hands. It cannot. I know, because I have tried, and I have told that story in The Sixty Per Cent. Generative AI is some help, but it cannot do the task, and the unpredictability built into the process means the gains are eaten up by the extra human oversight the AI itself makes necessary. What matters here is not the conclusion but the shape of the road to it, a road that hundreds, perhaps thousands, of people building agentic AI are travelling right now. The closer you try to get to real precision, the more erratic the results become, and the more every change, to a prompt, to a dataset, to a layer of governance, carries consequences you did not intend. You are forever “almost there”. The problem is not the prompt, and not the data; nothing, in that sense, is broken. The system is doing exactly what its design requires, and that is the whole of the problem.

What Shannon Left Out

In 1948 Claude Shannon published the paper that founded information theory, and in doing so gave the twentieth century one of its most useful tools. Shannon’s entropy measures how much a signal varies, how much surprise it carries, how many bits you need to pin it down. It is the reason your telephone calls connect and your files compress and your messages cross the world intact. It is one of the great ideas.

Shannon, though, was solving a specific problem, the problem of moving a message faithfully from one place to another, and to solve it cleanly he deliberately set aside a question that did not matter for his purposes. He set aside meaning. His theory measures how much a signal varies. It says nothing about why it varies. For a communication channel that omission is not a flaw; it is the whole point. The telephone wire does not need to understand your words to carry them. It only needs to reproduce them.

The trouble begins when you take the same mathematics and apply it to a system that is not carrying a signal but acting on one. A doctor reading a scan, a pilot reading an instrument, a machine reading an image: these are not channels passing a message along. They are cognitive processes that must commit to a decision. For anything that has to commit, the question Shannon set aside, the question of why a signal varies, turns out to be the most important question there is. Cognitive Information Theory is, at bottom, an attempt to put that question back.

Philosophy’s Physics

Something stranger is going on here than a gap in an old theory, and it changes what kind of project this is.

For the whole of human history we have studied thinking from the inside. The only instrument available was the thinking mind examining itself: introspection, reflection, the patient description of what it is like to reason. This is how philosophy approached cognition, and it did so with great power. Kant’s Critique of Pure Reason, published in 1781, is the supreme example, an attempt to map the very architecture of thought from within, to set out the categories and the forms the mind must bring to any experience at all. That method was not wrong. It was one of the great achievements of the human intellect. It was, however, the only method there was, and it had a particular shape. It studied cognition by studying how we think. It never had occasion to ask, from the outside, what information is, and what a process is obliged to do with it.

Artificial intelligence changed the instruments available. Psychology and neuroscience had already taken cognition outside the introspecting mind, with experiments and, later, with imaging; what they could not do was build a cognitive process, take it apart, watch it fail and rerun it. For the first time we can. To make sense of why the built ones fail, we have been pushed towards a question philosophy never had to ask: not how do we think, but what is information, considered from the standpoint of something that must act on it. Cognitive Information Theory belongs to that question. It approaches cognition not through introspection but through information, from the outside in rather than the inside out.

An older relationship captures the shift. Mathematics is unconstrained by the observed world. A mathematician can build any consistent structure, and the overwhelming majority of the structures mathematics contains describe nothing in particular. What ties a portion of that freedom to reality is physics: physics is the discipline that asks which of the mathematically possible worlds is the one we actually inhabit, and in doing so it holds mathematics against the observed. Philosophy, and the philosophy of mind above all, has had the same freedom and lacked the same constraint. It could construct any internally coherent account of thought, and there was nothing outside the account, nothing in the observed world, to decide between rival pictures. Cognitive Information Theory may do for philosophy what physics does for mathematics: supply the external constraint, the point of contact with something measurable, that tells you which of the possible accounts of cognition the world will actually bear.

Physics has not finished constraining mathematics, and this work has barely begun to constrain philosophy. The older discipline will go on doing what it has always done, and should, because a great deal of what philosophy asks is not the sort of question information can answer. The claim I am making is narrower than it sounds: not that there was never an outside to the philosophy of mind, which the laboratories have supplied for a century, but that on one question, what a mind must do with what it cannot know, there is now a way to be wrong that can be measured rather than argued.

Two Kinds of Not Knowing

Here is the idea that the whole framework rests on. When a signal is uncertain, that uncertainty can come from two completely different sources, and they require completely different responses.

The first kind is genuine randomness in the world. Toss a fair coin and the outcome is uncertain, but the uncertainty is not hiding anything. There is no fact about which way the coin will land that you are failing to perceive. The variation is in the world itself. Call this noise. The theory’s word for it is entropy, and the primer on this site uses that word; noise is the everyday one and will serve here. A blurry photograph, the hiss on an old recording, the scatter of a measurement taken with an imperfect instrument: in each case the uncertainty is real and irreducible, but it points to nothing beyond itself.

The second kind is altogether different. It arises when many distinct causes produce the same appearance. A shadow on a chest scan may be consistent with several conditions, some harmless and one not, all of which can present in exactly the same grey smudge. A fever is the body’s response to a hundred different infections. The written word “lead” is one thing on the page and two things in the mind, the metal and the act of going first, and only the surrounding sentence tells you which. In all these cases the signal is not noisy at all. It may be perfectly sharp. The uncertainty is not in how clearly you can see it but in how many different realities are compatible with what you saw. Call this ambiguity.

Noise and ambiguity feel the same from the inside. Both register as not knowing. They could not be more different in what they demand. When the uncertainty is noise, gathering more of the same kind of data helps; you average the hiss away. When the uncertainty is ambiguity, more of the same data does not help at all, because every additional look shows you the same smudge that was compatible with several causes to begin with. What you need then is a different kind of information entirely, or the wisdom to know you cannot yet decide.

Now the uncomfortable result, the one that took me a long time to accept. If a system represents its uncertainty as a single number, a confidence score, a probability, one figure standing for how sure it is, then that single number cannot tell the two kinds of not knowing apart. It is not that this is difficult. It is that it is impossible in principle. Any single quantity computed from the data will assign the same value to a situation that is merely noisy and a situation that is ambiguous in the deeper sense. The information that would distinguish them has been thrown away in the act of compressing everything down to one figure. We call this non-identifiability, and it is the root from which the failures grow. A system that carries its doubt as a single number is structurally blind to the difference between a world that is random and a world that is undecided. When it meets either, it does the only thing it can do. It commits, and because it has no way to represent the kind of doubt that should give it pause, it commits with confidence. That confident commitment, repeated across a whole task, is what “almost there” is made of.

None of this arrives in an empty field, and it would be dishonest to present it as though it did. Machine learning has distinguished for years between aleatoric uncertainty, the randomness in the data, and epistemic uncertainty, the model’s own ignorance, and Alex Kendall and Yarin Gal set the distinction out for deep networks in 2017. Out-of-distribution detection tries to notice when an input is unlike anything the model was trained on; conformal prediction attaches a guaranteed coverage to a set of answers rather than to a single one; abstaining classifiers and the learning-to-defer literature build systems that decline and hand the case to a person. The field did not fail to build an “I don’t know”; it built several. What the two kinds add is a claim about where the line falls and why it cannot be crossed from inside: that ambiguity is not epistemic uncertainty in a new costume, because it survives perfect prediction, and that no single number, however well calibrated, can carry it, which is a statement about the geometry of the problem rather than about the training of the model. Out-of-distribution detection deserves a word on its own, because it is the neighbour most easily mistaken for this. It catches one way that entropy can arrive, the input unlike anything the model has seen. The entropy that costs most sits inside the distribution or close to it, where a novelty detector sees nothing, and what the theory asks for is the signal in that case as well as the fact of it: not only that uncertainty is present but which kind, and what it means. The neighbours measure how sure the model should be. This is a claim about which kind of unsure it is.

Why a Bigger Model Cannot Help

The natural response, and it was my response for a long while, is to say: very well, but surely a more powerful system will learn to handle this. Give it enough examples and enough capacity and it will sort the noisy cases from the ambiguous ones and the brittleness will melt away as it always has.

The objection deserves its full due, because it is the most successful prediction in the recent history of the field. Again and again, people have pointed at something AI could not do and declared it a wall, and again and again the wall has fallen to more scale. The scaling story has earned its confidence. Much of what looks like uncertainty in a hard problem really is reducible, really is just complicated rather than fundamentally undecided, and at that, scale has been close to miraculous. I do not dispute any of it.

The two kinds of irreducible uncertainty I have described, though, are not features of the model. They are features of the world and of the problem. The randomness in a fair coin is not in the observer; no amount of study removes it. Ambiguity is stranger still: you can build a system that predicts the appearances perfectly, that is never once surprised by what it sees, and it will still have no way to recover which of several causes produced any given appearance, Perfect prediction does not dissolve ambiguity. The two can coexist completely. This is why the limit cannot be optimised away. Scale clears the reducible undergrowth, and clears it brilliantly, but underneath there is a floor, set by the structure of the problem, and below that floor there is nothing more to be had.

The evidence for this is no longer just theoretical. When we and others have gone looking, the predicted failures show up where the theory says they should, at stable rates that do not improve with scale, and a simple safeguard that watches for exactly the disagreement the single number hides can cut errors dramatically: in a study Ken and I published, a layer of that kind wrapped around an ordinary object detector lifted its accuracy on the cases it found hardest from about 67 per cent to about 78 per cent, and the experiment is set out in The Mathematical Principles of Information. More tellingly still, when models are tested on how long they can sustain coherent reasoning, raw size stops predicting much of anything, and that result is in the next section. Both point at something hopeful: if scale is not the lever, then architecture is, and architecture is something we can choose.

The Slow Leak

So far I have described how a system can be wrong in a single judgement. The more insidious failure happens over time.

Consider any process that proceeds in steps, where each step builds on the last: a chain of reasoning, an autonomous agent carrying out a long task, a drone navigating a complex and changing operational theatre over hours. At each step there is some small probability that the system slips out of alignment with what it was actually meant to do. On a single step this probability may be tiny, small enough to ignore. The problem is that the steps do not average; they multiply. To stay on task for the whole run, the system has to stay on task at every step in turn, and a long sequence of high probabilities multiplied together becomes a low one. Stretch the horizon far enough and the chance of reaching the end still aligned drifts towards zero. We call this distribution drift, and the unsettling part is that scale shrinks the per-step error but never drives it to zero in an open world, so a longer horizon always wins in the end. Worse, once such a system has drifted, it usually cannot tell that it has, and the error cascades through everything that follows.

This is why a system can score brilliantly on a test of single questions and then collapse when asked to do something that takes 200 consecutive decisions. The agentic era, the one we are now entering, in which AI does not answer one question but carries out long chains of interdependent actions, runs directly into this wall.

When we built a benchmark to measure it, testing how many steps of a fully deterministic task a model could sustain before its end-to-end accuracy fell apart, the results were clarifying. Across 29 current models from eight providers, the differences were not what scale would predict. One large frontier model, with hundreds of billions of parameters, collapsed after roughly six steps. A much smaller model, built specifically to reason in a structured way, sustained well over 100. The decisive factor was not size or provider but architecture: whether the model had been built to hold and check its own reasoning. Under the strictest scoring, no model stayed fully coherent. Every one of them, eventually, drifted. This is not a defect awaiting a patch. It is a structural property of any system that commits to actions while carrying its uncertainty as a single number, and the only remedy is to change the structure.

Where Being Right Is Not Optional

For a great deal of what AI does, none of this matters very much. If a system that recommends films is confidently wrong, you scroll past. The stakes are low and the horizon is short.

There is, though, a growing class of applications where being right is not a nicety, and it is exactly here that the theory earns its keep. A model reading a medical scan, where a confidently wrong answer on an ambiguous shadow is the difference between a missed diagnosis and a needless intervention. A system in avionics or industrial control, where a precision target is not an aspiration but a certification requirement and a failure can cost lives. An autonomous vehicle or a drone operating for hours in an environment that keeps changing, where the slow leak of drift has time to do its work. In all of these, the very things that make ordinary AI impressive on average, its fluency, its confidence, its willingness to answer, become liabilities.

What Cognitive Information Theory offers these domains is not a better model but a different way of deciding what to build, and the reframing is practical. Begin by stating how right you must be and over how long a horizon, and treat those as the primary facts of the problem rather than hoping scale will deliver them. Then ask whether any available system, given the irreducible uncertainty the task actually contains, can clear that bar at all; you can often answer this on paper, before committing a year of engineering to something that was never going to work. Budget for the irreducible: where genuine ambiguity exists, expect a ceiling below perfection and design around it instead of optimising into a wall. Where scale cannot reach, add a different kind of structure, a layer that holds the system’s intent, watches for drift, checks the answer against what was actually wanted, and, above all, retains the right to abstain. The single capacity these confident machines most conspicuously lack is the capacity to decline, to say “these two possibilities are indistinguishable to me; I will not guess.” A system that can say that is doing something its confident cousin cannot, and in the places where being right matters, that refusal is worth more than any amount of fluency.

Entropy as a Signal

It was in trying to specify that refusal that the work turned, for me, from a theory of machines into something closer to a theory of mind.

The whole tradition I had inherited treats uncertainty as failure. Noise to be filtered, doubt to be resolved, error to be optimised away. The implicit picture is that a perfect cognitive system would be one that had eliminated uncertainty altogether. Once you take seriously that some uncertainty is irreducible, that picture stops making sense. If uncertainty cannot be removed, then a system whose only relationship to it is to suppress it is not perfect; it is blind. The alternative is to treat uncertainty not as a defect but as a signal, a piece of information in its own right, telling you something about the situation you are in.

The signal is not one thing. The arrival of unexpected uncertainty can mean at least three different things, and a competent system has to tell them apart. It can mean danger: the situation has moved outside anything the system can handle, and the right response is restraint, to slow down, defer, or hand off rather than barrel ahead. It can mean correctable drift: the world has shifted a little from what was expected, but in a way the system can adjust to without tearing anything up. Or it can mean opportunity: a persistent, structured surprise that does not go away with small adjustments, the kind that signals there is something real here that the system’s current picture of the world simply does not contain, and the right response is to investigate and, perhaps, to rebuild. A mind that collapses these three into one, that treats every surprise as an alarm, or every surprise as a chance to adapt, is either paralysed or reckless. The skill is in the distinction. Once you see it stated that way, the governance of uncertainty starts to look less like a failure of cognition and a lot more like its substance.

What It Might Mean to Think

I used to imagine thinking as the steady accumulation of certainty. The educated mind, in this picture, is the one that has resolved its doubts, gathered its facts, and arrived at firm ground. Cognitive Information Theory suggests something closer to the reverse: that thinking is not the elimination of uncertainty but its governance, and that the mark of a cognitive system is not how little it doubts but how well it handles what it cannot know.

The evidence for this, oddly, begins in the cradle. Long before an infant has language, concepts, or anything we would call reasoning, it reacts to the unexpected. The careful longitudinal work of Jerome Kagan and others at Harvard showed that babies as young as four months differ reliably in how strongly they respond to novelty, to the violation of what they have come to expect. Sensitivity to not knowing comes first, before all the machinery we usually associate with thought. Development, on this reading, is not the silencing of that sensitivity but the slow acquisition of ways to govern it: learning which surprises to fear, which to adjust to, and which to lean into. The same deviation that overwhelms an infant can, in an adult, prompt restraint, or recalibration, or curiosity. What matures is not the absence of uncertainty but the sophistication of its government.

This brings us back to Kant, and to a use of him I have set out at length in What AI Reveals About Human Sentience, Free Will and Creativity: time and space as forms the mind brings to experience rather than things it learns from it, and time, read through this framework, as the registering of change that matters. A system to which nothing is ever surprising has, in a meaningful sense, no experience of time passing at all. It is a small example of the outside constraint at work: an old account of the mind, met by a new question about information, and found to survive the encounter changed but intact.

Run the same idea forward through evolution and a striking pattern appears. Every time living things got better at taming one kind of uncertainty, they generated a new kind. Early organisms faced the uncertainty of the physical world: temperature, chemistry, the movement of predators. Group living tamed some of that and produced a new uncertainty, the unpredictability of other agents, the social. Memory and planning tamed some of that and produced the uncertainty of the future, the temporal. Language and symbol tamed some of that and produced the uncertainty of meaning itself. Uncertainty is not eliminated as minds grow more capable; it migrates, from the world outside to the world the mind itself creates. At the far end of this process the mind becomes its own largest source of uncertainty, generating, through memory and imagination and the modelling of other minds modelling it, far more doubt than the environment ever supplied. A great deal of what we call inner life is the work of governing uncertainty we manufactured ourselves.

This way of looking turns out to illuminate corners of human experience that seem, at first, to have nothing to do with information theory. Creativity, on this account, is what happens when no amount of small adjustment will resolve a persistent uncertainty and the underlying picture has to be rebuilt rather than tuned, a structural revision rather than a parameter change. Anxiety looks like uncertainty about the future that will not resolve and cannot be set down. Art, with its rhythm and its narrative and its form, looks like a technology for governing uncertainty at the scale of a single person, imposing just enough structure on time and meaning to make them bearable; and the great philosophical and religious systems look like the same work carried out at the scale of a whole culture, stabilising expectations across millions of people and many lifetimes. These are ways of seeing rather than claims to have explained anything away, but the through-line is real, and it is the same line that runs through the confident machine: a cognitive system is defined by what it does with what it cannot know.

There is even a sober answer here to the hardest question of all, the question of sentience. Rather than asking whether a system has some metaphysical inner spark, the framework asks how deeply and how persistently it has to integrate uncertainty across all these domains, physical and social and temporal and symbolic, simply to hold itself together. On that view sentience is not a switch that flips but a matter of degree, of integration depth, and it makes no promises about subjective experience, about what any of it feels like from the inside. That question remains open.

A Theory That Can Be Wrong

What this framework does not do matters as much as what it does, because I distrust theories that explain everything, and a theory of cognition that claimed to be complete would be the first thing I would disbelieve.

Uncertainty does not explain the mind. The governance of uncertainty is a constraint that any mind, biological or artificial, operating in a world it cannot fully specify, must satisfy, and that this constraint shows up everywhere from a misfiring classifier to an anxious afternoon to the rise and fall of cognition over evolutionary time. That is a large claim, and it has the decency to be wrong in identifiable ways. If intelligence could be shown to grow with no corresponding pressure from uncertainty, if scale alone turned out to dissolve the irreducible cases after all, if the distinctions I have drawn collapsed under measurement, the framework would be in serious trouble, and it should be. Its strength lies not in its reach but in its willingness to be constrained. There is real work still to do on how you measure these things outside the clean world of information, on how much of the story this actually carries, and on the perennial danger, when a word like entropy is involved, of mistaking a metaphor for a mechanism.

I keep returning, all the same, to those artefacts, and to the years spent trying to make a machine produce them reliably. We had built our systems to be fluent and confident, and so the one thing they could not do was the thing the work most needed, which was to stop at the point where they could not actually tell, and say so. The capacity to say “I do not know, and I know which kind of not knowing this is” looked, for most of the history of this field, like a weakness to be engineered out. I have come to think it is the thing we were trying to build all along, and that we mistook it for a flaw because we had inherited a theory in which doubt was only ever failure. The machine that could tell me which kind of not knowing it was in is the one I would have trusted with the artefacts, and it is the one this work is trying to build.