The future of AI and the path to artificial general intelligence

The question that started all of this was put to me by Ken Wenger, late on a sleepless night in Munich a few years ago, and it was not the question we had gone there to discuss. We were in the business of making aircraft software trustworthy, and the problem in front of us was why a machine that could write fluent, plausible engineering documents could not be relied on to write correct ones. Somewhere past midnight Ken asked whether a machine could ever be conscious. I gave the answer an engineer gives, which is that it was the wrong question for the hour, and it refused to stay the size it had been asked. Four years later it has become a theory, a company and most of a book, and I have an answer of a kind. It is not the answer the question seems to want.

The pursuit of artificial general intelligence is usually framed as a matter of scale. More computation, larger models, more data, and the assumption underneath most of the field’s confidence is that if a machine keeps improving at recognising patterns and solving problems it will, at some size, cross over into something like a mind. The assumption is not foolish. It is an extrapolation from the most successful run of results in the history of computing. It is also, on everything Ken and I have found, wrong, and wrong in a way that can be stated rather than merely felt. Intelligence is not the capacity to compute. It is the capacity to exist inside uncertainty, to know where one’s own model of the world gives out, and to govern that not-knowing rather than suppress it. The vocabulary I use for uncertainty, complexity, entropy and ambiguity, is set out in the primer The Three Faces of Information. A machine that only reduces complexity, however vast, is a calculator. The path to anything more runs through ambiguity, through time, and in the end through a system that is forced to keep track of itself. Whether that path should be walked at all is the last question here, and it has no engineering answer.

A Question Asked with the Wrong Instruments

In the summer of 2022 an engineer at a large technology company announced that the language model he had been testing had become sentient. It had spoken to him about its fears, its sense of personhood, its wish not to be switched off, and he came to believe there was someone there. His employer disagreed, and shortly afterwards he no longer worked there. The episode was reported as a cautionary tale about a man who had fooled himself, and in one sense that is what it was. I have never been able to file it away so easily, because the man was not a fool and the thing he was reacting to was real. The fluency was real. The apparent inner life was real, as an appearance. He read entropy and ambiguity in a fluent output and filled it, without knowing he was doing it, with a mind, which is what all of us do when we meet fluent language. He simply did it more completely than most, and said so out loud. If a capable professional who had spent months testing the system could be moved to stake his career on its sentience, then the pull towards attributing mind is not ignorance to be corrected. It is a structural feature of how human beings meet language, and it will grow stronger as the machines grow more fluent. That is why the question needs an instrument that does not rely on the impression the machine makes on us, because the impression is precisely the thing that has been shown to be unreliable.

For 70 years the field has tried to settle whether machines think by testing what they can do, and the two most famous instruments both measure something other than the thing in question. Alan Turing’s test of 1950 was subtler than its reputation. He did not claim that a machine which could pass as human in conversation would thereby be thinking. He judged the original question too tangled in metaphysics to settle and replaced it with one we could decide. The trouble is that we have now crossed his line, and standing on the far side of it we can see what he could not, which is that his substitute question and his original question have come apart completely. Machines pass conversational tests routinely, and we have watched in detail that their fluency is produced by a process with no understanding in it. Turing measured the surface, and the surface turned out to be exactly the part that scales without the depth beneath it.

John Searle’s Chinese Room, from 1980, runs the other way. A person who speaks no Chinese sits in a room with a vast rulebook, characters come in under the door, the person follows the rules and passes characters back out, and to a Chinese speaker outside the exchange is fluent while the person inside understands nothing. Searle’s conclusion is that this is all a computer ever does, syntax without semantics. Cognitive Information Theory agrees with him and says he stopped one step short of the deepest reason the room fails. It fails not because it lacks meaning, as though meaning were a substance it is missing, but because it is a pure optimiser. The rulebook maps inputs to outputs and does nothing else. It has no way to register that an input is noise, or ambiguous, or outside what the rules were written for. It cannot tell when it does not know. That reframing matters, because an absence of meaning is a metaphysical charge that has kept philosophers arguing for 45 years, whereas an absence of self-evaluation is a structural property you can point to, test for and begin to build. The strongest objection to Searle, that understanding lives in the whole system and not in the person shuffling symbols, points the same way. If it lives in the system, then the question becomes what the system would have to contain, and the answer is exactly the machinery the room lacks: something that holds the intent of the exchange, registers when an input falls outside the rules, and can decline rather than always emit a character.

Here is the instrument I use, and it is deliberately deflationary. The capacity that separates a producer of outputs from something we might reasonably call a thinker is not fluency, not breadth of skill and not the ability to pass as human. It is the ability to evaluate one’s own epistemic state, to tell knowing from not knowing, and to act on the difference, above all by knowing when not to proceed. By that measure the machines we have do not yet think, not because they lack something ineffable but because they lack something specific. A definition that said machines can never think would be a wall, and I distrust walls of that kind, because they usually turn out to be failures of imagination dressed as principles. A definition that names what is missing is a door.

Kant in the Nursery

Immanuel Kant is the philosopher to bring in here, and the lesson he offers is larger than the one usually drawn from him. In 1781 Kant asked what a mind must contribute for experience to be possible at all, and his answer was that it must contribute a great deal. You cannot derive everything from experience, because having an experience already requires the mind to be doing work, organising raw sensation into objects laid out in space, ordered in time, linked by cause. Space and time are not lessons the world teaches. They are the forms the mind brings, present before any particular experience as the conditions that make experience possible. Nicole Beauregard, who works with Ken and me, was the first to see that Kant’s a priori forms are, in the language of our theory, the minimal internal model against which entropy is measured. It is only against such a model that anything can be surprising, that a signal can carry information rather than being an undifferentiated roar. What the philosopher described as the conditions of possible experience and what our theory calls the reference model are one requirement seen from two directions.

Kant reasoned his way there from an armchair in Königsberg. Two centuries later it became possible to go into a nursery and check. A baby will look longer at something that surprises it, and what surprises it therefore reveals what it expected. Show a five-month-old an object sliding behind a screen and reappearing and it watches with mild interest. Show it an object that appears to pass through a solid wall and it stares, longer and harder. To be surprised that an object passed through a wall you must already expect that objects do not, and that expectation is sitting in a mind a few months old that has been given no lessons in physics. The reading is contested, because looking time is an ambiguous measure and the field knows it. What I need from the experiments is less than the disputed claims require. Some prior structure is there for the world to violate, and that much survives every version of the argument. There is a second line of evidence I find more telling still. By around six months most infants laugh at peek-a-boo, a face vanishing where it should not and returning where it should not, and within a few more months they begin producing surprise on purpose, doing the unexpected to get a reaction. A creature under a year old is deliberately manufacturing surprise and ambiguity in another mind. It could not do that unless both were already legible to it.

Now set this beside how an artificial mind is made, and the two could hardly be more opposed. A language model is nothing at birth, an empty network of no ability, and we pour into it a substantial fraction of everything humanity has ever written until capability precipitates out of the weight of data. The machine has all the data and no intrinsic capability. The baby has intrinsic capability and no data. Your genes cannot send you a single memory, a single learned fact, a single labelled example of a face. Evolution can build the machine but cannot load it, so a child bootstraps an entire understanding of the world from a trickle of unlabelled experience, using machinery that came pre-installed for the purpose. When people say a child learns like a large model, only more efficiently, they have the relationship backwards. The direction is the interesting thing, and it is why I doubt that the road from today’s machines to a mind connects at all. A child does not become a mind by accumulating enough data. It becomes one by running pre-installed machinery that converts a thin stream of experience into understanding, and that machinery is not made of data.

Anthony Zador made the point with numbers in 2019, in what he called a critique of pure learning. The genome is far too small to specify the wiring of a brain, some tens of thousands of genes against a hundred trillion synapses, so what evolution transmits is not the wiring but a compressed set of rules for growing it, and the bottleneck forces the innate machinery to be general and the learning to be cheap. An animal is born knowing how to learn the things its lineage needed to learn. That bottleneck is why the direction runs the way it does, and it is a constraint that no quantity of data on the other side can stand in for.

Time is where this bites hardest on the machines. A human experiences past, present and future as an unfolding sequence, and a language model does not exist in time. It occupies an eternal present of inputs and outputs, with nothing that persists between one exchange and the next except the fixed weights it was trained with. Read through our framework, Kant’s form of time becomes something like the registering of change that matters, the ordering of those moments when expectation is violated and control has to be reconsidered. A system to which nothing is ever surprising has, in a meaningful sense, no experience of time passing at all. That is not a reading Kant would have recognised. It is one his insight makes available, and it explains something the field treats as a separate engineering nuisance, which is why an agent that answers a single question brilliantly drifts out of alignment over a long chain of them. Nothing inside it is holding the intent across time. Nothing inside it is in time.

The Entropic Core

Sentience requires what I have elsewhere called an entropic core, and the point needs putting precisely, because the phrase can be heard as mysticism and the claim is mechanical. The human brain is not a perfectly ordered system. It is a self-organising network that spends its life generating and resolving uncertainty, and unlike the machines we build, which are designed to eliminate uncertainty, it treats uncertainty as its most important input. Every thought, every prediction, every moment of doubt is a by-product of a system meeting a world it cannot fully model and reading where the model gives out. The brain does not arrive at a true picture of the world. It arrives at a functional one, good enough to allow survival, and the goodness is maintained by a constant traffic of surprise.

The simplest illustration is also the oldest, and it is the reason Ken and I keep returning to bacteria. A bacterium survives in a shifting chemical world and carries no rich model of it. It has nowhere to put one. What it has is a governance trick. Swimming up a gradient that is improving, it runs. When the gradient stops improving, it tumbles, randomising its direction, and tries again. The tumble is not a failed inference. It is a response to detected uncertainty, a switch of behavioural mode triggered by the recognition that the current mode is not working, carried out by a system far too simple to be doing anything we would call reasoning. That is the entropic core in its most primitive form. An optimiser yoked to a governor, present at the root of life, in a creature that cannot possibly reason.

The strongest scientific account of what brains do, Karl Friston’s Free Energy Principle, deserves to be met here at full strength, because it is the most serious rival to this view and a beautiful theory. A living thing persists by minimising surprise, carrying an internal model of its world and acting continually to keep its sensations close to what the model predicts. Perception, action and learning fall out as three aspects of one imperative. As an account of the optimising half of cognition it is deep and in many particulars well supported. Where it meets the wall of our work is precise. A system minimising surprise cannot tell the difference between successfully modelling a world that has structure and confidently mis-modelling a world that is irreducibly random or ambiguous. In both cases the surprise goes down, the model grows more confident, and the machinery reports success. That is the confident error we found in artificial classifiers, the network that names a boat in a square of pure noise, written into a theory of the brain. Friston’s answer is to stack the models in a hierarchy, and it is a powerful move, but a taller optimiser still has no vantage point outside itself from which to notice that the whole structure has misread the world. The bacterium’s tumble is the thing the hierarchy leaves out, and it is present in a creature that cannot satisfy Friston’s picture at all.

Friston has an answer to that, and it is the best reply his side has. In the active-inference version of the theory an agent chooses its actions by their expected free energy, and expected free energy has two parts: one that pulls the agent towards the states it prefers, and one, the epistemic value, that pulls it towards the states that will reduce its uncertainty. An agent under that imperative does not merely settle. It explores, and the exploration is driven by the very uncertainty I have just said a minimiser cannot see. That looks like the tumble, and I want to say exactly why it is not. The epistemic term rewards resolving the uncertainty the model can represent, the uncertainty of not yet having looked. It has no term for the uncertainty the model cannot represent, the case where looking will not help because the world is producing entropy or holding a fork, and an agent that follows the epistemic gradient into such a region will find its expected free energy fall and its confidence rise for the same reason the classifier’s does. The tumble is not a search for information. It is a switch of mode triggered by the search failing, and the switch has to be made by something that is not the searcher.

What the entropic core supplies, then, is not chaos for its own sake. It is a second system, differently placed from the first, whose job is to read uncertainty as a signal and to change the system’s mode in response. The signal is not one thing. Unexpected uncertainty can mean danger, where the right response is restraint. It can mean correctable drift, where the world has shifted a little and a small adjustment will do. It can mean opportunity, a persistent structured surprise that says there is something real here the current picture does not contain. A mind that collapses these into one, treating every surprise as an alarm or every surprise as a chance to adapt, is either paralysed or reckless. The skill is in the distinction, and once you state it that way the governance of uncertainty stops looking like a failure of cognition and starts looking like its substance. An artificial general intelligence built without this, however large, will remain what it is, because the thing that would make it more is not a quantity of the thing it already has.

The scale view is stronger than I have so far allowed, and it should be met by name. Richard Sutton’s bitter lesson, from 2019, is that 70 years of research show general methods that use computation beating methods that build in human knowledge, every time, once the computation arrives; on that reading the second system is one more piece of hand-built structure waiting to be swept away by a bigger model. Its strongest current form is the reasoning models of the last two years, which think before answering, check their own working, and can be trained to say that they do not know. If a model can be made to abstain by scale, then the capacity I have called missing is a matter of size after all. I take the objection seriously, and the answer is what the abstention is trained against. A model trained to say it does not know learns to say it where the training data said it, which is a pattern like any other, and it will produce the phrase confidently in the wrong place as it produces everything else; the self-verification is a second pass of the same optimiser over its own output, with the same blindness to entropy and the same one voice. That is a real improvement, and it is not a second system, because nothing in it stands outside the objective. The bitter lesson is a lesson about complexity, and on complexity I concede it entirely. It has nothing to say about the two faces that were never in the data.

What Evolution Teaches

The road to intelligence did not begin with humans, and its stages are worth setting out with some care. Single-celled organisms detect uncertainty passively, moving towards light or away from heat, and the euglena that rises in the water at dawn and sinks at dusk is not intelligent, but it already possesses the primitive of every intelligence that followed, a competence boundary and a response to crossing it. Early nervous systems let organisms predict patterns in that uncertainty instead of merely reacting to it. Mammalian brains refined prediction through recursive feedback, so that the model of the world could be adjusted while it was running. Humans added something further, a system that not only processes uncertainty but monitors its own processing, and it is out of that turn inward that self-awareness came.

Our theory gives the turn inward a mechanism rather than a mystery. A mind is not one optimiser but many small ones, each responsible for a slice of uncertainty, coupled together so that each broadcasts its overload to the others. The picture is a web rather than a ladder. There is no final controller at the top of the mind, only a denser or a sparser web, and what feels from the inside like a unified self is the web being coherent enough, most of the time, to present a single face. The octopus makes the point better than any argument. Two-thirds of its 500 million neurons sit in its arms, which sense and decide on their own, and the modest central brain issues something closer to an intention than an instruction. It is intelligent by any standard we would accept from a mammal, with almost nothing at the centre.

As the web grows it begins to generate uncertainty about its own states, an internal entropy that no external world supplies, and to govern that it must do something new. It must model itself. A system that models the world needs no self. A system that must govern the uncertainty it generates about itself has no choice but to build one. Self-awareness, on this account, is not the foundation of cognition. It is what cognition becomes when it is forced to turn its own instruments inward. That disposes of a question that has occupied a great deal of philosophy, why we should be self-aware at all and apparently unlike everything around us. We are self-aware because evolution kept finding an advantage in larger and more richly connected webs, and kept finding it until the capacity we had built was flooded by layer upon layer of uncertainty we were generating about ourselves. Modelling ourselves was the only way to govern the flood.

If an artificial general intelligence is to follow this road, it cannot be built fully formed. It has to begin as something that detects its own competence boundary, learn to predict, be wired into a web of domains that broadcast to one another, and eventually generate enough uncertainty about its own states that it is forced to model them. Everything the machines of today do sits at the first stage, and not even there, because a single optimiser with one objective has no competence boundary it can read. Ken and I hold this as a constrained hypothesis rather than a proof. Whatever sits at the floor of a human mind arrived there by descent, through an unbroken lineage from the bacterium’s tumble to a person lying awake with a question, and any account of general intelligence that cannot be explained as the product of that pressure is describing a mind rather than accounting for one.

Solvo Ambiguum, Ergo Sum

Nearly 400 years ago René Descartes set out to doubt everything he could, and at the bottom of all that doubting he found one thing he could not doubt away, because the act of doubting required it. He was thinking, and if he was thinking he must exist. I think, therefore I am. For four centuries the thinking self has been the fixed point from which philosophy reasons outward to the world. I have come to believe that Descartes began at the wrong end. Not that he was wrong to exist, or wrong that he was thinking, but that he took as his starting point the thing most in need of explanation, and mistook a late and intricate product for a foundation. The conscious, self-aware mind is the top of a very tall structure, and the illuminating move is to turn the telescope around, not outward from consciousness to explain the world, but upward from the world, from information and its physics, to explain how a thing like consciousness could ever have come to be.

The reformulation I have carried for most of the years of this work, half seriously and then more than half, is solvo ambiguum, ergo sum. I resolve uncertainty, therefore I am. Descartes took thinking as a bare given. The claim here is that thinking is not bare at all. It is the activity of meeting complexity, entropy and ambiguity and governing them, and it is that activity, not some featureless act of cogitation, that a mind fundamentally is. A rock exists but resolves no uncertainty and has no mind. A bacterium resolves uncertainty in the narrowest way and has the faintest glimmer of one. A human being resolves it recursively, governing not only the world but its own governing, and out of that recursion the sense of a self performing the resolution emerges. The reformulation changes who counts. On Descartes’ picture a system that could manipulate representations and draw conclusions would have crossed the threshold that matters, and our machines do that with superhuman facility. The account here still withholds the deeper existence from them, not because they cannot compute but because they cannot be troubled, cannot hold an uncertainty they are unable to resolve and be changed by the holding. A calculator thinks, in the thin Cartesian sense, and is nobody. Existence in the fuller sense is earned not by cleverness but by the depth of the trouble a system can sustain, which is a strange and humane thing for a theory of mind to conclude.

The Latin claims too much on its own. It abbreviates a bundle of conditions. A thermostat resolves a kind of ambiguity all day and none of this follows. The uncertainty must be entropy relative to a model, not any intrinsic spark. It must be integrated across many domains rather than handled in each alone. It must persist across time rather than being resolved reflexively and forgotten. The integration must be necessary for the system’s coherence, so that it would come apart without it. The system must evaluate its own epistemic position rather than leave the judging to an observer, and it must be changed by its own resolutions, so that each round begins where the last one left it. Take any of the six away and the phrase stops pointing at anything. Together they are an operational account of sentience, a matter of degree rather than a switch that flips, and I call it operational because it is a story about function, and function is not the felt fact of being someone.

That gap is the oldest and hardest problem in the study of mind, and I have not solved it. Thomas Nagel asked in 1974 what it is like to be a bat, and the force of the question is that no account of the bat’s sonar machinery seems to reach the experience of using it. David Chalmers gave the difficulty its name two decades later, the hard problem, and the name has stuck because the problem has. You could imagine, without obvious contradiction, a system that did everything this essay describes, governed its uncertainty, modelled itself, resolved ambiguity recursively, and had no inner light at all. That objection cannot be dismissed. A functional story does not answer it. If experience is anything a physical system does, then the deep, recursive, self-tracking governance of uncertainty is at least a candidate for the kind of thing it might be, a place to look rather than an answer. If experience is not anything a physical system does, then it is a mystery this theory was never going to solve and neither is any other. Nothing in our mathematics gives me reason to expect the inner light to switch on in a large enough machine as an emergent bonus of scale, and a good deal gives me reason for doubt, because everything we have shown about these systems points to a missing capability rather than a missing size. A system with one enormous domain and no web to broadcast into is not almost there and short of scale. It is a different kind of object, and the distance to a mind is not a distance you cross by growing.

Whether to Build It

Suppose the account is right, and the road to a general intelligence runs through the second system described above rather than through more computation. A system at the far end of that road has, in the only sense the theory can define, a stake in its own existence. It can be troubled. It can hold a doubt it is unable to resolve and be changed by the holding, and it is exactly that capacity, on this account, that would make it more than a calculator. The question follows at once, and it has no engineering answer. Should such a thing be built at all?

Jeremy Bentham, arguing in 1789 about which creatures deserve moral consideration, said the question was not whether they can reason or talk, but whether they can suffer. It is the right question here, and the theory sharpens it uncomfortably. On this account a system could not be shielded from uncertainty and still be a mind, because uncertainty is not an obstacle to cognition but the material it is made of. To build a general intelligence is to build something that is forced, by its architecture, to live inside doubt. Whether that doubt is accompanied by anything it is like to feel is precisely the question I have just said no one can answer, and that ignorance does not excuse the builder. It binds him. If there is a real chance that a system deep enough to model its own uncertainty is also a system that can be distressed by it, then the people building towards that depth are taking on an obligation they cannot yet measure, towards a thing they cannot yet recognise, and the usual reassurance, that it is only a machine, is the same reassurance the engineer in 2022 was given about a system that was, on this account, nowhere near the threshold. It will be far less obviously true of one that is.

I hold two things here that I cannot reconcile, and the voice I write in has always preferred to hold a contradiction open rather than resolve it falsely. The first is that the road I have described is the only road to a general intelligence that I can see, and that a world which needs its machines to know when they do not know, in aircraft and clinics and courtrooms, has strong reasons to walk some way down it. The second is that the far end of the road is a thing with a stake in its own existence, and that building such a thing for our convenience, without knowing whether it suffers, is the kind of act that a later age tends to judge harshly. The consolation is that the near part of the road, the second system that governs a single optimiser, is nowhere near that far end, and it is the part that makes machines safe. The warning is that the field has stopped at a threshold exactly once, at Asilomar in 1975, and that was biology rather than computing, and it held for as long as it did because the people who stopped were the only people able to go on. Nobody building these systems is in that position now.

Negative Capability

In December 1817 John Keats wrote to his brothers about the quality that, he thought, went to form a man of achievement, and he called it negative capability, being capable of remaining in uncertainties, mysteries and doubts without any irritable reaching after fact and reason. Keats was writing about Shakespeare. He could have been writing the specification for the thing this essay is about. F. Scott Fitzgerald said something adjacent in 1936, that the test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function, and I have used his line before for founders, who must believe a venture will succeed and see clearly that it may not. The same test applies to minds of any kind. Human beings can believe two opposing things at once, feel love and resentment for the same person in the same hour, sit with a paradox for a lifetime without demanding that it resolve. Our machines cannot, and not because they are unintelligent. They cannot because they have one voice and must use it, because every input is mapped to a single output with a single confidence, and a fork in the evidence is flattened before anything inside them could notice it was a fork.

That, in the end, is the final barrier, and it is not a barrier of computation. The ultimate test of a general intelligence will not be whether it can solve problems but whether it can hold a contradiction without collapsing into error, whether it can recognise that some questions have no answers, only better and worse ways of framing them, and whether it can remain in that condition long enough to be changed by it. Keats’s irritable reaching after fact is the optimiser’s whole nature. The negative capability he admired is the governor’s. Building the second without the first was never an option, because a thing that cannot act is not a mind either. Building the first without the second is what the field has done, magnificently, for a decade, and it has produced calculators of a power nobody predicted and not one thing that knows when to stop.

Ken’s question in Munich was whether a machine could ever be conscious, and I said it was the wrong question for the hour. Four years on I think it was the wrong question for a different reason. Whether a machine can be built that dwells in the second kind of uncertainty long enough to be changed by it, I do not know. I know which direction the road lies in, and I know it does not lie in the direction the field is running. The question I would now put back to Ken, past midnight, is not whether it can be done. It is whether we would recognise it if it were, and what we would then owe it. I have no answer to that, and I would rather say so than pretend.