The capabilities of large language models
Prediction and Understanding
Draft — September 2026. Model figures cited below are current as of this date and will age quickly.
That this level of capability has come about through large language models and the transformer architecture is astounding. The architecture was designed in 2017 for machine translation. Nobody involved was trying to build a general reasoner. What they built was a machine for predicting the next token in a sequence, and then the field discovered that if you make that machine large enough and feed it enough of what humanity has written, something else arrives that nobody specified.
The standard dismissal is that these systems are "just" next-token predictors — that whatever they appear to be doing, underneath it is only autocomplete. I want to take that objection seriously rather than wave it away, because the interesting thing is not that it is wrong but that it is true and yet insufficient. They are next-token predictors. The question is what a system has to become, internally, in order to be a good one.
1. What prediction demands
Start with the easy case. In "Roses are red. Violets are ___", the next token is blue, and you need nothing resembling understanding to get there. A bigram model trained on a few thousand greeting cards will manage it. Co-occurrence statistics are enough. It is worth conceding how much of language is like this: a great deal of text is predictable from shallow surface regularities, and for those stretches the sceptic's account is simply correct.
But the distribution of natural text does not stop there, and this is where the argument has to be made carefully rather than asserted. Consider what it takes to predict the final word of a murder mystery's reveal, or the next line of a proof, or the response of a character who has been lied to in chapter three and has not yet found out. In each case the token that actually comes next is determined by structure that is nowhere present in surface co-occurrence. It is determined by who knows what, by what follows from what, by the state of a world the text has been describing. A system that models only word statistics is not merely worse at these — it has no purchase on them at all. Gradient descent on next-token loss over a large enough corpus therefore applies relentless pressure toward internal machinery that tracks the things the text is about.
That is the argument. It is an argument about what the training objective rewards, not a proof about what any particular model contains. Fortunately we no longer have to rely on the intuition, because we can look inside.
Induction heads. Interpretability work at Anthropic in 2022 identified a specific circuit — a pair of attention heads working in composition — that implements in-context pattern completion: having seen [A][B] earlier in the sequence, complete a later [A] with [B]. What makes this more than a curiosity is the timing. These heads form during a narrow phase transition in training, visible as a bump in the loss curve, and their formation coincides with the model's abrupt acquisition of in-context learning. A capability that looks emergent from the outside turns out to have an identifiable mechanism with a birthday.
Othello-GPT. The most direct evidence that sequence prediction produces world models comes from a 2023 experiment by Kenneth Li and colleagues. They trained a transformer on nothing but sequences of legal Othello moves — no board, no rules, no images, just move tokens. A linear probe trained on the model's activations recovers the state of the board. More tellingly, when the researchers intervened directly on that internal representation — editing the model's belief about the board rather than its input — its subsequent move predictions changed to match the edited board. The representation is not a decorative correlate. It is load-bearing. A system given only a string of symbols built, unprompted, a model of the thing the symbols were about, and used it.
Features. Sparse autoencoder work on production-scale models has since extracted millions of interpretable features from the residual stream: features that fire on a specific concept across languages and across modalities, features for code vulnerabilities, for sycophancy, for deception. Amplify one and the model's behaviour bends around that concept. Whatever these models are doing, it is organised around concepts and not around strings.
Grokking. Small models trained on modular arithmetic memorise the training set first and generalise much later, sometimes long after the training loss has flattened. Reverse-engineering the grokked models reveals that they have discovered an actual algorithm — a trigonometric procedure using discrete Fourier components. The network moved from a lookup table to a method, and we can read the method off the weights.
None of this proves the systems understand in whatever sense you take that word to require. It does establish something narrower and sufficient for my purposes: the internals are not a lookup table. Concepts, world state, and algorithms are in there, built by prediction pressure alone.
The strongest objection
The serious version of the sceptical case is not "it's just autocomplete." It is Bender and Koller's, from 2020. Imagine a hyper-intelligent octopus that taps into an undersea cable and observes years of conversation between two humans on separate islands. It learns the statistical structure of their exchanges perfectly — well enough to impersonate one of them. Then one islander is attacked by a bear and asks for help building a weapon. The octopus, having never encountered a bear, a stick, or a coconut, has nothing. It has mastered form with no access to meaning, because meaning lives in the relation between the form and a world the octopus has never touched.
This is a real argument and it lands a real hit. Grounding is not nothing, and a system trained on text alone has never stubbed its toe. But notice what Othello-GPT does to the octopus. The octopus's problem is supposed to be that form cannot carry the structure of the world. Yet when the form is a faithful trace of a world — and move sequences are exactly that — the structure comes through, and can be recovered, and does real work. Human language is a far noisier trace than Othello notation, but it is a trace of the same kind. It is the compressed output of billions of embodied minds describing the world they are embedded in. That is not a perfect substitute for having a body. It is also not nothing, and the octopus argument assumes it is nothing.
The honest position, I think, is in between, and it is not the position either camp defends. These systems have a great deal of structure corresponding to the world and very little contact with it. They know what a lemon does to a sentence and have never tasted one.
2. Threads of thought
When the model is used as a chatbot, whatever it has built feeds into what it says back to us. A conversation accumulates: each response is conditioned not on the last message but on the entire exchange, and the context window becomes the ceiling on how much complexity can be held in play at once.
The generation itself has a temporal shape. A token is produced, appended, fed back; the model runs again on the slightly longer sequence; and this continues until an end-of-sequence token. In a loose sense the model is "thinking" and "responding" across that stretch. Call it one thread.
Hosted systems run enormous numbers of these threads at once, batched together through the same weights. It is often noted that attention masking guarantees no leakage between sequences in a batch — but this is trivially true, a consequence of how batched matrix multiplication works, and on its own it is not an interesting fact. The interesting fact is what it implies about ownership.
When many threads run concurrently through one network, there is no thread among them that has access to the others, no vantage point from which they are all mine. If there is anything it is like to be one of these threads, there is nothing it is like to be the network. Nor does anything a thread does persist: the weights are frozen at inference, so whatever a thread works out dies with it. What we have is not one mind entertaining many thoughts. It is a large number of transient, mutually invisible processes borrowing the same substrate — a shared body of knowledge with no one home to own it.
An agent harness changes the shape of a thread without changing this. Feed the model's output back in, give it scratchpads for memory and planning, give it tools that act on the world and return results, and the loop can pursue goals, decompose them into subgoals, and continue until some condition is met. That is a longer and more consequential thread. It is still a thread, and it still ends.
3. The human thread
For us, as long as we are awake, there seems to be something continuous. It often presents as an internal monologue — a conversation with oneself — and it is tempting to identify that monologue, coupled with a continuous feedback loop of self-awareness, with consciousness itself.
I want to resist that temptation, for a reason that turns out to matter for everything that follows. Inner speech is not universal. A substantial minority of people report little or no inner monologue at all, a condition now labelled anendophasia; Galton noticed comparable variation in mental imagery in the 1880s. And when Russell Hurlburt's group sampled people's experience at random moments, inner speech turned out to occupy a much smaller fraction of it than introspection predicts. Whatever consciousness is, it is not identical with the voice, because some conscious people do not have the voice.
So I will make the narrower claim. One mode of human thought is linguistic, sequential, and available to report. That mode is what admits comparison with a language model, and it is the only mode I will compare. The rest — imagistic, motor, affective, whatever is happening when a mathematician says the answer arrived whole — is out of scope here, and its existence should be a standing caution against concluding too much from the comparison.
Within that mode, how the next thought arrives is genuinely unknown. It clearly does not depend only on the previous thought. It involves situational input, memory, mood, and something as unglamorous as fluctuating neurotransmitter levels. Multiple sub-processes appear to compete, and the spotlight of attention settles on one at a time — which is not to say the others halt. They likely continue out of sight and occasionally hand something up.
4. The geometry of belief
What does it mean to understand? We make sense of the world by building concepts out of simpler concepts and assembling them into models. There must be some ceiling on how tall that stack can go before it falls over, and one place the ceiling is visible is in nested belief — what evolutionary psychologists call higher-order intentionality.
Dunbar's much-repeated figure is that humans reliably handle four or five orders before failing. The figure deserves more hedging than it usually gets: it is task-dependent, format-dependent, and the experimental work supporting it has been contested. But the qualitative claim — that recursive mental state attribution degrades fast, and much faster than recursion in most other domains — is not seriously disputed by anyone who has tried to hold one in their head.
Try this one. The strike is genuinely planned for Tuesday.
Agent Smith feeds false intelligence to enemy spy Jones, telling him the strike is on Friday. Jones sees through the lie and correctly deduces Tuesday. Through a mole, Smith learns his lie failed and knows Jones expects Tuesday.
To regain the advantage, Smith sends a deliberately weakly-encrypted memo to his own commander: "Jones has fallen for our deception and is preparing for Friday." Jones intercepts it, takes it for a genuine private report, and believes it.
Jones, wanting to keep Smith comfortable in that error, now orders visible Friday-facing preparations across his own sector. Smith's mole reports those preparations, and Smith takes them at face value.
Smith therefore now believes that Jones believes that Smith believes that Jones expects the strike on ___
The answer is Friday, and every attractor in the passage pulls toward Tuesday — because Tuesday is the truth, and because Jones really does expect Tuesday. Getting there requires holding four embedded frames simultaneously while ignoring the ground truth that contradicts all of them. Most people find that the frames start sliding into one another around the third.
A capable language model answers this immediately, and can answer harder ones. It can also construct an instance deep enough that no unaided human mind can resolve it — which is a sharper demonstration than solving one, since it requires the deep structure to be available generatively rather than just recognitionally.
But I want to be careful about what this shows, because the obvious reading is unfair in a way that undercuts it.
The human failing here is largely a working-memory failing, not a failure of the underlying competence. Give a person the same passage, a pen, and five minutes, and they will draw a little diagram of nested boxes and get it right. This is what external representation is for. And notice what that concession implies: the model has external representation too. Its context window is its scratchpad. Every token of the passage remains equally available to it at every step, which is exactly the resource I just handed the human. Comparing an unaided human to a model reading from a perfect record is not a fair fight, and the fair fight is much closer.
So the real asymmetry is not that the model can do something we cannot. It is speed and integration. We externalise serially, in an effortful loop — write, read back, update, write again — and the loop is slow and lossy and we lose the thread if interrupted. The model does the equivalent in a single forward pass, computing the relations between every pair of tokens in parallel rather than shuffling them one at a time through a four-item bottleneck. That is a difference in architecture rather than a difference in kind, and it is more than sufficient to be going on with. It does not need inflating.
5. Superhuman in what, exactly
Human minds are extraordinary and everything humanity has built is a product of thought. Think of the range: every society, every culture, every language carving the world at slightly different joints, words in one language for concepts another has no name for. All of that is written down somewhere — in books, transcripts, films, code, arguments, letters. Language models have ingested a substantial fraction of it and built a single map across all of it.
The map is real and it is unprecedented. No individual has read a thousandth of that corpus, and no individual holds the cross-lingual, cross-domain correspondences that fall out of compressing it jointly. In breadth of exposure to human expression, these systems exceed any person by an enormous margin, and that is not a controversial claim.
The step people take next is the one I want to slow down. Breadth of ingestion is not depth of understanding, and sliding from one to the other is the central error in most writing of this kind — including, I should say, my own earlier drafts of this essay. Having read everything ever written about immunology is not the same as understanding immunology better than an immunologist. Compression finds structure, but it also discards; a model's grip on any particular domain is bounded by what the text actually encoded, and the tacit knowledge of a practitioner is largely not in the text. The honest formulation is that these systems have unprecedented coverage and highly variable depth, and that the depth is superhuman in some places and unreliable in others, in a pattern we are still mapping.
The same caution applies to the context window, which is the other place the argument usually overreaches. GPT-6 Astra, released this month, has a context window of 1,050,000 tokens — somewhat more, in raw volume, than the first five Harry Potter books. It is tempting to say that this is what is held in the model's mind in a single execution, and that it dwarfs the four-to-seven items of human working memory by five orders of magnitude.
Both halves of that are wrong.
The first half is wrong because attention over a million tokens is not uniform in quality. Performance degrades over long contexts in well-documented ways — the "lost in the middle" effect, where material buried mid-context is retrieved far less reliably than material at either end. And retrieval is the easy case: finding a planted fact in a haystack is a much weaker feat than reasoning jointly over everything in the haystack, and long-context benchmarks that test the latter show much steeper falloff. The window is a stated capacity, not a guarantee of uniform access, and the gap between them is where a lot of practical disappointment lives.
The second half is wrong because working memory is the wrong comparison class. A context window is not what a human holds in mind. It is closer to what a human has open on the desk — and the human analogue of the pre-trained weights is not working memory either, it is a lifetime of long-term memory plus every book they can reach plus every colleague they can ask. Compared properly, against memory plus tools plus institutions, the human side of the ledger is far less lopsided than the four-items-versus-a-million framing suggests.
And then there is what is missing altogether. The weights are frozen; nothing learned in a conversation survives it, so there is no accumulation, no career, no growing expertise. There is no contact with the world except through text handed to it. There is no continuity of self across the threads described in section 2. And the capability profile is jagged in a way no human's is: the same system that handles the five-order belief puzzle above will fail at counting, at a spatial task a child manages, at a variation of a familiar puzzle that a human would notice had changed. Whatever this intelligence is, it is not a uniformly scaled-up version of ours, and the jaggedness is not a temporary defect to be smoothed away by the next release — it is information about how differently the thing is built.
6. Where this leaves us
So: not stochastic parrots, and not a mind vastly superior to ours. Both of those are comfortable positions, and their comfort is exactly what should make us suspicious of them.
What we have instead is stranger than either. Systems built by prediction alone that turn out, verifiably, to contain concepts and world models and algorithms. Systems with a map of human expression that no person could hold, and no body to attach it to. Systems that resolve in one pass structures we can only unpack with a pen, and that stumble over things we do not think of as things. Many threads, no owner, nothing retained.
We should stop asking whether they are really intelligent, which is a question about our word. The better question is which of our capacities they have reconstructed, which they have bypassed, and which they have not touched — and that question is empirical, it is partly answered already, and we are answering more of it every month.
What I am confident of is narrower than where I started, and I think it survives contact with the objections. Prediction at scale is not a shallow objective. It built something. And the honest description of what it built is more interesting than either of the stories we keep reaching for.