Next-Token Predictor Is An AI's Job, Not Its Species

Scott Alexander explores the central misconception that large language models (LLMs) are merely 'next token predictors,' arguing that this view confuses different levels of explanation. Drawing a parallel between human cognition and AI development, he illustrates how both are shaped by high-level optimization processes—evolution for humans, corporate profit motives for AI—while their inner workings involve complex, abstract representations like predictive coding in the brain and helical manifolds in AI. He emphasizes that just as humans don't consciously think about survival or reproduction when solving math problems, AIs don't 'think' in terms of next-token prediction when generating responses. Instead, both use sophisticated, often unintuitive internal mechanisms to model the world. The episode uses the example of Claude’s line-breaking behavior, which relies on 6D helical manifolds, to show how AI systems develop strange but effective computational structures. Ultimately, Alexander contends that labeling AI as a 'stochastic parrot' is a category error, akin to saying humans are just survival machines because evolution optimized for reproduction. The real intelligence lies in the high-level thought processes, not the low-level training mechanics.