The Monk's Error, The Model's Echo

In the ninth century a monk in a scriptorium copies a manuscript by candlelight, and somewhere in the third chapter his hand slips, turning one word into another that sounds almost the same. The error is small enough to survive uncorrected, since no one thinks to check a text against a memory of a memory. Two centuries later a different monk copies that copy, and the slip is no longer a slip, it is simply what the text says, because no one alive has seen the original to correct it.

Something structurally similar is now happening at a speed no scriptorium ever managed. Large language models are trained on enormous samples of text drawn from the internet, and an increasing share of that text is itself generated by earlier language models rather than written by anyone at all. The next generation of systems is, in part, learning the world from the previous generation's guesses about the world.

Researchers have a name for the resulting drift, model collapse, the tendency for a system trained repeatedly on its own outputs to lose the rare, unusual, and statistically marginal information that made its training data resemble the actual richness of human experience. Each generation smooths slightly toward its own average, confident where it should be uncertain, until the tails of the distribution, the odd fact, the minority view, the exception that proved a rule, quietly go missing.

The next generation of systems is, in part, learning the world from the previous generation's guesses about the world.

It would be tempting to treat this as a novel danger unique to machines, but human culture has run versions of the same experiment for a long time. Oral traditions drift with each retelling. Newspapers cite other newspapers citing an original report that no living reporter can locate. Encyclopedias have occasionally been caught copying each other's errors so faithfully that a fabricated entry survived across editions for years, cited as though it were independent confirmation of itself.

The difference, if there is one, is not that machines recycle information while humans do not, but that human recycling has historically kept at least an intermittent tether to the world outside text, a traveler correcting a rumor, an experiment overturning a citation, a witness contradicting a legend. Language models, by contrast, encounter the world exclusively through text about the world, and as that text increasingly originates from other models rather than from perception, the tether loosens without anyone deciding to cut it.

What erodes first is not accuracy in any single fact but the internet's usefulness as a kind of ground truth, the background assumption that enough independent observers, writing independently, will collectively triangulate toward something real. When independence itself becomes uncertain, when a claim's frequency across the web may reflect how often a model repeated itself rather than how often reality was witnessed, verification stops being a matter of counting sources and becomes a much harder problem of tracing lineage.

Perhaps the more unsettling implication is how much of this predicament was already latent in human knowledge before any model existed. Most of what any person believes rests on testimony trusting other testimony, a chain few of us ever trace back to a direct encounter with the fact itself, whether the claim concerns a distant war, a historical date, or the safety of a medicine never personally tested. The scriptorium's error and the model's collapse are not aberrations in an otherwise clean system of knowing. They may simply be the moment the system's oldest vulnerability became visible enough, at last, to name.