Model Collapse: What Happens When AI Trains on AI Output

Model collapse is a degenerative process in which generative AI models trained on data produced by earlier AI models gradually lose information about the real world. Each generation learns from a slightly distorted copy of reality made by the one before it. Rare details disappear first, and over many generations the output becomes repetitive and drifts away from the original data.

The term comes from a paper by Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal, published in Nature on 24 July 2024 as “AI models collapse when trained on recursively generated data”. Its formal definition: “Model collapse is a degenerative process affecting generations of learned generative models, in which the data they generate end up polluting the training set of the next generation.” An earlier version appeared on arXiv in May 2023 under the title “The Curse of Recursion: Training on Generated Data Makes Models Forget”.

Why it matters now

Large language models are trained mostly on text scraped from the web, and the web is filling with AI output. Studies using AI detectors estimate that about half of newly published English-language articles are now primarily AI-generated (see our analysis of AI slop). Every future model that trains on a fresh web crawl therefore takes in some share of machine-made text, which is exactly the loop the Nature paper describes. Its authors conclude that data about genuine human interaction will be “increasingly valuable” as a result.

How model collapse happens

The paper identifies “three specific sources of error compounding over generations”:

Statistical approximation error. The primary cause. Each model learns from a finite sample, so rare events may simply not appear in it. What the sample misses, the next model never sees.
Functional expressivity error. A secondary cause. A model has limited capacity to represent the true distribution. The paper’s example is fitting a mixture of two Gaussians with a single Gaussian: the shape is lost no matter how much data you have.
Functional approximation error. Also secondary. Training methods themselves introduce bias, for example through the structure of gradient descent or the choice of objective.

Each error on its own is small. Repeated over generations, they compound.

Early and late collapse

The paper separates two stages:

Early model collapse: “the model begins losing information about the tails of the distribution”. Rare words, unusual facts and minority cases fade first, while average performance can look fine.
Late model collapse: “the model converges to a distribution that carries little resemblance to the original one, often with substantially reduced variance.” Output becomes narrow, repetitive and often nonsensical.

In the language-model experiment, the researchers fine-tuned Meta’s small OPT-125m model on Wikipedia-derived text (wikitext2), then trained each new generation on the previous one’s output. In one example, a passage about English parish church towers had turned into a list of jackrabbit varieties by the ninth generation.

It is not limited to text

Image models show the same effect. Rice University researchers led by Sina Alemohammad and Richard Baraniuk (arXiv July 2023, ICLR 2024) studied “self-consuming” image generators and found that “without enough fresh real data in each generation”, future models are “doomed to have their quality (precision) or diversity (recall) progressively decrease”. They named it Model Autophagy Disorder, or MAD, “by analogy to mad cow disease”, and reported that it appears “in just a few generations”.

Can it be avoided?

The research points to one main factor: whether synthetic data replaces real data or accumulates alongside it.

Gerstgrasser and co-authors from Stanford and other institutions (arXiv April 2024) found that “replacing the original real data by each generation’s synthetic data does indeed tend towards model collapse”, while “accumulating the successive generations of synthetic data alongside the original real data avoids model collapse.” With accumulation, test error has a finite upper bound however many generations pass.
Dohmatob and co-authors pushed the other way in “Strong Model Collapse” (ICLR 2025), reporting that in their theoretical setting “even the smallest fraction of synthetic data (e.g., as little as 1 per 1000) can still lead to model collapse”, and that larger models can amplify the effect up to a point.

Synthetic data is not banned by this research. Microsoft’s Phi series is a well-known example of using it deliberately: phi-1 (2023) was trained partly on GPT-3.5-generated “textbooks” and exercises, and the Phi-4 report (December 2024) says the model “strategically incorporates synthetic data throughout the training process”. The difference is curation: synthetic data generated for a purpose, filtered, and mixed with real data, rather than unlabeled AI text scraped back in by accident.

	Model collapse	Model drift	Overfitting
What goes wrong	Each generation trained on AI output loses information	A deployed model’s accuracy falls as real-world data changes	A model memorizes its training data and fails on new data
Cause	Recursive training on generated data	The world changes after training	Too little data or too complex a model
Timescale	Across model generations	During deployment	Within one training run

See model drift and overfitting for those terms.

FAQ

What is model collapse in AI?

The degradation that occurs when generative models are trained on content made by earlier models. Over generations they lose rare information, become more repetitive and drift away from the real data distribution.

Who discovered model collapse?

The term was introduced by Ilia Shumailov and co-authors from Oxford, Cambridge, Imperial College London, the University of Toronto, the Vector Institute and other institutions, first on arXiv in May 2023 and then in Nature in July 2024.

Is model collapse happening to ChatGPT or other current models?

There is no public evidence that current frontier models have collapsed. Labs curate and filter training data, and research shows that keeping real data in the mix prevents collapse. The concern is about the long term, as AI-generated text becomes a growing share of the web.

Does using synthetic data always cause model collapse?

No. Research shows collapse comes mainly from replacing real data with synthetic data. Accumulating synthetic data alongside real data, and curating it carefully, avoids it in the studied settings, although some theoretical work finds even small fractions can cause harm.

What is Model Autophagy Disorder (MAD)?

A name for model collapse in image generators, coined by Rice University researchers in 2023 by analogy to mad cow disease: models fed on their own output degrade in quality or diversity within a few generations.

The wider context: AI Slop by the Numbers: How Much of the Internet Is Now Machine-Made.