AI Models Trained on AI-Generated Data Quickly Produce Nonsense
AI models can collapse according to a new study, but what does that mean for the future of AI, and how can it be avoided?
Training AI models on AI-generated text quickly leads to the models producing nonsense, according to a study. This phenomenon, called model collapse, could halt the development of large language models (LLMs) as they run out of human-generated training data and AI-generated text increases across the internet.
Caution Needed With Training Data
"We need to be very careful about what ends up in our training data," says Zakhar Shumaylov, co-author and AI researcher at the University of Cambridge. Otherwise, "things will always go wrong." The study showed that the problem of model collapse is universal, affecting language models of all sizes that use uncurated data, as well as simple image generators and other types of AI.
Model Collapse in Practice
The researchers used an LLM to create Wikipedia-like texts and then trained new versions of the model on the previous model's text. As AI-generated information, called synthetic data, contaminated the training set, the model's output became nonsense. The ninth version of the model ended a Wikipedia article about English church towers with an essay on the colors of jackrabbit tails.
Human Data Is Crucial
The study also showed that learning from AI-generated text caused models to forget information that appeared least often in their dataset, making the output more homogeneous. This is especially concerning for AI models meant to represent all groups fairly, since low-probability events often relate to marginalized groups.
The Inbreeding Analogy
The problem is compared to inbreeding within a species, says Hany Farid, a computer scientist at the University of California, Berkeley. "If a species inbreeds with its own offspring and doesn't diversify its gene pool, it can lead to the collapse of the species."
Steps to Avoid Collapse
Model collapse doesn't mean LLMs stop working, but the cost of building them will increase. The researchers found that when they trained each model on 10% real data alongside synthetic data, the collapse happened more slowly. To avoid model collapse, developers may need to find ways to keep AI-generated data separate from real data, such as watermarking, and filtering AI-generated text before it re-enters the dataset.
Summary
Model collapse is a serious threat to the development of AI, but with caution and the right measures, it can be managed. To ensure AI continues to improve, we need to make sure training draws on a diversity of human-generated data.