Why AI Is Training on Its Own Garbage (and How to Fix It)
Why AI Is Training on Its Own Garbage (and How to Fix It)
https://towardsdatascience.com/why-ai-is-training-on-its-own-garbage-and-how-to-fix-it/
Publish Date: 2026-04-08 12:30:00
Source Domain: towardsdatascience.com
in AI for a while, you are probably an LLM/Agent/Chat user, but have you ever asked yourself how these tools will be trained in the near future, and what if we have already used up the data we need to train models? Many theories say that we are running out of high-quality, human-generated data to train our models.
New content goes up every day, that’s a reality, but an increasing share of what gets added daily is itself AI-generated. So if you keep training on public web data, you’re eventually training on the outputs of your own predecessors. The snake eating its tail. Researchers call this phenomenon Model Collapse, where AI models start learning from the errors of their predecessors until the whole system degrades into nonsense.
But what if I told you we aren’t actually running out of data? We’ve just been looking in the wrong place.
In this article, I am going to break down the key insights from this brilliant paper.
The Web We Already use and the Web That Matters
Most of us consider the web as a unique source of information. In reality, there are at least two.
There is the Surface Web: the indexed, public world like what we find on Reddit, Wikipedia, and news sites. This is what we’ve already scraped and overused for years to train the mainstream AI models of today. Then, there is what we call the Deep Web, and here I’m not talking about the “Dark Web” or anything illegal.
The Deep Web is simply everything behind a login or a firewall. It refers to anything online that isn’t publicly indexed. It could be your hospital’s patient portal, your bank’s internal dashboard, enterprise document archives, private databases, and years of email sitting behind a login screen. Normal, boring, but incredibly valuable data.
Many studies suggest the Deep Web is orders of magnitude larger than the surface web. More importantly, it is crucially better quality data. Compared to surface web content, which can be noisy, full of…
