AI chatbots are breaking the web—and forcing a 404 makeover

More than half of Americans now use a chatbot, with an increasing number of people replacing search engines with large language model (LLM)-powered chat queries to navigate the web and find answers. In general, the quality of these outputs is improving as the underlying models get better.

However, the challenge of processing so much information means AI models sometimes misfire, hallucinating details—including website URLs. One analysis of 18,000 landing page visits from ChatGPT found that the AI system included the wrong URL in roughly one out of every 33 links. Sometimes, the mistake is just a single letter off; other times, the URL is entirely fabricated. The result is that website hosts are beginning to adapt, trying to capture this traffic and redirect it to the intended target.

From designing new 404 error pages that explain what may have gone wrong and offer contextual links, to creating on-the-fly content—often with the help of AI tools themselves—various strategies are being deployed. These efforts aren’t quite AI engine optimization, the anticipated successor to traditional search engine optimization (SEO), but they reflect a growing urgency as users shift from Google searches to chatbot interactions. They’re an attempt to acknowledge and improve the experience of those who land on websites via AI hallucinations or errors.

Jack Arturo, who runs WP Fusion, a WordPress plug-in, has noticed the impact of artificial intelligence on traffic to his site. Data shared with Fast Company shows that in September 2024, ChatGPT was responsible for less than 0.5% of his referral traffic. “A year ago, it didn’t really exist as a referrer,” he explains. “But if we go for the last 30 days, it’s number three—well, actually, really number two, after Google, because these are direct visits,” he says.

Most of the AI chatbot traffic Arturo receives is directed to live pages. But in line with broader trends, about three out of every 100 links are broken. Until recently, those users landed on WP Fusion’s standard 404 page, similar to many others that have existed on the web for decades. It informs visitors that something went wrong and the page can’t be found, suggesting they use the on-site search tool to locate what they’re looking for. Still, it seemed like a missed opportunity to auto-redirect a chatbot-referred visitor to something more relevant than a generic “Ooops” message.

To address that, Arturo built an AI-powered system that generates content for nonexistent pages that users were sent to. “We know what they’re looking for based on the title,” Arturo explains. “Why not just generate something for them?”

The system took only about an hour to set up, but Arturo believes the time was well spent and expects future enhancements will be, too. “We see the traffic increasing consistently from those sources,” he says. “I think it’s worth spending some time and getting it right, and then we’ve open-sourced it, so ideally, other people would contribute.”

SEO experts agree that website owners will need to adapt as AI chatbots become more embedded in everyday browsing. “LLMs have a tendency to hallucinate URLs that don’t exist, so if the user clicks on them, they will land on your 404 page,” says SEO expert Lily Ray. This makes it important for site owners to rethink how 404 pages function and to guide users toward the content they were originally seeking.

“The best way to provide an optimal UX [user experience] on 404 pages is to provide messaging about the content not being found and include links to important pages on the site,” says Ray, who also suggests incorporating standard site navigation into the 404 page template.

Arturo, for his part, acknowledges this is yet another case of AI systems creating new challenges for those not involved in their development. “It would be nice if it sent them to the correct page,” he says. But he adds that even AI companies may not fully understand how their models behave and may not be able to fix the issue right away. “Until they fix it—if they can fix it—we just have to kind of adapt and work around it.”