AI data wars push Reddit to block the Wayback Machine

As the battle to train artificial intelligence models becomes more intense and Reddit’s rich content library becomes more valuable, the social media giant has taken steps to block the Internet Archive from indexing its pages.

While the Wayback Machine has historically recorded all Reddit pages, comments and user profiles, the company has put limits on what the system can scrape. Moving forward, it will only be permitted to archive the site’s home page, which shows popular posts and news headlines of the day, but no user comments or post history.

The action comes as Reddit has become increasingly protective of the content on its site. Reddit, in May, announced it had struck a deal with OpenAI to use its content to help train ChatGPT. It previously announced a similar deal with Google – and blocked other search engines from crawling the site after that deal unless they struck financial agreements with Reddit as well.

AI companies that are less well-financed, however, have reportedly been using the Internet Archive to scrub the site’s previous posts and train their large language models from that content.

Reddit spokesperson Tim Rathschmidt, in a statement, told Fast Company “Internet Archive provides a service to the open web, but we’ve been made aware of instances where AI companies violate platform policies, including ours, and scrape data from the Wayback Machine. Until they’re able to defend their site and comply with platform policies (e.g., respecting user privacy, re: deleting removed content) we’re limiting some of their access to Reddit data to protect redditors.”

Reddit shares were higher Tuesday, gaining more than 3% in midday trading, hitting $228. Year to date, the company’s stock is up 38%.

Reddit’s legal battles meet its AI ambitions

In June, Reddit sued Anthropic, claiming the AI company behind the Claude chatbot was scraping the Reddit site.

“In July 2024, Anthropic claimed, in response to Reddit’s public protests regarding Anthropic’s misuse of Reddit content, that it had blocked its bots from accessing Reddit. Not so,” the suit reads. “Anthropic’s bots continued to hit Reddit’s servers over one hundred thousand times. … Unlike its competitors, Anthropic has refused to agree to respect Reddit users’ basic privacy rights, including removing deleted posts from its systems.”

(Anthropic has denied the accusations.)

Reddit’s latest defensive act against AI scraping comes as the company is focusing more on its own AI initiatives. Last December, the company rolled out Reddit Answers, an AI-powered tool that will summarize conversations and posts on the site, letting users bypass traditional search engines. That AI product is now used by six million people, the company said in its second quarter earnings announcement, up from one million in the first quarter.

Reddit is planning to use that momentum, as well as the significant use of its own internal search engine (which the company says services 70 million users per week) to challenge Google and other popular search tools.

“The world and the internet are rapidly changing, and I believe Reddit has a once-in-a-generation opportunity,” said CEO Steve Huffman on an earnings call following the earnings. “We’re unifying [search and Reddit Answers] into a single search experience. We’re going to bring that front and center in the app. So, whether you’re a new user opening the app for the first time or returning user opening the app, that search box will be present immediately for users who open the app looking for something specific.”

While Reddit’s efforts in the search space will include AI components, the company said it hopes to differentiate itself from the growing number of AI search engines by highlighting the human component.

“Conversation and connection are becoming more valuable and rare,” said Huffman. “In a world increasingly dominated by algorithms and automation, the need for human voices has never been greater.”