Reddit has recently implemented a significant policy change that affects how search engines and artificial intelligence (AI) bots access its content. According to a report from 404 Media, Reddit has updated its robots.txt file to restrict the crawling of its website by most search engines and AI bots unless they enter into a paid partnership. This move has led to notable consequences for users of search engines other than Google.

Impact on Search Engines

As of July 1, Reddit's updated robots.txt file has led to a situation where only Google is currently able to index and display recent Reddit content in search results. Google has reportedly paid $60 million to Reddit for access to its data, which allows it to continue showing updated Reddit results. In contrast, alternative search engines such as Bing, DuckDuckGo, Mojeek, and Qwant are now unable to provide recent Reddit content in their search results. While older Reddit content still appears in these search engines, they are unable to crawl the site for new posts and comments due to the new restrictions.

Microsoft has confirmed that Bing's ability to crawl Reddit has been halted. A Microsoft spokesperson stated that "Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site." This reflects the broader impact of Reddit's decision on search engines other than Google.

Reddit's Stance on Web Crawlers

Reddit's policy update is part of a broader crackdown on web scraping, which is using bots to extract data from websites. The company has introduced these restrictions to curb the unauthorized use of its data for training AI models. Ben Lee, Reddit's chief legal officer, explained that the new policy is intended to signal to companies that do not have agreements with Reddit that they should not access its data. This policy requires search engines to agree to paid partnerships if they wish to continue indexing recent content from Reddit.

Reddit has clarified that the changes are unrelated to its recent partnership with Google. A Reddit spokesperson, Tim Rathschmidt, emphasized that the company has been in discussions with multiple search engines but has not reached agreements with all of them due to issues related to enforceable promises regarding the use of Reddit content, particularly for AI purposes.

Read Also: You may soon get Snapchat-like 3D face filters in this Google app, heres what we know so far