According to the Internet Infrastructure Provider CloudFlare, AI startups’ stumps are raw scraping content from websites that explicitly indicate they don’t want to be scraped away.
On Monday, CloudFlare published a survey that found that AI startups ignored blocks and observing their raw or scraping activities. The Network Infrastructure giant has accused them of obscuring their identity when trying to scrape web pages “to avoid website preferences,” CloudFlare researchers wrote.
AI products like Prplexity offer rely on gobbling large amounts of data from the Internet, and AI startups have repeatedly scraped text, images and videos from the Internet without permission to make the product work. Recently, the website has tried to fight back using the Web Standard Robots.txt file. It tries to tell search engines and AI companies whether they can index their efforts that they have seen a wide range of results.
According to CloudFlare, it appears they are willing to bypass these blocks by changing the “user agent” of the bot.
“This activity was observed across tens of thousands of domains and millions of requests per day. We were able to fingerprint this crawler using a combination of machine learning and network signals,” read CloudFlare’s post.
Perplexity spokesman Jesse Dwyer dismissed the CloudFlare blog post as “sales pitch” and added an email to TechCrunch that said it “indicates that the content was not accessed.” In a follow-up email, Dwyer insisted on the CloudFlare blog a bot named “Not us.”
CloudFlare said the action was first noticed after customers complained that they were baffled and raw and rubbed the site in distress, especially to block known bots in Prplexity. CloudFlare then ran tests to check and confirmed that the confusion was avoiding these blocks.
TechCrunch Events
San Francisco
|
October 27th-29th, 2025
“Perplexity observed that it uses not only declared user agents, but also a common browser that impersonates Google Chrome on MacOS when declared crawlers are blocked,” CloudFlare said.
The company also said it has created Perplexity bots from its verified list and added new techniques to block them.
CloudFlare has recently taken a public stance against AI Crawlers. Last month, CloudFlare announced the launch of a market that will allow website owners and publishers to claim AI scrapers to visit their sites. CloudFlare CEO Matthew Prince sounded the alarm at the time, saying that AI was breaking the internet, particularly the publisher’s business model. Last year, CloudFlare launched a free tool to prevent bots from shaking websites to train AI.
This is not the first time that confusion has been accused of rubbing without permission.
Last year, news outlets such as wired claim that confusion was plagiarizing their content. A few weeks later, Perplexity CEO Aravind Srinivas was unable to answer immediately when asked to provide a definition of plagiarism in an interview with Devin Coldewey of The TechCrunch at The Disrupt 2024 Conference.
Source link
