Web infrastructure provider Cloudflare has been pushing for increased regulation in the AI space since earlier this year when it launched a marketplace that allows websites to charge AI bots for scraping their content.
The company’s chief executive, Matthew Prince, said he was in London to speak to Britain’s Competition and Markets Authority (CMA), which is proposing stricter rules for how Google can compete in the AI race given its search dominance.
Earlier this month, the CMA granted Google special status in the search and advertising markets due to its “substantive and established” position. The move will allow regulators to impose stricter regulations not only on search and advertising, but also on areas such as Google’s AI Overview, AI Mode, Discover feed, Top Stories, and News tab.
Prince said that while Cloudflare is not in the AI business itself, it has many relationships with AI companies themselves, which puts it in a good position to make recommendations.
“We’re not directly involved in the fight. We’re not an AI company,” Prince said at the Bloomberg Tech Conference in London this week. “We are not a media publisher, but a network between media publishers. 80% of AI companies are our customers,” he added.
Cloudflare’s president believes Google should compete on a level playing field with other AI companies, but he said that’s not the case right now. Rather, Google uses its existing web crawler, in addition to its search engine, to crawl content for AI products and services. This gives Google an unfair advantage, Prince said.
“Google is saying, ‘We have an absolute God-given right to all the content in the world, even if we don’t pay for it. Because look what we’ve done over the last 27 years,'” Prince explained. “We’re saying you can use the same crawler that you use for search to power the system. And if you want to opt out of either, you have to opt out of both,” he noted.
tech crunch event
san francisco
|
October 27-29, 2025
This is clearly not feasible for most companies, especially those in the media industry, where losing search means losing about 20% of revenue, the executive said.
“But it gets worse. Blocking Google’s crawlers also blocks Google’s ad safety team, which means ads will stop working on all platforms. This is not just the beginning,” Prince added.
Google bundles its own crawler, which gives you access to content that you would have to pay for from other companies like Anthropic, OpenAI, and Perplexity.
“The problem is we’re effectively handing the game over to Google,” he said.
The solution, Prince said, is to foster more competition in the market, potentially with thousands of AI companies competing to buy content from thousands of media companies and millions of small businesses. He suggested that the UK’s CMA flagging Google as a potential regulatory target was a thoughtful move and shows they recognize Google’s unique advantages.
Cloudflare also provided CMA with data showing how Google’s crawler works and why it’s nearly impossible for other players to replicate Google’s success.
Prince isn’t the only one sharing this opinion these days. Last month, Neil Vogel, CEO of People, the nation’s largest digital and print publisher that operates more than 40 media brands, said essentially the same thing. In an interview, he called Google the “bad guy” and said that because of the way its crawlers were combined, media companies had no choice but to let Google crawl their sites looking for AI content.
Vogel’s company has adopted Cloudflare’s solution to block AI crawlers that don’t pay, and he said contract negotiations are underway with several large LLM providers and insisted the system is working.
Source link
