Cloudflare blocks AI crawlers to support content creators

Cloudflare has implemented a new default setting to block AI crawlers from accessing website content without explicit permission or compensation, making it the first internet infrastructure provider to do so.

With this change, every new customer and domain on Cloudflare's platform will start with a setting that blocks AI crawlers by default, shifting the responsibility to AI companies to request access and clarify the crawler's intended purpose, such as training, inference, or search. This new approach replaces the previous opt-out system with an opt-in model, giving more power to content creators and publishers over the use of their work.

Cloudflare is also developing a feature called "Pay Per Crawl," which would allow content creators to request payment from AI companies seeking to use their content, thereby creating potential new revenue streams. This move addresses concerns about AI companies scraping web content without consent or compensation—a practice that many publishers and stakeholders argue threatens the future economic sustainability of the internet.

Shifting value in online content

The longstanding model of the internet has been based on a cycle in which search engines index web content, drive traffic to original websites, and provide revenue to creators through advertising. However, according to Cloudflare, the growing use of AI crawlers that extract information for large language models and other generative applications has disrupted this cycle by delivering answers without redirecting users to the original source. This change means creators may lose both the financial benefits and audience engagement historically generated by their work.

Matthew Prince, Cloudflare's Co-founder and CEO, commented,

"If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow's AI founders, and the future of the web itself. Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone."

This sentiment has been echoed by several publishers and content platforms. Roger Lynch, CEO of Condé Nast, stated,

"Cloudflare's innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything, they want for free, it opens the door to sustainable innovation built on permission and partnership. This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable."

Neil Vogel, CEO of Dotdash Meredith, remarked,

"We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements. We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web."

Renn Turiano, Chief Consumer and Product Officer at Gannett Media, noted,

"As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorised scraping and the use of our original content without fair compensation is critically important. As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP."

Other technology companies have also spoken in support of the new permission-based system. Bill Ready, CEO of Pinterest, stated,

"Creators and publishers around the world leverage Pinterest to expand their businesses, reach new audiences and directly measure their success. As AI continues to reshape the digital landscape, we are committed to building a healthy Internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive."

Steve Huffman, Reddit's Co-founder and CEO, pointed out,

"AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what. The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare's efforts are a step in the right direction for everyone."

Vivek Shah, CEO of Ziff Davis, added,

"We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders — the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination."

Default enforcement

Prior to this change, Cloudflare had already offered a one-click option to block AI crawlers since mid-2024. Over a million customers have enabled this option. With the latest move, every new website signing up to Cloudflare will be prompted to decide whether to allow or deny AI crawler access, streamlining the decision process and ensuring the default favours content owner control.

Industry support

More than 30 publishers, media, and technology companies have voiced their support for the new permission-based crawling model, including ADWEEK, The Associated Press, TIME, The Atlantic, Reddit, Pinterest, Quora, Sky News Group, and Universal Music Group, among others.

Will Lee, CEO of ADWEEK, stated,

"As the front page and homepage for marketing, advertising and media industry leaders, ADWEEK's position has been clear that we must be compensated for our investment grade journalism and information. I am thrilled Cloudflare has created a marketplace and mechanism that will enable us to properly participate in the promise LLMs have for our industry."

Paul Edmondson, CEO of The Arena Group, said,

"We think of our writers and content creators as entrepreneurs. Their work deserves protection. By blocking unauthorized AI crawlers, Cloudflare is not just defending content – it's defending the future of creators and storytellers. This is a vital move toward a digital economy built on trust, permission and fair value."

Additional key supporters include BuzzFeed, PMC, Quora, Stack Overflow, News/Media Alliance, and Webflow, all of whom have commented on the significance of the move for the digital economy and the rights of creators.

Technical implementation

Cloudflare has also put forward new ways for AI bots to authenticate themselves and for webmasters to identify them, including participating in the development of industry protocols for bot identification and authentication. These mechanisms aim to increase transparency in the operation of AI crawlers, allowing website owners to make more informed choices about access to their content.

AI companies are now required to obtain clear, explicit permission from websites prior to scraping content for AI training or generation purposes. Existing and future customers of Cloudflare can review and modify their crawler settings as required.