Cloudflare to block AI bots from scraping

P

Paul B

XenForo moderator
Staff member
Internet firm Cloudflare will start blocking artificial intelligence crawlers from accessing content without website owners’ permission or compensation by default, in a move that could significantly impact AI developers’ ability to train their models.

Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow AI crawlers, effectively giving them the ability to prevent bots from scraping data from their websites.
Click to expand...

 
Blocking AI scraping might protect your data in the short term, but it’s a losing battle unless you’re a truly unique source for the data. Most content on the internet isn’t that exclusive. Your competitors who allow scraping will dominate AI-driven responses which are the first responses on nearly all search results now. Those responses, even with attribution links, may not drive the same traffic as organic search results once did, but they still offer some exposure. Blocking AI entirely risks being left out of the conversation altogether, as users increasingly rely on AI summaries over direct site visits.

The trade-off is real: allowing scraping means your content fuels AI models, potentially reducing direct traffic, but blocking it could make you invisible in AI-driven search ecosystems which right now are growing fast.

It also wouldn't surprise me that Google's algorithms might favor AI-friendly sites in the additional non-AI search results.

Until there is a a more universal service in place for sites to gain financial benefit from your scraped data, I do think it is best to still let AI in. You don't want to be the site left out. Sites like Reddit, X already have financial agreements in place. Cloudflare is launching an experiment today for just this, it's called "Pay per Crawl". Website owners in the experiment can choose to let AI crawlers, on an individual basis, scrape their site at a set rate, a micropayment for every single “crawl", or allow them in for free, or block them altogether. It will be interesting to see where this leads.
 
 
You must log in or register to reply here.

Similar threads

tribedude
Blocking AI bots from scraping your database
Replies
3
Views
548
digitalpoint
digitalpoint
Back
Top Bottom