Cloudflare "block ai trainers" rule is blocking Googlebot

[kontrabass]

Thought I'd share this here as I love my fellow community managers. If you run Cloudflare with "block AI trainers", check your Google search console now. Our indexed URL's went from over 1MM to less than 1,000, we are effectively wiped from their index.

Report That Cloudflare AI Bot Blocking Prevents Googlebot From Indexing Sites

Anecdotal report claims that blocking AI training bots with Cloudflare prevents Googlebot from indexing websites.

www.searchenginejournal.com

 

[rdn]

Do you mean this option?

 

[Rhody]

Theyre still getting through to mine, with cloudflare. The settings for that are a bit confusing, and I believe it is changing again next month.

 

[GCT2007]

I ****ed this up on my end and it destroyed my SEO. Fixed it but now I'm having to crawl back.

 

[smallwheels]

If you run Cloudflare with "block AI trainers", check your Google search console now. Our indexed URL's went from over 1MM to less than 1,000, we are effectively wiped from their index.

Technically Cloudflare is correct here as Google does use their crawl results for both: Search as well as AI training. As Google does not have separate bots for those purposes and does also not offer website owners to opt out of AI training CF treating the Google Bot as AI training bot is correct. Google has ditched "don't be evil" years ago and is now effectively blackmailing website owners to tolerate the use of their content for AI training if they want to be listed in Google search.

Personally I think CF's step is good as they are big enough with their customer base to break Google's power. If Cloudflare customers to leave this setting and Google does not change it's policy this will lead to Google Search becoming way less userful and this is not what Google should aim at.

 

[kontrabass]

Technically Cloudflare is correct here as Google does use their crawl results for both: Search as well as AI training. As Google does not have separate bots for those purposes and does also not offer website owners to opt out of AI training CF treating the Google Bot as AI training bot is correct. Google has ditched "don't be evil" years ago and is now effectively blackmailing website owners to tolerate the use of their content for AI training if they want to be listed in Google search.

Personally I think CF's step is good as they are big enough with their customer base to break Google's power. If Cloudflare customers to leave this setting and Google does not change it's policy this will lead to Google Search becoming way less userful and this is not what Google should aim at.

This next step was not supposed to go into effect until next month. Our search presence was being wiped out starting August 10th. I guess we’re the lucky ones to have an early rollout.

I’m afraid everyone needs Google, and are going to allow training. There’s no other option.

We also have a nice new feature in CF though - “precursor”. Anyone else seeing this in their waf security settings? Looks very promising to stop sophisticated bots.

 

[Mr Lucky]

Does anyone know if GSC notifies that this is happening, ie when Googlebot is blocked by CF?

 

[smallwheels]

I’m afraid everyone needs Google, and are going to allow training. There’s no other option.

I don't think so. Google has always been and is still free to play fair and not to combine search indexing with AI training. Apple only has one bot for both as well but (if I remember correctly) if you set "User-agent: Applebot-Extended" to disallowed in robots.txt they will use the crawl results for search but not for AI training and it has been like that for years.

Somewhat astonishing that people jump through loops to avoid their content to be grabbed by AI crawlers but are absolutely willing to bend over for Google doing the same thing. Google are misusing their market position badly. Their search is dependent from having a comprehensive index of the web. If people block the combined Google bot they will have to change their behavior rather sooner than later as it will massively and quickly influence their reputation and the quality of search results. It is not that website owners are powerless and dependent from Google - Google is dependent from website oner cooperating even more b/c w/o that their business model is toast and - even worse - with search results becoming incomplete and unreliable people will turn to other search engines.

So in the end it is a game of power and Cloudflare gives the huge amount of powerless website owners the opportunity to align their power against Googles behavior. Obviously, if you bend over for Google Google will win.

 

[kontrabass]

Does anyone know if GSC notifies that this is happening, ie when Googlebot is blocked by CF?

I received no notification, but I’m not sure if GSC has some notification options I wasn’t using. I wish I had notification rather than finding out manually after a couple weeks of declining ad revenue. Our GA4 is so overloaded with bots I don’t get any reliable info from that.