Blocking AI bots from scraping your database

[tribedude]

I am using various robots.txt lines to block specific bots but I think some are ignoring that.

Google has just informed me that our domain won't get indexed unless we allow their AI training bots to roam free in the domain. Other Canadian publishers have also received this notification. We have no access to Google AI products in Canada, so this is doubly sketchy, I think.

I am thinking of taking the board completely private.

What are the best ways to stop AI bots from scraping your content? Any tips or suggestions?

Thanks and happy holidays everyone.

 

[Suzanne O]

Via your hosts cpanel there is a good way of dealing with it.
Cloudflare turnstile is good too

 

[MentaL]

Maybe incorperate some traffic throttling that way its not blocked but limited.

 

[digitalpoint]

Via your hosts cpanel there is a good way of dealing with it.
Cloudflare turnstile is good too

You can also flat out block traffic before it gets to your origin server based on whatever you want (IP, country, user agent, etc.)

I use a Cloudflare rule to block spiders that don't respect robots.txt (see this post).