https://www.reddit.com/r/interestingas****/s/DCGY99YXLi
Karma.
Code:
https://www.reddit.com/r/interestingas****/s/DCGY99YXLi**** = for unlawful carnal knowledge
Open AI ignored my robots.txt and then this happened
- Thread starter Digital Doctor
- Start date
Karma.
**** = for unlawful carnal knowledge
I've been thinking for years robots.txt is actually quite useless.
Doesn't Cloudflare have a feature to discourage or block AI bots?
Doesn't Cloudflare have a feature to discourage or block AI bots?
It does. Robots.txt definitely won't work for AI stuff. They are ignoring it and changing their strings constantly to avoid being blocked. It should actually be illegal.
I don't trust robots.txt after I had a site get systematically removed from Google over the course of three months...because I had a crawl-delay directive in it. Which had been there for a decade (where they ignored it anyway). Removed it, and the pages slowly came back.
I do have AI bot blocking active on Cloudflare. I wish there was a way some sites with a lot of data would "poison" all the AI bots with false information, to the point that their services become useless. I'm tired of having "AI" shoved in my face with every application I open now--it's not even "intelligence." It's just an updated algorithm.
I do have AI bot blocking active on Cloudflare. I wish there was a way some sites with a lot of data would "poison" all the AI bots with false information, to the point that their services become useless. I'm tired of having "AI" shoved in my face with every application I open now--it's not even "intelligence." It's just an updated algorithm.
There are a ton of AI poisoning projects going, so don't fret. I found a good thread in one of the posts here: https://news.ycombinator.com/
With a list of half a dozen. I can't find the post now but you are welcome to search... there are a bunch out there, so fret not.
I use a combination of robots.txt and ai.txt.
How to Prevent AI from Mining Your Site's Data - Abhishek Deshpande
Do you want to share your website's public content with AI robots for Generative AI or other Artificial intelligence for AI Training? If No, then this article
whoisabhi.com
I'm with you there. I hate underhand stuff like that.
