Open AI ignored my robots.txt and then this happened

[Digital Doctor]

https://www.reddit.com/r/interestingas****/s/DCGY99YXLi

Karma.

https://www.reddit.com/r/interestingas****/s/DCGY99YXLi

**** = for unlawful carnal knowledge

 

[ekool]

It looks like the post was deleted

 

[Wildcat Media]

I've been thinking for years robots.txt is actually quite useless.

Doesn't Cloudflare have a feature to discourage or block AI bots?

 

[ekool]

I've been thinking for years robots.txt is actually quite useless.

Doesn't Cloudflare have a feature to discourage or block AI bots?

It does. Robots.txt definitely won't work for AI stuff. They are ignoring it and changing their strings constantly to avoid being blocked. It should actually be illegal.

 

[Wildcat Media]

I don't trust robots.txt after I had a site get systematically removed from Google over the course of three months...because I had a crawl-delay directive in it. Which had been there for a decade (where they ignored it anyway). Removed it, and the pages slowly came back.

I do have AI bot blocking active on Cloudflare. I wish there was a way some sites with a lot of data would "poison" all the AI bots with false information, to the point that their services become useless. I'm tired of having "AI" shoved in my face with every application I open now--it's not even "intelligence." It's just an updated algorithm.

 

[Suzanne O]

I'll never use AI for my forum. Never.

 

[ekool]

I don't trust robots.txt after I had a site get systematically removed from Google over the course of three months...because I had a crawl-delay directive in it. Which had been there for a decade (where they ignored it anyway). Removed it, and the pages slowly came back.

I do have AI bot blocking active on Cloudflare. I wish there was a way some sites with a lot of data would "poison" all the AI bots with false information, to the point that their services become useless. I'm tired of having "AI" shoved in my face with every application I open now--it's not even "intelligence." It's just an updated algorithm.

There are a ton of AI poisoning projects going, so don't fret. I found a good thread in one of the posts here: https://news.ycombinator.com/

With a list of half a dozen. I can't find the post now but you are welcome to search... there are a bunch out there, so fret not.

 

[bzcomputers]

I use a combination of robots.txt and ai.txt.

Spawning

Data Governance for Generative AI

spawning.ai

How to Prevent AI from Mining Your Site's Data - Abhishek Deshpande

Do you want to share your website's public content with AI robots for Generative AI or other Artificial intelligence for AI Training? If No, then this article

whoisabhi.com

 

[FTL]

It does. Robots.txt definitely won't work for AI stuff. They are ignoring it and changing their strings constantly to avoid being blocked. It should actually be illegal.

I'm with you there. I hate underhand stuff like that.