Crazy amount of guests

[Mike Fara]

There's a few options but check out https://iplists.firehol.org/ which can help you integrate blocklists into a custom solution. Almost always what it is just scrapers and people training AI models. Like previous people indicated, Cloudflare is very good for handling automated traffic, but there are some things you can do manually if you have your own VPS/root access.

 

[smallwheels]

One common trend we've seen so far is that all the problems are members using their iphones on safari. Nobody else has a problem.

Probably these are then using Apple's privacy relay which is basically a VPN, but limited to Safari on Apple devices. I see this getting used by my users a lot as well. IP Threat monitor has an option to let those pass and - as the IP-lists are publicly available - it should be possible to integrate something like that in CF, too.

 

[Anthony Parsons]

I think one of the biggest issues not being discussed here is, how much traffic does AI send a site nowadays? If you block the known entities from providing your site as a recommendation because it can't access it, then are you losing out?

Just putting this out there, which is why I have all AI bots on CF as allow, because a lot of people are now turning to AI in place of Google. I get better answers asking Claude or ChatGPT than I often do asking Google. You ask them a specific question and there is no nonsense listings in return, just what fits your question to them.

 

[lazy llama]

I think one of the biggest issues not being discussed here is, how much traffic does AI send a site nowadays? If you block the known entities from providing your site as a recommendation because it can't access it, then are you losing out?

Cloudflare gather stats on this, crawl-to-refer ratios, here are their stats from the past week:



So Anthropic and OpenAI are pretty horrendous.....

 

[smallwheels]

I think one of the biggest issues not being discussed here is, how much traffic does AI send a site nowadays? If you block the known entities from providing your site as a recommendation because it can't access it, then are you losing out?

Just putting this out there, which is why I have all AI bots on CF as allow, because a lot of people are now turning to AI in place of Google. I get better answers asking Claude or ChatGPT than I often do asking Google. You ask them a specific question and there is no nonsense listings in return, just what fits your question to them.

Surely, what the best strategy is depends from how exclusive and how high quality the content of your forum is and how eager you are for growth or visitors. In general AI providers grab your content and give nothing back in exchange. Some are worse than others but overall you will lose: If AI has grabbed your content and serves it there is no need to visit your forum for people asking questions, even if there is a backlink.

Personally I do not care too much about new users and I clearly want to protect my forum content as it is pretty high quality and a lot of it is exclusive and cannot be found anywhere. How silly would it be to give that advantage away and even more for free? Plus enabling the AIs to grab all kinds of personal information that forum users may post with all potentially negative effects this may have.

So I do block scrapers and most AI agents. Depending from the AI and how well structured it's bots are it is sometimes possible to let the searh bot through while blocking the training bot. When in doubt I rather block completely. Furthermore I've set hard limitations to visibility for guests: They have always not been able to see some parts of the forum but most of it was freely accessible. I've changed this a while ago and now as a guest you can only see the first post of a thread and on top of that there are even more areas of the forum not accessible to guests.

Until now this has tremendously fostered registrations and at had least in the first months no negative impact on search engine ranking. I did not check this recently because I don't care too much. I do have a working community, I gain new users - what else could I want?

Surely, running a non-commercial forum helps, but I am part of the "lock them out as good as possible" cohort.