Crazy amount of guests

[Alvin63]

Although I'm not sure a referer would always work in this situation as it would with a uri path when they have already been "referred" from another page on the site or possibly from google. So quite a few genuine guests could get challenged. And my priority has been to have no genuine guests or users getting challenges and keep the site normal and functional for genuine guests. So I think I'm going to pause the rule for now.

The difference the rule made so far was to greatly reduce the CSR rate for my final rule 5 country challenge. However it didn't reduce the CSR rate for my uri path rule 4 - ie the bots can still solve the challenge and get into the guest list and scrape. So the ones going directly to Whats new are still solving the challenge, the bots hitting different paths (not many of those) go directly to rule 5 and are stopped there. But it's rule 4 where they are getting through - they solve the challenges.

Basically I'd rather block a few countries that are unlikely to come to my site, than have any genuine guests challenged. If that's what it comes to and hopefully it's not a permanent thing.

 

[Anthony Parsons]

Referrer is the one thing they fake most commonly. Search this thread about that.

Actions

Learn about actions supported by the Rules language, including Block, Skip, and Log.

developers.cloudflare.com

Non-Interactive Challenge:
Useful for ensuring that bots and spam cannot access the requested resource; browsers, however, are free to satisfy the challenge automatically. The client that made the request must pass a non-interactive Cloudflare challenge before proceeding. If successful, Cloudflare accepts the matched request; otherwise, it is blocked.

Everyone here is trying to stop automation causing havoc within their website, and that is what non-interactive does well without ever showing a real person a box to click. This is turnstile without the box. You allow cf.known.bots to bypass, you allow logged in users to bypass, ie. POST, everything else gets blocked at DNS.

You should use what you feel comfortable using, that is my opinion. I am just outlining what I use and my recommendation. Each to their own IMHO.

 

[Alvin63]

Referrer is the one thing they fake most commonly. Search this thread about that.

Actions

Learn about actions supported by the Rules language, including Block, Skip, and Log.

developers.cloudflare.com

Non-Interactive Challenge:
Useful for ensuring that bots and spam cannot access the requested resource; browsers, however, are free to satisfy the challenge automatically. The client that made the request must pass a non-interactive Cloudflare challenge before proceeding. If successful, Cloudflare accepts the matched request; otherwise, it is blocked.

Everyone here is trying to stop automation causing havoc within their website, and that is what non-interactive does well without ever showing a real person a box to click. This is turnstile without the box. You allow cf.known.bots to bypass, you allow logged in users to bypass, ie. POST, everything else gets blocked at DNS.

You should use what you feel comfortable using, that is my opinion. I am just outlining what I use and my recommendation. Each to their own IMHO.

Thank you. That is really helpful. I think I need to work out a strategy going forwards - ie how far down do I want to get the bot numbers, versus accepting a higher level than usual that isn't so high as to cause issues. Rather than trying to beat all of them.

 

[smallwheels]

I think I need to work out a strategy going forwards - ie how far down do I want to get the bot numbers, versus accepting a higher level than usual that isn't so high as to cause issues. Rather than trying to beat all of them.

As said way earlier in this thread: There are two main reasons why one could dislike bots: They steal your content and they may influence the performance/response time of the forum. Possibly no one likes the latter. The effects of the first one come way later with a delay: If the content of your forum (including personal details of your members that they may have posted) ends up in AI-models there is no need to visit your forum any more plus the privacy of your users is doomed. To be fair: There are also people that consider public what is posted and believe that w/o being present with content in AI models your forum stops do be relevant and to be found. Depending from what you believe you may have two main goals for dealing with bots:

1. avoid that they influence the performance
2. avoid that they steal your content

As always, there are many ways to Rome and which one you should take heavily depends how important which goal is for you and also, which level you want to achieve in respect to each goal and what you are willing to sacrifice to reach your goal(s). I.e. @Anthony Parsons ist only interested in the performance topic while I rank the avoidance of content theft equally high. This is why we come to different strategies. Performance can be tackled via server resources, software tuning, firewalls (inc. Anubis and Fail2Ban), various XF add ons and things like Cloudflare. Of those, Cloudflare is the easiest and cheapest way (needs the least knowledge and the smallest time investment).
The easiest and fastest way against content theft would be to lock the forum so that only registered users can see the content. This would work, but probably be too much of a price as discoverability is limited. So I ended up limiting public visibility to the first post of a thread and hid quite a bunch of forums and other content from public view. Which btw. has let my registrations triple. Furthermore I am blocking pretty radical as one successful scraping of exclusive content that has taken years to create and then ends up in an AI creates a harm that cannot be undone. The value of my forum aside of the conversation aspect lies in the amount and quality of structured content within and aside of the discussions that cannot be found elsewhere - however, that is a special situation, that may or may not be different in other forums. However: "A little content theft" goes in the direction of "a little pregnant" regarding the possible outcome. Wich means: I try to beat as many of the scrapers as possible and invested lots of time - I can't beat every scraper, but I can limit the possible damage. Completely different strategy from "only" to care about the performance of the forum. The tools used may be identical to a degree, how you deal with them differs.

 

[smallwheels]

everything else gets blocked at DNS.

No. It would be helpful in a technical forum to keep the terms correct. Whenever a client hits Cloudflare DNS resolution has already happened. Coudflare combines DNS, CDN and WAF and the blocking happens on the WAF. A DNS is like a phonebook, the CDN is a callcenter and the WAF is the ability of the call center staff to only accept calls that are legitimate while discarding anything else and to only forward calls to you that the call center staff cannot answer with the content you gave them. As with a real call center, the staff may sometimes fail.

 

[puterfixer]

There are also people that consider public what is posted

Yes and no. Without going too deep into the legalese of copyright:

• Forum members posting stuff are the copyright holders of their contents from the second they create a message; by submitting that message to your forum, they also grant the forum a free and unlimited license to process, store, modify and redistribute that message - without that you'd have zero legal ground to store that message in the database and to make it available to others to read;
• You as forum administrator can decide on your own the conditions and limitations for licensing access to this copyrighted content to forum members and visitors, including the permissions to use the content for any other purpose than reading it on their screen in their browser with their own two eyes; you can explicitly forbid saving that contents elsewhere, republishing it, or using it for AI training, to name a few. For this you must have a very explicit copyright notice published. This gives you legal ground for fighting bots etc. and to enforce those boundaries through technical means.

Publishing a copyright notice won't stop all the idiots who think that, if it's visible on the internet, it's free to take. But it clarifies a legal position and fundaments the defensive actions.

 

[Alvin63]

As said way earlier in this thread: There are two main reasons why one could dislike bots: They steal your content and they may influence the performance/response time of the forum. Possibly no one likes the latter. The effects of the first one come way later with a delay: If the content of your forum (including personal details of your members that they may have posted) ends up in AI-models there is no need to visit your forum any more plus the privacy of your users is doomed. To be fair: There are also people that consider public what is posted and believe that w/o being present with content in AI models your forum stops do be relevant and to be found. Depending from what you believe you may have two goals for dealing with bots:

1. avoid that they influence the performance
2. avoid that they steal your content

As always, there are many ways to Rome and which one you should take heavily depends how important which goal is for you and also, which level you want to achieve and what your are willing to sacrifice to reach you goal(s). I.e. @Anthony Parsons ist only interested in the performance topic while I rank the avoidance of content theft equally high. This is why we come to different strategies. Performance can be tackled via server resources, software tuning and things like Cloudflare. Of those, Cloudflare is the easiest and cheapest way (needs the least knowledge and the smallest time investment).
The easiest and fastest way against content theft would be to lock the forum so that only registered users can see the content. This would work, but probably be too much of a price as discoverability is limited. So I ended up limiting public visibility to the first post of a thread and hid quite a bunch of forums and other content from public view. Which btw. has let my registrations triple. Furthermore I am blocking pretty radical as one successful scraping of exclusive content that has taken years to create and then ends up in an AI creates a harm that cannot be undone. The value of my forum aside of the conversation aspect lies in the amount and quality of structured content within and aside of the discussions that cannot be found elsewhere - however, that is a special situation, that may or may not be different in other forums. However: "A little content theft" goes in the direction of "a little pregnant" regarding the possible outcome. Wich means: I try to beat as many of the scrapers as possible and invested lots of time - I can't beat every scraper, but I can limit the possible damage. Completely different strategy from "only" to care about the performance of the forum. The tools used may be identical to a degree, how you deal with them differs.

I think @Anthony Parsons was just trying to help me find an alternative to country blocking and it's a good alternative. I'm just getting my head around the various Cloudflare challenges in respect to the rule.

There are also some useful suggestions there about limiting what guests can see at forum level that I could consider also - I haven't done a lot on that side. I do have a limited guest viewing addon, which leads to a request to register, and does get a lot of registrations - so prior to the limited views the forum is fairly open. And of course the limited guest addon won't work for bots. So what worked before, isn't necessarily working now. A lot to think about.

 

[smallwheels]

And of course the limited guest addon won't work for bots.

Why wouldn't it? I am using this one by @AndyB:

First post only

Dec 18, 2017

Show first post only to guests.

• AndyB
• Add-ons [2.x]

It does the job and nothing to complain about. However: It may influence your Google ranking as obviously search engine bots also only see the first post. As Google is the only search engine relevant for me and does weirdo things regarding ranking constantly anyway I decided not to bother. My ranking may have gotten formally worse but registrations tripled, so nothing to complain. There is also a "plus" version of the add on, the difference seems to be the possibility for setting a permission (which I don't need):

First post only plus

Sep 11, 2020

Show first post only. Requires account upgrade to view replies in selected forums.

• AndyB
• Add-ons [2.x]

 

[Alvin63]

Why wouldn't it? I am using this one by @AndyB:

First post only

Dec 18, 2017

Show first post only to guests.

• AndyB
• Add-ons [2.x]

It does the job and nothing to complain about. However: It may influence your Google ranking as obviously search engine bots also only see the first post. As Google is the only search engine relevant for me and does weirdo things regarding ranking constantly anyway I decided not to bother. My ranking may have gotten formally worse but registrations tripled, so nothing to complain. There is also a "plus" version of the add on, the difference seems to be the possibility for setting a permission (which I don't need):

First post only plus

Sep 11, 2020

Show first post only. Requires account upgrade to view replies in selected forums.

• AndyB
• Add-ons [2.x]

Well my google rankings are important so I'm not sure about those As for the limited guest viewing - it clearly doesn't stop bots as they are going direct to /whats-new/posts where I have a rule/challenge set up.

 

[smallwheels]

Publishing a copyright notice won't stop all the idiots who think that, if it's visible on the internet, it's free to take. But it clarifies a legal position and fundaments the defensive actions.

That's why I have one (and also a horrendous fee payed out in ai.txt, lmms.txt and robots.txt for the case one ignores the disallowance of scraping and using the forum content in an AI or in any way commercially). Clearly, the bots don't bother and neither do AI companies, but in case there will ever be action I will be in a better position (hopefully).

However: My statement regarding "public content is public anyway" was targeting more some forum owners that have that position (which is fine, mine is however different).

 

[smallwheels]

As for the limited guest viewing - it clearly doesn't stop bots as they are going direct to /whats-new/posts where I have a rule/challenge set up.

It is intended to protect the content and rise registration, not to optimize for performance. As soon as robots would open a thread linked in "what's new" they would see no more than the first post. Can't blame a tool for doing what it is intended to do and not doing, what it was never intended to do. ;-)

 

[Alvin63]

So the rule @Anthony Parsons - I would like to re-enable it and use the non interactive challenge, having looked into it - but have been weighing up the pros and cons of potentially leaving a member stuck on a cloudflare page if they had an old browser or ad blocker etc, versus blocking a few countries, four of whom I would rather not block. So what I'm thinking is to possibly have your rule but with a whitelist for Uk, Europe, and USA, so visitors from those countries bypass the rule. Now I know that means more bots would get past the rule but there are still two other rules below that and the vast majority are from other countries than those.

Edit - I have done that set it to non-interactive challenge and - added this - and haven't added a referer. For certain European countries, US, Canada and Uk. Analysis showed about a third of them would get past the rule if these were whitelisted - time will tell.

and (not ip.src.country in {"GB" "IE" "FR" "DE" "NL" "BE" "LU" "CH" "AT" "LI" "MC" "ES" "PT" "AD" "DK" "SE" "NO" "FI" "IS" "IT" "GR" "CY" "SM" "VA" "PL" "CZ" "TR" "US" "CA"})

It does mean that any genuine guests from countries that aren't whitelisted could get challenged twice - once for this rule and then also for my country challenge rule ......... still needs some thinking about.

 

[Alvin63]

So rule orders for Guests not from blocked countries or whitelisted countries in rule 3:

Rules 1 and 2 Country block and ASN block

3) Fake browser rule - Non Interactive challenge - pass to forum page and get a referer cookie
4) Whats-new, search and members with referer - Interactive challenge - they bypass this because they have a referer cookie
5) Country challenge list - Interactive challenge

So - if they bypass rule 4 because they have a referer - do they stop there or go straight to rule 5 and get an Interactive challenge? I think they would because a Non interactive challenge first is a lower permission than an Interactive challenge.

So they could potentially get challenged twice before even accessing the forum - because would the rule 5 challenge kick in before they even got to a forum page to get the referer?

I guess I would either have to make rule 3 Interactive or Rule 5 non interactive to match rule 3.

 

[puterfixer]

You are overthinking these rules and contradicting yourself, creating rules from exceptional cases that undermine the entire principles of having defensive rules in the first place. Don't pamper users so much, they are responsible for updating their software and there's nothing wrong in asking them to put a little effort to fulfill their wishes of connecting to a forum. They are (or should be) fully functioning adults who should be capable of meeting you halfway. Your responsibity is with the big picture, not with rolling out the red carpet and handholding a doofus who still has Windows 95 and a Chrome from 20 years ago, set up not to auto update. The normality of life is continuous change, and those who can't keep up will have to learn what "survival of the fittest" means.

 

[Alvin63]

You are overthinking these rules and contradicting yourself, creating rules from exceptional cases that undermine the entire principles of having defensive rules in the first place. Don't pamper users so much, they are responsible for updating their software and there's nothing wrong in asking them to put a little effort to fulfill their wishes of connecting to a forum. They are (or should be) fully functioning adults who should be capable of meeting you halfway. Your responsibity is with the big picture, not with rolling out the red carpet and handholding a doofus who still has Windows 95 and a Chrome from 20 years ago, set up not to auto update. The normality of life is continuous change, and those who can't keep up will have to learn what "survival of the fittest" means.

Good point! I was thinking more of adblockers and script checkers than doofus's I was just trying to work out if my existing rules need changing now I've added the new rule three for fake browsers.

 

[Alvin63]

Anyway doesn't matter if some countries outside my usual membership get two challenges and I've left everything as is. Main thing is my bot numbers have gone up again.

So the new fake browser rule is catching a lot, but is effectively just reducing the number that get to the next rule 4 - the whats new, search, members path challenge and the solve rate for rule 4 is still 16% - no difference. From the json though, all the bots hitting rule 4 have full chrome, up to date browsers including the latest version. And 16% of them also seem to solve the challenge.

So nothing really has changed - except the guest list is back up 1800 and a whole load of new obscure countries. It went right up to 2,400 again. Adjusted Rule 5 country challenge to include most of the world, whitelisting UK, USA and some of Europe, as the new countries were bypassing it and numbers went back to 500 ish.

I finally found precursor in settings and turned it on. However, on the free plan I think it just monitors and no rules allowed to be set.

Also turned on AI labyrinth and will see if either of those gets the sophisticated bots that solve challenges.

 

[Suzanne O]

Just got an email from cloudflare regarding this thread.

 

[Wildcat Media]

Honestly, some of these rules I've tried and few have made any difference. But, things seem to be tamed for us about as good as they can be for now, so I'm not complaining.

 

[Wildcat Media]

The latest from Cloudflare regarding the mixed-use bots, from today's email:

Have it both ways: stay discoverable in search while disallowing AI training

Cloudflare is giving site owners a way to stay discoverable while disallowing AI training. New controls and an Accountable designation establish a shared model with Apple, Google, and Microsoft.

blog.cloudflare.com

I am blocking as many CF confirmed bot categories as I can right now, so these new settings will help me fine tune my rules a bit more. In fact, I was about to update my robots.txt with a lot of these changes and turns out, CF automatically added what I wanted, and then some. No it doesn't cover the rogue bots, but these will tame the confirmed bots somewhat.

A sample of the settings:

 

[Anthony Parsons]

if they bypass rule 4 because they have a referer - do they stop there or go straight to rule 5

Rules are in order. If rule 1 is to block bad bots, so a bad bot from china gets caught, and rule 2 is to block china, the bad bot never gets to rule 2, as rule 1 was triggered for it. Now a real person from china, is not a bad bot so they skip rule 1 but get caught in rule 2, where you are blocking everyone from china. If you're going to block a country, put them first, that way they work outright and immediately. Your other rules go after blocking, that way you aren't relying on further rules to fix any issues you block in previous rules, ie. if you block china, then you don't need anything later to block a china bot, china is blocked. Done.

Now China can be calculating and shift their bots through Hong Kong and residential proxies... but that is a China thing specifically, so whilst not the best example, the above is just an example of rule ordering.

You are overthinking these rules and contradicting yourself, creating rules from exceptional cases that undermine the entire principles of having defensive rules in the first place.

@Alvin63 you really need to take the above quote from @puterfixer and apply it. You're new to this it seems, and getting swamped with choices, chopping and changing things near daily.

Identify the biggest problem, tackle that first and foremost.

Then measure that action over a full 24-48hrs.

You aren't going to stop everything bad ever hitting your website. The biggest issue is typically automation, and the worst is automation hitting anything /search, as that is load bearing to your server far more than page loading with caching.

As you are using cloudflare, you only really need to do two things initially, shared hosting or not.

1. Download @digitalpoint cloudflare addon, enable guest page caching for a short duration, ensure it is all working correctly. (AI can help you confirm it is working) Working guest caching will remove about 60% server resources for guest viewing. Work that out for your sites traffic, duration wise.

2. One WAF rule to protect /search

Those two things protect your server resources the most. Then you tackle all the nasty stuff hitting your server.

At the end of the day, shared hosting, anything hitting your IP directly becomes your hosts problem, not yours. If they aren't interested in protecting their clients from attacks that their setup hinders a client from handling themselves, that is a them problem, not a you problem. Possibly becoming one of the most important questions to ask hosting before signing up, what protections do they have in place for bad bot behaviour hitting client IP's.

Yes, unfortunately, your users suffer that negligence. This is why a VPS is really required, and it sucks IMO that is needed today to control this issue where hosts aren't doing the job for their own hardware. Well... some are, some aren't.