Crazy amount of guests

[smallwheels]

I don't know how it allows search engines through, or which ones, that will be in the developer's code, which gets updated now and then.

Which basically means you do not know how well and how reliable it works and if at all. I'd expect that it should be explained in the documentation of the add on and ideally be configurable in the backend. In IP Threat Monitor it looks i.e. like that:



Pretty brilliant, if you ask me.

 

[Alvin63]

Bots will fake known search bots anyway no doubt. Just found one today in the Robots list. Robot Google Inspection tool - but the IP address was Cox Communications in Nevada.

 

[smallwheels]

Bots will fake known search bots anyway no doubt. Just found one today in the Robots list. Robot Google Inspection tool - but the IP address was Cox Communications in Nevada.

The point is: With proper detection via RDNS and IP-ranges this would not come through. With primitive relying on the easy to fake user agent it will. That is the whole point. But anyway a view limiter is not designed to defend against bots, it is rather a tool to rise registrations. It can block bots that circumvent other protection mechanisms from scraping loads, but the add on you use probably not so much as well.

 

[Wildcat Media]

I don't know if I'd ever use this, but theoretically...

(lower(http.user_agent) contains "google" and not ip.src.asnum in {15169})

...with an interactive challenge (requires human intervention to solve) or a block could filter out bots faking the "google" user agents. One could add other ASNs and their user agent strings (Bing, etc.) to filter out more fake bots.

EDIT: Keep in mind that cf.bot_management.verified_bot and cf.client.bot both check that the verified bots are coming from "official" servers.

I wish we could set up a rule like this for logging purposes only, just to see out of curiosity how many of the bots are from Google (and others) and how many are faking the user agent string.

 

[Wildcat Media]

I noticed we can look at analytics beyond 24 hours now...up to 30 days.

The past three days for us--note this huge bots warm, and how it has calmed down since then. Note that I did nothing to the rules in the past week. That's a huge spike! Yet our origin didn't even notice.



In comparison, for scale, the past 24 hours. I hate to say this is getting boring, but...nothing is happening where I need to adjust any rules for a while. I haven't had a single notification from monit that our CPU% and load averages are above normal. But when I do get those notifications, that is usually an indicator that I need to revisit our CF security analytics and see if there are new swarms to deal with.

 

[smallwheels]

I hate to say this is getting boring, but...nothing is happening where I need to adjust any rules for a while. I haven't had a single notification from monit that our CPU% and load averages are above normal. But when I do get those notifications, that is usually an indicator that I need to revisit our CF security analytics and see if there are new swarms to deal with.

The numbers are requests I assume? With a ratio of 1,65 Million mitigated vs. 3,85 Million served I am wondering if not a relevant part of those served requests is still bots. Cannot judge but the percentage mitigated seems somewhat low, based on silly wild guessing.

 

[Wildcat Media]

Some of the bot traffic is from legitimate search engine scrapers, and a few other we allow.

Traffic is low enough that I haven't been concerned what the ratios are for mitigated/cached/origin. Lately we're riding at about 2x as many guests as there are logged-in members, which historically is about right where we'd been for many years before these AI bot swarms started. That's really where I see the improvement, along with ongoing server load averages not varying much throughout the day. (I started this silly log several years ago where I used uptime to write to a log file once per minute, just for a quick reference.)

I think the most sane way to go about this is to find the combination that keeps traffic to manageable limits, and not worry about swatting down every bot that comes along just because it shows up in the logs. But I will still admit that when I was first adjusting and creating these rules, it took me a while to get it right. And having to revise it when we started seeing the residential proxies.

I will also say that nobody else's rules in this thread absolutely changed our filtering, and some barely made a dent...but all sites and traffic are different. I incorporated many of the examples above with some of my own filtering, and this is where I'm at.

But based on those analytics charts I post, I almost get the feeling the bots are taking a nice deep breath, waiting to deliver their next onslaught...

 

[Alvin63]

Some of the bot traffic is from legitimate search engine scrapers, and a few other we allow.

Traffic is low enough that I haven't been concerned what the ratios are for mitigated/cached/origin. Lately we're riding at about 2x as many guests as there are logged-in members, which historically is about right where we'd been for many years before these AI bot swarms started. That's really where I see the improvement, along with ongoing server load averages not varying much throughout the day. (I started this silly log several years ago where I used uptime to write to a log file once per minute, just for a quick reference.)

I think the most sane way to go about this is to find the combination that keeps traffic to manageable limits, and not worry about swatting down every bot that comes along just because it shows up in the logs. But I will still admit that when I was first adjusting and creating these rules, it took me a while to get it right. And having to revise it when we started seeing the residential proxies.

I will also say that nobody else's rules in this thread absolutely changed our filtering, and some barely made a dent...but all sites and traffic are different. I incorporated many of the examples above with some of my own filtering, and this is where I'm at.

But based on those analytics charts I post, I almost get the feeling the bots are taking a nice deep breath, waiting to deliver their next onslaught...

I have that feeling too - it’s not nice feeling they control that. A big swarm one night and you wonder how high it will go - and then low numbers for a few days.

 

[puterfixer]

I think the most sane way to go about this is to find the combination that keeps traffic to manageable limits, and not worry about swatting down every bot that comes along just because it shows up in the logs.

"Don't let 'perfect' get in the way of 'good'."

 

[smallwheels]

I think the most sane way to go about this is to find the combination that keeps traffic to manageable limits, and not worry about swatting down every bot that comes along just because it shows up in the logs. But I will still admit that when I was first adjusting and creating these rules, it took me a while to get it right. And having to revise it when we started seeing the residential proxies.

I will also say that nobody else's rules in this thread absolutely changed our filtering, and some barely made a dent...but all sites and traffic are different. I incorporated many of the examples above with some of my own filtering, and this is where I'm at.

Absolutely. And so are the goals, possiblities and strategies regarding bot fighting. What has made this thread so valuable for many months was that we were all sharing the same problem (if to vastly different amounts), all starting more or less at the same level having to develop understanding and measures from scratch. Quite a bunch of people were highly motivated and exploring the issue from different perspectives, trying out different strategies and approaches, shared what they found and everything was openly discussed.

Sadly this has changed a bit as some now not only seem to have found "their" solution but have started to promote their solution as the only valid one and their goals as the only valid ones. New people have joined the thread w/o bothering to read and learn what the collective state of knowledge is and the thread has started to loop, discussing from scratch what has already been excessively discussed and solved months ago and snake oil is praised. So sadly, the usefulness of the thread has massively decreased in my eyes as has the discussion level.

For my part, server load has only one or two times been affected. Easy, as I have not in the slightest hit by waves of the size others have experienced. While it is impossible to hunt down every bot I still do see protection of my content as one main goal. As with you my setup has somewhat stabilized for the moment and adjustments have barely been necessary over the last months. But then it has been very calm over the last weeks - it has always been waves, but currently it feels rather suspiciously calm.

 

[Alvin63]

I think the thread is here to help people as well. It isn't just a case of reading but getting responses at particular times - which is what threads are. People come along at different stages and what works for one forum might not work for another.

 

[smallwheels]

I think the thread is here to help people as well. It isn't just a case of reading but getting responses at particular times

Sure. Fact is: The amount of new information has become less in this thread over the last weeks, partly b/c there is not too much new as the behavior of the bots has roughly been explored and is well known, they are calm at the moment and their behavior is somewhat predictable in general. To explore further would mean the ratio between work and time invested vs. immediate impactful outcome would be worse than in the beginning - normal - and it would need really sorrowful analysis and decent understanding. Most people seem more or less happy with the plateau we are currently on knowlegewise and not willing to invest more. Fair enough.

It is also normal that people join that are new to the topic. So it comes - as often in forums - down to the question: What can be expected that new people invest upfront? In general, better forums follow the guideline "tell something new", not something that has already endlessly been discussed in the very same thread. It is a question of respect towards others to read before you post. It is simply boring for all the others (and sometimes even annoying) when things that have been discussed and finally solved with agreement after long and sometimes painful discussions are again brought up as brand new in complete ignorance of the existing tree of knowledge in the very same thread. And it gets worse if this ignorance then leads to false claims and sometimes grotesque statements. I mean: This is a very technical thread, not a social chit-chat. And it had for the most part reached a very high level.

 

[Alvin63]

Bots are lower in the house tonight at around 500,000. However there are a lot less being mitigated by Cloudflare. The solve rates of bots on challenges is getting higher. My main rule, the solve rate is 28%. So the bots are winning again as more is being served by the server.

 

[Alvin63]

Sure. Fact is: The amount of new information has become less in this thread over the last weeks, partly b/c there is not too much new as the behavior of the bots has roughly been explored and is well known, they are calm at the moment and their behavior is somewhat predictable in general. To explore further would mean the ratio between work and time invested vs. immediate impactful outcome would be worse than in the beginning - normal - and it would need really sorrowful analysis and decent understanding. Most people seem more or less happy with the plateau we are currently on knowlegewise and not willing to invest more. Fair enough.

It is also normal that people join that are new to the topic. So it comes - as often in forums - down to the question: What can be expected that new people invest upfront? In general, better forums follow the guideline "tell something new", not something that has already endlessly been discussed in the very same thread. It is a question of respect towards others to read before you post. It is simply boring for all the others (and sometimes even annoying) when things that have been discussed and finally solved with agreement after long and sometimes painful discussions are again brought up as brand new in complete ignorance of the existing tree of knowledge in the very same thread. And it gets worse if this ignorance then leads to false claims and sometimes grotesque statements. I mean: This is a very technical thread, not a social chit-chat. And it had for the most part reached a very high level.

It didn't start out as a "very technical thread".

 

[Anthony Parsons]

I don't know if I'd ever use this, but theoretically...

My previous post does this, a better way IMO, using cf.client.bot: ((http.user_agent contains "Googlebot" or http.user_agent contains "bingbot" or http.user_agent contains "Applebot" or http.user_agent contains "DuckDuckBot" or http.user_agent contains "Google-InspectionTool" or http.user_agent contains "Mediapartners-Google" or http.user_agent contains "AdsBot-Google" or http.user_agent contains "YandexBot" or http.user_agent contains "Baiduspider") and (not cf.client.bot))

 

[puterfixer]

the usefulness of the thread has massively decreased in my eyes as has the discussion level

On the other hand, thanks to this thread, we learned that:

• we are not alone in facing a massive, disruptive force that's suffocating online communities - there are supportive forum admins that drive a positive discussion forward, bringing in their wealth of expertise and best practices
• there are multiple ways to address the problem, and each can find whatever combination of measures works for them
• there are technical mitigation measures that can be implemented, with varying degrees of complexity, cost and learning curves - fortunately including some free and easy ways for us small fish
• there are some important decisions that each has to evaluate about which bots are allowed and under which constraints, and which carry consequences
• this is not a one-time thing but something that is expected to continuously evolve, so it is useful to keep an eye on monitoring solutions and on the discussions sharing developments in bots' behavior
• it's also very important not to let this become a rabbit hole that consumes your entire energy and focus - draw some limits for what is "good enough", make any corrections in baby steps, with purpose and with enough time afterwards to observe the consequences, and take healthy breaks to enjoy peaceful life away from the problem.

Wishing all here a sunny weekend ahead, without letting a single thought about AI scrapers ruin a good time.

Also, not to forget: layers

 

[Levina]

What has made this thread so valuable for many months was that we were all sharing the same problem (if to vastly different amounts), all starting more or less at the same level having to develop understanding and measures from scratch. Quite a bunch of people were highly motivated and exploring the issue from different perspectives, trying out different strategies and approaches, shared what they found and everything was openly discussed.

Sadly this has changed a bit as some now not only seem to have found "their" solution but have started to promote their solution as the only valid one and their goals as the only valid ones. New people have joined the thread w/o bothering to read and learn what the collective state of knowledge is and the thread has started to loop, discussing from scratch what has already been excessively discussed and solved months ago and snake oil is praised. So sadly, the usefulness of the thread has massively decreased in my eyes as has the discussion level.

I think you need to come off your high horse, smallwheels, and relax. This thread is now 81 pages long. Someone just coming in, looking for help isn’t going to wade through the whole thing but just present their problem, ask for help. And not everyone gets it all right away. Some of the stuff will be over people’s heads for a while. God knows it was over mine and I’m still learning. So don’t give people the equivalent of the old RTFM because the same questions may have been addressed and answered already. It’s irrelevant because we are all different, have different forums, different objectives and there is no one size fits all. So be kind instead.

 

[Wildcat Media]

My previous post does this, a better way IMO, using cf.client.bot: ((http.user_agent contains "Googlebot" or http.user_agent contains "bingbot" or http.user_agent contains "Applebot" or http.user_agent contains "DuckDuckBot" or http.user_agent contains "Google-InspectionTool" or http.user_agent contains "Mediapartners-Google" or http.user_agent contains "AdsBot-Google" or http.user_agent contains "YandexBot" or http.user_agent contains "Baiduspider") and (not cf.client.bot))

I do it similarly myself. I shorten/combine some of them, similar to lower(http.user_agent) contains "google".

 

[smallwheels]

This thread is now 81 pages long.

Indeed. And this is one of the reasons why posting noise is harmful to the thread. Bots are a topic that is complex - there are no turnkey solutions (no matter what Cloudflare advertizes), no one pager that makes you an expert. The whole thing is a constantly moving target and to deal with it needs a systematic approach and at least a minimal level of technical understanding and the willingness to learn. In this thread quite a couple of experienced people have collectively built up knowledge and strategies and this went well b/c there was barely any noise up to a certain point.

Someone just coming in, looking for help isn’t going to wade through the whole thing

This surely won't get better if this thread is filled with noise and false claims. It undermines the probability that anyone is willing and able to learn from it.

And not everyone gets it all right away. Some of the stuff will be over people’s heads for a while. God knows it was over mine and I’m still learning.

Sure, there is no problem with that. The problem starts when the same people that lack understanding don't follow the advice given and even start to give advice to others themselves only minutes later, based on a more than shallow understanding of the topic or draw wild conclusions and post wild statements throughout the whole forum, behaving like an expert, while at the same time lacking even the basics of the basics of knowledge about the topic. This is a perfect example of the Dunning-Kruger effect and it does cause harm.



It is simply impossible to become an expert in the topic of bots and server management incl. the various technologies involved within one or two weeks - that is not a personal short coming but lies in the complex nature of the topic. It is also not possible to gain understanding of the topic w/o investing relevant amounts of time in learning and taking a steep and sometimes frustratingly hard learning curve. The content of this thread eases up this learning curve dramatically, you just have to read it - still it needs learning and investing time to build up knowledge. The thread may be long but in comparison to finding out everything out by other means yourself it saves months of time.

It is however easy to just grab a few buzzwords w/o any understanding and chit chat around, taking the whole thing as a kind of adventure game while having absolutely no idea what one is talking about and ignoring all hints and not answering questions. Obviously it also not illegal to behave like that - but it destroys the thread and does harm to basically all other forum members that want a valuable, deep, effecitive, honest and well founded discussion about a very complex mainly technical topic that leads to learning and usable, validated outcomes.

It is however possible to join this thread and participate as a newbie, ask questions, get help, share experiences, gain more and more knowledge and understanding and become able to deal with bots to a certain degree w/o becoming a deep expert and also w/o destroying the thread. A matter of attitude, respect and awareness.

It is completely fine to be new to a topic and forum members should support newbies. Also in this thread and it has happened here before w/o issues. But on the other hand newbies should - in their deepest own interest - not destroy and poison expert level technical discussions with loads of noise. Asking questions, sharing experiences - all fine. But once a thread like this gets too boring and too annoying for the more knowledgable to participate it will harm everyone including the newbies. There will be no more progress as there are no more experts and left are newbies who will give ill-led advice to other newbies. The classical death spiral of forums: Unrealiable statements by people of questionable qualification but with high self esteem and a strong opinion. Just read through the last couple of pages of this thread. This is rather a personal diary of Alvin63, lacking any technical details, context or useful hints to others than a thread where one could learn from. Just noise.

I mean: In real life you would, as a non expert, probably not join a dense expert level discussion about quantum physics by a group of physicist that stand together, ask them about a very special physics problem that you have and constantly tell them things like: "I just found out gravity exists, I saw an apple falling from a tree", "Apples consist of quantum, too and this is b/c the earth is flat" and other things on that level and, while ignoring their advice, only minutes later start to teach others about quantum physics as if you were an expert. It would be considered rude and idiotic. In a forum it is possible to bridge such knowledge gaps w/o issues and join the physicist's discussion w/o issues and beneficial - yet it is a question of attitude how you deal with it.

If the only statement regarding this problem is

It didn't start out as a "very technical thread".

there seems not to be the slightest bit of awareness about the problem and the topic of the thread. Sadly, the bot problem does not get solved through chit chat, anecdotes or frequent chaotic configuration changes w/o any learning curve or any systematic approach.

I think I will take a break from this thread. It is absolutely fine if you like it as it is - I more and more don't, it has lost it's value for me for the most part. Absolutely not worth participating here any more at the moment.

 

[Alvin63]

I noticed we can look at analytics beyond 24 hours now...up to 30 days.

The past three days for us--note this huge bots warm, and how it has calmed down since then. Note that I did nothing to the rules in the past week. That's a huge spike! Yet our origin didn't even notice.

View attachment 345483

In comparison, for scale, the past 24 hours. I hate to say this is getting boring, but...nothing is happening where I need to adjust any rules for a while. I haven't had a single notification from monit that our CPU% and load averages are above normal. But when I do get those notifications, that is usually an indicator that I need to revisit our CF security analytics and see if there are new swarms to deal with.

View attachment 345484

Those are HUGE numbers! 1.4 million (but at least mitigated