Crazy amount of guests

[BrettC]

This is so crucial. If we use ai to summarise something we can't be bothered to read, it is very obvious that we are in no position to verify its accuracy. And yet so many use ai in that way, and have total confidence in it because it's, er, ai.

It comes down to the old fallacy I read it in the paper (or saw it on telly) so it must be true.

While we're getting a bit OT here.... AI does have its use-cases, but you have to know how to direct and guide the AI 'agent' correctly. Otherwise, just as you've effectively said, using AI without verification (or general knowledge in the field) gains a false confidence in it.

 

[puterfixer]

Some people are unfortunately too lazy to move eyeballs left and right to read, then think through the information and finally make their own opinion. From this group, some failed upwards and started demanding "executive summaries" in e-mails because they're "busy" and "time is valuable" and they focus on "strategic thinking" not details. They've got peons to worry about those, digest them and spit out a very superficial summary. To them all, AI's confidently incorrect summaries are the next level of an enabler for their skimming through life and consequently making misinformed decisions. It will get to the point where "if AI is doing the entire thinking and suggesting actions, why do we pay you to be here" will be a very valid question. But that's another tragedy.

AI (actual intelligence) would figure out that there are some points in the current discussion that have not been explicitly stressed but are important for a noob's understanding of why admins need to set filters. You can't assume that the average forum dweller has an understanding of bots and how (copyrighted) information on forums is valuable to spammers, scammers or AI model training, and scraping it for electronic processing is essentially a breach of copyright and fair use. It is no surprise that big AIs bought physical books by the millions, scanned them and then destroyed them because this was a scenario which lawyers could defend in court against copyright. Others, like facebook, grabbed stuff in digital format directly through piracy - torrents and the like, and didn't get even a slap on the wrist. The point is, forum admins have broader responsibilities than breaking up fights among members, and the visibility on that work behind the scenes is still a gap in explaining why filtering is necessary to protect a site's best interests.

 

[Azaly]

Cloudflare is great. It saved me during bot surges: I'd just enable CAPTCHAs for non-target countries for a day or two. Caching helped too. But CF is now blocked in some countries. So I bought a second server as a frontend for filtering and caching. However, running CAPTCHAs during an attack drains resources fast. It's a shame CF lacks GeoDNS, and Argo is too pricey for my project. Honestly, I have no idea what the hell to do. The IPs are unique and not from cloud servers. It's just infected devices worldwide.

 

[BrettC]

Cloudflare is great. It saved me during bot surges: I'd just enable CAPTCHAs for non-target countries for a day or two. Caching helped too. But CF is now blocked in some countries. So I bought a second server as a frontend for filtering and caching. However, running CAPTCHAs during an attack drains resources fast. It's a shame CF lacks GeoDNS, and Argo is too pricey for my project. Honestly, I have no idea what the hell to do. The IPs are unique and not from cloud servers. It's just infected devices worldwide.

Since you have a second server (assuming VPS, Colo, dedi or otherwise), you should look into implementing Anubis: https://anubis.techaro.lol/

Once you get it up and running, it is extremely versatile in its configuration - and can save you a ton of headache.

 

[Azaly]

Since you have a second server (assuming VPS, Colo, dedi or otherwise), you should look into implementing Anubis

Tested already. My users hate me for this and it not so light as I hoped.

 

[ES Dev Team]

I'm not saying "told you so" (Fall 2025), but well... we all knew something was going on. It's well known that a third party has been mass-scraping the internet for just about anything and everything.

I am redeeming told-you-so points on a daily basis these days and am loosing interest in collecting on this one.



Cloudflare is slowly showing cracks and i think it's due to a software monoculture problem: the dominant thing gets studied and attacked more.
Adoption progressively increases the risk to the internet as a system.

What i don't particularly like seeing in my logs is being attacked via cloudflare owned IP addresses and Cloudflare completely dismissing my detailed reports. If you use cloudflare, you are probably whitelisting it. That's a small problem that could get bigger.


My alternative is coming together. The hard problem is achieving good performance and low memory consumption at the same time. I had a lot of false starts solving that. APCU turned out to be a bust. It's looking like memcached can produce a system that can process 50-100k requests/sec, with a high % of the traffic being unique, on a 4 core system with about 2.5gb of ram to store records on 10 million IP addresses.

I have a bit more feasibility testing to do before i can call it a thing. I'll report in later on how this is going.

 

[ES Dev Team]

Tested already. My users hate me for this and it not so light as I hoped.

What was the problem with anubis?

I know it is a bit of a fiasco to setup, and not particularly friendly to visitors out of the box, but other than that?

 

[Wildcat Media]

Cloudflare is slowly showing cracks and i think it's due to a software monoculture problem:

Certain operating systems had the same issue in the past. Yeah, they could get all sorts of viruses, but when the majority of users are on a single OS, of course the hackers are going to attack the most popular to get the most bang for the buck.

And, even the earliest CAPTCHA mechanisms were easily fooled.

It's what I've said about security--both I.T. and physical/in person. I can secure my house all I want to. I have high-res surveillance cameras outdoors. I fortified our locks and hinges, from rekeying the locks with security pins to using strong latch plates and beefier hinge screws. There are other countermeasures in place for windows and doors, as well as other electronic measures. Yet if someone wants to get in, they're going to get in. Tossing a landscape brick through the doorwall is a five-second task.

At least for physical security, one of the goals is to make a potential intruder decide the work isn't work the risk, and move on to neighbors who pose a much easier target. I don't see that happening in regards to our bot handling, unfortunately--they'd rather just hammer away at us relentlessly until they find a crack.

 

[Azaly]

I know it is a bit of a fiasco to setup, and not particularly friendly to visitors out of the box, but other than that?

Page loading stuck for too many good users.

 

[ES Dev Team]

At least for physical security, one of the goals is to make a potential intruder decide the work isn't work the risk, and move on to neighbors who pose a much easier target. I don't see that happening in regards to our bot handling, unfortunately--they'd rather just hammer away at us relentlessly until they find a crack.

Here's my current thinking.

Effective cybersecurity is making the cost of an attack high enough that an attacker is discouraged from carrying it out.
The problem here is that we also need to be nice to legitimate users in the process.
Cloudflare and Anubis both make for a pretty bad welcome mat.

From the server, you can impose a relatively small cost of bandwidth, memory, and CPU to a web visitor which would be large for a bot operator.
If someone doesn't want to pay that, then bye, felicia.

If enough websites are imposing costs, it's going to be more economically difficult to do web scraping at all!

If the solution can be easier to setup than cloudflare or anubis ( not hard, both are multi step processes ), then more websites can participate in punishing scrapers.

For that reason i'm working on a system that's easier to distribute and install. Because i think it is possible to stop this crap.

 

[ES Dev Team]

Page loading stuck for too many good users.

Sad, anyone else had this problem?

 

[BrettC]

Page loading stuck for too many good users.

Could you expand on that? It sort of sounds like a slight misconfiguration occurred between the ingest server -> anubis -> backend server in relation to cookie settings or redirect settings.

 

[Azaly]

It sort of sounds like a slight misconfiguration occurred between the ingest server -> anubis -> backend server in relation to cookie settings or redirect settings.

You might be right, but they also observed issues on the official Anubis website when I asked them to check it for testing purposes. Аt that time I decided to stick with geo-blocking.
upd: in my Xenforo installation the bottleneck is redirects to posts not threads themselves. I have no idea why bots check them so agressivly.

 

[Anthony Parsons]

If enough websites are imposing costs, it's going to be more economically difficult to do web scraping at all!

If the solution can be easier to setup than cloudflare or anubis ( not hard, both are multi step processes ), then more websites can participate in punishing scrapers.

Interesting idea. Call me intrigued.

 

[Anthony Parsons]

They're slowly starting to come back... having removed all protections to see what happens.

 

[Anthony Parsons]

Fairly decent overview/summary of the thread. Some 2c's worth of feedback...

Yup. Without giving it specific instructions, like to read all the points of issue and solution, group them, sift them for duplicates and similarities, do this, do that, etc. I didn't do any of that. No doubt if I did, it would have been different again. I wanted to test what the latest AI could do to build a document for this thread, which talks about blocking ai crawlers, without giving it any deep instruction, just, go look at all the pages, build a document to summarise the content, issues and solutions. It was very limited instruction, where I agree, you need to give them detailed instruction for more focused outcomes. Then George outlined the 2nd and 3rd pass idea to see how it could improve upon itself, so I ran that too, and it works. Again, no detailed instruction, just go look at your work and see if you missed anything, anything can be improved upon.

 

[Anthony Parsons]

Xenforo installation the bottleneck is redirects to posts not threads themselves.

That is an interesting point, which I discovered myself, as I have a previous domain I used to use for many years, which does a thread redirect to the updated domain location. That was being hammered. I stuck a global managed challenge on that redirect, excluding known bots, and it wiped out a few hundred thousand daily uniques.

 

[puterfixer]

My domain is so far back to its pre-assault levels, although there is a bit of unusual traffic during the night that I didn't look into (yet).



Regarding AI summaries, I've learned at work (huge global IT/consulting firm, partners with all tech providers, clients pretty much in all industries, so far encourages employees to learn to use AI as a tool not as a mandatory corporate cult) that it helps to define in the AI prompt the persona and the deliverables/outcome. The persona can make a massive difference in the way it works, because it defines so much about the "personality" AI will emulate:

• "You are an IT security expert, which takes pride in being thorough in details, concise in summarizing complex topics, and using reputable external sources as references where needed. You need to produce a summary of current risk mitigation strategies for defending an online forum against bots while protecting access of legitimate human visitors, and a suggested action plan or checklist for a forum administrator with priorities and options for the different layers of protection that might be involved." vs.
• "You are a technical IT consultant with excellent communication and documentation skills, tasked to explain a technical topic to a non-technical audience, in a way that explains why forum administrators must work to protect access to the limited resources of a shared hosting server, why the forum contents is worth protecting, how the constraints might impact forum members' experience, and what they should do or avoid doing. Generate 3 proposals for a short (<10,000 characters) forum post from slightly different perspectives that might better fit different audience types."

On making bots' traffic expensive, I'm not sure how that could be doable. The economy of the internet works at such a massive scale that it's hard to put a dent in the profitability of abuse. Take spam for example: a curated database of a few tens of millions of e-mail addresses costs like $200, and spam sending servers are also cheap. With just a few hundred bucks you can reach millions of people, to sell them... overpriced pillows, or frying pans, or pills. Any idiot that answers gives you say $30 in revenue. It takes 20-30 gullible idiots out of millions to break even, and there are far more idiots responding to spam and scams. Spam still exists at such a large scale because it is massively profitable, and no filtering or end user education or banks' fraud detection cannot curb this phenomenon down. Scraper bots don't even have financial institutions breathing down their necks or a fake web shop to maintain for visitors, they just grab the free stuff publicly accessible and resell it multiple times at a cost of a few beers per month per VPS.

 

[smallwheels]

The persona can make a massive difference in the way it works, because it defines so much about the "personality" AI will emulate:

• "You are an IT security expert, which takes pride in being thorough in details, concise in summarizing complex topics, and using reputable external sources as references where needed. You need to produce a summary of current risk mitigation strategies for defending an online forum against bots while protecting access of legitimate human visitors, and a suggested action plan or checklist for a forum administrator with priorities and options for the different layers of protection that might be involved." vs.

could be fun to make a comparison run with the outcome of something like this:

"You are an over confident idiot with very high self esteem but absolutely no technical clue and no understanding of or interest in technical details. You avoid reading specifications or log files as this is boring and you would not understand either of them anyway. Yet you believe you are an expert in basically everything and want to explain experienced professional developers, security experts and system administrators what is going on in regards of bots, what to do about it and how."

 

[smallwheels]

The economy of the internet works at such a massive scale that it's hard to put a dent in the profitability of abuse. Take spam for example: a curated database of a few tens of millions of e-mail addresses costs like $200, and spam sending servers are also cheap. With just a few hundred bucks you can reach millions of people, to sell them... overpriced pillows, or frying pans, or pills. Any idiot that answers gives you say $30 in revenue. It takes 20-30 gullible idiots out of millions to break even, and there are far more idiots responding to spam and scams. Spam still exists at such a large scale because it is massively profitable, and no filtering or end user education or banks' fraud detection cannot curb this phenomenon down. Scraper bots don't even have financial institutions breathing down their necks or a fake web shop to maintain for visitors, they just grab the free stuff publicly accessible and resell it multiple times at a cost of a few beers per month per VPS.

The concepts have been around for ages: I.e. if sending a mail would cost a minimal amount of real money (let's say i.e. 1 Cent) it would not at all affect people wanting to send a mail to another person. People sending out newsletters or automated mails would think twice, as at some point cost would start to add up. People who send millions of spam emails per day would have to rethink their business model. And all of a sudden our inboxes would be empty and email would become a usable form of communication again.
Today, the cost of email is mainly at the receiver side: spam filtering, storage, sorting and the time to read all this stuff. Sending on the other hand is basically free. Once you turn this around all of a sudden the whole situation changes drastically, possibly for the better.
You could use the same principles in other areas as well, like i.e. access to web pages. With bot protection, one way that is actually used is computation cost. If you give a requesting client some extra work to compute it does not affect a normal user. It does however add up if you request thousands of URLs per minute. Far from being perfect or sufficient, but basically the same principle.