Crazy amount of guests

[Alvin63]

Is it just very large forums having this issues? Mine spiked for a while but has settled down again.

 

[ES Dev Team]

Mine spiked to 12k briefly but i saw 17k here.

90% of the activity is slamming the what's new page

msnbot/bingbot is the primary source of terror for me lately

 

[ES Dev Team]

I saw this site climb to 17k bots rapidly.. now mine's at 20k
What i'm seeing reminds me of that anubis breakthrough - so distributed that rate limiters won't work.



Generally the IPs are generally doing this or hitting what's new:



This is indefensible with fail2ban isn't it. I might have to move to cloudflare for a while until i finish my system

The one difference versus Xenforo.com and my site was that the blast was slightly less intense on Xenforo and was half the duration.

 

[Andy.N]

And just as suddenly, the number of crawlers have dropped over night. It has under 500 guests right now, compared to 30K+ the last few days.
Just weird.

 

[chillibear]

And just as suddenly, the number of crawlers have dropped over night.

Your training is complete young Padawan go hence from this place and create AI slop for the masses...

 

[Alvin63]

And just as suddenly, the number of crawlers have dropped over night. It has under 500 guests right now, compared to 30K+ the last few days.
Just weird.

Mine spiked and went back to normal as well, but much lower numbers than yours. Maybe they got what they needed and left! Or thought "nothing to see here - move along" Following on with the Jedi theme!

 

[ES Dev Team]

I ended up banning china mobile and i think some other ISP.
My numbers look great due to this racist policy.

But the one major source of this crap from China that i haven't shut off is now going haywire.
It's real browsers in a clickfarm, thus they show up in google analytics.
Normally the China count would be 1 user since our site is quite USA-centric.

I'm quite sure i'm going to ban the other China ISP as the amount of bandwidth is still enormous.



This is a beyond unacceptable abuse of our resources. The bandwidth consumed is 3tb/month and growing. Before AI-pocalypse, this number was under 500gb/month.

 

[Jja]

I'm quite sure i'm going to ban the other China ISP as the amount of bandwidth is still enormous.

Why not list China under challenge and not dealing with their ASNs? 99% of those bots wont solve the challenge and will move on to other domain

 

[smallwheels]

Why not list China under challenge and not dealing with their ASNs? 99% of those bots wont solve the challenge and will move on to other domain

You seem to have missed that @ES Dev Team does not use Cloudflare and also does not want to. Cloudflare is not a given - many people do not want to use it, i.e. for privacy reasons.

 

[Jja]

I might have to move to cloudflare for a while until i finish my system

What i thought.

Cloudflare is not a given - many people do not want to use it, i.e. for privacy reasons.

 

[ES Dev Team]

What i thought.

Doesn't hit very hard when someone who doesn't care about privacy or better uptime is mocking me

 

[Jja]

 

[BrettC]

What i thought.

A lot of companies intentionally do not utilize Cloudflare (or other like-typed providers) for varying reasons. Be it legal, privacy, datamining concerns, company decision, network stack 'issues', existing infrastructure already handles the needs, or even financial burden. There is nothing wrong with not using Cloudflare. I don't understand the tin-foil statement gif.

As robust as Cloudflare is, they do add another 'cog to the wheel', which means another potential point of failure. The same applies to 'cloud hosting' such as AWS, Azure, GCloud, etc... you're now fully reliant on their uptime/connectivity guarantees.


As for me, I finished implementing Anubis this morning on just about all websites that I manage (not all of these are Xenforo or forum related), and I've literally shut the door on all of these abusive LLM/AI Scrapers. The only virtual machines on my side of things are now just the load balancers/ingest servers and reverse proxy servers delegating, checking and terminating requests at Anubis. Nothing bot-like is making it to the actual websites, it is actual users and friendly search indexers hitting the backend now.

Self-hosting stuff, and learning the ins and outs of your software and hardware limitations and potential is huge in this day and age. A lot of companies are moving their entire stack 'to the cloud' without having optimizing their websites or web services. That becomes costly, quick. Cloudflare is no different in that instance when you're a large company/corporation.

 

[Jja]

Your point is valid, but as you see, some individuals are all the time pushing this "the big brother is watching us" agenda all the time.

Pushing their opinion at such frequency, aggressively, they do look like a foil heads here.

 

[ES Dev Team]

As for me, I finished implementing Anubis this morning on just about all websites that I manage (not all of these are Xenforo or forum related), and I've literally shut the door on all of these abusive LLM/AI Scrapers. The only virtual machines on my side of things are now just the load balancers/ingest servers and reverse proxy servers delegating, checking and terminating requests at Anubis. Nothing bot-like is making it to the actual websites, it is actual users and friendly search indexers hitting the backend now.

That's awesome to hear. what do your average guest counts look like before/after?
Keep us up to date, haven't heard of anyone else where running anubis.