Crazy amount of guests
- Thread starter Levina
- Start date
There are some IPs which should be whitelisted.
For example google - https://developers.google.com/static/search/apis/ipranges/googlebot.json
34.126.178.96/28
I'm experience this right now with 37K guests and I have cloudflare Pro plan with all the AI Bot turned on.
The server load is extremely high and bouncing around 100 where normal load would be 1-2.
I see the mariadbd server has very high load with 488 %CPU.
I tried to restart the sql server, etc but it does not help at all.
Any advice?
The server load is extremely high and bouncing around 100 where normal load would be 1-2.
I see the mariadbd server has very high load with 488 %CPU.
I tried to restart the sql server, etc but it does not help at all.
Any advice?
MentaL
Well-known member
serve static pages to guests
How? is it done via XF or Cloudflare?
MentaL
Well-known member
@digitalpoint has an addon that can automate it.
![[DigitalPoint] App for Cloudflare®](/community/data/resource_icons/8/8750.jpg?1692633961){kind=icon}
MentaL
Well-known member
"
- Support for caching certain content at the network edge(in Cloudflare data centers):
- HTML pages for guests"
Does it help with lowering the server load which is my #1 concern right now.
MentaL
Well-known member
if guests are accessing the server in a dynamic timeline (no cache), then yes. It'll reduce db and webserv load.
digitalpoint
Well-known member
Yes, because when a page is cached, the request doesn't even need to go to your origin server, it's served to the user (or bot) directly from Cloudflare's closest data center. It's kind of the point (at least partially).
I updated the addon to the latest version, config it with an API. The load still not budging from 100 with 35K guests onsite.
Is there a magic button somewhere that I can add to help deal with this?
digitalpoint
Well-known member
Enable the option to cache pages for guests (under Admin -> Cloudflare -> Edge caching).
Oh yeah baby!!!
You save the day, Shawn. Thanks very much.
The load is from 100+ is dropping to single digit now.
So this means without this edge caching offered by this addon, the 40K guests (mostly AI crawlers) would hammer my server all day long.
I also guess that simply turn on Anti-bot feature from Cloudflare would not do much.
ES Dev Team
Well-known member
consistently 2-3k guests here on fail2ban lately.
the secret sauce is good banlists
Glad you found an answer btw, that's a record amount of guests i've heard of on any site
the secret sauce is good banlists
Glad you found an answer btw, that's a record amount of guests i've heard of on any site
digitalpoint
Well-known member
Yep, and good bots (like Googlebot) as well as people finding your site via search (unregistered users) should see significantly faster load times since they are just being fed the content from geographically closer places (Cloudflare data centers). A single location on earth (your origin server) can be 20,000km away from the user and potentially traversing a zillion different networks/hops. Having the page cached 50 or 100km from users is going to always be faster.
Try browsing around your own site with Incognito to test it.
...but also can free up resources of your server that no longer needs to render every page request in full.
How another topic without your fail2banconsistently 2-3k guests here on fail2ban lately.
the secret sauce is good banlists
ES Dev Team
Well-known member
Just thinking here.
For members with similar issues, is there another option to deal with this when they are not aware of this addon and using cloudflare? I imagine a less technical site owner would just not be able to figure out the issue nor the solution.
Similar threads
- Question
