Dont get me started on this. My SSD filled server now has the value of a car.
Crazy amount of guests
- Thread starter Levina
- Start date
-
- Tags
- bot swarms guests
Don't even want to know current prices for server with that much storage. Though I am now curious
Dam $625 per 8TB drive! I nabbed a lot of spare NVMe drives just before the price hikes, my 14TB of NVMe drives were roughly 40% the cost of your 4x8TB NAS drives! I had a client get 3x servers each with 10x SSD/NVMe drives before price hikes - last checked a few months ago on hiked prices that would of been an extra US$3-5K/month per server. Probably even more now!
Sorry to hear about 5090
I've released a new version of my StandardLib add-on which implements a version of thelinkpostXF2.4 feature for XF2.1/XF2.2/XF2.3;
I've been considering adding an option to disable the redirect url that is getting added into the url but for now removing the session token should help.
Thanks this will come in handy!Meanwhile be doing more work on my Cloudflare Aggregated Analytics dashboards
It's insane and yeah...it's one of the reasons I hate (and absolutely refuse to use) AI. I lucked out in building my new video editing system in December 2024, and even then it cost me a bit since I needed a higher-end video card (to edit in real time without skipping/jumping), a lot of memory, and some decent sized NVMe drives for it. I couldn't even afford to build this same server now. My $210 memory modules were selling for $900+ a few months ago.
I asked this before but it was overlooked.
Why not just whitelist a handful of approved bots (legit scrapers identified in Cloudflare's bot categories) and issue a managed challenge to all guests?
My #5 rule in Cloudflare, which I enable only when we're getting crushed, does exactly that, and when it's in force we get a true number of members to guests, like a ratio of about 1:1.3. I can set the challenge timeout to a week, a month, whatever I want, so those same guests only have to periodically solve another challenge vs. every time they visit (provided they properly allow cookies).
The only hiccup I had was on one of my forums where I use R2 buckets to store images. It broke those. But I could probably fix that if I were motivated.
Sure it's inconvenient for some users but in just about all cases, nobody complains when we do this. (FYI--my rules are more extensive in what we block/challenge, but that final rule in Cloudflare is like the last line of defense.)
Why not just whitelist a handful of approved bots (legit scrapers identified in Cloudflare's bot categories) and issue a managed challenge to all guests?
My #5 rule in Cloudflare, which I enable only when we're getting crushed, does exactly that, and when it's in force we get a true number of members to guests, like a ratio of about 1:1.3. I can set the challenge timeout to a week, a month, whatever I want, so those same guests only have to periodically solve another challenge vs. every time they visit (provided they properly allow cookies).
The only hiccup I had was on one of my forums where I use R2 buckets to store images. It broke those. But I could probably fix that if I were motivated.
Sure it's inconvenient for some users but in just about all cases, nobody complains when we do this. (FYI--my rules are more extensive in what we block/challenge, but that final rule in Cloudflare is like the last line of defense.)
Well, sure you can do that, but it would be really annoying. The question in the end is: Whome for are you running your forum? Bots, humans (generically) or registered members? Since the rise of google we've come to an attitude, were sites on the internet (that was created for humans) have been optimized for machines in the first hand for ages (think about SEO and all the garbage text on websites that comes with it). In future, with AI becoming the middlemen between humans and websites, it seems that humans are unneeded and unwanted on websites - AI will scrape what's on them and then deliver it to humans that pay for the AI. At least, if you believe some people.
Then on the other hand: What would a forum be good for then? Who would actually use it? Who would create the content, that AIs then could scrape?
Possibly a wider amount of people simply walks away when it becomes too annoying to visit your website. Remember the week when this forum here ran on CF's "under attack mode" - using the forum was a total pita. Even now, with the new, current ruleset in place when you use the search even as a logged in user there comes reliably the CF captcha. Recommendations of XF enhanced search (as useless as they often are) are gone.
So if you want human visitors: don't annoy them more than necessary. If you want new forum members: they are typically guests, before they become members. If you forum does live from selling ads challenging every guest may kill your ad revenue.
Do you work for free?
Don't even want to know current prices for server with that much storage. Though I am now curious
if you're okay with recertified drives, serverpartdeals is your friend. Just added an additional 10tb seagate exos to my unraid server for $250
Good luck if you want an NVME though, Had a 4tb Samsung 990 that I bought for under $200 less than a year ago pro fail in my PC a couple months ago, thankfully while under warranty. Was looking into getting another one to have while waiting for the replacement and they were around $1k at the time, so decided I'd just stick with one of the 2tb ones I already have from the before times
All ajax calls will fail due to how cloudflare works.
What's most annoying is that they had the captcha interval around the default 30 minutes. This means visitors are constantly getting hit up. Also, no exclusion for being logged in.
Yikes why not keep them back then? I'm a hoarder for this gear. Still have draws full of DDR2/DDR3/DDR4 memory modules and CPUs/motherboards/GPUs from way back when I was into overclocking and PC memory reviews and beta testing memory for manufacturersI threw these in the skip 2 years ago, and they were fully decked out with 192GB of RAM and 1TB SAS drives
Might be worth something by 2028-2030 
Left that PC/OC scene before DDR5 memory came about though. Still nabbed 192GB of DDR5 as spare to ride out till 2030 before price hikes, they're now 2x to 6x the price!Indeed the 14TB NVMe drives I bought before hikes now cost 3.5x to 5x the price I paid! Also NVMe RMA warranty especially with Samsung is now problematic too - search for Louis Rossmann + 4TB Samsung RMA on Youtube - Samsung is just refunding the drive instead of RMA replacements!if you're okay with recertified drives, serverpartdeals is your friend. Just added an additional 10tb seagate exos to my unraid server for $250
Good luck if you want an NVME though, Had a 4tb Samsung 990 that I bought for under $200 less than a year ago pro fail in my PC a couple months ago, thankfully while under warranty. Was looking into getting another one to have while waiting for the replacement and they were around $1k at the time, so decided I'd just stick with one of the 2tb ones I already have from the before times
Back on topic
Indeed. Though with AI bots hammering sites, CF captchas will be more common and visitors probably will get use to it
luckily I got the replacement a couple weeks later
Mandated as EOL and not to be deployed. Anything < G7 went in the skip.
Google Search Console has the nerve to e-mail me monthly that there's some author tag missing somewhere in the data structure of the forum pages and, as a consequence, I'm getting penalized of sorts for not putting in the effort to make it convenient for Google's bots to ingest my forum data. Say what now? Since when do I work for Google, for free? How about Google can f right off.
Other mails complain that some pages are being blocked by robots.txt or its spiders are rejected from accessing them, and that's apparently a problem I should fix. Dude that is ON PURPOSE. I intentionally added some paths in there that I don't want your spiders to index. But somehow I have no option to ask Google to remove any data cached, it's all in the most cumbersome, inconvenient triple backflip kind of way to get anything OFF Google.
I am so done with Google.
Yeah, but that doesn't make them less annoying.Though with AI bots hammering sites, CF captchas will be more common and visitors probably will get use to it
I really wish it would be possible to distinguish trusted logged in accounts from others at CF WAF level zo bypass challenges for them, but that's not possible (out of the box).
Yeah. They marketed their "don't be evil" motto pretty well and pretty credible - until they ditched it and replaced it by "we don't give a f*** - we know everything about you, you depend from us, do what we say, give us all your data and we do whatever we want with it as well as with you, worm that you are. We rule the world and you are our slave, don't complain but comply."
Google is one company that i unconditionally make happy, even if their requests are ridiculous, because they provide benefit to me, my customer, and the people that use their product.
Can't say that for many big tech companies.
The achilles heel of cloudflare is that it's a proxy service, and has no interface to information that the application knows.
Can't say that for many big tech companies.
The achilles heel of cloudflare is that it's a proxy service, and has no interface to information that the application knows.
JWx & snippets could (partly) bridge that gap.
Last edited:
Yes, you can theoretically code it in. I imagine that's going to come at some kind of compute/network/latency cost, in addition to engineering time. Some data must be sent to cloudflare.
If you don't come up with a clever way to manage the added cost, you could be doubling the usage of some resource when you're getting hit with a wave of unique IP addresses.
If you don't come up with a clever way to manage the added cost, you could be doubling the usage of some resource when you're getting hit with a wave of unique IP addresses.
I got told recently in this thread pretty clearly that information from inside the application would not be necessary at all for filtering bots.
If guests are challenged once a week or once a month, they'll be used to it. Our forum is the type that has a lot of repeat visitors. I whitelist the scrapers I want visiting the site beforehand (especially Google and Bing), so that's not an issue. Logged-in users are whitelisted, so they don't encounter the challenge once they stay logged in. Honestly I don't see a problem doing this--as you say, since the bots are already hammering our sites (and denying them access when the server chokes up), they would probably rather click a box for a challenge rather than not visit the site at all because it won't load, or takes a minute or two to load a page.Indeed. Though with AI bots hammering sites, CF captchas will be more common and visitors probably will get use to it
I don't have the technical skills to code anymore (long story) but there is some sort of CF "pre-clearance" which I'm thinking would need to be installed with an add-on or, better yet, have it baked into XF's core as an option during login (or first visit by a guest...not quite sure how it works). With that pre-clearance cookie in their browsers, it would help ease the pain of waiting or clicking on a single checkbox.
We do have a very small number of vocal complainers (one whining that he obviously needed a bleeding-edge computer and an OS and browser updated within the last five minutes
) due to Cloudflare, but we are actually happier that they just stay out of the forum anyway. I mean, we're talking maybe one in several hundred who can't help but criticize anything/everything that doesn't work for them. Beyond that, what users don't like is when CF's captcha leads them into an endless loop; that however would be a problem on their end, I would think. They may be one of the tinfoil hat conspiracy theorists who think the entire Internet is out to get them, and have everything disabled on their browsers, as well as a few anti-virus/anti-malware suites stacked up on their computer.
On my own Internet provider, AT&T fiber, I was getting challenged by at least half the sites I visited, and for no apparent reason. And obviously, I've done nothing but try to visit their sites. With some, I was getting challenged every time I'd start a new session with them. This seems to have let up a little, but I'm sure all of us will be encountering more challenges as time goes on. Whether it's Cloudflare or not, these challenges will probably become the new normal for using the web anyway.
As I know that most of these sites are doing it to stop being hammered by the scrapers and bots, I don't mind clicking a simple checkbox if I need to.
