Hey everyone,
I'm managing a discussion forum in an Eastern European language. It is for people interested in practicing a particular sport, fairly off the radar and very unusual to get attention.
In recent few weeks I've noticed a massive spike in the number of guests hitting the forum, from what used to be maybe 8-10 from time to time to values in the 3-4 thousands. These came in waves during the day, it's nearly nobody online then suddely 4000, then quiet again. Now it's roughly continuous every time I look.
The traffic to the hosting goes through CloudFlare, primarily because the attachments are in R2 storage and the level of filtering and security in transit to the server is a side benefit. This has been working for years this way. On the hosting itself, CloudFlare subnets are whitelisted in .htaccess and all other traffic targeting the server IP (v4/v6) directly, bypassing CloudFlare, is blocked.
I already did a lot of work to identify and control bots. Long story short:
But then why do random forum threads get hits on exactly the same URL at the same time, but from 2 different locations in the world, in countries that do not speak the forum language? Why would someone in Bangladesh and someone else in Chile be interested in a topic that is very specific to another place in the world? Always 2, always at the same time, always from different places... Something is very very weird.
In recent days this shifted from hitting on legitimate URLs to page after page of errors. XenForo does not show the URL or referral, just that it's an error for an unknown page. The hosting is obfuscating visitor IPs in logs in their interpretation of compliance with GDPR, so that's useless for any in-depth tracing. But at least I do see thousands upon thousands of hits directly on the cookies page, with all sorts of parameters, without an apparent first connection to the forum root to establish a server session.
To me, this does not look at all like normal human behavior, and more like agents doing stuff as orchestrated by a Command and Control system. But I am lacking the tools to see and identify potential distributed attempts to brute force or find vulnerabilities in the software.
How can I get XenForo to show the URL which returned an error?
I know that the default answer is "the forum is doing what is intended and providing responses in line with the guests' forum permissions", and that would have been a fairly reasonable answer some years ago. But the golden days of the internet are long gone, the by default trust in clients following conventions and standards is gone, it is now a wild west where human-originated traffic is a teeny tiny minority among bot traffic.
The purpose of hosting a forum is for people to be allowed to read contents and to engage in a community. Search engine spiders were tolerated because they'd get a bit of advertising and my forum would get exposure. But Google started returning AI summaries that many times is sufficient to make people NOT click on the links to read the source sites, and recently announced that it's moving away from returning search results, instead will keep the user in an AI conversation and just feed answers there.
I have absolutely zero obligation to allow bots to ingest copyrighted material as training data for a commercial AI model, for free and without any previous agreement of any kind, from which I get nothing in return other than a fat bill from the hosting. So I do need the ability to get ahead in this cat-and-mouse game with a bit more than "works as designed".
The same behavior I have noticed on a second forum I manage, which is an extreme niche community, private funded with no advertising, no new members for years, kept out of search engines etc., and it's also being hammered - but at least there I could basically make the entire forum private, guests don't even see the forum categories.
Has anyone else noticed this recently? How did you (successfully?) handle it?
Many thanks in advance.
I'm managing a discussion forum in an Eastern European language. It is for people interested in practicing a particular sport, fairly off the radar and very unusual to get attention.
In recent few weeks I've noticed a massive spike in the number of guests hitting the forum, from what used to be maybe 8-10 from time to time to values in the 3-4 thousands. These came in waves during the day, it's nearly nobody online then suddely 4000, then quiet again. Now it's roughly continuous every time I look.
The traffic to the hosting goes through CloudFlare, primarily because the attachments are in R2 storage and the level of filtering and security in transit to the server is a side benefit. This has been working for years this way. On the hosting itself, CloudFlare subnets are whitelisted in .htaccess and all other traffic targeting the server IP (v4/v6) directly, bypassing CloudFlare, is blocked.
I already did a lot of work to identify and control bots. Long story short:
- CloudFlare bot categories are being already handled there, there's some bot fighting mode enabled and captchas and so on
- I have a few dozen AS numbers and TOR filtered out, to prevent connections from major cloud providers
- Many bots completely disregard robots.txt directives or identifying themselves through user-agent
But then why do random forum threads get hits on exactly the same URL at the same time, but from 2 different locations in the world, in countries that do not speak the forum language? Why would someone in Bangladesh and someone else in Chile be interested in a topic that is very specific to another place in the world? Always 2, always at the same time, always from different places... Something is very very weird.
In recent days this shifted from hitting on legitimate URLs to page after page of errors. XenForo does not show the URL or referral, just that it's an error for an unknown page. The hosting is obfuscating visitor IPs in logs in their interpretation of compliance with GDPR, so that's useless for any in-depth tracing. But at least I do see thousands upon thousands of hits directly on the cookies page, with all sorts of parameters, without an apparent first connection to the forum root to establish a server session.
To me, this does not look at all like normal human behavior, and more like agents doing stuff as orchestrated by a Command and Control system. But I am lacking the tools to see and identify potential distributed attempts to brute force or find vulnerabilities in the software.
How can I get XenForo to show the URL which returned an error?
I know that the default answer is "the forum is doing what is intended and providing responses in line with the guests' forum permissions", and that would have been a fairly reasonable answer some years ago. But the golden days of the internet are long gone, the by default trust in clients following conventions and standards is gone, it is now a wild west where human-originated traffic is a teeny tiny minority among bot traffic.
The purpose of hosting a forum is for people to be allowed to read contents and to engage in a community. Search engine spiders were tolerated because they'd get a bit of advertising and my forum would get exposure. But Google started returning AI summaries that many times is sufficient to make people NOT click on the links to read the source sites, and recently announced that it's moving away from returning search results, instead will keep the user in an AI conversation and just feed answers there.
I have absolutely zero obligation to allow bots to ingest copyrighted material as training data for a commercial AI model, for free and without any previous agreement of any kind, from which I get nothing in return other than a fat bill from the hosting. So I do need the ability to get ahead in this cat-and-mouse game with a bit more than "works as designed".
The same behavior I have noticed on a second forum I manage, which is an extreme niche community, private funded with no advertising, no new members for years, kept out of search engines etc., and it's also being hammered - but at least there I could basically make the entire forum private, guests don't even see the forum categories.
Has anyone else noticed this recently? How did you (successfully?) handle it?
Many thanks in advance.
Last edited:
