How I reduced the huge guest/bot traffic with Cloudflare and protected the origin server
After experiencing the same huge increase in guest traffic described in this thread, I thought I would document exactly what I did, step by step, in case it helps other XenForo administrators.
This is not intended as a universal set of blocking rules. The important part is the method:
- Analyse the server access logs.
- Put the website behind Cloudflare.
- Create targeted Cloudflare rules based on what the logs actually show.
- Prevent direct access to the origin server, so Cloudflare cannot simply be bypassed.
1. What we found in the access logs
The forum suddenly went from normal guest numbers to several hundred concurrent guests, sometimes close to 1,000, including during the night.
The raw web server logs made the problem much easier to understand.
During one period we analysed:
- 25,055 HTTP requests reached the origin.
- 22,938 of them (91.6%) used exactly the same suspicious User-Agent.
- The requests were distributed across thousands of different IP addresses.
So blocking individual IP addresses was clearly not going to be an effective solution.
In our case, we also found some heavily represented ASNs and a very specific browser User-Agent being reused across thousands of IPs.
Your traffic will probably be different, so I strongly recommend analysing your own access logs instead of copying someone else's IP/ASN blocking rules.
2. Put the website behind Cloudflare
The first step was to proxy the website through Cloudflare.
Instead of:
Code:
Visitor → Origin server → XenForo
we now have:
Code:
Visitor → Cloudflare → Origin server → XenForo
Only the web hostnames need to be proxied.
Mail, FTP, control panel and other services can remain DNS-only if required by your hosting setup.
Once DNS propagation is complete, normal HTTP/HTTPS traffic should therefore reach Cloudflare before reaching the web server.
3. Create targeted Cloudflare security rules
I did NOT immediately challenge the entire XenForo installation.
Instead, I looked at the access logs and created targeted rules.
In our case these included:
- Blocking one ASN responsible for a significant amount of the abusive traffic.
- Managed Challenge for
/whats-new/.
- Managed Challenge for
/search/.
- Managed Challenge for the exact suspicious User-Agent identified in the access logs.
For example, a path rule can look like:
Code:
starts_with(http.request.uri.path, "/search/") and not cf.client.bot
Action:
The
not cf.client.bot part allows Cloudflare verified bots to be excluded from the rule.
For the suspicious User-Agent, I used an exact match rather than challenging all visitors to XenForo threads.
That rule alone eventually recorded more than 18,000 events with a 0% Challenge Solve Rate.
I would NOT recommend blindly copying a User-Agent, ASN or country block from another forum.
Use your own logs to identify what is actually attacking your site.
4. The remaining problem: Cloudflare can potentially be bypassed
This is the part that I think is particularly important.
Putting your DNS behind Cloudflare does not necessarily make your origin server inaccessible.
If someone knows the origin IP, they may potentially connect directly to it while supplying the correct hostname.
Instead of:
Code:
Bot
|
v
Cloudflare
|
v
Origin
the bot could attempt:
Code:
Bot ──────────────> Origin
In that situation, your Cloudflare WAF/challenge rules are irrelevant because Cloudflare never sees the request.
Ideally, this should be prevented at firewall/server level by only allowing Cloudflare to access HTTP/HTTPS.
However, this was not possible with my shared hosting environment.
So I used a Cloudflare custom request header combined with
.htaccess.
5. Create a secret header in Cloudflare
In Cloudflare, go to the rules section and create a Request Header Transform Rule.
The terminology may vary slightly depending on the Cloudflare interface, but the objective is:
Code:
Incoming requests: All requests
Modify request header:
Operation: Set static
Header name: X-My-Origin-Auth
Value: <LONG RANDOM SECRET>
Do NOT use
X-My-Origin-Auth as your real production header name just because it appears in this example.
Choose your own header name.
More importantly, generate a long random value and keep it secret.
Conceptually Cloudflare will now send this to the origin:
Code:
X-My-Origin-Auth: a-long-random-secret-value
Do not put the blocking rule into
.htaccess yet.
First test that this actually works.
6. Create a temporary PHP test
Create a temporary file in the web root, for example:
test-cloudflare.php
Contents:
PHP:
<?php
if (!empty($_SERVER['HTTP_X_MY_ORIGIN_AUTH'])) {
echo 'CLOUDFLARE_OK';
} else {
echo 'CLOUDFLARE_ABSENT';
}
If you chose a different HTTP header name, adjust the PHP variable accordingly.
For example:
becomes:
in PHP's
$_SERVER array.
This PHP file does NOT display the secret. It only tells us whether the header exists.
7. Test normal access through Cloudflare
Visit:
Code:
https://www.example.com/test-cloudflare.php
through your normal browser.
You should see:
If you see:
STOP HERE.
Do not add the blocking rule.
Your Cloudflare Transform Rule is either not working or the header is not reaching the origin.
8. Test direct access to the origin
Now we want to deliberately bypass Cloudflare.
You need to know your origin server IP for this test.
With curl, you can force a connection directly to the origin while preserving the correct hostname/SNI.
For example:
Code:
curl --resolve www.example.com:443:ORIGIN_IP https://www.example.com/test-cloudflare.php
On Windows you may need:
Code:
curl.exe --resolve www.example.com:443:ORIGIN_IP https://www.example.com/test-cloudflare.php
This request goes directly to the origin rather than resolving the hostname normally through Cloudflare.
The expected result is:
So before doing any blocking, you should have successfully demonstrated:
Code:
Normal access through Cloudflare:
CLOUDFLARE_OK
Forced direct access to origin:
CLOUDFLARE_ABSENT
9. Only now block direct access in .htaccess
Once BOTH tests above are successful, the origin can be configured to reject requests that do not contain the correct secret.
Near the beginning of
.htaccess, before the normal XenForo rewrite rules, add:
Code:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP:X-My-Origin-Auth} !^YOUR_RANDOM_SECRET$
RewriteRule ^ - [F,L]
</IfModule>
Replace:
X-My-Origin-Auth
with your own header name, and:
YOUR_RANDOM_SECRET
with EXACTLY the same random value configured in Cloudflare.
The check should validate the actual secret, not simply check whether the header exists.
Otherwise somebody could potentially bypass the protection simply by adding an empty/fake header with the correct name.
10. Test the website immediately
First visit the forum normally.
It should work exactly as before:
Code:
Visitor
|
v
Cloudflare
|
| Adds secret
v
Origin
|
| Secret accepted
v
XenForo
If the website returns 403 through normal access, immediately remove/disable the new
.htaccess rule and check the Cloudflare header configuration.
11. Verify that direct origin access is now blocked
Repeat the direct-origin curl test, this time against the homepage:
Code:
curl -I --resolve www.example.com:443:ORIGIN_IP https://www.example.com/
On Windows:
Code:
curl.exe -I --resolve www.example.com:443:ORIGIN_IP https://www.example.com/
The expected result is now:
while:
continues to work normally.
At that point you have effectively achieved:
Code:
INTERNET
|
v
CLOUDFLARE
|
WAF / Challenges
|
Adds secret header
|
v
ORIGIN
|
Is secret valid?
/ \
YES NO
| |
v v
XenForo 403
And a crawler trying to bypass Cloudflare gets:
Code:
Bot → Origin IP → No valid secret → 403 Forbidden
12. Clean up
Once everything has been verified, delete:
test-cloudflare.php
It is no longer needed.
Keep a copy of the working
.htaccess before making future changes, and obviously keep the Cloudflare secret private.
Important considerations
This method is useful when you cannot restrict the origin to Cloudflare IP ranges at firewall/server level, for example on some shared hosting platforms.
It also means that Cloudflare becomes part of the availability path for the website. If the Cloudflare Transform Rule is removed or stops adding the header while the
.htaccess restriction remains active, the origin will return 403.
So document the setup somewhere for yourself. Six months from now it is very easy to forget why that strange custom header exists in
.htaccess.
Also be careful if the same document root serves other hostnames that are NOT proxied through Cloudflare. The
.htaccess rule above would block them too. In that situation the rule should be scoped to the appropriate hostname(s).
Finally, this doesn't magically identify or eliminate every crawler.
What it does is ensure that automated traffic can no longer simply bypass the filtering layer by connecting directly to the origin.
From there, Cloudflare rules can actually do their job, and you can adjust them based on what your own access logs show.
In our case this has been considerably more effective than trying to chase thousands of rotating IP addresses.
