I've also been experiencing massive problems with guest traffic over the past few days.
During the day, I usually have around 50 to 70 real, logged-in users and between 500 and 1,500 guests online. However, yesterday I reached a new record of almost 47,000 guests. My forum became extremely slow, and I frequently encountered HTTP 429 errors.
My website is hosted in Germany, and my hosting provider initially advised me to use .htaccess to allow only German IP addresses, as the traffic was coming from countries all over the world.
I managed to resolve the issue by allowing access from Germany, all its neighboring countries, and several other European countries.
After adjusting my .htaccess configuration, the number of guests online has dropped to around 1,000, and the forum is running normally again.
Important: This is not a recommendation, just a description of what worked for me.
Here is my .htaccess configuration with some additional optimizations to ensure that Google and Bing can still crawl and index my website:
Code:
# Mod_security can interfere with uploading content such as attachments.
# If you experience upload issues, remove the "#" from the lines below.
#<IfModule mod_security.c>
# SecFilterEngine Off
# SecFilterScanPOST Off
#</IfModule>
# Use default Apache error responses
ErrorDocument 401 default
ErrorDocument 403 default
ErrorDocument 404 default
ErrorDocument 405 default
ErrorDocument 406 default
ErrorDocument 500 default
ErrorDocument 501 default
ErrorDocument 503 default
# Custom PHP configuration path
SetEnv PHPRC /home/www/doc/85366/mydomain.com/www/
<IfModule mod_rewrite.c>
RewriteEngine On
# 1. Immediate bot protection:
# Block style-variation requests without a valid internal referrer
RewriteCond %{REQUEST_URI} ^/misc/style-variation [NC]
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?mydomain\.com [NC]
RewriteRule .* - [F,L]
# Block style-variation requests containing outdated bot timestamps
RewriteCond %{REQUEST_URI} ^/misc/style-variation [NC]
RewriteCond %{QUERY_STRING} t=(1788|1794) [NC]
RewriteRule .* - [F,L]
# Block AI crawlers from accessing a specific thread (ID 2028)
RewriteCond %{HTTP_USER_AGENT} (GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|CCBot|PerplexityBot|Bytespider|Google-Extended) [NC]
RewriteRule ^threads/.*\.2028/?$ - [F,L]
# Standard XenForo rewrite rules
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
RewriteRule ^(data/|js/|styles/|install/|favicon\.ico|crossdomain\.xml|robots\.txt) - [NC,L]
RewriteRule ^.*$ index.php [NC,L]
</IfModule>
<IfModule mod_maxminddb.c>
MaxMindDBEnable On
MaxMindDBFile DB /usr/local/share/maxminddb/GeoLite2-Country.mmdb
MaxMindDBEnv MM_COUNTRY_CODE DB/country/iso_code
# 1. Allowed countries (selected European countries)
SetEnvIf MM_COUNTRY_CODE "^(DE|DK|PL|CZ|AT|CH|FR|LU|BE|NL|LI|IT|ES|HR|GR|GB)$" AllowAccess
# 2. Allow AI systems, search engines, and social media link previews
SetEnvIfNoCase User-Agent "(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|DuckDuckBot|YandexBot|WhatsApp|facebookexternalhit|Facebot|Twitterbot|TelegramBot|Discordbot)" AllowAccess
<RequireAll>
# Always block explicitly denied IP addresses (Alibaba Cloud spam)
Require not env BadBot
# Always block known cloud hosting and scraper networks
# Including Alibaba Cloud and DigitalOcean EU/US
Require not ip 47.79.
Require not ip 178.62.
Require not ip 134.209.
Require not ip 167.99.
Require not ip 159.65.
Require not ip 165.22.
Require not ip 142.93.
Require not ip 68.183.
Require not ip 46.101.
Require not ip 209.38.
Require not ip 137.184.
Require not ip 159.223.
Require not ip 164.90.
Require not ip 164.92.
# At least ONE of the following conditions must be met:
<RequireAny>
# Allowed countries, AI crawlers, and social media previews
Require env AllowAccess
# Googlebot and Google services (listed IP ranges)
Require ip 66.249.64.0/19
Require ip 64.233.160.0/19
Require ip 72.14.192.0/18
Require ip 209.85.128.0/17
Require ip 216.239.32.0/19
Require ip 2001:4860:4801::/48
# Bingbot and Microsoft Copilot (listed IP ranges)
Require ip 157.55.39.0/24
Require ip 207.46.13.0/24
Require ip 40.77.167.0/24
Require ip 52.167.144.0/24
Require ip 13.66.139.0/24
Require ip 2a01:111:f400::/48
</RequireAny>
</RequireAll>
</IfModule>
And here my robots.txt:
Code:
# robots.txt
# General rules for search engines
User-agent: *
Disallow: /admin.php
Disallow: /install/
Disallow: /search/
Disallow: /misc/
Disallow: /members/
Disallow: /whats-new/
Disallow: /find-threads/
Disallow: /account/
Disallow: /login/
Disallow: /register/
Disallow: /conversations/
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /posts/*/share
Disallow: /help/bb-codes/
Disallow: /help/cookies/
# Aggressive SEO crawlers and scraper bots
User-agent: AliyunBot
Disallow: /
User-agent: Alibaba
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: SpiderLing
Disallow: /
User-agent: SketchEngine
Disallow: /
# AI crawlers: Public forum content remains accessible
# System, search, and member areas are blocked
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /admin.php
Disallow: /install/
Disallow: /search/
Disallow: /misc/
Disallow: /members/
Disallow: /whats-new/
Disallow: /find-threads/
Disallow: /account/
Disallow: /login/
Disallow: /register/
Disallow: /conversations/
Disallow: /online/
Disallow: /posts/*/reactions
Disallow: /posts/*/share
Disallow: /help/bb-codes/
Disallow: /help/cookies/
Disallow: /help/impressum/
Disallow: /help/privacy-policy/
Disallow: /help/terms/
As I mentioned before, please don't just copy and use this code unless you know exactly what you're doing!
Maybe someone can suggest improvements to my .htaccess and robots.txt configurations. Any feedback or suggestions are welcome!