Crazy amount of guests

Interesting observations.

A couple of weeks ago during a round of bot storms, I noted previously that accesses via Linux were abnormally high.

Looking at HTTP version in analytics today, for a period of one week from 9/16 to 9/23, I notice the bot storms are preferring HTTP/2. The blue peaks coincide with bot swarms.

1790777938160.webp

1790777954601.webp

This is interesting though. The bot swarms used Linux, but notice the shift to Windows and MacOSX for our last big bot swarm on the right. I put a rule in place to raise a managed challenge to systems using Linux (excluding xf_user, which is whitelisted) and that tamed down some of the accesses; very, very few users in this forum are tech savvy enough to know what Linux/Ubuntu/etc. is, yet those few only get a managed challenge before they log in. No complaints from users!

1790778025678.webp

No surprises with the source browser. Most are passing "Chrome" as a user-agent, and note the small uptick in Edge on the largest peak on the right. Interesting that Firefox isn't in the top 5, but it did land at #6 when looking at the statistics for that week.

1790778229479.webp

If CF used a weighting system, assigning points to indicators like this (use of Linux and HTTP/2, for example), that could help raise a challenge. (Enterprise accounts might do this?)

Our current week has been boring, which I'm glad for! 😁
 
BrettC said:
Anubis, when configured correctly, does a proper pimp-slap to the majority of fake clients. Just have to be mindful of the challenge level/difficulty. It's quite customizable and can be set to auto-bypass clients with a xf cookie active too.

Make sure you have the honeypot feature active, hook it up to Prometheus and chart it with grafana. Sit back and enjoy the ai scraper poisoning. 😅
Click to expand...

Do you have some reference on how to configure the cookie part?
 
ES Dev Team said:
Do you have some reference on how to configure the cookie part?
Click to expand...
Indeed. For the utmost starters: Anubis operates on the basis of "Crap in, crap out" aka FIFO rulesets, aka top-down rules. It will stop processing rules per-client after a match.

The bots: configuration 'list' will always override the weight: clauses when a bots: clause segment net-ends in a challenge operation. I'm going to give you a slightly-filtered configuration from start to finish, as I'm not one for giving partial answers.

This first example is a generic example, without cookie validation checking:

Environment file:

Code: 
BIND_NETWORK=tcp
BIND=0.0.0.0:8924
TARGET=http://192.168.22.7:80
POLICY_FNAME=/etc/anubis/policy/website-name-here.yaml
CUSTOM_REAL_IP_HEADER=X-Real-IP
COOKIE_DOMAIN=.website-name-here.tld
COOKIE_EXPIRATION_TIME=504h
COOKIE_PREFIX=websitenameherebotcheck
COOKIE_SECURE=true
COOKIE_PARTITIONED=false
METRICS_BIND=0.0.0.0:9924
METRICS_BIND_NETWORK=tcp
WEBMASTER_EMAIL=staff@website-name-here.tld
versionFlag=false
ED25519_PRIVATE_KEY_HEX_FILE=/etc/anubis/key/key-name-here.hex

policy yaml file:
YAML: 
bots:
#  - name: allow-index-viewing
#    path_regex: ^/$
#    action: ALLOW
  - name: allow-api
    path_regex: ^/api/.*$
    action: ALLOW
  - name: allow-xenforo-assets
    path_regex: ^/(css\.php$|js/.*$|data/local/icons/.*$)
    action: ALLOW
  - import: /etc/anubis/data/crawlers/_allow-good.yaml
  - import: /etc/anubis/data/clients/discord-embed.yaml
  - import: /etc/anubis/data/crawlers/alibaba-cloud.yaml
  - import: /etc/anubis/data/crawlers/tencent-cloud.yaml
  - import: /etc/anubis/data/bots/aggressive-brazilian-scrapers.yaml
  - import: /etc/anubis/data/bots/ai-catchall.yaml
  - import: /etc/anubis/data/bots/headless-browsers.yaml
  - import: /etc/anubis/data/common/keep-internet-working.yaml
  - import: /etc/anubis/data/common/allow-private-addresses.yaml
  - import: /etc/anubis/data/common/rfc-violations.yaml
  - import: /etc/anubis/data/asn/as132203.yaml
  - import: /etc/anubis/data/datacenters/google-cloud-all.yaml
  - name: challenge-all
    action: CHALLENGE
    path_regex: .*
    challenge:
      algorithm: fast
      difficulty: 4
store:
  backend: valkey
  parameters:
    url: "redis://192.168.22.11:6379/42"

The imports are mostly of all default anubis rulesets. asn/ and datacenters/ are my custom additions which generally translate to DENY, or extremely harsh Challenge rules. When all of those ALLOWS and import options are processed, if none are met, a global challenge takes place at a level of 4. Yes, storing each client session into redis/valkey is a must - very handy for logging/analytics on top of the built in anubis metrics.

Now, for the more complex 'item'... the env file above is just about the same for the following configuration...

YAML: 
bots:
#  - name: allow-index-viewing
#    path_regex: ^/$
#    action: ALLOW
  - name: allow-api
    path_regex: ^/api/.*$
    action: ALLOW
  - name: allow-xenforo-assets
    path_regex: ^/(css\.php$|js/.*$|data/local/icons/.*$)
    action: ALLOW
  - import: /etc/anubis/data/crawlers/_allow-good.yaml
  - import: /etc/anubis/data/clients/discord-embed.yaml
  - import: /etc/anubis/data/crawlers/alibaba-cloud.yaml
  - import: /etc/anubis/data/crawlers/tencent-cloud.yaml
  - import: /etc/anubis/data/bots/aggressive-brazilian-scrapers.yaml
  - import: /etc/anubis/data/bots/ai-catchall.yaml
  - import: /etc/anubis/data/bots/headless-browsers.yaml
  - import: /etc/anubis/data/common/keep-internet-working.yaml
  - import: /etc/anubis/data/common/allow-private-addresses.yaml
  - import: /etc/anubis/data/common/rfc-violations.yaml
  - import: /etc/anubis/data/asn/as132203.yaml
  - import: /etc/anubis/data/datacenters/google-cloud-all.yaml
  - name: xf-cookie-bypass
    action: ALLOW
    path_regex: .*
    expression:
      all:
        - '"Cookie" in headers'
        - headers["Cookie"].contains("xf_user")
  - name: upstream-weight-boost-05
    action: WEIGH
    expression: '"X-Weight-Adjust" in headers && headers["X-Weight-Adjust"] == "5"'
    weight:
      adjust: 5
  - name: upstream-weight-boost-10
    action: WEIGH
    expression: '"X-Weight-Adjust" in headers && headers["X-Weight-Adjust"] == "10"'
    weight:
      adjust: 10

thresholds:
  - name: WeightOfFive-Level5Difficulty
    expression: weight = 5
    action: CHALLENGE
    challenge:
      algorithm: fast
      difficulty: 5
  - name: WeightOfTen-Level10Difficulty
    expression: weight = 10
    action: CHALLENGE
    challenge:
      algorithm: fast
      difficulty: 10
  - name: default-Level3Difficulty
    expression: weight < 5
    action: CHALLENGE
    challenge:
      algorithm: fast
      difficulty: 3

store:
  backend: valkey
  parameters:
    url: "redis://192.168.22.11:6379/44"

YAML: 
  - name: xf-cookie-bypass
    action: ALLOW
    path_regex: .*
    expression:
      all:
        - '"Cookie" in headers'
        - headers["Cookie"].contains("xf_user")
is the part of interest for you in the example. This rule will (or at least does on my end) point-blank bypass any challenge issued. I intentionally added path_regex for everything on a domain name, as I was having issues making it play nice a good while back.
 
Cloudflare seems to have changed it's colour scheme for analytics, just to confuse issues - mitigated used to be orange! Now it's pink and "served by Cloudflare" is now orange.

Bot numbers have been low-ish recently 400k to 500k and low in my guest list for a few days - 150 to 250. The biggest help seems to have been blocking fake robots. Yet it looks like there was a big spike at server level this morning.

This is my block list

(ip.src.asnum in {132203 45090 133478 139341 132591 134103 137876 58835 45102 24429 37963 38369 45104 134963 60924 136188 7497 9393 62429 10099 136907 55990 151610 141180 56041 56046 58453 56045 56040 132510 23724 58461 134774 24400 4811 17621 56044 24444 27665 55836 24547 11664 47589 329227 134739 17557 17882 42961 57764 132525 21299 7713 9930 24757 24445 24556 8359 27725 5384 2609 140308 38266 56042 4812 56047 27789 27775 11816 9329 21003 151491 28118 327934 30986 27800 28287 4134 4809 4837 9929 4808 9808 58807 4788 27699 28573 264663 265944 272112 12389 60095 8452 24863 36992 45899 45758 7552 38019 137542 63526 136255 136442 132148 136780 136210 1547 6810 1680 15344 51375 212616} and not cf.client.bot) or (ip.src.country in {"BD" "BY" "BR" "CN" "EG" "ID" "IQ" "IL" "KZ" "KG" "MY" "MX" "MM" "PH" "RU" "TH" "VN" "DO"} and not cf.client.bot) or (http.user_agent contains "meta-externalagent") or (http.user_agent contains "meta-webindexer") or (http.user_agent contains "meta-externalads") or (http.user_agent contains "Googlebot" or http.user_agent contains "bingbot" or http.user_agent contains "Applebot" or http.user_agent contains "DuckDuckBot" or http.user_agent contains "Google-InspectionTool" or http.user_agent contains "AdsBot-Google" or http.user_agent contains "GPTBot" or http.user_agent contains "ClaudeBot" or http.user_agent contains "PerplexityBot" or http.user_agent contains "CCBot" or http.user_agent contains "facebookexternalhit" or http.user_agent contains "Twitterbot") and (not cf.client.bot)

Cloudflare.webp
 
Last edited:
Orange is a painful colour online... the burgundy is much easier on the eyes to me. Saying that, I don't use CF analytics though, no online analytics at all, actually.
 
ES Dev Team said:
I checked my Xenforo forum today after 4 days of using Anubis.
Unique IP address count is 1.5k, also 1/10-1/20 what it was.
Bandwidth down 1/4 of what it was before.

Look forward to tuning it :)
Click to expand...
Forgot to mention, for the weight adjustment rules, you'll need to set (on nginx anyways) proxy_set_header X-Weight-Adjust "10";. The 10 can be another digit, so long as it's configured as a weight value on Anubis's thresholds section.

Makes the entire WAF very dynamic once you configure the location /some/location {} blocks aptly. For example, you can force location /search/ to a weight of 5 for clients lacking a xf_user cookie, while the other location / block could even default to 3 or 4.

It sounds a bit confusing, but once you get the idea of it, it works pretty well. Anubis is a sure-fire leap from the GUI-friendly of Cloudflare's stuff, but hey... some of us like the hands-on method to working with this stuff, right? :D
 
Our swarm has moved to using fake safari browsers, so trying this WAF rule set to managed challenge in addition to @Anthony Parsons chrome/mozilla rules. Those rules, btw, are blocking over 4MM request per day with 0% challenge success rate :cool:

Code: 
(
    http.user_agent contains "Mozilla/5.0"
    and http.user_agent contains "Version/26."
    and http.user_agent contains "Safari/"
    and not http.user_agent contains "Chrome/"
    and not cf.client.bot
    and
    (
        not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode")
        or
        not any(lower(http.request.headers.names[*])[*] == "accept-language")
    )
)

Is anyone else using chatgpt to analyze raw nginx access logs? It's quite interesting and informative. I'm late to the AI game, but I've been feeding it compressed nginx log files daily and the knowledge it retains and offers every day is remarkable.
 
Last edited:
kontrabass said:
Our swarm has moved to using fake safari browsers, so trying this WAF rule set to managed challenge in addition to @Anthony Parsons chrome/mozilla rules. Those rules, btw, are blocking over 4MM request per day with 0% challenge success rate :cool:

Code: 
(
    http.user_agent contains "Mozilla/5.0"
    and http.user_agent contains "Version/26."
    and http.user_agent contains "Safari/"
    and not http.user_agent contains "Chrome/"
    and not cf.client.bot
    and
    (
        not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode")
        or
        not any(lower(http.request.headers.names[*])[*] == "accept-language")
    )
)

Is anyone else using chatgpt to analyze raw nginx access logs? It's quite interesting and informative. I'm late to the AI game, but I've been feeding it compressed nginx log files daily and the knowledge it retains and offers every day is remarkable.
Click to expand...
I've had the fake safari clients for quite some time. The majority are via residential proxies. Most don't provide proper / traditional client headers, just a botched or incorrect user agent. From my own findings, absolutely no JavaScript support.
 
kontrabass said:
Our swarm has moved to using fake safari browsers
Click to expand...
I just keep coming back to someone taking it personally against xenforo forums. Post a rule to block the current fake browsers, they switch to safari and try and spoof that. It's not like the internet is talking about fake browser CF rules - yes, I googled that first.

Still seems like someone has it out for xenforo forums to me... but that is just me. Any who... my take on this one, pretty much the same thing, just a slightly different way of going about it. My version targets all fake safari, not just version 26.

Code: 
(
  (
    (
      http.user_agent contains "Chrome/1"
      and not any(lower(http.request.headers.names[*])[*] == "sec-ch-ua")
    )
    or
    (
      http.user_agent contains "Mozilla/5.0"
      and (http.user_agent contains "Chrome/1" or http.user_agent contains "Firefox/1")
      and not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode")
    )
    or
    (
      http.user_agent contains "Mozilla/5.0"
      and (http.user_agent contains "Chrome/1" or http.user_agent contains "Firefox/1")
      and not any(lower(http.request.headers.names[*])[*] == "accept-language")
    )
    or
    (
      http.user_agent contains "Mozilla/5.0"
      and http.user_agent contains "Version/"
      and http.user_agent contains "Safari/"
      and not (http.user_agent contains "Chrome/")
      and not (http.user_agent contains "Chromium/")
      and (
        not any(lower(http.request.headers.names[*])[*] == "sec-fetch-mode")
        or not any(lower(http.request.headers.names[*])[*] == "accept-language")
      )
    )
  )
  and not (
    http.cookie contains "xf_user="
    or http.cookie contains "xf_session="
  )
)
 
ES Dev Team said:
Forums are full of human generated text and one of the last places an AI training operation will find clean ( little to no LLM output ) data on the open web.
The more text you feed AI, the smarter it gets. So they are big targets.
Click to expand...
Plus two other things:
• in oppostite to books, newspapers, PR materials and normal webpages forums do use a way broader and more normal human laguage. Which probably helps AI to understand humans better and also to answer in a less "machine" style/lexicon language.
• regarding the content forums also show a way broader set of topics and opinions plus the content is often perfectly suited to what LLMs are used: To answer questions and show the range of possible answers.

So forums seem indeed a perfect target for AI training.
 
Last night we saw a significant PHP overload from 11:15 onwards exhausting memory resources and CPU. This PHP overload stemmed from 20,000 HTTPS requests originating primarily from the XenForo forum hosted on our website. From looking at the high volume we can see these seemed to be distributed across many client IP addresses but using similar mobile browser identities following similar search request patterns "Wandzee".
It has been recommend to add rate limiting upon search forum requests to avoid regular incidents of this variety.

What would be the best way to go about this?
We have a Cloudflare pro account.
 
webbouk said:
It has been recommend to add rate limiting upon search forum requests to avoid regular incidents of this variety.

What would be the best way to go about this?
Click to expand...
The easiest thing would be to set the permissions for search to "no" for guests, but this is not rate limiting and obviously creates colateral damage for genuine guests. But it is done within seconds and solves the issue for /search.

Other than that it may be a welcome opportunity to generically fine tune your CF setup to lock bots out before they hit your server as bot waves may also find other ways to overload your server. However: I'm not using CF, so no clue how to rate limit there. But within this thread, especially over the last pages, countless example configs for CF that proved to be working have been posted.
 
webbouk said:
Last night we saw a significant PHP overload from 11:15 onwards exhausting memory resources and CPU. This PHP overload stemmed from 20,000 HTTPS requests originating primarily from the XenForo forum hosted on our website. From looking at the high volume we can see these seemed to be distributed across many client IP addresses but using similar mobile browser identities following similar search request patterns "Wandzee".
It has been recommend to add rate limiting upon search forum requests to avoid regular incidents of this variety.

What would be the best way to go about this?
We have a Cloudflare pro account.
Click to expand...
If they are residential proxies (which seems common) then I found rate limiting didn't do anything. Because they switch IP addresses every couple of seconds.

I would turn on superbotfightmode if you have pro. I have botfightmode on and custom rules as well. Botfightmode is catching loads - the ones without proper headers mainly. The rest get dealt with by custom rules.

If setting up custom rules, a block rule needs to come before a challenge rule. I would suggest blocking China and Myanmar and ASN's in Singapore for Tencent, Huawei and Alibaba (cloud centres) and then the challenge rule, like Anthony suggested above.

My second custom rule is a challenge rule like Anthony's but I also included "/members/", "/misc/cookies/", "/threads/", "/ams/", "/posts/" and I also added a referer (meaning genuine guests don't get challenged) however that does let some bots through as some fake a referer.

I think you'll need superbotfightmode as well to cut the numbers down - you can set it to block "definitely automated" or "likely automated" - which should help with the percentage that can solve challenges. Blocking China (I blocked a few other eastern countries too but it depends if you have members from there) made a huge difference to start with, and still makes a big difference, but they then moved to Myanmar and residential proxies worldwide in my case.

After a tip on here, I downloaded the raw server logs and uploaded them to Claude to analyse. It could help if there is a specific swarm from one asn or country so you know what to block - but ultimately all it showed for me was - a lot of residential proxies from 6 main agents somewhere. Although it did give me some names to block as well.

There are those on here who know a lot more but ..........

1) Superbotfightmode
2) Country/ASN custom block rule
3) Challenge uri paths rule

I also have AI labyrinth turned on and precursor turned on - not sure if they're helping much or not.
 
Last edited:
You must log in or register to reply here.

Similar threads

O
  • Question Question
XF 2.2 Huge amount of guests
Replies
6
Views
189
infinitysnapz
infinitysnapz
Wildcat Media
Unusually high number of guests (4x to 5x normal)
Replies
8
Views
222
Tumpe
Tumpe
Tyson
Possible to limit amount of threads a guest can view?
Replies
4
Views
1K
Junior
J
Back
Top Bottom