I managed to retrieve my full September access log, which gives me a much clearer picture of what happened before and after the crawler activity started.
I had AI analyse the complete log and break the traffic down by day, including the suspicious Chrome/macOS User-Agent I mentioned earlier.
The result is quite revealing:
* September 8th was the day I progressively introduced the Cloudflare protections, so it isn't a clean before/after day.
** September 9th is only a partial day, approximately 00:02–07:26.
The change appears to have started very clearly on September 4th.
Before that, the forum was averaging approximately:
Then suddenly:
What I find particularly interesting is what happens when the suspicious User-Agent is removed from the figures.
For September 4th:
And for September 7th:
That puts the remaining traffic almost exactly back into the normal range seen before September 4th.
So this gives me much more confidence that the huge increase wasn't simply legitimate visitors or normal search engine activity.
The number of IP addresses involved is also quite remarkable.
On September 7th:
And on September 5th:
So trying to deal with this by blocking individual IP addresses would have been almost pointless.
September 9th is where things become particularly interesting.
Between approximately 00:02 and 07:26, the origin log contains only:
The suspicious Chrome/macOS crawler was still present:
So every request carrying that particular signature during this period was rejected.
The crawler hasn't necessarily gone away. It is still trying, but those attempts are no longer successfully crawling the site in the same way.
There was another interesting surprise in the remaining traffic: Bingbot.
Of those 877 origin requests:
The Bingbot traffic came from 89 different IP addresses.
Some of the most active addresses were in ranges such as:
These are consistent with Microsoft/Bing crawler infrastructure, so this appears to be genuine Bing crawling rather than simply another bot using a fake Bingbot User-Agent.
I haven't individually reverse-DNS/forward-DNS verified every one of the 89 addresses, so I'm not claiming absolute verification of every request, but the IP distribution is consistent with legitimate Bingbot traffic.
This changes the interpretation of the 877 requests quite significantly.
More than three quarters of the origin traffic this morning appears to be Bing crawling the forum, while the crawler that caused the original problem accounted for another 12.7% and all of those requests were rejected.
That leaves only:
outside those two easily identifiable groups.
So the overall timeline now looks like this:
This also gives me something I didn't have before: a real baseline.
Before the crawler wave started, my forum was receiving roughly 5,000–9,000 origin requests per day.
The really striking part is that if I remove the suspicious crawler traffic from September 4th and September 7th, the numbers fall almost exactly back into that range.
I'll now leave the configuration unchanged and collect a complete 24-hour log for September 9th.
That should give me a much cleaner before/after comparison, but the first few hours are already looking very different from September 4th–8th.
I had AI analyse the complete log and break the traffic down by day, including the suspicious Chrome/macOS User-Agent I mentioned earlier.
The result is quite revealing:
Code:
Date Total requests Suspicious UA Percentage
---------------------------------------------------------
Sep 1 8,632 113 1.3%
Sep 2 5,398 99 1.8%
Sep 3 6,928 147 2.1%
Sep 4 44,413 34,678 78.1%
Sep 5 65,536 48,666 74.3%
Sep 6 53,266 34,102 64.0%
Sep 7 50,972 43,724 85.8%
Sep 8* 27,121 24,387 89.9%
Sep 9** 877 111 12.7%* September 8th was the day I progressively introduced the Cloudflare protections, so it isn't a clean before/after day.
** September 9th is only a partial day, approximately 00:02–07:26.
The change appears to have started very clearly on September 4th.
Before that, the forum was averaging approximately:
Code:
6,986 origin requests/dayThen suddenly:
Code:
Sep 3 → 6,928
Sep 4 → 44,413
Sep 5 → 65,536
Sep 6 → 53,266
Sep 7 → 50,972What I find particularly interesting is what happens when the suspicious User-Agent is removed from the figures.
For September 4th:
Code:
44,413 total requests
-34,678 suspicious requests
------------------------------
9,735 other requestsAnd for September 7th:
Code:
50,972 total requests
-43,724 suspicious requests
------------------------------
7,248 other requestsThat puts the remaining traffic almost exactly back into the normal range seen before September 4th.
So this gives me much more confidence that the huge increase wasn't simply legitimate visitors or normal search engine activity.
The number of IP addresses involved is also quite remarkable.
On September 7th:
Code:
43,724 suspicious requests
17,160 different IP addressesAnd on September 5th:
Code:
48,666 suspicious requests
19,505 different IP addressesSo trying to deal with this by blocking individual IP addresses would have been almost pointless.
September 9th is where things become particularly interesting.
Between approximately 00:02 and 07:26, the origin log contains only:
Code:
877 total requestsThe suspicious Chrome/macOS crawler was still present:
Code:
111 suspicious requests
111 different IP addresses
111 HTTP 403 responsesSo every request carrying that particular signature during this period was rejected.
The crawler hasn't necessarily gone away. It is still trying, but those attempts are no longer successfully crawling the site in the same way.
There was another interesting surprise in the remaining traffic: Bingbot.
Of those 877 origin requests:
Code:
680 identifying as Bingbot 77.5%
111 suspicious Chrome/macOS 12.7%
86 everything else 9.8%
---------------------------------------
877 totalThe Bingbot traffic came from 89 different IP addresses.
Some of the most active addresses were in ranges such as:
Code:
40.77.167.x
52.167.144.xThese are consistent with Microsoft/Bing crawler infrastructure, so this appears to be genuine Bing crawling rather than simply another bot using a fake Bingbot User-Agent.
I haven't individually reverse-DNS/forward-DNS verified every one of the 89 addresses, so I'm not claiming absolute verification of every request, but the IP distribution is consistent with legitimate Bingbot traffic.
This changes the interpretation of the 877 requests quite significantly.
More than three quarters of the origin traffic this morning appears to be Bing crawling the forum, while the crawler that caused the original problem accounted for another 12.7% and all of those requests were rejected.
That leaves only:
Code:
86 other requests in approximately 7 hours 24 minutesoutside those two easily identifiable groups.
So the overall timeline now looks like this:
Code:
Sep 1 ── 8.6k ┐
Sep 2 ── 5.4k ├── Normal traffic
Sep 3 ── 6.9k ┘
Sep 4 ── 44.4k ┐
Sep 5 ── 65.5k │
Sep 6 ── 53.3k ├── Major crawler wave
Sep 7 ── 51.0k │
│
Sep 8 ── 27.1k ┘ Protections progressively introduced
Sep 9 ── 877 requests (00:02–07:26)
|
+-- 680 Bingbot
+-- 111 suspicious Chrome/macOS -> all HTTP 403
+-- 86 other requestsThis also gives me something I didn't have before: a real baseline.
Before the crawler wave started, my forum was receiving roughly 5,000–9,000 origin requests per day.
The really striking part is that if I remove the suspicious crawler traffic from September 4th and September 7th, the numbers fall almost exactly back into that range.
I'll now leave the configuration unchanged and collect a complete 24-hour log for September 9th.
That should give me a much cleaner before/after comparison, but the first few hours are already looking very different from September 4th–8th.
![[XTR] IP Threat Monitor](/community/data/resource_icons/10/10134.jpg?1763724168){kind=icon}
