smallwheels
Well-known member
- Affected version
- 2.2.17
XF 2.2.17
I went through the weblog of my server on the hunt for 404s and found a lot of entries caused by the bing indexing bot. Among them was this one:
msnbot-40-77-167-149.search.msn.com - - [08/Apr/2025:15:15:17 +0200] "GET /tags/start/ HTTP/1.1" 404 9472 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 315 10202
The URL "/tags/start/" exists (there is a tag "start") and I can open it w/o issue. It seems only accessble when you are logged in (which bingbot is not). If I call the URL when not logged in I do indeed get a 404 - a bit strange, I would have expected a 401 here.
I am a little bit in sorrow that this way bing is collecting hundreds of 404s on my webpage from tags alone and this might negatively influence the SEO ranking. While technically those should not be 404s, as the URL exists, just the unlogged user does not have enough privileges to access it.
I went through the weblog of my server on the hunt for 404s and found a lot of entries caused by the bing indexing bot. Among them was this one:
msnbot-40-77-167-149.search.msn.com - - [08/Apr/2025:15:15:17 +0200] "GET /tags/start/ HTTP/1.1" 404 9472 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 315 10202
The URL "/tags/start/" exists (there is a tag "start") and I can open it w/o issue. It seems only accessble when you are logged in (which bingbot is not). If I call the URL when not logged in I do indeed get a 404 - a bit strange, I would have expected a 401 here.
I am a little bit in sorrow that this way bing is collecting hundreds of 404s on my webpage from tags alone and this might negatively influence the SEO ranking. While technically those should not be 404s, as the URL exists, just the unlogged user does not have enough privileges to access it.
