Crawl Budget Stats in Google Search Console

arn

Well-known member
Crawl budget has been a point of discussion for forums. Essentially, how much of Google's crawl of your site is "wasted" on non Response 200 results.

I'm curious what people's stats are for Xenforo forums specifically.

If you go to Google Search Console and go to Settings -> Crawl Stats, you should see this breakdown. So, of all our crawls on our forums, 69% are proper results. Whereas 14% are 301 redirects.

I don't believe 304 is a big deal.

Screen Shot 2020-11-30 at 4.52.06 PM.png
 

briansol

Well-known member
I have a ton due to https switch over, as well as using www's in the past. So all those old internal links get redirected, sometimes twice :(

a query to update that is pending, but i don't see it as being a massive determent

crawlstats.png.


I should also mention that I have a LOT of 301's where it looks like the bot is just trying IDs, eg

.com/threads/nnnnn/

hits which of course redirect to

.com/threads/title-here-nnnnnn/
 

Max Fridman

Active member
@arn your crawl budget changed over time?

Im trying to evaluate ours with 20% 4XX errors... for me 20% 4XX is a lot only for Xenforo. But maybe it isn't. Suggestions?

googlecrawl.png
 

arn

Well-known member
@arn your crawl budget changed over time?

Im trying to evaluate ours with 20% 4XX errors... for me 20% 4XX is a lot only for Xenforo. But maybe it isn't. Suggestions?

View attachment 250477

That's a lot, imo. Are you pointing to private / member-only sections? That would be permissions errors. Maybe also make sure your sitemap isn't pointing to stuff that's not publicly crawl able.

Here's my latest:

Screen Shot 2021-04-18 at 3.12.11 PM.png

200 actually went down. Not sure why.

But 301 is better, which was intentional.

Not sure if there's anything to do about 304s.

arn
 

Max Fridman

Active member
That's a lot, imo. Are you pointing to private / member-only sections? That would be permissions errors. Maybe also make sure your sitemap isn't pointing to stuff that's not publicly crawl able.

Ok, ill check, thanks.

Not sure if there's anything to do about 304s.

304 are "non modified content" so cached versions that Google can use, nothing to worry about.

I saw you changed your sitemap, in the end it helped with the Crawling Stats? and how did you do that?
 

Anatoliy

Well-known member
Im trying to evaluate ours with 20% 4XX errors... for me 20% 4XX is a lot only for Xenforo. But maybe it isn't. Suggestions?
I had about 10%, now it's 1%.
I tried that new "reply before registering" feature. Then I turned it off. But Google picked already "reply" links, and a crawler recieved 4xx.
Also if you have "no" for "can see a member profile" for unregistered, that would return 4xx to a crawler, too.
 

Max Fridman

Active member
I had about 10%, now it's 1%.
I tried that new "reply before registering" feature. Then I turned it off. But Google picked already "reply" links, and a crawler recieved 4xx.
Also if you have "no" for "can see a member profile" for unregistered, that would return 4xx to a crawler, too.

I use the reply before registering feature. Yes my members profile are private by default.

Should be possible to optimize and clean up this report so i can see if there are problems on our site.
 
Top