Optimizing Google Crawl Budget

A

Alfuzzy

Well-known member
I'm trying to optimize my sites Google crawl budget. My understanding is...items can be added to a sites robots.txt file to prevent Googlebot from crawling/indexing unnecessary items...and thus better utilize the Google crawl budget for the important stuff.

Can some of the experts out there please list what you feel:

  • Should be included in the robots.text file to prevent Googlebot from wasting crawl budget.
  • Should NOT be included in a robots.text file in regards to crawl budget...to make sure Googlebot DOES crawl/index it.

If there are other considerations for optimizing Google crawl budget (other than the robots.txt file)...would really appreciate those insights as well.

Thanks:)
 
robots.txt

Code: 
User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /members/
Disallow: /admin.php
Disallow: /tags/
Allow: /
 
Thanks Much DJ.

Looking at what you listed.../tags/...this was one I've been wondering about (and my robots.txt file does not have). I will add it.

I'm curious about disallowing "posts". Is this because "threads" is already crawled...and crawling/indexing "posts" is a double-count of the exact same content?

Or even more than double...since an individual thread could have 10 posts or more...and each post would have a crawl count of one (10 for the whole thread).

Thanks:)
 
tags would depend upon how properly they are used on your forum. if they are free for all, it makes no sense cluttering search results with them. i recently disabled tags for members and also made them noindex because of the same reason. if you have limited number of tags that are used judiciously, there is no harm in letting them index i suppose.
 
Alfuzzy said:
I'm curious about disallowing "posts". Is this because "threads" is already crawled...and crawling/indexing "posts" is a double-count of the exact same content?

Or even more than double...since an individual thread could have 10 posts or more...and each post would have a crawl count of one (10 for the whole thread).

Thanks:)
Click to expand...
Yes exactly.
 
djbaxter said:
robots.txt

Code: 
User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /members/
Disallow: /admin.php
Disallow: /tags/
Allow: /
Click to expand...

One more question please. What does the Allow: / mean?

Does this mean everything else (anything not Disallowed)...is ok to crawl/index?
 
djbaxter said:
robots.txt

Code: 
User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /members/
Disallow: /admin.php
Disallow: /tags/
Allow: /
Click to expand...

I modified my sites robots.txt file to reflect the parts mentioned above that were not already in the robots.txt file.

One more question. Should each of the items in the list be a separate sub-directory on my server for my XF install?

For example (if the XF install resides in the public_html directory on my server)...I'm not seeing a public_html/tags/ or public_html/posts/ sub-directory.

I'm familiar with how my old vBulletin 4 server file structure worked...maybe the XF file structure is different.

Just want to be sure the Googlebot is able to find these things properly.

Thanks
 
Good deal thanks. I think I did hear something about this somewhere else...but wanted to be 100% sure.

Is this more like a "virtual" directory system (not actual directories on the server)?

Also...if I wanted to add the sitemap location to my robots.txt file...what would be the proper "sitemap" statement?

Thanks again.:)
 
Alfuzzy said:
Do I use


or


The reason why I ask is...there's a difference between what DJBaxter mentioned in post #2...and what member "Mouth" linked in post #10 above.

DJ's robots.txt does not have the "community" part...and the link "Mouth" posted does have the "Community" part.

Thanks
Click to expand...
The robots.txt line for the sitemap if your forum is in the root directory is this:

Sitemap: https://{YOURSITE}.com/sitemap.xml

If the forum is in a subdirectory, it would look like this:

Sitemap: https://{YOURSITE}.com/{FORUMDIRECTORY}/sitemap.xml

Using sitemap.php will also work:

ICODE]Sitemap: https://{YOURSITE}.com/sitemap.php[/ICODE]

Sitemap: https://{YOURSITE}.com/{FORUMDIRECTORY}/sitemap.php

The line posted above by @Mouth is the one for Xenforo, where the actual forum is in a subdirectory called "community".

And the line posted by @Chromaniac in post #14 above lacks the Sitemap: for robots.txt.
 
Awesome. Thanks so much for explaining in detail DJ. Also thanks for clarifying what was linked in post #10. I thought that was the actual robots.txt for xenforo.com (here)...and figured the "Community" part was the way Xenforo had things structured/organized (but now I know 100%).

Yes my XF install is in the root directory...and I will use the first statement you mentioned above.

Thanks!:)
 
I believe because you can use sitemap.xml rather than sitemap.php if you have friendly URLs enabled.
 
You must log in or register to reply here.

Similar threads

arn
Crawl Budget Stats in Google Search Console
Replies
12
Views
1K
DeltaHF
DeltaHF
K
  • Question Question
XF 1.5 403 Google Crawler Errors
Replies
5
Views
1K
dking
dking
snoopy5
  • Question Question
XF 1.5 Google Crawl error 403 on disallowed pages
Replies
3
Views
826
Mike
Mike
tommydamic68
  • Question Question
XF 1.5 Google - crawl errors
Replies
7
Views
2K
tommydamic68
tommydamic68
MstrfBlng
I noticed my Google crawl statistics. Loading speed Xenforo vs vBulletin 5
Replies
0
Views
1K
MstrfBlng
MstrfBlng
Back
Top Bottom