1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.1 robots.txt

Discussion in 'XenForo Questions and Support' started by tekgirl, Jan 2, 2012.

  1. tekgirl

    tekgirl Active Member

    I have read quite a bit about the robots.txt file and still am confused about what files/directories shold be disallowed. Can someone help?

    Thanks
     
  2. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    graham_w and Mt.Rev like this.
  3. tekgirl

    tekgirl Active Member

    Thanks, Jake!
     
  4. Mt.Rev

    Mt.Rev Member

    Thanks Jake
     
  5. craigiri

    craigiri Well-Known Member

    I know this is an old thread, but wonder if that robots.txt file is complete....well, maybe some depends on add-ons!

    But I am showing a vast number of google crawl errors on conversations.....

    Also showing on bookmarks, but - yes- that's an add-on.

    Strangely enough, the standard directory block doesn't seem to stop the conversation errors, so I am now trying a more complete URL like:
    Disallow: /talk/conversations/add?to=*
    Disallow: /talk/bookmarks/*
    I also blocked the conversation direction in another line....but most of the crawler goog errors have that other text after them.
    The second line is for bookmarks (add-on).
    I will update this message after I watch the goog crawl errors for a week or two.
     
  6. MagnusB

    MagnusB Well-Known Member

    I usually disallow when I see things in GWT that shouldn't be there (like a year ago I saw my styles folder in phpBB was indexed). On top of that I have disallowed quite a bit more of the default stuff:
    Code:
    Disallow: /account/
    Disallow: /admin.php
    Disallow: /ajax/
    Disallow: /attachments/
    Disallow: /conversations/
    Disallow: /data/
    Disallow: /forums/-/
    Disallow: /forums/tweets/
    Disallow: /goto/
    Disallow: /help/
    Disallow: /internal_data/
    Disallow: /js/
    Disallow: /library/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /misc/contact/
    Disallow: /members/
    Disallow: /online/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /posts/
    Disallow: /search/
    Disallow: /styles/
    Allow: /
    In addition I have disallowed my Intro forum, cause allot of the threads there are just repeating them self.
     
    Jake Bunce likes this.
  7. craigiri

    craigiri Well-Known Member

    Will disallowing attachments make it so that goog does not index attachments (full size) inside posts?
     
  8. MagnusB

    MagnusB Well-Known Member

    Yes, any attachment will not be indexed then, I don't remember if the ones inserted wont be indexed, I know the thumbnails are directly hotlinked, the inserted ones I am not sure about.
     
  9. craigiri

    craigiri Well-Known Member

    Google and Bing images are quite important in SOME searching, so folks should be careful about disallowing attachments if that is true.

    In our case, we want goog to get the images and associate them with what is in the text nearby (which I think is how they do some of their categorization)
     
  10. MagnusB

    MagnusB Well-Known Member

    I don't see any traffic from image search, and my site isn't image heavy, so I decided to save bandwidth instead. If you are getting traffic from image search, it might be a different story.
     
  11. craigiri

    craigiri Well-Known Member

    How did you check your google stats for images? The regular report doesn't have them, but after searching around on google I saw an article which pointed to webmaster tools where you can at least see queries and clicks on your images searches.....
    If the enclosed pic is anywhere near correct, most forums will not want to disable attachments!
    Screen Shot 2013-01-27 at 5.49.18 PM.png
     
  12. MagnusB

    MagnusB Well-Known Member

    None of my top ranking threads contain any images, and as I said, my site isn't image heavy (a total of 150 ish attachments, and allot of them is me posting screenshots on new site features). As I said, if image search is important to you, allow them. Be sure to allow guests to view attachments, cause that is why I blocked it (to prevent those 403 errors in GWT).
     

Share This Page