1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots File

Discussion in 'XenForo Questions and Support' started by lostincable, Mar 10, 2013.

  1. lostincable

    lostincable Active Member

    Hi all, I have imported from VBulletin and had a couple of questions about the Robots.txt file.

    Firstly I am using the redirects script for VB4 to redirect threads and forums and my URL's are working perfectly but I am wondering whether the redirect will tell google that the link has moved vs thinking it is duplicate content i.e 301 redirect?

    Second I am using the below robots file and was wondering whether this is acceptable and whether there is maybe a reason to not block any of the below or maybe even things that are missed?

    I got this from the sitemaps mod...

    Code:
    User-agent: *
    Disallow: /account/
    Disallow: /help/
    Disallow: /login/
    Disallow: /lost-password/
    Disallow: /misc/style/
    Disallow: /online/
    Disallow: /recent-activity/
    Disallow: /register/
    Disallow: /search/
    Disallow: /admin.php
    Disallow: /index.php?account/
    Disallow: /index.php?help/
    Disallow: /index.php?login/
    Disallow: /index.php?lost-password/
    Disallow: /index.php?misc/style/
    Disallow: /index.php?online/
    Disallow: /index.php?posts/
    Disallow: /index.php?recent-activity/
    Disallow: /index.php?register/
    Disallow: /index.php?search/
    Disallow: /admin.php
    Allow: /
    Cheers!
     
  2. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    The redirect scripts issue a 301 response which instructs Google that the content has permanently moved.

    Here is the robots file on this site for reference:

    http://xenforo.com/robots.txt
     

Share This Page