1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.2 robots.txt

Discussion in 'XenForo Questions and Support' started by tommydamic68, Sep 19, 2013.

  1. tommydamic68

    tommydamic68 Well-Known Member

    Can anyone steer me in the right direction for a good robots.txt for my site?

    Thanks in advance!
  2. Brogan

    Brogan XenForo Moderator Staff Member

  3. System0

    System0 Active Member

    The allow command is not necessary in the Xenforo robots.txt file, as search engines are allowed to crawl unless you specify otherwise. The allow rule should be used for allowing files and directories that are inside a directory that has been disallowed.
  4. Michael Berg

    Michael Berg New Member

    So as worse case I can copy the text of the xenforo/robots.txt and create a new file and upload to the root of my domain>

    google webmaster tools is saying that i don't have a robots.txt file (which is true) It also says latest version seen 8/13/2014--that means I used to have one??

    also there is a robots.php file comes along with the sitemap add-on--is that related at all?

Share This Page