XF 1.2 robots.txt

[tommydamic68]

Can anyone steer me in the right direction for a good robots.txt for my site?

Thanks in advance!

 

[Paul B]

You already posted in the existing thread: http://xenforo.com/community/threads/robots-txt.16735/page-2#post-637428

Default robots.txt is here: http://xenforo.com/robots.txt

Edit it to suit your needs.

 

[System0]

The allow command is not necessary in the Xenforo robots.txt file, as search engines are allowed to crawl unless you specify otherwise. The allow rule should be used for allowing files and directories that are inside a directory that has been disallowed.

 

[Michael Berg]

So as worse case I can copy the text of the xenforo/robots.txt and create a new file and upload to the root of my domain>

google webmaster tools is saying that i don't have a robots.txt file (which is true) It also says latest version seen 8/13/2014--that means I used to have one??

also there is a robots.php file comes along with the sitemap add-on--is that related at all?