Discussion in 'XenForo Questions and Support' started by tommydamic68, Sep 19, 2013.
Can anyone steer me in the right direction for a good robots.txt for my site?
Thanks in advance!
You already posted in the existing thread: http://xenforo.com/community/threads/robots-txt.16735/page-2#post-637428
Default robots.txt is here: http://xenforo.com/robots.txt
Edit it to suit your needs.
The allow command is not necessary in the Xenforo robots.txt file, as search engines are allowed to crawl unless you specify otherwise. The allow rule should be used for allowing files and directories that are inside a directory that has been disallowed.
So as worse case I can copy the text of the xenforo/robots.txt and create a new file and upload to the root of my domain>
google webmaster tools is saying that i don't have a robots.txt file (which is true) It also says latest version seen 8/13/2014--that means I used to have one??
also there is a robots.php file comes along with the sitemap add-on--is that related at all?
Separate names with a comma.