XF 1.2 robots.txt


Active member
The allow command is not necessary in the Xenforo robots.txt file, as search engines are allowed to crawl unless you specify otherwise. The allow rule should be used for allowing files and directories that are inside a directory that has been disallowed.

Michael Berg

New member
So as worse case I can copy the text of the xenforo/robots.txt and create a new file and upload to the root of my domain>

google webmaster tools is saying that i don't have a robots.txt file (which is true) It also says latest version seen 8/13/2014--that means I used to have one??

also there is a robots.php file comes along with the sitemap add-on--is that related at all?