• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.2 robots.txt

System0

Active member
#3
The allow command is not necessary in the Xenforo robots.txt file, as search engines are allowed to crawl unless you specify otherwise. The allow rule should be used for allowing files and directories that are inside a directory that has been disallowed.
 
#4
So as worse case I can copy the text of the xenforo/robots.txt and create a new file and upload to the root of my domain>

google webmaster tools is saying that i don't have a robots.txt file (which is true) It also says latest version seen 8/13/2014--that means I used to have one??

also there is a robots.php file comes along with the sitemap add-on--is that related at all?