Why no robots.txt by default with XF 1.5?

[sajal]

There's no robots.txt in XF 1.5? What should be the default contents??

 

[Martok]

robots.txt isn't provided, that's an individual choice for each site what to allow or block from being crawled. The one that XenForo uses is here https://xenforo.com/robots.txt.

 

[sajal]

Thanks @Martok

 

[sajal]

@Martok, seems that it's related XF2, can you reference the one for XF1.5?

 

[Martok]

It's probably similar for XF 1.5 except for the whats-new entry which I think used to be find-new. Other than that, search this forum for other threads on robots.txt or look for XenForo 1.5 sites to see what their robots.txt looks like.

 

[sajal]

Thanks @Martok!