XF 2.2 Does XF has a default robots.txt file?

KhanTastic

Active member
Hi friends,

Recently I got so much spam on my XF forum, I updated it to the latest version just now and everything looks fine. I deleted all the spamming posts and blocked the spam users. I just would like to know if XF has any default robots.txt file as I can not see in my root folder and I just created one by seeing the codes from a website in google search engine. I am not sure if my robots.txt is correct or not.
Because my forum is not indexing at all in search engine, the posts are not getting any views. I am thinking maybe my robots.txt is not good and blocking search engine from crawling my site.

I would appreciate if someone can confirm to me if XF really has a robots.txt file by default, if it has then can you paste the codes here for me so that I can replace it on my robots.txt file. Or should I totally get rid of robots.txt file which I created, does XF really need that robots.txt file?

Thank you so much in advance, awaiting yours.

Best regards,
Cheers!
 
What's the difference between sitemap.php and sitemap.xml? The content seems the same, which one i have to use for SE?
They are identical, you can use one or the other. Google still prefers the .xml sitemap extension.

 
You just create one to suit your site and needs.

This is the one from here: https://xenforo.com/robots.txt
Just to say that yesterday Bing reported the indexing problem to me because I have used the robots.txt recommended on this site and used by many others users.

XenForo IndexNow is using links with the format https://sitename.com/posts/123456/ and in the robots.txt file there is Disallow: /posts/
So be careful because this robots.txt blocks all IndexNow links!

I hope I was helpful...
 
Back
Top Bottom