XF 2.2 Robots.txt is not accessible or doesn't exist

[JasonBrody]

Hi All,
Though I had never checked it earlier, the concern was raised after I changed my hosting.
Recently, Google Console has not been indexing my site, and its sitemap has also not been read since the last few weeks.
I am getting robots blocked and redirection error messages too.

When I checked for robots.txt file (ex: https://yourdomain.com/robots.txt ) then found that it doesn't exist/accessible. What should I do? By default, the XenForo package doesn't provide any robots.txt files.

Please suggest.

 

[Paul B]

Just create one.

This is ours - adjust to suit: https://xenforo.com/robots.txt

 

[JasonBrody]

Just create one.

This is ours - adjust to suit: https://xenforo.com/robots.txt

Does it exist/created by default ? if yes, it's possible that somehow the file got deleted ?

 

[Paul B]

XF does not ship with a robots.txt file.

 

[Mendalla]

I am getting robots blocked and redirection error messages too.

That doesn't sound like a missing robots.txt, though. Absent robots.txt, robots should have wide open access to your site since there would be no controls on them unless you've got them set somewhere else like a firewall. The redirection errors, too, suggest something else to me but I'm not a total expert on web stuff since we have other people managing the webservers at work.

 

[JasonBrody]

What if I re-install from this stage ? will I lost all data ?

 

[Mendalla]

What if I re-install from this stage ? will I lost all data ?

As @Brogan said, the missing robot.txt isn't going to be fixed this way since the file is not part of XF. Download the one he supplied and then upload it to the root directory of your site. e.g. mine (which is blank since I don't really use it) is in public_html. cPanel File Manager or similar should allow you to do this if you have admin access to the web server.

 

[JasonBrody]

Thanks @Mendalla @Brogan , let me create and check the google console indexing issues if they have anything to do with the robots.txt or not.
I will update the results here if it sorts it out.

 

[JasonBrody]

Just create one.

This is ours - adjust to suit: https://xenforo.com/robots.txt

@Brogan this includes sitemap is .xml file, what is recommended ? XML or PHP one ?

 

[mrdoitnyce]

@Brogan this includes sitemap is .xml file, what is recommended ? XML or PHP one ?

where do i get the sitemap xml file?

 

[Chromaniac]

should be visible from here: forum.domain/admin.php?logs/sitemap/. basically points to forum.domain/sitemap.php. and you can use sitemap.xml as well. both works fine.