How can I add robots/bots to my forum?
E.g.: Total: 318 (members: 27, guests: 223, robots: 68)
Robots.txt and sitemaps should always be in synch. In other words: if the xenforo sitemap sends a URL to search engines, then it should not be...
Noticed on this websites robots.txt that their disallow ended with an "Allow /"
I think it would be a good idea if there was an option to make the sitemap follow robots.txt rules. It would mean the sitemap more accurately...
Separate names with a comma.