I'm migrating from vb3 to XF next week and for the longest time I've had a robots.txt file which blocked bots from indexing thin content like member pages etc. Does XF suggest this too? Is anyone blocking thin sections of their forum? thanks!
@Mike I'd leave admin.php out of that file. You are just basically advertising the location of your back end entry point and rogue bots exist that crawl deliberately looking for disallowed entry points. Ok, it's secure, but why advertise where it is? It's not like they are likely to come across a link to the admin area anywhere on a public page so why would a bot even try and crawl it?
I've seen various discussions about it and concluded it's really down to personal preference. Some prefer not to allow on the basis it's "thin" content and doesn't do your site any favours (dilutes your your overall indexed page pool); others prefer to have them indexed on the basis that any page is a potential entry point for visitors.