XF 1.4 Does XF recommend a robots.txt file

[dethfire]

I'm migrating from vb3 to XF next week and for the longest time I've had a robots.txt file which blocked bots from indexing thin content like member pages etc. Does XF suggest this too? Is anyone blocking thin sections of their forum? thanks!

 

[Paul B]

It's up to you.

This is the one in place here: https://xenforo.com/robots.txt

 

[dethfire]

Does anyone block member pages? I have 200k+ members. Can't imagine Google wants to index them.

 

[CTXMedia]

There are further examples and discussion here too: https://xenforo.com/community/threads/whats-your-robots-txt-file-look-like-xenforo.66493/

 

[Rob]

@Mike I'd leave admin.php out of that file. You are just basically advertising the location of your back end entry point and rogue bots exist that crawl deliberately looking for disallowed entry points. Ok, it's secure, but why advertise where it is? It's not like they are likely to come across a link to the admin area anywhere on a public page so why would a bot even try and crawl it?

 

[CTXMedia]

Does anyone block member pages? I have 200k+ members. Can't imagine Google wants to index them.

I've seen various discussions about it and concluded it's really down to personal preference. Some prefer not to allow on the basis it's "thin" content and doesn't do your site any favours (dilutes your your overall indexed page pool); others prefer to have them indexed on the basis that any page is a potential entry point for visitors.

 

[Paul B]

I block /members/ on my site.

 

[dethfire]

It's up to you.

This is the one in place here: https://xenforo.com/robots.txt

It appears this file is now missing?

 

[Paul B]

It has been restored.