1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.4 Does XF recommend a robots.txt file

Discussion in 'XenForo Questions and Support' started by dethfire, Sep 11, 2014.

  1. dethfire

    dethfire Well-Known Member

    I'm migrating from vb3 to XF next week and for the longest time I've had a robots.txt file which blocked bots from indexing thin content like member pages etc. Does XF suggest this too? Is anyone blocking thin sections of their forum? thanks!
     
  2. Brogan

    Brogan XenForo Moderator Staff Member

    Niekiletta and dethfire like this.
  3. dethfire

    dethfire Well-Known Member

    Does anyone block member pages? I have 200k+ members. Can't imagine Google wants to index them.
     
  4. CyclingTribe

    CyclingTribe Well-Known Member

  5. Rob

    Rob Well-Known Member

    @Mike I'd leave admin.php out of that file. You are just basically advertising the location of your back end entry point and rogue bots exist that crawl deliberately looking for disallowed entry points. Ok, it's secure, but why advertise where it is? It's not like they are likely to come across a link to the admin area anywhere on a public page so why would a bot even try and crawl it?
     
  6. CyclingTribe

    CyclingTribe Well-Known Member

    I've seen various discussions about it and concluded it's really down to personal preference. Some prefer not to allow on the basis it's "thin" content and doesn't do your site any favours (dilutes your your overall indexed page pool); others prefer to have them indexed on the basis that any page is a potential entry point for visitors.
     
    dethfire likes this.
  7. Brogan

    Brogan XenForo Moderator Staff Member

    I block /members/ on my site.
     
    dethfire likes this.

Share This Page