1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.4 Preventing Search Engine from crawling members list

Discussion in 'XenForo Questions and Support' started by CommanderMadi, Feb 6, 2015.

  1. CommanderMadi

    CommanderMadi Member

    On my board, I don't give the guest a privilege to view users profile. As robots are techincally guests, I found 25 URL crawl errors in google webmaster, all related to denial of access to members list and users profiles. What should I do to prevent google to crawl here? Does it require a robot.txt?
  2. djawir

    djawir Member

    yes using robots.txt

    Disallow: /members/
    ibrian likes this.
  3. CommanderMadi

    CommanderMadi Member

    What user-agent?
  4. melbo

    melbo Well-Known Member

    Use a *

    See example:

    User-agent: *
    Disallow: /community/find-new/
    Disallow: /community/account/
    Disallow: /community/attachments/
    Disallow: /community/goto/
    Disallow: /community/posts/
    Disallow: /community/login/
    Disallow: /community/admin.php
    Allow: /
    Sitemap: https://xenforo.com/community/sitemap.php
    CommanderMadi likes this.
  5. CommanderMadi

    CommanderMadi Member

    Thanks man.
  6. creativeforge

    creativeforge Active Member

    Sorry for awakening a sleeping thread, but where do you enter these values?
  7. Andrej

    Andrej Well-Known Member

    The Web Robots Pages
  8. creativeforge

    creativeforge Active Member

    So... it doesn't sound too strong a deterrent to keep robots from crawling 'members lists.' Is this what most people do here? What other option is there?
    Last edited: Jul 11, 2015

Share This Page