1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. This forum has been archived. New threads and replies may not be made. All add-ons/resources that are active should be migrated to the Resource Manager. See this thread for more information.

My robots.txt file

Discussion in 'Tips and Guides [Archive]' started by Puntocom, Oct 19, 2010.

  1. Puntocom

    Puntocom Well-Known Member

    I have this since I don't want members pages, search, help, misc, login, register pages indexed from Google, and only threads and forums. Maybe the members section is useful for some sites.... anyway, I prefer to get indexed only the pages with revelant content since Google starts indexing only some pages from new sites.

    I installed XenForo on root directory (/), so my robots.txt is:

    User-agent: *
    Disallow: /misc
    Disallow: /help
    Disallow: /search
    Disallow: /members
    Disallow: /register
    Disallow: /login
    Disallow: /online
    Disallow: /lost-password
    Disallow: /recent-activity
     
  2. Michael

    Michael Active Member

    Thanks for the share!
     
    Puntocom likes this.
  3. anotheralias

    anotheralias Well-Known Member

    Thanks! I had totally forgotten about the robots.txt page.
     
  4. Ra

    Ra Active Member

    /threads/*/poll
     
  5. DSF

    DSF Well-Known Member

    What ist correct? :oops:

    ...
    Disallow: /misc
    or
    Disallow: /forum/misc
     
  6. Puntocom

    Puntocom Well-Known Member

    Depending on where you installed XenForo.
     
  7. DSF

    DSF Well-Known Member

    In this case in /forum
    I think, a leading "/" ist ok?

    Disallow: /forum/misc
     
  8. Puntocom

    Puntocom Well-Known Member

    Yes.
     
  9. Dietmar

    Dietmar Member

    Thanks for this useful hint! :)
     
    Puntocom likes this.
  10. Luciano

    Luciano Member

    Sorry for this stupid question...

    when you disallow /members you dont want memberlist to be indexed.. I understand that
    But you also dont let search engines index /members/username, ie. the user profiles
    At times where in vb2 you only had the name and the amount of posts, it was understandable,

    but today you have richer content in profiles with images, texts etc.. And users are ever so happy when they find their profile listed in google.
    Now my questions:
    1. Am I missing something? (because i saw that in nearly ALL boards, the profiles are excluded from indexing - There must be a reason i am missing)

    2. Is there a way to exclude the memberlist but allow the profiles?

    Luc
     
  11. Puntocom

    Puntocom Well-Known Member

    # of indexed pages is limited, specially in starting sites, so I prefer to get indexed only forums and threads.
     
    Blandt likes this.
  12. kuyenmotdivad

    kuyenmotdivad Active Member

    Should there be something added to stop search engines going into folders / files?

    Code:
    Disallow: /data/
    Disallow: /internal_data/
    Disallow: /js/
    Disallow: /library/
    Disallow: /styles/
    Disallow: /admin.php
    Disallow: /admindav.php
    
     
  13. Decent60

    Decent60 Well-Known Member

    Allowing bots is a double-edge sword. Sure, allowing your posts and members showing up on Google will get more people on your site (especially if they are trying to locate friend), however, it also allows adbots to search your site and display ads relevant to content found on your site. There are also bots that want to strictly mine your site for info about something/someone. Protecting your user's privacy is part of your job as a host. If they are willing to divulge information to fellow members, that's their priority.
     
  14. Luciano

    Luciano Member

    thanks for the info, but IMHO when you encounter bots that:
    I dont think those bots will respect the robots.txt file.

    But for the rest, I think it would be a nice addon to have: an extra setting:
    Allow Search Engines to index your profile: YES/NO
    adding a noindex,nofollow tag on that particular page if set to NO
    Luc
     
  15. Puntocom

    Puntocom Well-Known Member

    I haven't got indexed any of these files without adding them to robots.txt. admin.php has the meta noindex.
     
  16. Brogan

    Brogan XenForo Moderator Staff Member

    I have it on good authority that if there's a page that shouldn't be indexed, it is marked it up as such.

    So a robots.txt file isn't required unless you want to specifically block pages which are allowed to be indexed, such as member profiles.
     
  17. Rigel Kentaurus

    Rigel Kentaurus Well-Known Member

    Today I crafted mine. I don't use mod_rewrite because I don't think it is necessary for SEO anymore

    Code:
    User-agent: *
    Disallow: /index.php?help/*
    Disallow: /index.php?misc/style*
    Disallow: /index.php?members/*
    Disallow: /index.php?online/*
    Disallow: /index.php?XenStaff/*
    Disallow: /index.php?profile-posts/*
    Disallow: /index.php?register/*
    Disallow: /index.php?search/*
    Disallow: /index.php?login/*
    Disallow: /index.php?lost-password/*
    Disallow: /index.php?recent-activity/*
    
    That should do it, for now :) I am still debating if I should index member pages. For now they are disabled.
    Now I only need a sitemap ...
     
  18. Michael Dance

    Michael Dance Active Member

    very handy, when installing how do you select the non folder so its like your index.php?..../ ?
    i prefere that way
     
  19. Blandt

    Blandt Well-Known Member

    Are you not using pretty URLs ?
     
  20. Peggy

    Peggy Well-Known Member

    Excellent info, thanks!
     

Share This Page