• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.
  • This forum has been archived. New threads and replies may not be made. All add-ons/resources that are active should be migrated to the Resource Manager. See this thread for more information.

My robots.txt file

Puntocom

Well-known member
#1
I have this since I don't want members pages, search, help, misc, login, register pages indexed from Google, and only threads and forums. Maybe the members section is useful for some sites.... anyway, I prefer to get indexed only the pages with revelant content since Google starts indexing only some pages from new sites.

I installed XenForo on root directory (/), so my robots.txt is:

User-agent: *
Disallow: /misc
Disallow: /help
Disallow: /search
Disallow: /members
Disallow: /register
Disallow: /login
Disallow: /online
Disallow: /lost-password
Disallow: /recent-activity
 
#10
Sorry for this stupid question...

when you disallow /members you dont want memberlist to be indexed.. I understand that
But you also dont let search engines index /members/username, ie. the user profiles
At times where in vb2 you only had the name and the amount of posts, it was understandable,

but today you have richer content in profiles with images, texts etc.. And users are ever so happy when they find their profile listed in google.
Now my questions:
1. Am I missing something? (because i saw that in nearly ALL boards, the profiles are excluded from indexing - There must be a reason i am missing)

2. Is there a way to exclude the memberlist but allow the profiles?

Luc
 
#12
Should there be something added to stop search engines going into folders / files?

Code:
Disallow: /data/
Disallow: /internal_data/
Disallow: /js/
Disallow: /library/
Disallow: /styles/
Disallow: /admin.php
Disallow: /admindav.php
 

Decent60

Well-known member
#13
Sorry for this stupid question...

when you disallow /members you dont want memberlist to be indexed.. I understand that
But you also dont let search engines index /members/username, ie. the user profiles
At times where in vb2 you only had the name and the amount of posts, it was understandable,

but today you have richer content in profiles with images, texts etc.. And users are ever so happy when they find their profile listed in google.
Now my questions:
1. Am I missing something? (because i saw that in nearly ALL boards, the profiles are excluded from indexing - There must be a reason i am missing)

2. Is there a way to exclude the memberlist but allow the profiles?

Luc
Allowing bots is a double-edge sword. Sure, allowing your posts and members showing up on Google will get more people on your site (especially if they are trying to locate friend), however, it also allows adbots to search your site and display ads relevant to content found on your site. There are also bots that want to strictly mine your site for info about something/someone. Protecting your user's privacy is part of your job as a host. If they are willing to divulge information to fellow members, that's their priority.
 
#14
thanks for the info, but IMHO when you encounter bots that:
There are also bots that want to strictly mine your site for info about something/someone.
I dont think those bots will respect the robots.txt file.

But for the rest, I think it would be a nice addon to have: an extra setting:
Allow Search Engines to index your profile: YES/NO
adding a noindex,nofollow tag on that particular page if set to NO
Luc
 

Puntocom

Well-known member
#15
Should there be something added to stop search engines going into folders / files?

Code:
Disallow: /data/
Disallow: /internal_data/
Disallow: /js/
Disallow: /library/
Disallow: /styles/
Disallow: /admin.php
Disallow: /admindav.php
I haven't got indexed any of these files without adding them to robots.txt. admin.php has the meta noindex.
 

Brogan

XenForo moderator
Staff member
#16
I have it on good authority that if there's a page that shouldn't be indexed, it is marked it up as such.

So a robots.txt file isn't required unless you want to specifically block pages which are allowed to be indexed, such as member profiles.
 

Rigel Kentaurus

Well-known member
#17
Today I crafted mine. I don't use mod_rewrite because I don't think it is necessary for SEO anymore

Code:
User-agent: *
Disallow: /index.php?help/*
Disallow: /index.php?misc/style*
Disallow: /index.php?members/*
Disallow: /index.php?online/*
Disallow: /index.php?XenStaff/*
Disallow: /index.php?profile-posts/*
Disallow: /index.php?register/*
Disallow: /index.php?search/*
Disallow: /index.php?login/*
Disallow: /index.php?lost-password/*
Disallow: /index.php?recent-activity/*
That should do it, for now :) I am still debating if I should index member pages. For now they are disabled.
Now I only need a sitemap ...