Adding NoIndex to Some Pages

J

JK Maken

New member
Hi,

I just did a search on my domain name on Google and realised that a few redundant pages are being indexed.

How can I add NoIndex Follow to the meta for the following types of pages:
  • mydomain.com/help/*
  • mydomain.com/search/*

URL such as the ones below are appearing in my Google index:
  • mydomain.com/search/member?user_id=1
  • mydomain.com/search/?type=profile_post
  • mydomain.com/help/bbcodes

Thanks!
 
Sort by date Sort by votes
JK Maken said:
Hi,

I just did a search on my domain name on Google and realised that a few redundant pages are being indexed.

How can I add NoIndex Follow to the meta for the following types of pages:
  • mydomain.com/help/*
  • mydomain.com/search/*
URL such as the ones below are appearing in my Google index:
  • mydomain.com/search/member?user_id=1
  • mydomain.com/search/?type=profile_post
  • mydomain.com/help/bbcodes
Thanks!
Click to expand...
Search this forums for robots.txt, some people have posted their examples
 
Upvote 0 Downvote
I have already added to following to my robots.txt for the relevant user agents

Disallow: /help/
Disallow: /search/

But, what I was looking for is how I an edit the template (which template, or other methods etc) so that I can add the No Index Follow to the meta for the above pages.
 
Upvote 0 Downvote
JK Maken said:
I have already added to following to my robots.txt for the relevant user agents

Disallow: /help/
Disallow: /search/

But, what I was looking for is how I an edit the template (which template, or other methods etc) so that I can add the No Index Follow to the meta for the above pages.
Click to expand...
That is just redundant, if you have the robots.txt like that, you have disallowed google from indexing that. It will take quite some time before they will remove it from their index though, but they will not recrawl it.
 
Upvote 0 Downvote
Hi,

Even after disallowing the pages in robots.txt, some of them still show in the search results. The meta description beneath them will become "A description for this result is not available because of this site’s robots.txt"

So I think it is good practice to add NoIndex to the meta. Unfortunately, I don't have a good idea on how to do so in Xenforo.

Source: https://developers.google.com/webmasters/control-crawl-index/docs/faq
However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.
Click to expand...
 
Upvote 0 Downvote
Hi There!

First off... this is not "Johnstevens21", but his SEO consultant posting on his behalf.

Afraid to say, but JK is 100% correct on this. Robots.txt will never remove a URL from Google's index (or prevent it from getting indexed) no matter how long you wait. The only way to do so is with the meta robots noindex tag which needs to go in the <head> of pages you don't want indexed.

I've located the templates for certain page types (member profiles etc). Is it safe to say I can just add the normal HTML of the tag;

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

I can add that to the template, or do I need any special coding? I am totally brand new to Xenforo (but not brand new to SEO :) ) ...so some confirmation on that would be excellent.

Thanks!
-Dan
 
Upvote 0 Downvote
Johnstevens21 said:
Hi There!

First off... this is not "Johnstevens21", but his SEO consultant posting on his behalf.

Afraid to say, but JK is 100% correct on this. Robots.txt will never remove a URL from Google's index (or prevent it from getting indexed) no matter how long you wait. The only way to do so is with the meta robots noindex tag which needs to go in the <head> of pages you don't want indexed.

I've located the templates for certain page types (member profiles etc). Is it safe to say I can just add the normal HTML of the tag;

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

I can add that to the template, or do I need any special coding? I am totally brand new to Xenforo (but not brand new to SEO :) ) ...so some confirmation on that would be excellent.

Thanks!
-Dan
Click to expand...

Find the root template of the page:

http://xenforo.com/community/threads/1-0-0-b1-how-to-identify-the-root-template-of-a-page.5591/

And add this to the top:

Code: 
<xen:container var="$head.robots">
	<meta name="robots" content="noindex" /></xen:container>
 
Upvote 0 Downvote
Upvote 0 Downvote
Jake Bunce said:
Find the root template of the page:

http://xenforo.com/community/threads/1-0-0-b1-how-to-identify-the-root-template-of-a-page.5591/

And add this to the top:

Code: 
<xen:container var="$head.robots">
    <meta name="robots" content="noindex" /></xen:container>
Click to expand...

I tried using this method to noindex the tag pages generated by xentag but when I view the page source the meta tag doesn't appear. Does it only display to robots? If not does xenforo version 1.4.3 require a different method?
 
Upvote 0 Downvote
TheBigK said:
Alright - so if I go to 'member_view' and add the code to the top of the template - will it make all the member profiles 'noindex' but 'follow'?
Click to expand...

Yes, it will be "follow". The "follow" tag is actually redundant; only the "nofollow" tag changes anything and is not to be used internally like "noindex".
 
Upvote 0 Downvote
Problem with adding noindex to some pages is that canonical tags will still be there. John Mueller said not to have a canonical tag and a noindex tag on the same page.
 
Upvote 0 Downvote
I would also like to know how can we noindex particular forums, pages and member profiles?
Is there an addition for that? Or it has to be done manually? How come this is not implemented in the XF? Why would anyone want members to be shown in search?

I would like to know how you guys use <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> on your forums? Where do you put it? Can it be done on attachments and other junk pages?
 
Upvote 0 Downvote
@Jake Bunce Jake, can you please clarify this one more time :

Can you tell us which templates should we edit and what code should we put and where in order to :
  • NOINDEX, NOFOLLOW : Members profile, particular forum and subforum?
 
Upvote 0 Downvote
Pavle123 said:
@Jake Bunce Jake, can you please clarify this one more time :

Can you tell us which templates should we edit and what code should we put and where in order to :
  • NOINDEX, NOFOLLOW : Members profile, particular forum and subforum?
Click to expand...

This post by @Jake Bunce will show you how to find the appropriate template and what code to include. I typically put it near the top of the template. Once you do, fire up a browser like Chrome with the no-index plugin. The resulting page should show a no-index. Why would you NOFOLLOW? I think noindex, follow is sufficient.
 
Upvote 0 Downvote
You must log in or register to reply here.

Similar threads

Steffen
Not a bug Please add missing rel="canonical" meta tags to some pages (login, register, lost-password, help, poll results)
Replies
10
Views
1K
TLDR
TLDR
TheBigK
  • Suggestion Suggestion
Lack of interest Allow ability to add 'noindex' to member profile pages
Replies
6
Views
974
DJ XtAzY
DJ XtAzY
Back
Top Bottom