• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Adding NoIndex to Some Pages

#1
Hi,

I just did a search on my domain name on Google and realised that a few redundant pages are being indexed.

How can I add NoIndex Follow to the meta for the following types of pages:
  • mydomain.com/help/*
  • mydomain.com/search/*

URL such as the ones below are appearing in my Google index:
  • mydomain.com/search/member?user_id=1
  • mydomain.com/search/?type=profile_post
  • mydomain.com/help/bbcodes

Thanks!
 

Rigel Kentaurus

Well-known member
#2
Hi,

I just did a search on my domain name on Google and realised that a few redundant pages are being indexed.

How can I add NoIndex Follow to the meta for the following types of pages:
  • mydomain.com/help/*
  • mydomain.com/search/*
URL such as the ones below are appearing in my Google index:
  • mydomain.com/search/member?user_id=1
  • mydomain.com/search/?type=profile_post
  • mydomain.com/help/bbcodes
Thanks!
Search this forums for robots.txt, some people have posted their examples
 
#3
I have already added to following to my robots.txt for the relevant user agents

Disallow: /help/
Disallow: /search/

But, what I was looking for is how I an edit the template (which template, or other methods etc) so that I can add the No Index Follow to the meta for the above pages.
 

MagnusB

Well-known member
#4
I have already added to following to my robots.txt for the relevant user agents

Disallow: /help/
Disallow: /search/

But, what I was looking for is how I an edit the template (which template, or other methods etc) so that I can add the No Index Follow to the meta for the above pages.
That is just redundant, if you have the robots.txt like that, you have disallowed google from indexing that. It will take quite some time before they will remove it from their index though, but they will not recrawl it.
 
#5
Hi,

Even after disallowing the pages in robots.txt, some of them still show in the search results. The meta description beneath them will become "A description for this result is not available because of this site’s robots.txt"

So I think it is good practice to add NoIndex to the meta. Unfortunately, I don't have a good idea on how to do so in Xenforo.

Source: https://developers.google.com/webmasters/control-crawl-index/docs/faq
However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.
 
#7
Hi There!

First off... this is not "Johnstevens21", but his SEO consultant posting on his behalf.

Afraid to say, but JK is 100% correct on this. Robots.txt will never remove a URL from Google's index (or prevent it from getting indexed) no matter how long you wait. The only way to do so is with the meta robots noindex tag which needs to go in the <head> of pages you don't want indexed.

I've located the templates for certain page types (member profiles etc). Is it safe to say I can just add the normal HTML of the tag;

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

I can add that to the template, or do I need any special coding? I am totally brand new to Xenforo (but not brand new to SEO :) ) ...so some confirmation on that would be excellent.

Thanks!
-Dan
 

Jake Bunce

XenForo moderator
Staff member
#8
Hi There!

First off... this is not "Johnstevens21", but his SEO consultant posting on his behalf.

Afraid to say, but JK is 100% correct on this. Robots.txt will never remove a URL from Google's index (or prevent it from getting indexed) no matter how long you wait. The only way to do so is with the meta robots noindex tag which needs to go in the <head> of pages you don't want indexed.

I've located the templates for certain page types (member profiles etc). Is it safe to say I can just add the normal HTML of the tag;

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

I can add that to the template, or do I need any special coding? I am totally brand new to Xenforo (but not brand new to SEO :) ) ...so some confirmation on that would be excellent.

Thanks!
-Dan
Find the root template of the page:

http://xenforo.com/community/threads/1-0-0-b1-how-to-identify-the-root-template-of-a-page.5591/

And add this to the top:

Code:
<xen:container var="$head.robots">
	<meta name="robots" content="noindex" /></xen:container>
 

Jake Bunce

XenForo moderator
Staff member
#11
It will add the "noindex" meta tag to the head. I'm not sure what you mean by "follow". It just adds that meta tag which instructs search engines not to index that page.
 
#13
Find the root template of the page:

http://xenforo.com/community/threads/1-0-0-b1-how-to-identify-the-root-template-of-a-page.5591/

And add this to the top:

Code:
<xen:container var="$head.robots">
    <meta name="robots" content="noindex" /></xen:container>
I tried using this method to noindex the tag pages generated by xentag but when I view the page source the meta tag doesn't appear. Does it only display to robots? If not does xenforo version 1.4.3 require a different method?
 
#15
Alright - so if I go to 'member_view' and add the code to the top of the template - will it make all the member profiles 'noindex' but 'follow'?
Yes, it will be "follow". The "follow" tag is actually redundant; only the "nofollow" tag changes anything and is not to be used internally like "noindex".
 

dethfire

Well-known member
#16
Problem with adding noindex to some pages is that canonical tags will still be there. John Mueller said not to have a canonical tag and a noindex tag on the same page.
 

Pavle123

Active member
#18
I would also like to know how can we noindex particular forums, pages and member profiles?
Is there an addition for that? Or it has to be done manually? How come this is not implemented in the XF? Why would anyone want members to be shown in search?

I would like to know how you guys use <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> on your forums? Where do you put it? Can it be done on attachments and other junk pages?
 

Pavle123

Active member
#19
@Jake Bunce Jake, can you please clarify this one more time :

Can you tell us which templates should we edit and what code should we put and where in order to :
  • NOINDEX, NOFOLLOW : Members profile, particular forum and subforum?
 

cmeinck

Well-known member
#20
@Jake Bunce Jake, can you please clarify this one more time :

Can you tell us which templates should we edit and what code should we put and where in order to :
  • NOINDEX, NOFOLLOW : Members profile, particular forum and subforum?
This post by @Jake Bunce will show you how to find the appropriate template and what code to include. I typically put it near the top of the template. Once you do, fire up a browser like Chrome with the no-index plugin. The resulting page should show a no-index. Why would you NOFOLLOW? I think noindex, follow is sufficient.