1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Adding NoIndex to Some Pages

Discussion in 'XenForo Questions and Support' started by JK Maken, Oct 24, 2012.

  1. JK Maken

    JK Maken New Member

    Hi,

    I just did a search on my domain name on Google and realised that a few redundant pages are being indexed.

    How can I add NoIndex Follow to the meta for the following types of pages:
    • mydomain.com/help/*
    • mydomain.com/search/*

    URL such as the ones below are appearing in my Google index:
    • mydomain.com/search/member?user_id=1
    • mydomain.com/search/?type=profile_post
    • mydomain.com/help/bbcodes

    Thanks!
     
  2. Rigel Kentaurus

    Rigel Kentaurus Well-Known Member

    Search this forums for robots.txt, some people have posted their examples
     
    Jake Bunce likes this.
  3. JK Maken

    JK Maken New Member

    I have already added to following to my robots.txt for the relevant user agents

    Disallow: /help/
    Disallow: /search/

    But, what I was looking for is how I an edit the template (which template, or other methods etc) so that I can add the No Index Follow to the meta for the above pages.
     
  4. MagnusB

    MagnusB Well-Known Member

    That is just redundant, if you have the robots.txt like that, you have disallowed google from indexing that. It will take quite some time before they will remove it from their index though, but they will not recrawl it.
     
    Jake Bunce likes this.
  5. JK Maken

    JK Maken New Member

    Hi,

    Even after disallowing the pages in robots.txt, some of them still show in the search results. The meta description beneath them will become "A description for this result is not available because of this site’s robots.txt"

    So I think it is good practice to add NoIndex to the meta. Unfortunately, I don't have a good idea on how to do so in Xenforo.

    Source: https://developers.google.com/webmasters/control-crawl-index/docs/faq
     
  6. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Google will update eventually now that you have disallowed those URLs. Just gotta wait.
     
  7. Johnstevens21

    Johnstevens21 New Member

    Hi There!

    First off... this is not "Johnstevens21", but his SEO consultant posting on his behalf.

    Afraid to say, but JK is 100% correct on this. Robots.txt will never remove a URL from Google's index (or prevent it from getting indexed) no matter how long you wait. The only way to do so is with the meta robots noindex tag which needs to go in the <head> of pages you don't want indexed.

    I've located the templates for certain page types (member profiles etc). Is it safe to say I can just add the normal HTML of the tag;

    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

    I can add that to the template, or do I need any special coding? I am totally brand new to Xenforo (but not brand new to SEO :) ) ...so some confirmation on that would be excellent.

    Thanks!
    -Dan
     
    IPF, XenBurger and TheLaw like this.
  8. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Find the root template of the page:

    http://xenforo.com/community/threads/1-0-0-b1-how-to-identify-the-root-template-of-a-page.5591/

    And add this to the top:

    Code:
    <xen:container var="$head.robots">
    	<meta name="robots" content="noindex" /></xen:container>
    
     
    equinox, otto, Bugfix and 3 others like this.
  9. Johnstevens21

    Johnstevens21 New Member

    Jake

    Sweeeettt!! Perfect, thank you!!

    -Dan
     
  10. TheBigK

    TheBigK Well-Known Member

  11. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    It will add the "noindex" meta tag to the head. I'm not sure what you mean by "follow". It just adds that meta tag which instructs search engines not to index that page.
     
    TheBigK likes this.
  12. dethfire

    dethfire Well-Known Member

    Pavle123 likes this.
  13. cherylferraro

    cherylferraro Member

    I tried using this method to noindex the tag pages generated by xentag but when I view the page source the meta tag doesn't appear. Does it only display to robots? If not does xenforo version 1.4.3 require a different method?
     
  14. cherylferraro

    cherylferraro Member

    Nevermind @Jake Bunce ... the meta is displaying on the main tag page now. My guess is that the pages are cached so it will also display on the other pages when they are refreshed.
     
    Last edited: Jan 7, 2015
  15. HammerDown

    HammerDown Active Member

    Yes, it will be "follow". The "follow" tag is actually redundant; only the "nofollow" tag changes anything and is not to be used internally like "noindex".
     
  16. dethfire

    dethfire Well-Known Member

    Problem with adding noindex to some pages is that canonical tags will still be there. John Mueller said not to have a canonical tag and a noindex tag on the same page.
     
  17. cherylferraro

    cherylferraro Member

    I was not aware of this... Thanks for sharing.
     
  18. Pavle123

    Pavle123 Active Member

    I would also like to know how can we noindex particular forums, pages and member profiles?
    Is there an addition for that? Or it has to be done manually? How come this is not implemented in the XF? Why would anyone want members to be shown in search?

    I would like to know how you guys use <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> on your forums? Where do you put it? Can it be done on attachments and other junk pages?
     
  19. Pavle123

    Pavle123 Active Member

    @Jake Bunce Jake, can you please clarify this one more time :

    Can you tell us which templates should we edit and what code should we put and where in order to :
    • NOINDEX, NOFOLLOW : Members profile, particular forum and subforum?
     
  20. cmeinck

    cmeinck Well-Known Member

    This post by @Jake Bunce will show you how to find the appropriate template and what code to include. I typically put it near the top of the template. Once you do, fire up a browser like Chrome with the no-index plugin. The resulting page should show a no-index. Why would you NOFOLLOW? I think noindex, follow is sufficient.
     

Share This Page