1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.5 How to disallow crawling complete RM and one Node through robots.txt

Discussion in 'XenForo Questions and Support' started by Sunka, Aug 22, 2016.

  1. Sunka

    Sunka Well-Known Member

    What to add in robots.txt file to disallow Google and other spiders from crawling complete Resources manager and one complete Category (included all in it)?
    Thanks (y)
     
  2. Mike

    Mike XenForo Developer Staff Member

    If you just want to disallow the entire RM, then you can block /resources/. However, if you are trying to block a single category, you can't do this via robots.txt. You would need to modify the resource_view template:

    Code:
    <xen:if is="{$resource.resource_category_id} == 123">
    <xen:container var="$head.noindex"><meta name="robots" content="noindex" /></xen:container>
    </xen:if>
    (I haven't tested this, so please check the code is being output when you expect.)
     
    Sunka likes this.
  3. Sunka

    Sunka Well-Known Member

    To add this into robots.txt
    Disallow: /resources/

    I mean category in forum, not category in Resources manager.
    Example:
    somepage.com/categories/categoryname.23/
     
  4. Mike

    Mike XenForo Developer Staff Member

    Ah, the concept is similar but a different template/condition. You'd be checking thread_view and acting based on {$forum.node_id} .
     
    Sunka likes this.
  5. Sunka

    Sunka Well-Known Member

    If node ID is for example 123
    This will be right code:

    Code:
    <xen:if is="{$forum.node_id} == 123">
    <xen:container var="$head.noindex"><meta name="robots" content="noindex" /></xen:container>
    </xen:if>
    Where to put that code in thread_view template?
     
  6. Mike

    Mike XenForo Developer Staff Member

    That should be correct. It can go anywhere.
     
    Sunka likes this.

Share This Page