1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Automated no-index of light pages (zero replies, thin content)

Discussion in 'Forum Management' started by cmeinck, Jun 7, 2013.

  1. cmeinck

    cmeinck Well-Known Member

    Google seems intent on pushing UGC (user generated content) down in the search results. There are some active conversations in their product forums where Google reps have suggested that sites de-index UGC that is light (zero replies, thin content). As a forum admin, this sounds sort of insane. 88k discussions and almost 1 million messages, this is impossible to do without some level of automation. It would also have to tie in with whatever sitemap tool you are using.

    Google's algorithms could change for the better, but I'm not so sure. I think as forum admins, we need to spoon feed our best content and leave the thin stuff as no-index. I'd welcome ideas on how to best accomplish this going forward with XenForo.

    Source, Discussion on Google's Product Forums
     
  2. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Stick with what works:

    http://en.wikipedia.org/wiki/PageRank

    Focus on promoting your highest quality content. As a hypothetical example, if you were running a tech forum you might have a separate forum with technical guides. Those guides are moderated and approved. Then you can go out posting links to your guides on other highly ranked sites with relevant content. That kind of exposure is what increases your page rank.
     
    0ptima likes this.
  3. MagnusB

    MagnusB Well-Known Member

    I disallow indexing on thin content forums, such as introduction forums. Also thinking about doing it on the offtopic forum, but from time to time there are interesting discussions there, so I have not yet decided on that.
     
  4. feldon30

    feldon30 Well-Known Member

    Wouldn't it be extremely trivial to add no-index to threads with only 1 post based on subforum?
     
    cmeinck likes this.
  5. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    feldon30 likes this.
  6. cmeinck

    cmeinck Well-Known Member


    I agree with this, but what Google's asking for now is to not waste their time with thin content. Let's take my site for example. I've got 88k threads. I'm sure that some percentage are thin, low-quality posts. Google's algo now takes into account thin content. If there was a way for me to set no-index to threads with only one post, it automates the process and provides me with a better chance at succeeding. Point being that lower quality, thin content could impact your great threads.

    I'd love to find ways of customizing the sitemap tools we have (created by 3rd parties) to remove these from the sitemaps.

    My personal opinion is that we can no longer trust that Google will understand that forums have some thin content, along with robust discussion. I think we'll continue to see forums getting squeezed by these algos.

    Daniweb is a perfect example.
     
    Mr Lucky likes this.
  7. cmeinck

    cmeinck Well-Known Member

    Are you using robots.txt or is there a way to set threads within those forums to no-index?
     
  8. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    cmeinck likes this.
  9. MagnusB

    MagnusB Well-Known Member

    I use robots.txt, was the simplest way and I wanted the entire introduction forum out of the way.
     
  10. cmeinck

    cmeinck Well-Known Member

  11. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    You would have to check the node_id when adding that to thread_view.
     
  12. cmeinck

    cmeinck Well-Known Member

    Let's say I want threads in nodes 81 and 80 to be set to nonidex using the code you provided here.

    Code:
    <xen:container var="$head.robots">
        <meta name="robots" content="noindex" /></xen:container>
    How would you add this snippet to thread_view so that it appears in the head?
     
  13. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Code:
    <xen:if is="in_array({$thread.node_id}, array(80,81))">
    
    <xen:container var="$head.robots">
        <meta name="robots" content="noindex" /></xen:container>
    
    </xen:if>
    
     
    cmeinck and Marcus like this.
  14. cmeinck

    cmeinck Well-Known Member

    I've tried adding it to thread_view and receive the following error.

    Screen Shot 2013-07-22 at 9.31.12 AM.png

    I'm grabbing the nodes from view-source and this code I've found:
    Screen Shot 2013-07-22 at 9.32.46 AM.png

    Here's what I was using:

    Code:
    <xen:if is="in_array({$thread.node_id}, array(83,80))">
    
    <xen:container var="$head.robots">
        <meta name="robots" content="noindex" /></xen:container>
    </xen:if>
     
  15. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Use this:

    Code:
    <xen:if is="in_array({$thread.node_id}, array(80,81))">
    
    <xen:container var="$head.robots">
        <meta name="robots" content="noindex" /></xen:container>
    
    </xen:if>
    
    It's the same thing, minus an invisible character that somehow creeped in there.
     
    cmeinck likes this.
  16. cmeinck

    cmeinck Well-Known Member

    Thanks @Jake Bunce. This is invaluable to me. Such an awesome fix to help me combat Google's Panda. Between this thread_view code and member_view, I've effectively removed a ton of non-relevant content from Google's index. (y)(y)

    Hoping one of the sitemap devs comes up with an option to not include threads from selected forum categories. I'm submitting Off Topic to Google in my sitemap and then telling them not to index it.
     
  17. cmeinck

    cmeinck Well-Known Member

    Is there a way to do this for a specific forum category? I've been able to tackle the threads. For example, all threads within Off Topic are now set to noindex. However, the category itself is not set to noindex. Setting the category to noindex would complete the job. Would this work if I used it on the forum_view template?

    Code:
    <xen:if is="in_array({$thread.node_id})">
    
    <xen:container var="$head.robots">
        <meta name="robots" content="noindex" /></xen:container>
    
    </xen:if>
     
  18. cmeinck

    cmeinck Well-Known Member

    A couple of questions regarding this topic. Is it possible to have something in place that would automate the addition of a noindex tag to threads that have zero replies and are X number of days old. If so, who could contact about creating the custom code required to make an add-on or script for this purpose. It should also work in reverse. If an older thread receives a reply, than it should remove the noindex tag, allowing it to be indexed in Google.

    I'd gladly pay for this to happen.
     
    ForestForTrees and RoldanLT like this.
  19. Live Free

    Live Free Active Member

    Do we know that there's any real benefit from no-indexing thin content? So having thin content hurts quality content on the same url?
     
  20. cmeinck

    cmeinck Well-Known Member

    Thin content is one of the many factors which can lead to a Panda penalty. Based on my research into the topic, there are forums which have been penalized, but have come back by noindexing zero reply content.

    I'm just hoping that a. this is possible and b. someone will take me up on the offer.
     
    ForestForTrees likes this.

Share This Page