XF 1.3 Xenforo and Google index problem

[DRaver]

I have a question for XenForo topics redirect.
First you have to take a look at the robots.txt from xenforo.com

Disallow: /community/posts/

Now I have made a google site query on xenforo.com ( site:xenforo.com/community/posts/) and this is the result: 3910 entries in Google.



WOW, around 4000 Hits although that should block the robots.txt these entries yet.

But that's not the main problem.

As I see, all /community/posts/..../ are 301 redirect to the real url but how can be these "post-sites" in Google index if they redirect?
This makes no sense.
Is this perhaps not a proper set 301 redirect?
That could be bad for the ranking in Google.

 

[Liam W]

As you can see, the URLs state that the robots.txt file stops Google from getting a description.

 

[DRaver]

But the title is in.

With a 301 redirect the site result in Google should be ZERO. That is my real problem.

 

[DRaver]

No other answers or solutions?

 

[Chris D]

Think of it this way. It's nothing to do with XenForo. XenForo has no control over what search engines decide to show or not show. A quick Google of the answer brings up this page:

http://www.sitepoint.com/why-pages-disallowed-in-robots-txt-still-appear-in-google/

Basically, Google can see these URLs. They're linked to everywhere. So it does make them available in its search results. What it does abide by, however, is that the content of the page is not indexed -- or at least not shown to be indexed. And that's basically all you can expect from using Disallow. You're Disallowing the content to be indexed, and Google is abiding by that. You'd have to physically search for community/posts/ to get any hits on it.

It has zero effect on ranking.

And yes, it is a proper 301 redirect:

 

[DRaver]

@Chris D my Problem ist not the URL´s blocked by robots.txt.

My problem is, why are post urls in the google index although every "post url" is a 301 redirect.
So a real 301 say to the bot. "Hey that content is not here. It is on this position. Forget the old position and follow the new path".
According to my understanding, with a real 301, the count of post url, in the google index should be zero and should be replaced by the real target. Right?

Why not with Xenforo?

 

[Chris D]

You would have to ask your question to Google.

There's no such thing as a 301 redirect that isn't "real". But that's academic anyway, because, as you can see from my screenshot, the HTTP response code issued when going to those links IS 301. So what Google then does with that is entirely down to Google. What more can XenForo do about that?

 

[DRaver]

I ask because I do not have that characteristics under all my other projects and that are hundrets of projects.
Something must be different and I just want to know what.

 

[Chris D]

Ask Google.

Here's one possible answer: http://webmasters.stackexchange.com/a/45618

 

[Chris D]

Also, unless all of your other projects are XenForo, of course things are going to behave differently.

One thing that XenForo has is a great deal of visibility of these links. /posts/ links are found in quoted posts, they're found on the forum index, they're easily accessible from quite a few places.

When Google sees these links it will log them. But, as per the declaration in robots.txt, it will not index them.

There really isn't a big issue here.

Quite frankly, I'm bored of seeing posts on the subject (not just this thread). Nothing is going to affect your SEO as much as your own content. And you should be dedicating your time to that instead of worrying about inane details that do not matter.

 

[DRaver]

I think XenForo is not so bad optimized for SEO, but there are still many points for improvement. For example the internal linking is not optimal.