XF 2.1 [Google Webmasters/Console] - Warnings blocked by robots.txt

[Albanian]

Hi,

If i look in my Google Webmaster Console i am getting 66 warnings because there are some urls indexed by Google but blocked by robots.txt

The urls are almost the same and are like :

https://www.######.com/posts/1324631/reactions

https://www.######.com/posts/1324427/reactions

https://www.######.com/posts/1324040/reactions

Did somebody experience this also, or does somebody know how i can fix this. And what is the exactly reason that /post are standard blocked in robots by xenforo. Has this something to do with the amount of unrelated posts in your forum that google doesnt like?

Kind Regards

 

[djbaxter]

It's a combination of two things:

1. avoiding duplicate content (e.g., the thread is already indexed so you don't need each individual post in that thread indexed); and

2. more importantly in this case avoiding having googlebot crawling things that shouldn't be indexed anyway and wouldn't be indexed even if googlebot could crawl them. like smilies.

Google Search Console warns you about blocked URLs but that doesn't necessarily mean it's a problem. It's only a problem if you see something being blocked that you don't want blocked. Most of these warnings can be ignored.

 

[Albanian]

It's a combination of two things:

1. avoiding duplicate content (e.g., the thread is already indexed so you don't need each individual post in that thread indexed); and

2. more importantly in this case avoiding having googlebot crawling things that shouldn't be indexed anyway and wouldn't be indexed even if googlebot could crawl them. like smilies.

Google Search Console warns you about blocked URLs but that doesn't necessarily mean it's a problem. It's only a problem if you see something being blocked that you don't want blocked. Most of these warnings can be ignored.

Thanks for making it clear