XF 1.4 Site map

[bsdinsight]

Hi All

I am not sure how long Xenforo Site Map submit to Google, but I can see my site from Sep 15, 2014 - can you help me to fix it?

 

[Martok]

You have something in your robots.txt file that disallows Google from crawling, yet you apparently are submitting links in sitemap to crawl that they aren't allowed to due to this.

I've checked your robots.txt file and can't see anything obvious but you've not given full details of the URLs that are blocked - we can only see part of them.

 

[bsdinsight]

Thanks for your information.

My robots.txt file is:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /account/
Disallow: /cdn-cgi/
Disallow: /cgi-bin/
Disallow: /conversations/
Disallow: /doubleclick/
Disallow: /draft/
Disallow: /find-new/
Disallow: /goto/
Disallow: /login/
Disallow: /posts/
Disallow: /unanswered/

Sitemap: http://forums.bsdinsight.com/sitemap.php

what information do you need more, please tell me and I can upload for you to help me to fix it.

 

[Martok]

what information do you need more, please tell me and I can upload for you to help me to fix it.

The screenshot you posted gives examples of the links that are blocked. However we can't see the full links, only the first part. So, what is blocked?

 

[bsdinsight]

thanks you

Here are 3 blocked links

• http://forums.bsdinsight.com/members/123macc321.217/
  
• http://forums.bsdinsight.com/members/12chomsao.2569/
  
• http://forums.bsdinsight.com/members/16nhannt.2482/

 

[Martok]

Is it reporting that member profiles are blocked then? Are there other URLs besides these that it reports as blocked? I ask because in your screenshot it shows 3144 problems, yet you only have 1580 members.

Have you edited the robots.text file recently? Is Google still reporting these URLs as blocked, or only back in September?

 

[bsdinsight]

Is it reporting that member profiles are blocked then? Yes
Are there other URLs besides these that it reports as blocked? No, I can see only 3 links.

Have you edited the robots.text file recently? No
Is Google still reporting these URLs as blocked, or only back in September? Only in Sep

Do you want to logon into my Google account? I can give you.

 

[Martok]

If it's not continuing to report these as blocked then I wouldn't worry about it. Just keep an eye on it to see if it happens again.

 

[bsdinsight]

it means there are no issues - is it correct?

 

[Martok]

I wouldn't worry about it if it's not currently reporting these as blocked but keep an eye on it over the next few weeks.

 

[Mike]

I would note that this is not related to when Google last read the sitemap; that will be listed elsewhere. It's related to specific errors it found on the sitemap at some point in time. It's possible those errors have since been rectified, but it certainly seems to show them for historical purposes.