• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.4 Site map

bsdinsight

Active member
#1
Hi All

I am not sure how long Xenforo Site Map submit to Google, but I can see my site from Sep 15, 2014 - can you help me to fix it?


forum.jpg
 

Martok

Well-known member
#2
You have something in your robots.txt file that disallows Google from crawling, yet you apparently are submitting links in sitemap to crawl that they aren't allowed to due to this.

I've checked your robots.txt file and can't see anything obvious but you've not given full details of the URLs that are blocked - we can only see part of them.
 

bsdinsight

Active member
#3
Thanks for your information.

My robots.txt file is:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /account/
Disallow: /cdn-cgi/
Disallow: /cgi-bin/
Disallow: /conversations/
Disallow: /doubleclick/
Disallow: /draft/
Disallow: /find-new/
Disallow: /goto/
Disallow: /login/
Disallow: /posts/
Disallow: /unanswered/

Sitemap: http://forums.bsdinsight.com/sitemap.php
what information do you need more, please tell me and I can upload for you to help me to fix it.
 

Martok

Well-known member
#4
what information do you need more, please tell me and I can upload for you to help me to fix it.
The screenshot you posted gives examples of the links that are blocked. However we can't see the full links, only the first part. So, what is blocked?
 

Martok

Well-known member
#6
Is it reporting that member profiles are blocked then? Are there other URLs besides these that it reports as blocked? I ask because in your screenshot it shows 3144 problems, yet you only have 1580 members.

Have you edited the robots.text file recently? Is Google still reporting these URLs as blocked, or only back in September?
 

bsdinsight

Active member
#7
Is it reporting that member profiles are blocked then? Yes
Are there other URLs besides these that it reports as blocked? No, I can see only 3 links.

Have you edited the robots.text file recently? No
Is Google still reporting these URLs as blocked, or only back in September? Only in Sep

Do you want to logon into my Google account? I can give you.
 

Martok

Well-known member
#8
If it's not continuing to report these as blocked then I wouldn't worry about it. Just keep an eye on it to see if it happens again.
 

Martok

Well-known member
#10
I wouldn't worry about it if it's not currently reporting these as blocked but keep an eye on it over the next few weeks.
 

Mike

XenForo developer
Staff member
#11
I would note that this is not related to when Google last read the sitemap; that will be listed elsewhere. It's related to specific errors it found on the sitemap at some point in time. It's possible those errors have since been rectified, but it certainly seems to show them for historical purposes.