• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.4 Sitemap warnings

Stuart Wright

Well-known member
#1
Folks, Google webmaster tools is telling me
Sitemap contains urls which are blocked by robots.txt. Count: 39,915
Example: Sitemap:www.avforums.com/sitemap.php?c=34 Value:https://static.avforums.com/data/avatars/l/111/111117.jpg?1299678671

But my sitemap, copied from Xenforo's, does not block /data/avatars
Here is my robots.txt:
User-agent: *
Disallow: /forums/find-new/
Disallow: /forums/account/
Disallow: /forums/attachments/
Disallow: /forums/goto/
Disallow: /forums/posts/
Disallow: /forums/login/
Disallow: /forums/admin.php
Allow: /

Sitemap: https://www.avforums.com/sitemap.php
Confused. Where is the problem, please?
 

Chris D

XenForo developer
Staff member
#2
It's technically correct.

The robots.txt it is referring to is at static.avforums.com:

https://static.avforums.com/robots.txt

That disallows everything in terms of the static. subdomain. It may be ok to just leave it. Or adjusting the robots.txt on the static subdomain to allow /data/avatars/ may be necessary.

It's worth noting, by the way, that your robots.txt is wrong. Most of those URLs don't exist on your site. XenForo is installed in the root. Should be /find-new/ /account/ /attachments/ etc.