XF 1.4 Sitemap warnings

Stuart Wright

Well-known member
Folks, Google webmaster tools is telling me
Sitemap contains urls which are blocked by robots.txt. Count: 39,915
Example: Sitemap:www.avforums.com/sitemap.php?c=34 Value:https://static.avforums.com/data/avatars/l/111/111117.jpg?1299678671

But my sitemap, copied from Xenforo's, does not block /data/avatars
Here is my robots.txt:
User-agent: *
Disallow: /forums/find-new/
Disallow: /forums/account/
Disallow: /forums/attachments/
Disallow: /forums/goto/
Disallow: /forums/posts/
Disallow: /forums/login/
Disallow: /forums/admin.php
Allow: /

Sitemap: https://www.avforums.com/sitemap.php
Confused. Where is the problem, please?
 
It's technically correct.

The robots.txt it is referring to is at static.avforums.com:

https://static.avforums.com/robots.txt

That disallows everything in terms of the static. subdomain. It may be ok to just leave it. Or adjusting the robots.txt on the static subdomain to allow /data/avatars/ may be necessary.

It's worth noting, by the way, that your robots.txt is wrong. Most of those URLs don't exist on your site. XenForo is installed in the root. Should be /find-new/ /account/ /attachments/ etc.
 
  • Like
Reactions: HWS
Back
Top Bottom