Sadik B
Well-known member
I received this email after converting my site to xenforo.
Upon checking I found that the resource download url were causing authorization errors for Google's crawler.
I have over 12K+ files in resource manager and this is causing a havoc with crawl errors in WMT.
I blocked this by putting this simple rule in robots.txt
Disallow: /resources/*/download
So if you use the RM, you should also add this to your robots.txt.
Upon checking I found that the resource download url were causing authorization errors for Google's crawler.
I have over 12K+ files in resource manager and this is causing a havoc with crawl errors in WMT.
I blocked this by putting this simple rule in robots.txt
Disallow: /resources/*/download
So if you use the RM, you should also add this to your robots.txt.
