• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

RM 1.1 Add Download Block to robots.txt for Resource Manager

Sadik B

Well-known member
#1
I received this email after converting my site to xenforo.

WMT_Email.jpg

Upon checking I found that the resource download url were causing authorization errors for Google's crawler.

crawlErrors.gif

I have over 12K+ files in resource manager and this is causing a havoc with crawl errors in WMT.

I blocked this by putting this simple rule in robots.txt

Disallow: /resources/*/download

So if you use the RM, you should also add this to your robots.txt.