RM 1.1 Add Download Block to robots.txt for Resource Manager

Sadik B

Well-known member
I received this email after converting my site to xenforo.

WMT_Email.webp

Upon checking I found that the resource download url were causing authorization errors for Google's crawler.

crawlErrors.gif

I have over 12K+ files in resource manager and this is causing a havoc with crawl errors in WMT.

I blocked this by putting this simple rule in robots.txt

Disallow: /resources/*/download

So if you use the RM, you should also add this to your robots.txt.
 
Back
Top Bottom