1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

RM 1.1 Add Download Block to robots.txt for Resource Manager

Discussion in 'Resource Manager Support' started by Sadik B, Jul 16, 2015.

  1. Sadik B

    Sadik B Well-Known Member

    I received this email after converting my site to xenforo.

    WMT_Email.jpg

    Upon checking I found that the resource download url were causing authorization errors for Google's crawler.

    crawlErrors.gif

    I have over 12K+ files in resource manager and this is causing a havoc with crawl errors in WMT.

    I blocked this by putting this simple rule in robots.txt

    Disallow: /resources/*/download

    So if you use the RM, you should also add this to your robots.txt.
     

Share This Page