• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.5 Robots.txt blocking access to attachments. Good or bad?

Stuart Wright

Well-known member
#1
I'm wondering what the reason is behind blocking access to attachments in the robots.txt file.
What are the disadvantages of allowing the indexing of attachments?
 

Steve F

Well-known member
#5
If an attachment is in a post viewable by a bot (or guest) but the URL to that attachment is behind a permission to view (or not view by guest) then it will certainly show an error in google webmaster tools (crawl errors).
 

Stuart Wright

Well-known member
#6
If an attachment is in a post viewable by a bot (or guest) but the URL to that attachment is behind a permission to view (or not view by guest) then it will certainly show an error in google webmaster tools (crawl errors).
That's not the case with us. So is there any other reason to block them?
 
#8
I don't block attachment crawls on any of my sites. The reason being, i've never seen anyone successfully link to an attachment where guest view permissions have been disabled from the usergroup permissions side. I have also been told by one of Google's Dev Engineers, their robots by default no longer crawl forums for other than the topic thread meta and body post meta (May 2016). Makes sense as i used to be able to find people by username through Google Search. Now, you have to be more far more specific with your long-tail search.

I do digital marketing by trade and spend a lot of time working with GA Pro, Search Console and Adobe Analytics among many other tools. I do not see any crawl errors and have not for the last 15 or so years with many of my personal sites. Why would you want to give a guest those benefits anyhow? In many cases, they are basically taking without permission for their own personal use, if not... maybe even as a competitor. If they cannot be bothered to even become a member of a community which has provided them a bounty, why support it? This has always been a personal pet peeve of mine

The permissions have always worked for me and never had to the disable in robots. Your mileage may vary of course...
 

Brent W

Well-known member
#9
People continue to over think SEO.

Code:
User-agent: *
Allow: /

User-agent: BoardReader
Disallow: /
Across all of our sites.

Google is smart. Just let it do its thing.