Is there a good reason to Disallow: /attachments/ in robots.txt?

[Stuart Wright]

On the basis that it's a good thing if Google can index attachments on your site - predominantly images - then why would you Disallow: /attachments/ ?

 

[Ozzy47]

Just a guess, maybe some admins don’t want their images indexed by google, or they are afraid of the bots using up bandwidth.

 

[Kevin]

On the basis that it's a good thing if Google can index attachments on your site - predominantly images - then why would you Disallow: /attachments/ ?

Do you allow guests to view attachments? Remember, the Google bot is a guest and if you have attachment viewing for guests turned off then the bot will be running into permission errors when trying to retrieve the attachment URLs.

 

[StarArmy]

Remember, the Google bot is a guest

IIRC, bots are in the BOTS usergroup and you can set special permissions for them.

 

[Kevin]

IIRC, bots are in the BOTS usergroup and you can set special permissions for them.

Bots usergroup?

 

[Chris D]

There is no bots usergroup by default in XF and @Kevin's previous post was accurate - bots are treated the same as guests.

 

[StarArmy]

Sorry, I must've imported that group from my previous software (phpBB3).

 

[jgaulard]

On the basis that it's a good thing if Google can index attachments on your site - predominantly images - then why would you Disallow: /attachments/ ?

The actual attachments path is /data/attachments/. The clickable path to open the attachment is just /attachments/. Two different paths. By blocking /attachments/ in the robots.txt file, you're not blocking any uploaded attachments, just the clickable paths to them. It's set up a little weird. The actual page (lightbox) that the image shows in is actually considered a page in itself.