Adding Disallow: /attachments/ to robots.txt

[JoyFreak]

If I add:

Disallow: /attachments/

to my Robots.txt.. Does that mean attachments such as images won't get index'd on search engines?

I've seen XenForo using it in their robots.txt but I've also noticed attachments on here getting index'd, so it has me bit confused.

 

[Jeremy P]

to my Robots.txt.. Does that mean attachments such as images won't get index'd on search engines?

Yeah.

I've seen XenForo using it in their robots.txt but I've also noticed attachments on here getting index'd, so it has me bit confused.

Are you sure? Attachments aren't visible to guests here to the best of my knowledge. Attachment thumbnails are visible to guests even when attachments themselves aren't, so I suppose it's possible those might get indexed.

 

[JoyFreak]

Yeah.


Are you sure? Attachments aren't visible to guests here to the best of my knowledge. Attachment thumbnails are visible to guests even when attachments themselves aren't, so I suppose it's possible those might get indexed.

Try this in private window, it definitely view able and found on Google: https://xenforo.com/community/attac...ty_threads_1-000-messages-156559_-png.188488/

 

[dethfire]

If you don't disallow you'll get a soft 404s in Google Search Console for each attachment. XF is just built wrong for this.

 

[JoyFreak]

If you don't disallow you'll get a soft 404s in Google Search Console for each attachment. XF is just built wrong for this.

I haven't had any errors for attachments in GSC and I haven't disallowed it since launch of my board. So I don't know how much of that is true.

 

[Jeremy P]

Try this in private window, it definitely view able and found on Google: https://xenforo.com/community/attac...ty_threads_1-000-messages-156559_-png.188488/

Ah, I guess some attachments are visible to guests then. Not sure why they would be indexed by Google.

 

[JoyFreak]

Ah, I guess some attachments are visible to guests then. Not sure why they would be indexed by Google.

Yeah strange behaviour.

 

[dethfire]

I haven't had any errors for attachments in GSC and I haven't disallowed it since launch of my board. So I don't know how much of that is true.

Odd, I have 85k soft 404s and they are all attachment images.

 

[JoyFreak]

Odd, I have 85k soft 404s and they are all attachment images.

It might just be random. I get some soft 404s for other URLs randomly and I just validate a fix and they pass after a week. It’s odd behaviour but try it.

 

[dethfire]

It might just be random. I get some soft 404s for other URLs randomly and I just validate a fix and they pass after a week. It’s odd behaviour but try it.

soft 404s can't be validated

 

[JoyFreak]

soft 404s can't be validated

Oh must be another error for me then. Definitely check that out though because you shouldn’t be having that problem!