XF 1.4 Stuff blocked by robot.txt gets indexed

[Pavle123]

Few members on the forum suggested a while ago that blocking stuff with robots.txt will get search engine not to index those things.

However take a look at attachments that got indexed. It looks pretty bad in search engines. I would like to keep my indexed things nice, clean and content-rich, but these sort of pages bother me.

Is there a way to noindex attachments? I´ve been using *******´s Advanced NoIndex, but they do not have option for that.

 

[Mike]

And yet nobody from Support team seams to care to explain to us how to put rel="nofollow" on those useless pages that gets indexed.

Aside from the fact that we can't provide official support for template edits, I explained the general process above. It's no different elsewhere: find the template containing the link to the content you're referring to and add the nofollow attribute.

 

[Pavle123]

@Mike, I want to noindex those pages not nofollow them. Sorry I made a mistake in previous post.

But you already said that putting
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Is not possible in attachments? Am I right? Can this somehow be fixed in future updates?

 

[Mike]

As attachments aren't HTML pages, you can't put that HTML tag in. You may be able to add a HTTP header, but that is something that would require custom development. It's not something we would do by default.

The whole thing with attachment indexing is really a moot point if you don't allow guests to view attachments as the attachment URL will be returning a 403 and that shouldn't be indexed.

 

[Pavle123]

The whole thing with attachment indexing is really a moot point if you don't allow guests to view attachments as the attachment URL will be returning a 403 and that shouldn't be indexed.

@Mike So, is this the way to do it? Please check image bellow and tell me. Or should it be done somewhere else?

 

[Mike]

If Guests can't view an attachment, then it can't be index, so that should be sufficient (assuming you're setting it on the right group, right forum if applicable, etc).