Robots and noindex of Xenforo Media Gallery pages?

[arn]

What are people doing with robots.txt and no index on media gallery pages? Having just synced my entire attachment database, I don't think abruptly tossing up 800k thin content pages seems like a good idea.

robots.txt is not a great option since it also blocks link previews from Twitter, Facebook etc...

So that leaves "no index". But probably actually "no index, no follow" -- since i don't really want Google spending time crawling 800k no index pages. Anyone doing anything specific or just letting Google at it? If I was starting from scratch I don't think I'd worry about it.

arn

 

[Old Nick]

Hello, some news about this ?
How to prevent search engines to index medias from Gallery ?
Thanks.

 

[Old Nick]

Is this sufficient to prevent GG to index medias ?

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /media/
Disallow: /admin.php
Allow: /

 

[arn]

That will prevent Google from crawling it, but not necessarily the same as no-index'ing it.

Looking at what I did, I added this:

<xf:head option="robots"><meta name="robots" content="noindex, nofollow" /></xf:head>

to

xfmg_category_view
xfmg_media_index
xfmg_media_user_index
xfmg_media_view
xfmg_whats_new_media

That will prevent indexing and also some crawling

 

[Old Nick]

Thank you very much for sharing this.