Robots and noindex of Xenforo Media Gallery pages?

arn

Well-known member
What are people doing with robots.txt and no index on media gallery pages? Having just synced my entire attachment database, I don't think abruptly tossing up 800k thin content pages seems like a good idea.

robots.txt is not a great option since it also blocks link previews from Twitter, Facebook etc...

So that leaves "no index". But probably actually "no index, no follow" -- since i don't really want Google spending time crawling 800k no index pages. Anyone doing anything specific or just letting Google at it? If I was starting from scratch I don't think I'd worry about it.

arn
 
Is this sufficient to prevent GG to index medias ?

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /media/
Disallow: /admin.php
Allow: /
 
That will prevent Google from crawling it, but not necessarily the same as no-index'ing it.

Looking at what I did, I added this:

<xf:head option="robots"><meta name="robots" content="noindex, nofollow" /></xf:head>

to

xfmg_category_view
xfmg_media_index
xfmg_media_user_index
xfmg_media_view
xfmg_whats_new_media

That will prevent indexing and also some crawling
 
Top Bottom