dethfire
Well-known member
I've never seen these before. What are they used for? Does Google recommend using these?User-agent: Adsbot-Google
Disallow:
User-agent: Googlebot-Mobile
Disallow:
I've never seen these before. What are they used for? Does Google recommend using these?User-agent: Adsbot-Google
Disallow:
User-agent: Googlebot-Mobile
Disallow:
User-agent: Mediapartners-Google
Disallow:
User-agent: AhrefsBot
Disallow: /
User-agent: Baidu
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider-video
Disallow: /
User-agent: Baiduspider-image
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: EasouSpider
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: linkdexbot
Disallow: /
User-agent: linkdexbot-mobile
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: meanpathbot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: proximic
Disallow: /
User-agent: Rogerbot
Disallow: /
User-agent: SiteBot
Disallow: /
User-agent: sogou
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: spbot
Disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: Twiceler
Disallow: /
User-agent: URLAppendBot
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: Yeti
Disallow: /
User-Agent: *
Disallow: /?page=
Disallow: /find-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Disallow: /members/
Disallow: /conversations/
Allow: /
Sitemap: http://mysite.co.uk/sitemap.php
No i hadn't either, i think i got them from this thread or one linked here. So thought i would add them, which i did today. I guess seeing as its essentially allowing them it could not do any harm.
Actually looking at it more i don't think i need the mobile bot as i have User-Agent: * which allows all bots to crawl, and the adsbot looks like it is for adsense so for me i will be removing those now.
This is what i have now.
Code:User-agent: AhrefsBot User-agent: Baidu User-agent: Baiduspider User-agent: Baiduspider-video User-agent: Baiduspider-image User-agent: Cliqzbot User-agent: Diffbot User-agent: DotBot User-agent: EasouSpider User-agent: Exabot User-agent: linkdexbot User-agent: linkdexbot-mobile User-agent: magpie-crawler User-agent: meanpathbot User-agent: MJ12bot User-agent: NaverBot User-agent: omgilibot User-agent: proximic User-agent: Rogerbot User-agent: SiteBot User-agent: sogou User-agent: sogou spider User-agent: Sogou web spider User-agent: spbot User-agent: trendictionbot User-agent: Twiceler User-agent: URLAppendBot User-agent: Yandex User-agent: YoudaoBot User-agent: Yeti Disallow: / User-Agent: * Disallow: /?page= Disallow: /find-new/ Disallow: /account/ Disallow: /attachments/ Disallow: /goto/ Disallow: /posts/ Disallow: /login/ Disallow: /admin.php Disallow: /members/ Disallow: /conversations/ Allow: / Sitemap: http://mysite.co.uk/sitemap.php
Just remember, an entry in robots.txt isn't really a "block" against those visitors, it is just a request to the visitor that they may or may not honor. Legitimate 'bots' will honor the requests.I have Baidu blocked via robots.txt yet they still visit my site
Slight improvement/fix by removing all the unnecessary multiple disallow's
http://www.robotstxt.org/norobots-rfc.txt -> section 3.2When I researched this I couldn't find a definitive answer as to whether it was advisable to group or not group disallows in this way
Been working fine for me for years!If anyone has any experience to offer in this regard - to confirm if grouping disallows works as expected on your server - then please let us know.
Code:User-agent: BoardReader Disallow: /
User-agent: *
Disallow: /account/
Disallow: /admin.php
Disallow: /attachments/
Disallow: /conversations/
Disallow: /find-new/
Disallow: /goto/
Disallow: /login/
Disallow: /members/*/trophies
Disallow: /misc/style
Disallow: /posts/
Disallow: /register/
Disallow: /search/
Allow: /
Sitemap: https://www.gamingforums.net/sitemap.php
It stops search engines seeing anything yourforum.tld/attachments. Thumbnails for attachments are internal_data I think? It makes them ignore anything to do with the attachments directory.Hi,
Can anyone explain to me why are you all using
Disallow: /attachments/ in robots.txt?
Does this block images being shown in search engine or it has nothing to do with that?
Out of interest, why does Xenforo's own robots.txt file, along with many other Xenforo site (including @Brogan's) disallow /find-new/ ?
If you noindex you must also remove the canonical tagAs discussed million times, admins should find a way to let us put rel noindex tag, but nobody seams to care about that until forums begin to receive messages like that.
We use essential cookies to make this site work, and optional cookies to enhance your experience.