My robots.txt file

[Michael Dance]

Are you not using pretty URLs ?

not sure. i didnt change anything when i was installing it.

 

[Simon]

I added also:

Disallow: /attachments - As I dont want images to be indexed.

I also see that quite a few pages with /goto are being indexed. Not sure to add that to disallow or not?

 

[Digital Doctor]

Here's another robots.txt

most search engines look for robots.txt in the site root.
it basically tells them what not to index.
this is my robots.txt:

User-agent: *
Disallow: /community/attachments/
Disallow: /community/misc/
Disallow: /community/help/
Disallow: /community/search/
Disallow: /community/members/
Disallow: /community/register/
Disallow: /community/login/
Disallow: /community/online/
Disallow: /community/lost-password/
Disallow: /community/recent-activity/
Allow: /

 

[Peggy]

Does anyone use a robots.txt file for your blogs?

 

[Deleted member 745]

This is the file that XF uses, Any idea what the /-/ does?

User-agent: *
Disallow: /community/find-new/
Disallow: /community/forums/-/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

 

[Paul B]

It's not actually shown to guests, but it's the mark all forums read link.