Puntocom
Well-known member
I have this since I don't want members pages, search, help, misc, login, register pages indexed from Google, and only threads and forums. Maybe the members section is useful for some sites.... anyway, I prefer to get indexed only the pages with revelant content since Google starts indexing only some pages from new sites.
I installed XenForo on root directory (/), so my robots.txt is:
User-agent: *
Disallow: /misc
Disallow: /help
Disallow: /search
Disallow: /members
Disallow: /register
Disallow: /login
Disallow: /online
Disallow: /lost-password
Disallow: /recent-activity
I installed XenForo on root directory (/), so my robots.txt is:
User-agent: *
Disallow: /misc
Disallow: /help
Disallow: /search
Disallow: /members
Disallow: /register
Disallow: /login
Disallow: /online
Disallow: /lost-password
Disallow: /recent-activity