Robots.txt

And the /community is correct in the .txt file then?
I think you need to have a bit of confidence in yourself rather than have to feel you need to check everything. :)

If you look at Brogan's earlier post, this should help you to write the correct paths for your robots.txt file based on wherever you have yours installed. :)

This is the robots.txt file at XenForo.com, which is installed in the /community directory.

http://xenforo.com/robots.txt

User-agent: *
Disallow: /community/find-new/
Disallow: /community/forums/-/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /
 
How do I keep pages like smilies, trophies, notable members, lost password, and other silly stuff like that from being indexed?
 
This is the robots.txt file at XenForo.com, which is installed in the /community directory.

http://xenforo.com/robots.txt

User-agent: *
Disallow: /community/find-new/
Disallow: /community/forums/-/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /


I'm slightly baffled.

If the robots/txt is installed in the /community directory as opposed to the root, then why is /community/ included in the disallow. I thought that should be there if the file is i the root, but not the folder itself ???
 
The software is installed in the /community directory.
Robots.txt is always in the root.
 
No typo.

The second half of the sentence refers to where the software is installed.
 
how to add robot.txt to my forum, can i use this ones from Xenforo and just change site URL
 
so just need add this and upload it to my server

User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /
 
i find this tutorial
1) Edit this file to change "/forum/" to the correct relative path from your base URL, for example if your forum was at "domain.com/sites/community", then you'd use "/sites/community/"
# 2) Rename the file to 'robots.txt' and move it to your web root (public_html, www, or htdocs)
# 3) Edit the file to remove this comment (anything above the dashed line, including the dashed line
#
# NOTES:
# Even though wild cards and pattern matching are not part of the robots.txt specification, many search bots understand and make use of them
#------------------------ REMOVE THIS LINE AND EVERYTHING ABOVE SO THAT User-agent: * IS THE FIRST LINE ------------------------------------------
 
Quick question ... is this information found in this thread still relevant for version 1.4.x of Xenforo? I am trying to find a way for search engines to stop searching member profiles. I have already set the permissions for the user group Unregistered / Unconfirmed to Not Set (No) ... do I need to do more than that to stop search engines from searching member profiles?
 
Back
Top Bottom