Robots.txt

[Martok]

Xenforo install is is public_html/community - so would this be correct then?

No. robots.txt goes in your root, not the directory that Xenforo is installed in. public_html presumably is your root.

 

[tommydamic68]

Got it! Thanks!

 

[tommydamic68]

No. robots.txt goes in your root, not the directory that Xenforo is installed in. public_html presumably is your root.

And the /community is correct in the .txt file then?

 

[Martok]

And the /community is correct in the .txt file then?

I think you need to have a bit of confidence in yourself rather than have to feel you need to check everything.

If you look at Brogan's earlier post, this should help you to write the correct paths for your robots.txt file based on wherever you have yours installed.

This is the robots.txt file at XenForo.com, which is installed in the /community directory.

http://xenforo.com/robots.txt

User-agent: *
Disallow: /community/find-new/
Disallow: /community/forums/-/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

 

[jauburn]

How do I keep pages like smilies, trophies, notable members, lost password, and other silly stuff like that from being indexed?

 

[Jeremy]

Add a Disallow directive with the path to the pages you want to disallow.

 

[Mr Lucky]

This is the robots.txt file at XenForo.com, which is installed in the /community directory.

http://xenforo.com/robots.txt

User-agent: *
Disallow: /community/find-new/
Disallow: /community/forums/-/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

I'm slightly baffled.

If the robots/txt is installed in the /community directory as opposed to the root, then why is /community/ included in the disallow. I thought that should be there if the file is i the root, but not the folder itself ???

 

[Paul B]

The software is installed in the /community directory.
Robots.txt is always in the root.

 

[Brent W]

Just know that attachments entry in robots.txt will keep all of your attachments out of Google Images...

 

[Mr Lucky]

The software is installed in the /community directory.
Robots.txt is always in the root.

That's what I thought, but this says the robots.txt is in /community

This is the robots.txt file at XenForo.com, which is installed in the /community directory.

So that is a typo I presume.

 

[Paul B]

No typo.

The second half of the sentence refers to where the software is installed.

 

[Lexy]

how to add robot.txt to my forum, can i use this ones from Xenforo and just change site URL

 

[Paul B]

Create a text file named robots.txt, populate it, and upload it to your server.

 

[Lexy]

so just need add this and upload it to my server

User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

 

[Paul B]

Is your forum installed in /community?

 

[Lexy]

yap i see this, no its not installed in community

 

[Paul B]

Then use the correct path for where your forum is installed.

 

[Lexy]

so community its /myforrumname/find-new/ andthis is not goes inside public_html

 

[Lexy]

i find this tutorial
1) Edit this file to change "/forum/" to the correct relative path from your base URL, for example if your forum was at "domain.com/sites/community", then you'd use "/sites/community/"
# 2) Rename the file to 'robots.txt' and move it to your web root (public_html, www, or htdocs)
# 3) Edit the file to remove this comment (anything above the dashed line, including the dashed line
#
# NOTES:
# Even though wild cards and pattern matching are not part of the robots.txt specification, many search bots understand and make use of them
#------------------------ REMOVE THIS LINE AND EVERYTHING ABOVE SO THAT User-agent: * IS THE FIRST LINE ------------------------------------------

 

[BizBuzz]

Quick question ... is this information found in this thread still relevant for version 1.4.x of Xenforo? I am trying to find a way for search engines to stop searching member profiles. I have already set the permissions for the user group Unregistered / Unconfirmed to Not Set (No) ... do I need to do more than that to stop search engines from searching member profiles?