1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

test site block all spiders

Discussion in 'XenForo Questions and Support' started by mmoore5553, Apr 24, 2012.

  1. mmoore5553

    mmoore5553 Well-Known Member

    I forgot where the setting is to block all search engines and spiders . Let me know where it is . Also if i just make a robot.txt to disallow all is that the best way to not duplicate content.
  2. mmoore5553

    mmoore5553 Well-Known Member

    I feel like a newbie lol
  3. mmoore5553

    mmoore5553 Well-Known Member

    I just did this in my robots.txt

    User-agent: *
    Disallow: /

    is that all i need to do ? Which i put in my sub directory with my test board.
  4. MagnusB

    MagnusB Well-Known Member

    Yes, that will stop all robots that respects robots.txt. Your best way to stop it from being indexed though is to prevent the link from being found, or protect the directory in some manner.

  5. mmoore5553

    mmoore5553 Well-Known Member

    Are you talking about just password protecting the folder ? Would that be enough to block everything. I just do not want duplicate content.
  6. MagnusB

    MagnusB Well-Known Member

    Yes, then they wouldn't get access to the test folder without the password, and probably end up with a 403 error or something like that. I have that, plus disallow / and never put the link to my test site anywhere a crawler can see it.
    mmoore5553 likes this.
  7. mmoore5553

    mmoore5553 Well-Known Member

    thanks i never thought of password protecting it. Will do that now.
  8. mrGTB

    mrGTB Well-Known Member

    It should be placed in ROOT directory (regardless), and if using a sub-folder for forum board you can add this to only cover that forum folder (only).

    Or to cover everything, just leave as you have it now already - uploaded into same ROOT area again.

    But as said above, only spiders that follow it will obey. Many don't and will ignore what's in your robots.txt file. If you want to use htaccess to password protect a folder, this site is very good for doing it easy. http://tools.dynamicdrive.com/password/

Share This Page