test site block all spiders

[mmoore5553]

I forgot where the setting is to block all search engines and spiders . Let me know where it is . Also if i just make a robot.txt to disallow all is that the best way to not duplicate content.

 

[mmoore5553]

I feel like a newbie lol

 

[mmoore5553]

I just did this in my robots.txt

User-agent: *
Disallow: /

is that all i need to do ? Which i put in my sub directory with my test board.

 

[MagnusB]

Yes, that will stop all robots that respects robots.txt. Your best way to stop it from being indexed though is to prevent the link from being found, or protect the directory in some manner.

http://www.robotstxt.org/faq/prevent.html

 

[mmoore5553]

Are you talking about just password protecting the folder ? Would that be enough to block everything. I just do not want duplicate content.

 

[MagnusB]

Yes, then they wouldn't get access to the test folder without the password, and probably end up with a 403 error or something like that. I have that, plus disallow / and never put the link to my test site anywhere a crawler can see it.

 

[mmoore5553]

thanks i never thought of password protecting it. Will do that now.

 

[mrGTB]

I just did this in my robots.txt

User-agent: *
Disallow: /

is that all i need to do ? Which i put in my sub directory with my test board.

It should be placed in ROOT directory (regardless), and if using a sub-folder for forum board you can add this to only cover that forum folder (only).

User-agent: *
Disallow: /forum/

Or to cover everything, just leave as you have it now already - uploaded into same ROOT area again.

User-agent: *
Disallow: /

But as said above, only spiders that follow it will obey. Many don't and will ignore what's in your robots.txt file. If you want to use htaccess to password protect a folder, this site is very good for doing it easy. http://tools.dynamicdrive.com/password/