• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

test site block all spiders

mmoore5553

Well-known member
#1
I forgot where the setting is to block all search engines and spiders . Let me know where it is . Also if i just make a robot.txt to disallow all is that the best way to not duplicate content.
 

mmoore5553

Well-known member
#3
I just did this in my robots.txt

User-agent: *
Disallow: /

is that all i need to do ? Which i put in my sub directory with my test board.
 

mmoore5553

Well-known member
#5
Are you talking about just password protecting the folder ? Would that be enough to block everything. I just do not want duplicate content.
 

MagnusB

Well-known member
#6
Yes, then they wouldn't get access to the test folder without the password, and probably end up with a 403 error or something like that. I have that, plus disallow / and never put the link to my test site anywhere a crawler can see it.
 

mrGTB

Well-known member
#8
I just did this in my robots.txt

User-agent: *
Disallow: /

is that all i need to do ? Which i put in my sub directory with my test board.

It should be placed in ROOT directory (regardless), and if using a sub-folder for forum board you can add this to only cover that forum folder (only).
User-agent: *
Disallow: /forum/

Or to cover everything, just leave as you have it now already - uploaded into same ROOT area again.
User-agent: *
Disallow: /

But as said above, only spiders that follow it will obey. Many don't and will ignore what's in your robots.txt file. If you want to use htaccess to password protect a folder, this site is very good for doing it easy. http://tools.dynamicdrive.com/password/