XF 1.4 XF & Google

[Cambion]

While I am rebranding my site(it still exists lol) under it's new name and work on the new design and what not a bit, I am wondering how to block Google and all search engines from Viewing and ranking said forum in their searches?

As it stands, it is member only right now at the moment and i prefer to keep it that way although it is already ranking and registering for the new name(it's a temporary name lol) faster than i expected.

So how do I block google/bing for the time being and when I am ready to be crawled, how do I remove it exactly?

Someone told me that making the forums themselves only visible by logged in Users was the key but i know for a fact there was a code or something you use to block bots from access/crawling your site.

 

[rdn]

Password protect the entire forum or use robots.txt to block all bots.

 

[Cambion]

Password protect the entire forum or use robots.txt to block all bots.

can you elaborate on both.

how do you password protect and will that allow my active users to still be on the site? What benefits does it have

and what exactly is robots.txt and how do I use it

 

[rdn]

If you're on Cpanel: http://www.inmotionhosting.com/supp...rotect-a-directory-in-my-control-panel-cpanel

Robots.txt http://www.robotstxt.org/faq/prevent.html

 

[TJA]

In robots.txt put in

User-agent: *
Disallow: /

 

[Cambion]

If you're on Cpanel: http://www.inmotionhosting.com/supp...rotect-a-directory-in-my-control-panel-cpanel

Robots.txt http://www.robotstxt.org/faq/prevent.html

In robots.txt put in

User-agent: *
Disallow: /

okay thank you both. It seems like robot.txt is the way to go then right? If I password protected that makes it where my active users have to remember a username and password to access the forum.

robot.txt just blocks google in general...but all active members can still freely visit my site.

I can just delete robots.txt when I want to be crawled right?

 

[rdn]

I can just delete robots.txt when I want to be crawled right?

Yes.

 

[TJA]

It seems like robot.txt is the way to go then right?

Yes

robot.txt just blocks google in general...but all active members can still freely visit my site.

No, what I gave you blocks ALL robots crawling your site.

I can just delete robots.txt when I want to be crawled right?

You can but wouldn't recommend it, you will need to adjust it to disallow some robots like Baiduspider for example, if you do a search on here there's plenty of threads with recommendations on what to put in your robots.txt file.

 

[Cambion]

Yes.

Yes


No, what I gave you blocks ALL robots crawling your site.


You can but wouldn't recommend it, you will need to adjust it to disallow some robots like Baiduspider for example, if you do a search on here there's plenty of threads with recommendations on what to put in your robots.txt file.

While it is on my mind, I don't have to go through EACH forum and do the user permissions do I? LOL. I have like 80 some forums

Or can i go into ACP under Unregistered and remove the permission to view node?

I guess my question is where do I make it member only the individual nodes by making them Private or do I do it through usergroup permissions and just mess with the unregistered group

 

[Mike]

If you're removing all access for guests, just remove the general "view" permission.

 

[Cambion]

If you're removing all access for guests, just remove the general "view" permission.

Ah thanks mike!