1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.4 XF & Google

Discussion in 'XenForo Questions and Support' started by Cambion, Sep 13, 2014.

  1. Cambion

    Cambion Well-Known Member

    While I am rebranding my site(it still exists lol) under it's new name and work on the new design and what not a bit, I am wondering how to block Google and all search engines from Viewing and ranking said forum in their searches?

    As it stands, it is member only right now at the moment and i prefer to keep it that way although it is already ranking and registering for the new name(it's a temporary name lol) faster than i expected.

    So how do I block google/bing for the time being and when I am ready to be crawled, how do I remove it exactly?

    Someone told me that making the forums themselves only visible by logged in Users was the key but i know for a fact there was a code or something you use to block bots from access/crawling your site.
     
  2. RoldanLT

    RoldanLT Well-Known Member

    Password protect the entire forum or use robots.txt to block all bots.
     
  3. Cambion

    Cambion Well-Known Member

    can you elaborate on both.

    how do you password protect and will that allow my active users to still be on the site? What benefits does it have

    and what exactly is robots.txt and how do I use it
     
  4. RoldanLT

    RoldanLT Well-Known Member

  5. TJA

    TJA Well-Known Member

    In robots.txt put in

    User-agent: *
    Disallow: /
     
    Cambion likes this.
  6. Cambion

    Cambion Well-Known Member

    okay thank you both. It seems like robot.txt is the way to go then right? If I password protected that makes it where my active users have to remember a username and password to access the forum.

    robot.txt just blocks google in general...but all active members can still freely visit my site.

    I can just delete robots.txt when I want to be crawled right?
     
  7. RoldanLT

    RoldanLT Well-Known Member

    Yes.
     
  8. TJA

    TJA Well-Known Member

    Yes

    No, what I gave you blocks ALL robots crawling your site.

    You can but wouldn't recommend it, you will need to adjust it to disallow some robots like Baiduspider for example, if you do a search on here there's plenty of threads with recommendations on what to put in your robots.txt file.
     
  9. Cambion

    Cambion Well-Known Member

    While it is on my mind, I don't have to go through EACH forum and do the user permissions do I? LOL. I have like 80 some forums :p

    Or can i go into ACP under Unregistered and remove the permission to view node?

    I guess my question is where do I make it member only the individual nodes by making them Private or do I do it through usergroup permissions and just mess with the unregistered group
     
  10. Mike

    Mike XenForo Developer Staff Member

    If you're removing all access for guests, just remove the general "view" permission.
     
    optrex likes this.
  11. Cambion

    Cambion Well-Known Member

    Ah thanks mike!
     

Share This Page