XF 2.1 Google Cache

[stub]

When I did a site:domain.com for my domain on google, I was kinds shocked how much stuff was cached, which I'd like not to be cached. Here is what I found..

index.php (well ok)
forums/ (well ok)

but how do I make all these non- searchable

members/
categories/
help/cookies/
help/terms/
misc/contacts/
whats-new/
help/privacy-policy/
members/names/
members/list/

and I'm presuming there might be others too.

What is the best method to prevent google for searching all these folders?

 

[djbaxter]

Create a robots.txt file:

User-agent: *
Disallow: /members/
Disallow: /categories/
Disallow: /help/cookies/
Disallow: /help/terms/
Disallow: /whats-new/
Disallow: /help/privacy-policy/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Allow: /

Upload this to your forum root.

 

[Paul B]

A combination of permissions for the unregistered group and robots.txt.

 

[stub]

Create a robots.txt file:

User-agent: *
Disallow: /members/
Disallow: /categories/
Disallow: /help/cookies/
Disallow: /help/terms/
Disallow: /whats-new/
Disallow: /help/privacy-policy/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Allow: /

Upload this to your forum root.

OK. Done that. Is this a comprehensive list, or does this cover only what I mentioned and there are others which should be included. I noticed you combined my /members/ folders. I presume I can do the same for /help/

This was only after my board had been open for less than a week with only a few admin posts. Which is why I speculated there might be others?

 

[stub]

A combination of permissions for the unregistered group and robots.txt.

Would you have anything else I might need to add to the robots.txt file? What are you suggesting needs to be changed from the default unregistered settings. All I want is that unregistered users can read the forums. Will that cause search engines to go crazy too?

 

[djbaxter]

OK. Done that. Is this a comprehensive list, or does this cover only what I mentioned and there are others which should be included. I noticed you combined my /members/ folders. I presume I can do the same for /help/

I think it applies to subfolders, yes. Someone will no doubt correct me if that's incorrect.

This is what https://xenforo.com uses:

User-agent: *
Disallow: /community/whats-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

Sitemap: https://xenforo.com/community/sitemap.xml

 

[eL_]

Create a robots.txt file:

User-agent: *
Disallow: /members/
Disallow: /categories/
Disallow: /help/cookies/
Disallow: /help/terms/
Disallow: /whats-new/
Disallow: /help/privacy-policy/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php
Allow: /

Upload this to your forum root.

@djbaxter set that on my root page folder, any setting after that?

 

[djbaxter]

Assuming your forum is set up in the root, that should be okay.

See

Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers

Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.

support.google.com

and from that source see also

robots.txt report - Search Console Help

See whether Google can process your robots.txt filesThe robots.txt report shows which robots.txt files Google found for the top 20 hosts on your site, the last time they were crawled, and any warnings

support.google.com

 

[eL_]

@djbaxter thank you.

 

[eL_]

@djbaxter is this correct?

​

 

[bzcomputers]

@djbaxter is this correct?

View attachment 218101​

That would be incorrect. You need to remove the last line "Allow: /".



Having that line at the bottom is unneeded and could potentially override all the Disallow lines above it depending on the search engine/bot.

 

[Astra]

Why close forum posts from search robots?
Disallow: /posts/