Whats your robots.txt file look like? - Xenforo

[Pavle123]

If you noindex you must also remove the canonical tag

Hi @dethfire do you mind clarifying this a bit?
I have seen that on your great forum, you have noindex to members pages, can you please tell me how you did it? I am trying to do the same. I have also seen that you do not disallow members in your robots.txt, which is something I am trying to achieve. Thanks in advance.

 

[Sap1ent]

As my URLs have a slightly different structure to most here (i.e. http://www.myforum.com/index.php?threads/whats-happened-at-this-community-school.5215/) probably because I imported from phpbb.

In this case is it just a case of adding lines like this to the robots txt instead:

Disallow: /index.php?attachments/

I don't want to mess up and stop google crawling my site!

 

[rdn]

As my URLs have a slightly different structure to most here (i.e. http://www.myforum.com/index.php?threads/whats-happened-at-this-community-school.5215/) probably because I imported from phpbb.

No, just enable Use Full Friendly URLs on admin option

If you enable this option, the links generated by the system will not include "index.php?". However, to enable this, mod_rewrite must be available and an appropriate .htaccess file must be in place.

/admin.php?options/list/seo

 

[Sap1ent]

Thanks for taking the time out to answer, but i'm a bit scared by that solution as i've only just managed to get my site fully working again by tweaking the .htaccess file!

I don't think it will cause a vast amount of problems for me...but it could!

 

[Sap1ent]

No, just enable Use Full Friendly URLs on admin option

/admin.php?options/list/seo

I bit the bullet and it has worked a treat, thanks a lot!

 

[dethfire]

How many pages are being blocked by your robots? Google WMT says I have near a million blocked pages with this:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /admin.php

Sitemap: https://www.physicsforums.com/sitemap.php

 

[Robust]

User-agent: *
Disallow: /account/
Disallow: /admin.php
Disallow: /articles/settings
Disallow: /attachments/
Disallow: /conversations/
Disallow: /find-new/
Disallow: /goto/
Disallow: /login/
Disallow: /lost-password/
Disallow: /members/*/trophies
Disallow: /members/*/recent-content
Disallow: /misc/style
Disallow: /posts/
Disallow: /register/
Disallow: /online/
Disallow: /search/

Sitemap: https://www.gamingforums.net/sitemap.php

Most recently.

 

[dethfire]

If there is no robots file with install, does XF recommend not using one?

 

[Paul B]

It's entirely up to you whether you use one or not and how you define it.
It's not specifically related to your XenForo installation, it is applied at the root, which may already have one.

 

[Mr Lucky]

If there is no robots file with install, does XF recommend not using one?

As Brogan said, it is not necessary fro xenforo to work. But it can be useful for SEO as (theoretically) it would stop Google from indexing unnecessary pages that are either not useful in searches or could possibly (according to some experts) even be harmful for SEO.

 

[usAdultAds]

As Brogan said, it is not necessary fro xenforo to work. But it can be useful for SEO as (theoretically) it would stop Google from indexing unnecessary pages that are either not useful in searches or could possibly (according to some experts) even be harmful for SEO.

I want to say it was one person on this form that has 1+/m posts, and he said he does not use robots.txt...
I would not go so far to say it will be a negative impact; The impact of less traffic from blocked pages...
Can you call that a negative impact? I have never had a site banned for having low quality pages...
some pages may not get indexed for obvious reasons, other then that, just dont expect to get higher
rankings that you may get from other indexed pages. Even low quality pages could bring in some traffic,
but you will certainly kill that off by blocking everthing from here to there. After all, if a million post
forum has no need for a robots.txt then that is good enough for me. Your robots file will not make your
site a success or failure, as your robots file does not determine your popularity....

The same can be said for VB SEO, everyone rushed out to get it because they felt it was a must have, and
if they did not get the addon, then they were doomed for failure. VB SEO did not determine their fate...
It was the forum owner that failed, having or not having vb seo had nothing to do with it, over the years
i have run across many successful forums that did not use any type of addons for seo urls.....

 

[Mr Lucky]

I
I would not go so far to say it will be a negative impact; The impact of less traffic from blocked pages...
Can you call that a negative impact? I have never had a site banned for having low quality pages...

I'm not sure about sites being "banned," but they can certainly get penalised. I'm not quite clear on whether Google panda penalised just pages or a whole site, I think that isopen to debate.
But when you see things like this: Thin content with little or no added value - Search Console Help I believe there is no harm in erring on the cautious side.

My site was hit badly by panda a few years back. By dealing with thin content by improving the content and by and reviewing robots.txt I doubled the visitors within a few months and got the site back to where it was.

Could be coincidence, or maybe not.

The thing is it makes no sense to say there is a site with 1 million posts but no robots.txt and use that as a reason for not needing robots.txt on any site. Who knows, if they had a well optimised robots.txt then maybe they'd havea lot more than 1 million posts.

 

[Nuno]

Hi

What about /members?
Mines is disallowed because I have a great amount of warnings in m search console members profiles.

 

[DieselMinded]

whats still applicable? Host just recommend i add

User-agent: *
Crawl-delay: 5

to my robot.txt

to help with my site speed

 

[Diana]

I want to be able to block search engines from crawling the forum to assure my members their privacy. I don't care about SEO as membership is by word-of-mouth. Will listing them in robot.txt do it?

 

[Paul B]

Not all search engines respect the robots.txt file.

Remove view permission from guests if you want to prevent non-logged in visitors from accessing content.

 

[whynot]

I want to be able to block search engines from crawling the forum to assure my members their privacy....... Will listing them in robot.txt do it?

No, it won't. Some of them will respect it.
Make your nods invisible to visitors, search engines cannot acess them.

 

[Diana]

Not all search engines respect the robots.txt file.

Remove view permission from guests if you want to prevent non-logged in visitors from accessing content.

I've removed viewing permissions for guests. I even deactivated the forum to see what happens. Google bots are still able to access threads (though probably only the headings and not the content as it shows an "error" when I deactivate).

 

[Diana]

No, it won't. Some of them will respect it.
Make your nods invisible to visitors, search engines cannot acess them.

But it looks like they still are. Mostly, google accesses the calendar, but once in a while they still access a thread.

 

[whynot]

Is Google treated as a member of your forum because of some addon?

Are you using AdSense?