XF 2.0 Noindex members pages

cdub

Well-known member
Has anyone found the best way to do this yet? Noindex on member pages? I have it set that guests can't see member profiles so of course I'm getting a ton of 403 forbidden crawl errors.
 

arn

Well-known member
Has anyone found the best way to do this yet? Noindex on member pages? I have it set that guests can't see member profiles so of course I'm getting a ton of 403 forbidden crawl errors.

if you have guests unable to see the members pages, you don't need the NoIndex directive... because Google will never see it.

arn
 

cdub

Well-known member
if you have guests unable to see the members pages, you don't need the NoIndex directive... because Google will never see it.

arn
I'm getting alot of crawl errors. I know Google won't see it... I want it to stop trying to and telling me there's something wrong with the site. Maybe it's not a real problem though.
 

arn

Well-known member
I'm getting alot of crawl errors. I know Google won't see it... I want it to stop trying to and telling me there's something wrong with the site. Maybe it's not a real problem though.

Yeah, I hear you. Did you make sure to turn off user pages in the Sitemap settings? So you're not telling Google to look at them.

The only other way to prevent Google from crawling them would b

1. Robots.txt blocking /members/ -- but then Google will complain that they are robots blocked
2. No-Follow all links to /members/ -- controversial choice. But would presumably hint to google not to follow. Google will prob still complain.

I do think Google eventually "gives up" on the 403 pages, so should be a relatively small amount of your crawl budget. You can grep your server logs to see how much time Google is actually spending on your /members/ pages. it might make you feel better.

arn
 

jgaulard

Active member
Per Google this is not the preferred method.

From my experience, these "Indexed, though blocked by robots.txt" errors inside of Google Search Console come about six months late. They're generally warnings that deal with pages that once weren't blocked by robots.txt and were indexed by Google at some point and then became blocked later on. I really don't pay attention to those particular warnings.
 

jgaulard

Active member
Yeah, I hear you. Did you make sure to turn off user pages in the Sitemap settings? So you're not telling Google to look at them.

The only other way to prevent Google from crawling them would b

1. Robots.txt blocking /members/ -- but then Google will complain that they are robots blocked
2. No-Follow all links to /members/ -- controversial choice. But would presumably hint to google not to follow. Google will prob still complain.

I do think Google eventually "gives up" on the 403 pages, so should be a relatively small amount of your crawl budget. You can grep your server logs to see how much time Google is actually spending on your /members/ pages. it might make you feel better.

The worst part is that Google will complain about the pages being blocked by robots.txt, the noindex pages and the 403 pages. I've been watching these graphs go up and down in Google Search Console and I haven't quite figured out which is the lesser of all three evils yet. From what I gather, I think Googlebot will eventually give up on all three. The problem is, giving up takes a very long time sometimes.
 

Chromaniac

Well-known member
ok. this does not seem to be mentioned in this thread. but it has been mentioned several times in other threads. so just posting this in case it helps someone.

<xf:head option="metaNoindex"><meta name="robots" content="noindex" /></xf:head>

this is the magical code that can be added to individual templates where you want to include the noindex meta tag.

for member pages... this can be added to these:
  • member_view
  • member_about
  • member_latest_activity
  • member_recent_content

From the looks of it, XenForo already has this code in the last two templates so only the first two needs to be edited!
 
Last edited:
Top