Issue with noindex meta tags being inconsistent with sitemap inclusion/exclusion options

[bzcomputers]

Affected version

2.2.15

There are inconsistency issues in XenForo setting noindex meta tags for pages that have sitemap inclusion/exclusion options.

Example 1 - Tags:
A. Turn tags off: tag page shows no permission notice, but it does not include the noindex meta tag (not expected for a non-content notice page)
B. Turn tags on: tag pages include the noindex meta tag (as expected)
C. Turn tags on and select them to be included in sitemap: tag pages still include the noindex meta tag (not expected)

Example 2 - Members:
A. Select members pages to be included in sitemap: member pages do not include the noindex meta tag (as expected)
B. Select members pages to be excluded from sitemap: member pages still do not include the noindex meta tag (not expected)

 

[Kirby]

A. Turn tags off: tag page shows no permission notice, but it does not include the noindex meta tag (not expected for a non-content notice page)

That no permission page has status code 403; it doesn't really need noindex as 403 pages are not indexed anyway.

C. Turn tags on and select them to be included in sitemap: tag pages still include the noindex meta tag (not expected)

Hmm ... doesn't seem to be the case (at least on xenforo.com):

sitemaps

xenforo.com

B. Select members pages to be excluded from sitemap: member pages still do not include the noindex meta tag (not expected)

Why should a page have noindex just because it is not included in the sitema?
It is perfectly fine for a page to be indexable without being included in the sitemap.
But pages that are not indexable should not be included in sitemaps.

 

[bzcomputers]

C. Turn tags on and select them to be included in sitemap: tag pages still include the noindex meta tag (not expected)

Hmm ... doesn't seem to be the case (at least on xenforo.com):

​

Appears it is just the main tag page: /community/tags/ that still contains the noindex tag

Why should a page have noindex just because it is not included in the sitema?
It is perfectly fine for a page to be indexable without being included in the sitemap.
But pages that are not indexable should not be included in sitemaps.

It is true, it is perfectly fine for a page to be indexable without being included in the sitemap. That being said, any admin that purposefully removes a page/group of pages from the sitemap does not typically want those pages to still be indexed.

 

[bzcomputers]

There has been conditional coding in the member_view template available now for at least 4 years that XenForo is still not using. This conditional, if satisfied, would allow for member pages to be set to "noindex". Which is what quite a few admins have been wanting to have any easy option to do for years. (see posts linked below)

This code exists in member_view:

<xf:if is="!$user.isSearchEngineIndexable()">
    <xf:head option="metaNoindex"><meta name="robots" content="noindex" /></xf:head>
</xf:if>

Seems the above conditional could either be linked to the User / member pages sitemap content option or in a separate option located near near the sitemap option.

It has definitely been wanted for years...

XF 2.2 Thread 'What affects the !$user.isSearchEngineIndexable() conditional?'

Dec 30, 2020

Hi,

I want to prevent that Google indexes member profiles and while looking at the member_view template, I noticed that there already is a conditional for this:

<xf:if is="!$user.isSearchEngineIndexable()">
    <xf:head option="metaNoindex"><meta name="robots" content="noindex" /></xf:head>
</xf:if>

However, I'm wondering, what is necessary to make this conditional "true"? I haven't found an option to not allow indexing of member profiles, so what exactly affects this conditional? Of course I can just get rid of the conditional statement and my issue is solved, but I would...

• Michael_IO
• Replies: 3
• Forum: XenForo questions and support

XF 2.2 Thread 'No index'

Dec 18, 2021

If I don’t want my member pages to be indexed, what would be the best way to do it?

1. There’s an option to uncheck/disable ‘User’ content to be included in the sitemap, in options. Not sure if this has anything to do with it.

2. There’s also a NoIndex code wrapped by a conditional in the member_view. Removing the condition could do it.

3. Use the robots.txt instead.

Out of the 3 methods, what would be best practice?

• JoyFreak
• Replies: 12
• Forum: XenForo questions and support

• 

XF 2.2 Thread 'How can I stop members from getting indexes?'

Feb 16, 2022

Users are not checked in sitemap options, the following commands are added in robots txt, but it still gets indexed.

Disallow: /kullanicilar
Disallow: /kullanicilar/

https://www.iyiforumlar.net/robots.txt

• Garfield™
• Replies: 4
• Forum: XenForo questions and support

Yes, it is a simple template edit, but if half the code already exists lets just use it. An easy update/fix.