XF 2.2 Does xenforo add 'X-Robots-Tag' noindex http header to sitemap files in certain cases?

[dethfire]

I'm trying to figure out why half my sitemap files have a 'X-Robots-Tag' noindex http header. Does xenforo have some functionality that controls this? I don't have anything in my cloudflare rules, no addons, nothing in templates, nothing in htaccess. The first 9 sitemaps are fine, anything 10 and above are xrobot noindex.

 

[MattW]

# grep -iR X-Robots *
XF/Sitemap/Renderer.php:                $response->header('X-Robots-Tag', 'noindex');

 

[dethfire]

huh, I see it in the file now. Looks like it's in the stock file too. Is it controlled by the indexing rules?

 

[Jeremy P]

It's set for every sitemap. It has no bearing on whether Google can crawl it, it just prevents it from being listed in search results (though Google is generally smart enough not to index them by itself).

 

[dethfire]

yep, just trying to figure out why Google won't successfully fetch any sitemaps beyond 9. It's driving me crazy.

 

[James]

yep, just trying to figure out why Google won't successfully fetch any sitemaps beyond 9. It's driving me crazy.

I've actually seen a htaccess fix for this on the forum. Let me find it.

Here you go.

XF 2.2 Post in thread 'GSC only reads in the first 9 sitemap files...'

Jan 17, 2023

Thanks Chris - I did have a look at existing but couldn't see anything really similar.

As a test, I've submitted the missing sitemaps manually - I'll give it a day or so for them to be accepted (or not) and report back

Ok reporting back.

Manually submitting them as single sitemaps didn't work either - Google just gave the same don't fetch error.

However, I was pondering this again today and wondered if it really could be just the fact the number has 2 digits.

So easy test, convert everything over 9 into hex (10 = A, 11 = B, 12 = C etc) and submitted again - low and behold it...

• benFF

• 
•