• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Partial fix Google search console: sitemap pending due to query string param

CTXMedia

Formerly CyclingTribe
#43
I've deleted my sitemap.php from GSC for now because it is perpetually pending; I'll keep an eye on indexed pages and see if there is a positive change or not in the coming weeks.
So far so good - numbers of indexed pages are steadily coming up.

It does seem to only affect multi-page sitemaps though as my other, single page (less than 50,000 URLs) sites seem to work okay.

Specifically Google doesn't seem to be loading / using the split-up maps referred to in the parent sitemap.php for some reason.

Submit them individually by directly entering their URL and they display fine and GSC shows number of crawled pages / images etc. but submit the parent and the child maps are perpetually pending.

Since it is affecting quite a number of us (from the replies in this thread) are there any good/free sitemap add-ons that we could possibly use in the meantime until the issue with the built-in maps is resolved? (Not suggesting it's an XF issue, Google is clearly not indexing them properly at the mo so it may be fixed at some future point).
 
Last edited:

Mike

XenForo developer
Staff member
#46
If anything, it would appear to potentially be a Google change that affects us. The sitemap output is specified and we follow the spec. It could be query string related, though we'll have to see what options we have to resolve that (that work in all cases).
 

CTXMedia

Formerly CyclingTribe
#50
I've deleted my sitemap.php from GSC for now because it is perpetually pending; I'll keep an eye on indexed pages and see if there is a positive change or not in the coming weeks.
In the past couple of weeks, since removing the sitemap from GSC, I've seen a steady daily increase in the number of indexed pages, with today's being around seven thousand more than when I removed it.
 

dutchbb

Well-known member
#51
Same here, had a serious decrease in pages crawled and now going up same as pages indexed. Was searching for weeks what i could be. So a 'bad' sitemap is negative for crawling and indexing?

Strange thing is i can't find anything wrong with the sitemap and also no changes on google's end.
 
Last edited:

Mike

XenForo developer
Staff member
#55
Adding that is a well-documented and recommended practice. It's done by common frameworks as well as XF.
 

CTXMedia

Formerly CyclingTribe
#56
That's what I meant.
It doesn't correlate with my XF upgrade and sitemap problem timeline either: Skipped XF 1.5.14 (so didn't have the noindex tag); queued a card in Trello for upgrade on 26 Sep (sitemap issue started on 21 Sep); upgraded to 1.5.15a on 6 Oct (where the noindex will have been added).

So no, it doesn't seem to be related.
 

Mike

XenForo developer
Staff member
#58
The issue is definitively the query string for individual indexes. It's not totally clear why. Perhaps Google is just ignoring sitemaps with a query string or they're ignoring the query string and thus assuming that sitemap.php?c=3 and sitemap.php are the same thing (the former is a sitemap, the latter a sitemap index).

I have a change for XF2 right now that changes the individual sitemaps to be sitemap-3.xml (or similar). This depends on friendly URLs being enabled; if they're not enabled, we can't guarantee that rewrites are supported and thus this is difficult. I'll look into the viability of backporting this to XF1.

If friendly URLs aren't enabled, the situation is difficult. We have no guaranteed ability to write files to the root directory and we can't serve the sitemap from a sub-directory as the specification is explicit as to what URLs are allowed; if you have example.com/dir/sitemap.xml, your urls can only relate to example.com/dir/. This, if we wrote it to something like /community/data/, we couldn't actually include URLs from outside that based on the spec. (Beyond that, there's no guarantee that the data output is on the same domain as the rest of the site, which is another problem.)

Thus, I'm not sure if there's a clear solution for that situation yet.
 
Last edited:

Mike

XenForo developer
Staff member
#59
I'm attaching a unified diff for XF1 if you'd like to apply the equivalent fix there to confirm that this resolves the problem for other people.
 

Attachments