So far so good - numbers of indexed pages are steadily coming up.
It does seem to only affect multi-page sitemaps though as my other, single page (less than 50,000 URLs) sites seem to work okay.
Specifically Google doesn't seem to be loading / using the split-up maps referred to in the parent sitemap.php for some reason.
Submit them individually by directly entering their URL and they display fine and GSC shows number of crawled pages / images etc. but submit the parent and the child maps are perpetually pending.
Since it is affecting quite a number of us (from the replies in this thread) are there any good/free sitemap add-ons that we could possibly use in the meantime until the issue with the built-in maps is resolved? (Not suggesting it's an XF issue, Google is clearly not indexing them properly at the mo so it may be fixed at some future point).
If anything, it would appear to potentially be a Google change that affects us. The sitemap output is specified and we follow the spec. It could be query string related, though we'll have to see what options we have to resolve that (that work in all cases).
In the past couple of weeks, since removing the sitemap from GSC, I've seen a steady daily increase in the number of indexed pages, with today's being around seven thousand more than when I removed it.
It doesn't correlate with my XF upgrade and sitemap problem timeline either: Skipped XF 1.5.14 (so didn't have the noindex tag); queued a card in Trello for upgrade on 26 Sep (sitemap issue started on 21 Sep); upgraded to 1.5.15a on 6 Oct (where the noindex will have been added).
The issue is definitively the query string for individual indexes. It's not totally clear why. Perhaps Google is just ignoring sitemaps with a query string or they're ignoring the query string and thus assuming that sitemap.php?c=3 and sitemap.php are the same thing (the former is a sitemap, the latter a sitemap index).
I have a change for XF2 right now that changes the individual sitemaps to be sitemap-3.xml (or similar). This depends on friendly URLs being enabled; if they're not enabled, we can't guarantee that rewrites are supported and thus this is difficult. I'll look into the viability of backporting this to XF1.
If friendly URLs aren't enabled, the situation is difficult. We have no guaranteed ability to write files to the root directory and we can't serve the sitemap from a sub-directory as the specification is explicit as to what URLs are allowed; if you have example.com/dir/sitemap.xml, your urls can only relate to example.com/dir/. This, if we wrote it to something like /community/data/, we couldn't actually include URLs from outside that based on the spec. (Beyond that, there's no guarantee that the data output is on the same domain as the rest of the site, which is another problem.)
Thus, I'm not sure if there's a clear solution for that situation yet.